SDSC’s new supercomputer, Expanse, went into full production on December 7, 2020 as a follow-on to Comet. Although the transition should be straightforward for most users, there are some important differences in both the hardware and software between these two systems. We will start with an overview of the Expanse architecture, which is based on AMD’s 64-core EPYC Rome processor and NVIDIA’s V100 GPU device. Moving from Intel to AMD processors introduces a new set of compilers, math libraries and tools. We’ll explain how to use the AMD software stack to build and run applications for optimal performance. GPU applications will port easily to Expanse and the upgrade from Comet’s P100s to the newer V100s generally results in significantly reduced run times. With regards to job submission, we continue to use the Slurm workload manager. Partition names are left unchanged, but job scripts will have to be updated to reflect the larger core counts and, for highly scalable applications, maximum job size. We also introduce a new charging model that fairly takes into account the usage of all resources (memory, CPU, GPU). The tutorial concludes with a discussion of interactive computing using the Expanse Portal and efficiently moving data from Comet to Expanse.
Overview/User Environment- 9:00 AM - 9:30 AM - Overview of System and Allocations
- Bob Sinkovits, Director for Scientific Computing Applications
- 9:30 AM - 10:15 AM - Slurm differences, Running Jobs
- Mahidhar Tatineni, Director of User Services
- 10:15 AM - 10:25 AM - 10-minute break
- 10:25 AM – 11:25 AM – Software module environment, basic compilation and optimization
- Marty Kandes, Computational and Data Science Research Specialist
- 11:25 AM - 11:35 AM - 10-minute break
- 11:35 AM - 11:45 AM - Job Charging, TRES
- Nicole Wolter, Computational and Data Science Research Specialist
- 11:45 AM - 12:00 PM - Using the Expanse portal
- Subhashini Sivagnanam, Senior Computational and Data Science Specialist
- 12:00 PM - 12:15 PM - Interactive computing and running Jupyter Notebooks
- Mary Thomas, Computational Data Scientist
- 12:15 PM - 12:30 PM - Data Transfer Mechanisms
- Nicole Wolter, Computational and Data Science Research Specialist
- 12:30 PM – 1:00 PM - Q&A
