06: High-Performance Computing (HPC)

Although today’s handy laptops perform many advanced and computationally intensive tasks, projects involving Big Data require significantly more resources. That need is satisfied by the HPC infrastructure, built from a network of computing clusters combined with immense memory. Access to these resources is remote, so job submission and data preview occurs through an interface on any local computing machine from any (allowed) geolocation. The HPC infrastructure is a shared community space, so you might want to familiarize yourself with the usage policy to avoid disrupting peer work.

Table of contents

1. Introduction to HPC infrastructure

• 1.1 ACCESS Program
  • 1.1.1 XSEDE Supercell Storage
• 1.2 SCINet Network
  • 1.2.1 Atlas Computing Cluster
  • 1.2.2 Ceres Computing Cluster
  • 1.2.3 Juno Storage
• 1.3 ISU HPC
  • 1.3.1 Condo Computing Cluster
  • 1.3.2 Nova Computing Cluster
  • 1.3.3 LSS Storage

2. Remote Access to HPC Resources

• 2.1 Virtual Private Network (VPN) Connection
• 2.2 Secure Shell Connection (SSH)
  • 2.2.1 SSH shortcuts and password-less login
• 2.3 Open On Demand (OOD) Connection

(see more in section 7: Data Acquisition: Remote Data Access ⤴)

3. Setting up Your Home Directory for Data Analysis

• 3.1 .bashrc example file

4. Software Available on HPC

• 4.1 Accessing pre-Installed Modules
• 4.2 Accessing Software via Package Manager
• 4.3 Installing Custom Programs in User Space

5. Introduction to Job Scheduling

• 5.1 SLURM: Basics of Workload Manager
  • 5.1.1 Introduction to SLURM
  • 5.1.2 Creating SLURM Job Submission Scripts
  • 5.1.3 Submitting Dependency Jobs using SLURM
  • SLURM Commands CheatSheet
• 5.2 PBS: Portable Batch System
  • 5.2.1 Creating PBS Job Submission Scripts
  • 5.2.2 Submitting Dependency Jobs using PBS
  • PBS Commands CheatSheet

6. Introduction to GNU Parallel

7. Introduction to Containers

• 7.1 Apptainer
  • 7.1 Singularity (archival)
  • 7.1.1 Creating Containers using Singularity
  • 7.1.2 Modifying Existing Containers
  • 7.1.3 Singularity on your Mac via Vagrant
• 7.2 Docker

---

Homepage Prior Section Next Section