High-Performance Computing (HPC)

Although today’s handy laptops perform many advanced and computationally intensive tasks, projects involving Big Data require significantly more resources. That need is satisfied by the HPC infrastructure, built from a network of computing clusters combined with immense memory. Access to these resources is remote, so job submission and data preview occurs through an interface on any local computing machine from any (allowed) geolocation. The HPC infrastructure is a shared community space, so you might want to familiarize yourself with the usage policy to avoid disrupting peer work.

Table of contents

1. Setting up your home directory for data analysis

• .bashrc example file

2. Introduction to HPC infrastructure

• 2.1 XSEDE Supercomputer
  • XSEDE Supercell Storage
• 2.2 SCINet Network
  • Atlas Computing Cluster
  • Ceres Computing Cluster
  • Juno Storage
• 2.3 ISU HPC
  • Condo Computing Cluster
  • Nova Computing Cluster
  • LSS Storage

3. Secure Shell Connection (SSH)

• 3.1 SSH Shortcuts
• 3.2 Password-less SSH

4. Remote Data Access
(see more in section 7: Data Acquisition)

5. Available Software

• 5.1 Accessing pre-Installed Modules
• 5.2 Installing Custom Programs in User Space

6. Introduction to Job Scheduling

• 6.1 SLURM
  • SLURM Cheatsheet
  • Tutorial: Creating SLURM Job Submission Scripts
  • Tutorial: Submitting Dependency Jobs using SLURM
• 6.2 PBS
  • PBS Cheatsheet
  • Tutorial: Creating PBS Job Submission Scripts
  • Tutorial: Submitting Dependency Jobs using PBS

7. Introduction to GNU Parallel

8. Introduction to Containers

• 8.1 Singularity
  • Tutorial: Creating Containers using Singularity
  • Tutorial: Modifying Existing Containers
  • Tutorial: Singularity on your Mac via Vagrant
• 8.2 Docker

---

Homepage Prior Section Next Section