Shell and Job Scripting course notes

This course is intended to teach how to use the bash shell efficiently to researchers using a HPC cluster, with a goal-oriented approach (backward lesson design, popularized by Greg Wilson’s Teaching Tech Together and references therein).

Teaching style: Type along

Some of the content in this lesson is a collection of notions, but the main style of this lesson is practical and following along, trying to type the commands while the instructor does it.

The instructor will follow these notes, so that if you get lost you can get back to it.

Objectives

1. Ease the use of the shell and the terminal in general

2. Stimulate discussion with the audience to collect use cases

3. Show useful patterns to carry out common tasks in (slurm) job scripts

Suggestions welcome!

Suggestions/criticism are welcome, so feel free to open issues and pull requests to this repository (see “Edit on GitHub” icon in the upper right corner).

Setup to follow along and for the exercises

You can clone this repository

git clone https://github.com/mmesiti/shell-job-scripting-scc.git 

and use the examples directory to follow along.

The job scripting exercises and their solutions are in the exercises directory.

Sources and Inspiration

• The already existing slides and the exercises of the HPC-DIC courses by SCC/SCS at KIT;

• The “Shell scripts and tips” course by NRIS;

• The linux shell tutorial by Aalto Science IT department (very detailed course, goes deeper than what we are trying to do here);

• The software carpentry Shell Novice Lesson;

• The Extra Unix Shell Material; from the Carpentries Incubator;

• The Bash manual.

Table of Content

Motivation

• Why should one learn the shell?
  • Advantages
  • Limitations

Intermediate

• Job Scripting and best practices
  • What is a Bash script?
    • Where do scripts come from?
    • Structure of a Bash script: best practices
    • Adapt it to for use with sbatch
• Dealing with repetition
  • Iterating on multiple files/directories with for loops
  • xargs instead of for
  • Non integer parameter scans with seq
• Composability
  • Streams and redirections
    • Clarification: stdin vs command arguments
  • Pipes
  • Variables
  • Subshells and export
• SSH ergonomics
  • SSH keys
    • Generating a ssh keypair and registering it
  • The .ssh/config file
• Tips and Tricks for interactive use
• Workflow discussion

Advanced

• Understanding the Bash manual
• More Bash features and their composability
  • More streams and redirections
    • Use a file as stdin for a process: <
    • stderr into stdout or vice versa
    • Process substitution: <()
    • Compound commands
  • Bash functions
  • Arrays
• Tips and Tricks for interactive use
  • Multitasking et similia
    • Using multiple applications at the same time
    • Alternative: tmux
• Defensive programming and debugging
  • Debug and error checking
    • Debugging: set -x
    • Error when using an unset variable: set -u
    • Fail on error: set -e
  • Automatic check for common problems
• Unix tools for text file manipulation and exploration
  • “Vertical” text manipulation
  • Horizontal Manipulations
  • When cut does not cut it: awk
  • Bulk text manipulation with sed
  • Following a running program
    • The tee command
    • The tail -f command
• SLURM Workflows
  • Repeating jobs with job arrays
  • Job chaining using SLURM dependencies
  • Job chaining by recursive sbatch invocations
  • Workflow managers
• Simple Parallelism patterns in the shell
• Tools for embarrassingly Parallel Workloads
  • Parbatch
  • Similar Projects

Extra

• Jobscript Examples
  • Dealing with walltime limits on your own terms
  • Using a local/private FileSystem
  • Job chaining
    • Using recursive sbatch
    • Using Slurm dependencies
• Instructor notes
  • Rationale
  • Known issues
  • Schedule