SCC/SCS Shell And Job Scripting

Motivation

  • Why should one learn the shell?
    • Advantages
    • Limitations

Intermediate

  • Job Scripting and best practices
    • What is a Bash script?
      • Where do scripts come from?
      • Structure of a Bash script: best practices
      • Adapt it to for use with sbatch
  • Workflow discussion
  • Composability
    • Streams and redirections
      • Clarification: stdin vs command line arguments
    • Pipes
    • Variables
    • Yet another way to reuse the output of a command
  • Dealing with repetition
    • Iterating on multiple files/directories with for loops
    • Non integer parameter scans with seq
  • Simple Parallelism patterns in the shell
  • Tools for embarrassingly Parallel Workloads
    • Parbatch
    • Similar Projects
  • SLURM Workflows
    • Repeating jobs with job arrays
    • Workflow managers
  • Tips and Tricks for interactive use
  • SSH ergonomics
    • The .ssh/config file
    • SSH keys
      • Generating a ssh keypair and registering it
    • ssh-agent
    • Security on HPC is important!

Advanced

  • Examples of job scripts
    • Dealing with walltime limits on your own terms
    • Job chaining using recursive sbatch
    • Setting dependencies between jobs using Slurm
    • An excerpt of a job using a local/private FileSystem
  • Understanding the Bash manual
  • Control flow in Bash: if, while and traps
    • Control flow statements
    • if-else statements
    • &&, || and !
    • While loops
    • Dealing with signals: traps
      • When is a trap executed?
  • More Bash features and their composability
    • More streams and redirections
      • Use a file as stdin for a process: <
      • stderr into stdout or vice versa
      • Process substitution: <()
      • Compound commands
    • Subshells and export
    • Bash functions
    • Arrays
    • Dealing with repetition: pipe into xargs instead of for
  • Defensive programming and debugging
    • Debug and error checking
      • Fail on error: set -e
      • Error when using an undefined variable: set -u
      • Debugging: set -x
    • Automatic check for common bad practices and antipatterns
  • Tips and Tricks for interactive use
    • Multitasking et similia
      • Using multiple applications at the same time
      • Alternative: tmux
  • Unix tools for manipulation, exploration and monitoring of text files
    • “Vertical” text manipulation
    • Horizontal Manipulations
    • When cut does not cut it: awk
    • Bulk text manipulation with sed
    • Following a running program
      • The tee command
      • The tail -f command

Extra

  • Instructor notes
    • Rationale
    • Known issues
    • Schedule
SCC/SCS Shell And Job Scripting
  • Search

© Copyright .

Built with Sphinx using a theme provided by Read the Docs.