SCC/SCS Shell And Job Scripting

Motivation

  • Why should one learn the shell?
    • Advantages
    • Limitations

Intermediate

  • Job Scripting and best practices
    • What is a Bash script?
      • Where do scripts come from?
      • Structure of a Bash script: best practices
      • Adapt it to for use with sbatch
  • Dealing with repetition
    • Iterating on multiple files/directories with for loops
    • xargs instead of for
    • Non integer parameter scans with seq
  • Composability
    • Streams and redirections
      • Clarification: stdin vs command arguments
    • Pipes
    • Variables
    • Subshells and export
  • SSH ergonomics
    • SSH keys
      • Generating a ssh keypair and registering it
    • The .ssh/config file
  • Tips and Tricks for interactive use
  • Workflow discussion

Advanced

  • Understanding the Bash manual
  • More Bash features and their composability
    • More streams and redirections
      • Use a file as stdin for a process: <
      • stderr into stdout or vice versa
      • Process substitution: <()
      • Compound commands
    • Bash functions
    • Arrays
  • Tips and Tricks for interactive use
    • Multitasking et similia
      • Using multiple applications at the same time
      • Alternative: tmux
  • Defensive programming and debugging
    • Debug and error checking
      • Debugging: set -x
      • Error when using an unset variable: set -u
      • Fail on error: set -e
    • Automatic check for common problems
  • Unix tools for text file manipulation and exploration
    • “Vertical” text manipulation
    • Horizontal Manipulations
    • When cut does not cut it: awk
    • Bulk text manipulation with sed
    • Following a running program
      • The tee command
      • The tail -f command
  • SLURM Workflows
    • Repeating jobs with job arrays
    • Job chaining using SLURM dependencies
    • Job chaining by recursive sbatch invocations
    • Workflow managers
  • Simple Parallelism patterns in the shell
  • Tools for embarrassingly Parallel Workloads
    • Parbatch
    • Similar Projects

Extra

  • Jobscript Examples
    • Dealing with walltime limits on your own terms
    • Using a local/private FileSystem
    • Job chaining
      • Using recursive sbatch
      • Using Slurm dependencies
  • Instructor notes
    • Rationale
    • Known issues
    • Schedule
SCC/SCS Shell And Job Scripting
  • Search

© Copyright .

Built with Sphinx using a theme provided by Read the Docs.