Basic SLURM example

The latest version of this example can be found at: https://github.com/reproducible-reporting/stepup-queue/tree/main/docs/examples/slurm-basic/

This example shows how to use StepUp to run job scripts, which can be either manually written (static) or generated from a template (dynamic). Since these jobs only take a few seconds and don’t perform any computations, they allow for a quick demonstration of StepUp Queue’s features.

Files

.
├── dynamic-template.sh
├── fail
│   └── slurmjob.sh
├── pass
│   └── slurmjob.py
├── plan.py
└── README.md

plan.py is a Python script that defines the workflow:

#!/usr/bin/env python3

from stepup.core.api import mkdir, render_jinja, static
from stepup.queue.api import sbatch

# Two examples of a static job script, i.e. already present on disk.
static("pass/", "pass/slurmjob.py")
sbatch("pass", ext=".py")
static("fail/", "fail/slurmjob.sh")
sbatch("fail")

# Example of job scripts generated from a template.
static("dynamic-template.sh")
for i in range(1, 4):
    mkdir(f"dynamic{i}/")
    render_jinja("dynamic-template.sh", {"field": i}, f"dynamic{i}/slurmjob.sh")
    # You can use the rc option to load an environment before calling sbatch.
    # Use this only if it cannot be done in the job script itself.
    sbatch(f"dynamic{i}/", rc="module swap cluster/doduo")

The job fail/slurmjob.sh is a static job script that fails with a non-zero exit code, which is correctly handled by StepUp Queue:

#!/usr/bin/env bash
#SBATCH --job-name fail
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

echo "This job will fail"
exit 1

The job pass/slurmjob.py shows how to write a Job script in Python:

#!/usr/bin/env python3
#SBATCH --job-name pass
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

from time import sleep

print("Hello from static job")
sleep(5)
print("Goodbye from static job")

The file dynamic-template.sh is a template from which actual job scripts are generated:

#!/usr/bin/env bash
#SBATCH --job-name 'dyn{{ field }}'
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

echo "Hello from dynamic job {{ field }}"
sleep 5
echo "Goodbye from dynamic job {{ field }}"

Note that render_jinja can be used to render any kind of text-based file from a template, such as inputs to computational tools, configuration files, etc.