Basic SLURM example

The latest version of this example can be found at: https://github.com/reproducible-reporting/stepup-queue/tree/main/docs/examples/slurm-basic/

This example shows how to use StepUp to run job scripts, which can be either manually written (static) or generated from a template (dynamic). Since these jobs only take a few seconds and don’t perform any computations, they allow for a quick demonstration of StepUp Queue’s features.

Files

.
├── dynamic-template.sh
├── fail
│   └── slurmjob.sh
├── parts-template.sh
├── pass
│   └── slurmjob.py
├── plan.py
└── README.md

plan.py is a Python script that defines the workflow:

#!/usr/bin/env python3

from stepup.core.api import render_jinja, static
from stepup.queue.api import sbatch

# Two examples of a static job script, i.e. already present on disk.
static("pass/slurmjob.py")
sbatch("pass", ext=".py")
static("fail/slurmjob.sh")
sbatch("fail")

# Example of job scripts generated from a template.
static("dynamic-template.sh")
for i in range(1, 4):
    render_jinja("dynamic-template.sh", {"field": i}, f"dynamic{i}/slurmjob.sh")
    # You can use the rc option to load an environment before calling sbatch.
    # Use this only if it cannot be done in the job script itself.
    sbatch(f"dynamic{i}", rc="module swap cluster/doduo")

# Example of a job that computes several parts and skips parts that already exist.
# When NUM_PARTS is increased, the job script changes.
# The completeness check then finds missing parts, so the job is resubmitted,
# and the new job only computes the missing parts.
# Other changes to the job script, which do not add parts, do not lead to a resubmission.
NUM_PARTS = 3
static("parts-template.sh")
render_jinja(
    "parts-template.sh",
    # The wall time in minutes is estimated from the number of parts.
    {"num_parts": NUM_PARTS, "minutes": 1 + NUM_PARTS // 6},
    "parts/slurmjob.sh",
)
paths_part = [f"part{i}.txt" for i in range(NUM_PARTS)]
sbatch(
    "parts",
    out=paths_part,
    onchange="resubmit",
    complete=" && ".join(f"test -f {path}" for path in paths_part),
)

The job fail/slurmjob.sh is a static job script that fails with a non-zero exit code, which is correctly handled by StepUp Queue:

#!/usr/bin/env bash
#SBATCH --job-name fail
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

echo "This job will fail"
exit 1

The job pass/slurmjob.py shows how to write a Job script in Python:

#!/usr/bin/env python3
#SBATCH --job-name pass
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

from time import sleep

print("Hello from static job")
sleep(5)
print("Goodbye from static job")

The file dynamic-template.sh is a template from which actual job scripts are generated:

#!/usr/bin/env bash
#SBATCH --job-name 'dyn{{ field }}'
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

echo "Hello from dynamic job {{ field }}"
sleep 5
echo "Goodbye from dynamic job {{ field }}"

Note that render_jinja can be used to render any kind of text-based file from a template, such as inputs to computational tools, configuration files, etc.

Resubmitting a Job Only When Outputs Are Missing

The job generated from parts-template.sh computes a number of parts, each of which is written to a separate output file. Parts whose output already exists are skipped:

#!/usr/bin/env bash
#SBATCH --job-name parts
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time={{ minutes }}

# Compute all parts that do not exist yet.
# Parts completed by a previous job are skipped,
# so that this job can be resubmitted after more parts were added.
NUM_PARTS={{ num_parts }}
for ((i = 0; i < NUM_PARTS; i++)); do
    if [[ -f "part${i}.txt" ]]; then
        echo "Skipping part ${i}"
        continue
    fi
    echo "Computing part ${i}"
    sleep 5
    # The output is renamed at the end, so that an interrupted part is not mistaken as complete.
    echo "Result of part ${i}" > "part${i}.txt.tmp"
    mv "part${i}.txt.tmp" "part${i}.txt"
done

This job is submitted with onchange="resubmit" and a complete command, which checks that all output files exist. The following scenarios illustrate how the two work together:

1. Run the workflow with NUM_PARTS = 3 in plan.py. The job computes parts 0, 1 and 2.
2. Increase NUM_PARTS to 5 and run the workflow again. The job script has changed, and the complete command fails because part3.txt and part4.txt do not exist yet. The job is therefore resubmitted, and the new job only computes parts 3 and 4.
3. Add a comment to parts-template.sh and run the workflow again. The job script has changed again, but the complete command now succeeds. The existing job is kept and nothing is submitted.

Without the complete command, the third scenario would resubmit a job that has nothing to do, which still has to wait in the queue. With onchange="ignore" instead, the second scenario would fail, because the existing job did not produce the new outputs.

The wall time requested by the job is estimated from the number of parts, assuming that none of them is complete yet. When parts are skipped, the job simply ends before its wall time limit.