Blocked Steps

As discussed in a previous tutorials on Optional Steps and Build Targets, StepUp has several mechanisms to ignore certain steps. As a rule, however, StepUp will always try to execute all steps, and not doing so is the exception.

A valid reason for ignoring some steps is illustrated in the following schematic:

     File          In development           File                Costly
╔═════════════╗     ┌──────────┐     ╔═════════════════╗     ┌──────────┐
║  input.txt  ║ --> │  Step 1  │ --> ║  converted.txt  ║ --> │  Step 2  │
╚═════════════╝     └──────────┘     ╚═════════════════╝     └──────────┘

Imagine that Step 2 is very expensive and you are developing a script for Step 1. In practice, it takes several iterations to get Step 1 working properly. This can be verified by analyzing the file converted.txt or by running unit tests.

To avoid executing Step 2 at every iteration in the development of Step 1, you can block this step. Blocking is achieved by assigning an undefined resource to the step, e.g. resources="gate". Because the scheduler does not know about a resource named gate, the step remains permanently pending until you remove the argument. Blocked steps are intended to be a temporary measure, and to be reverted once you’re done with Step 1.

Blocking a step has some consequences:

• A blocked step remains in the PENDING state, meaning that outdated output files are not cleaned up automatically.
• At the end of the build phase, blocked steps are reported as a reminder, grouped by resource name with a count of how many steps each resource blocks. (The same grouping applies to steps blocked on unavailable inputs, grouped by file.) These counts are per-root totals: a step blocked on two different roots is counted under both, so the counts across the report can add up to more than the total number of pending steps. When build targets are in use and the blocked step is not needed to produce any of the given targets, it stays silently PENDING like any other unneeded step.
• Subsequent steps, which use outputs of blocked or pending steps, also remain pending.

Example

Example source files: docs/advanced_topics/blocked_steps/

The following plan.py illustrates the blocking mechanism. Note that the copy commands are too cheap to justify blocking, so this is just an example illustrating the mechanism.

#!/usr/bin/env python3
from stepup.core.api import copy, run

run("echo hello > a.txt", shell=True, out="a.txt")
copy("a.txt", "b.txt", resources="gate")
copy("b.txt", "c.txt")

Make this plan executable and run it with StepUp:

chmod +x plan.py
sb -j 1

You should get the following terminal output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
   START │ echo hello > a.txt
 SUCCESS │ echo hello > a.txt
DIRECTOR │ Ran 2 job(s).
 WARNING │ 2 step(s) remained pending.
──────────────────────────── Insufficient resources ────────────────────────────
                gate: 1 unit(s) needed, none available  2 step(s)
──────────────────────────────────── Remedy ────────────────────────────────────
Increase resources with --resources or lower the step requirements.
Run `stepup browse` and search for a name above to see the steps involved.
────────────────────────────────────────────────────────────────────────────────
 WARNING │ Skipping file cleanup due to incomplete build
 WARNING │ Check logs: .stepup/warning.log
DIRECTOR │ See you!

Try the Following

• Run sb -r gate to provide the gate resource and allow the blocked step to run. The output files b.txt and c.txt will be created.

• Next, run sb without the gate resource. Although the copy commands are not executed, obviously, but their outputs (b.txt and c.txt) are not cleaned up either. This is the expected behavior because automatic cleaning is only performed when all (non-optional) steps have been executed successfully.

• Remove the resources="gate" argument and then make the last copy command optional. Rerun sb and the output of the optional step (c.txt) will be removed. Because all non-optional steps have been executed successfully, the automatic cleaning mechanism was triggered.