Skip to content

Blocked Steps

As discussed in a previous tutorial on Optional Steps, StepUp has several mechanisms to ignore certain steps. As a rule, StepUp will always try to execute all steps, and not doing so is the exception.

A valid reason for ignoring some steps is illustrated in the following schematic:

     File          In development           File                Costly
╔═════════════╗     ┌──────────┐     ╔═════════════════╗     ┌──────────┐
║  input.txt  ║ --> │  Step 1  │ --> ║  converted.txt  ║ --> │  Step 2  │
╚═════════════╝     └──────────┘     ╚═════════════════╝     └──────────┘

Imagine that Step 2 is very expensive and you are developing a script for Step 1. In practice, it takes several iterations to get Step 1 working properly. This can be verified by analyzing the file converted.txt or by running unit tests.

To avoid executing Step 2 at every iteration in the development of Step 1, you can block this step. Blocking is achieved by assigning an undefined resource to the step, e.g. resources="blocked". Because the scheduler does not know about a resource named blocked, the step remains permanently pending until you remove the argument. Blocked steps are intended to be a temporary measure, and to be reverted once you’re done with Step 1.

Blocking a step has some consequences:

  • A blocked step remains in the PENDING state, meaning that outdated output files are not cleaned up automatically.
  • At the end of the build phase, all currently blocked steps are listed as a reminder.
  • Subsequent steps, which use outputs of blocked or pending steps, also remain pending.

Example

Example source files: docs/advanced_topics/blocked_steps/

The following plan.py illustrates the blocking mechanism. Note that the copy commands are too cheap to justify blocking, so this is just an example illustrating the mechanism.

#!/usr/bin/env python3
from stepup.core.api import copy, run

run("echo hello > ${out}", shell=True, out="a.txt")
copy("a.txt", "b.txt", resources="blocked")
copy("b.txt", "c.txt")

Make this plan executable and run it with StepUp:

chmod +x plan.py
sb -j 1

You should get the following terminal output:

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 3.2.3.post54)
   STARTUP │ (Re)initialized boot script
     PHASE │ build
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ echo hello > a.txt
   SUCCESS │ echo hello > a.txt
   WARNING │ 2 step(s) remained pending ...
───────────────────────────────── PENDING Step ─────────────────────────────────
Reason                required resources exceed maximum available
Command               cp -p a.txt b.txt
Inputs         BUILT  a.txt
Outputs      AWAITED  b.txt
Resource     blocked  1
───────────────────────────────── PENDING Step ─────────────────────────────────
Reason                required inputs are unavailable
Command               cp -p b.txt c.txt
Inputs       AWAITED  b.txt
Outputs      AWAITED  c.txt
────────────────────────────────────────────────────────────────────────────────
   WARNING │ Skipping file cleanup due to incomplete build
   WARNING │ Check logs: .stepup/warning.log
  DIRECTOR │ See you!

Try the Following

  • Remove the resources="blocked" argument, run StepUp, add it back, and run StepUp again. Although the copy commands are no longer executed, their outputs (b.txt and c.txt) are not cleaned up. This is the expected behavior because automatic cleaning is only performed when all (non-optional) steps have been executed successfully.

  • Remove the resources="blocked" argument, run StepUp, and then make the last copy command optional. In this case, the output of the optional step (c.txt) will be removed.