Skip to content

Optional Steps

By default, StepUp will build all steps. As an exception, steps can be made optional by adding the optional=True option. This is the opposite of most build tools, where a step only runs when it is needed, directly or indirectly, by a target given on the command line.

The reason for this difference is that conventional build tools work with rigid predefined graphs. They introduce some flexibility by allowing users to specify which steps are targets on the command line. This gives the user some control over which part of the graph is executed, but also shifts to them the responsibility of knowing which targets a given task actually needs.

StepUp keeps this responsibility with the build tool instead. The basic premise is that all outdated or missing outputs need to be (re)built, and it is StepUp’s job to figure out which steps that requires. That said, several legitimate mechanisms exist for running only part of the workflow when this is genuinely useful. These are supported by StepUp as follows:

  • One can define steps conditionally, e.g., as in the tutorial Glob Conditional. Such conditionals are controlled by external factors and are picked up by your plan.py without manual interventions.

  • One can make steps optional, as in this tutorial. This is useful when multiple steps are defined in a loop, as in the Glob Patterns in static() tutorial, of which not all steps are required for the end result. Use this feature wisely: Defining thousands of steps when only a few are actually used, is obviously inefficient.

  • One can build a subset of targets, by listing one or more output paths on the command line, as shown in the next tutorial. This is an inclusion mechanism: only the steps needed to produce the given targets run, and the rest stay PENDING. It is useful when you temporarily want to work on a small part of a much larger workflow, e.g., for debugging purposes. An exact-file target reaches an optional step just like any other step: naming its output explicitly is enough to build it, regardless of its declared need. A directory target is more conservative: it only elevates steps whose declared need is DEFAULT, so an optional step’s output sitting under a targeted directory is not, by itself, a reason to build it.

  • One can also block steps, as shown in a later tutorial. This acts more as a longer-term exclusion filter: it keeps one or more steps pending indefinitely, e.g., while a downstream part of the workflow is still under development or too expensive to run on every iteration, until you explicitly unblock them again.

Steps that define other steps, declare static files, or otherwise extend the workflow, should not be made optional. Until such steps are executed, StepUp has no idea what output these steps will generate, which it would need to decide that an optional step is required by another step.

Example

Example source files: docs/advanced_topics/optional_steps/

The example below uses the call() feature introduced in the Call tutorial to create a somewhat entertaining example. However, practically all step-generating functions support the optional argument, and can thus be made optional in the same way.

Create a first script generate.py that generates sequences of the logistic map for different values of the parameter r:

#!/usr/bin/env python3
from stepup.core.call import driver


def run(out: list[str], r: float):
    x = 0.1
    with open(out[0], "w") as fh:
        for _ in range(100):
            print(f"{x:10.5f}", file=fh)
            x = r * x * (1 - x)


if __name__ == "__main__":
    driver()

Then, write a plot.py script that plots only one of these sequences:

#!/usr/bin/env python3
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from stepup.core.call import driver


def run(inp: list[str], out: list[str]):
    mpl.rc_file(inp[0])
    seq = np.loadtxt(inp[1])
    fig, ax = plt.subplots()
    ax.plot(seq)
    ax.set_xlabel("n")
    ax.set_ylabel("x_n")
    fig.savefig(out[0])


if __name__ == "__main__":
    driver()

The plan.py file adds steps for both scripts, but makes the data generation optional:

#!/usr/bin/env python3
from stepup.core.api import call, static

static("generate.py", "plot.py", "matplotlibrc")

for case_r in [2.2, 2.8, 3.2, 3.8]:
    call("./generate.py", "run", out=f"logmap_{case_r:5.3f}.txt", r=case_r, optional=True)

call(
    "./plot.py",
    "run",
    inp=["matplotlibrc", "logmap_3.200.txt"],
    out="plot_logmap.png",
)

Finally, make the scripts executable and run StepUp:

chmod +x generate.py plot.py plan.py
sb -j 1

You should get the following output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
   START │ ./generate.py run '{"r": 3.2, "out": ["logmap_3.200.txt"]}'
 SUCCESS │ ./generate.py run '{"r": 3.2, "out": ["logmap_3.200.txt"]}'
   START │ ./plot.py run '{"inp": ["matplotlibrc", "logmap_3.200.txt"], "out": ["plot_logmap.png"]}'
 SUCCESS │ ./plot.py run '{"inp": ["matplotlibrc", "logmap_3.200.txt"], "out": ["plot_logmap.png"]}'
DIRECTOR │ Ran 3 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

Note that, in this case, it would be trivial to modify the generate.py script to only generate the sequence of interest. Whenever such a simpler approach is possible, it is always preferable. However, in more complex use cases, it is not always possible to figure out which steps are going to be needed or not. In such situations, optional steps can be convenient.

Try the Following

  • Remove the optional=True keyword argument from the call() for generate.py in plan.py and rerun the plan. As expected, additional text files with sequences will be created.

  • Restore the optional=True keyword argument and rerun the plan. As expected, the Automatic Cleaning feature removes the outputs that were generated by steps that are no longer present in the workflow.