Optional Steps¶

By default, StepUp will build all steps. As an exception, steps can be made optional by adding the optional=True option. This is the opposite of most build tools, where steps are only executed when they are targets.

The reason for this difference is that conventional build tools work with rigid predefined graphs. They introduce some flexibility by allowing users to specify which steps are targets on the command line. This gives the user some control over which part of the graph is executed.

StepUp offers such flexibility differently. The basic premise is that all outdated or missing outputs need to be (re)built. It is the responsibility of the build tool to figure out which steps need executing. This responsibility should not be shifted to users by expecting them to specify targets. That said, some legitimate exceptions exist, in which ignoring steps is a desirable feature. These are supported by StepUp as follows:

One can define steps conditionally, e.g., as in the tutorial Static Glob Conditional. Such conditionals are controlled by external factors and are picked up by your plan.py without manual interventions.
One can make steps optional, as in this tutorial. This is useful when multiple steps are defined in a loop, as in the Static Glob tutorial, of which not all steps are required for the end result. Use this feature wisely: Defining thousands of steps when only a few are actually used, is obviously inefficient.
As shown in the next tutorial, one may also block steps, as a temporary measure to speed up the edit-build cycle.

Steps that define other steps, declare static files, or otherwise extend the workflow, should not be made optional. Until such steps are executed, StepUp has no idea what output these steps will generate, which it would need to decide that an optional step is required by another step.

Example¶

Example source files: docs/advanced_topics/optional_steps/

The example below uses the call() feature introduced in the Call tutorial to create a somewhat entertaining example. However, practically all step-generating functions support the optional argument, and can thus be made optional in the same way.

Create a first script generate.py that generates sequences of the logistic map for different values of the parameter r:

#!/usr/bin/env python3
from stepup.core.call import driver


def run(out: list[str], r: float):
    x = 0.1
    with open(out[0], "w") as fh:
        for _ in range(100):
            print(f"{x:10.5f}", file=fh)
            x = r * x * (1 - x)


if __name__ == "__main__":
    driver()

Then, write a plot.py script that plots only one of these sequences:

#!/usr/bin/env python3
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from stepup.core.call import driver


def run(inp: list[str], out: list[str]):
    mpl.rc_file(inp[0])
    seq = np.loadtxt(inp[1])
    fig, ax = plt.subplots()
    ax.plot(seq)
    ax.set_xlabel("n")
    ax.set_ylabel("x_n")
    fig.savefig(out[0])


if __name__ == "__main__":
    driver()

The plan.py file adds steps for both scripts, but makes the data generation optional:

#!/usr/bin/env python3
from stepup.core.api import call, static

static("generate.py", "plot.py", "matplotlibrc")

for case_r in [2.2, 2.8, 3.2, 3.8]:
    call("./generate.py", "run", out=f"logmap_{case_r:5.3f}.txt", r=case_r, optional=True)

call(
    "./plot.py",
    "run",
    inp=["matplotlibrc", "logmap_3.200.txt"],
    out="plot_logmap.png",
)

Finally, make the scripts executable and run StepUp:

chmod +x generate.py plot.py plan.py
sb -j 1

You should get the following output:

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 3.2.3.post54)
   STARTUP │ (Re)initialized boot script
     PHASE │ build
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./generate.py run '{"r": 3.2, "out": ["logmap_3.200.txt"]}'
   SUCCESS │ ./generate.py run '{"r": 3.2, "out": ["logmap_3.200.txt"]}'
     START │ ./plot.py run '{"inp": ["matplotlibrc", "logmap_3.200.txt"], "out": ["plot_logmap.png"]}'
   SUCCESS │ ./plot.py run '{"inp": ["matplotlibrc", "logmap_3.200.txt"], "out": ["plot_logmap.png"]}'
  DIRECTOR │ Trying to delete 0 outdated output(s)
  DIRECTOR │ See you!

Note that, in this case, it would be trivial to modify the generate.py script to only generate the sequence of interest. Whenever such a simpler approach is possible, it is always preferable. However, in more complex use cases, it is not always possible to figure out which steps are going to be needed or not. In such situations, optional steps can be convenient.

Try the Following¶

Remove the optional=True keyword argument from the call() for generate.py in plan.py and rerun the plan. As expected, additional text files with sequences will be created.
Restore the optional=True keyword argument and rerun the plan. As expected, the Automatic Cleaning feature removes the outputs that were generated by steps that are no longer present in the workflow.