Optional Steps¶

By default, StepUp will build all steps created. As an exception, steps can be made optional by adding the optional=True option. This is the opposite of most build tools, where steps are only executed when they are targets.

The reason for this difference is that conventional build tools work with rigid predefined graphs. By accepting command-line arguments with target steps, they introduce some flexibility: this lets the user control which part of the graph is executed.

StepUp offers such flexibility in a different way. The basic premise is that all outdated or missing outputs need to be (re)built. It is the responsibility of the build tool to figure out which steps need executing. This responsibility should not be shifted to users by expecting them to specify targets. That said, some legitimate exceptions exist, in which ignoring steps is a desirable feature. These are supported by StepUp as follows:

One can define steps conditionally, e.g., as in the tutorial Static Glob Conditional. Such conditionals are controlled by external factors and are picked up by your plan.py without manual interventions.
One can make steps optional, as in this tutorial. This is useful when multiple steps are defined in a loop, as in the Static Glob tutorial, of which not all steps are required for the end result. Use this feature wisely: It is obviously inefficient to define a few thousand steps of which only a handful are needed.
As shown in the next tutorial, one may also block steps, as a temporary measure to speed up the edit-build cycle.

Example¶

Example source files: advanced_topics/optional_steps/

The example below uses the script() feature introduced in Script (Single Case) and Script (Multiple Cases) to create a somewhat entertaining example. However, practically all step-generating functions support the optional argument, and can thus be made optional in the same way.

Create a first script generate.py that generates sequences of the logistic map for different values of the parameter r:

#!/usr/bin/env python
from stepup.core.script import driver


def cases():
    yield 2.2
    yield 2.8
    yield 3.2
    yield 3.8


CASE_FMT = "logmap_{:5.3f}"


def case_info(r):
    return {"out": f"logmap_{r:5.3f}.txt", "r": r}


def run(out, r):
    x = 0.1
    with open(out, "w") as fh:
        for _ in range(100):
            print(f"{x:10.5f}", file=fh)
            x = r * x * (1 - x)


if __name__ == "__main__":
    driver()

Then, write a plot.py script that plots only one of these sequences:

#!/usr/bin/env python
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from stepup.core.script import driver


def info():
    r = 3.2
    return {
        "inp": ["matplotlibrc", f"logmap_{r:5.3f}.txt"],
        "out": "plot_logmap.png",
    }


def run(inp, out):
    mpl.rc_file(inp[0])
    seq = np.loadtxt(inp[1])
    fig, ax = plt.subplots()
    ax.plot(seq)
    ax.set_xlabel("n")
    ax.set_ylabel("x_n")
    fig.savefig(out)


if __name__ == "__main__":
    driver()

The plan.py file adds steps for both scripts, but makes the data generation optional:

#!/usr/bin/env python
from stepup.core.api import script, static

static("generate.py", "plot.py", "matplotlibrc")
script("generate.py", optional=True)
script("plot.py")

Finally, make the scripts executable and run StepUp:

chmod +x generate.py plot.py plan.py
stepup -n -w1

You should get the following output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./generate.py plan --optional
   SUCCESS │ ./generate.py plan --optional
     START │ ./plot.py plan
   SUCCESS │ ./plot.py plan
     START │ ./generate.py run -- 'logmap_3.200'
   SUCCESS │ ./generate.py run -- 'logmap_3.200'
     START │ ./plot.py run
   SUCCESS │ ./plot.py run
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

Note that, in this case, it would be trivial to modify the generate.py script to only generate the sequence of interest. Whenever such a simpler approach is possible, it is always preferable. However, in more complex use cases, it is not always possible to figure out which steps are going to be needed or not. In such situations, optional steps can be convenient.

Try the Following¶

Remove the optional=True keyword argument and rerun the plan. As expected, additional text files with sequences will be created.
Restore the optional=True keyword argument and rerun the plan. As expected, the Automatic Cleaning feature removes the outputs that were generated by steps that are no longer present in the workflow.