Amending Steps¶

Every step in StepUp can inform the director process of additional inputs or environment variables it uses, or of additional (volatile) outputs it creates. However, defining an amended input will fail when that file is not yet built nor known as a static file. In this case, the step will exit early and will be rescheduled by the director process when the amended input becomes available.

The amend() function implements this feature and is convenient in various scenarios:

This is useful for handling nested dependencies where an initial input file references secondary inputs that are not yet generated. Because these secondary files might, once generated, point to a third tier of inputs, the full chain of dependencies cannot be mapped out in advance.

This creates a chicken-and-egg scenario where the build system cannot discover what files are needed until it actually runs the steps to generate them.

A classic example of this is compiling a complex LaTeX document, where the main file pulls in sub-chapters that rely on figures or data plots generated by external scripts.

Another use case is that some steps may take their default configuration from environment variables if some command-line options are missing. In this case, amend() can be used to specify the environment variables used.
Some steps may produce a list of volatile outputs, some of which are difficult to know upfront. Such volatile outputs can be specified using amend() once they have been created.

To use amend() effectively, use it as early as possible, before accessing additional input files or creating unforeseen output files. Also, try to amend as many possible step arguments in a single call. This will avoid the following problems:

Trying to read from a file that hasn’t been created yet.
Unintentionally overwriting a file, e.g. a STATIC file that you have written by hand.
Performing unnecessary work.

(A call to amend may mean that the step is interrupted and restarted later. This is particularly important when the step is time-consuming or when it uses stepup.core.api functions to extend the workflow.)

Rescheduling a step multiple times.

To the best of our knowledge, there is no equivalent of amend() in other build tools. Some features in Ninja cover what can be achieved with amend().

Example¶

Example source files: docs/advanced_topics/amending_steps/

This example intentionally creates a simple scenario with an amended input. This is a somewhat silly example to illustrate the concept. You may achieve the same result without amending, because you have full control over all scripts in the example.

Create the following plan.py, where the first step is a script that will discover it needs an additional input.

#!/usr/bin/env python3
from stepup.core.api import run, static

static("step.py")
run("./step.py", inp=["step.py", "sources.txt"])
run("echo input.txt > ${out}", shell=True, out="sources.txt")
run("echo You better read this. > input.txt", shell=True, out="input.txt", optional=True)

In addition, create a file input.txt with some arbitrary contents and the following step.py script:

#!/usr/bin/env python3
from stepup.core.api import amend

# Parse the sources.txt file
with open("sources.txt") as fh:
    paths_inp = fh.read().split()

# Request the additional input.
amend(inp=paths_inp)

# Write all files from source.txt to the standard output.
# This part is reachable only if the amended input is present.
for path_inp in paths_inp:
    print(f"Contents of {path_inp}:")
    with open(path_inp) as fh:
        print(fh.read())

Make the scripts executable and fire up StepUp to see how it deals with the amended input:

chmod +x step.py plan.py
sb -j 1

You should get the following terminal output:

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 3.2.3.post54)
   STARTUP │ (Re)initialized boot script
     PHASE │ build
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ echo input.txt > sources.txt
   SUCCESS │ echo input.txt > sources.txt
     START │ ./step.py
RESCHEDULE │ ./step.py
──────────────── Rescheduling due to unavailable amended inputs ────────────────
input.txt
────────────────────────────────────────────────────────────────────────────────
     START │ echo You better read this. > input.txt
   SUCCESS │ echo You better read this. > input.txt
     START │ ./step.py
   SUCCESS │ ./step.py
─────────────────────────────── Standard output ────────────────────────────────
Contents of input.txt:
You better read this.
────────────────────────────────────────────────────────────────────────────────
  DIRECTOR │ Trying to delete 0 outdated output(s)
   WARNING │ Check logs: .stepup/warning.log
  DIRECTOR │ See you!

The output shows that ./step.py first stops early due to the missing file input.txt. As a result, it becomes clear that input.txt is required, so StepUp schedules the optional step to generate this requested input. After input.txt has been created, StepUp runs ./step.py again.

Try the Following¶

Run StepUp again without making any changes. As expected, all steps are skipped. The .stepup/graph.db file also stores the amended information, so these don’t need to be rediscovered later.
Modify the plan.py file to include a second amended input, for example, other.txt. Run StepUp with these changes. Because sources.txt contains a new file, StepUp will try re-running ./step.py, which will amend new inputs that require the step to be rescheduled again.