Amending steps¶

Every step in StepUp can inform the director process of additional inputs or environment variables it uses, or of additional (volatile) outputs it creates. However, defining an amended input will fail when that file is not yet built nor known as a static file. In this case, the step can end early and will be rescheduled by the director process when the amended input becomes available.

The amend() function implements this feature and is convenient in various scenarios:

This is particularly useful when a step uses input files that refer to other input files. It may happen that some of these other input files still need to be generated by other steps. Once built, the other inputs may again refer to more input files, etc. Such dependencies cannot be discovered in advance, simply because not all inputs are available before running the steps that generate them. (This is a common scenario when writing LaTeX documents of which parts are generated by scripts.)
Another use case is that some steps may take their default configuration from environment variables if some command-line options are missing. In this case, amend can be used to specify the environment variables used.
Some steps may produce a list of volatile outputs, some of which are difficult to know upfront. One may specify such volatile outputs with amend after they have been created.

To the best of our knowledge, there is no equivalent of amend in other build tools. Some features in Ninja cover what can be achieved with amend.

Example¶

Example source files: advanced_topics/amending_steps/

This example intentionally creates a simple scenario with an amended input. This is a somewhat silly example to illustrate the concept. You may achieve the same result without amending, because you have full control over all scripts in the example.

Create the following plan.py, where the first step is a script that will discover it needs an additional input.

#!/usr/bin/env python
from stepup.core.api import static, step

static("step.py")
step("./step.py", inp=["step.py", "sources.txt"])
step("echo input.txt > ${out}", out="sources.txt")
step("echo You better read this. > input.txt", out="input.txt", optional=True)

In addition, create a file input.txt with some arbitrary contents and the following step.py script:

#!/usr/bin/env python
from stepup.core.api import amend

# Parse the sources.txt file
with open("sources.txt") as fh:
    paths_inp = fh.read().split()

# Write all files from source.txt to the standard output.
keep_going = amend(inp=paths_inp)
if keep_going:
    # This branch is only executed if the amended input is present.
    for path_inp in paths_inp:
        print(f"Contents of {path_inp}:")
        with open(path_inp) as fh:
            print(fh.read())

Make the scripts executable and fire up StepUp to see how it deals with the amended input:

chmod +x step.py plan.py
stepup -n -w1

You should get the following terminal output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ echo input.txt > sources.txt
   SUCCESS │ echo input.txt > sources.txt
     START │ ./step.py
RESCHEDULE │ ./step.py
──────────────── Rescheduling due to unavailable amended inputs ────────────────
input.txt
────────────────────────────────────────────────────────────────────────────────
     START │ echo You better read this. > input.txt
   SUCCESS │ echo You better read this. > input.txt
     START │ ./step.py
   SUCCESS │ ./step.py
─────────────────────────────── Standard output ────────────────────────────────
Contents of input.txt:
You better read this.
────────────────────────────────────────────────────────────────────────────────
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

The output shows that ./step.py first stopped early due to the missing file input.txt. Once this became clear, StepUp scheduled the optional step to generate the requested input. Later, StepUp ran ./step.py again.

Try the Following¶

Run StepUp again without making any changes. As expected, all steps are skipped. The workflow.mpk.xz file also stores the amended information, so these don’t need to be rediscovered later.
Modify the plan.py file to include a second amended input, for example, other.txt. Run StepUp with these changes. Because sources.txt contains a new file, StepUp will try re-running ./step.py, which will amend new inputs that require the step to be rescheduled again.