Dynamic Dependencies¶

Every step in StepUp can inform the director process of additional inputs or environment variables it uses, or of additional (volatile) outputs it creates. These are generally referred to as dynamic dependencies because they are discovered at runtime.

A step amends itself with dynamic dependencies by sending them to the director process using the amend() function. Once registered, the dynamic dependencies are treated on equal footing as the initial ones. However, as soon as an initial input file changes and StepUp reruns, the dynamic dependencies are discarded and must be rediscovered by running the step. (This is needed because the dynamic dependencies typically depend on the contents of the initial inputs.)

Amending a step with a dynamic input will fail if that input is not yet available (built by a step or declared static). In this case, the step exits early and its execution is deferred by the director process until the required inputs become available.

Dynamic dependencies are convenient in various scenarios:

They are useful for handling nested dependencies where an initial input file references secondary inputs that are not yet generated. Because these secondary files might, once generated, point to a third tier of inputs, the full chain of dependencies cannot be mapped out in advance.

This creates a chicken-and-egg scenario where the build system cannot discover what files are needed until it actually runs the steps to generate them.

A classic example of this is compiling a complex LaTeX document, where the main file pulls in sub-chapters that rely on figures or data plots generated by external scripts.
Another use case is that some steps may take their default configuration from environment variables if some command-line options are missing. In this case, amend() can be used to specify the environment variables used.
Some steps may produce a list of volatile outputs, some of which are difficult to know upfront. Such volatile outputs can be specified using amend() once they have been created.

amend() is safe to call after an input file has already been read: if a dynamic input has been built too recently to be trusted (e.g. a producer step was still writing it while it was being read), the step is still deferred instead of failing outright.

While correctness is always guaranteed, it is the safest and most efficient to call amend() as early as possible, before accessing dynamic input files or creating dynamic outputs, whenever that is practical. In addition, try to amend the step with as many dynamic dependencies as possible in a single call. By following these good practices, you can avoid the following problems:

Trying to read from a file that hasn’t been created yet.

When you run amend(inp=...) before reading the file, and StepUp knows that the file is not yet available, the amend() call will raise an exception, preventing the step (or some wrapped program) from trying to read the file and failing.
Unintentionally overwriting a file by calling amend(out=...) or amend(vol=...) only after writing to these outputs.

When you call amend() first and the files are registered as static files or outputs of other steps, an exception is raised. This is a safety check to prevent overwriting files that belong to other steps, but this only works if you call amend() before writing to the files.
Performing unnecessary work.

A call to amend may mean that the step is interrupted and restarted later. Early calls to amend() are particularly important when the step is time-consuming or when it uses stepup.core.api functions to extend the workflow.
Deferring a step too many times.
As a safety net, a step that is deferred too many times in a row without succeeding will eventually fail instead of being deferred forever. The limit is configurable with the --defer-cap option, see Configuration.

Note that there are two different deferring mechanisms in StepUp:
1. The amend(inp=...) hits a file that has not been built yet. (This is an “unavailable input”.)
2. The amend(inp=...) hits a file that has been built by another step that completed after the current step started. If the amend() call is made after the file has been read, StepUp cannot guarantee correctness and will therefore defer the step.
Both types of deferring are counted towards the defer cap.

When you know some of a step’s dynamic dependencies while writing plan.py, you can declare them there as ordinary inp, env, out or vol arguments, even though the step also amends itself with them. The step is then not dispatched before those inputs are available, so it is deferred less often, while the step’s own amend() call still covers the general case in which the dependencies are not known upfront. The step needs no knowledge of what was declared for it: amend() silently ignores anything that the step already declares. Each argument is matched against its own kind only, so amending an out path that was declared as vol (or vice versa) is still an error.

To the best of our knowledge, there is no equivalent of amend() in other build tools. Some features in Ninja’s generator rule cover what can be achieved with amend().

Example¶

Example source files: docs/advanced_topics/dynamic_dependencies/

This example intentionally creates a simple scenario in which a step is amended with an extra input. This is a somewhat silly example to illustrate the concept. You may achieve the same result without amending, because you have full control over all scripts in the example.

Create the following plan.py, where the first step is a script that discovers that it needs an additional input.

#!/usr/bin/env python3
from stepup.core.api import run, static

static("step.py")
run("./step.py", inp=["step.py", "sources.txt"])
run("echo input.txt > sources.txt", shell=True, out="sources.txt")
run("echo Abracadabra! > input.txt", shell=True, out="input.txt", optional=True)

In addition, create a file input.txt with some arbitrary contents and the following step.py script:

#!/usr/bin/env python3
from stepup.core.api import amend

# Parse the sources.txt file
with open("sources.txt") as fh:
    paths_inp = fh.read().split()

# Request the additional input.
amend(inp=paths_inp)

# Write all files from source.txt to the standard output.
# This part is reachable only if the requested inputs are present.
for path_inp in paths_inp:
    print(f"Contents of {path_inp}:")
    with open(path_inp) as fh:
        print(fh.read())

Make the scripts executable and fire up StepUp to see how it deals with the amended step:

chmod +x step.py plan.py
sb -j 1

You should get the following terminal output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
   START │ echo input.txt > sources.txt
 SUCCESS │ echo input.txt > sources.txt
   START │ ./step.py
DEFERRED │ ./step.py
────────────────────────── Unavailable dynamic inputs ──────────────────────────
input.txt
────────────────────────────────────────────────────────────────────────────────
   START │ echo Abracadabra! > input.txt
 SUCCESS │ echo Abracadabra! > input.txt
   START │ ./step.py
 SUCCESS │ ./step.py
─────────────────────────────── Standard output ────────────────────────────────
Contents of input.txt:
Abracadabra!
────────────────────────────────────────────────────────────────────────────────
DIRECTOR │ Ran 5 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

The output shows that ./step.py first stops early due to the missing file input.txt. As a result, it becomes clear that input.txt is required, so StepUp schedules the optional step to generate this requested input. After input.txt has been created, StepUp runs ./step.py again.

Try the Following¶

Run StepUp again without making any changes. As expected, all steps are skipped. The .stepup/graph.db file also stores the dynamic dependencies, so these don’t need to be rediscovered later.
Modify the plan.py file so that ./step.py is amended with a second input, for example, other.txt. Run StepUp with these changes. Because sources.txt contains a new file, StepUp will try re-running ./step.py, which will amend the step with new inputs that require the step to be deferred again.