Skip to content

Dependencies

This tutorial demonstrates how StepUp tracks dependencies.

Example

Example source files: docs/getting_started/dependencies/

The following plan.py defines two steps, with the second making use of the output from the first.

#!/usr/bin/env python3
from stepup.core.api import graph, run

run(r'printf "Monday frown\nCoffee smile\n" > story.txt', shell=True, out="story.txt")
run("grep Coffee story.txt", inp="story.txt")
graph("graph")

The command is sent to the director exactly as written: StepUp does not scan it for placeholders or perform any substitution on it. When the paths are known in advance, as in this example, you can simply type them into the command string. When they are computed dynamically (e.g. a list of paths, or a path built from other variables), use shq() to shell-quote and embed them, e.g. run(f"grep Coffee {shq(inp)}", inp=inp).

Embedding a step’s own paths this way normally means naming the path list twice, which forces an extra variable definition. To avoid that, the command may also be a callable that builds the command from the paths, so they are written exactly once:

run(lambda out: f"./gen.py {shq(out)}", out=["out1.txt", "out2.txt"])
run(lambda inp, out: f"cat {shq(inp)} > {shq(out)}", shell=True, inp=["a.md", "b.md"], out="all.md")

The callable may declare any subset of the parameters inp, out and vol, matched by name, and it is called once while the step is defined. It receives the paths after environment variable substitution and normalization, so the command text and the declared paths are guaranteed to be derived from the same values. With run() and plan(), the callable’s inp does not include the local executable that is detected in the command and added as an input: that executable is derived from the command, which does not exist yet when the callable is called, and it already appears as the first word of the command itself.

The graph() function writes the graph in a few formats, which are used for visualization below.

Now run StepUp with up to 2 steps in parallel:

sb -j 2

You will see the following output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
   START │ printf "Monday frown\nCoffee smile\n" > story.txt
DIRECTOR │ Wrote graph to graph.txt, graph_provenance.dot and graph_dependency.dot
 SUCCESS │ printf "Monday frown\nCoffee smile\n" > story.txt
   START │ grep Coffee story.txt
 SUCCESS │ ./plan.py
 SUCCESS │ grep Coffee story.txt
─────────────────────────────── Standard output ────────────────────────────────
Coffee smile
────────────────────────────────────────────────────────────────────────────────
DIRECTOR │ Ran 3 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

Although StepUp allows 2 steps to run in parallel, it executes your run steps sequentially, since it knows that the output of the first step will be used by the second.

Note, however, that the echo commands are already started before ./plan.py has finished. This is the expected behavior: even without a complete overview of all the build steps, StepUp will start the steps for which it has sufficient information.

Graphs

The plan.py script writes a few files to analyze and visualize the graphs StepUp uses internally. The file graph.txt is a detailed human-readable version of .stepup/graph.db:

root:
             product   file:plan.py
             product   step:./plan.py

file:plan.py
               state = CONFIRMED
              digest = 52f76e6e 44aa9a2f 2495d98a bc23e312 ac826bda eeaec4cf 28d9fe49 ec7603c9
             creator   root:
                sink   step:./plan.py

step:./plan.py
               state = RUNNING
                need = PLAN
             creator   root:
              source   file:plan.py
             product   step:grep Coffee story.txt
             product   step:printf "Monday frown\nCoffee smile\n" > story.txt

step:printf "Monday frown\nCoffee smile\n" > story.txt
               state = RUNNING
                need = DEFAULT
             creator   step:./plan.py
             product   file:story.txt
                sink   file:story.txt

file:story.txt
               state = PLANNED
             creator   step:printf "Monday frown\nCoffee smile\n" > story.txt
              source   step:printf "Monday frown\nCoffee smile\n" > story.txt
                sink   step:grep Coffee story.txt

step:grep Coffee story.txt
               state = PENDING
                need = DEFAULT
             creator   step:./plan.py
              source   file:story.txt

This text format may not always be the most convenient way to understand how StepUp connects all the steps and files. A more intuitive picture can be created with GraphViz using the .dot files as input. The figures below were created using the following commands:

dot -v graph_provenance.dot -Tsvg -o graph_provenance.svg
dot -v graph_dependency.dot -Tsvg -o graph_dependency.svg

The workflow in StepUp consists of two graphs involving (a subset of) the same set of nodes: the dependency graph and the provenance graph.

Dependency Graph

This graph shows how information is passed from one node to the next as the steps are executed.

graph_dependency.svg

This is an intuitive graph showing the execution flow. A similar graph is used by most other build tools.

Provenance Graph

This one shows who created each node in the graph:

graph_provenance.svg

This diagram can be challenging to interpret and requires further explanation. Each node in StepUp’s workflow is created by exactly one other node, except for the Root node, which is its own creator. In this example, there are three nodes that create other nodes:

  • The root node is an internal node controlled by StepUp. Upon startup, StepUp creates root and a few other nodes by default:

    • The initial plan.py file
    • The initial ./plan.py step (with working directory ./).

  • The ./plan.py step creates two nodes, see the two run() function calls in the plan.py script above.

    • The grep ... step.
    • The echo ... step.

  • The echo ... step creates one output file: story.txt.

This provenance graph is used by StepUp to decide which steps to keep and which to clean up. After some files have changed and StepUp is run again, some nodes may no longer be created. These “old” nodes will still exist in the database as “detached” nodes, i.e. without a creator.

After all steps have been successfully completed, StepUp will remove detached nodes that are not sources to other steps. When output file nodes are deleted, the corresponding files are also removed from disk. StepUp ensures safe removal: it will only delete files if it confirms they were created in a previous run and if their file hash still matches the one recorded when they were originally built.

If some steps use detached nodes as input, those steps will remain pending, resulting in an incomplete build and blocking the removal of the detached nodes.

Example:

  • After you modify plan.py and rerun StepUp, it will see that the file has changed and therefore detaches all nodes created by the old plan.py.
  • When StepUp runs the new plan.py, it may recreate some of the old nodes in exactly the same way, in which case the detached nodes will simply be restored, along with all of their products and related information.
  • If some nodes are not recreated, they will remain detached, and will be removed after a complete and successful build.
  • The new plan.py can also define new nodes, which simply extend the graph.
  • Nodes that are recreated with different properties will override any existing detached nodes.

Exploration of the Graph in a Web Browser

You can explore the graph interactively in a web browser using the stepup browse command. Run the following command in the same directory as above:

stepup browse

This will show the following output:

Server started http://localhost:8000
Press Ctrl+C to stop the server.

Open this link in a web browser to inspect every node in the graph. The web server will load the graph in memory and will only reload it when requested.

Try the Following

  • Run sb -j 2 again. As expected, the steps are now skipped.

  • Modify the grep command to select the first line (matching Monday) and run sb -j 2 again. The echo commands are skipped as they have not changed.

  • Change the order of the two steps in plan.py and run sb -j 2. The step ./plan.py is executed because the file has changed, but the echo and grep steps are skipped. This shows that plan.py is nothing but a plan, and it does not execute the steps itself. When plan.py is executed, it simply sends instructions to the director process.

  • Rename the file story.txt to lines.txt (in both steps) and restart StepUp. The old story.txt output file will be automatically removed from disk, as it is an intermediate output file whose node becomes detached and cleaned up.