Skip to content

StepUp Core: A Dynamic Build Tool for Python

StepUp is a dynamic build tool and a modern alternative to Make. Its defining feature is that workflow generation and execution are unified: a plan.py script defines the initial build steps. While the workflow is being executed, any step can add more steps and dependencies, based on the outputs built so far. This makes StepUp ideal for builds where the full set of dependencies cannot be determined in advance.

This is the documentation for StepUp Core, the basic framework for StepUp, without any domain-specific extensions. Domain-specific features are implemented in extension packages. Currently, there are:

  • StepUp RepRep extension for creating reproducible reports: papers, presentations, theses, etc.
  • StepUp Queue submits jobs to a SLURM scheduler.

Quick Visual Impression of StepUp

StepUp in the Wild

A real-world example of StepUp is the AutoCorrelation Integral Drill (ACID) dataset. This project uses a StepUp workflow to regenerate a 43 GB collection of 15360 synthetic time series used to benchmark algorithms that compute the autocorrelation integral. The workflows generate the dataset, package and upload it to Zenodo, and validate the results.

Why Was StepUp Created?

StepUp is a greenfield project inspired by similar tools, such as Ninja, pydoit and tup.

The defining feature of StepUp is that it treats the generation and execution of the build workflow as one and the same thing. This may sound abstract, so let’s break it down by reviewing how build tools work and have evolved over time.

Traditional build tools run programs in parallel using a fixed workflow as input: you must define this workflow in advance by writing all steps and their dependencies in a text file, such as a Makefile. In practice, these workflow definitions are rarely written manually. Instead, they are often generated by other tools, such as CMake or Automake, which handle the configuration and discovery of build steps. This separation into generation and execution simplifies the build tool, but it also prevents steps from being defined using the output of previous steps.

More modern build tools, such as Bazel, Meson and Buck2 have abandoned the traditional separation between build generator and executor. They introduce a domain-specific language (DSL), such as Starlark, to specify the build steps. These DSLs are designed to be powerful for build tasks, but are limited in what they can do for security reasons.

For software compilation, established build tools usually make acceptable assumptions, and workarounds exist for certain exceptions, see for example depfile, deps, dyndep and generator rule in Ninja. In build scenarios other than software compilation, e.g., building a scientific publication from LaTeX sources and raw data, these workarounds are too limited. For example, if a LaTeX source contains \input commands with TeX files generated by a script, one cannot know in advance whether these generated TeX files will reference additional input files, e.g., figures. In such cases, it is natural to determine all inputs of a LaTeX document on the fly instead of doing so in advance. (StepUp RepRep’s predecessor, RepRepBuild, generated build instructions for Ninja and addressed this problem with an elaborate generator rule.)

StepUp overcomes such difficulties by taking a different approach. The stepup command starts a background process that can receive build steps from any step in the build process, via Remote Procedure Calls (RPCs). It uses this information to extend its workflow, which is internally represented as a partial directed acyclic graph. This process is bootstrapped by an initial plan.py script containing the first RPC calls. Each build step can use intermediate results to add new information to the workflow. Steps can even be added rather late in the build, if this is necessary to correctly define them.

The program tup deserves a special mention in this brief review. StepUp’s algorithm for rebuilding steps (in response to changed inputs) resembles that of tup. Tup traverses upwards through the dependency graph, and StepUp adopted this pattern. The “Up” part of StepUp’s name acknowledges this source of inspiration, while “Step” reflects how StepUp defines operations as individual steps.

Other Noteworthy Features

  • 🐍 Python Scripting: Build scripts are written in Python (in so-called plan.py files), instead of introducing a new domain-specific language (DSL).

  • 🕸️ Partial Directed Acyclic Graph (PDAG) Execution: StepUp supports execution of partially known DAGs, similar to tup. Execution can begin before the complete workflow graph is defined. Similarly, the cost of an incremental build is proportional to the number of steps that need to be rebuilt, not the total number of steps in the workflow.

  • 🔄 Dynamic Workflows: StepUp makes no distinction between the generation and execution of the workflow. Any step can extend the workflow with new files and steps based on runtime information. Dynamic workflows in StepUp even go a step further: a running step can dynamically register newly discovered dependencies (inputs, outputs, or environment variables) on the fly. If a dynamic input is missing when requested, the step is automatically deferred until its dependencies are built by other steps.

  • 🗓️ Advanced Scheduling: StepUp can schedule steps in parallel to prioritize steps that are on the critical path of the workflow. In addition, one can define resources and resource limits for steps, which StepUp will respect when scheduling. For instance, this can be used to limit the number of concurrent steps using a GPU.

  • 🌐 Graph Visualization Webserver: The stepup browse command launches a small local web server and opens your browser to interactively explore the build graph. It serves a snapshot of the workflow database, letting you inspect every node (files and steps), their states, and the dependencies connecting them. This makes it easy to understand and debug how a workflow is wired together.

  • 👁️ Watching File Changes: StepUp always runs background processes (a director and a step executor) to execute steps, and a terminal frontend to control or interrupt the build. The director starts with a build phase to execute steps in parallel until the build is complete. When StepUp completes the build, it can switch to a watch phase to register file changes. When the user requests a rerun, it knows exactly which part of the DAG needs to be rebuilt. This allows efficient edit-build iterations to incrementally build and refine a project.

  • 🧹 Automatic Cleanup: Old outputs are automatically removed when the steps creating those files are removed from the workflow. This cleanup is only performed after a complete and successful build.

  • ⏳ Progress Bar: The StepUp terminal user interface provides easy-to-follow progress information.

  • 📥 Input Declaration: A file must either be declared static (user-provided) or built (created by a step) before it can be used as input for steps. Static declarations are uncommon in other build tools and allow StepUp to correctly execute steps with partial knowledge of the workflow.

  • 🔑 File Hashing: If a step’s input files have changed, a file hash is used to determine whether the file is different from a previous run before the step is re-run. This prevents unnecessary step executions in two common scenarios:

    • A file is changed and then reverted to its original state.
    • Switching between branches in Git.

  • 💲 Environment Variable Dependencies: Steps can have environment variables as dependencies, so that they are flagged for re-execution after the variables change.

  • 🧩 Pattern Matching: Rich pattern-matching rules make it easy to multiplex a step over multiple similar inputs.

  • 💾 Optional Outputs: Steps do not need to have output files. They will be rerun if inputs have changed since the last run.

When to Use StepUp?

Many build and workflow tools overlap with StepUp, so whether it fits depends on what you need. StepUp sits between the two categories, and is a good fit if you want either:

  • a build tool with features more typical of workflow tools, such as dynamic workflows, advanced scheduling, file hashing, rich metadata, or automatic cleanup; or
  • a workflow tool with features more typical of build tools, such as complex dependencies, pattern matching, or incremental building.

A few use cases are out of scope by design:

  • Very short steps (a few milliseconds each). The overhead of scheduling, file hashing, and process execution adds up. Workflow tools built on long-running pilot processes or threads are faster here, but risk carrying side effects from one step to the next, whereas StepUp prioritizes robustness and correctness.
  • Huge graphs (hundreds of thousands of steps or more). StepUp’s SQLite storage overhead becomes significant. A pure in-memory graph (with specialized data structures) can handle larger workflows more efficiently, at the cost of giving up the crash recovery that SQLite provides.