Skip to content

Welcome to StepUp Core

StepUp is a simple, powerful and universal build tool, a modern alternative to Make.

StepUp, like most build tools, schedules and executes commands in parallel. The scheduling takes into account that input files for a command must be available before starting it. Build tools also keep track of which other commands can create these files.

This is the documentation for StepUp Core, the basic framework for StepUp, without any domain-specific functionality. Domain-specific features are implemented in extension packages. Currently, there is only the StepUp RepRep extension for creating reproducible reports: papers, presentations, theses, etc.

What Does StepUp Look Like?

The following screen recording provides a quick visual impression of StepUp’s terminal user interface.

Why Was StepUp Core Created?

StepUp is a greenfield project inspired by similar tools, such as Ninja, pydoit and tup.

The defining feature of StepUp is that it treats the generation and execution of the build graph as one and the same thing. This may sound abstract, so let’s clarify this by reviewing how build tools work and have evolved over time.

Traditional build tools run programs in parallel, which you must define in advance by writing all steps and their dependencies in a text file, such as a Makefile. In practice, humans rarely write such files. Instead, they are often generated by other tools, such as CMake or Automake, which handle the configuration and discovery of build steps. This separation into generation and execution simplifies the build tool, but it also prevents steps from being defined using information from the output of previous steps.

More modern build tools, such as Bazel, Meson and Buck2 have also abandoned the traditional separation between build generator and executor. They introduce a domain-specific language (DSL), such as Starlark, to specify the build steps. These DSLs are designed to be powerful for build tasks, but are limited in what they can do for security reasons.

For software compilation, established build tools usually make acceptable assumptions, and workarounds exist for certain exceptions, see for example depfile, deps, dyndep and generator rule in Ninja. In build scenarios other than software compilation, e.g., building a scientific publication from LaTeX sources and raw data, these workarounds are too limited. For example, if a LaTeX source contains \input commands with TeX files generated by a Python script, it cannot be decided in advance whether these generated files reference additional input files, e.g., figures. In such cases, it is natural to determine all inputs of a LaTeX document on the fly instead of doing so in advance. (StepUp RepRep’s predecessor, RepRepBuild, generated build instructions for Ninja and addressed this problem with an elaborate generator rule.)

StepUp overcomes such difficulties by taking a different approach. The stepup command starts a background process that can receive build steps from any step in the build process, via Remote Procedure Calls (RPCs). It uses this information to extend its workflow, which is internally represented by a partial directed acyclic graph. This process is bootstrapped by an initial plan.py script containing the first RPC calls. Each build step can use intermediate results to add new information to the workflow. Steps can even be added rather late in the build, if this is necessary to correctly define such steps.

The program tup deserves a special mention in this brief review. StepUp’s algorithm for rebuilding steps (in response to changed inputs) strongly resembles that of tup. The build algorithm in both programs traverse upwards through the build graph. The “Up” part of StepUp’s name acknowledges this inspiration, with “Step” reflecting how StepUp defines operations as individual steps.

Other noteworthy features

  • StepUp build scripts are written in Python, in so-called plan.py files.

  • StepUp supports partial directed acyclic graph (PDAG) execution, similar to tup. StepUp is radical in the sense that it always assumes partial knowledge of the DAG. For example, at startup it will already run steps before it has complete knowledge of the workflow.

  • StepUp always runs background processes (a director and several workers) to execute steps, and a terminal frontend to control or interrupt the build. The director starts with a run phase to execute steps in parallel until the build is complete. When StepUp completes the build, it switches to a watch phase to register file changes. When the user requests a rerun, it knows exactly which part of the DAG needs to be rebuilt. This allows efficient edit-build iterations to incrementally build and refine a project.

  • A file must either be declared static (written by the user) or built (created by a step) before it can be used as input for steps. StepUp will never use a file without knowing if it is static or built. Static file declarations are uncommon in other build tools and allow StepUp to correctly execute steps with partial knowledge of the workflow.

  • Old outputs are automatically removed when the steps creating those files are removed from the workflow. This cleanup is only performed after a completely successful build.

  • Rich pattern-matching rules make it easy to multiplex a step over multiple similar inputs.

  • Steps do not need to have output files. They will be rerun if inputs have changed since the last run.

  • Environment variables can be defined as dependencies, so that steps will rerun if they depend on variables that have changed.

  • If a step’s input files have changed, a file hash is used to determine whether the file is different from a previous run before the step is re-run. This prevents unnecessary step executions in two common scenarios:

    • A file is changed and then reverted to its original state.
    • Switching between branches in Git.

  • The StepUp terminal user interface provides easy-to-follow progress information.

  • A keyboard shortcut can be defined in most IDEs to start a new run phase of an active StepUp director. The command python -c 'from stepup.core.interact import run; run()' will instruct the director to run pending steps due to file changes.

  • While a step is running, it can inform StepUp that it needs additional inputs, in which case the step will be rescheduled for later execution (after the additional inputs have become available). Similarly, a step can define additional outputs during its execution.

When to use StepUp?

  • The software compilation assumptions of other build tools get in the way.
  • You want to define the build graph in a language you already know: Python.
  • Writing out all dependencies upfront is difficult or impossible.
  • You enjoy living on the edge.