Skip to content

StepUp Configuration

StepUp can be configured using configuration files, environment variables, and command-line options. When a stepup tool starts, e.g. sb, it will load its settings in the following order, with later settings overriding earlier ones:

  • /etc/stepup.toml (system-wide configuration file)
  • ~/.config/stepup.toml (user-wide configuration file)
  • ${STEPUP_ROOT}/.stepup.toml (project-specific configuration file)
  • ${STEPUP_ROOT}/stepup.toml (project-specific configuration file)
  • ${STEPUP_ROOT}/pyproject.toml (project-specific configuration file, with settings under the [tool.stepup] section)
  • ${STEPUP_ROOT}/stepup-local.toml (project-specific configuration file, intended for local overrides that are not committed to version control)
  • Environment variables (STEPUP_*)
  • Command-line options

On Windows, ~ is the profile directory of the user, e.g. C:/Users/me, and /etc/stepup.toml lies in the root of the current drive, e.g. C:/etc/stepup.toml.

Every setting belongs to a subcommand, e.g. build, and is placed under a subcommand-specific section, e.g. [build] (or [tool.stepup.build] in pyproject.toml). The top level of a config file (or [tool.stepup] in pyproject.toml) is reserved for settings shared by all subcommands, of which there are currently none.

Example stepup.toml:

[build]
jobs = 4
log_level = "INFO"
watch = true

Having multiple configuration files is convenient but can be confusing. StepUp provides a stepup config tool to help you understand which settings are in effect. This tool reads all the configuration files and environment variables, and shows the merged settings as a single, informative TOML file, including comments about the source of each setting.

Positional command-line arguments (targets to build or paths to clean) cannot be set through configuration files or environment variables.

Configuration Errors

A subcommand refuses to start when anything is wrong with a configuration file: invalid TOML syntax, an unknown section, a key that is not a setting of the section it appears in, or a value that a setting cannot use. The value of an environment variable is checked in the same way, but its name is not: see Unrecognized Environment Variables. All problems are reported at once, so that a single run tells you everything to fix. Each one names the config file the way stepup config does, relative to the working directory when it lies below it:

$ sb
ERROR: Problems with the StepUp configuration:
  ./stepup.toml: unsupported key 'speed' in section [build]
  ./stepup.toml: unknown section [buidl] (did you mean 'build'?)
Run 'stepup config' to inspect the configuration.

This sets the first bit of the return code, without a Python traceback unless STEPUP_DEBUG is set. Problems are shown in red when the terminal supports color.

The stepup config subcommand is the one exception: it still prints the configuration it loaded, with the source of each setting. This makes it usable precisely when the configuration is broken. Each problem is shown on the line of the setting, section or config file it concerns, for the same stepup.toml as above:

# Config files (lowest to highest priority):
#   MISSING: /etc/stepup.toml
#   MISSING: ~/.config/stepup.toml
#   MISSING: .stepup.toml
#   FOUND:   ./stepup.toml
#   MISSING: ./pyproject.toml
#   MISSING: ./stepup-local.toml
# Environment variables: STEPUP_*

[buidl]  # <-- ERROR: ./stepup.toml: unknown section [buidl] (did you mean 'build'?)
jobs = 2  # ./stepup.toml

[build]
speed = 4  # ./stepup.toml  <-- ERROR: unsupported key 'speed' in section [build]

The problems are written in comments, so the output remains valid TOML. A problem that has no line of its own is listed afterwards on standard error. This happens when a config file with a higher priority overrides the setting at fault, and when a line already carries another problem, because a second comment cannot be opened on the same line.

Unrecognized Environment Variables

A misspelled key in a config file is an error, but a misspelled STEPUP_* variable is not. The STEPUP_* namespace is shared with the variables that configure StepUp’s internals, listed under StepUp Core Module Environment Variables, which no subcommand defines an option for. Rejecting every name that is not a setting would therefore reject StepUp’s own environment.

To make a typo visible anyway, stepup config lists the STEPUP_* variables in three groups, by what each of them does:

# Configuration environment variables:
#   STEPUP_BUILD_JOBS = "4"

# StepUp Core module environment variables:
#   STEPUP_ROOT = "/home/user/project"

# Unrecognized environment variables, without effect:
#   STEPUP_BUILD_JBOS = "4"

When a setting does not take effect, the last group is the first place to look. It holds every STEPUP_* variable that StepUp does not act on, which for a variable you set deliberately means a misspelled name. Variables of an extension package appear there as well when they configure that package’s internals rather than one of its settings.

StepUp Core Module Environment Variables

Some environment variables affect StepUp’s internals even when it is just imported as a Python library. These can only be set via environment variables, and cannot be configured through config files or command-line options.

STEPUP_MAX_OUTPUT_SIZE

The maximum size of standard output and standard error stored in the workflow database. The default is 0, meaning unlimited (no truncation). This limit only affects what is persisted, not the terminal output. When a stream exceeds the limit, it is truncated on a UTF-8 character boundary and a [output truncated at N bytes] line is appended.

STEPUP_PATH_FILTER

A list of filters for determining whether to retain or ignore an automatically detected dependency. The items are separated by a colon on POSIX systems and by a semicolon on Windows, as in PATH. Each item starts with a + or - sign, followed by a path prefix to be used for matching. The items in the filter are processed in order, and the first match determines the action. If the matching path prefix is preceded by a -, the dependency is ignored. If it is preceded by a +, the dependency is retained and rewritten relative to ${STEPUP_ROOT}. A path in the filter can be absolute or relative to ${STEPUP_ROOT}, but matching is always done based on absolute paths. A prefix is matched as a plain string, not as a sequence of path components, so a filter item also matches paths whose first differing character is not a separator. For example, -venv ignores dependencies under venv and under venv311, which is convenient when several virtual environments sit next to each other, but +data also retains dependencies under database. A trailing separator does not restrict an item to one directory, because it is removed when the prefix is resolved to an absolute path.

The default is -.venv:-venv:-.tox:-.nox:-.direnv:-.pixi:-node_modules, which ignores the top-level directories in which tools install dependencies. Because every prefix is anchored at ${STEPUP_ROOT}, the default cannot ignore an environment nested deeper in the tree, such as packages/foo/.venv. Setting this variable replaces the default in full, so an environment directory has to be listed again when the variable is used to retain files outside ${STEPUP_ROOT}.

Regardless of whether a filter is defined, the filters :+.:-/ are always appended.

This feature can be used for several purposes:

  • You may have source files that are not part of the StepUp project, but are used in the project and edited frequently. In this case, steps that depend on these external files will be rerun when you change the external source files.

  • You have a virtual environment with many packages installed, but you don’t want to include them in the dependency graph for performance reasons. (The default filter already does this for the most common environment directories.)

STEPUP_ROOT

The root directory containing the top-level plan.py file. If not set, StepUp will look for this file in the current directory.

STEPUP_RPC_TRANSPORT

The kind of socket through which stepup build, the director, the steps and the other stepup subcommands talk to each other. This is a setting for developers of StepUp, to test both transports on Linux and macOS. End users can leave it unset. unix selects Unix domain sockets, which is the default on platforms that have them. tcp is the default and the only accepted value on Windows. With tcp, the director and stepup build listen on TCP sockets on the loopback interface (127.0.0.1). Every local user can connect to such a socket, so a client must first send a random token, which it reads from a file that only the user running stepup build can access. Whoever obtains this token can run code as that user.

On Windows, this file is exactly as private as the temporary directory in which StepUp creates it. Python 3.12.4 and later, and 3.11 releases from 3.11.10 on, give that directory access rules of its own, which only admit its owner, SYSTEM and administrators. Older versions let it inherit the access rules of the directory named by TEMP (CVE-2024-4030 of Python). These rules are private by default, because TEMP lies in the profile of the user, but not when TEMP is set to a directory that other users can read, e.g. a directory in the root of a drive, such as C:\Temp. On Windows, use Python 3.12.4 or later, or a 3.11 release from 3.11.10 on, and in particular when TEMP has been changed.

Only stepup build reads this variable: the steps and the other subcommands find out which transport is in use by themselves.

STEPUP_SHELL

The POSIX shell that runs the commands of steps created with shell=True, and local shell scripts on platforms that cannot execute them directly. The value is the path of the shell, or its name, which is then looked up in PATH. StepUp starts it as SHELL -c COMMAND. The default is /bin/sh on Linux and macOS. When STEPUP_SHELL is not set on Windows, StepUp uses the first of:

  1. sh on PATH, unless it lies in the Windows directory, which holds the launchers of WSL.
  2. bin\sh.exe of the Git installation whose git is on PATH.
  3. bin\sh.exe of the installation of Git for Windows that is recorded in the registry, for the current user first and for all users second.

StepUp only searches when a step needs a shell, and when nothing is found, that step fails with a message that lists what was tried. See POSIX Shell on Windows for the shells that can be used. Only stepup build and steps that call run_subprocess() with shell=True read this variable. Changing it does not make StepUp rerun steps.

STEPUP_SYNC_RPC_TIMEOUT

The number of seconds a synchronous RPC client waits for a response from the director, which also bounds the connection setup that the first call triggers. The default is 600 seconds. Set this to a smaller value if you want to detect deadlocks more quickly. A negative or zero value disables the timeout, which some calls use because the director answers them only when the workflow is ready for it.

STEPUP_WATCHER

The library that watches the file system in watch mode, i.e. with the --watch or --watch-first option of stepup build. This is a setting for developers of StepUp, to test both libraries on Linux. End users can leave it unset. On Linux, the default is asyncinotify, and watchfiles can be selected after installing it with pip install 'stepup[watchfiles]'. On other platforms, watchfiles is the default and the only accepted value. Without watch mode, this variable is ignored.

STEPUP_WATCHER_RECURSIVE

How the watchfiles library watches the directories below the one in which StepUp runs: 1 for a single recursive watch, 0 for a separate watch on every directory of interest. This is a setting for developers of StepUp, to test both ways on every platform. End users can leave it unset. The default is 1 on Windows, where a watch on a directory makes it impossible to rename its parent directories, and 0 on other platforms. A watched directory outside the directory in which StepUp runs always gets a watch of its own. This variable is ignored without watch mode and when STEPUP_WATCHER is asyncinotify.

On Linux, a recursive watch follows symbolic links, and removing a symbolic link to a watched directory stops the reports of changes in that directory, and possibly in the directories below it, until StepUp is restarted. This was observed with watchfiles 1.3.0, and is caused by a bug in the Rust library notify on which watchfiles builds, see notify#1038.

Settings for All Subcommands

Each entry below lists the config file key, environment variable, and command-line option separated by slashes, where applicable.

STEPUP_DEBUG

Set to 1 to enable debugging features and strict consistency checks. This changes the default log level of stepup build to DEBUG (see log_level below) and will require internal consistency checks to pass, rather than applying corrections to overcome the inconsistencies. (Every such inconsistency is due to a bug, which should be fixed eventually.) It also makes the scan of .stepup/director.log at the end of a build fatal: a logged error or a coroutine, task or thread left dangling is reported as an error and sets the internal error bit of the return code, instead of only being reported as a warning. It disables the shortening of error reports described in Failing Steps: a failing step then prints its complete traceback, including StepUp’s own frames and the traceback of the director process. A mistake that any subcommand reports as a short ERROR: message keeps its traceback, which is the only way to find out where it was raised. Finally, when a plan.py or a console script runs outside StepUp, the functions of stepup.core.api print the calls they would send to the director on standard error, instead of ignoring them silently. This variable cannot be set through config files or command-line options.

Settings for sb or stepup build

These settings are stored under the [build] section in config files (or [tool.stepup.build] in pyproject.toml). Each entry below lists the config file key, environment variable, and command-line option separated by slashes, where applicable. The settings are grouped as in the output of stepup build --help, alphabetically within each group.

Build Control

clean / STEPUP_BUILD_CLEAN / --clean, --no-clean

Set to false to disable automatic cleaning of outdated output files. By default, StepUp automatically removes old output files that are no longer created by any step in the workflow.

duration / STEPUP_BUILD_DURATION / --duration, --no-duration

Set to false to disable recording of step wall times as their duration for future runs. StepUp uses the duration information to prioritize the execution of steps with the longest critical path (in time units) to any terminal node. (This is stored in the database as the _tail_time field of each step.) While accurate durations are generally beneficial, they can result in non-deterministic execution order, which can be undesirable in some cases, such as testing.

When disabled, durations remain unchanged from the previous run or stay at their initial value if the step has never been executed before. The initial value can be provided when defining the step, or defaults to 1.0 if not provided. With the 1.0 default, the _tail_time of a step degrades to the number of steps in the longest path to any terminal node.

explain_rerun / STEPUP_BUILD_EXPLAIN_RERUN / --explain-rerun, -e, --no-explain-rerun

Set to true to explain for every step with recording info why it cannot be skipped.

fix_epoch / STEPUP_BUILD_FIX_EPOCH / --fix-epoch, --no-fix-epoch

If set to true (the default), the SOURCE_DATE_EPOCH environment variable will be set to a fixed value of 315532800 (corresponding to 1980-01-01 00:00:00 UTC) for all step executions. This is useful for ensuring reproducible builds: Many tools and libraries recognize SOURCE_DATE_EPOCH and use it instead of the current time for timestamps in generated files. If the variable is already set in the environment, it will be used as-is.

jobs / STEPUP_BUILD_JOBS / --jobs, -j

The maximum number of steps to run concurrently. When given as a floating point number, the value is multiplied by the number of available CPU cores. The default is 1.0.

keep_going / STEPUP_BUILD_KEEP_GOING / --keep-going, -k, --no-keep-going

Set to true to keep dispatching new steps after another step has failed, as long as their own inputs remain available (like make -k). By default (false), the scheduler starts draining after the first failure: steps already running are still allowed to finish, but no new steps are started.

defer_cap / STEPUP_BUILD_DEFER_CAP / --defer-cap

Maximum number of times a step can be deferred (since it last succeeded) before it is reported as failed instead of parked pending again. This guards against livelocks where a step’s dynamic inputs keep flip-flopping. The default is 100, deliberately generous.

offline / STEPUP_BUILD_OFFLINE / --offline, --no-offline

Set to true when remote data cannot be reached, e.g. without a network connection. STEPUP_DIRECTOR_I then keeps the value of the previous start, so a step that runs at every start of the director does not run merely because StepUp started. Such a step keeps its result of the previous start, and so do the steps that use its outputs. A step that must run for another reason, e.g. because an input changed or because it failed, runs as usual. The director also exports the setting as STEPUP_BUILD_OFFLINE to the steps it runs, with the value 1 or 0, so that a step that does run can avoid the network, as described in Steps That Use the Network. The next start without this setting gives STEPUP_DIRECTOR_I a new value again, so every step that runs at every start looks at the remote data again. In watch mode, the setting holds for the lifetime of the director. Working Offline shows an example.

progress / STEPUP_BUILD_PROGRESS / --progress, --no-progress

Set to false to disable the progress bar in the terminal user interface. This can be useful to simplify and reduce the output.

resources / STEPUP_BUILD_RESOURCES / --resources, -r

A comma-separated list of resource names and available quantities to be used for scheduling decisions. For example, resources = "gpu:2,cpu:4" indicates that there are 2 GPUs and 4 CPUs available. Any resource labels can be used, and the available quantity can be any positive integer. Note that resource specifications from config files, the environment variable, and one ore more CLI options (from left to right, e.g. -r gpu:2 -r cpu:4) are merged together.

watch / STEPUP_BUILD_WATCH / --watch, -w, --no-watch

Set to true to enable watch mode. In watch mode, StepUp will monitor the file system for changes and rerun affected steps after pressing the r key in the terminal user interface.

watch_first / STEPUP_BUILD_WATCH_FIRST / --watch-first, -W, --no-watch-first

Set to true to automatically rerun affected steps when relevant file changes are observed, without needing to press the r key. This implies watch = true.

Execution Environment

cgroup / STEPUP_BUILD_CGROUP / --cgroup, --no-cgroup

This setting controls whether StepUp will run the director (and all its child processes running steps) use cgroup isolation. When enabled, peak memory usage of the director process and all its child processes is measured. Only available on Linux with cgroup v2 enabled and if systemd-run is available. Exceptions are raised if cgroup isolation is requested but not working. Off by default.

forkserver / STEPUP_BUILD_FORKSERVER / --forkserver, --no-forkserver

Set to true to use a forkserver for Python step execution, which reduces startup overhead. This is enabled by default on Linux. Enabling it on a platform without the forkserver start method, such as Windows, is an error.

preload_modules / STEPUP_BUILD_PRELOAD_MODULES / --preload-modules

A comma-separated list of Python modules to pre-load into the forkserver. Only has effect when forkserver = true. Use this to reduce per-step startup time when all (or most) steps import the same large modules. For example, preload_modules = "numpy,scipy" pre-loads NumPy and SciPy into the forkserver so that each Python step forked from it inherits them at zero import cost. By default, no additional modules are pre-loaded (only internal StepUp modules are pre-loaded).

Diagnostics and Profiling

joblog / STEPUP_BUILD_JOBLOG / --joblog, --no-joblog

Set to true to record job-execution events to .stepup/joblog.csv, one row per event, with columns time_ns, job_i, event, description. The file is truncated and rewritten at the start of every build phase. Each job produces four events: - CREATED (by the scheduler), - STARTED and ENDED (by the executor), - COMPLETED (observed by the scheduler, freeing a slot for the next job). Comparing the timestamps across these events, and deriving the number of concurrently running jobs from them, helps diagnose scheduler/executor dispatch overhead.

log_level / STEPUP_BUILD_LOG_LEVEL / --log-level, -l

The level of the messages that the director writes to .stepup/director.log. Possible values are DEBUG, INFO, WARNING, ERROR, and CRITICAL. The default is WARNING, or DEBUG when STEPUP_DEBUG is set. The director also exports the level as STEPUP_BUILD_LOG_LEVEL to the steps it runs.

perf / STEPUP_BUILD_PERF / --perf

Set to a frequency in Hz to enable performance monitoring of the director process with the Linux perf profiler. Enabling it on any other platform is an error. See the section on Profiling in the development documentation for more details.

sqllog / STEPUP_BUILD_SQLLOG / --sqllog, --no-sqllog

Set to true to enable SQLite debug logging. Each execute() / executemany() call appends a timing row to .stepup/sqllog.csv as it happens. A .stepup/sqllog.json index (query text, call site, query plan, and the query_i id referenced by the CSV rows) is written when the director exits.

yappi / STEPUP_BUILD_YAPPI / --yappi, --no-yappi

Set to true to profile the director process with the Yappi profiler. See the section on Profiling in the development documentation for more details.

The targets to build (see Build Targets) are positional command-line arguments and cannot be set through config files or environment variables. When no targets are given, the full default workflow is built.

Settings for stepup clean

These settings are stored under the [clean] section in config files (or [tool.stepup.clean] in pyproject.toml). Each entry below lists the config file key, environment variable, and command-line option separated by slashes, where applicable.

all / STEPUP_CLEAN_ALL / --all, -a

Set to true to remove outputs of any step in the workflow, not just detached outputs (those for which no corresponding step exists anymore). Whenever a file is removed, outputs depending on it are also removed. The default is false.

commit / STEPUP_CLEAN_COMMIT / --commit, -c

Set to true to actually remove files and directories instead of performing a dry run. By default, stepup clean only prints what would be removed without deleting anything.

safe / STEPUP_CLEAN_SAFE / --unsafe, -u

Set to false to also remove output files that have been modified after their creation in the workflow. By default (true), modified files are skipped and reported as a warning. Note that the CLI flag is --unsafe, which is the negation of the config key safe.

The paths to consider for cleanup are positional command-line arguments and cannot be set through config files or environment variables. When no paths are given, the current directory is used.

Settings for stepup browse

These settings are stored under the [browse] section in config files (or [tool.stepup.browse] in pyproject.toml). Each entry below lists the config file key, environment variable, and command-line option separated by slashes, where applicable.

port / STEPUP_BROWSE_PORT / --port

The port number for the local web server that serves the build graph browser. The default is 8000. After starting, the server is accessible at http://localhost:<port>/.

Environment Variables in Step Execution

The following environment variables are set when a step executes. These are mainly relevant for StepUp’s internals, but can be useful for users implementing workflows. Note that anything in the step execution (sub)processes is also affected by the internal environment variables described above.

HERE and ROOT

These are documented in the tutorial on HERE and ROOT variables. They relate the step’s working directory to the directory where StepUp was started. They are not updated when a process changes its working directory, except by tools that explicitly do so for a child process. In Python, use get_here() and get_root() instead, which are always up to date.

STEPUP_DIRECTOR_I

An integer incremented every time the director starts, unless offline is set. A step that depends on this variable runs at every start of the director, which suits a step that looks at something StepUp does not watch, such as a remote server. With offline, the director keeps the value of the previous start. In watch mode, the value stays the same for the lifetime of the director, so such a step does not run again in later build phases of the same director process.

Interactions with remote data must be skipped when a step sees STEPUP_BUILD_OFFLINE=1, because even if STEPUP_DIRECTOR_I remains the same, a step may run for other reasons, such as a changed input or a failure in a previous run.

Declare the dependency with env="STEPUP_DIRECTOR_I" when the step is defined, or with getenv("STEPUP_DIRECTOR_I") from inside the step.

A step must not use the value of STEPUP_DIRECTOR_I for anything. It is only allowed to declare a dependency on it. The director increments the value before any step runs, and stores it in the workflow database. A step does not always see the value increase by 1 for several reasons, such as a failed dependency or an interrupted director. Furthermore, the value is reset to 1 when .stepup/graph.db is removed (manually or due to a change in database format).

The Remote Data tutorial shows an example.

STEPUP_JOB_I

A unique integer id for the current job running the step. Unlike a step’s own (stable) index, this changes every time the step is (re)started, e.g. after being deferred. This is mainly relevant for StepUp and has little significance for users implementing workflows.

STEPUP_STEP_INP_DIGEST

A hex-formatted digest of all the inputs to the step (including environment variables used). This is useful in special cases. For example, it can be used to decide if cached results from a previously interrupted run of the step are still valid. It can also be useful when a step submits a job to an external scheduler, to decide if a previously submitted job is still valid.

STEPUP_STEP_NEED

The declared need level of the currently executing step, as one of the strings OPTIONAL, DEFAULT, TARGET, or PLAN. This is used internally by StepUp to detect workflow authoring errors, such as registering a planning step (need=Need.PLAN) from inside a non-planning step.