StepUp Configuration¶
StepUp can be configured using configuration files, environment variables, and command-line options.
When a stepup tool starts, e.g. sb,
it will load its settings in the following order, with later settings overriding earlier ones:
/etc/stepup.toml(system-wide configuration file)~/.config/stepup.toml(user-wide configuration file)${STEPUP_ROOT}/.stepup.toml(project-specific configuration file)${STEPUP_ROOT}/stepup.toml(project-specific configuration file)${STEPUP_ROOT}/pyproject.toml(project-specific configuration file, with settings under the[tool.stepup]section)${STEPUP_ROOT}/stepup-local.toml(project-specific configuration file, intended for local overrides that are not committed to version control)- Environment variables (
STEPUP_*) - Command-line options
On Windows, ~ is the profile directory of the user, e.g. C:/Users/me,
and /etc/stepup.toml lies in the root of the current drive, e.g. C:/etc/stepup.toml.
Every setting belongs to a subcommand, e.g. build,
and is placed under a subcommand-specific section,
e.g. [build] (or [tool.stepup.build] in pyproject.toml).
The top level of a config file (or [tool.stepup] in pyproject.toml)
is reserved for settings shared by all subcommands, of which there are currently none.
Example stepup.toml:
Having multiple configuration files is convenient but can be confusing.
StepUp provides a stepup config tool to help you understand which settings are in effect.
This tool reads all the configuration files and environment variables,
and shows the merged settings as a single, informative TOML file,
including comments about the source of each setting.
Positional command-line arguments (targets to build or paths to clean) cannot be set through configuration files or environment variables.
Configuration Errors¶
A subcommand refuses to start when anything is wrong with a configuration file:
invalid TOML syntax, an unknown section, a key that is not a setting
of the section it appears in, or a value that a setting cannot use.
The value of an environment variable is checked in the same way,
but its name is not: see Unrecognized Environment Variables.
All problems are reported at once, so that a single run tells you everything to fix.
Each one names the config file the way stepup config does,
relative to the working directory when it lies below it:
$ sb
ERROR: Problems with the StepUp configuration:
./stepup.toml: unsupported key 'speed' in section [build]
./stepup.toml: unknown section [buidl] (did you mean 'build'?)
Run 'stepup config' to inspect the configuration.
This sets the first bit of the return code,
without a Python traceback unless STEPUP_DEBUG is set.
Problems are shown in red when the terminal supports color.
The stepup config subcommand is the one exception:
it still prints the configuration it loaded, with the source of each setting.
This makes it usable precisely when the configuration is broken.
Each problem is shown on the line of the setting, section or config file it concerns,
for the same stepup.toml as above:
# Config files (lowest to highest priority):
# MISSING: /etc/stepup.toml
# MISSING: ~/.config/stepup.toml
# MISSING: .stepup.toml
# FOUND: ./stepup.toml
# MISSING: ./pyproject.toml
# MISSING: ./stepup-local.toml
# Environment variables: STEPUP_*
[buidl] # <-- ERROR: ./stepup.toml: unknown section [buidl] (did you mean 'build'?)
jobs = 2 # ./stepup.toml
[build]
speed = 4 # ./stepup.toml <-- ERROR: unsupported key 'speed' in section [build]
The problems are written in comments, so the output remains valid TOML. A problem that has no line of its own is listed afterwards on standard error. This happens when a config file with a higher priority overrides the setting at fault, and when a line already carries another problem, because a second comment cannot be opened on the same line.
Unrecognized Environment Variables¶
A misspelled key in a config file is an error, but a misspelled STEPUP_* variable is not.
The STEPUP_* namespace is shared with the variables that configure StepUp’s internals,
listed under StepUp Core Module Environment Variables,
which no subcommand defines an option for.
Rejecting every name that is not a setting would therefore reject StepUp’s own environment.
To make a typo visible anyway, stepup config lists the STEPUP_* variables
in three groups, by what each of them does:
# Configuration environment variables:
# STEPUP_BUILD_JOBS = "4"
# StepUp Core module environment variables:
# STEPUP_ROOT = "/home/user/project"
# Unrecognized environment variables, without effect:
# STEPUP_BUILD_JBOS = "4"
When a setting does not take effect, the last group is the first place to look.
It holds every STEPUP_* variable that StepUp does not act on,
which for a variable you set deliberately means a misspelled name.
Variables of an extension package appear there as well
when they configure that package’s internals rather than one of its settings.
StepUp Core Module Environment Variables¶
Some environment variables affect StepUp’s internals even when it is just imported as a Python library. These can only be set via environment variables, and cannot be configured through config files or command-line options.
STEPUP_MAX_OUTPUT_SIZE-
The maximum size of standard output and standard error stored in the workflow database. The default is
0, meaning unlimited (no truncation). This limit only affects what is persisted, not the terminal output. When a stream exceeds the limit, it is truncated on a UTF-8 character boundary and a[output truncated at N bytes]line is appended. STEPUP_PATH_FILTER-
A list of filters for determining whether to retain or ignore an automatically detected dependency. The items are separated by a colon on POSIX systems and by a semicolon on Windows, as in
PATH. Each item starts with a+or-sign, followed by a path prefix to be used for matching. The items in the filter are processed in order, and the first match determines the action. If the matching path prefix is preceded by a-, the dependency is ignored. If it is preceded by a+, the dependency is retained and rewritten relative to${STEPUP_ROOT}. A path in the filter can be absolute or relative to${STEPUP_ROOT}, but matching is always done based on absolute paths. A prefix is matched as a plain string, not as a sequence of path components, so a filter item also matches paths whose first differing character is not a separator. For example,-venvignores dependencies undervenvand undervenv311, which is convenient when several virtual environments sit next to each other, but+dataalso retains dependencies underdatabase. A trailing separator does not restrict an item to one directory, because it is removed when the prefix is resolved to an absolute path.The default is
-.venv:-venv:-.tox:-.nox:-.direnv:-.pixi:-node_modules, which ignores the top-level directories in which tools install dependencies. Because every prefix is anchored at${STEPUP_ROOT}, the default cannot ignore an environment nested deeper in the tree, such aspackages/foo/.venv. Setting this variable replaces the default in full, so an environment directory has to be listed again when the variable is used to retain files outside${STEPUP_ROOT}.Regardless of whether a filter is defined, the filters
:+.:-/are always appended.This feature can be used for several purposes:
-
You may have source files that are not part of the StepUp project, but are used in the project and edited frequently. In this case, steps that depend on these external files will be rerun when you change the external source files.
-
You have a virtual environment with many packages installed, but you don’t want to include them in the dependency graph for performance reasons. (The default filter already does this for the most common environment directories.)
-
STEPUP_ROOT-
The root directory containing the top-level
plan.pyfile. If not set, StepUp will look for this file in the current directory. STEPUP_RPC_TRANSPORT-
The kind of socket through which
stepup build, the director, the steps and the otherstepupsubcommands talk to each other. This is a setting for developers of StepUp, to test both transports on Linux and macOS. End users can leave it unset.unixselects Unix domain sockets, which is the default on platforms that have them.tcpis the default and the only accepted value on Windows. Withtcp, the director andstepup buildlisten on TCP sockets on the loopback interface (127.0.0.1). Every local user can connect to such a socket, so a client must first send a random token, which it reads from a file that only the user runningstepup buildcan access. Whoever obtains this token can run code as that user.On Windows, this file is exactly as private as the temporary directory in which StepUp creates it. Python 3.12.4 and later, and 3.11 releases from 3.11.10 on, give that directory access rules of its own, which only admit its owner, SYSTEM and administrators. Older versions let it inherit the access rules of the directory named by
TEMP(CVE-2024-4030 of Python). These rules are private by default, becauseTEMPlies in the profile of the user, but not whenTEMPis set to a directory that other users can read, e.g. a directory in the root of a drive, such asC:\Temp. On Windows, use Python 3.12.4 or later, or a 3.11 release from 3.11.10 on, and in particular whenTEMPhas been changed.Only
stepup buildreads this variable: the steps and the other subcommands find out which transport is in use by themselves. STEPUP_SHELL-
The POSIX shell that runs the commands of steps created with
shell=True, and local shell scripts on platforms that cannot execute them directly. The value is the path of the shell, or its name, which is then looked up inPATH. StepUp starts it asSHELL -c COMMAND. The default is/bin/shon Linux and macOS. WhenSTEPUP_SHELLis not set on Windows, StepUp uses the first of:shonPATH, unless it lies in the Windows directory, which holds the launchers of WSL.bin\sh.exeof the Git installation whosegitis onPATH.bin\sh.exeof the installation of Git for Windows that is recorded in the registry, for the current user first and for all users second.
StepUp only searches when a step needs a shell, and when nothing is found, that step fails with a message that lists what was tried. See POSIX Shell on Windows for the shells that can be used. Only
stepup buildand steps that callrun_subprocess()withshell=Trueread this variable. Changing it does not make StepUp rerun steps. STEPUP_SYNC_RPC_TIMEOUT-
The number of seconds a synchronous RPC client waits for a response from the director, which also bounds the connection setup that the first call triggers. The default is 600 seconds. Set this to a smaller value if you want to detect deadlocks more quickly. A negative or zero value disables the timeout, which some calls use because the director answers them only when the workflow is ready for it.
STEPUP_WATCHER-
The library that watches the file system in watch mode, i.e. with the
--watchor--watch-firstoption ofstepup build. This is a setting for developers of StepUp, to test both libraries on Linux. End users can leave it unset. On Linux, the default isasyncinotify, andwatchfilescan be selected after installing it withpip install 'stepup[watchfiles]'. On other platforms,watchfilesis the default and the only accepted value. Without watch mode, this variable is ignored. STEPUP_WATCHER_RECURSIVE-
How the
watchfileslibrary watches the directories below the one in which StepUp runs:1for a single recursive watch,0for a separate watch on every directory of interest. This is a setting for developers of StepUp, to test both ways on every platform. End users can leave it unset. The default is1on Windows, where a watch on a directory makes it impossible to rename its parent directories, and0on other platforms. A watched directory outside the directory in which StepUp runs always gets a watch of its own. This variable is ignored without watch mode and whenSTEPUP_WATCHERisasyncinotify.On Linux, a recursive watch follows symbolic links, and removing a symbolic link to a watched directory stops the reports of changes in that directory, and possibly in the directories below it, until StepUp is restarted. This was observed with watchfiles 1.3.0, and is caused by a bug in the Rust library
notifyon which watchfiles builds, see notify#1038.
Settings for All Subcommands¶
Each entry below lists the config file key, environment variable, and command-line option separated by slashes, where applicable.
STEPUP_DEBUG-
Set to
1to enable debugging features and strict consistency checks. This changes the default log level ofstepup buildtoDEBUG(seelog_levelbelow) and will require internal consistency checks to pass, rather than applying corrections to overcome the inconsistencies. (Every such inconsistency is due to a bug, which should be fixed eventually.) It also makes the scan of.stepup/director.logat the end of a build fatal: a logged error or a coroutine, task or thread left dangling is reported as an error and sets the internal error bit of the return code, instead of only being reported as a warning. It disables the shortening of error reports described in Failing Steps: a failing step then prints its complete traceback, including StepUp’s own frames and the traceback of the director process. A mistake that any subcommand reports as a shortERROR:message keeps its traceback, which is the only way to find out where it was raised. Finally, when aplan.pyor a console script runs outside StepUp, the functions ofstepup.core.apiprint the calls they would send to the director on standard error, instead of ignoring them silently. This variable cannot be set through config files or command-line options.
Settings for sb or stepup build¶
These settings are stored under the [build] section in config files
(or [tool.stepup.build] in pyproject.toml).
Each entry below lists the config file key, environment variable, and command-line option
separated by slashes, where applicable.
The settings are grouped as in the output of stepup build --help,
alphabetically within each group.
Build Control¶
clean/STEPUP_BUILD_CLEAN/--clean,--no-clean-
Set to
falseto disable automatic cleaning of outdated output files. By default, StepUp automatically removes old output files that are no longer created by any step in the workflow. duration/STEPUP_BUILD_DURATION/--duration,--no-duration-
Set to
falseto disable recording of step wall times as theirdurationfor future runs. StepUp uses the duration information to prioritize the execution of steps with the longest critical path (in time units) to any terminal node. (This is stored in the database as the_tail_timefield of each step.) While accurate durations are generally beneficial, they can result in non-deterministic execution order, which can be undesirable in some cases, such as testing.When disabled, durations remain unchanged from the previous run or stay at their initial value if the step has never been executed before. The initial value can be provided when defining the step, or defaults to
1.0if not provided. With the1.0default, the_tail_timeof a step degrades to the number of steps in the longest path to any terminal node. explain_rerun/STEPUP_BUILD_EXPLAIN_RERUN/--explain-rerun,-e,--no-explain-rerun-
Set to
trueto explain for every step with recording info why it cannot be skipped. fix_epoch/STEPUP_BUILD_FIX_EPOCH/--fix-epoch,--no-fix-epoch-
If set to
true(the default), theSOURCE_DATE_EPOCHenvironment variable will be set to a fixed value of315532800(corresponding to 1980-01-01 00:00:00 UTC) for all step executions. This is useful for ensuring reproducible builds: Many tools and libraries recognizeSOURCE_DATE_EPOCHand use it instead of the current time for timestamps in generated files. If the variable is already set in the environment, it will be used as-is. jobs/STEPUP_BUILD_JOBS/--jobs,-j-
The maximum number of steps to run concurrently. When given as a floating point number, the value is multiplied by the number of available CPU cores. The default is
1.0. keep_going/STEPUP_BUILD_KEEP_GOING/--keep-going,-k,--no-keep-going-
Set to
trueto keep dispatching new steps after another step has failed, as long as their own inputs remain available (likemake -k). By default (false), the scheduler starts draining after the first failure: steps already running are still allowed to finish, but no new steps are started. defer_cap/STEPUP_BUILD_DEFER_CAP/--defer-cap-
Maximum number of times a step can be deferred (since it last succeeded) before it is reported as failed instead of parked pending again. This guards against livelocks where a step’s dynamic inputs keep flip-flopping. The default is
100, deliberately generous. offline/STEPUP_BUILD_OFFLINE/--offline,--no-offline-
Set to
truewhen remote data cannot be reached, e.g. without a network connection.STEPUP_DIRECTOR_Ithen keeps the value of the previous start, so a step that runs at every start of the director does not run merely because StepUp started. Such a step keeps its result of the previous start, and so do the steps that use its outputs. A step that must run for another reason, e.g. because an input changed or because it failed, runs as usual. The director also exports the setting asSTEPUP_BUILD_OFFLINEto the steps it runs, with the value1or0, so that a step that does run can avoid the network, as described in Steps That Use the Network. The next start without this setting givesSTEPUP_DIRECTOR_Ia new value again, so every step that runs at every start looks at the remote data again. In watch mode, the setting holds for the lifetime of the director. Working Offline shows an example. progress/STEPUP_BUILD_PROGRESS/--progress,--no-progress-
Set to
falseto disable the progress bar in the terminal user interface. This can be useful to simplify and reduce the output. resources/STEPUP_BUILD_RESOURCES/--resources,-r-
A comma-separated list of resource names and available quantities to be used for scheduling decisions. For example,
resources = "gpu:2,cpu:4"indicates that there are 2 GPUs and 4 CPUs available. Any resource labels can be used, and the available quantity can be any positive integer. Note that resource specifications from config files, the environment variable, and one ore more CLI options (from left to right, e.g.-r gpu:2 -r cpu:4) are merged together. watch/STEPUP_BUILD_WATCH/--watch,-w,--no-watch-
Set to
trueto enable watch mode. In watch mode, StepUp will monitor the file system for changes and rerun affected steps after pressing therkey in the terminal user interface. watch_first/STEPUP_BUILD_WATCH_FIRST/--watch-first,-W,--no-watch-first-
Set to
trueto automatically rerun affected steps when relevant file changes are observed, without needing to press therkey. This implieswatch = true.
Execution Environment¶
cgroup/STEPUP_BUILD_CGROUP/--cgroup,--no-cgroup-
This setting controls whether StepUp will run the director (and all its child processes running steps) use cgroup isolation. When enabled, peak memory usage of the director process and all its child processes is measured. Only available on Linux with cgroup v2 enabled and if
systemd-runis available. Exceptions are raised if cgroup isolation is requested but not working. Off by default. forkserver/STEPUP_BUILD_FORKSERVER/--forkserver,--no-forkserver-
Set to
trueto use a forkserver for Python step execution, which reduces startup overhead. This is enabled by default on Linux. Enabling it on a platform without the forkserver start method, such as Windows, is an error. preload_modules/STEPUP_BUILD_PRELOAD_MODULES/--preload-modules-
A comma-separated list of Python modules to pre-load into the forkserver. Only has effect when
forkserver = true. Use this to reduce per-step startup time when all (or most) steps import the same large modules. For example,preload_modules = "numpy,scipy"pre-loads NumPy and SciPy into the forkserver so that each Python step forked from it inherits them at zero import cost. By default, no additional modules are pre-loaded (only internal StepUp modules are pre-loaded).
Diagnostics and Profiling¶
joblog/STEPUP_BUILD_JOBLOG/--joblog,--no-joblog-
Set to
trueto record job-execution events to.stepup/joblog.csv, one row per event, with columnstime_ns,job_i,event,description. The file is truncated and rewritten at the start of every build phase. Each job produces four events: -CREATED(by the scheduler), -STARTEDandENDED(by the executor), -COMPLETED(observed by the scheduler, freeing a slot for the next job). Comparing the timestamps across these events, and deriving the number of concurrently running jobs from them, helps diagnose scheduler/executor dispatch overhead. log_level/STEPUP_BUILD_LOG_LEVEL/--log-level,-l-
The level of the messages that the director writes to
.stepup/director.log. Possible values areDEBUG,INFO,WARNING,ERROR, andCRITICAL. The default isWARNING, orDEBUGwhenSTEPUP_DEBUGis set. The director also exports the level asSTEPUP_BUILD_LOG_LEVELto the steps it runs. perf/STEPUP_BUILD_PERF/--perf-
Set to a frequency in Hz to enable performance monitoring of the director process with the Linux perf profiler. Enabling it on any other platform is an error. See the section on Profiling in the development documentation for more details.
sqllog/STEPUP_BUILD_SQLLOG/--sqllog,--no-sqllog-
Set to
trueto enable SQLite debug logging. Eachexecute()/executemany()call appends a timing row to.stepup/sqllog.csvas it happens. A.stepup/sqllog.jsonindex (query text, call site, query plan, and thequery_iid referenced by the CSV rows) is written when the director exits. yappi/STEPUP_BUILD_YAPPI/--yappi,--no-yappi-
Set to
trueto profile the director process with the Yappi profiler. See the section on Profiling in the development documentation for more details.
The targets to build (see Build Targets) are positional command-line arguments and cannot be set through config files or environment variables. When no targets are given, the full default workflow is built.
Settings for stepup clean¶
These settings are stored under the [clean] section in config files
(or [tool.stepup.clean] in pyproject.toml).
Each entry below lists the config file key, environment variable, and command-line option
separated by slashes, where applicable.
all/STEPUP_CLEAN_ALL/--all,-a-
Set to
trueto remove outputs of any step in the workflow, not just detached outputs (those for which no corresponding step exists anymore). Whenever a file is removed, outputs depending on it are also removed. The default isfalse. commit/STEPUP_CLEAN_COMMIT/--commit,-c-
Set to
trueto actually remove files and directories instead of performing a dry run. By default,stepup cleanonly prints what would be removed without deleting anything. safe/STEPUP_CLEAN_SAFE/--unsafe,-u-
Set to
falseto also remove output files that have been modified after their creation in the workflow. By default (true), modified files are skipped and reported as a warning. Note that the CLI flag is--unsafe, which is the negation of the config keysafe.
The paths to consider for cleanup are positional command-line arguments and cannot be set through config files or environment variables. When no paths are given, the current directory is used.
Settings for stepup browse¶
These settings are stored under the [browse] section in config files
(or [tool.stepup.browse] in pyproject.toml).
Each entry below lists the config file key, environment variable, and command-line option
separated by slashes, where applicable.
port/STEPUP_BROWSE_PORT/--port-
The port number for the local web server that serves the build graph browser. The default is
8000. After starting, the server is accessible athttp://localhost:<port>/.
Environment Variables in Step Execution¶
The following environment variables are set when a step executes. These are mainly relevant for StepUp’s internals, but can be useful for users implementing workflows. Note that anything in the step execution (sub)processes is also affected by the internal environment variables described above.
HEREandROOT-
These are documented in the tutorial on
HEREandROOTvariables. They relate the step’s working directory to the directory where StepUp was started. They are not updated when a process changes its working directory, except by tools that explicitly do so for a child process. In Python, useget_here()andget_root()instead, which are always up to date. STEPUP_DIRECTOR_I-
An integer incremented every time the director starts, unless
offlineis set. A step that depends on this variable runs at every start of the director, which suits a step that looks at something StepUp does not watch, such as a remote server. Withoffline, the director keeps the value of the previous start. In watch mode, the value stays the same for the lifetime of the director, so such a step does not run again in later build phases of the same director process.Interactions with remote data must be skipped when a step sees
STEPUP_BUILD_OFFLINE=1, because even ifSTEPUP_DIRECTOR_Iremains the same, a step may run for other reasons, such as a changed input or a failure in a previous run.Declare the dependency with
env="STEPUP_DIRECTOR_I"when the step is defined, or withgetenv("STEPUP_DIRECTOR_I")from inside the step.A step must not use the value of
STEPUP_DIRECTOR_Ifor anything. It is only allowed to declare a dependency on it. The director increments the value before any step runs, and stores it in the workflow database. A step does not always see the value increase by 1 for several reasons, such as a failed dependency or an interrupted director. Furthermore, the value is reset to1when.stepup/graph.dbis removed (manually or due to a change in database format).The Remote Data tutorial shows an example.
STEPUP_JOB_I-
A unique integer id for the current job running the step. Unlike a step’s own (stable) index, this changes every time the step is (re)started, e.g. after being deferred. This is mainly relevant for StepUp and has little significance for users implementing workflows.
STEPUP_STEP_INP_DIGEST-
A hex-formatted digest of all the inputs to the step (including environment variables used). This is useful in special cases. For example, it can be used to decide if cached results from a previously interrupted run of the step are still valid. It can also be useful when a step submits a job to an external scheduler, to decide if a previously submitted job is still valid.
STEPUP_STEP_NEED-
The declared need level of the currently executing step, as one of the strings
OPTIONAL,DEFAULT,TARGET, orPLAN. This is used internally by StepUp to detect workflow authoring errors, such as registering a planning step (need=Need.PLAN) from inside a non-planning step.