Supported Platforms¶
StepUp runs on Linux, macOS and Windows. It is developed on Linux, which also has the most features. This page explains what StepUp does the same on every platform, which features are limited to some of them, and what you have to take care of for a workflow to run on all of them.
What StepUp Does the Same Everywhere¶
StepUp is platform independent in everything it does itself:
- A path has
/as its only separator, in the workflow graph, in the results of the API and in the variablesHEREandROOT. copy()copies a file without starting a program.- A local script with a shebang line runs with the interpreter on that line, even when no POSIX shell is installed.
- The output of a step is recorded as UTF-8 with
\nline endings. - The files that StepUp writes are UTF-8 with
\nline endings. This includes the files in.stepup/,graph.txtand the files written bydumpns()andrender_jinja().
Whatever happens outside the plan.py files, e.g. in shell commands and scripts,
is for you to make reproducible across platforms,
because StepUp does not change what the programs of a step do.
The differences you are most likely to run into are:
-
Line endings and encodings of the files that scripts write. On Windows, Python ends the lines of a text file with
\r\nby default, and Python 3.14 and older write it in the ANSI code page, e.g.cp1252. Open text files withopen(path, "w", encoding="utf-8", newline="\n"). When a shell redirects the output of a Python script to a file, e.g../work.py > out.txt, the script has to configure its standard output in the same way: -
Line endings of files in Git. On Windows, Git converts the line endings of text files to
\r\nwhen it checks them out, unless it is told otherwise. A shell script with\r\nline endings fails in bash, and every converted file has another digest and size than on Linux and macOS. Add a file.gitattributeswith the following line to the root of the repository:If you only want to protect shell scripts, use
*.sh text eol=lfinstead. -
Executable bits in Git. Windows has no executable bit, so a script that you add to Git on Windows is not executable after a checkout on Linux or macOS. Mark it with
git add --chmod=+x script.py. - Case of file names.
The default file systems of macOS and Windows do not distinguish upper and lower case,
while StepUp compares paths as strings.
For StepUp,
Data.csvanddata.csvare two different files, even when they are the same file on disk. Spell each path the same way throughout a workflow, and do not create files whose names only differ in case. - The programs of shell commands.
A shell command runs the programs that its shell finds on
PATH: the GNU tools on Linux, the BSD tools on macOS, and on Windows those of the shell you installed. They do not all accept the same options, e.g.sed -ineeds an argument on macOS.
Features per Platform¶
| Feature | Linux | macOS | Windows |
|---|---|---|---|
Shell for shell=True |
/bin/sh |
/bin/sh |
Git for Windows or another |
| Watch mode | inotify | watchfiles | watchfiles |
| Forkserver for Python steps | on by default | off by default | not available |
--cgroup |
with cgroup v2 and systemd-run |
not available | not available |
--perf |
available | not available | not available |
Suspending with Ctrl+Z |
available | available | not available |
| Connections between processes | Unix domain sockets | Unix domain sockets | TCP on 127.0.0.1 |
Copies made by copy() |
share data where possible, keep holes | keep holes | plain copy |
Enabling a feature on a platform that does not have it is an error, which StepUp reports before the build starts. The libraries for watch mode are compared in How Watch Mode Observes Changes, suspending a build is explained in Suspending a Build, and the other features are explained in Configuration.
Linux¶
Linux needs nothing beyond the installation.
WSL counts as Linux,
with one limitation in watch mode:
WSL 2 does not report changes to files on a Windows drive, such as /mnt/c,
when Windows programs make them.
This is a limitation of WSL 2 itself,
which Microsoft tracks in microsoft/WSL#4739,
and StepUp has no way around it.
Keep a project that you edit with Windows programs in the file system of WSL,
or run StepUp on Windows itself.
macOS¶
macOS needs nothing beyond the installation.
Shell commands run with /bin/sh, and they find the BSD versions of the usual Unix tools,
which differ in some options from the GNU versions on Linux.
Windows¶
StepUp runs on Windows natively, without WSL. Shell steps need a POSIX shell, see POSIX Shell on Windows, and the installation lists the minimal version of Python for Windows, below which other users of the computer may be able to connect to a running build.
Paths¶
StepUp writes paths with /, also on Windows, e.g. C:/Users/me/project/data.csv,
always with an upper-case drive letter
and without the prefix \\?\ of an extended-length path.
Windows accepts / in all its file functions,
so these paths can be passed to open() and to every other function of Python.
All paths that StepUp returns are instances of Path
from stepup.core.path, which keeps this spelling when you join or change them.
Import Path from stepup.core.path instead of from the path library,
because the latter gives mixed separators on Windows:
path.Path("data/sub") / "file.txt" is data/sub\file.txt.
You will notice this convention in the following situations:
-
Comparing paths. A path from StepUp is not equal to
str()of apathlib.Pathof the same file, which has backslashes on Windows. Convert the latter withPathfromstepup.core.pathbefore comparing them, or comparepathlib.Pathobjects: -
Paths in commands. Embed a path in a command with
shq(), which writes it with/on every platform and quotes it when needed. A command is text, which StepUp passes on exactly as you wrote it. Its words are split with the rules of a POSIX shell, also when it runs without a shell, and in those rules a backslash escapes the next character. A native path that you put into a command with an f-string therefore loses its backslashes, without any error:import pathlib from stepup.core.api import run, shq src = pathlib.Path("data") / "table.csv" # Wrong: on Windows, analyze.py receives the argument datatable.csv. run(f"./analyze.py {src}", inp=src) # Right: analyze.py receives data/table.csv on every platform. run(f"./analyze.py {shq(src)}", inp=src) -
Programs that need backslashes.
os.path.normpath(path)gives the native spelling of a path, for a program that needs it. Put such a path into a command withshlex.quote(), which keeps its backslashes, and not withshq(), which writes/again. - Lists of paths.
The items of
STEPUP_PATH_FILTERand of a variable read withgetenv(..., multi=True)are separated by;on Windows, as inPATH, because:is part of a path likeC:/data. On Linux and macOS, the separator is:.
Commands Without a Shell¶
On Linux and macOS, the operating system finds the program of a command and reads the shebang line of a script. Windows does neither, so StepUp does both itself and follows the behavior of Linux and macOS where it can:
- A program without a directory, e.g.
typst, is looked up on thePATHof the step, with the extensions ofPATHEXT, e.g.typst.exe. Unlike incmd.exe, it is never looked up in the current directory. Write./tool.exefor a program there. - A program with a directory, e.g.
./tool, is looked up in the working directory of the step, with an extension ofPATHEXTwhen the file does not exist without one. - A batch file (
*.bator*.cmd) runs withcmd.exe, and StepUp quotes its arguments so that they arrive unchanged. An argument with a line break cannot be passed to a batch file, and makes the step fail. - A file whose name has none of the extensions in
PATHEXTand that starts with a shebang line runs with the interpreter on that line. StepUp only uses the name of that interpreter, without its directory, and supports#!/bin/NAME,#!/usr/bin/env NAMEand#!/usr/bin/env -S NAME ARGS. As on Linux, everything afterNAMEis a single argument, except withenv -S. The other options ofenvare not supported. The namespythonandpython3stand for the Python that runs StepUp. The namesshandbashstand for the POSIX shell. Any other name is looked up on thePATHof the step.
A script whose name ends in .py runs with the Python that runs StepUp, on every platform.
Shell Commands¶
A shell command runs in the POSIX shell that StepUp finds, which is usually the bash of Git for Windows. That bash behaves differently from bash on Linux in a few ways:
- It converts arguments that look like POSIX paths to Windows paths
when it starts a program that is not part of Git or MSYS2, such as
python.exe. For example,/tmp/xbecomesC:/Users/me/AppData/Local/Temp/xand--prefix=/usrbecomes--prefix=C:/Program Files/Git/usr. Relative paths and paths that start with a drive letter stay as they are, so the paths of StepUp are not affected. This conversion is what allows the result of e.g.mktemp -dto be passed to a Windows program. To pass such an argument unchanged, setMSYS_NO_PATHCONV=1for that one command, e.g.MSYS_NO_PATHCONV=1 ./tool.exe /opt. - It drops every carriage return from a command, also inside quotes.
A shell command with a literal
\rtherefore does something else than on Linux and macOS. -
It has no
python3. A virtual environment on Windows has apython.exeand nopython3.exe. StepUp starts a script with apython3shebang line with its own Python, but when a shell starts the same script, e.g. in the shell step./work.py > out.txt, the shell looks forpython3onPATHand fails. StepUp does not provide this program, because what a shell command needs is up to you. The following lines, run in Git Bash with the virtual environment activated, create a wrapper namedpython3in the directory of the environment’spython.exe:bindir="$(dirname "$(type -P python)")" printf '#!/bin/sh\nexec python "$@"\n' > "${bindir}/python3"The wrapper runs the first
pythononPATH, which is that of the virtual environment while it is activated. The wrapper can also be placed in any other directory on thePATHofstepup build.
Stopping a Build¶
A Windows console has no signals as Linux and macOS know them, which changes how a build is stopped:
Ctrl+CandCtrl+Breakabort a build.Ctrl+Zdoes not suspend a build. The keyboard commands only work when StepUp runs in a Windows console.- A step that is interrupted ends at once, without cleaning up:
a Python step gets no
KeyboardInterrupt, and atrapin a shell step does not run. - Processes that a step leaves running end together with
stepup build. - When
stepup buildis killed, everything it started ends at once.
The details are in Using StepUp in a Windows Console.
Files¶
- A file that another program holds open cannot be overwritten or removed. This is often the case for an output that is shown in a viewer, e.g. a PDF file. A step that writes such a file then fails, and so does the removal of an output that is no longer needed. Close the file, or use a viewer that does not hold the files it shows open.
- Paths are limited to 260 characters, unless long paths are enabled in Windows, which takes administrator rights. This limit includes the directory of the project. Even with long paths enabled, a program that a step runs may not support them.
- Windows has no permission bits.
Every file counts as executable, so a script needs no
chmod +x, but it still needs its shebang line. - Creating a symbolic link takes a privilege, which only administrators have by default, and every user when the Developer Mode of Windows is on.
Output and Encodings¶
- A Python step on Python 3.14 and older writes its output in the ANSI code page,
unless it is configured otherwise.
StepUp reads the output of every step as UTF-8 and drops what is not valid UTF-8,
so non-ASCII characters may be missing from the output that StepUp shows.
Set the environment variable
PYTHONUTF8=1before you startstepup build, so that Python steps write UTF-8. Python 3.15 writes UTF-8 by default. - The output of the
stepupcommand is UTF-8 with\nline endings when it is redirected to a file or a pipe. The PowerShell 5.1 that comes with Windows decodes the output of every program with[Console]::OutputEncodingand writes redirected output in UTF-16, sostepup build > out.txtgives a garbled file there. Redirect incmd.exe, in Git Bash or in PowerShell 7.4 or later instead, or set[Console]::OutputEncoding = [System.Text.Encoding]::UTF8in PowerShell 5.1, which gets the characters right in a file that is still UTF-16.