Remote Data¶
StepUp is designed to work with files on disk (and environment variables) that it can see. It cannot, by itself, check data on a remote server, or any other resource that it cannot see. However, sometimes a workflow needs to receive such data, or send it. StepUp’s workaround is to run certain steps unconditionally, even if their visible dependencies did not change, so the step itself can interact with the remote resource and act when needed. This tutorial shows how to make a step run at every start.
This feature of StepUp is mainly for developers of extension packages, such as StepUp RepRep. However, it can also be used as such in an ordinary workflow.
No dependencies through remote resources
StepUp does not track dependencies through remote resources. They are completely opaque to StepUp and the mechanism below does not even assume they exist. For example, if step A sends data to a remote, and step B fetches that remote data again, StepUp will not know that B depends on A. It may thus run them in the wrong order, and it does not make B pending when the upload of A has changed. This means that interaction with remote resources must sit at the edges of a workflow: initial data coming in or final data going out.
The Variable STEPUP_DIRECTOR_I¶
The director sets the environment variable STEPUP_DIRECTOR_I for every step it runs.
Its value is a unique integer every time the director starts.
A step can depend on this variable like on any other one,
as explained in the tutorial on Environment Variables:
with env="STEPUP_DIRECTOR_I" when the step is defined,
or with getenv("STEPUP_DIRECTOR_I") from inside the step.
At every start, StepUp finds a new value of the variable,
so it runs such a step again instead of skipping it.
Like any other step, it only runs when it is needed and when its inputs are available.
In watch mode, the value stays the same for the lifetime of the director,
so the step does not run again in later build phases because of this variable.
The value does not identify a start,
because a start with --offline keeps the value of the previous one,
and a new .stepup/graph.db starts again at 1.
The variable is only meant to be tracked,
so a step must not use its value for anything else, e.g. as a key for data outside .stepup/.
The reference
describes the variable in full.
Example: A Download Step¶
Example source files: docs/advanced_topics/remote_data/
In this example, the directory remote/ stands in for a remote server, so no network is needed.
No step declares this directory, so StepUp does not see when it changes,
just as it cannot see changes on a real server.
Create it with some data:
The following plan.py defines a download step that runs at every start,
and a step that summarizes the downloaded data:
#!/usr/bin/env python3
from stepup.core.api import run, static
static("download.py", "report.py")
run("./download.py", env="STEPUP_DIRECTOR_I", out="data.txt")
run("./report.py", inp="data.txt", out="report.txt")
The download step compares the checksum of the remote file with that of its output,
and copies the file only when they differ.
The part of main() that handles offline mode is explained in Working Offline.
#!/usr/bin/env python3
"""Download `data.txt` from the remote, unless the local copy is up to date."""
import hashlib
import os
import shutil
import sys
REMOTE = "remote/data.txt"
LOCAL = "data.txt"
def checksum(path: str) -> str | None:
"""Return the SHA-256 checksum of a file, or `None` if it does not exist."""
try:
with open(path, "rb") as fh:
return hashlib.file_digest(fh, "sha256").hexdigest()
except FileNotFoundError:
return None
def main():
local = checksum(LOCAL)
if os.environ.get("STEPUP_BUILD_OFFLINE") == "1":
if local is None:
sys.exit(f"Working offline, so {LOCAL} cannot be downloaded.")
print(f"Working offline, keeping {LOCAL}.")
return
# This mimics the usage of a research-grade data server, like iRODS,
# which can report the checksum, so downloads of unchanged data can be skipped.
if checksum(REMOTE) != local:
shutil.copyfile(REMOTE, LOCAL)
if __name__ == "__main__":
main()
The script report.py writes the summary:
#!/usr/bin/env python3
"""Summarize the downloaded data."""
with open("data.txt", encoding="utf-8") as fh:
values = [float(line) for line in fh]
with open("report.txt", "w", encoding="utf-8", newline="\n") as fh:
print(f"Number of values: {len(values)}", file=fh)
print(f"Average: {sum(values) / len(values):.2f}", file=fh)
Make the scripts executable and run StepUp:
The first build runs all steps:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
STARTUP │ (Re)initialized boot script
PHASE │ build
START │ ./plan.py
SUCCESS │ ./plan.py
START │ ./download.py
SUCCESS │ ./download.py
START │ ./report.py
SUCCESS │ ./report.py
DIRECTOR │ Ran 3 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
Run sb -j 1 again without changing anything.
The UPDATED line shows that STEPUP_DIRECTOR_I has a new value, so the download step runs,
but it finds that data.txt is up to date and leaves it untouched.
StepUp then checks report.py, finds that its input did not change, and skips it:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
UPDATED │ STEPUP_DIRECTOR_I ='1' ➜ ='2'
STARTUP │ Checking 5 file(s) for changes
PHASE │ build
START │ ./download.py
SUCCESS │ ./download.py
SKIP │ ./report.py
DIRECTOR │ Ran 1 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
Now add a value to the remote data and run StepUp again:
The download step copies the new data, so report.py runs as well:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
UPDATED │ STEPUP_DIRECTOR_I ='2' ➜ ='3'
STARTUP │ Checking 5 file(s) for changes
PHASE │ build
START │ ./download.py
SUCCESS │ ./download.py
START │ ./report.py
SUCCESS │ ./report.py
DIRECTOR │ Ran 2 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
A next build without changes skips report.py again:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
UPDATED │ STEPUP_DIRECTOR_I ='3' ➜ ='4'
STARTUP │ Checking 5 file(s) for changes
PHASE │ build
START │ ./download.py
SUCCESS │ ./download.py
SKIP │ ./report.py
DIRECTOR │ Ran 1 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
Only the download step runs in every build,
and StepUp skips everything downstream as long as the downloaded file remains the same.
The comparison of checksums in download.py avoids repeating the download itself,
which matters for large files.
If the remote server does not support checksums, the download must be repeated every time,
and StepUp will still skip report.py when the file is unchanged.
A download step that learns its outputs from the remote
declares them with amend(out=...) in every run,
also when it downloads nothing, because StepUp drops them when the step runs again.
Rules for Steps That Run at Every Start¶
-
Fast and bounded when there is nothing to do. The step runs in every build, so its check delays every build. The check costs at most a few requests, each with a timeout. Do not touch the network when there is nothing to check, e.g. when credentials are absent. Fail when credentials are present and the remote cannot be reached, because the user asked for something that cannot be done. With
STEPUP_BUILD_OFFLINE=1, do not touch the network at all, as explained in Working Offline. -
No secrets in what StepUp stores or shows. Read a secret, such as an access token, with
os.environ, never with StepUp’sgetenv(),step(env=...)oramend(env=...). StepUp stores the value of every variable that a step depends on in.stepup/graph.db, and shows how it changed, at startup and with--explain-rerun, e.g. asUPDATED │ MY_TOKEN ='old' ➜ ='new'. Likewise, never write a secret to an output or to the standard output or error. -
Undeclared state stays private. A step that runs at every start may read a secret, the remote,
STEPUP_BUILD_OFFLINEor files of its own. It does not need to declare them, because StepUp starts it again anyway.That said, an always-run step may declare dependencies to StepUp, so it runs at the right time in the workflow.
-
Throttled where needed. Limit the number of steps that use the same server at the same time with resources, because a server may refuse or slow down too many connections at once.
Working Offline¶
Without network access, a step that looks at the remote at every start fails.
The offline setting, e.g. sb --offline,
lets such steps keep the results of the previous start:
STEPUP_DIRECTOR_Ikeeps its value of the previous start, so no step runs merely because StepUp started. A step that runs at every start keeps its result of the previous start, and so does everything downstream of it.- A step that runs anyway, because something else changed,
finds
STEPUP_BUILD_OFFLINE=1in its environment, and does not touch the network. It runs, for example, when its script changed, or when it failed at the previous start because the server could not be reached. - The next start without
--offlinegivesSTEPUP_DIRECTOR_Ia new value, so every step that runs at every start looks at the remote again.
To see this in the example, change the remote data and run StepUp offline:
No step runs, so the change of the remote data goes unnoticed:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
DIRECTOR │ Working offline, STEPUP_DIRECTOR_I stays at 4
STARTUP │ Checking 5 file(s) for changes
PHASE │ build
DIRECTOR │ Ran 0 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
The next start without --offline downloads the new data, and report.py runs:
DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.2.0)
UPDATED │ STEPUP_DIRECTOR_I ='4' ➜ ='5'
STARTUP │ Checking 5 file(s) for changes
PHASE │ build
START │ ./download.py
SUCCESS │ ./download.py
START │ ./report.py
SUCCESS │ ./report.py
DIRECTOR │ Ran 2 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!
When the download step does run offline,
the first lines of its main() function keep data.txt and say so in the output.
When data.txt does not exist, the step fails with a message that says why,
instead of a network error after a timeout.
The download step reads STEPUP_BUILD_OFFLINE with os.environ, without declaring it.
The step runs at every start without --offline,
so a result that it produced offline is never kept online.
Declaring the variable would make the step run at the first start with --offline,
because the variable changed from 0 to 1,
which is exactly what offline mode avoids.
A step that does not run at every start and behaves differently offline
declares the variable, like any variable that it reads.
Steps That Use the Network
asks every extension package to honor STEPUP_BUILD_OFFLINE in this way.
Try the Following¶
-
Change the docstring of
download.pyand runsb -j 1 --offline. The download step runs, because its script changed, and prints that it keepsdata.txt. StepUp then skipsreport.py. -
Remove
data.txtand runsb -j 1 --offline. The download step fails, because it cannot download the data offline. Runsb -j 1to download the data again.
Limits¶
- Once per start, not per build phase.
In watch mode, a step that runs at every start does not run again in later build phases,
e.g. after you press
r. To look at the remote again, stop StepUp withqand start it again. - Environment variables are read at startup only, including secrets. The steps get the environment of the director as it was when it started, so a token that you set after starting StepUp is only seen after a restart.
- A step that does not handle offline mode fails offline when something else makes it run,
e.g. a change of its script.
The
--offlinesetting only stops steps from running merely because StepUp started.