Glob Conditional¶

A wildcard-free pattern passed to glob() is a convenient existence probe: glob() never raises for a zero-match pattern, so it can be used directly in a conditional expression. Declare the match with static() only once the probe succeeds:

from stepup.core.api import glob, static

if glob("dataset/bigfile.txt"):
    # The file exists: declare it and run plan A.
    static("dataset/bigfile.txt")
    ...
else:
    # The file is not available: run plan B instead.
    ...

A similar conditional would not work with static() directly, because it would raise an exception when the file does not exist.

The probe is always well-behaved under the end-of-phase check: when the file exists, the same branch that probes also declares it with static(), and when it does not exist, there is no match at all and nothing to justify.

Example¶

Example source files: docs/getting_started/glob_conditional/

Let’s simulate a scenario where dataset/, if it exists, is remote storage with a huge dataset. Plan A is to extract useful information from the dataset. However, there may be reasons why this is not always possible or desirable:

Not all your collaborators may have access to this storage at all times.
The extraction is slow or expensive otherwise.

Plan B is to use the results of the extraction from a previous run and declare them as static files.

Create the following plan.py:

#!/usr/bin/env python3
from stepup.core.api import glob, run, static

if glob("dataset/bigfile.txt"):
    static("dataset/bigfile.txt", "expensive.py")
    run("./expensive.py", inp="dataset/bigfile.txt", out="average.txt")
else:
    static("average.txt")

run("cat average.txt", inp="average.txt")

For this example, the script expensive.py is not expensive at all. It just serves as an illustration of a more realistic scenario where this script may do some non-trivial work. In this example, expensive.py just computes the average of all numbers in dataset/bigfile.txt and writes out the result to average.txt:

#!/usr/bin/env python3

total = 0.0
count = 0
with open("dataset/bigfile.txt") as fh:
    for line in fh:
        total += float(line)
        count += 1

with open("average.txt", "w") as fh:
    print(f"{total / count:f}", file=fh)

Now put some values in dataset/bigfile.txt, e.g.:

1.2
2.3
7.9
1.8
-0.3

To run the example, make the scripts executable and fire up StepUp:

chmod +x expensive.py plan.py
sb -j 1

You should get the following output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
   START │ ./expensive.py
 SUCCESS │ ./expensive.py
   START │ cat average.txt
 SUCCESS │ cat average.txt
─────────────────────────────── Standard output ────────────────────────────────
2.580000
────────────────────────────────────────────────────────────────────────────────
DIRECTOR │ Ran 3 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

Now, simulate the situation where the dataset is absent by renaming the directory:

mv dataset tmp
sb -j 1

The new output reveals that the dataset is completely ignored while the file average.txt is still used:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ Checking 4 file(s) for changes
 DELETED │ dataset/bigfile.txt
 STARTUP │ Checking 1 nglob(s) for new or deleted matches
 DELETED │ dataset/bigfile.txt
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
    SKIP │ cat average.txt
DIRECTOR │ Ran 1 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

Since the file average.txt did not change, the step cat average.txt is skipped.

Practical Considerations¶

For simplicity’s sake, the example involves few calculations. In a more realistic setting, the step cat average.txt is replaced by several scripts that create visualizations of the information extracted from the large dataset. Tweaking these visualizations for clarity usually takes some iterations, for which access to the large dataset is not necessary.
A StepUp project practically always resides in a Git repository. While the files extracted from the large dataset can be reproduced easily, it may still be relevant to commit them into the Git repository:
- Not all collaborators may have access to the dataset, but you still want them to be able to reproduce a part of the workflow.
- In the long run, the large dataset might be removed because it is too big and old to keep around. The extracted data then become a relevant and compact subset that can be easily stored for longer periods.