Running Python Scripts¶

In principle, you can incorporate any Python script into a StepUp workflow using the runsh() function. However, it is recommended to use the runpy() function instead. It will automatically detect thje modules imported by the script, and if they correspond to local files in your workflow, the step is amended with these files as required inputs. This way, if the local modules have changed, StepUp will know the script needs to run again.

Example¶

The following example plan.py file shows how to run a Python script with StepUp.

#!/usr/bin/env python3
from stepup.core.api import runpy, static

static("work.py", "helper.py")
runpy("./work.py 3")

Create a work.py script that can be executed:

#!/usr/bin/env python3
import sys

from helper import message

for _ in range(int(sys.argv[1])):
    print(message)

Finally, create a helper.py script that is imported by work.py:

message = "Repetition helps to get the message across."

Make the appropriate scripts executable and run StepUp:

chmod +x plan.py work.py
stepup boot -n 1

You should see the following output:

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp 3.0.0)
   STARTUP │ (Re)initialized boot script
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ runpy ./plan.py
   SUCCESS │ runpy ./plan.py
     START │ runpy ./work.py 3
   SUCCESS │ runpy ./work.py 3
─────────────────────────────── Standard output ────────────────────────────────
Repetition helps to get the message across.
Repetition helps to get the message across.
Repetition helps to get the message across.
────────────────────────────────────────────────────────────────────────────────
  DIRECTOR │ Trying to delete 0 outdated output(s)
  DIRECTOR │ Stopping workers
  DIRECTOR │ See you!

Try the Following¶

Change the value of the variable message in helper.py and rerun StepUp with stepup boot -n 1. Only work.py is rerun, since plan.py has not changed.

Notes¶

You can control which imports are treated as “local” with the STEPUP_PATH_FILTER environment variable. See Environment Variables.
StepUp also provides more sophisticated interfaces for running (Python) scripts, which are described in the following sections:
Although not common, it is also possible to dynamically generate the Python file helper.py. In this case, the work.py script should only be executed after a previous step has created helper.py. For this to work, you must explicitly add the inp argument:
```
runpy("./generate.py > ${out}", out=["helper.py"])
runpy("./work.py", inp=["helper.py"])
```
The reason for this is that the running work.py will fail if helper.py does not exist. By adding the inp option, StepUp knows not to run work.py until helper.py is available.