Variable Substitution¶

StepUp does not substitute environment variables is the command (first argument) of the step() function. As discussed in the tutorial on environment variables, the executing shell takes care of such substitutions.

However, environment variables in all path-like arguments (e.g. workdir, inp, out and vol) of functions that take such arguments (step(), amend() etc.) are automatically substituted. This substitution takes place before the commands are sent to the director process and all used variables are communicated to the director with an amend() call.

If a script needs an environment variable elsewhere, the function getenv() is recommended: It returns the value of the variable and calls amend() to tell the director that the current step depends on this variable.

Example¶

Example source files: advanced_topics/variable_substitution/

Create a plan.py with the following contents:

#!/usr/bin/env python
from stepup.core.api import static, step

static("step.py", "src_${MYVAR}.txt")
step("./step.py < ${inp} > ${out}", inp="src_${MYVAR}.txt", out="dst_${MYVAR}.txt")

In addition, create a script step.py as follows:

#!/usr/bin/env python
import sys

from stepup.core.api import getenv

myname = repr(getenv("MYNUM"))
print(f"MYNUM={myname}\n")
print("Read from stdin:")
print(sys.stdin.read())

Make the Python scripts executable and run them as follows:

chmod +x plan.py step.py
MYVAR=foo stepup -n -w1

You should get the following terminal output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./step.py < src_foo.txt > dst_foo.txt
   SUCCESS │ ./step.py < src_foo.txt > dst_foo.txt
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

The file dst_foo.txt will contain the following:

MYNUM=None

Read from stdin:
This is src_foo

As shown in this example, the function getenv() returns None when a variable does not exist (or any other default you specify). When using variables like ${MYVAR} in path-like arguments, the variable must exist or an exception is raised.

Try the Following¶

Run StepUp without defining MYVAR: stepup -n -w1. As explained above, this raises an exception.
Run StepUp by also defining MYNUM: MYVAR=foo MYNUM=1 stepup -n -w1. Now the string '1' is shown in the output dst_foo.txt. Note that environment variables are always strings, and need to be converted to other types if needed.