Environment Variables

When defining a step, one can specify the environment variables it uses (not their values). When restarting StepUp with a different value for any of these variables, StepUp will know that it has to rerun the step instead of skipping it, even if input files have not changed.

One can only change an environment variable by stopping StepUp, changing the variable, and then starting StepUp again. Environment variables cannot be modified while StepUp is running.

Example

Example source files: docs/advanced_topics/environment_variables/

Create the following plan.py:

#!/usr/bin/env python3
from stepup.core.api import run

run("echo ${MYVAR}", env="MYVAR", shell=True)

Make it executable and run StepUp with a specific value of the variable:

chmod +x plan.py
MYVAR=foo sb -j 1

You will see the following output:

        0/0 |   DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 3.2.3.post54)
        0/0 |    STARTUP │ (Re)initialized boot script
        0/1 |      PHASE │ build
        0/1 |      START │ ./plan.py
        1/2 |    SUCCESS │ ./plan.py
        1/2 |      START │ echo ${MYVAR}
        2/2 |    SUCCESS │ echo ${MYVAR}
─────────────────────────────── Standard output ────────────────────────────────
foo
────────────────────────────────────────────────────────────────────────────────
        2/2 |   DIRECTOR │ Trying to delete 0 outdated output(s)
        2/2 |   DIRECTOR │ See you!

The variable substitution is performed in the step’s execution environment. StepUp will not try to substitute ${MYVAR} before starting the step. The special variables ${inp} and ${out} are exceptions to this rule, as discussed in the tutorial on dependencies.

Try the Following

• Repeat MYVAR=foo sb -j 1 without making changes. You will see that the echo step is skipped as expected.

• Now run MYVAR=bar sb -j 1. This time, the variable change will cause the step to be executed.

Injecting Environment Variables

Besides working with external environment variables, you can also inject environment variables into the command of a step. For example:

msg = "hello"
run(f"MESSAGE={msg} " + "echo ${MESSAGE}")

Note that this is conceptually very different and it practically serves a different purpose. It just sets the value of the variable before the echo command is executed. Now, it would be incorrect to add the argument env="MESSAGE" to the run() call, because this step will not be sensitive to the value of MESSAGE defined outside StepUp.

Note that this syntax for setting variables in a command is supported by StepUp with both shell=True and shell=False, but the implementation is very different:

• With shell=True, the variable is set by the shell before the command is executed. This is completely opaque to StepUp, which just sees the command as a string.

  Because StepUp does not know which variables are set, it will not check their validity.

• With shell=False, a normal Python subprocess call as such would fail. StepUp extracts the variable assignment from the command, and stores it in the workflow database. When the executor runs the command, it passes the variables into the env=... argument of the subprocess call.

  In this case, StepUp will raise an error if the variable included in the command also appears in the env argument of the run() call, because now the mistake can be detected unambiguously.