Skip to content

Script (Single Case)

StepUp Core implements a simple script protocol for defining scripts that combine planning and execution in a single source file. This can be more convenient than putting a lot of detail in the plan.py file.

Script Protocol

The script() protocol itself is rather simple. The following line in plan.py:

script("executable", "sub")

is roughly equivalent to:

step("./executable plan", inp="executable", workdir="sub")

Note that the use of a subdirectory is not required. The ./executable plan step is expected to define additional steps to actually run something useful with the executable. A common scenario is to plan a single ./executable run step with appropriate inputs and outputs.

When the optional=True keyword argument is given to the script() function, it executes ./executable plan --optional. The script protocol requires that all run steps created by this planning step should then receive the optional=True keyword argument. Note that the plan step itself is never an optional step: It is always executed.

Script driver

StepUp implements a driver function in the module stepup.core.script that greatly facilitates writing Python scripts that adhere to the script protocol.

It can be used in two ways:

  1. To run the executable for just one specific case of inputs and outputs (this tutorial).

  2. To run the same script with multiple combinations of inputs and outputs (next tutorial).

Single Case Script Driver

A Python script using the driver for a single case has the following structure.

#!/usr/bin/env python
from stepup.core.script import driver

def info():
    return {
        "inp": ..., # a single input path or a list of input paths
        "out": ..., # a single output path or a list of input paths
        "static": ..., # declare a static file or a list of static files
        "stdout": ..., # redirect the standard output to a file (StepUp 1.3.0)
        "stderr": ..., # redirect the standard error to a file (StepUp 1.3.0)
        "just_any": "argument that you want to add",
    }

def run(inp, out, just_any):
    ...

if __name__ == "__main__":
    driver()

  • The info function provides the data necessary to plan the execution of the script. It is executed when calling the script as ./script.py plan.

    Note

    All dictionary items are optional. The info function can even return an empty dictionary. The keys inp, out, static, stdout and stderr affect the planning the run part, as explained in the comments above.

  • The run function is called to perform the useful work and is executed when the script is executed as ./script.py run.

    Note

    The run function can have any argument defined in the dictionary returned by info, but it does not have to specify all of them. The argument list of run can contain fewer arguments (or even none at all).

Example

Example source files: getting_started/script_single/

Consider a script that has parameters defined in a config file config.json, which may be used by multiple script, e.g. for reasons of consistency. For this example, the configuration contains a number of steps and a frequency in arbitrary units, serialized in a JSON file:

{
  "nstep": 100,
  "freq": 0.1
}

This file is used in a script generate.py as follows:

#!/usr/bin/env python
import json

import numpy as np

from stepup.core.script import driver


def info():
    return {
        "inp": "config.json",
        "out": ["cos.npy", "sin.npy"],
    }


def run(inp, out):
    with open("config.json") as fh:
        config = json.load(fh)
    nstep = config["nstep"]
    freq = config["freq"]
    np.save(out[0], np.cos(2 * np.pi * freq * np.arange(nstep)))
    np.save(out[1], np.sin(2 * np.pi * freq * np.arange(nstep)))


if __name__ == "__main__":
    driver()

Also, add the following plan.py file:

#!/usr/bin/env python
from stepup.core.api import script, static

static("generate.py", "config.json")
script("generate.py")

Finally, make the Python scripts executable and give StepUp a spin:

chmod +x generate.py plan.py
stepup -n -w1

You should see the following output on screen:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./generate.py plan
   SUCCESS │ ./generate.py plan
     START │ ./generate.py run
   SUCCESS │ ./generate.py run
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

As expected, this creates two files: cos.npy and sin.npy.

Try the Following:

  • Modify the file config.json and re-run StepUp. The planning is skipped because the script itself did not change. Only the run function is called to work with the updated config.json.

  • Delete one of the outputs and rerun StepUp. Again, the planning is skipped and the computation is repeated to recreate the missing output.

  • Create a new module utils.py with a compute function to calculate the cosine and sine arrays with parameters nstep and freq. Import this module into generate.py, use it in run and re-run StepUp. This will automatically make utils.py an input for the planning and running of generate.py. Test this by making a small change to utils.py and re-running it. (Note that local imports inside the run function will not be identified automatically and are therefore not recommended.)