Function calls¶

Note

This feature was introduced in StepUp 2.0.0.

Warning

This feature is experimental and may change significantly in future 2.x releases.

The call() function is a function wrapper for local executable scripts. These scripts can be written in any language, as long as they adhere to the “call protocol” described below. StepUp provides a driver() function in the module stepup.core.call to facilitate the implementation of Python scripts that adhere to the call protocol.

Call protocol¶

In its simplest form, the following use of call() in a plan.py script

call("executable", parameter="value")

is roughly equivalent to

step("./executable '{\"parameter\": \"value\"}'", inp="executable")

(It also works without any parameters.) The script executable is expected to decode the JSON given on the command-line, and then use these parameters.

The call() function supports many optional arguments that control with which options the executable is called. In a nutshell, the parameters can also be provided through an input file. By default, this is a JSON file, but PICKLE is provided as a fallback in case the types of the parameters are not compatible with JSON. When the executable script produces a “return value”, it should write this to an output file, either in JSON or PICKLE format.

Because of the delayed execution, the call() function cannot return this value. If you are familiar with Python’s builtin concurrent.futures module, you can think of the output file of the script as the Future object that is returned by concurrent.futures.Executor.submit(). The call() function returns a StepInfo object from which you can extract the output file path.

To fully support the call() protocol, the executable must be able to handle the following command-line arguments:

JSON_INP: The JSON-encoded parameters.
--inp=PATH_INP: As an alternative to the previous, a file with parameters in JSON or PICKLE format.
--out=PATH_OUT: The output file to use (if there is a return value), either in JSON or PICKLE format.
--amend-out: If given, the executable must call amend(out=PATH_OUT) before writing the output file.

Call driver¶

StepUp implements a driver() function in the module stepup.core.call that greatly facilitates writing Python scripts that adhere to the call protocol. The usage of this driver() function is illustrated in the example below.

Note that the driver() function also detects local modules that are imported in the script, and amends these as required inputs. Changes to modules imported in your Python script will automatically trigger a re-run of the script. By default, only the modules inside STEPUP_ROOT are treated as dependencies. You can specify additional directories in the STEPUP_EXTERNAL_SOURCES environment variable.

Example¶

Example source files: docs/getting_started/call/

First create a wavegen.py script as follows:

#!/usr/bin/env python3
from math import pi, sin

from stepup.core.call import driver


def run(freq: float, duration: float, rate: int = 44100):
    """Generate a sine wave with given frequency, duration, and sample rate."""
    return [sin(2 * pi * freq * i / rate) for i in range(int(duration * rate))]


if __name__ == "__main__":
    driver()

Then create a plan.py script as follows:

#!/usr/bin/env python3
from stepup.core.api import call, static

static("wavegen.py")
call("wavegen.py", out="wave1.json", freq=440.0, duration=1.0)
call("wavegen.py", out="wave2.json", freq=380.0, duration=0.5, rate=22050)

Finally, make the Python scripts executable and start StepUp:

chmod +x wavegen.py plan.py
stepup -n 1

You should see the following output on screen:

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp 2.0.4)
   STARTUP │ (Re)initialized boot script
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./wavegen.py '{"freq": 440.0, "duration": 1.0}' --out=wave1.json
   SUCCESS │ ./wavegen.py '{"freq": 440.0, "duration": 1.0}' --out=wave1.json
     START │ ./wavegen.py '{"freq": 380.0, "duration": 0.5, "rate": 22050}' --out=wave2.json
   SUCCESS │ ./wavegen.py '{"freq": 380.0, "duration": 0.5, "rate": 22050}' --out=wave2.json
  DIRECTOR │ Trying to delete 0 outdated output(s).
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

Practical considerations¶

As shown by the example, the driver() function takes care of parsing the command-line arguments in the call protocol and amending the output. You don’t need to worry about this in your script. You simply add the run() function that takes the parameters as arguments and returns the result.
Scripts that use the driver() function can be run as standalone scripts. This is useful for debugging and testing.
Scripts that adhere to the call protocol can be reused across the entire workflow. If you want to place it in the top-level directory and execute it in any other directory, you can use the ROOT variable:
```
call("${ROOT}/wavegen.py", freq=235, duration=1.0, out="sine.json")
```