Distributed Plans¶

When your project grows, defining the entire workflow in a single plan.py file may become inconvenient. Especially when working with nested directories for different parts of the project, it may be convenient to distribute the workflow over multiple plan.py files.

Example¶

Example source files: getting_started/distributed_plans/

Create a simple example with a top-level plan.py as follows:

#!/usr/bin/env python
from stepup.core.api import plan, static

static("sub/", "sub/plan.py", "part1.txt", "sub/part2.txt")
plan("sub/")

The top-level plan defines a few static files and then calls another plan in sub/. Create a file sub/plan.py as follows:

#!/usr/bin/env python
from stepup.core.api import step

step("cat part2.txt", inp="part2.txt")
step("cat ../part1.txt", inp="../part1.txt")

Also create two files part1.txt and sub/part2.txt with a bit of text. Make both plans executable and run StepUp as follows:

chmod +x plan.py sub/plan.py

You will get the following output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ ./plan.py  # wd=sub/
   SUCCESS │ ./plan.py  # wd=sub/
     START │ cat part2.txt  # wd=sub/
   SUCCESS │ cat part2.txt  # wd=sub/
─────────────────────────────── Standard output ────────────────────────────────
This is part 2.
────────────────────────────────────────────────────────────────────────────────
     START │ cat ../part1.txt  # wd=sub/
   SUCCESS │ cat ../part1.txt  # wd=sub/
─────────────────────────────── Standard output ────────────────────────────────
This is part 1.
────────────────────────────────────────────────────────────────────────────────
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

Practical Considerations¶

The main benefit of having multiple plan.py files is to improve the logical structure of your project. It may also be helpful when a part of your plan.py is computationally demanding, in which case it can be factored out so that it does not slow down the rest of the build. However, ideally, the plan.py scripts execute quickly, leaving the hard work to other steps.
When there are multiple plan.py files, keep in mind that their order of execution cannot be relied upon. They are executed in parallel, and their relative starting times depend on factors unknown a priori, such as system load and number of workers.