Cyclic dependencies

Cyclic dependencies are defined in StepUp as closed loops in the dependency graph. Formally, such a loop is defined as a set of supplier ➜ consumer edges (as introduced under Edges) that can be followed such that one arrives back at the starting point. If you construct cyclic dependencies in a plan.py, an error message is generated.

The stepup.core.api enforces acyclic dependencies in the provenance graph and cannot be introduced accidentally by the user.

Example

Create the following plan.py, which is StepUp’s equivalent of a snake biting its own tail:

#!/usr/bin/env python3
from stepup.core.api import copy

copy("a.txt", "b.txt")
copy("b.txt", "a.txt")

Make the plan executable and give it a try as follows:

chmod +x plan.py
sb -j 1

You will get the following terminal output showing that this plan won’t work.

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 3.2.3.post54)
   STARTUP │ (Re)initialized boot script
     PHASE │ build
     START │ ./plan.py
      FAIL │ ./plan.py
──────────────────────────────── Failed command ────────────────────────────────
./plan.py  # exit=1
──────────────────────────────── Standard error ────────────────────────────────
Traceback (most recent call last):
  File "stepup/core/executor.py", line 152, in _forkserver_entry
    runpy.run_path(cmd, run_name="__main__")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 294, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "./plan.py", line 5, in <module>
    copy("b.txt", "a.txt")
    ~~~~^^^^^^^^^^^^^^^^^^
  File "stepup/core/api.py", line 888, in copy
    return step(
        "cp -p ${inp} ${out}",
    ...<4 lines>...
        shell=False,
    )
  File "stepup/core/api.py", line 412, in step
    to_check = RPC_CLIENT.call.step(
        _get_step_i(),
    ...<9 lines>...
        env_overrides,
    )
  File "stepup/core/rpc.py", line 543, in __call__
    _handle_error(body, name, args, kwargs)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "stepup/core/rpc.py", line 74, in _handle_error
    raise RPCError(f"An exception was raised in the server during the call {fmt_call}: \n\n{body}")
stepup.core.exceptions.RPCError: An exception was raised in the server during the call step(3, 'cp -p b.txt a.txt', [Path('b.txt')], [], [Path('a.txt')], [], Path('.'), 32, {}, False, {}):
Traceback (most recent call last):
  File "stepup/core/rpc.py", line 210, in _handle_request
    result = await result
             ^^^^^^^^^^^^
  File "stepup/core/director.py", line 511, in step
    return self.workflow.define_step(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        creator,
        ^^^^^^^^
    ...<9 lines>...
        env_overrides=env_overrides,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "stepup/core/workflow.py", line 827, in define_step
    file.add_supplier(step)
    ~~~~~~~~~~~~~~~~~^^^^^^
  File "stepup/core/file.py", line 277, in add_supplier
    idep = super().add_supplier(supplier)
  File "stepup/core/trellis.py", line 426, in add_supplier
    raise CyclicError("New relation introduces a cyclic dependency")
stepup.core.exceptions.CyclicError: New relation introduces a cyclic dependency
────────────────────────────────────────────────────────────────────────────────
   WARNING │ 1 step(s) failed.
   WARNING │ 1 step(s) remained pending ...
─────────────────────────────── Detached inputs ────────────────────────────────
             AWAITED  a.txt
───────────────────────────────── PENDING Step ─────────────────────────────────
Reason                creator is not RUNNING or SUCCEEDED
Command               cp -p a.txt b.txt
Inputs       AWAITED  (a.txt)
Outputs      AWAITED  b.txt
────────────────────────────────────────────────────────────────────────────────
   WARNING │ Skipping file cleanup due to incomplete build
   WARNING │ Check logs: .stepup/fail.log .stepup/warning.log
  DIRECTOR │ See you!