Cyclic dependencies

Cyclic dependencies are defined in StepUp as closed loops in the dependency graph. Formally, such a loop is defined as a set of edges that can be followed from supplier to consumer, such that one arrives back at the starting point. If you construct cyclic dependencies in a plan.py, an error message is generated.

In theory, one could also have cycles in the provenance graph, but it is not possible to create such loops with the functions in stepup.core.api.

Example

Create the following plan.py, which is StepUp’s equivalent of a snake biting its own tail:

#!/usr/bin/env python3
from stepup.core.api import copy

copy("a.txt", "b.txt")
copy("b.txt", "a.txt")

Make the plan executable and give it a try as follows:

chmod +x plan.py
stepup -n 1

You will get the following terminal output showing that this plan won’t work.

  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp 2.0.4)
   STARTUP │ (Re)initialized boot script
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
      FAIL │ ./plan.py
────────────────────────────────── Step info ───────────────────────────────────
Command               ./plan.py
Return code           1
──────────────────────────────── Standard error ────────────────────────────────
Traceback (most recent call last):
  File "docs/advanced_topics/cyclic_dependencies/./plan.py", line 5, in <module>
    copy("b.txt", "a.txt")
    ~~~~^^^^^^^^^^^^^^^^^^
  File "stepup/core/api.py", line 548, in copy
    return step("cp -aT ${inp} ${out}", inp=path_src, out=path_dst, optional=optional, block=block)
  File "stepup/core/api.py", line 330, in step
    RPC_CLIENT.call.step(
    ~~~~~~~~~~~~~~~~~~~~^
        _get_step_key(),
        ^^^^^^^^^^^^^^^^
    ...<8 lines>...
        block,
        ^^^^^^
    )
    ^
  File "stepup/core/rpc.py", line 444, in __call__
    _handle_error(body, name, args, kwargs)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "stepup/core/rpc.py", line 69, in _handle_error
    raise RPCError(f"An exception was raised in the server during the call {fmt_call}: \n\n{body}")
stepup.core.exceptions.RPCError: An exception was raised in the server during the call step('step:./plan.py', 'cp -aT b.txt a.txt', [Path('b.txt')], [], [Path('a.txt')], [], Path('./'), False, None, False):
Traceback (most recent call last):
  File "stepup/core/rpc.py", line 192, in _handle_request
    result = await result
             ^^^^^^^^^^^^
  File "stepup/core/director.py", line 399, in step
    self.workflow.define_step(
    ~~~~~~~~~~~~~~~~~~~~~~~~~^
        creator,
        ^^^^^^^^
    ...<8 lines>...
        block=block,
        ^^^^^^^^^^^^
    )
    ^
  File "stepup/core/workflow.py", line 699, in define_step
    file.add_supplier(step)
    ~~~~~~~~~~~~~~~~~^^^^^^
  File "stepup/core/file.py", line 135, in add_supplier
    idep = super().add_supplier(supplier)
  File "stepup/core/cascade.py", line 374, in add_supplier
    raise CyclicError("New relation introduces a cyclic dependency")
stepup.core.exceptions.CyclicError: New relation introduces a cyclic dependency
────────────────────────────────────────────────────────────────────────────────
   WARNING │ 1 step(s) failed.
   WARNING │ 1 step(s) remained pending due to incomplete requirements
─────────────────────────────── Orphaned inputs ────────────────────────────────
             AWAITED  a.txt
───────────────────────────────── PENDING Step ─────────────────────────────────
Command               cp -aT a.txt b.txt
Working directory     ./
Inputs        STATIC  ./
             AWAITED  (a.txt)
Outputs      AWAITED  b.txt
────────────────────────────────────────────────────────────────────────────────
   WARNING │ Skipping cleanup due to incomplete build.
   WARNING │ Check logs: .stepup/fail.log .stepup/warning.log
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!