Cyclic dependencies

Cyclic dependencies are defined in StepUp as closed loops in the “supplier ➜ consumer” graph. One may construct these in a plan.py, which will result in an error message.

In theory, one could also have cycles in the “creator ➜ product” graph, but these are excluded by construction and therefore not relevant to discuss in the tutorial.

Example

Create the following plan.py, which is StepUp’s equivalent of a snake biting its own tail:

#!/usr/bin/env python
from stepup.core.api import copy

copy("a.txt", "b.txt")
copy("b.txt", "a.txt")

Make the plan executable and give it a try as follows:

chmod +x plan.py
stepup -n -w1

You will get the following terminal output showing that this plan won’t work.

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
      FAIL │ ./plan.py
────────────────────────────────── Step info ───────────────────────────────────
Command               ./plan.py
Return code           1
──────────────────────────────── Standard error ────────────────────────────────
Traceback (most recent call last):
  File "/home/toon/univ/reprep/stepup-core/docs/advanced_topics/cyclic_dependencies/./plan.py", line 5, in <module>
    copy("b.txt", "a.txt")
  File "/home/toon/univ/reprep/stepup-core/stepup/core/api.py", line 517, in copy
    return step("cp -aT ${inp} ${out}", inp=path_src, out=path_dst, optional=optional, block=block)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toon/univ/reprep/stepup-core/stepup/core/api.py", line 355, in step
    step_key = RPC_CLIENT.call.step(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toon/univ/reprep/stepup-core/stepup/core/rpc.py", line 448, in __call__
    _handle_error(body, name, args, kwargs)
  File "/home/toon/univ/reprep/stepup-core/stepup/core/rpc.py", line 69, in _handle_error
    raise RPCError(f"An exception was raised in the server during the call {fmt_call}: \n\n{body}")
stepup.core.exceptions.RPCError: An exception was raised in the server during the call step('step:./plan.py', 'cp -aT b.txt a.txt', [Path('b.txt')], [], [Path('a.txt')], [], Path('./'), False, None, False):
Traceback (most recent call last):
  File "/home/toon/univ/reprep/stepup-core/stepup/core/rpc.py", line 194, in _handle_request
    result = call(*bound.args, **bound.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toon/univ/reprep/stepup-core/stepup/core/director.py", line 376, in step
    return self._workflow.define_step(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toon/univ/reprep/stepup-core/stepup/core/workflow.py", line 622, in define_step
    self.create_file(step_key, out_path, FileState.PENDING)
  File "/home/toon/univ/reprep/stepup-core/stepup/core/workflow.py", line 375, in create_file
    self.supply(creator_key, file_key)
  File "/home/toon/univ/reprep/stepup-core/stepup/core/cascade.py", line 493, in supply
    self.check_cyclic(supplier_key, consumer_key)
  File "/home/toon/univ/reprep/stepup-core/stepup/core/cascade.py", line 447, in check_cyclic
    self.report_cyclic(src_key, dst_key)
  File "/home/toon/univ/reprep/stepup-core/stepup/core/cascade.py", line 422, in report_cyclic
    raise CyclicError("\n".join(lines))
stepup.core.exceptions.CyclicError: New relation introduces cyclic dependency
src = step:cp -aT b.txt a.txt
dst = file:a.txt
cycle:
  step:cp -aT b.txt a.txt
  file:b.txt
  step:cp -aT a.txt b.txt
  file:a.txt
────────────────────────────────────────────────────────────────────────────────
   WARNING │ 1 step(s) failed, see error messages above
   WARNING │ Scheduler is put on hold. Not reporting pending steps.
   WARNING │ Skipping cleanup due to incomplete build.
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
   WARNING │ Dissolving the workflow due to an exceptions while the graph was being changed.
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!