Volatile Outputs¶

It may happen that steps produce auxiliary outputs that are not really of interest, but rather occur as a side effect. For example, LaTeX is notoriously productive in terms of output files. Some of these files will change with every run, e.g., because they contain timestamps.

It is useful to inform StepUp of the existence of such volatile files, so they can be cleaned up when appropriate. However, there is no point in computing file hashes for them, as these files are not used as inputs later and may change for no good reason. One may pass a list of such files to the vol argument of the step() function.

Example¶

Example source files: advanced_topics/volatile_outputs/

Create the following plan.py, with a single step that produces a trivially volatile output:

#!/usr/bin/env python
from stepup.core.api import step

step("date > date.txt", vol="date.txt")

Make the plan executable and run it as follows:

chmod +x plan.py
stepup -n -w1

The file date.txt will contain the current time. You should get the following terminal output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ date > date.txt
   SUCCESS │ date > date.txt
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

Try the Following¶

Remove the file date.txt and run StepUp again. You will see that the step gets ignored: StepUp does not care much about the state of volatile files. It only keeps track of them, so they can be removed when needed.
Manually recreate the file date.txt with some arbitrary contents, and run StepUp. Again, the step gets skipped because the contents of the volatile date.txt are not considered when deciding if a step is outdated.
Comment out the step in plan.py and run StepUp again. Because the step is removed, the volatile output is also removed.