Static Deferred Glob

When dealing with massive datasets comprising tens of thousands of files, it doesn’t make sense to render all of them as static when only a handful will actually be utilized. StepUp addresses this issue with the deferred glob feature, which makes previously unknown files static when:

1. they are used as inputs of a new step and
2. they match a deferred glob pattern.

Example

Example source files: advanced_topics/static_deferred_glob/

Create two text files with some content: foo.txt and bar.txt, and also the following plan.py:

#!/usr/bin/env python
from stepup.core.api import glob, step
from stepup.core.interact import graph

glob("*.txt", _defer=True)
step("cat foo.txt", inp="foo.txt")
graph("graph")

Run the plan interactively with StepUp:

chmod +x plan.py
stepup -n -w1

You should get the following screen output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ cat foo.txt
   SUCCESS │ cat foo.txt
─────────────────────────────── Standard output ────────────────────────────────
This is foo.
────────────────────────────────────────────────────────────────────────────────
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!

As expected, foo.txt is used as a static file. Of course, this would also have been the case without the _defer=True option. The key difference is that with _defer=True, StepUp does not create a list of all matching *.txt files. This can be seen when examining the file graph.txt, which has no trace of bar.txt:

root:
             version = v1
             creates   file:./
             creates   file:plan.py
             creates   step:./plan.py

file:plan.py
                path = plan.py
               state = STATIC
          created by   root:
            consumes   file:./
            supplies   step:./plan.py

file:./
                path = ./
               state = STATIC
          created by   root:
            supplies   file:foo.txt
            supplies   file:plan.py
            supplies   step:./plan.py
            supplies   step:cat foo.txt

step:./plan.py
             workdir = ./
             command = ./plan.py
               state = RUNNING
          created by   root:
            consumes   file:./
            consumes   file:plan.py
             creates   dg:'*.txt'
             creates   step:cat foo.txt

dg:'*.txt'
          created by   step:./plan.py
             creates   file:foo.txt

file:foo.txt
                path = foo.txt
               state = STATIC
          created by   dg:'*.txt'
            consumes   file:./
            supplies   step:cat foo.txt

step:cat foo.txt
             workdir = ./
             command = cat foo.txt
               state = QUEUED
          created by   step:./plan.py
            consumes   file:./
            consumes   file:foo.txt

The node dg:'*.txt; in the graph (green octagon in the figures below) is the result of adding the _defer=True option. This node will create static files as they are needed by other steps. The deferred glob is ideal when there are a large number of files that could match the pattern, of which most are irrelevant for the build. For example, there could be thousands of .txt files in this scenario, but this would not have any effect on the resources consumed by StepUp.

The supplier graph:

The creator graph:

Try the Following

• When using deferred globs, steps cannot create outputs that match the deferred glob. This would mean that a built file could be made static when used as input later, which is clearly inconsistent. Try causing this error by adding a step copy("foo.txt", "foo2.txt").

• Remove the _defer=True option and inspect the corresponding graph.txt. You should see that bar.txt is now indeed included in the graph.