Static Named Glob

Conventional glob patterns support a handful of different wildcards. For advanced use cases, StepUp also supports an in-house extension called “named glob”. For example, the following pattern will only match files with matching strings at the placeholders.

prefix_${*name}_something_${*name}.txt

The following will match:

prefix_aaa_something_aaa.txt
prefix_bbb_something_bbb.txt

The following won’t:

prefix_aaa_something_bbb.txt
prefix_bbb_something_aaa.txt

Named globs are often useful when working with files distributed over multiple directories, each having a central file that repeats a part of the directory name.

Example

Example source files: advanced_topics/static_named_glob/

In the example below, each directory represents a chapter from course notes, containing source files for individual sections. In a realistic setting, one could envision building a PDF presentations from LaTeX sources instead. To keep the example independent of StepUp RepRep, text files will be copied to Markdown files, which will then be concatenated.

Create the following directory layout with markdown files:

ch1/
ch1/sec1_1_introduction.txt
ch1/sec1_2_objectives.txt
ch2/
ch2/sec2_1_mathematical_requisites.txt
ch2/sec2_2_theory.txt
ch3/
ch3/sec3_1_applications.txt
ch3/sec3_2_discussion.txt
ch4/sec4_1_summary.txt

Create the following plan.py:

#!/usr/bin/env python
from stepup.core.api import copy, glob, mkdir, step

# Make all chapter directories static
glob("ch*/")

# Enforce consistent chapter numbers throughout the match,
# ignoring inconsistent txt files.
md_chapter = {}
for match in glob("ch${*ch}/sec${*ch}_${*sec}_${*name}.txt", ch="[0-9]", sec="[0-9]"):
    path_txt = match.single
    path_md = path_txt[:-3] + "md"
    copy(path_txt, path_md)
    md_chapter.setdefault(match.ch, []).append(path_md)

# Concatenate all markdown files per chapter
mkdir("public/")
for ch, paths_md in md_chapter.items():
    step("cat ${inp} > ${out}", inp=paths_md, out=f"public/ch{ch}.md")

Note that the substrings matching the named glob patterns are accessible as attributes of the NGlobMatch object. For example, match.ch is the chapter number (as a string).

Make the plan executable and run StepUp:

chmod +x plan.py
stepup -n -w1

You should get the following output:

  DIRECTOR │ Listening on /tmp/stepup-########/director
  DIRECTOR │ Launched worker 0
     PHASE │ run
     START │ ./plan.py
   SUCCESS │ ./plan.py
     START │ cp -aT ch1/sec1_1_introduction.txt ch1/sec1_1_introduction.md
   SUCCESS │ cp -aT ch1/sec1_1_introduction.txt ch1/sec1_1_introduction.md
     START │ cp -aT ch1/sec1_2_objectives.txt ch1/sec1_2_objectives.md
   SUCCESS │ cp -aT ch1/sec1_2_objectives.txt ch1/sec1_2_objectives.md
     START │ cp -aT ch2/sec2_1_mathematical_requisites.txt ch2/sec2_1_mathematical_requisites.md
   SUCCESS │ cp -aT ch2/sec2_1_mathematical_requisites.txt ch2/sec2_1_mathematical_requisites.md
     START │ cp -aT ch2/sec2_2_theory.txt ch2/sec2_2_theory.md
   SUCCESS │ cp -aT ch2/sec2_2_theory.txt ch2/sec2_2_theory.md
     START │ cp -aT ch3/sec3_1_applications.txt ch3/sec3_1_applications.md
   SUCCESS │ cp -aT ch3/sec3_1_applications.txt ch3/sec3_1_applications.md
     START │ cp -aT ch3/sec3_2_discussion.txt ch3/sec3_2_discussion.md
   SUCCESS │ cp -aT ch3/sec3_2_discussion.txt ch3/sec3_2_discussion.md
     START │ cp -aT ch4/sec4_1_summary.txt ch4/sec4_1_summary.md
   SUCCESS │ cp -aT ch4/sec4_1_summary.txt ch4/sec4_1_summary.md
     START │ mkdir -p public/
   SUCCESS │ mkdir -p public/
     START │ cat ch1/sec1_1_introduction.md ch1/sec1_2_objectives.md > public/ch1.md
   SUCCESS │ cat ch1/sec1_1_introduction.md ch1/sec1_2_objectives.md > public/ch1.md
     START │ cat ch2/sec2_1_mathematical_requisites.md ch2/sec2_2_theory.md > public/ch2.md
   SUCCESS │ cat ch2/sec2_1_mathematical_requisites.md ch2/sec2_2_theory.md > public/ch2.md
     START │ cat ch3/sec3_1_applications.md ch3/sec3_2_discussion.md > public/ch3.md
   SUCCESS │ cat ch3/sec3_1_applications.md ch3/sec3_2_discussion.md > public/ch3.md
     START │ cat ch4/sec4_1_summary.md > public/ch4.md
   SUCCESS │ cat ch4/sec4_1_summary.md > public/ch4.md
  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
  DIRECTOR │ Stopping workers.
  DIRECTOR │ See you!