Skip to content

Named Glob

Conventional glob patterns support a handful of different wildcards. For advanced use cases, StepUp also supports an in-house extension called “named glob”. A named wildcard is written as ${*name}, and repeating the same name within one pattern constrains both occurrences to match the same substring. For example, the following pattern will only match files with matching strings at the placeholders.

prefix_${*name}_something_${*name}.txt

The following will match:

prefix_aaa_something_aaa.txt
prefix_bbb_something_bbb.txt

The following won’t:

prefix_aaa_something_bbb.txt
prefix_bbb_something_aaa.txt

Named globs are especially valuable for coordinating output across multiple directories where a consistent naming pattern links them together.

Restricting What a Named Wildcard May Match

By default, a named wildcard matches *, i.e. anything a plain wildcard would match. The keyword arguments of glob() override that sub-pattern for individual names:

ng = glob("ch${*ch}/sec${*ch}_${*sec}_${*name}.txt", ch="[0-9]", sec="[0-9]")

Here, ch and sec may only match a single digit, while name still matches anything. This is what makes the example below ignore inconsistently named files.

Accessing the Captured Substrings

Iterating over the result of glob() yields NamedGlobMatch objects when the pattern contains named wildcards, and plain Path objects when it does not. The captured substrings are attributes of the match, and match.single is the matching path:

for match in ng:
    print(match.ch, match.sec, match.single)

A few methods are useful when the default iteration mode is not what you need:

  • ng.matches() and ng.files() force iteration over NamedGlobMatch objects or over Path objects, respectively.
  • ng.single() asserts that there is exactly one match and returns its path.

Matching Directories by Name

When you only need to discover directories by name, without declaring a static tree or the whole directory, combine a named wildcard with a file inside each directory that shares its name. For example, given a set of Typst documents organized one per directory:

typst-report/report.typ
typst-summary/summary.typ

the directories can be discovered by name with:

ng = glob("typst-${*name}/${*name}.typ")
static(ng)
for match in ng:
    ...  # match.name is "report", "summary", ...

This matches a file, not the directory itself, so it works with or without a static tree, while match.name still gives you the enclosing directory’s name.

Example

Example source files: docs/getting_started/named_glob/

In the example below, each directory represents a chapter from course notes, containing source files for individual sections. The files in this example are just placeholders and the operations in the workflow are just mockups, but the structure is realistic enough to illustrate the use of named globs. In a realistic setting, one could envision building PDF presentations from LaTeX sources instead, using StepUp RepRep to drive your LaTeX or Typst builds.

Create the following directory layout with markdown files:

ch1/
ch1/sec1_1_introduction.txt
ch1/sec1_2_objectives.txt
ch2/
ch2/sec2_1_mathematical_requisites.txt
ch2/sec2_2_theory.txt
ch3/
ch3/sec3_1_applications.txt
ch3/sec3_2_discussion.txt
ch4/sec4_1_summary.txt

Create the following plan.py:

#!/usr/bin/env python3
from stepup.core.api import copy, glob, run, shq, static

# Enforce consistent chapter numbers throughout the match,
# ignoring inconsistent txt files.
md_chapter = {}
ng = glob("ch${*ch}/sec${*ch}_${*sec}_${*name}.txt", ch="[0-9]", sec="[0-9]")
static(ng)
for match in ng:
    path_txt = match.single
    path_md = path_txt[:-3] + "md"
    copy(path_txt, path_md)
    md_chapter.setdefault(match.ch, []).append(path_md)

# Concatenate all markdown files per chapter
for ch, paths_md in md_chapter.items():
    path_out = f"public/ch{ch}.md"
    run(f"cat {shq(paths_md)} > {shq(path_out)}", shell=True, inp=paths_md, out=path_out)

Note that the substrings matching the named glob patterns are accessible as attributes of the NamedGlobMatch object. For example, match.ch is the chapter number (as a string).

Make the plan executable and run StepUp:

chmod +x plan.py
sb -j 1

You should get the following output:

DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp Core 4.0.0)
 STARTUP │ (Re)initialized boot script
   PHASE │ build
   START │ ./plan.py
 SUCCESS │ ./plan.py
   START │ cp -p ch1/sec1_1_introduction.txt ch1/sec1_1_introduction.md
 SUCCESS │ cp -p ch1/sec1_1_introduction.txt ch1/sec1_1_introduction.md
   START │ cp -p ch1/sec1_2_objectives.txt ch1/sec1_2_objectives.md
 SUCCESS │ cp -p ch1/sec1_2_objectives.txt ch1/sec1_2_objectives.md
   START │ cp -p ch2/sec2_1_mathematical_requisites.txt ch2/sec2_1_mathematical_requisites.md
 SUCCESS │ cp -p ch2/sec2_1_mathematical_requisites.txt ch2/sec2_1_mathematical_requisites.md
   START │ cp -p ch2/sec2_2_theory.txt ch2/sec2_2_theory.md
 SUCCESS │ cp -p ch2/sec2_2_theory.txt ch2/sec2_2_theory.md
   START │ cp -p ch3/sec3_1_applications.txt ch3/sec3_1_applications.md
 SUCCESS │ cp -p ch3/sec3_1_applications.txt ch3/sec3_1_applications.md
   START │ cp -p ch3/sec3_2_discussion.txt ch3/sec3_2_discussion.md
 SUCCESS │ cp -p ch3/sec3_2_discussion.txt ch3/sec3_2_discussion.md
   START │ cp -p ch4/sec4_1_summary.txt ch4/sec4_1_summary.md
 SUCCESS │ cp -p ch4/sec4_1_summary.txt ch4/sec4_1_summary.md
   START │ cat ch1/sec1_1_introduction.md ch1/sec1_2_objectives.md > public/ch1.md
 SUCCESS │ cat ch1/sec1_1_introduction.md ch1/sec1_2_objectives.md > public/ch1.md
   START │ cat ch2/sec2_1_mathematical_requisites.md ch2/sec2_2_theory.md > public/ch2.md
 SUCCESS │ cat ch2/sec2_1_mathematical_requisites.md ch2/sec2_2_theory.md > public/ch2.md
   START │ cat ch3/sec3_1_applications.md ch3/sec3_2_discussion.md > public/ch3.md
 SUCCESS │ cat ch3/sec3_1_applications.md ch3/sec3_2_discussion.md > public/ch3.md
   START │ cat ch4/sec4_1_summary.md > public/ch4.md
 SUCCESS │ cat ch4/sec4_1_summary.md > public/ch4.md
DIRECTOR │ Ran 12 job(s).
DIRECTOR │ Trying to remove 0 deletable file(s) and empty director(y|ies)
DIRECTOR │ See you!

Notes on static()

Two details of the interaction with static() are worth spelling out:

  • Named wildcards work in static() too, and the back-reference constraint applies there as well, so static("ch${*ch}/sec${*ch}_*.txt") is a valid declaration. There are limits, though: static() takes no keyword arguments, so a named wildcard there always uses the default sub-pattern *, and the captured substrings are not part of the return value. Go through glob() when you need either.

  • static() accepts the return value of glob() directly. The call static(ng) declares exactly what ng matched, without registering the pattern a second time, because glob() already did that. This is the composition used in the example plan above, and it is what makes query-then-declare, and the probe-then-declare idiom, cheap.