Design Principles¶

This document records the reasoning behind the shape of Snipwise, including the alternatives that were considered and turned down. It is written to facilitate the future extension of Snipwise, and it assumes that the user documentation the home page links to has been read. What Snipwise does is specified there and is not repeated here.

The scope and the invariants come first, because every decision after them is an argument that they still hold.

Scope¶

Snipwise copies fragments from one text file into another, possibly with a rendering step in between to adjust the formatting. It does not generate anything, it does not read the environment, and the result of snipwise check depends only on the files it is looking at.

This limited goal separates Snipwise from a templating tool such as Cog, which can also generate new text by executing code. While Cog is more general, and can thus do whatever Snipwise can do, reproducing Snipwise with Cog would take more work to set up and maintain.

Snipwise infers two decisions when an author has not declared them: which snippets exist, read from the markers that use them, and which copy of a snippet is the original. Neither inference invents text. Both only pick between texts that the project already contains, and both exist to make Snipwise usable with less configuration.

Invariants¶

Seven properties hold throughout, and the sections that follow are largely the reasoning that keeps them true.

Snipwise edits character ranges and never reserializes a file, so that a fix leaves the formatting of the rest of the file exactly as it was.
The template context is closed. A template sees the snippet and the results of the scan, and nothing else.
Unknown configuration keys are errors, so that later keys are safe to add.
Snipwise 1.0 executes no code from the repository. A snipwise.md file is a document, and reading it cannot run anything.
snipwise check is a comparison of the files that the patterns of the rules select, and nothing more. snipwise fix may also read those same files as they are at HEAD, and may be told with --original which copy wins, but neither command is interactive and neither reaches outside that selection.
Every text Snipwise writes was already in the project. Inference and --original choose between texts, they never compose one.
A snippet needs two places that hold it. A name that fewer than two places hold keeps nothing consistent, and is refused rather than reported.

Where Configuration Lives¶

Configuration belongs to the destination, never to the snippet.

Everything a snippet could seem to need, which file it goes into, where in that file, and how it is transformed, is a property of the place that receives it, because the same snippet reaches different destinations in different shapes. The configuration of Snipwise is therefore bundled in one TOML block, and a snippet section holds a name and a text and nothing else.

The alternative, a settings block inside each snippet section, was considered and not adopted. It puts the configuration next to the snippet it mentions, which reads well for a single snippet, but it scatters the snippet-to-file mapping over every snippet section, and it gives each such section two code blocks.

Strict Keys, Lenient Names¶

The schema refuses a key it does not know and accepts any name a marker gives it. That is not an inconsistency.

A key belongs to a vocabulary that Snipwise defines and can enumerate, so an unknown key means the author expected a behaviour that this version does not have. Refusing it is what makes it safe to add keys later, because an old Snipwise meeting a new configuration will complain instead of silently ignoring the new key.

A name belongs to a vocabulary that the author defines, where Snipwise has no list to check against.

One guard has to be explicit in exchange. With no list of headings to check a garbled name against, the name is defined by the marker grammar alone: one token of letters, digits, hyphens, underscores and dots, and nothing else. A marker with no name, or with a name that does not match, is malformed input. The rest of the marker line stays free for a remark.

The Shape of a Rule¶

The scope of a rule is a set of file-and-snippet pairs rather than a set of files, which is what makes a per-snippet render possible in a file that holds several snippets. Rules are therefore unordered and may overlap freely, because what has to be unambiguous is the pair and not the file. The one precedence there is, a narrowed rule outranking a rule that claims everything, is derived from what a rule claims rather than from where its table sits, so that the rules stay unordered. Without it, giving one snippet of a file its own render would force every other snippet of that file into a rule of its own.

A pattern that matches no file is refused, because a rule that selects nothing is almost always a typo or a forgotten rename, and reporting it costs nothing on a glob that has already been expanded. An exclude pattern is not held to that, since an exclusion that removes nothing removes nothing, and a file that a rule keeps out of its scope is not always present. What is refused is a rule left with no file at all once its exclusions are applied, because that is again a rule that selects nothing. A rule that has to subtract from its own patterns is usually two rules, but a broad patterns with one file taken out is common enough to deserve the key.

One Key for the Snippets of a Rule¶

The snippets key narrows a rule, and it names what the regions of a rule hold when nothing else can. The two readings are the same statement. A regex rule whose expression has no name group finds regions that carry no name of their own, and the single snippet such a rule claims is the snippet every one of those regions holds, so a second key for naming would only repeat what snippets already says. Two keys also pulled apart what belongs together: a rule that named its snippet did not thereby claim it, so a file holding that snippet beside marked blocks needed the name written twice, once to name the regions and once to keep another rule from claiming them.

The json scanner says both of those things in insert and therefore refuses snippets. An entry names the snippet of the value it addresses, and the entries together are every region such a rule can find, so the rule is narrowed to them without the author writing a name a second time. A list beside the entries could only repeat them, contradict them or reach past them, and the last of those is the one that does damage: a name claimed without an entry to write it takes the region holding it away from the rule that does find it, and a region whose owner never writes it is a block that has silently stopped being checked.

The Ladder of Scanners¶

The values of the scanner key form a ladder of increasing power, so that a project pays for the generality it uses and no more.

The whole scanner is the bottom rung, where a rule claims a file and searches for nothing. An expression says the same thing as (?s)\A(?P<content>.*)\Z, and it is a value of its own because the least general claim a rule can make should not be reachable only through the most general scanner. The warning that comes with an expression, that it may replace more of a file than intended, is answered here by the shape of the rule rather than by the care of its author, and the spelling it replaces has two mistakes that a scanner value cannot make: a missing (?s) leaves the file unowned until a lone copy is reported somewhere else, and (?m)^(?P<content>.*)$ turns every line of the file into a copy of the snippet. It gives up the messages that justify markers below, and needs none, because a scan that searches for nothing cannot fail.

Conceptually, the markers scanner could have been a special case of the regex scanner. It produces the same kind of region with the same kind of context. It is implemented on its own because an expression that fails to match leaves a block silently unowned, while a scanner written for markers can say that a begin marker is never closed, that an end marker closes a different name, or that a block sits inside another one. Those messages are what makes a mistyped marker easy to fix, which justifies the extra code.

The regex scanner takes its flags inline in the expression, so the schema needs no key for them. Together with the template of a rule, this scanner covers custom identification and custom replacement without executing any code, which is what keeps invariant 4 affordable.

The json scanner addresses values with RFC 6901 pointers, chosen over a query syntax because a pointer resolves to exactly one node, needs no dependency, and cannot silently match more places than intended. A pointer that does not resolve is malformed input, because creating the missing member would make Snipwise generate structure rather than copy text. Two entries that address the same value, or one that addresses a part of what another addresses, are refused because the file would otherwise depend on the order in which the entries were written. Both are properties of the pointers alone, so both are refused while snipwise.md is read.

Locating a JSON Value¶

Invariant 1 requires the character span of the addressed value in a JSON file. Snipwise finds it with json.JSONDecoder.raw_decode, which is a public method of the built-in json module. Every hard token, which is to say strings with their escapes, numbers and nested containers, is delegated to it, and what remains to be written is whitespace skipping and the punctuation of an object or an array.

The locator never decides what is valid JSON. The file is parsed with json.loads first, and its error is reported when that fails, so the locator only ever runs on text that a real parser has already accepted, using the primitives that accepted it.

Duplicate keys are the one case where json.loads and json.JSONDecoder.raw_decode disagree, because json.loads keeps the last member of that name and a forward scan finds the first. A duplicate along the path of a pointer is therefore an error rather than a silent choice.

A grammar-based alternative such as Lark is workable and pure Python, but it buys roughly sixty lines at the price of a dependency in every pre-commit environment, for the benefit of a single scanner, and with worse messages than the standard library gives away. A tree-sitter grammar costs compiled wheels per platform, which a tool that installs anywhere should not take on.

Writing a JSON Value¶

Invariant 1 holds in the other direction as well. The replacement is one value and nothing around it, so the key, the comma and the whitespace of the member it belongs to are never rewritten.

The layout inside the replacement is the only real choice, and only lines has to make it, because an array of strings can be written on one line or on one line per element. It is answered by taking the layout of the array being replaced, which keeps the indentation and the line endings the author gave it. A value that is not an array, and an array holding nothing, say nothing about a layout and give a single line. That is the layout which disturbs a file least, and an author who wants another one writes it out once, after which it is the layout that is matched.

A json fragment is spliced exactly as the template wrote it, so the layout stays with the author there. Only the whitespace around the fragment is dropped, because it is not part of the value and because the span that addresses a value holds none of it either, so a fix that left it in would never bring the file up to date. The fragment is parsed before it is written, and a bare scalar is accepted, because a scalar is a JSON value like any other.

Rendering¶

A rule renders with a Jinja template rather than with a list of named filters. One key is enough because Jinja already supplies the chaining, the naming and the arguments that a filter list would have to reinvent, and because the same key then serves the regex scanner, where the replacement is generated from the match groups as well as from the snippet.

The template context is closed. It holds the snippet and the results of the identification, and nothing else. This is the rule that keeps snipwise check a comparison rather than a build step.

The environment uses StrictUndefined, so that a mistyped name fails instead of rendering nothing. It keeps the trailing newline and it escapes nothing, because a snippet is text and the file receiving it is not necessarily HTML. Jinja supplies the plumbing but none of the useful filters, so the domain filters are ours to write and register.

What the context holds beyond content is up to the scanner. A scanner that lays its own result out, as markers does with the indentation of the begin marker, keeps that out of the context, because a value the template can apply and the scanner applies again is a doubling waiting to happen.

A Render May Reduce, Never Complete¶

A render applies formatting rules and may discard information. A template dresses a snippet in what its destination needs, and it is free to leave behind whatever that destination cannot use. Syntax is discardable like anything else: plain takes the Markdown off a paragraph that becomes a docstring, unwrap takes the line structure off a paragraph that becomes one line, and a template that strips the quotes off a TOML fragment is doing the same kind of thing. Discarding is safe because the original stays where it is and only a copy is written in the reduced shape.

Adding information is beyond the scope of a render. The quoting, separators and comment markers that a template writes are formatting, spelled out in the rule and belonging to the destination, which is Where Configuration Lives applied to the text rather than to the keys. Content is another matter. A date, a version number or an item that the snippet does not contain would be text that no copy in the project holds, which invariant 6 rules out, and the closed context is what leaves a template no way to reach it.

A filter therefore earns its place by reducing a snippet and never by completing one. plain may parse Markdown because it throws the markup away, whereas a filter that wrote TOML or YAML would reserialize a value on the author's behalf and decide a layout that invariant 1 leaves with the file holding it. The reduction is one-way in exchange: a rule that renders can never hold the original, because reading a discarded shape backwards would be a guess rather than a copy. See The Candidate Set.

The Opinions of the `plain` Filter¶

It refuses more than it renders. Paragraphs, inline markup and lists have one plain text form each, and everything else has several: a heading is a line, or a line and a rule under it, or a line in capitals, and a code block is verbatim text that no longer says it is code. A filter that picks one of those picks it for every project that will ever use it, so a construct with no single answer fails by name instead, which turns an unanswered question into a message rather than into a surprise that reaches a destination file. This is why the filter parses with the table rule turned on while snipwise.md itself is read as plain CommonMark: a table is not CommonMark, and a table that is not parsed is a paragraph of pipes, which is exactly the kind of thing that would arrive in a docstring unannounced.

It keeps the line structure of the snippet. A Markdown renderer reflows, because HTML decides where a line ends. Here nothing downstream does, and a project that writes its prose in semantic line breaks has already made the decision. Folding the reflow into plain would make two transformations impossible to ask for separately, so a soft break stays a newline and plain | unwrap is how a destination asks for one line.

A link gives its text. The address is the part of a link that a plain text destination cannot act upon, and a rule that wants it can say so with replace or with a snippet of its own.

A list is renumbered and gets a dash. The source markers of a list are markup like any other, and a plain text list has no renderer left to do the counting, so 3. 3. 3. in a README would arrive as three items numbered three. The dash is ASCII, which a docstring or a comment block all take without asking.

It takes no arguments. One rendering is one thing to document and one thing to test, and an argument stays a compatible addition for the day a project reports needing one.

Declaring a Source¶

A [[sources]] rule is what keeps the inference of the next section a convenience rather than a foundation. A project that never wants to be asked a question writes these rules and gets the determinism that a definition in snipwise.md gives.

Two keys of a [[targets]] table have no meaning here and are refused. A render asks Snipwise to read a transformation backwards, which a transformation has no inverse for. The json scanner, and the insert key with it, is refused for the reason that a JSON value can never hold the original either: the rule addresses the value instead of finding it, so nothing in the document says that it holds a text somebody wrote there. See The Candidate Set.

That a snippet has at most one source is a stricter answer than requiring several to agree, which would only fail on the day the files drift apart, that is to say at the moment nobody can tell which one was meant. A region being a source or a destination and never both keeps a check a single pass and makes a cycle impossible to express. Propagation is one-directional, which is not a limitation waiting to be lifted but the property that makes a render safe.

Snippets without a Definition¶

A snippet defined in snipwise.md and used in two files is stored three times, once as the original and twice as a copy. The third copy is what removes every ambiguity, and it is also what the author pays for on every edit. That price is worth paying for a snippet that several files share on equal terms, but not for a snippet that plainly belongs to one of them.

None of what follows touches a project whose snippets are all defined in snipwise.md. Such a project has an original for every snippet and is never asked a question.

The Candidate Set¶

Candidacy is decided by reversibility rather than by configuration: a block can be the original only if the text of the snippet can be recovered from it. A snippet with no candidate block anywhere is an error, reported while the targets are scanned and not when a difference eventually appears, because such a snippet asks Snipwise to keep copies of a text that is written nowhere in a form Snipwise is able to read.

Recovering the text from the winning block also normalises it, so snipwise fix rewrites the winner along with the receivers. This is a single correction rather than a moving target, because the recovered text renders back to itself from then on.

Which Copy Wins¶

The four answers fall into two kinds. A definition in snipwise.md and a [[sources]] rule are declarations, made by the author and read from the configuration. The agreement of the candidates and the comparison with the last commit are inferences, reached only when the author has declared no original.

The order of the two declarations is a formality, because a snippet that is both defined in snipwise.md and claimed by a [[sources]] rule is an error naming both, rather than a precedence of one over the other. Rules are few and written by hand, and two originals for one snippet is a mistake in every project that manages to contain it.

The order of the two inferences is not a formality. Agreement is a question about the working tree alone, so snipwise check can ask it and reach the same answer that snipwise fix does, which is what keeps the two commands agreeing wherever they can. Only the comparison with the last commit reaches outside the working tree, and only snipwise fix ever does. Invariant 5 is worded around that split.

The Comparison with `HEAD`¶

The last answer is a three-way merge in which the commit at HEAD is the base, which is why a block that does not exist there takes no part in it. Leaving such a block out of the comparison entirely is what lets the merge answer with the base itself: when the copies that existed at HEAD all still hold what they held there, that text is what the block added since receives.

The committed text is scanned with the rules as they are now, because the base is a text and not a configuration. A rule that changed in the same commit therefore reads the old file with the new rules, which is the intent, since the question is what the copies of a snippet held and not how they were found at the time.

Snipwise reads the HEAD version of a file through git, and only of files that patterns already selects, so invariant 5 holds at that revision too. Running git is not running code out of the repository, so invariant 4 is untouched as well. The blob is read with git cat-file rather than with git show, which is the narrower door of the two: a repository that defines a smudge filter should not be able to make Snipwise run it, even though that filter is git's own configuration rather than something snipwise.md points at. The price is that a repository which does rewrite files on checkout has a working tree that differs from its blobs for reasons of its own, which makes a block in such a file look edited at every run. That is accepted rather than repaired, because repairing it means applying the filters, which is the thing the narrow door avoids. Every way this rule can fail degrades to an absence of evidence rather than to an error.

Snipwise Never Asks¶

Being asked which copy wins would save the second half of a two-step repair, which is a small thing next to what refusing to prompt buys: nothing can hang a build that happens to allocate a pseudo-terminal, the command behaves the same everywhere, the decision is recorded in the shell history rather than in a moment that leaves no trace, there is no second verb for hooks to migrate to, and no pseudo-terminal anywhere in the test suite.

The order in which the two could be adopted is what settles it. Prompting when a terminal is attached only ever changes a case that exits 2, so it stays a compatible addition. Withdrawing it once hooks have come to rely on it would not be. What it waits for is a report that the second command is a nuisance in practice, rather than the suspicion that it might be one.

What Snipwise Reports¶

A name that no heading has to declare moves work from the parser to the count of its parties. That count is the only thing standing between a mistyped marker and a block that goes stale, so Snipwise reports on every name it has seen and not only on the ones it wants changed. The --verbose listing answers the two questions that a required heading would have answered implicitly, which are what names this project has and where each of them lives.

A snippet with fewer than two parties is doing nothing, and is far more often a typo than an intention, which is what the count is for. A definition in snipwise.md is one of the two parties rather than a promise that a name is meant, so a definition that nothing receives is as much a mistake as a lone copy is.

A Lone Copy Does Not Reach a Commit¶

A lone copy is the state every snippet passes through on the way to having two copies, which is an argument for expecting it and not for letting it be committed. The state belongs to the minutes in which a snippet is being written, and Snipwise is run by a pre-commit hook, where those minutes are over. A snippet that still has one place by the time a commit is made is a mistake often enough that refusing it is the better default.

A warning is the softer answer, and it is not enough, because a hook that passes prints nothing. A report the author has to remember to go and read by hand is not a defence against the mistake it exists to catch, and it is the only defence there is, because a name that no heading defines is a snippet like any other rather than an error.

The exit code is 2 and not 1. Code 1 says that a block differs from what it should hold, and here no block does. Code 2 is what Snipwise already uses for input it will not act on, including input it could act on and refuses to, such as a snippet with two source regions or one that is both defined and sourced. A lone copy belongs with those, because it is actionable and it is very probably a typo.

Nothing is rewritten, which is what code 2 means everywhere else in Snipwise. The price is that a lone copy in one corner of a project holds back the repair of every snippet in the rest of it. That is accepted, because the alternative is a run that writes files and fails anyway, which exists nowhere else in the tool.

snipwise fix does not repair a lone copy either, and cannot. The two ways out are to give the snippet a second place or to take away the place it has, and choosing between them is a decision about what the project is for. Snipwise never asks and never guesses, so it reports the choice and stops.

A refusal here is a rule that only 1.0 can adopt. Turning a report into a refusal turns an exit of 0 into an exit of 2, so a project that has pinned a revision and passes today would start failing on upgrade. After 1.0 this part of Snipwise may only move the other way, from refusing a name to accepting it, which is why a stricter rule about names is declined rather than reserved.

The counts behind the check and the listing cover the whole selection even when file names are given on the command line, because a count made over the named files alone would report a lone copy for every snippet whose other copies simply were not named, which is the normal case under pre-commit. Reading the whole selection is what a check of a snippet without a definition does already, so this asks for nothing that was not being read.

Reserved for Later¶

Each of these is a compatible addition, which is why none of them is in 1.0.

The python scanner. Custom scan and render written by the user. Whether the code is referenced by path or embedded in snipwise.md is left open, and the implementation of the other scanners should settle which is worth having. A referenced module is lintable, testable and importable; an embedded block keeps the project in one file. If a block is ever embedded, it is marked by the info string of the code block and not by the title of its heading, because a heading that is not a snippet name is documented as free for the author to use.
A source that reads a value out of a JSON file, which needs the inverse of shape before it can be written.
snipwise.toml. Should the configuration outgrow a fenced code block, reading it from a separate file when that file exists breaks nothing.
Options on a marker line, for a setting that applies to one block rather than to a rule, written as a single token naming a rule, behind a sigil:

<!-- snipwise.md BEGIN tagline @one-line -->

A key=value syntax is the shape to avoid. Values would need quoting, quoting would need escaping, and all of it would sit in a host comment that already has a closing delimiter to survive. A bare name keeps the marker free of grammar and leaves the template in the TOML, where it is readable and reusable. The sigil also reserves the slot without any help from the parser, because nobody has reason to write @word directly after a snippet name.

A rule narrowed with snippets already covers rendering per file and per snippet, so this is only needed when two blocks of the same snippet in the same file want different renderings, which can wait for a real report.

Abandoned Ideas¶

Settings that Require a Heading¶

A key such as declare = "required" would refuse a name that snipwise.md does not mention. It asks for exactly the bookkeeping a copying tool exists to remove: a heading per snippet, maintained by hand, and a rename to be made in two places. The lone-copy error already refuses the shape that a mistyped marker takes most often, and what the setting would add on top of that is the partial rename that leaves two blocks behind, which is rare enough that a --verbose listing is the proportionate answer. A project that wants the guarantee can still have it by writing the headings anyway.

Several Settings Blocks¶

Prose between chunks of configuration is what TOML comments are for, and a snipwise.md file is expected to contain example code blocks, including the one that snipwise init writes. Parsing every block before the first level-2 heading as settings would incorrectly read the example as settings. Were it ever wanted, the safe form marks a block by its info string and never by its position, which stays compatible with a lenient parser because an unlabelled example goes on being ignored.

A Cache of the Last Observed State¶

A file recording the text of every block at the end of the last run would extend the HEAD comparison to projects without git, and to a second edit within one commit. It is declined because it is right only while it is fresh. A block the cache never saw, a branch switch, a git stash, a checkout by a colleague, a run that somebody skipped: each of these leaves a base that never existed in the tree. The failure is silent, because a stale base still names a single winner with full confidence, whereas HEAD is a base that git maintains, that both sides of a pre-commit hook agree on, and that is wrong only when the working tree is wrong. The durable form of remembering a decision is a [[sources]] rule, which is why the report that reaches an ambiguity prints one.