Behaviour Probes¶

The design of Animo depends on many behaviours of typst 0.15.0 and of the browsers it targets. Findings records each of these behaviours, as verified by experiment. See Where the Specification Lives for where to find Findings. A new release of typst or of a browser could change any of these behaviours.

A probe asserts such a behaviour itself, rather than a feature that happens to depend on it. The difference shows when a test fails. Suppose that a new typst release stops emitting data-typst-label for labelled blocks. A feature test then reports that reveal animations are broken, while a probe reports that typst no longer emits the attribute. The first failure leads to debugging Animo, and the second leads to the release notes of typst.

A probe therefore has these properties:

• it asserts one entry of Findings, and names that entry in its module docstring;
• it is independent of every Animo feature, and imports nothing from src/;
• where its claim is a comparison, it states why the comparison is not vacuous. The check that the comparison is not vacuous is often a separate probe.

Probes live in probes/ and run in the same three tiers as the feature tests, through the same fixtures. See Testing for the tiers.

pytest probes             # all of them, against the pinned typst
pytest probes -m browser  # the ones that need a browser, in every engine available

A separate workflow also runs the probes every week against the newest typst release. This workflow blocks no pull request, and reports its results in the summary of the run.

Failing Probes¶

A failing probe indicates that a documented behaviour of typst or a browser has changed, and the weekly workflow exists to detect such changes. Never fix a failing probe by relaxing its assertion. Take these steps instead.

1. Decide whether the finding was wrong when it was written, or whether a newer release has changed the behaviour.
2. Edit the entry in Findings to say what holds now, and on which release it was observed. A finding is a claim about a point in time, and it is written as one.
3. Update the probe in the same commit. When the behaviour that the probe guarded is gone, delete the probe together with its finding, rather than leaving the probe skipped.
4. Check what else depended on the finding. Every entry in Findings supports some part of the design, and the design document says which part.

New Findings and Their Probes¶

Work on the design or the implementation sometimes depends on a behaviour of typst or of a browser that Findings does not record yet. That behaviour becomes a new finding, and gets a probe in the following steps.

1. Add the finding to Findings, with the measured numbers and the release on which they were observed.
2. Write the probe in the module that covers that part of the design, or in a new probes/test_<topic>.py for a new topic. The module docstring names the finding, and the docstring of the test says what would break without the behaviour.
3. Build the mechanism by hand, without Animo. For example, a probe for the region footprint rule calls layout and measure itself, so that a failure cannot come from a bug in Animo.
4. Add a row to the table in Probe Modules.

Two rules of thumb apply. Prefer an assertion inside the document (#assert) over one in Python, because a failing #assert points at the line where the behaviour differs. Also prefer comparing two values from the same test over comparing one value with a hardcoded number. A hardcoded number also records details that the finding is not about, such as the metrics of the fonts that typst embeds. The probe then fails when such a detail changes, even though the finding still holds. For example, the probe that checks whether measure gives the same size in both targets takes the size from the HTML compilation and passes it to the paged compilation through --input. A typst release with different font metrics changes the size in both targets alike, so this probe fails only when the two targets disagree.

Answers per Engine¶

Several probes hold a table with one expected answer per engine, because the three engines do not always agree. Every entry of such a table has to come from a run in the engine that the entry names. An entry copied from another table or from another engine is an assumption that looks like a measurement. With such an entry, a browser probe can pass while the behaviour it guards is absent.

The risk is largest for webkit, because webkit runs only in a container outside macOS and the debian family. See Which Engines Run Where. Measure in webkit before adding a webkit entry to a table:

pytest --browser webkit

Until that measurement exists, the table says that the webkit entry is unmeasured, and the probe asserts only what was measured. For an unmeasured engine, a probe may accept either of two answers, as long as it names the answer for each engine that was measured, because that is still a claim based on measurements. An invented number is not such a claim.

Observations That Cannot Be Probed¶

An observation that no probe can assert does not belong in Findings, because nothing would notice when a release of typst or of a browser changes it. A measured cost, such as the time a deck takes to compile or to paint, belongs in benchmarks/ and in Performance. A fact about the source code or the history of typst can appear in an entry only as background. The probe of that entry asserts the behaviour that follows from the fact.

Probe Modules¶

The table lists the probe module for each entry in Findings.