Skip to content

The release checklist runs three baselines against four rule-bearing ledgers, so expected_since_2.1.0.toml is audited by nothing #503

Description

@derek73

Rationale

The release checklist in root AGENTS.md opens with:

Run the differential gate at EVERY baseline that has a ledger with rules, and
confirm each exits 0.

and then gives three copy-pasteable commands against four rule-bearing
ledgers. DEFAULT_BASELINE = "2.2.0", so the three cover 1.4.0, 2.0.0 and 2.2.0.
--baseline 2.1.0 is reached by no line.

Measured 2026-09-03: all four ledgers carry rules — 75 / 66 / 59 / 1 [[change]]
entries at 1.4.0 / 2.0.0 / 2.1.0 / 2.2.0. So expected_since_2.1.0.toml and its
59 rules are audited by nothing a releaser executes.

What that ledger misses

Three checks are per-ledger and run only inside compare.py, so a baseline
nobody runs gets none of them:

The last one is why this is being filed now rather than left as a comment. #497
added a check whose whole purpose is that a recorded value nobody re-measures
agrees with itself forever, and at 2.1.0 nobody re-measures it.

Why it is latent today, and what arms it

expected_since_2.1.0.toml's _RECORDED_DIFFS section is empty, so the shape
check has nothing to miss there right now. #501 is open on whether the 2.x
sections stay empty — it records six contested diffs across the two 2.x ledgers
with no pins, one of them at 2.1.0. The first 2.1.0 row to land would be checked
by nothing in the documented process.

The dormancy and over-declaration gaps are live now, not latent.

Options

  1. Add the fourth command. One line in the checklist. It is the obvious fix
    and it costs a sub-second run — measured, a whole gate run is ~0.35s at 1.4.0
    and ~0.57s at 2.2.0, and all four back to back are ~2s.
  2. Make the checklist derive the list rather than enumerate it — loop the
    expected_since_*.toml glob, so a fifth ledger at the next release is covered
    without anyone remembering. Slightly more to paste; cannot go stale.
  3. Leave it and say so. If 2.1.0 is deliberately out of the release audit,
    the "EVERY baseline that has a ledger with rules" line should say so instead,
    since it currently promises what the list does not do.

Option 2 is the one that matches what the codebase does elsewhere — the corpus
glob in compare.py is deliberately a glob rather than a list, with the comment
"a corpus that has to be asked for by name is one that stops being run".

Recompute

grep -c '^\[\[change\]\]' tools/differential/expected_since_*.toml
grep -n 'baseline' AGENTS.md
python -c "import re;print(re.search(r'DEFAULT_BASELINE = \"([^\"]+)\"',open('tools/differential/compare.py').read()).group(1))"

Provenance

Found by a design-docs-reviewer pass on #502 (PR for #497). AGENTS.md
carries a note recording the gap; this issue is what gives it an owner, and the
note cites it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

docsDocumentation fixes and updatestooling

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions