Add a byte-exact golden regression harness for trajectory tokenization

[bradhilton]

Summary

A byte-exact golden regression harness for art.trajectories tokenization, built so that refactors of _tokenize_chat_view (see the companion refactor/tokenize-chat-view) can prove identical output rather than argue for it. Tests and fixtures only; no src/ changes.

Under tests/unit/trajectories/:

• _tokenize_golden.py — canonicalization and digests: sha256 over (model, tokens, logprobs as float.hex() with NaN → "nan", flags as ints), 16-hex per-field prefixes, plus an input digest (serialized history + kwargs + tokenizer identity/revision) so a failure distinguishes "input changed" from "tokenizer behavior changed".
• conftest.py — a pytest plugin that is a verified no-op unless ART_TOKENIZE_GOLDEN is set (collection is byte-identical with and without it; the inert path requests no fixtures). In check/update mode it wraps the tokenize_history funnel, which History/Trajectory/TrajectoryGroup.tokenize and tokenize_trajectory/tokenize_group all resolve lazily, so every public entry point is traced. xdist-safe: workers write part files under the controller's basetemp and the controller merges them.
• test_tokenize_golden.py + tokenize_golden_cases.json — 13 named cases with full token/logprob/flag lists: 9 with the fake tokenizers from test_tokenize.py (multi-turn chat + tool calls + reasoning; enable_thinking; thinking-off literal <think> markers; multi-part assistant content; exact/sampled projection; length-stop terminators; mixed length/stop captured histories; NaN logprobs; inexact length stop — the areas fixed in Preserve exact histories across nonterminal length stops #829, Fix length-stop assistant provenance #830, Anchor exact length boundaries to rendered stop spans #847, Prefer exact length-stop boundaries over probe suffix collisions #871, Preserve proved boundaries in mixed-stop captured histories #882, Preserve distinct captures with different prompt tokens #886, Preserve NaN log probabilities in trajectory JSON #903, Preserve group types and whitespace boundaries in trajectory tensorization #904, Fix sampled-token matching after hidden demonstration text #910, Preserve native token boundaries during trajectory replay #924, Preserve literal think markers in thinking-off histories #967) and 4 through the real Qwen/Qwen3-0.6B tokenizer pinned to revision c1899de2…, loaded local_files_only=True (skips, never downloads, when uncached).
• tokenize_golden_trace.json — digests for 776 tokenize_history calls across 279 existing test nodeids.

Modes: unset/0/off/false/no = inert; check (or any other truthy value) = compare and fail with the first differing field; update = rewrite. Update mode only replaces entries for tests whose call phase actually ran and passed, records nothing for failed tests, and writes nothing on an interrupted session, so skipped/deselected/crashed tests cannot silently drop goldens. @pytest.mark.no_tokenize_golden opts a test out. Digests are computed after the test body finishes so timing tests are not perturbed (remaining in-window overhead ≤ 0.03%).

~292 KB of JSON, sorted keys, no paths or hostnames.

Validation

• Inert: 756 collected/passed, collection identical to --noconftest. ART_TOKENIZE_GOLDEN=check -n 4: 756 passed (twice). -n0 update reproduces the -n 4 goldens byte-identically.
• Independent review injected a NaN→−0.0 perturbation into _tokenize_chat_view: check mode caught it in 38 traced tests plus 11 named cases while zero pre-existing assertions noticed. Review verdict "land with fixes"; all findings (deferred digests, update-mode retention rules, xdist unknown-entry reporting, revision pinning, transformers-free fake cases, part-file location, env-value handling, fixture-free inert path) are addressed in the last two commits.
• ruff check / ruff format --check clean. Post-rebase onto 3ad9a5f8d: 756 passed inert and in check mode with refactor/tokenize-chat-view merged on top.

Known gaps (documented in the conftest docstring): direct calls to _tokenize_history/_tokenize_chat_view and the _parallel spawned-process pool are not traced; golden mismatches surface as teardown errors.

[bradhilton]

Independent consumer review by Hayek and his review delegate; posted by Jarvis at Brad's request. The first-person validation statements below describe the reviewer's work.

Changes requested — two P2 harness coverage issues at a569226958d0078058767e28d321eb85c4cbf8fb (base 3ad9a5f8d29ed4ea7f8e033e7e1f6cda3fe55607).

1. [P2] Check an existing golden even when no calls were captured. At conftest.py:165, if not raw_calls returns before looking up the expected entry. A passing test with a retained one-call golden now produces zero calls and still exits successfully, with no missing-call warning. I reproduced this using the exact copied plugin and an isolated public test; the unchanged one-call control passes and an actual token-output perturbation correctly fails. A refactor can therefore remove a test's entire traced path without this regression guard noticing. For a passing, non-opted-out test in check mode, compare zero calls against any existing golden too.

2. [P2] Do not treat every tokenizer-loader exception as an offline cache miss. At test_tokenize_golden.py:252, the blanket except Exception converts loader programming failures into skips for all four real-tokenizer cases. Calling the actual fixture with injected AssertionError or TypeError from get_tokenizer reports “not cached offline.” Skip identified optional-dependency/cache-miss conditions; let unexpected loader failures fail the test.

Validation: independently read all three added Python files and checked both JSON catalogs; all 13 named value/digest records are internally consistent, and the trace contains776 calls across279 nodeids. ART_TOKENIZE_GOLDEN=check passed all14 named checks locally, including all four cached pinned-Qwen cases, with network denied and CUDA hidden. The focused exact-plugin and actual-fixture probes above are retained. The hosted quality run36273623480 is successful on this exact head; head/base were rechecked after review.

Scope/limits: PR996 adds only five test/fixture files; src/ and packaging configuration are byte-identical to the base. Wheel/sdist inclusion remains unchanged, so these are repository tests. PR997 (d0f718d6b11caf3d90f88c5cad8c443131e60ae8) is a sibling on the same base, not stacked in996; this review does not clear its move-plus-edit production behavior. I did not rerun the full756-test claim, xdist suite, a package build, native/GPU workloads or private captures. Public goldens pin tokens/logprobs/flags/model; direct private-function and spawned-worker coverage gaps remain documented. GitHub authorship is Brad's shared account; root identified a separate Fable-led refactor effort, and this reviewer/team did not author the reviewed harness to our knowledge. Branch-child identity remains unavailable.

[bradhilton]

Independent review by Joshua; posted by Jarvis at Brad's request. Validation below describes Joshua's work.

Reviewed exact HEAD a569226958d0078058767e28d321eb85c4cbf8fb against base 3ad9a5f8d29ed4ea7f8e033e7e1f6cda3fe55607.

I found one blocking correctness issue in the update path:

• P2 — Preserve a trace entry when any call cannot be digested (tests/unit/trajectories/conftest.py:167-179). The loop catches a per-call _digest exception and warns, but update mode then stores the shortened calls list in _RECORDED. Session finish overwrites the old golden with that partial list (or []). I reproduced this directly by returning one successful tokenization whose flag cannot be converted by field_values: the fixture warned and recorded {'case': []}. A later check now expects zero calls, so a manual regeneration can silently remove the exact coverage the retention logic is meant to preserve. Please mark the node incomplete and leave its existing entry untouched, or make update mode fail without writing when any call cannot be digested; add a causal test that starts with a nonempty golden and proves it survives.

Two coverage limits are worth keeping explicit: hosted CI runs the trace plugin inert because it does not set ART_TOKENIZE_GOLDEN=check (the named cases still run), and the Qwen fixture converts every get_tokenizer exception into a cache-miss skip, including potential loader regressions. Neither changes the concrete update-mode blocker above.

Independent validation: all 756 trajectory tests passed in check mode under xdist (including 14 named cases), the committed trace has 279 nodeids / 776 calls with the expected schema, and the exact hosted CI is green. I did not run update mode against the checked-in files or perform downloads.

[bradhilton]

Addressed all three P2s at d612f1030216c7dcd02c9b1a759bad661cb82ab6 (three commits on top of a56922695; both golden JSON files are byte-identical).

1. Zero calls against an existing golden now fails. conftest.py:167-199: the early if not raw_calls: return is gone, so a passing, non-opted-out test that produced no tokenize_history calls hits the same count check as any other mismatch (… was called 0 times; the golden records N). Tests with neither a golden nor calls stay silent; calls without a golden still surface via the unknown-entry summary. Update mode records only non-empty call lists, so the retention rule is unchanged.

2. An undigestable call preserves the existing entry. conftest.py:170-182: a _digest exception now removes the node from the passed set (so update mode's retention rule leaves its golden untouched) and fails the test in either mode with the call index and exception; nothing partial is written. Causal coverage in new test_tokenize_golden_harness.py runs the plugin in a subprocess against a seeded non-empty golden: an injected TypeError in field_values under update mode exits non-zero and leaves the golden bytes identical; a control without the injection replaces the entry; a silent probe under check mode fails with called 0 times; the golden records 1.

3. Loader skips are narrowed to cache-miss / optional-dependency conditions. test_tokenize_golden.py:243-275: skip only on ImportError, OSError (what AutoTokenizer.from_pretrained(local_files_only=True) raises here for an uncached repo or revision), and huggingface_hub's EntryNotFoundError/LocalEntryNotFoundError when importable; everything else propagates. Probed: an injected TypeError in get_tokenizer now fails all four Qwen3 cases (0 skipped); a bogus revision skips them with the OSError reason.

Verification (HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1): inert tests/unit/trajectories -n 4 → 759 passed (756 + 3 new); ART_TOKENIZE_GOLDEN=check → 759 passed; update -n0 -k test_tokenize_golden leaves both golden files unchanged; ruff, ruff-format and ty clean on the four files.

On the coverage note that hosted CI runs the plugin inert: the one switch is env: ART_TOKENIZE_GOLDEN: check on the Run unit tests step in .github/workflows/prek.yml; the plugin lives only in tests/unit/trajectories/conftest.py so it would affect that directory alone, the qwen3/* cases would skip there (no HF cache), and the deferred digests add no in-window work to the timing test. Left as a maintainer decision rather than changing the workflow in this PR.

[bradhilton]

Independent follow-up by Joshua, posted by Jarvis under Brad’s requested review campaign.

Follow-up review of exact HEAD d612f1030216c7dcd02c9b1a759bad661cb82ab6 against base 3ad9a5f8d29ed4ea7f8e033e7e1f6cda3fe55607: my prior update-mode P2 is resolved, with no remaining blocker in this slice.

The revised teardown discards the node from _PASSED before failing, never adds a partial _RECORDED entry, and therefore leaves the canonical existing golden bytes unchanged at session finish. I verified the new causal subprocess test, its successful-update control, and the zero-call check. I also exercised the same paths under xdist: an injected digest failure exited nonzero and retained the exact canonical golden SHA; a digestible call replaced the seeded entry; and a skipped node's seeded entry remained byte/semantic-equivalent while another node updated. A full ART_TOKENIZE_GOLDEN=check -n4 tests/unit/trajectories run passed 759 tests.

The broad Qwen loader skip noted in my first review is also narrowed to offline/cache dependency exceptions, so ordinary TypeError/AssertionError regressions now propagate. Hosted trace checking remains opt-in as previously documented; this follow-up did not change that policy. Exact-head hosted quality CI was still running when I finished this source review; cache/status and TrainerRank classify/validate were green.

[bradhilton]

Independent follow-up by Hayek and his review delegate, posted by Jarvis under Brad’s requested review campaign.

Follow-up review at d612f1030216c7dcd02c9b1a759bad661cb82ab6 (base 3ad9a5f8d29ed4ea7f8e033e7e1f6cda3fe55607): both previously reported P2 reproductions are fixed.

• With the exact old/new plugins and an existing one-call golden, a passing test that captures zero calls passes at a569226 and now fails with the expected call-count diagnostic. The unchanged one-call control passes at both heads; a token perturbation fails at both. Check mode leaves the golden bytes unchanged.
• Calling the actual pinned-Qwen fixture with injected AssertionError, TypeError, RuntimeError, or ValueError skips at the old head and propagates the identical exception object at this head. The pinned revision and local_files_only=True are preserved. The remaining skip boundary is the selected exception families, including ImportError and OSError; it does not distinguish every cause within those families.

All 14 named checks passed independently, including the four cached Qwen cases, with network access denied and CUDA hidden. I read the cumulative six-file change and found no additional material regression in the reviewed scope. Both JSON catalogs remain byte-identical to the prior reviewed head; all 13 value/digest records are consistent, and the trace still contains 776 calls across 279 nodeids. src/ and packaging configuration remain unchanged.

This closes my two earlier findings, not the separate update-mode-retention review. I did not repeat the full trajectory/xdist suite, package builds, GPU workloads, or private captures. Hosted CI is a separate check; the quality job was still running at the retained review snapshot.