S2: parser conformance across 18 harness versions - #21
Merged
Conversation
The capture adapter bets on two things: that the transcript format is stable enough for one parser to span releases, and that the version key is there to key on. S1 and S3 both run through that adapter, so it is upstream of results already published. The schema census that chose the subject is committed alongside, and its findings are recorded in the pre-registration as inputs rather than results. 18 harness versions, 323,430 records; the four content-bearing record types appear in all 18 with stable core key sets, and the churn is additive and concentrated in `system` metadata the IR does not need. One census finding shapes the bar. 35% of records carry no `version` field — twelve bookkeeping types that never carry one, against four content types that always do, with no type inconsistent. The design selects parsers by "the version field each record carries", so a fail-closed rule written literally would reject every real transcript in the corpus. Clause 3 requires both directions: refuse an unknown version, admit a versionless bookkeeping record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
A version-keyed parser that extracts exactly what S3 showed the Distiller needs — human turns, tool calls with their inputs, tool results, model, cwd, and the link from a subagent sidecar back to the tool call that spawned it. A measuring instrument, not the product's capture adapter. Conformance over the local store: 1681 files, 100.00% parsed, none refused, all 18 versions. Zero dangling tool_results across 209,701 content records, so the pairing invariant the Session IR wants to rest on is a property of real transcripts rather than an assumption. The redaction pass needed three fixes before it met its clause, and the planted corpus earned its keep by being awkward on purpose: - a password containing `&` escaped a value class that stopped at punctuation; - `AWS_SECRET_ACCESS_KEY=` never matched, because `_` is a word character so `\bsecret\b` cannot match a keyword buried in an env-var name — that one would have missed real env dumps, not just the plant; - base64-wrapped secrets decoded to an invalid length, because anchoring `\b` after the `=` padding excluded the padding from the match. Now 17/17 caught, 0 false positives on the benign corpus, and every caught value verifiably gone from the redacted text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
The planted corpus scored 17/17 with zero false positives on a benign set,
and the benign set was the problem. Scanning the real store fired on 50.9% of
files, and a masked sample of those hits showed the detector was largely
finding its own shadow:
- `openai-key` matched the word "risk-": `sk-` followed by 32 word characters
is satisfied by every `2026-03-27-risk-disclosure.md` in the corpus;
- `assigned-secret` matched `${POSTGRES_PASSWORD:?must be set}`, a shell
default that references a secret rather than containing one;
- `url-credentials` matched `${POSTGRES_PASSWORD}@host`, the same mistake in a
connection string.
Narrowing those required one correction of its own: an initial "a path is not a
credential" rule matched any `word/word` and swallowed
`wJalrXUtnFEMI/K7MDENG/…`, because an AWS secret key contains slashes and looks
exactly like a relative path. The rule now needs a leading separator or a
trailing extension.
Every one of those false positives is now a permanent case in the benign
corpus. The invented benign set contained not one of them, which is the point:
a detector is only as honest as its hard cases, and these came from data the
author did not choose.
Also reports invariant violations per version, which clause 2 asked for and
which separates standing format properties from release churn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
All four pre-registered clauses met, and the fourth was mis-specified. One parser read 1,681 files across all 18 harness versions with nothing refused, because every key it reads is present in every version for every record type. Tool-call pairing is exact: zero dangling tool_results across 35,510. The Session IR can rest on that. Three things it cannot rest on. `session_id` is not a duplicate of `sessionId` — in all 15 disagreeing files it names a sibling transcript, so it is a resume pointer and treating the two as synonyms misattributes sessions. 29 files move between working directories mid-session, across 15 of 18 versions, so workspace pinning cannot assume one. And 4.4% of subagent sidecars resolve to no tool call anywhere in their project, so subagent work has to be reported as unattributed rather than dropped. The fail-closed rule needs restating. 35% of records carry no version field at all, so selecting parsers by "the version field each record carries" rejects every real transcript when implemented literally. And versions arrive every 3.9 days and overlap, so strict per-version allowlisting breaks capture about twice a week while buying little — every field the parser needs was stable across all 18. Clause 4 passes and is the wrong gate. 17/17 on plants, but scanning the real store fires on 50.3% of files, and the largest contributors are code: a pricing module's `*_micro_per_token=` field names, matched for ending in "token". Recall against secrets the author planted is easy; precision on real transcripts is the hard part, and a redactor that flags one file in two either blocks every distillation or trains users to waive findings. No fixture corpus was produced, for that reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
Both published measurements run through the transcript-reading layer, so its ability to read real sessions across harness versions is upstream of them and belongs in public. Leads with the result and then with what did not survive: one parser read all 1,681 local transcripts across all 18 versions, and three assumptions failed — a session id that is really a resume pointer, sessions that move between working directories, and subagent transcripts that cannot be traced to the call that started them. The redaction paragraph is deliberately the most limiting thing in the section. Exams are meant to be shareable and the scanner is not trustworthy enough yet, so no shareable fixtures were produced. Saying that plainly now is cheaper than discovering it after someone exports one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
GitHub's push protection refused the first push of this branch, correctly, on the Slack plant in the planted-secret corpus. The literals are now assembled from fragments at run time, so no contiguous string in the file matches a scanner — the detector finds nothing in its own source — and the plants still score 17/17. The block is itself a finding, recorded in the results. A committed planted-secret corpus fights every scanner it meets: the host's push protection, the repository's own secret scanning, and whatever the developer runs locally. The only ways through are to allowlist real detections or to turn scanning off, and both are worse than the corpus is worth. The design proposes exactly this corpus running in CI, so it has to be synthesised at test time. History was rewritten to remove the literal from the two commits that carried it, rather than allowlisting a real detection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last outstanding Phase 0 item. Both published measurements run through the layer that reads a harness's own transcripts, so its ability to read real sessions across releases is upstream of them.
All four pre-registered clauses met — and the fourth was mis-specified.
One parser spans every version
All 18 versions (
2.1.142…2.1.220), 1,681 files, nothing refused. Every key the parser reads is present in every version for every record type; churn is additive and lands insystemmetadata a replay never reads. Tool-call pairing is exact — zero danglingtool_results across 35,510. The Session IR can rest on that.Three assumptions that did not survive
session_idis notsessionId. In all 15 disagreeing files it names a sibling transcript — it is a resume/fork pointer. Treating them as synonyms merges unrelated sessions. S3's subject session was one of these files.The fail-closed rule needs restating
35% of records (113,911) carry no
versionfield — twelve bookkeeping types that never carry one, four content types that always do, no type inconsistent. Selecting parsers by "theversionfield each record carries", implemented literally, rejects every real transcript.And versions arrive every 3.9 days and overlap. Strict per-version allowlisting breaks capture roughly twice a week while buying little, since every field needed was stable across all 18. Recommendation in the summary: fail closed on unknown record type and missing core keys, parse unknown versions, and mark capture fidelity unverified — vocabulary the design already has.
Clause 4 passes and is the wrong gate
17/17 on plants, reached only after three corrections the awkward plants forced. But the clause never measured precision, and scanning the real store fires on 50.3% of files. A masked sample showed the biggest contributors are code: a pricing module's
input_micro_per_token=/output_micro_per_token=/cached_input_micro_per_token=field names, matched for ending in "token". A redactor that flags one file in two either blocks every distillation or trains users to waive findings.Three FP classes found against the real corpus and fixed —
risk-disclosurematching an OpenAI key,${VAR:?must be set}shell defaults,${VAR}@hosttemplated URLs — each now a permanent benign case, because the invented benign set contained none of them.No fixture corpus was produced, for that reason.
Two findings from mistakes made while measuring
Review notes
2.1.xover 66 days. The design mentions a "2.0 inline-sidechain era"; no such transcripts exist here, so that era is unmeasured.🤖 Generated with Claude Code
https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ