Phase 1: the run seam and the Claude Code reference runner (ADR-0008) - #43
Merged
Conversation
The second seam. port.Run drives a harness through work it has already
done once; the Claude Code reference adapter implements it against print
mode, and a shared conformance kit checks the contract half.
The guarantees were measured rather than read. Three probes against the
real binary settled what the documentation could not:
- --max-turns exists and works, though `claude --help` does not list
it. An earlier draft declared turn caps unenforceable on the strength
of that omission and would have refused a cap it could keep.
- Neither cap bounds anything. A run capped at $0.001 cost $0.0047; a
run capped at one turn reported two. A cap is a threshold noticed
after it is crossed.
- Both cap hits exited 1, the same status as a crash. The exit code
cannot be the field that decides, so the harness's own subtype is.
That last one is why StopReason exists and why exactly one of its values
returns true from Decides(). ERROR is not FAIL, moved out of a reviewer's
memory and into the type.
Guarantees are declared per cap and per assignment field, and asking for
something undeclared is refused before a process starts by a shared
port.Refuse — a dropped cap would produce a sitting that looked clean and
a budget nobody kept. The check runs one way, like the capability check
it mirrors.
Query.Native closes the loop from a sitting back to a Session without
either adapter importing the other. It matches on the transcript's file
name: across 400 sampled transcripts the name was the session identifier
every time, and matching on contents would mean opening every file in a
store that held 2,485.
The conformance kit deliberately stops at the process boundary. Reading a
transcript is free; administering a sitting starts a coding agent against
a real model on somebody's key, and a kit that spends a contributor's
money to prove good behaviour is a kit that gets deleted. It checks
declarations and refusals; the fixtures under testdata are verbatim
results from the three real runs.
ADR-0008 records the decision. The Invigilator is not designed here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
Three independent reviews, one lens each: runtime correctness, contract
and invariants, tests and claims. Two blockers, both proven rather than
argued, and both in the gap between what this seam declares and what it
does.
The wall cap did not hold. exec.CommandContext kills the child it
started, and the wait then reads the output pipes to end of file, which
never arrives while a grandchild still holds them. Against a 150ms cap a
forking stub left Sit blocked past 20 seconds. Worse, the previous test
comment named this mechanism and wrote around it with `exec sleep 30`, so
EnforcementSupervised was a promise whose one failure mode was the case
the test avoided. cmd.WaitDelay fixes it and the new test fails without
it.
The environment was an open configuration channel. cmd.Env was never set,
so the harness inherited everything — including ANTHROPIC_MODEL,
ANTHROPIC_BASE_URL and CLAUDE_CODE_USE_BEDROCK, all of which it reads. A
developer with ANTHROPIC_MODEL exported would have got sittings run
against a model no report names, on a tool whose purpose is noticing when
the model changed. Now an allowlist, with the two consequences stated:
env-only Bedrock and interactive sign-in both fail loudly instead of
quietly measuring something else.
Also from the reviews:
- Assignment.Dir was governed by nothing. An empty value meant the
harness ran in Assayer's own working directory, which is the user's
repository. Refuse rejects it, and a new default-deny test over
Assignment's fields makes the next ungoverned field fail rather than
ship.
- A negative cap was silently dropped by every `> 0` guard — the one
direction that costs money, and reachable from an orchestrator
computing a wall cap as time-until-deadline.
- Sitting.Usage.Wall meant two different things. A sitting measures
process duration; Usage.Wall means a transcript's timestamp span,
which for an interactive golden session includes hours of idle.
Sitting.Wall is now its own field. Sitting.Turns joins it, so a
report can say "stopped at 2 turns of 1" without resolving a session
that may not resolve.
- The token test could not fail: success.json reads nothing from cache,
so the sum was indistinguishable from one ignoring the field that
carries 96.5% of input. Asserted against turns.json instead.
- Query.Native was bound by nothing — no kit check, and the reference
adapter ignored it, so any adapter could too.
- The stderr bound is documented as keeping the tail; keeping the head
passed. The stubs broke on any checkout path containing a space.
- Refuse panicked on a nil assignment, on an interface taking a
pointer.
- "Never calls Sit on a path that can reach a harness" was not true of
the kit, and "both cap hits exited 1" was asserted in five places
with no record in the repo. The first is reworded, the second is
written down in testdata/README.md beside the fixtures it describes.
ADR-0008 gains the couplings it should have declared: tool names and the
settings blob are harness vocabulary riding through the leaf, and a token
cap is missing from Caps. Seven remaining findings are filed as #44–#50.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
The only untested statement left in the run adapter was the line choosing what to execute when no binary is named — which is the path a composition root takes, so a typo in the harness's name would have shipped. The stub goes on PATH rather than being named directly, because the lookup is the thing under test. Writing it turned up the environment allowlist working: substituting PATH outright left the stub unable to find cat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second adapter seam.
port.Runadministers a sitting; the Claude Code reference adapter implements it against print mode; a shared conformance kit checks the contract half. Nothing consumes it yet — the Invigilator is deliberately not designed here.What the binary said that the documentation did not
Three real invocations, ~$0.020 total, and each one changed the design:
--max-turns 1on a task needing a toolsubtype: error_max_turns, exit 1claude --help. An earlier draft declared turn caps unenforceable on that omission and would have refused a cap it could keep.--max-budget-usd 0.001EnforcementNativesays so explicitly.The turn cap also reported
num_turns: 2against a cap of 1, and a budget-stopped run came back with its token counts zeroed while the cost was real. Both are pinned as tests rather than fixed.The shape
assay.Assignment/Sitting/Guarantees/Caps— domain types, since the Examiner, Ledger and Reporter all consume a sitting.assay.StopReasonwithDecides()true for exactly one value.ERROR ≠ FAILmoved out of a reviewer's memory and into the type.port.Refuse— shared, so "never silently drop a cap" is not an invariant every implementer re-derives. An undeclared cap or field is refused before a process starts, so a refusal costs nothing.port.Query.Native— closes the loop from a sitting back to aSessionwithout either adapter importing the other, leaving ADR-0005 untouched. Matches on the transcript's file name: 400 of 400 sampled transcripts are named for the session they hold, and matching on contents would mean opening every file in a store that held 2,485. Fails closed.--bareplus an empty--setting-sourcesis the hermeticity keystone.--fallback-modelis never passed, asserted as an absence.The conformance kit's honest limit
VerifyRunnever callsSiton a path that can reach a harness. Reading a transcript is free; administering a sitting starts a coding agent against a real model on somebody's key, and a kit that spends a contributor's money to prove an adapter well behaved is a kit that gets deleted from the test run. It checks declarations and refusals only, and says so in its own doc comment — the same wayVerifyalready admits it checks the contract half and not whether an adapter parses anything.Whether the adapter drives the harness correctly is proved in its own tests: three verbatim recorded results under
testdata, plus shell stubs for the wall cap, cancellation, a missing binary, unreadable output, and stderr bounding.A
ReferenceRunjoins the fictional capture adapter — a command-template runner that guarantees almost nothing, so the kit's refusal checks have something to check. The measured adapter guarantees everything, which would have made them vacuous.Verification
make verifygreen: build, vet, gofmt, race, coverage 89.7% (threshold 80), golangci-lint 0 issues, govulncheck 0.Known open edge
The guarantee check runs one way, like the capability check it mirrors: over-reach is caught, a guarantee declared and never kept is not. Same open edge as the capture side, tracked as #39.