Report how each arm reached the docs, and how often it got nothing

[leggetter]

Records the documentation-access gap from #61 rather than equalising it, per the decision that a skill pointing at the single source of truth is the skill working.

report-results gains a per-experiment breakdown: calls that reached a page, calls that came back empty, and the tool mix behind both. No new data is collected — buildDocsResult already captured source, pages and resultChars per call. This just reads it.

Printing it per experiment changed the conclusion

Pooled by arm, it reads as a skills effect: 296 calls in -no-skills against 83 in +skills, 58% empty against 37%. That is what #61 asserts. Split by experiment on the 25 August snapshot:

claude-code-sonnet-5              21 calls    0 empty (0%)   [web_fetch 12, shell_fetch 9]
claude-code-sonnet-5-no-skills    64 calls    0 empty (0%)   [web_fetch 43, web_search 19, shell_fetch 2]
codex-gpt-5.4-mini                43 calls   26 empty (60%)  [web_search 31, shell_fetch 12]
codex-gpt-5.4-mini-no-skills     196 calls  144 empty (73%)  [web_search 176, shell_fetch 20]
codex-gpt-5.6                     19 calls    5 empty (26%)  [shell_fetch 12, web_search 7]
codex-gpt-5.6-no-skills           36 calls   30 empty (83%)  [web_search 33, shell_fetch 3]

Both Claude arms are at zero. Claude Code reaches documentation with web_fetch, which returns a page; Codex leans on web_search, which frequently returns none.

A real skills effect does survive inside each Codex pair — 60% against 73%, and 26% against 83% — but it is the smaller term. The dominant variable is which agent CLI, and pooling two agents with different habits hid that.

That matters for the milestone-2 run: a skills delta computed across both agents will carry an agent-shaped artefact unless this is read per row.

Why record and not equalise

Equalising — telling both arms in the prompt that docs are fetchable over HTTP — would remove the mechanism along with the noise. If the skills are meant to point at the documentation rather than duplicate it, an agent fetching a docs URL because a skill named it is the thing we are trying to measure.

Recording keeps that visible and separates it from the part that genuinely is noise: an external search index returning nothing.

Verification

pnpm typecheck clean, pnpm -r test green (254 tests), output verified against the 25 August snapshot.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA