The -no-skills arm reaches documentation differently, so part of the skills delta is not skills

[leggetter]

Across every row of the current snapshot, the two arms do not read documentation the same way:

arm	docs calls	returned nothing	web_search	web_fetch	shell_fetch
+skills	83	31 (37%)	38	12	33
-no-skills	296	174 (58%)	228	43	25

The baseline makes 3.6× as many documentation calls and 77% of them are web_search, against 46% in the skills arm. More than half come back with pages: [].

Some of that gap is circular — an agent that is failing researches more. The tool mix is not: an agent that curls a docs URL gets the page, and an agent that queries a search index gets whatever the index feels like returning that day.

Where it was caught

verification-002 on the weak model, chased in #2. The scorer short-circuits when no ELEVENLABS source exists, and the failing baseline never created one:

execution	arm	calls	empty	tool
14 Aug	baseline	7	5	all web_search — passed
17 Aug	baseline	11	7	all web_search — failed
17 Aug	skills	3	0	all curl hookdeck.com/docs/sources.md — passed

Same prompt, same agent, same scorer, opposite results, and the difference between them is how many searches happened to return something. That was published for seven consecutive snapshots as a skills win (see #60 for why it looked like eight runs).

Why this matters more than one scenario

-no-skills is meant to be the control. AGENTS.md is deliberate that it keeps the CLI and a live API key, so a row differs from its neighbour "by skills and nothing else". It does not. It also differs by whether the agent fetches documentation or searches for it, and the search path fails most of the time.

Whatever the skills delta measures, part of it is currently "did the agent think to curl the docs" — a property of agent habit and an external search index, not of anything we ship. That contaminates the milestone-2 question directly.

Done when

We can say how much of the measured delta survives once documentation access is equalised, or rule the effect out. Options not yet chosen:

• Record it and report it — a per-row docs-call summary in report-results, so a delta that rests on failed searches is visible.
• Equalise it — state in the base prompt that docs are fetchable over HTTP, in both arms.
• Bound it — re-run the affected cells with the search path unavailable, and see which failures survive.

Related: #10 (this is the mechanism it asks to confirm, from data rather than inference), #60 (how one execution came to look like eight), #2 (where it surfaced).

[leggetter]

Prior art from the #57 survey — vercel-labs/agent-eval models exactly this as a controlled treatment, and its framing is the distinction this issue needs:

webResearch remains opt-in. It allows Claude Code's WebSearch/WebFetch, enables Codex live search (including custom AI Gateway providers), and enables OpenCode's Exa-backed websearch/webfetch tools. Whether an agent chooses to use an available research tool remains part of the measured behavior.

Availability is controlled; choice is measured. That is the line this issue was reaching for. Our arms currently differ in availability of habit rather than of tools — both can curl, and only the skills arm reliably does — which is neither controlled nor deliberately measured.

Two more details worth copying:

• They refuse to run rather than let a treatment drift silently. The fx adapter requires webResearch: true because fx exposes no prompt-free way to disable web tools, so Agent Eval rejects non-research fx runs "instead of silently changing the treatment".
• disableBundledSkills turns off skills shipped by Claude Code or Codex while leaving caller-installed project and user skills available. That is our BASELINE COMPROMISED problem — an agent self-installing a product skill into a -no-skills run — solved at the harness level instead of flagged after the fact in triage.

Their fingerprint scheme also treats webResearch and disableBundledSkills as a reuseCompatibilityFingerprint: a runtime boundary that is never carried forward between runs. See #60.

This does not change the recommendation here — record and report first, equalise only once we know the size of the effect — but it does mean the eventual fix has a well-tested shape to copy.

[leggetter]

Correction, and the recording is in — #64

Recording rather than equalising, per the decision that a skill pointing at the single source of truth is the skill working. report-results now prints per experiment how many documentation calls reached a page, how many came back empty, and the tool mix.

Printing it per experiment contradicts this issue's central claim. The figures at the top of this issue pool the arms across both agents. Split out, on the 25 August snapshot:

claude-code-sonnet-5              21 calls    0 empty (0%)   [web_fetch 12, shell_fetch 9]
claude-code-sonnet-5-no-skills    64 calls    0 empty (0%)   [web_fetch 43, web_search 19, shell_fetch 2]
codex-gpt-5.4-mini                43 calls   26 empty (60%)  [web_search 31, shell_fetch 12]
codex-gpt-5.4-mini-no-skills     196 calls  144 empty (73%)  [web_search 176, shell_fetch 20]
codex-gpt-5.6                     19 calls    5 empty (26%)  [shell_fetch 12, web_search 7]
codex-gpt-5.6-no-skills           36 calls   30 empty (83%)  [web_search 33, shell_fetch 3]

Both Claude arms are at zero. Claude Code reaches docs with web_fetch, which returns a page; Codex leans on web_search, which frequently returns none.

So the title of this issue is half wrong. The arms do read documentation differently, but the dominant variable is which agent CLI, not which arm. A real skills effect survives inside each Codex pair — 60% against 73%, 26% against 83% — and it is the smaller term.

I wrote the original claim from a pooled aggregate, which is the same error as reading a benchmark from a scoreboard: the summary had a mechanism in it that the breakdown does not support.

What this changes for milestone 2. A skills delta computed across both agents carries an agent-shaped artefact. It has to be read per row, which the new output now makes possible before anyone looks at a pass rate.

Leaving open: the recording is done, but the size of the surviving Codex-internal skills effect is not established, and that is what the done when asks for. The milestone-2 run should answer it.