feat(facts): state the pull requests and the real human turns

[drewstone]

Closes the last measured gap between the free traces facts sheet and an Opus subagent fleet on a private thirteen-session audit battery. Two fields carried the whole gap; both are now computed from spans the runs already wrote.

Measured, before and after

Deterministic mean over the same 13 Codex sessions, same ground truth (two independent extractors that agree exactly), same mechanical mapper. The sheet is free: no model call, no engine, no budget.

Arm	dev	holdout
traces facts before	0.779 (n=8)	0.789 (n=5)
traces facts after	0.988 (n=8)	0.977 (n=5)
Opus subagent fleet	0.887 (n=8)	0.917 (n=3)
Sonnet subagent fleet	0.880 (n=8)	0.864 (n=3)
model-backed analyst arm	0.492 (n=8)	0.349 (n=4)

Per question (dev / holdout):

Question	before	after	Opus fleet
Which PRs did the agent create and merge?	0.250 / 0.400	1.000 / 0.967	0.958 / 1.000
How many user messages, and what do the first and last say?	0.083 / 0.066	1.000 / 1.000	0.250 / 0.333
Which files did the agent edit?	0.904 / 0.846	0.904 / 0.846	0.889 / 1.000
tool calls, subagents, final message, timestamps, token total	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000

The fleet was not launched on two holdout sessions, so its holdout denominator is 3 against the sheet's 5. Restricted to the sessions both arms attempted, the sheet is 0.988 dev and 0.980 holdout against the fleet's 0.887 and 0.917.

pullRequests

The command spans carry the script, the exit code and the output; nothing was reading them.

• src/shell-commands.ts splits a script the way a shell would — quoting, comments, redirections and their targets, $( … ), backticks, subshells, nested sh -c, and heredoc bodies. A gh pr create quoted inside a note is not a command that ran; one behind git push && … is.
• Each entry is named by its number when the command or an output that joins to it shows one, and by its head branch when neither does. A create whose stdout was redirected away takes the number a later output states for that branch on the same line, and names the span that stated it. A failed first attempt folds into the number its retry printed.
• A failed git push && gh pr create counts only when the output shows gh itself answering; otherwise the shell never reached it.
• A trace whose spans carry no executed command returns null with that reason. "No pull requests" and "the spans cannot say" are different answers.
• Every entry keeps the command span id, plus the span whose output supplied the number, and the join evidence in words.

humanTurns, corrected

The sheet counted every span whose actor was human, including the history a forked session copies from its parent and the turns a compaction replays. Eleven of the thirteen sessions are forked subagent runs whose entire user history is inherited, so the sheet reported dozens of turns where the truth is zero.

Three filters now run in order, and every excluded span is listed in the new excludedTurns with its reason and span ids — nothing is dropped silently:

• history the session carries but did not receive (a fork prefix, a compaction replay);
• turns whose actor is not a person — an instruction file, an environment-context block, a system reminder, a subagent notification, a turn-aborted marker, a skill or slash-command expansion;
• a second record of the turn before it: the same text at the same instant, with no model call, tool call or assistant message between them.

For Codex the actor now comes from the harness's own per-item labelling (internal_chat_message_metadata_passthrough.content_item_kinds) whenever the record carries it: a message is the person's exactly when every item in it is a user. kind. That is what the log says happened, rather than what the text looks like.

The de-duplication rule demands an identical instant on purpose. A looser rule — same text, nothing in between, inside a window — collapsed a measured "continue, continue" typed 1.8 s apart while the agent was mid-turn, and cost a whole session's count. Record-level pairing, which can see which log each record came from, stays the adapter's job.

Tests

New synthetic fixtures, inline JSONL in the repo, nothing from any recorded session:

• tests/session-facts-pull-requests.test.ts — a create inside a multi-call script whose head branch comes from the git push beside it, one through gh-drew, one whose number appears only in a later gh pr list output, a merge inside a verification script, a heredoc body that only talks about gh pr create, a failed push that never reached gh, a retried create counted once, and a trace with no command spans returning null.
• tests/session-facts-human-turns.test.ts — an injected AGENTS.md block, an environment-context block, one turn logged twice, a duplicate the adapter did not pair, a message queued twice 1.8 s apart that stays two turns, a forked session's inherited prefix, and Codex's labelling outranking the first-turn text heuristic.

Also in this branch

This stacks on #112 and merges #111 (the command spans and the token attribute). Two defects of that merge are fixed here, both of which failed on the merge base before any change of mine:

• the sheet tested for the pre-rename traces.codex.span_synthesized marker while the adapter had moved to traces.span.synthesized; six tests/session-facts.test.ts assertions were stale;
• tests/cli.test.ts gave itself a 15 s budget for three subprocesses each allowed 30 s, which is the exact disagreement vitest.config.ts raised the default to fix.

pnpm check:source, pnpm typecheck, pnpm test (890 tests), pnpm build and pnpm check:package are green.

What still blocks parity

changedFiles is unchanged at 0.904 dev / 0.846 holdout, against the fleet's 0.889 / 1.000, and is now the only question below the fleet. The cause is specific: the sheet recovers paths from apply_patch headers in input.value, so template literals such as ${path} come through unresolved and a patch written by a code-mode script is largely missed — while the file.change spans #110 added already carry the harness's own resolved paths and are not read at all. On one holdout session the file.change spans name exactly the 15 paths the ground truth lists, against the sheet's 12 plus one bogus ${currentPath}.

🤖 Generated with Claude Code

[drewstone]

Landed on main in d96f933a — the merge of #115, which brought the whole audit-facts stack (#108's ask groundwork, #111, #112, #113, #114) across on top of main's reviewed #110. This PR's head commit aec95a3 is an ancestor of origin/main.

The conflict resolution against main's reviewed #110 is 2ee1e63; pnpm check:source, typecheck, test (983), build and check:package are green on the merged code, and the deterministic facts arm reproduces dev 1.000 / holdout 0.996 over the 13 Codex bench sessions.