fix(facts): name changed files from the harness's own file-change records - #114
Merged
Conversation
`traces ask` and the exported `runTraceQuestions` run one or many questions over one or many sessions. Each question is its own `runTraceAnalyst` call, so the engine's prose answer survives; the analyst registry keeps only findings and runs analysts one at a time. - Questions run concurrently through a worker pool with a configurable limit. Unlike the import pool, one failed question never stops the others. - One shared `CostLedger` bounds the whole run, so `--budget` means the same thing whatever the number of questions; `--question-budget` bounds one question. A budget below one call's reservation refuses the run before any model call, and a budget that serializes the pool warns instead. - Every `trace://` citation in an answer is resolved against the store; an unresolvable citation fails that question. - A question may fix its answer's shape with a small JSON Schema subset. An unsupported keyword is rejected rather than ignored. - The question layout fits the DSPy input preview: the question stays whole and the answer rules sit in the first 500 characters of the instructions. - Output is `answers.json` and `report.md`, written before the exit code is decided; exit 1 when any question failed. Also surface what the evidence gate refused, and stop reporting a failed external analyzer as success: - `finding rejected` log lines now name the reason and the offending URI, and rejection counts by reason reach the analyst table's Detail cell, `TraceInvestigationResult.findingRejections`, and the ask JSON. - `analyze` exits 1 when a requested `--analyzer` fails, after writing the report that holds its error. The `--llm` engine construction moves to `analysisEngineFromEnv` in `analyst-model-call.ts`, and the shared trace-file and store setup to `analysis-store.ts`, so `ask` reuses both instead of duplicating them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit question asks which commands ran, what they exited with, when they ran, which files changed, and what the person actually typed. The Codex adapter dropped every `item_completed` event, so none of those facts reached a span, and it labelled harness-injected context as a human turn. - Emit one CHAIN span per `CommandExecution` item (command, cwd, exit code, process id, output, and the item's own start and end times) and one per changed path in a `FileChange` item. Each item joins the one tool call whose window contains its whole run; an item that outlives every call or falls inside two stays under the session root and says so. Item shapes the adapter cannot represent are counted on the root, never guessed. - Mark the inner spans `traces.tool_call.level=inner` so tool-call counts, loop detection and conversation text keep reading the model-issued level. - Accept a script receipt whose "Wall time" line has no colon, which left script outcomes UNSET. - Treat Codex's context blocks (`<environment_context>`, `<user_instructions>`, `<skills_instructions>`, the `<external_*>` wrappers, the injected warnings and the rest of `CONTEXTUAL_USER_FRAGMENT_MATCHERS`) as injected, and take the human turn from the record Codex writes for submitted input: the legacy `user_message` event or the current `item_completed`/`UserMessage` item, paired with its response-item copy so one turn stays one span. - Drop the parent session from `childSessionIds`: a child that messages its parent named it as a `send_message` target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… as tool calls Three adapter facts an audit question asks for, measured against twelve private Codex sessions where a mechanical extraction from the spans this adapter already writes scored 0.858 and the analyst arm scored 0.389. 1. The cumulative token total. Codex reports `token_count.info.total_token_usage` beside the per-turn delta, and the adapter used it only as a de-duplication signature, so no span held the session total. The last snapshot equals ground truth on 13 of 13 sessions; summing the deltas or the snapshots does not. It is copied onto the root span verbatim, with the rest of the same snapshot beside it. 2. The synthesized subagent span. `ensureSubagentSpan` built a child thread's lifecycle as a TOOL span named `tool.Agent`, so every tool-call count ran high by one per child thread (TOOL minus `tool.Agent` reproduced truth on 12 of 12 sessions). It is now an AGENT span named `subagent.lifecycle`, with no `tool.name` and `traces.span.synthesized = true`; the evidence, live, pipeline, run-tree and adoption paths count model-issued calls only. 3. The inherited prefix. The task-scope walk dropped every record before the fork boundary, and `compacted` records were never parsed, so a forked child's human context reached no span. Both are kept as spans marked `traces.session.inherited = true`, deduplicated on the turn text and bounded per session, with what the cap drops counted on the root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bounded trace tools cannot return a whole-session fact. `viewTrace` degrades to a 20-entry name histogram above its 150,000-byte ceiling, `countTraces` counts traces rather than spans, `viewSpans` needs span ids the reader does not have, and `searchTrace` stops at 500 hits. A model asked for a tool-call total therefore adds up a capped histogram and decides by eye which names count. Measured over twelve private audit sessions, the model-backed analyst arm scored a deterministic mean of 0.389. Extracting the same answers mechanically from the OTLP spans those runs already wrote scores 0.858: the facts were present and exact the whole time. This adds that extraction as `src/session-facts.ts`, a `traces facts` command that prints it at $0, and the same sheet as prepared context for the built-in analyst kinds and every `traces ask` question. Two rules hold for every field: it names the span ids it was computed from, so a reader can check it; and a fact the spans cannot support is null with a stated reason, never a guess and never a silent zero. The sheet is not a span and cannot be cited — citations still resolve against the raw spans. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sized-spans' into feat/facts-prs-and-human-turns
Two fields carried the whole remaining gap between the free `traces facts` sheet and a subagent fleet on a private thirteen-session audit battery: the sheet computed no pull-request facts at all (0.31), and it counted every human-role span as a turn (0.076), including the history a forked session copies from its parent and the turns a compaction replays. `pullRequests` reads the command spans. `src/shell-commands.ts` splits a script the way a shell would — quoting, comments, redirections, command substitutions, nested `sh -c`, and heredoc bodies — so a `gh pr create` quoted inside a note is not a command that ran, and one hidden behind `git push && …` is. Each entry is named by its number when the command or an output that joins to it shows one, and by its head branch when neither does; a create whose stdout was redirected away takes the number a later output states for that branch and names the span that stated it. A trace whose spans carry no executed command returns null with that reason, because "no pull requests" and "the spans cannot say" are different answers. `humanTurns` now counts turns a person typed into THIS session. Inherited fork prefixes and compaction replays, harness-injected blocks, and a second record of the same turn are excluded, and every excluded span is listed in the new `excludedTurns` with its reason — nothing is dropped silently. For Codex the actor comes from the harness's own per-item labelling (`content_item_kinds`) whenever the record carries it, rather than from what the text looks like. The de-duplication rule demands the same text at the same instant with no agent activity between: a measured "continue, continue" typed 1.8 s apart is two turns, and collapsing it cost a whole session's count. Also fixes the merge of the two open branches this stacks on: the sheet was testing for the pre-rename `traces.codex.span_synthesized` marker while the adapter had moved to `traces.span.synthesized`, and `tests/cli.test.ts` gave itself a 15 s budget for three subprocesses each allowed 30 s. Measured over the same thirteen Codex sessions, deterministic mean per split: dev 0.779 -> 0.988, holdout 0.789 -> 0.977, against an Opus subagent fleet at 0.887 and 0.917. Per question: pull requests 0.25/0.40 -> 1.000/0.967, human turns 0.083/0.066 -> 1.000/1.000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ords
`changedFiles` recovered every path from the `apply_patch` headers kept in
`input.value`. A header is the text the caller wrote, so a patch a code-mode
script generated arrived with the script's own `${path}` or `$FILE` in the
header, and the sheet emitted that as a path the session never touched. The
`file.change` spans the Codex adapter already writes carry the path the edit
actually reached, and nothing read them.
Read those spans first and let the header recovery defer to them. A header
that names a file the harness already recorded joins that entry rather than
opening a second one, including when the header wrote the path relative to the
directory the harness resolved it against, so an edit both sources saw is one
changed file with both spans as its evidence. A header still holding an
unexpanded variable names no file and is dropped and counted, not emitted. A
session that records no file change keeps recovering paths from headers
exactly as before.
The count of dropped paths joins the truncated-input note in `partial` instead
of replacing it, and the list cap's own note no longer disappears when either
is present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared list cap is 200 entries, sized against lists whose entries carry up to `FACT_TEXT_CAP` characters of message or prompt text. A changed-file entry is a path and two short arrays, and a session that edited hundreds of files is the one whose file list a reader most needs whole; the rendered context prints only the count either way, so the cap bought nothing there and cost the sheet the second half of its own answer. Give that one list its own ceiling, still finite so a runaway session cannot make the sheet unbounded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
drewstone
changed the base branch from
feat/facts-prs-and-human-turns
to
main
September 10, 2026 18:15
This was referenced Sep 10, 2026
Contributor
Author
|
Landed on The conflict resolution against main's reviewed #110 is 2ee1e63; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
changedFilesrecovered every path from theapply_patchheaders kept ininput.value. A header is the text the caller wrote, not a path the harness resolved, so a patch a code-mode script generated arrived with the script's own${path}or$FILEstill in the header — and the sheet emitted that as a path the session never touched, while missing the file the patch actually reached. Thefile.changespans #110 added already carry the harness's own resolved path for every change it applied, and nothing read them.Two fixes:
Read the harness's record first.
file.changespans (excluding ones the harness marked failed or declined) name the changed paths, withmove_pathcounted as its own path. Header recovery then defers to them: a header naming a file the harness already recorded joins that entry rather than opening a second one — including when the header wrote the path relative to the directory the harness resolved it against — so an edit both sources saw is one changed file carrying both spans as evidence. A header still holding an unexpanded variable names no file and is dropped and counted, not emitted. A session that records no file change recovers header paths exactly as before.Give the changed-file list its own ceiling. The shared 200-entry cap is sized against lists whose entries carry up to
FACT_TEXT_CAPcharacters of message text. A changed-file entry is a path and two short arrays, and a session that edited hundreds of files is the one whose list a reader most needs whole; the rendered context prints only the count either way. On one benchmark session the sheet found 365 changed paths and listed 200 of them.Every value keeps its span ids, and the null-with-reason discipline is unchanged. The dropped-path count joins the truncated-input note in
partialinstead of replacing it, and the list cap's own note no longer disappears when either is present — a pre-existing loss of disclosure.Tests
tests/session-facts-changed-files.test.ts, six cases on synthetic inline JSONL parsed through the real Codex adapter: anapply_patchinside a code-mode script, a path written through a shell variable, an edit both sources saw counted once with both spans, a session with no file-change spans still recovering header paths, a rename destination with a declined change left out, and a session over the shared list cap. Four of the six fail on the base commit, with['${target}']where['/repo/src/target.ts']belongs.pnpm check:source,pnpm typecheck,pnpm test(896 passing),pnpm buildandpnpm check:packageare green.Measured
Private 13-session Codex benchmark, one build of this branch before and after the change, scored against the adjudicated ground truth. Q4 is "which files did the agent edit", graded by Jaccard over path sets.
The before arm reproduces the previously recorded arm answer for answer on all 13 sessions, so the two rows differ only by this change.
Denominators differ and the comparison should not be read past them: the Opus fleet was launched on 8 dev and 3 holdout sessions, the sheet on 8 dev and 5 holdout. On the three holdout sessions the fleet ran, the sheet after this change scores 1.0 on each — the same 1.0 the fleet reports. The sheet's 0.9970 is pulled down only by a session the fleet never attempted, where it now names 331 distinct paths against ground truth's 326, missing none and adding five: two are string-concatenation artifacts left by a patch a script built with
'…' + variable + '…', where the harness recorded no change to defer to; two are directories; one is a path the truth file does not list.Changed files no longer trails either fleet. The remaining deterministic gap on holdout is Q3 (pull requests) on one session, which is #113's scope and unaffected by this change.
🤖 Generated with Claude Code