feat(facts): land the session-facts audit stack (#108, #111, #112, #113, #114) - #115
Merged
Conversation
`traces ask` and the exported `runTraceQuestions` run one or many questions over one or many sessions. Each question is its own `runTraceAnalyst` call, so the engine's prose answer survives; the analyst registry keeps only findings and runs analysts one at a time. - Questions run concurrently through a worker pool with a configurable limit. Unlike the import pool, one failed question never stops the others. - One shared `CostLedger` bounds the whole run, so `--budget` means the same thing whatever the number of questions; `--question-budget` bounds one question. A budget below one call's reservation refuses the run before any model call, and a budget that serializes the pool warns instead. - Every `trace://` citation in an answer is resolved against the store; an unresolvable citation fails that question. - A question may fix its answer's shape with a small JSON Schema subset. An unsupported keyword is rejected rather than ignored. - The question layout fits the DSPy input preview: the question stays whole and the answer rules sit in the first 500 characters of the instructions. - Output is `answers.json` and `report.md`, written before the exit code is decided; exit 1 when any question failed. Also surface what the evidence gate refused, and stop reporting a failed external analyzer as success: - `finding rejected` log lines now name the reason and the offending URI, and rejection counts by reason reach the analyst table's Detail cell, `TraceInvestigationResult.findingRejections`, and the ask JSON. - `analyze` exits 1 when a requested `--analyzer` fails, after writing the report that holds its error. The `--llm` engine construction moves to `analysisEngineFromEnv` in `analyst-model-call.ts`, and the shared trace-file and store setup to `analysis-store.ts`, so `ask` reuses both instead of duplicating them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit question asks which commands ran, what they exited with, when they ran, which files changed, and what the person actually typed. The Codex adapter dropped every `item_completed` event, so none of those facts reached a span, and it labelled harness-injected context as a human turn. - Emit one CHAIN span per `CommandExecution` item (command, cwd, exit code, process id, output, and the item's own start and end times) and one per changed path in a `FileChange` item. Each item joins the one tool call whose window contains its whole run; an item that outlives every call or falls inside two stays under the session root and says so. Item shapes the adapter cannot represent are counted on the root, never guessed. - Mark the inner spans `traces.tool_call.level=inner` so tool-call counts, loop detection and conversation text keep reading the model-issued level. - Accept a script receipt whose "Wall time" line has no colon, which left script outcomes UNSET. - Treat Codex's context blocks (`<environment_context>`, `<user_instructions>`, `<skills_instructions>`, the `<external_*>` wrappers, the injected warnings and the rest of `CONTEXTUAL_USER_FRAGMENT_MATCHERS`) as injected, and take the human turn from the record Codex writes for submitted input: the legacy `user_message` event or the current `item_completed`/`UserMessage` item, paired with its response-item copy so one turn stays one span. - Drop the parent session from `childSessionIds`: a child that messages its parent named it as a `send_message` target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… as tool calls Three adapter facts an audit question asks for, measured against twelve private Codex sessions where a mechanical extraction from the spans this adapter already writes scored 0.858 and the analyst arm scored 0.389. 1. The cumulative token total. Codex reports `token_count.info.total_token_usage` beside the per-turn delta, and the adapter used it only as a de-duplication signature, so no span held the session total. The last snapshot equals ground truth on 13 of 13 sessions; summing the deltas or the snapshots does not. It is copied onto the root span verbatim, with the rest of the same snapshot beside it. 2. The synthesized subagent span. `ensureSubagentSpan` built a child thread's lifecycle as a TOOL span named `tool.Agent`, so every tool-call count ran high by one per child thread (TOOL minus `tool.Agent` reproduced truth on 12 of 12 sessions). It is now an AGENT span named `subagent.lifecycle`, with no `tool.name` and `traces.span.synthesized = true`; the evidence, live, pipeline, run-tree and adoption paths count model-issued calls only. 3. The inherited prefix. The task-scope walk dropped every record before the fork boundary, and `compacted` records were never parsed, so a forked child's human context reached no span. Both are kept as spans marked `traces.session.inherited = true`, deduplicated on the turn text and bounded per session, with what the cap drops counted on the root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bounded trace tools cannot return a whole-session fact. `viewTrace` degrades to a 20-entry name histogram above its 150,000-byte ceiling, `countTraces` counts traces rather than spans, `viewSpans` needs span ids the reader does not have, and `searchTrace` stops at 500 hits. A model asked for a tool-call total therefore adds up a capped histogram and decides by eye which names count. Measured over twelve private audit sessions, the model-backed analyst arm scored a deterministic mean of 0.389. Extracting the same answers mechanically from the OTLP spans those runs already wrote scores 0.858: the facts were present and exact the whole time. This adds that extraction as `src/session-facts.ts`, a `traces facts` command that prints it at $0, and the same sheet as prepared context for the built-in analyst kinds and every `traces ask` question. Two rules hold for every field: it names the span ids it was computed from, so a reader can check it; and a fact the spans cannot support is null with a stated reason, never a guess and never a silent zero. The sheet is not a span and cannot be cited — citations still resolve against the raw spans. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sized-spans' into feat/facts-prs-and-human-turns
Two fields carried the whole remaining gap between the free `traces facts` sheet and a subagent fleet on a private thirteen-session audit battery: the sheet computed no pull-request facts at all (0.31), and it counted every human-role span as a turn (0.076), including the history a forked session copies from its parent and the turns a compaction replays. `pullRequests` reads the command spans. `src/shell-commands.ts` splits a script the way a shell would — quoting, comments, redirections, command substitutions, nested `sh -c`, and heredoc bodies — so a `gh pr create` quoted inside a note is not a command that ran, and one hidden behind `git push && …` is. Each entry is named by its number when the command or an output that joins to it shows one, and by its head branch when neither does; a create whose stdout was redirected away takes the number a later output states for that branch and names the span that stated it. A trace whose spans carry no executed command returns null with that reason, because "no pull requests" and "the spans cannot say" are different answers. `humanTurns` now counts turns a person typed into THIS session. Inherited fork prefixes and compaction replays, harness-injected blocks, and a second record of the same turn are excluded, and every excluded span is listed in the new `excludedTurns` with its reason — nothing is dropped silently. For Codex the actor comes from the harness's own per-item labelling (`content_item_kinds`) whenever the record carries it, rather than from what the text looks like. The de-duplication rule demands the same text at the same instant with no agent activity between: a measured "continue, continue" typed 1.8 s apart is two turns, and collapsing it cost a whole session's count. Also fixes the merge of the two open branches this stacks on: the sheet was testing for the pre-rename `traces.codex.span_synthesized` marker while the adapter had moved to `traces.span.synthesized`, and `tests/cli.test.ts` gave itself a 15 s budget for three subprocesses each allowed 30 s. Measured over the same thirteen Codex sessions, deterministic mean per split: dev 0.779 -> 0.988, holdout 0.789 -> 0.977, against an Opus subagent fleet at 0.887 and 0.917. Per question: pull requests 0.25/0.40 -> 1.000/0.967, human turns 0.083/0.066 -> 1.000/1.000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ords
`changedFiles` recovered every path from the `apply_patch` headers kept in
`input.value`. A header is the text the caller wrote, so a patch a code-mode
script generated arrived with the script's own `${path}` or `$FILE` in the
header, and the sheet emitted that as a path the session never touched. The
`file.change` spans the Codex adapter already writes carry the path the edit
actually reached, and nothing read them.
Read those spans first and let the header recovery defer to them. A header
that names a file the harness already recorded joins that entry rather than
opening a second one, including when the header wrote the path relative to the
directory the harness resolved it against, so an edit both sources saw is one
changed file with both spans as its evidence. A header still holding an
unexpanded variable names no file and is dropped and counted, not emitted. A
session that records no file change keeps recovering paths from headers
exactly as before.
The count of dropped paths joins the truncated-input note in `partial` instead
of replacing it, and the list cap's own note no longer disappears when either
is present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared list cap is 200 entries, sized against lists whose entries carry up to `FACT_TEXT_CAP` characters of message or prompt text. A changed-file entry is a path and two short arrays, and a session that edited hundreds of files is the one whose file list a reader most needs whole; the rendered context prints only the count either way, so the cap bought nothing there and cost the sheet the second half of its own answer. Give that one list its own ceiling, still finite so a runaway session cannot make the sheet unbounded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main carries the reviewed squash of #110: inner spans built at read time (`PendingInnerSpan`, `placeInnerSpans`), `traces.codex.unmodeled_item_counts` split from `dropped_item_counts`, an additive `process.exit_code`, an item dedupe key scoped by task index, and a bounded-heap regression test. This branch carried #110's pre-review commit plus #108, #111, #112, #113 and #114 built on top of it. Resolution rule: code owned by #110 takes main's reviewed version; everything built after it takes the branch; where both changed the same function, both intents are kept. - src/adapters/codex.ts: main's read-time span construction, split item censuses and task-scoped dedupe key, alongside the branch's provenance imports, inherited-turn recording, `codexActor` kinds argument, session token totals and inherited-span counts. - src/adapters/codex-format.ts: main's `skipped` reason split; the branch's `compacted` payload shape (`replacement_history`, window ids). - src/analyze.ts: the branch's analysis-store helpers and session-facts context wrap main's `normalizeAnalystCitations`, so citations are normalized against the stored spans and the sheet still reaches `prepareContext`. - src/adapters/actor.ts, src/live.ts: the branch's version; main changed neither after #110. - tests/codex-command-facts.test.ts: main's reviewed assertions; the branch never changed that file after #110. Verified: install, check:source, typecheck, 983 tests, build and check:package green. The bounded-heap test passes here and still fails when src/adapters/codex.ts is reverted to 7bfc37d. The deterministic facts arm over the 13 Codex bench sessions reproduces dev 1.000 and holdout 0.996, with answers byte-identical to the pre-merge measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 10, 2026
drewstone
added a commit
that referenced
this pull request
Sep 10, 2026
`main` landed this PR's first commit (`55044df`) through the audit-facts stack (#115) and, in #112, extended `preparedContext` with the deterministic session-facts sheet. This PR's two review commits — `185f975` (an interrupted run keeps every bought answer) and `541e2eb` (a citation the model wrapped in Markdown emphasis resolves) — were not in that stack and are still only here. One conflict, in `ASK_RULES`. Both intents kept: rules 2-4 take main's wording, which tells the model the prepared context now ends with a SESSION FACTS sheet and that the sheet itself is not citable; the comment above rule 3 stays, because it records why that rule does not also legislate citation formatting — `traceCitationsInText` reads an emphasised citation. Verified: install, check:source, typecheck, 994 tests, build and check:package green. The deterministic facts arm still reproduces dev 1.000 and holdout 0.996 over the 13 Codex bench sessions, byte-identical to the measurement taken from #115's merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the rest of the audit-facts stack on top of main's reviewed
#110, as one merge so each PR's commit survives.What lands
feat(ask): answer free-form questions concurrently under one budget. Adds theaskcommand and moves the trace file, the file ceiling, and the source-bundle containment check into oneanalysis-storemodule every analysis path shares.fix(codex): carry the token total and stop counting synthesized spans as tool calls. Records the session token total from Codex's own usage snapshot and marks synthesized and inherited spans so tool counts and error ratios stop running high.feat(facts): compute a deterministic session-facts sheet from the spans. Addstraces facts: a zero-model, zero-cost sheet of exact session facts, also supplied to the agentic analysts as prepared context.feat(facts): state the pull requests and the real human turns. Reads pull requests from executed shell commands and separates the turns a person typed from the context the harness injected, using Codex's own per-item kinds.fix(facts): name changed files from the harness's own file-change records, plus its own list ceiling. Names changed files fromFileChangerecords instead of inferring them from command text, and bounds the list.#111was auto-closed when its base branch was deleted; its commit is in this merge.Merge resolution
maincarries the reviewed squash of#110; this branch carried#110's pre-review commit plus the five PRs above. Code owned by#110takes main's reviewed version — inner spans built at read time (PendingInnerSpan,placeInnerSpans),traces.codex.unmodeled_item_countssplit fromdropped_item_counts, the additiveprocess.exit_code, the task-scoped item dedupe key, and the bounded-heap regression test. Everything built after#110takes this branch, and where both changed the same function both intents are kept: main's read-time span construction alongside this branch's synthesized-span markers and token totals, and main'snormalizeAnalystCitationsinside this branch's session-facts context wrapper.Measured result
The deterministic facts arm answers eight audit questions per session over 15 recorded sessions, scored against adjudicated ground truth.
Measured from this merge over the 13 Codex sessions in the private manifest, deterministic on re-run and byte-identical to the pre-merge answers. The fleet arm is a model fleet with unmeasured token cost; this arm runs no model and costs $0.
The two Claude Code sessions score 0.384, and that is the expected result, not a regression: every fact the sheet reads — tool calls, subagent tasks, executed commands, file-change records, human turns, token totals — is emitted by the Codex adapter alone. Extending the sheet to Claude Code means teaching that adapter to emit the same facts.
Checks
pnpm install --frozen-lockfile,check:source,typecheck,test(983 tests, 68 files),buildandcheck:packageall green. Main's bounded-heap test passes here and still fails whensrc/adapters/codex.tsis reverted to the pre-review7bfc37d(the child process dies of heap exhaustion), so the review's repair survives the merge.🤖 Generated with Claude Code