diff --git a/README.md b/README.md index ff5d32c..307b6d5 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ Emitting the contract is the supported way to integrate a new system. The adapte - [Live stream](#live-stream) - [Watch a run tree](#watch-a-run-tree) - [Improvement engine](#improvement-engine) +- [Ask questions](#ask-questions) +- [Session facts](#session-facts) - [Session index](#session-index) - [Session bundle](#session-bundle) · [Two views](#two-views-two-consumers) - [Policy-mining evidence](#policy-mining-evidence) @@ -243,6 +245,8 @@ traces analyze --harness codex --current --latest-turn --workflow # current tur traces analyze --harness claude-code --session --latest-turn # latest task plus its subagents traces investigate --all --last 10 --out report.md # explicit investigation alias traces improve --all --last 10 --dir .traces/improvement +traces ask --harness codex --session --question "Which commands failed?" +traces facts --harness codex --session # the deterministic facts sheet, $0 traces analyze --all --since 2026-06-18 --out report.md traces validate spans.otlp.jsonl # conformance; exit 1 only when it is not a trace traces validate results/sessions --out conformance.md # a whole directory of exports @@ -291,11 +295,15 @@ See [Replay verification](./docs/replay-verify.md) for setup, semantics, and hon | `--cwd ` | Filter by working directory | | `--since ` | `upload`: window, `30m`/`2h`/`7d` or ISO (default 24h); `analyze`: ISO cutoff | | `--out ` | Write the report to a file | -| `--dir ` | `improve`: write the full artifact pack to this directory | +| `--dir ` | `improve`: write the full artifact pack to this directory; `ask`: write `answers.json` + `report.md` there | | `--otlp ` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `analyze`, `investigate`, `improve`, `stream` | | `--otlp-out ` | **WRITE** the OTLP artifact here (also evidence provenance / dry-run upload preview) | | `--format ` | File `analyze`, `export`, or `stream`: `auto`, `policy-evidence`, `sandbox-events`, `openinference`, `intelligence-spans`, or `chat-trajectory` | | `--llm` / `--budget ` | Enable agentic analysts (needs `TANGLE_API_KEY` + Python with `agent-eval-rpc[dspy]`) / cap their spend | +| `--question ` | `ask`: one question, repeatable. Kept short so the engine sees it whole | +| `--questions ` | `ask`: JSON array of questions — strings, or `{ id?, question, instructions?, answerSchema? }` | +| `--question-budget ` | `ask`: provider ceiling for ONE question; `--budget` is the ceiling shared by all of them | +| `--concurrency ` | `ask`: questions running at once (default 4); `import-codetracebench`: trajectories imported at once | | `--config ` | `analyze` / `investigate` / `improve` / `stream`: load BYO analysts, live analysts, and external analyzers | | `--interval ` / `--window ` | `watch` / live `stream`: poll seconds (sessions 5, run tree 2) / active-session window minutes (default 30) | | `--min-loop ` | Identical repeated calls before flagging a loop (default 3) | @@ -443,6 +451,91 @@ The config can export: Traces does not pretend that an action is a measured candidate. Use `agent-eval` to propose and compare candidate changes, `agent-runtime` to package an approved improvement, and `agent-interface` to represent profile edits. +## Ask questions + +`traces ask` answers free-form questions about the selected sessions. +Each question becomes its own recursive investigation over the same trace, and the questions run at the same time under one budget. + +```bash +traces ask --harness codex --session \ + --question "Which shell commands exited non-zero, and what were they?" \ + --question "What was the last thing the human asked for?" \ + --dir .traces/ask +``` + +Ask many questions from a file, and hold an answer to a shape a scorer can compare: + +```bash +traces ask --all --last 3 --questions questions.json --concurrency 6 --budget 2 --dir .traces/ask +``` + +```json +[ + "Which pull requests did this session open?", + { + "id": "merged-prs", + "question": "Which pull requests were merged?", + "instructions": "Count only merges the trace records, not merges the agent said it would do.", + "answerSchema": { "type": "array", "items": { "type": "integer" } } + } +] +``` + +An entry is a question string, or an object with `question` and an optional `id`, `instructions`, and `answerSchema`. +The schema accepts a small JSON Schema subset: `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, and `const`. +Any other keyword is rejected when the run starts, because a constraint that is silently ignored would let a wrong answer pass as checked. + +The command writes two artifacts to `--dir`: + +| File | Contents | +|---|---| +| `answers.json` | per question: the answer text, the parsed answer, every citation with whether it resolves, accepted findings, evidence-gate rejections by reason, model calls, tool calls, cost with provenance, and latency | +| `report.md` | the same run as readable Markdown: a summary line, a per-question table, then each answer with its citation and rejection notes | + +What it checks, and what it costs: + +- **Facts first.** Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context before its first model call, at $0. The sheet is not citable; it names the span ids behind each fact. +- **Citations.** Every `trace:///span/` URI in an answer is looked up in the trace. An answer that cites a span the trace does not hold fails. +- **Budget.** `--budget` is one ceiling shared by every question. `--question-budget` bounds one question. The run refuses to start when the budget cannot cover a single model call, and warns when the budget admits fewer concurrent calls than `--concurrency`. +- **Cost.** Each cost carries its provenance: `observed` from a provider receipt, `estimated` from token counts, or `uncaptured`. An uncaptured cost stays null; it never becomes zero. +- **Exit code.** `ask` writes both artifacts first, then exits 1 when any question failed, returned no answer, broke its schema, or cited a span that does not exist. A failed question never costs the other answers. + +`ask` uses the same engine and credentials as `--llm`, so it needs `TANGLE_API_KEY` and a Python interpreter with `agent-eval-rpc[dspy]`. +Do not pass `--llm`; the command is model-backed by definition. + +## Session facts + +`traces facts` prints the deterministic session-facts sheet: the answers a session audit needs first, computed straight from the spans. +No model call, no engine, no budget — it costs $0 and always returns the same sheet for the same spans. + +```bash +traces facts --harness codex --session # JSON on stdout +traces facts --harness codex --last 5 --format text # the short readable form +traces facts --otlp spans.otlp.jsonl --out facts.json +``` + +| Fact | What it is | +|---|---| +| `toolCalls` | TOOL spans the agent actually invoked. Synthesized subagent lifecycle spans are excluded and counted separately in `synthesizedToolSpans`, so the total is not high by the number of subagents | +| `toolCallsByName` | the same calls by tool name, so a category decision is the reader's, not a guess | +| `subagents` | every `spawn_agent` call with the task name the adapter recorded | +| `pullRequests` | the pull requests the commands created and merged, each named by number or head branch, with the command span and how the identity was joined. Scanned the way a shell reads the script, so a `gh pr create` inside a heredoc body is not a command that ran | +| `humanTurns` | `user.prompt` turns a person typed into this session, in order, with the timestamp. Inherited fork or compaction history, harness-injected blocks, and a second record of the same turn are excluded — each listed in `excludedTurns` with its reason and span ids, never silently dropped. `turnsByActor` shows every turn by actor so the filter is checkable | +| `finalMessages` | the last message of the session's own agent, and of each subagent task, kept apart | +| `changedFiles` | paths named by patch headers and file-editing tool arguments, with the operation | +| `firstRecordAt` / `lastRecordAt` | the trace's earliest span start and latest span end | +| `unreadRecords` | records the session reader could not parse, from the session's integrity receipt | +| `tokenTotal` | the harness's own cumulative token total, when a span carries `traces.session.total_tokens` | + +Two rules hold for every field: + +- **Every fact names its span ids.** `spanIds` lists the spans the value was computed from, so any number here can be opened and checked. The sheet itself is not a span and cannot be cited. +- **A fact the spans cannot support is `null` with its reason.** It is never guessed, and never a silent zero. `partial` marks a measured value that is known to be incomplete — a truncated patch, or a list above the entry cap. + +`facts` exits non-zero when a selected session cannot be read at all: a session that produced no record spans would otherwise print a sheet of zeros stating, in the sheet's own voice, that the session did nothing. + +The same sheet reaches the model-backed analysts as prepared context, before their first model call — see [Trace analysts](docs/trace-analysts.md#session-facts-as-prepared-context). + ## Session index `traces index` writes one general JSON catalog over the selected sessions. @@ -608,6 +701,8 @@ traces analyze --last 1 --analyzer prime traces analyze --last 1 --analyzer my-installed-command ``` +To ask your own question instead of the built-in kinds, use [`traces ask`](#ask-questions). + HALO returns a diagnosis report. Hodoscope samples distinct behaviors and marks every sample `needs_review`. Prime posts the full span projection to an OpenAI-compatible bridge (`TRACES_PRIME_BRIDGE_URL`, default `http://localhost:4181`) and returns validated findings with span evidence. @@ -630,6 +725,7 @@ See [`examples/external-engines.ts`](./examples/external-engines.ts). > `--llm` also needs a Python interpreter with `agent-eval-rpc[dspy]` installed, because agent-eval's model-backed analysts run through the DSPy RLM engine out of process; set `TRACES_PYTHON` to choose the interpreter. > The bridge protocol is version-locked: install the exact version matching this package's `@tangle-network/agent-eval` dependency (`pip install "agent-eval-rpc[dspy]==$(npm view @tangle-network/traces dependencies.@tangle-network/agent-eval)"`) — a skewed bridge kills every agentic analyst at startup. > When `--llm` was requested and every agentic analyst fails, `analyze`/`investigate`/`improve` still write the deterministic report, then exit 1 with each analyst's underlying error. +> A requested `--analyzer` behaves the same way: its error is written into the report, and `analyze` then exits 1 naming every external analyzer that failed. > Every deterministic command — `list`, `analyze` without `--llm`, `convert`, `index`, `inspect`, `export`, `evidence`, `stream`, `watch`, `analyze --supervisor-run-dir` — needs neither a key nor Python. ## Agent skills @@ -689,6 +785,7 @@ The CLI is a thin consumer of these exports. | `analyzeSpans` | `(spans, { registry?, ai?, budgetUsd? }) → AnalyzeResult` | run built-in analysts, or **your own** via `registry` | | `runTraceInvestigation` | `(TraceInvestigationOptions) → TraceInvestigationResult` | typed findings with actions/checks, execution facts, external analyzer output, and report | | `runTraceImprovement` | `(TraceImprovementOptions) → TraceImprovementResult` | writes the full findings, evidence, report, and trace artifact pack | +| `runTraceQuestions` | `(TraceQuestionsOptions) → TraceQuestionsResult` | ask many free-form questions of one span list, concurrently, under one shared cost ledger; keeps each answer and checks its citations | | `buildTraceFindingPacket` | `({ findings }) → TraceFindingPacket` | render any `AnalystFinding[]` without changing its schema | | `runTraceStoreInvestigation` | `({ traceStore }) → TraceStoreInvestigationResult` | run the same packet layer over a hosted/custom `TraceAnalysisStore` | | `loadTracesConfig` | `(path?) → TracesConfig \| undefined` | load BYO analysts and external analyzers | diff --git a/docs/trace-analysts.md b/docs/trace-analysts.md index 102ff6d..9d5a997 100644 --- a/docs/trace-analysts.md +++ b/docs/trace-analysts.md @@ -8,6 +8,8 @@ Keeping them separate prevents an exploratory model output from being reported a | What happened? | Built-in local checks | Findings from explicit trace facts | | Why might it have happened? | `--llm`, HALO, or a custom analyst | Findings or a diagnosis report with cited spans | | What behavior should we inspect? | Hodoscope | Samples marked `needs_review` | +| What exactly does this session say about X? | `traces ask` | An answer per question, with checked `trace://` citations | +| What does this session state, exactly and for free? | `traces facts` | A deterministic facts sheet, no model call | ## Start here @@ -56,6 +58,210 @@ Returned directories from another resumed Claude session are parsed and included The OpenInference file is the shared input for external engines. The original trace and exact cited span remain available for review. +## Ask free-form questions + +The built-in kinds ask fixed questions. +`traces ask` asks your own. + +```bash +traces ask --harness codex --session \ + --question "Which shell commands exited non-zero?" \ + --question "Which pull requests did the session open, and were they merged?" \ + --dir .traces/ask +``` + +Each question runs through agent-eval's `runTraceAnalyst` directly, not through the analyst registry. +That matters for three reasons. + +- The registry keeps only findings, so it discards the engine's prose answer. `ask` keeps the answer text verbatim. +- The registry runs analysts one at a time. `ask` runs questions concurrently, so wall time falls toward the slowest question instead of the sum. +- One shared `CostLedger` bounds the whole run, so `--budget` means the same thing whatever the number of questions. + +### Questions + +A question stays short on purpose. +The engine shows the model a preview of each long input: it keeps the first 500 and last 500 characters and drops the middle. +A question inside the limit therefore reaches the model whole, and the answer rules sit in the first 500 characters of the instructions, which the preview always keeps. +`ask` rejects an over-long question before it starts a model call and names the limit. +Put the detail in the entry's `instructions` field instead, which the model reads after the rules. + +Read many questions from a file: + +```json +{ + "questions": [ + "What was the last thing the human asked for?", + { + "id": "failed-commands", + "question": "Which shell commands exited non-zero?", + "instructions": "Report the command line and the exit code. Ignore commands the agent only proposed.", + "answerSchema": { + "type": "array", + "items": { + "type": "object", + "properties": { "command": { "type": "string" }, "exit_code": { "type": "integer" } }, + "required": ["command", "exit_code"] + } + } + } + ] +} +``` + +An `answerSchema` makes the answer one JSON value a scorer can compare field by field. +The supported keywords are `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `title`, and `description`. +Any other keyword is rejected when the run starts. +This package carries no JSON Schema library, and a constraint that is quietly ignored would let a wrong answer pass as checked. + +### What the run guarantees + +- Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context, before its first model call. It costs nothing and answers the counting questions the bounded trace tools cannot. +- Every `trace:///span/` URI in an answer is resolved against the store. An unresolvable citation fails that question. +- Findings the answer submits still pass the same evidence gate as the built-in kinds. Refused findings are counted by reason in both artifacts. +- A failed question never stops the others. Its failure is recorded on its own answer, and the remaining answers are written. +- The artifacts are written before the exit code is decided. `ask` exits 1 when any question failed, returned no answer, broke its schema, or cited a missing span. + +### Budget under concurrency + +`--budget` is the ceiling shared by every question; `--question-budget` is the provider ceiling for one question. +The ledger reserves each model call's maximum charge before the call runs, and releases the unused part when the call settles. +Two consequences follow. + +- A budget below one call's reservation refuses the run before any model call, rather than failing every question. +- A budget that admits fewer concurrent reservations than `--concurrency` still runs, and the report carries a warning naming how many concurrent calls it covers. + +A question the ledger refuses is reported as `budget-refused`, and the answers already produced are kept. + +### SDK + +```ts +import { analysisEngineFromEnv, runTraceQuestions, writeTraceQuestionsArtifacts } from '@tangle-network/traces' + +const result = await runTraceQuestions({ + questions: [ + { id: 'failed-commands', question: 'Which shell commands exited non-zero?' }, + { id: 'last-ask', question: 'What was the last thing the human asked for?' }, + ], + spans, + engine: analysisEngineFromEnv({ model: 'gpt-5.6-luna', maxCostUsd: 1 }), + concurrency: 4, + budgetUsd: 2, +}) +await writeTraceQuestionsArtifacts(result, '.traces/ask') +if (!result.ok) process.exitCode = 1 +``` + +`result.questions[i].answer` is the engine's prose, unedited. +`result.totals` carries the wall time, the summed question time, the peak concurrency, and the cost with its provenance. + +## Session facts + +`traces facts` computes a fixed set of session facts straight from the spans. +It runs no model, opens no engine, and spends nothing. +The same spans always produce the same sheet. + +```bash +traces facts --harness codex --session # JSON on stdout +traces facts --harness codex --last 5 --format text # the short readable form +traces facts --otlp spans.otlp.jsonl --out facts.json +``` + +### Why it exists + +The trace tools are bounded, and above their bounds they answer a different question than the one asked. +`viewTrace` returns a `≤20`-entry span-name histogram once a trace exceeds `perCallByteCeiling` (150,000 bytes). +`countTraces` counts traces, not spans. +`viewSpans` needs span ids the reader does not have yet. +`searchTrace` stops at 500 hits. + +A model asked "how many tool calls ran?" therefore adds up a capped histogram and decides by eye which names belong. +Measured over twelve private audit sessions, the model-backed analyst arm scored a deterministic mean of **0.389**. +Extracting the same answers mechanically from the OTLP spans those runs already wrote scored **0.858** — the facts were present and exact the whole time. +The sheet is that extraction, made part of the tool. + +### What it states + +| Field | Value | +| --- | --- | +| `toolCalls` | TOOL spans the agent invoked. Synthesized subagent lifecycle spans are excluded and counted in `synthesizedToolSpans` | +| `toolCallsByName` | the same calls by tool name | +| `subagents` | every `spawn_agent` call with the task name from `traces.codex.spawn_agent_path` | +| `pullRequests` | pull requests the commands created and merged, each named by number or head branch, with the command span and the join evidence | +| `humanTurns` | `user.prompt` turns a person typed into this session, in order, with timestamps | +| `excludedTurns` | every `user.prompt` turn `humanTurns` left out, grouped by the reason, with the span ids | +| `turnsByActor` | every `user.prompt` turn by actor, so the human filter is checkable | +| `finalMessages` | the last message of the session's own agent, and of each subagent task, separately | +| `changedFiles` | paths from `*** Add/Update/Delete/Move to File:` patch headers and from file-editing tool arguments | +| `firstRecordAt`, `lastRecordAt` | the trace's earliest span start and latest span end | +| `unreadRecords` | records the session reader could not parse | +| `tokenTotal` | the harness's cumulative total, when a span carries `traces.session.total_tokens` | + +Two rules hold for every field. + +- **Every fact names the span ids it came from.** A reader can open those spans and check the number. The sheet is not a span and cannot be cited. +- **A fact the spans cannot support is `null` with a stated reason.** It is never guessed and never a silent zero. `partial` marks a measured value known to be incomplete. + +`facts` exits non-zero when a selected session produced no record spans, rather than printing a sheet of zeros for a session it could not read. + +#### Pull requests + +The command spans carry the script, the exit code and the output. +`pullRequests` scans each script the way a shell would, so a `gh pr create` inside a heredoc body or a commit message is not counted as a command that ran. +A pull request is named by its number when the command or an output that joins to it shows one, and by its head branch when neither does. +A create whose stdout was redirected away takes the number a later output states for the same branch on the same line, and names the span that stated it. +A failed `git push && gh pr create` counts only when the output shows `gh` itself answering; otherwise the shell never reached it. +A trace whose spans carry no executed command returns `null` with that reason, because "no pull requests" and "the spans cannot say" are different answers. + +#### Human turns + +A user message is one the human typed into this session. +Three filters run in order, and each excluded span is listed in `excludedTurns` with its reason, never silently dropped: + +- history the session carries but did not receive — the prefix a fork copies from its parent, and the turns a compaction replays; +- turns whose actor is not a person — an instruction file, an environment-context block, a system reminder, a subagent notification, a turn-aborted marker, or a skill or slash-command expansion; +- a second record of the turn before it, meaning the same text at the same instant with no model call, tool call or assistant message between them. + +For Codex the actor comes from the harness's own per-item labelling (`internal_chat_message_metadata_passthrough.content_item_kinds`) whenever the record carries it: a message is the person's exactly when every item in it is a `user.` kind. + +### Session facts as prepared context + +The same sheet is supplied to the model-backed analysts before their first model call, through `TraceAnalystDefinition.prepareContext`. +It reaches the built-in kinds run by `analyze --llm`, `investigate`, and `improve`, and it reaches every `traces ask` question inside `PREPARED CONTEXT:`. + +- The sheet is bounded at `PREPARED_CONTEXT_BYTE_CEILING` (30,000 bytes), a fifth of the `perCallByteCeiling` of 150,000 the trace tools work to, so the rest of the budget stays available for the tool calls the model still makes. +- When the sheet does not fit, fields are shed from the largest downward and the shed is listed in `omitted_fields`. The tool-call counts are the last facts to go. +- Receiving the sheet changes an analyst's behavior, so its version carries `+session-facts.1`. `createTraceAnalyst` records `prepare_context` in the exact-run identity, and a changed prepared context must not hide behind an unchanged version. +- The sheet is not evidence. Citations still resolve against the raw spans, which is why every fact names its span ids rather than asking the model to trust the sheet. + +Pass `sessionFactsContext: false` to `analyzeSpans` to run an analyst without it. + +```ts +import { buildSessionFactsReport, computeSessionFacts, renderSessionFacts } from '@tangle-network/traces' + +const [facts] = computeSessionFacts(spans) +console.log(facts.toolCalls.value, facts.toolCalls.spanIds) +console.log(renderSessionFacts(buildSessionFactsReport(spans))) +``` + +## Evidence-gate rejections + +A model-backed analyst can submit a finding the evidence gate then refuses. +Without the reason, a report showing "0 findings" reads as "the model found nothing" when the truth may be "the gate refused everything it found". + +`analyze`, `investigate`, `improve`, and `ask` now carry those refusals: + +- the CLI log prints the reason and the offending URI on each `finding rejected` line; +- the analyst table's Detail cell names the reasons and their counts; +- `result.findingRejections` (investigation and improvement) and `answers.json` (`ask`) hold the counts per analyst and reason. + +The common reasons are an excerpt the cited span does not contain, a span the trace does not hold, and too few distinct citations for the kind. + +## External analyzer failures exit non-zero + +`--analyzer halo|hodoscope|prime|` promises that engine's output. +An analyzer that fails now writes its error into the report as before, and then `analyze` exits 1 naming every analyzer that failed. +Scripts that treated exit 0 as "the analyzer ran" were reading a report that said otherwise. + ## Codex tool outcomes Both function and custom tool outputs use the same status parser. diff --git a/src/adapters/actor.ts b/src/adapters/actor.ts index 7549abe..e74c2d9 100644 --- a/src/adapters/actor.ts +++ b/src/adapters/actor.ts @@ -161,11 +161,93 @@ export function claudeActor(args: { } /** - * Derive the actor for a Codex user message. Codex has no sidechain/userType, - * so it's text-only: synthetic markers → injected, first-turn agent-spawn - * brief → injected, otherwise human. + * Context blocks Codex writes as user-role messages and never reports as a user + * turn. The list mirrors `CONTEXTUAL_USER_FRAGMENT_MATCHERS` (openai/codex + * `codex-rs/core/src/context/contextual_user_message.rs`); the default matcher + * accepts a trimmed block that starts with the open marker and ends with the + * close marker, ignoring ASCII case (openai/codex + * `codex-rs/context-fragments/src/fragment.rs`). */ -export function codexActor(args: { text: string; isFirstUserTurn?: boolean }): Actor { +const CODEX_CONTEXT_BLOCKS: ReadonlyArray = [ + ['# AGENTS.md instructions', ''], + ['', ''], + ['', ''], + ['', ''], + ['', ''], + ['', ''], + ['', ''], + [''], + ['', ''], + ['', ''], +] + +/** Harness warnings Codex also injects as user-role text, matched by prefix. */ +const CODEX_CONTEXT_PREFIXES = [ + 'Warning: apply_patch was requested via ', + 'Warning: Your account was flagged for potentially high-risk cyber activity', + 'Warning: The maximum number of unified exec processes you can keep open is', +] as const + +const CODEX_EXTERNAL_CONTEXT = /^]+)>[\s\S]*<\/external_\1>$/ + +function startsWithIgnoringCase(text: string, prefix: string): boolean { + return text.slice(0, prefix.length).toLowerCase() === prefix.toLowerCase() +} + +function endsWithIgnoringCase(text: string, suffix: string): boolean { + return text.slice(-suffix.length).toLowerCase() === suffix.toLowerCase() +} + +/** Whether one text block of a Codex user-role message is harness context. */ +export function isCodexContextBlock(block: string): boolean { + const text = block.trim() + if (CODEX_CONTEXT_PREFIXES.some((prefix) => text.startsWith(prefix))) return true + if (CODEX_EXTERNAL_CONTEXT.test(text)) return true + return CODEX_CONTEXT_BLOCKS.some( + ([open, close]) => startsWithIgnoringCase(text, open) && endsWithIgnoringCase(text, close), + ) +} + +/** + * Codex's own label for each content item of a user-role message, from + * `internal_chat_message_metadata_passthrough.content_item_kinds`. + * + * Codex tags every item it puts in a user message with what the item is: + * `user.text` for what the person typed, and a namespaced kind for everything + * the harness added — `agents_md.instructions` for an AGENTS.md file, + * `environments.environment_context` for the environment block, + * `goal.internal_context`, `plugins.recommendations`, and so on. A message is + * one a person typed exactly when every item in it is a `user.` kind. + * + * This is the structural signal, so it decides on its own: it is what the + * harness recorded, not what the text looks like. Returns undefined when the + * record carries no kinds — older rollouts and event mirrors — and the text + * heuristics answer instead. + */ +export function codexKindsAreHuman(kinds: readonly unknown[] | undefined): boolean | undefined { + if (!Array.isArray(kinds) || kinds.length === 0) return undefined + if (!kinds.every((kind) => typeof kind === 'string')) return undefined + return kinds.every((kind) => (kind as string).startsWith('user.')) +} + +/** + * Derive the actor for a Codex user message. + * + * `kinds` is Codex's own per-item labelling and decides whenever the record + * carries it. Without it the decision is text-only: a Codex context block → + * injected, synthetic markers → injected, first-turn agent-spawn brief → + * injected, otherwise human. `blocks` are the message's separate text blocks; + * Codex treats the whole message as context when any one block is. + */ +export function codexActor(args: { + text: string + blocks?: readonly string[] + isFirstUserTurn?: boolean + kinds?: readonly unknown[] +}): Actor { + const recorded = codexKindsAreHuman(args.kinds) + if (recorded !== undefined) return recorded ? 'human' : 'injected' + if ((args.blocks ?? [args.text]).some(isCodexContextBlock)) return 'injected' if (textIsCmdOrInject(args.text)) return 'injected' if (textIsSynthetic(args.text)) return 'injected' if (args.isFirstUserTurn && looksLikeAgentPrompt(args.text)) return 'injected' diff --git a/src/adapters/codex-format.ts b/src/adapters/codex-format.ts index 0b181b2..ed1d510 100644 --- a/src/adapters/codex-format.ts +++ b/src/adapters/codex-format.ts @@ -44,6 +44,27 @@ export interface CodexLine { author?: string recipient?: string namespace?: string + /** `user_message` event text, and the summary on a `compacted` record. */ + message?: unknown + /** + * Conversation history a `compacted` record retained, in response-item + * shape. Codex writes the pre-compaction turns here and nowhere else, so + * this is the only surviving copy of what the human typed before the + * context was replaced. + */ + replacement_history?: ReadonlyArray<{ + type?: string + id?: string + role?: string + content?: unknown + internal_chat_message_metadata_passthrough?: { + content_item_kinds?: readonly unknown[] + } + }> + window_id?: string + previous_window_id?: string + first_window_id?: string + window_number?: number item?: { type?: string id?: string @@ -56,6 +77,9 @@ export interface CodexLine { } internal_chat_message_metadata_passthrough?: { turn_id?: string + /** What each content item of a user-role message is: `user.text` for the + * person's own words, a namespaced kind for anything the harness added. */ + content_item_kinds?: readonly unknown[] } source?: { subagent?: { @@ -121,6 +145,188 @@ export function codexSubagentActivity(line: CodexLine): CodexSubagentActivity | } } +/** A command Codex ran, from an `item_completed` event whose item is `CommandExecution`. */ +export interface CodexCommandExecution { + readonly itemId: string + /** Recorded argv (current builds) or command string, verbatim. */ + readonly command: readonly string[] | string + readonly cwd?: string + readonly processId?: string + readonly source?: string + readonly status?: string + readonly exitCode?: number + readonly output?: string | { readonly stdout?: string; readonly stderr?: string } + readonly outputFields: readonly string[] + readonly startedAtMs?: number + readonly completedAtMs?: number +} + +export interface CodexFileChangeEntry { + readonly path: string + /** Codex's change type (`add`, `delete`, `update`), verbatim. */ + readonly kind: string + readonly movePath?: string +} + +/** Files a patch changed, from an `item_completed` event whose item is `FileChange`. */ +export interface CodexFileChange { + readonly itemId: string + readonly changes: readonly CodexFileChangeEntry[] + readonly status?: string + readonly startedAtMs?: number + readonly completedAtMs?: number +} + +/** A turn a client submitted, from an `item_completed` event whose item is `UserMessage`. */ +export interface CodexUserMessage { + readonly itemId: string + readonly text: string +} + +/** + * One `item_completed` event, normalized. `skipped` carries the label the adapter + * counts when an item produces no span: the item type, or `:malformed` + * when a required field is missing. + */ +export type CodexCompletedItem = + | { readonly type: 'CommandExecution'; readonly item: object; readonly command: CodexCommandExecution } + | { readonly type: 'FileChange'; readonly item: object; readonly fileChange: CodexFileChange } + | { readonly type: 'UserMessage'; readonly item: object; readonly userMessage: CodexUserMessage } + | { readonly type: 'skipped'; readonly label: string } + +type JsonRecord = Record + +function recordValue(value: unknown): JsonRecord | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) ? value as JsonRecord : undefined +} + +function nonEmptyString(value: unknown): string | undefined { + return typeof value === 'string' && value.length > 0 ? value : undefined +} + +/** Codex defaults a missing `completed_at_ms` to 0, so only a positive time is a recorded time. */ +function epochMs(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) && value > 0 ? value : undefined +} + +function commandValue(value: unknown): readonly string[] | string | undefined { + if (typeof value === 'string') return value.length > 0 ? value : undefined + if (!Array.isArray(value) || value.length === 0) return undefined + return value.every((part) => typeof part === 'string') ? value as string[] : undefined +} + +function commandOutput(item: JsonRecord): Pick { + if (typeof item.aggregated_output === 'string') { + return { output: item.aggregated_output, outputFields: ['aggregated_output'] } + } + const stdout = typeof item.stdout === 'string' ? item.stdout : undefined + const stderr = typeof item.stderr === 'string' ? item.stderr : undefined + if (stdout === undefined && stderr === undefined) return { outputFields: [] } + return { + output: { ...(stdout === undefined ? {} : { stdout }), ...(stderr === undefined ? {} : { stderr }) }, + outputFields: [...(stdout === undefined ? [] : ['stdout']), ...(stderr === undefined ? [] : ['stderr'])], + } +} + +/** The rollout records a map keyed by path; `codex exec --json` records an array of `{path, kind}`. */ +function fileChangeEntries(value: unknown): CodexFileChangeEntry[] | undefined { + const entries: CodexFileChangeEntry[] = [] + if (Array.isArray(value)) { + for (const raw of value) { + const change = recordValue(raw) + const path = nonEmptyString(change?.path) + const kind = nonEmptyString(change?.kind) ?? nonEmptyString(change?.type) + if (!path || !kind) return undefined + const movePath = nonEmptyString(change?.move_path) + entries.push({ path, kind, ...(movePath ? { movePath } : {}) }) + } + } else { + const changes = recordValue(value) + if (!changes) return undefined + for (const [path, raw] of Object.entries(changes)) { + const change = recordValue(raw) + const kind = nonEmptyString(change?.type) ?? nonEmptyString(change?.kind) + if (path.length === 0 || !kind) return undefined + const movePath = nonEmptyString(change?.move_path) + entries.push({ path, kind, ...(movePath ? { movePath } : {}) }) + } + } + if (entries.length === 0) return undefined + return entries.sort((left, right) => (left.path < right.path ? -1 : left.path > right.path ? 1 : 0)) +} + +/** The text a `UserMessage` item carries; other input parts (images, audio) have no text. */ +function userMessageText(content: unknown): string { + if (!Array.isArray(content)) return '' + return content + .map((part) => { + const entry = recordValue(part) + return entry?.type === 'text' && typeof entry.text === 'string' ? entry.text : '' + }) + .join('') +} + +/** + * Normalize Codex's `item_completed` event for the command, file-change and + * user-message items. + * + * Field names follow `CommandExecutionItem`, `FileChangeItem`, `UserMessageItem` + * and `ItemCompletedEvent` in openai/codex `codex-rs/protocol`. The format drifts, + * so a required field that is missing or mistyped yields a counted `skipped` + * label rather than a span built from guessed values. + */ +export function codexCompletedItem(line: CodexLine): CodexCompletedItem | undefined { + if (line.type !== 'event_msg' || line.payload?.type !== 'item_completed') return undefined + const payload = line.payload as JsonRecord + const item = recordValue(payload.item) + const type = nonEmptyString(item?.type) ?? 'unknown' + if (!item || (type !== 'CommandExecution' && type !== 'FileChange' && type !== 'UserMessage')) { + return { type: 'skipped', label: type } + } + const itemId = nonEmptyString(item.id) + if (type === 'UserMessage') { + const text = userMessageText(item.content) + // An image-only or audio-only turn carries no text to record as a turn span. + if (!itemId || !text) return { type: 'skipped', label: `${type}:${itemId ? 'no_text' : 'malformed'}` } + return { type, item, userMessage: { itemId, text } } + } + const startedAtMs = epochMs(payload.started_at_ms) ?? epochMs(item.started_at_ms) + const completedAtMs = epochMs(payload.completed_at_ms) ?? epochMs(item.completed_at_ms) + const status = nonEmptyString(item.status) + const times = { + ...(startedAtMs === undefined ? {} : { startedAtMs }), + ...(completedAtMs === undefined ? {} : { completedAtMs }), + } + if (type === 'FileChange') { + const changes = fileChangeEntries(item.changes) + if (!itemId || !changes) return { type: 'skipped', label: `${type}:malformed` } + return { type, item, fileChange: { itemId, changes, ...(status ? { status } : {}), ...times } } + } + const command = commandValue(item.command) + if (!itemId || !command) return { type: 'skipped', label: `${type}:malformed` } + const cwd = nonEmptyString(item.cwd) + const processId = typeof item.process_id === 'number' && Number.isSafeInteger(item.process_id) + ? String(item.process_id) + : nonEmptyString(item.process_id) + const source = nonEmptyString(item.source) + const exitCode = typeof item.exit_code === 'number' && Number.isSafeInteger(item.exit_code) ? item.exit_code : undefined + return { + type, + item, + command: { + itemId, + command, + ...(cwd ? { cwd } : {}), + ...(processId ? { processId } : {}), + ...(source ? { source } : {}), + ...(status ? { status } : {}), + ...(exitCode === undefined ? {} : { exitCode }), + ...commandOutput(item), + ...times, + }, + } +} + export function contentToString(content: unknown): string { if (typeof content === 'string') return content if (Array.isArray(content)) { @@ -135,6 +341,17 @@ export function contentToString(content: unknown): string { return '' } +/** Each text block of a message, unjoined, so a per-block classifier sees block boundaries. */ +export function contentTextBlocks(content: unknown): string[] { + if (typeof content === 'string') return [content] + if (!Array.isArray(content)) return [] + return content.flatMap((item) => ( + item && typeof item === 'object' && typeof (item as { text?: unknown }).text === 'string' + ? [(item as { text: string }).text] + : [] + )) +} + export function timestampFromEpochMs(value: unknown): string | undefined { if (typeof value !== 'number' || !Number.isFinite(value)) return undefined const date = new Date(value) diff --git a/src/adapters/codex.ts b/src/adapters/codex.ts index 660266c..bbb97f3 100644 --- a/src/adapters/codex.ts +++ b/src/adapters/codex.ts @@ -14,7 +14,7 @@ * Shared by the codex-acp wrapper via alias (same rollout format). */ -import { sourceOf, textSources } from '../source-location.js' +import { type SourceReferences, sourceOf, textSources } from '../source-location.js' import { readdir, stat } from 'node:fs/promises' import { homedir } from 'node:os' @@ -34,11 +34,16 @@ import type { SpawnedChildResolution, } from '../types.js' import { codexActor } from './actor.js' -import { capText, userPromptSpan } from './conversation.js' +import { ACTOR_ATTR, capText, userPromptSpan } from './conversation.js' import { + type CodexCommandExecution, + type CodexCompletedItem, + codexCompletedItem, + type CodexFileChange, type CodexLine, codexSubagentActivity, type CodexTokenUsage, + contentTextBlocks, contentToString, latestTimestamp, multiAgentOperation, @@ -57,17 +62,58 @@ import { isCodexTaskBoundary, resolveCodexParentTask, } from './codex-task-scope.js' -import { recordToolOutput, toolIoAttributes } from './tool-io.js' +import { + INHERITED_SOURCE_ATTR, + INHERITED_SPAN_ATTR, + INHERITED_SPAN_COUNT_ATTR, + INHERITED_SPANS_OMITTED_ATTR, + type InheritedSpanSource, + isInheritedSpan, + SYNTHESIZED_SOURCE_ATTR, + SYNTHESIZED_SPAN_ATTR, +} from './provenance.js' +import { INNER_TOOL_CALL_LEVEL, recordToolOutput, TOOL_CALL_LEVEL_ATTR, toolIoAttributes } from './tool-io.js' export { CodexTaskScopeError } from './codex-task-scope.js' const SERVICE = 'codex' const SESSION_HEAD_LINES = 40 +/** + * Per-session ceiling on inherited spans. The prefix of a fork and the retained + * history of every `compacted` record are unbounded in a long rollout, and this + * adapter parses under a bounded heap. What the cap drops is counted on the root + * in `traces.session.inherited_spans_omitted`, never silently discarded. + */ +const MAX_INHERITED_SPANS = 200 + const CODEX_SOURCE_TRACE_ID = 'traces.codex.source_trace_id' const CODEX_SOURCE_SPAN_ID = 'traces.codex.source_span_id' const CODEX_SOURCE_PARENT_SPAN_ID = 'traces.codex.source_parent_span_id' +/** + * The harness's own cumulative token counter, carried verbatim. + * + * Codex reports `token_count.info.total_token_usage` beside the per-turn + * `last_token_usage` delta. The two are different numbers and neither derives + * the other: summing the deltas misses whatever the harness counted outside the + * recorded turns, and summing the cumulative snapshots multiplies the session + * total by the number of events. The last snapshot IS the session total, so it + * is copied onto the root span and never recomputed here. + */ +const SESSION_TOTAL_TOKENS = 'traces.session.total_tokens' +const SESSION_TOTAL_INPUT_TOKENS = 'traces.session.total_input_tokens' +const SESSION_TOTAL_OUTPUT_TOKENS = 'traces.session.total_output_tokens' +const SESSION_TOTAL_REASONING_TOKENS = 'traces.session.total_reasoning_tokens' +const SESSION_TOTAL_CACHED_INPUT_TOKENS = 'traces.session.total_cached_input_tokens' +const SESSION_TOTAL_TOKENS_SOURCE = 'traces.session.total_tokens_source' +const CODEX_TOTAL_TOKENS_SOURCE = 'codex.token_count.info.total_token_usage' + +/** Carry a reported counter only when it is a usable non-negative number. */ +function reportedCount(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : undefined +} + /** Convert Codex's readable span identities to fixed-width OTLP wire IDs. */ function normalizeCodexIds(spans: OtlpSpan[]): void { for (const item of spans) { @@ -167,8 +213,9 @@ function explicitOutputError(value: unknown, timeoutIsError = true): boolean | u const exitCode = numericStatus(header.match(/^Process exited with code[ \t]+(-?\d+)[ \t]*$/im)?.[1]) if (exitCode !== undefined) return exitCode !== 0 } - if (/^Script completed\s*\nWall time:/i.test(header)) return false - if (/^Script failed\s*\nWall time:/i.test(header)) return true + // Receipts print either "Wall time: 1.2 seconds" or "Wall time 1.2 seconds". + if (/^Script completed\s*\nWall time\b/i.test(header)) return false + if (/^Script failed\s*\nWall time\b/i.test(header)) return true const scriptExitCode = numericStatus(header.match(/^Script error:[ \t]*\r?\nExit code:[ \t]*(-?\d+)[ \t]*(?:\r?\n|$)/i)?.[1]) if (scriptExitCode !== undefined) return scriptExitCode !== 0 const commandExitCode = numericStatus(text.match(/^Command failed with exit code[ \t]+(-?\d+)\.?$/i)?.[1]) @@ -380,6 +427,189 @@ function closeSpanAt(target: OtlpSpan, sourceEndTime: string): void { target.end_time = sourceEndTime } +/** A completed item the adapter turns into spans; user messages take the turn path instead. */ +type CompletedItem = + & Extract + & { readonly recordTime: string } + +/** The time window of a model-issued call: its call record to its output record. */ +interface ToolWindow { + readonly startMs: number + readonly endMs: number +} + +/** + * How an item was placed. Codex item IDs never equal call IDs, so an item joins + * the one call whose window contains the item's whole run. An item that ran + * past every window (a command left running, then polled) or sits inside two + * windows (parallel calls) stays under the session root instead of a guess. + */ +type ItemJoin = 'call' | 'unmatched' | 'ambiguous' + +function joinItem( + windows: ReadonlyMap, + startMs: number, + endMs: number, +): { parent?: OtlpSpan; join: ItemJoin } { + const matches: OtlpSpan[] = [] + for (const [toolSpan, window] of windows) { + if (window.startMs <= startMs && endMs <= window.endMs) matches.push(toolSpan) + } + if (matches.length === 1) return { parent: matches[0], join: 'call' } + return { join: matches.length === 0 ? 'unmatched' : 'ambiguous' } +} + +function itemTimes( + item: { readonly startedAtMs?: number; readonly completedAtMs?: number }, + recordTime: string, +): { start: string; end: string; timeSource?: 'completed_only' | 'record' } { + const end = timestampFromEpochMs(item.completedAtMs) ?? recordTime + const start = timestampFromEpochMs(item.startedAtMs) ?? end + if (item.startedAtMs !== undefined && item.completedAtMs !== undefined) return { start, end } + return { start, end, timeSource: item.completedAtMs === undefined ? 'record' : 'completed_only' } +} + +function itemStatus( + status: string | undefined, + noun: string, + exitCode?: number, +): { code: OtlpSpan['status']['code']; message?: string } { + if (exitCode !== undefined && exitCode !== 0) return { code: 'ERROR', message: `${noun} exited ${exitCode}` } + if (status === 'failed' || status === 'declined') return { code: 'ERROR', message: `${noun} ${status}` } + if (exitCode === 0 || status === 'completed') return { code: 'OK' } + return { code: 'UNSET' } +} + +function itemSources(item: object, fields: readonly string[]) { + return fields.flatMap((field) => { + const reference = sourceOf(item, field) + return reference ? [reference] : [] + }) +} + +function innerItemAttributes( + type: CompletedItem['type'], + itemId: string, + join: ItemJoin, + timeSource: string | undefined, + status: string | undefined, +): Record { + return { + // Existing OTLP importers classify this marker as a container, not a call. + 'span.type': 'tool.execution', + [TOOL_CALL_LEVEL_ATTR]: INNER_TOOL_CALL_LEVEL, + 'traces.codex.item_type': type, + 'traces.codex.item_id': itemId, + 'traces.codex.item_join': join, + ...(timeSource ? { 'traces.codex.item_time_source': timeSource } : {}), + ...(status ? { 'traces.codex.item_status': status } : {}), + } +} + +interface ItemSpanContext { + readonly traceId: string + readonly rootId: string + readonly windows: ReadonlyMap +} + +/** + * One CHAIN span per command. Command text, cwd, and output stay in the tool + * I/O keys, which metadata-only upload strips and external redactors scrub. + */ +function commandSpan(context: ItemSpanContext, item: object, command: CodexCommandExecution, recordTime: string): OtlpSpan { + const { start, end, timeSource } = itemTimes(command, recordTime) + const { parent, join } = joinItem(context.windows, Date.parse(start), Date.parse(end)) + const status = itemStatus(command.status, 'command', command.exitCode) + const commandSpan = span({ + traceId: context.traceId, + spanId: `command:${command.itemId}`, + parentSpanId: parent?.span_id ?? context.rootId, + name: 'command.execution', + kind: 'CHAIN', + startTime: start, + status: status.code, + statusMessage: status.message, + service: SERVICE, + agent: SERVICE, + extra: { + ...toolIoAttributes({ + input: { command: command.command, ...(command.cwd ? { cwd: command.cwd } : {}) }, + inputSource: itemSources(item, ['command', ...(command.cwd ? ['cwd'] : [])]), + output: command.output, + outputSource: itemSources(item, command.outputFields), + }), + ...innerItemAttributes('CommandExecution', command.itemId, join, timeSource, command.status), + ...(command.exitCode === undefined ? {} : { 'process.exit_code': command.exitCode }), + ...(command.processId ? { 'traces.codex.process_id': command.processId } : {}), + ...(command.source ? { 'traces.codex.command_source': command.source } : {}), + }, + }) + closeSpanAt(commandSpan, end) + return commandSpan +} + +/** One CHAIN span per changed path; the path stays in `input.value` for the same reason as commands. */ +function fileChangeSpans(context: ItemSpanContext, item: object, fileChange: CodexFileChange, recordTime: string): OtlpSpan[] { + const { start, end, timeSource } = itemTimes(fileChange, recordTime) + const { parent, join } = joinItem(context.windows, Date.parse(start), Date.parse(end)) + const status = itemStatus(fileChange.status, 'file change') + return fileChange.changes.map((change, index) => { + const changeSpan = span({ + traceId: context.traceId, + spanId: `file-change:${fileChange.itemId}:${index}`, + parentSpanId: parent?.span_id ?? context.rootId, + name: 'file.change', + kind: 'CHAIN', + startTime: start, + status: status.code, + statusMessage: status.message, + service: SERVICE, + agent: SERVICE, + extra: { + ...toolIoAttributes({ + input: { path: change.path, kind: change.kind, ...(change.movePath ? { move_path: change.movePath } : {}) }, + inputSource: sourceOf(item, 'changes'), + }), + ...innerItemAttributes('FileChange', fileChange.itemId, join, timeSource, fileChange.status), + 'traces.codex.file_change_kind': change.kind, + }, + }) + closeSpanAt(changeSpan, end) + return changeSpan + }) +} + +/** A user turn awaiting its second record: Codex logs each typed turn as a response item and a `user_message` event. */ +interface UserTurnCandidate { + readonly span: OtlpSpan + readonly key: string + readonly task: number +} + +/** + * Legacy Codex prepends context to the submitted message and marks the typed + * text with this line (openai/codex `codex-rs/protocol/src/protocol.rs` + * `USER_MESSAGE_BEGIN`), so the two records of one turn can differ by a prefix. + */ +const USER_MESSAGE_BEGIN = '## My request for Codex:' + +function userTurnKey(text: string): string { + const begin = text.indexOf(USER_MESSAGE_BEGIN) + const typed = begin === -1 ? text : text.slice(begin + USER_MESSAGE_BEGIN.length) + return typed.trim().replace(/\s+/g, ' ') +} + +/** Remove and return the latest candidate with the same text in the same task. */ +function takeUserTurn(candidates: UserTurnCandidate[], key: string, task: number): UserTurnCandidate | undefined { + for (let index = candidates.length - 1; index >= 0; index -= 1) { + const candidate = candidates[index]! + if (candidate.key === key && candidate.task === task) return candidates.splice(index, 1)[0] + } + return undefined +} + +const USER_MESSAGE_EVENT_ATTR = 'traces.codex.user_message_event' + const verificationCommand = /\b(?:pnpm|npm|yarn|bun)\s+(?:run\s+)?(?:test|typecheck|lint|build|check)(?::[A-Za-z0-9:_-]+)?\b|\b(?:vitest|jest|pytest|tsc|biome|eslint|sha256sum|pdfinfo|pdftotext)\b|\bgo\s+test\b|\bcargo\s+(?:test|check|clippy|build)\b|\bgit\s+(?:status|diff|show|merge-tree)\b|\bgh-drew\s+pr\s+(?:view|checks)\b/i @@ -663,9 +893,207 @@ export class CodexAdapter implements HarnessTraceAdapter { let step = 0 let lastLlm = rootId let sawUserTurn = false + let sawInheritedTurn = false + let inheritedSpansEmitted = 0 + let inheritedSpansOmitted = 0 let lastCumulativeTokenUsage: string | undefined + let sessionTotalUsage: CodexTokenUsage | undefined let lastTimestamp: string | undefined const awaitingModel = model ? [] : [root] + const toolWindows = new Map() + const completedItems: CompletedItem[] = [] + const completedItemKeys = new Set() + const skippedItemCounts = new Map() + const countSkippedItem = (label: string): void => { + skippedItemCounts.set(label, (skippedItemCounts.get(label) ?? 0) + 1) + } + // Pairs the two records of one typed turn. A task index scopes the pairing, + // so the same short reply in two turns stays two turns. + let taskIndex = 0 + const unpairedUserItems: UserTurnCandidate[] = [] + const unpairedUserEvents: UserTurnCandidate[] = [] + const tasksWithUserEvents = new Set() + const inheritedTurnKeys = new Set() + /** + * Record one turn Codex reports as submitted input. Codex reports these + * turns and never its own context blocks: the legacy `user_message` event + * and the current `item_completed`/`UserMessage` item both come from the + * same filter (openai/codex `codex-rs/core/src/event_mapping.rs`), and the + * rollout carries one or the other by history mode (openai/codex + * `codex-rs/rollout/src/policy.rs`). The response-item copy of the same turn + * pairs with this record instead of becoming a second span. + */ + const recordSubmittedTurn = (raw: string, ts: string, contentSource: SourceReferences): void => { + const prompt = capText(raw) + if (!prompt) return + tasksWithUserEvents.add(taskIndex) + const key = userTurnKey(raw) + const recorded = takeUserTurn(unpairedUserItems, key, taskIndex) + if (recorded) { + recorded.span.attributes[USER_MESSAGE_EVENT_ATTR] = true + return + } + const actor = sessionRole === 'child' + ? 'agent' + : codexActor({ text: prompt, isFirstUserTurn: !sawUserTurn }) + sawUserTurn = true + const turnSpan = userPromptSpan({ + traceId, + spanId: `msg:${step}:user`, + parentSpanId: rootId, + startTime: ts, + content: prompt, + contentSource, + service: SERVICE, + agent: SERVICE, + step, + actor, + }) + turnSpan.attributes[USER_MESSAGE_EVENT_ATTR] = true + spans.push(turnSpan) + unpairedUserEvents.push({ span: turnSpan, key, task: taskIndex }) + step += 1 + } + /** Reserve a slot for one inherited span, counting what the cap turns away. */ + const claimInheritedSpan = (): boolean => { + if (inheritedSpansEmitted >= MAX_INHERITED_SPANS) { + inheritedSpansOmitted += 1 + return false + } + inheritedSpansEmitted += 1 + return true + } + /** + * A turn this session carries but did not receive: the prefix a fork copies + * from its parent, and the history a `compacted` record retains. Codex + * rewrites both into the child's rollout, and the task-scope walk used to + * drop them, so a forked child's human context reached no span at all and + * "what did the human ask for?" had no answer in the trace. + * + * The span is a normal `user.prompt` with its real actor, plus + * `traces.session.inherited` — a reader that wants the human's words finds + * them by name, and a count of THIS scope's turns excludes them by flag. + * + * Deduplicated on the turn text, because one retained turn is rewritten + * into every later `compacted` record. The key is bounded (length plus a + * text prefix) so a long session cannot grow the key set without bound. + */ + const recordInheritedTurn = ( + raw: string, + ts: string, + contentSource: SourceReferences, + source: InheritedSpanSource, + blocks?: readonly string[], + kinds?: readonly unknown[], + ): void => { + const prompt = capText(raw) + if (!prompt) return + const key = userTurnKey(raw) + const dedupKey = `${key.length}:${key.slice(0, 256)}` + if (inheritedTurnKeys.has(dedupKey)) return + if (!claimInheritedSpan()) return + inheritedTurnKeys.add(dedupKey) + const actor = codexActor({ text: prompt, blocks, isFirstUserTurn: !sawInheritedTurn, kinds }) + sawInheritedTurn = true + const turnSpan = userPromptSpan({ + traceId, + spanId: `inherited:${step}:user`, + parentSpanId: rootId, + startTime: ts, + content: prompt, + contentSource, + service: SERVICE, + agent: SERVICE, + step, + actor, + }) + turnSpan.attributes[INHERITED_SPAN_ATTR] = true + turnSpan.attributes[INHERITED_SOURCE_ATTR] = source + spans.push(turnSpan) + step += 1 + } + /** + * Records the parsed scope inherited rather than produced. Two shapes reach + * here: any line before the selected task boundary (the fork prefix), and a + * `compacted` record anywhere in the file. A compacted record carries the + * summary Codex replaced the context with, plus the history it retained. + */ + const recordInheritedContext = (l: CodexLine, ts: string): void => { + if (l.type === 'compacted') { + const payload = l.payload + if (!payload) return + const summary = capText(typeof payload.message === 'string' ? payload.message : '') + if (summary && claimInheritedSpan()) { + spans.push(span({ + traceId, + spanId: `inherited:${step}:compacted`, + parentSpanId: rootId, + name: 'session.compacted', + kind: 'CHAIN', + startTime: ts, + service: SERVICE, + agent: SERVICE, + step, + content: summary, + contentSource: textSources(payload, 'message'), + extra: { + [INHERITED_SPAN_ATTR]: true, + [INHERITED_SOURCE_ATTR]: 'compacted' satisfies InheritedSpanSource, + ...(typeof payload.window_number === 'number' ? { 'traces.codex.compaction_window_number': payload.window_number } : {}), + ...(payload.window_id ? { 'traces.codex.compaction_window_id': payload.window_id } : {}), + ...(payload.previous_window_id ? { 'traces.codex.compaction_previous_window_id': payload.previous_window_id } : {}), + }, + })) + step += 1 + } + for (const item of payload.replacement_history ?? []) { + if (!item || typeof item !== 'object') continue + if (item.type !== 'message' || item.role !== 'user') continue + recordInheritedTurn( + contentToString(item.content), + ts, + textSources(item, 'content'), + 'compacted', + contentTextBlocks(item.content), + item.internal_chat_message_metadata_passthrough?.content_item_kinds, + ) + } + return + } + if (l.type === 'response_item' && l.payload?.type === 'message' && l.payload.role === 'user') { + recordInheritedTurn( + contentToString(l.payload.content), + ts, + textSources(l.payload, 'content'), + 'pre-task-prefix', + contentTextBlocks(l.payload.content), + l.payload.internal_chat_message_metadata_passthrough?.content_item_kinds, + ) + return + } + if (l.type === 'event_msg' && l.payload?.type === 'user_message') { + recordInheritedTurn( + typeof l.payload.message === 'string' ? l.payload.message : '', + ts, + textSources(l.payload, 'message'), + 'pre-task-prefix', + ) + return + } + // Only the turn shape is decoded here: a prefix `FileChange` item would + // parse a whole diff this walk never records. + if (l.type === 'event_msg' && l.payload?.type === 'item_completed' && l.payload.item?.type === 'UserMessage') { + const completed = codexCompletedItem(l) + if (completed?.type === 'UserMessage') { + recordInheritedTurn( + completed.userMessage.text, + ts, + textSources(completed.item, 'content'), + 'pre-task-prefix', + ) + } + } + } const ensureSubagentSpan = ( threadId: string, agentPath: string, @@ -696,16 +1124,21 @@ export class CodexAdapter implements HarnessTraceAdapter { return existing } const subagentType = agentPath.split('/').filter(Boolean).at(-1) ?? 'subagent' + // The child thread's lifecycle, assembled from `sub_agent_activity` + // events — NOT a call the model issued. It was a TOOL span named + // `tool.Agent`, so every tool-call count (here and in any consumer that + // counts TOOL spans or `tool.name`) ran high by one per child thread. + // It is an AGENT span with no `tool.name`, marked synthesized, so a + // counter needs no name allowlist to get the model's tool calls right. const toolSpan = span({ traceId, spanId: `subagent:${threadId}`, parentSpanId: eventCallSpan?.span_id ?? lastLlm, - name: 'tool.Agent', - kind: 'TOOL', + name: 'subagent.lifecycle', + kind: 'AGENT', startTime: eventTime, service: SERVICE, agent: SERVICE, - tool: 'Agent', step, status: 'UNSET', extra: { @@ -716,6 +1149,9 @@ export class CodexAdapter implements HarnessTraceAdapter { agent_thread_id: threadId, }, }), + [SYNTHESIZED_SPAN_ATTR]: true, + [SYNTHESIZED_SOURCE_ATTR]: 'codex.sub_agent_activity', + 'traces.codex.subagent_type': subagentType, 'traces.codex.subagent_path': agentPath, 'traces.codex.subagent_thread_id': threadId, ...(!observedStart ? { 'traces.codex.subagent_start_missing': true } : {}), @@ -729,9 +1165,14 @@ export class CodexAdapter implements HarnessTraceAdapter { } let reachedCurrentTask = !selectedBoundary + let prefixTimestamp: string | undefined for await (const l of readJsonl(ref.path, jsonl)) { if (!reachedCurrentTask) { - if (!isCodexTaskBoundary(l, selectedBoundary!)) continue + if (!isCodexTaskBoundary(l, selectedBoundary!)) { + prefixTimestamp = validTimestamp(l.timestamp) ?? prefixTimestamp + recordInheritedContext(l, prefixTimestamp ?? root.start_time) + continue + } reachedCurrentTask = true if (options.taskScope === 'turn') { root.start_time = codexTaskBoundary(l)?.timestamp ?? root.start_time @@ -748,6 +1189,7 @@ export class CodexAdapter implements HarnessTraceAdapter { lastTimestamp = latestTimestamp(lastTimestamp, l.timestamp) const ts = validTimestamp(l.timestamp) ?? lastTimestamp ?? root.start_time if (l.type === 'event_msg' && l.payload?.type === 'task_started') { + taskIndex += 1 activeTaskTurnId = l.payload.turn_id ?? null root.status = { code: 'UNSET' } } else if (l.type === 'event_msg' && l.payload?.type === 'task_complete') { @@ -763,6 +1205,13 @@ export class CodexAdapter implements HarnessTraceAdapter { awaitingModel.length = 0 } else if (l.type === 'event_msg' && l.payload?.type === 'token_count') { const u = l.payload.info?.last_token_usage + // Read before the delta gate: a `token_count` event that reports no + // per-turn delta still advances the harness's cumulative counter, and + // the last snapshot in scope is the session total. + const reportedTotal = l.payload.info?.total_token_usage + if (reportedTotal && reportedCount(reportedTotal.total_tokens) !== undefined) { + sessionTotalUsage = reportedTotal + } if (u && (u.input_tokens || u.output_tokens)) { const cumulative = l.payload.info?.total_token_usage const cumulativeSignature = cumulative ? tokenUsageSignature(cumulative) : undefined @@ -790,6 +1239,10 @@ export class CodexAdapter implements HarnessTraceAdapter { lastLlm = id step += 1 } + } else if (l.type === 'compacted') { + // Compaction replaces the model's context with a summary and a retained + // history. Both are inherited context, not new turns in this scope. + recordInheritedContext(l, ts) } else if ( l.type === 'response_item' && (l.payload?.type === 'function_call' || l.payload?.type === 'custom_tool_call') @@ -848,6 +1301,7 @@ export class CodexAdapter implements HarnessTraceAdapter { const name = String(t.attributes['tool.name'] ?? '') const { status, pollOutcome } = outputStatus(name, l.payload) closeSpanAt(t, ts) + toolWindows.set(t, { startMs: Date.parse(t.start_time), endMs: Date.parse(ts) }) t.status = status if (pollOutcome) t.attributes['traces.poll.outcome'] = pollOutcome recordToolOutput(t, l.payload.output, sourceOf(l.payload, 'output')) @@ -864,9 +1318,31 @@ export class CodexAdapter implements HarnessTraceAdapter { if (requestId) t.attributes['traces.codex.agent_request_id'] = requestId } } + } else if (l.type === 'event_msg' && l.payload?.type === 'user_message') { + recordSubmittedTurn( + typeof l.payload.message === 'string' ? l.payload.message : '', + ts, + textSources(l.payload, 'message'), + ) } else if (l.type === 'event_msg') { const activity = codexSubagentActivity(l) - if (!activity) continue + if (!activity) { + const completed = codexCompletedItem(l) + if (completed?.type === 'skipped') { + countSkippedItem(completed.label) + } else if (completed?.type === 'UserMessage') { + recordSubmittedTurn(completed.userMessage.text, ts, textSources(completed.item, 'content')) + } else if (completed) { + const itemId = completed.type === 'CommandExecution' ? completed.command.itemId : completed.fileChange.itemId + const itemKey = `${completed.type}:${itemId}` + if (completedItemKeys.has(itemKey)) countSkippedItem(`${completed.type}:duplicate`) + else { + completedItemKeys.add(itemKey) + completedItems.push({ ...completed, recordTime: ts }) + } + } + continue + } const threadId = activity.agentThreadId const eventTime = timestampFromEpochMs(activity.occurredAtMs) ?? ts const eventCallSpan = toolByCallId.get(activity.eventId ?? '') @@ -953,26 +1429,35 @@ export class CodexAdapter implements HarnessTraceAdapter { } else if (l.type === 'response_item' && l.payload?.type === 'message' && l.payload.role === 'user') { // The human's prompt text. Codex drops the user turn from token events, // so capture it here as its own CHAIN span (no text → no span). - const prompt = textOf(l.payload.content) + const raw = contentToString(l.payload.content) + const prompt = capText(raw) if (prompt) { + const key = userTurnKey(raw) + // The user_message event already recorded this turn. + if (takeUserTurn(unpairedUserEvents, key, taskIndex)) continue const actor = sessionRole === 'child' ? 'agent' - : codexActor({ text: prompt, isFirstUserTurn: !sawUserTurn }) + : codexActor({ + text: prompt, + blocks: contentTextBlocks(l.payload.content), + isFirstUserTurn: !sawUserTurn, + kinds: l.payload.internal_chat_message_metadata_passthrough?.content_item_kinds, + }) sawUserTurn = true - spans.push( - userPromptSpan({ - traceId, - spanId: `msg:${step}:user`, - parentSpanId: rootId, - startTime: ts, - content: prompt, - contentSource: textSources(l.payload, 'content'), - service: SERVICE, - agent: SERVICE, - step, - actor, - }), - ) + const turnSpan = userPromptSpan({ + traceId, + spanId: `msg:${step}:user`, + parentSpanId: rootId, + startTime: ts, + content: prompt, + contentSource: textSources(l.payload, 'content'), + service: SERVICE, + agent: SERVICE, + step, + actor, + }) + spans.push(turnSpan) + unpairedUserItems.push({ span: turnSpan, key, task: taskIndex }) step += 1 } } else if (l.type === 'response_item' && l.payload?.type === 'message') { @@ -1003,8 +1488,53 @@ export class CodexAdapter implements HarnessTraceAdapter { `Codex turn ${JSON.stringify(options.taskTurnId)} does not exist in ${ref.path}`, ) } + // Where Codex recorded user_message events for a task, a user-role message + // without one is harness context, even under a wrapper not listed in actor.ts. + for (const candidate of unpairedUserItems) { + if (candidate.span.attributes[ACTOR_ATTR] !== 'human' || !tasksWithUserEvents.has(candidate.task)) continue + candidate.span.attributes[ACTOR_ATTR] = 'injected' + candidate.span.attributes['traces.codex.actor_evidence'] = 'no_user_message_event' + } + const itemContext: ItemSpanContext = { traceId, rootId, windows: toolWindows } + for (const completed of completedItems) { + if (completed.type === 'CommandExecution') { + spans.push(commandSpan(itemContext, completed.item, completed.command, completed.recordTime)) + } else { + spans.push(...fileChangeSpans(itemContext, completed.item, completed.fileChange, completed.recordTime)) + } + } + if (skippedItemCounts.size > 0) { + root.attributes['traces.codex.skipped_item_counts'] = JSON.stringify( + Object.fromEntries([...skippedItemCounts].sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))), + ) + } + if (sessionTotalUsage) { + const total = reportedCount(sessionTotalUsage.total_tokens) + if (total !== undefined) { + root.attributes[SESSION_TOTAL_TOKENS] = total + root.attributes[SESSION_TOTAL_TOKENS_SOURCE] = CODEX_TOTAL_TOKENS_SOURCE + // The rest of the same snapshot, so the parts and the total agree. + const input = reportedCount(sessionTotalUsage.input_tokens) + const output = reportedCount(sessionTotalUsage.output_tokens) + const reasoning = reportedCount(sessionTotalUsage.reasoning_output_tokens) + const cached = reportedCount(sessionTotalUsage.cached_input_tokens) + if (input !== undefined) root.attributes[SESSION_TOTAL_INPUT_TOKENS] = input + if (output !== undefined) root.attributes[SESSION_TOTAL_OUTPUT_TOKENS] = output + if (reasoning !== undefined) root.attributes[SESSION_TOTAL_REASONING_TOKENS] = reasoning + if (cached !== undefined) root.attributes[SESSION_TOTAL_CACHED_INPUT_TOKENS] = cached + } + } + if (inheritedSpansEmitted > 0) root.attributes[INHERITED_SPAN_COUNT_ATTR] = inheritedSpansEmitted + // A dropped record stays visible as a count: inherited context is bounded, + // and "the cap was hit" must not read as "there was nothing before this". + if (inheritedSpansOmitted > 0) root.attributes[INHERITED_SPANS_OMITTED_ATTR] = inheritedSpansOmitted if (selectedBoundary?.turnId) { - for (const item of spans) item.attributes['traces.codex.turn_id'] ??= selectedBoundary.turnId + // An inherited record predates the selected turn; stamping it with that + // turn id would claim it happened inside the turn. + for (const item of spans) { + if (isInheritedSpan(item.attributes)) continue + item.attributes['traces.codex.turn_id'] ??= selectedBoundary.turnId + } } closeSpanAt(root, lastTimestamp ?? root.start_time) normalizeCodexIds(spans) diff --git a/src/adapters/provenance.ts b/src/adapters/provenance.ts new file mode 100644 index 0000000..d9a3405 --- /dev/null +++ b/src/adapters/provenance.ts @@ -0,0 +1,47 @@ +/** + * Provenance markers for spans an adapter did NOT take from an action the agent + * performed inside the parsed scope. + * + * Two cases, and both used to be invisible: + * + * - SYNTHESIZED — the adapter built the span from harness lifecycle events, + * not from a call the model issued. A Codex subagent's start/finish stream + * is one span per child thread; recording it as a TOOL call named + * `tool.Agent` made every tool-call count high by exactly the number of + * child threads, and nothing on the span let a counter tell the difference. + * - INHERITED — the record belongs to context this session carries but did + * not produce: the prefix a fork copies from its parent, and the history a + * `compacted` record retains. Keeping the human's words is the point; + * counting them as turns of THIS scope is not. + * + * Both markers are additive attributes. A count that means "what the agent did + * in this scope" filters them out; a reader that wants the context selects them. + */ + +/** `true` on a span the adapter synthesized from lifecycle events. */ +export const SYNTHESIZED_SPAN_ATTR = 'traces.span.synthesized' + +/** What the synthesized span was built from, e.g. `codex.sub_agent_activity`. */ +export const SYNTHESIZED_SOURCE_ATTR = 'traces.span.synthesized_from' + +/** `true` on a span carrying context from outside the parsed scope. */ +export const INHERITED_SPAN_ATTR = 'traces.session.inherited' + +/** Where the inherited record came from: `pre-task-prefix` or `compacted`. */ +export const INHERITED_SOURCE_ATTR = 'traces.session.inherited_source' + +export type InheritedSpanSource = 'pre-task-prefix' | 'compacted' + +/** Count of inherited spans in the batch, stamped on the root span. */ +export const INHERITED_SPAN_COUNT_ATTR = 'traces.session.inherited_span_count' + +/** Inherited records the adapter's per-session cap dropped, stamped on the root. */ +export const INHERITED_SPANS_OMITTED_ATTR = 'traces.session.inherited_spans_omitted' + +export function isSynthesizedSpan(attributes: Readonly>): boolean { + return attributes[SYNTHESIZED_SPAN_ATTR] === true +} + +export function isInheritedSpan(attributes: Readonly>): boolean { + return attributes[INHERITED_SPAN_ATTR] === true +} diff --git a/src/adapters/tool-io.ts b/src/adapters/tool-io.ts index 900ff46..42ca933 100644 --- a/src/adapters/tool-io.ts +++ b/src/adapters/tool-io.ts @@ -5,6 +5,19 @@ import { sourceAttributes, SOURCE_ATTRIBUTE_PREFIX, type SourceReferences } from export const TOOL_IO_VALUE_MAX_BYTES = 16 * 1024 export const TOOL_IO_VALUE_KEYS = ['input.value', 'output.value'] as const +/** + * Marks work recorded inside a model-issued tool call, such as each command a + * code-mode script ran. These spans are CHAIN, not TOOL, and carry no + * `tool.name`, so every tool-call counter here and in agent-eval stays at the + * model-issued level; readers that want the inner facts select this value. + */ +export const TOOL_CALL_LEVEL_ATTR = 'traces.tool_call.level' +export const INNER_TOOL_CALL_LEVEL = 'inner' + +export function isInnerToolCall(attributes: Readonly>): boolean { + return attributes[TOOL_CALL_LEVEL_ATTR] === INNER_TOOL_CALL_LEVEL +} + interface ToolIoInput { inputSource?: SourceReferences outputSource?: SourceReferences diff --git a/src/adoption.ts b/src/adoption.ts index abe5758..1f142db 100644 --- a/src/adoption.ts +++ b/src/adoption.ts @@ -13,6 +13,7 @@ import { readFile } from 'node:fs/promises' import { join } from 'node:path' +import { isSynthesizedSpan } from './adapters/provenance.js' import { toolArgumentsFromAttributes } from './adapters/tool-io.js' import { indexSessionIdsByTrace, @@ -299,12 +300,17 @@ export async function analyzeAdoption(spans: readonly OtlpSpan[], opts: Adoption addCounts(skillDocumentReads, skillDocuments) } const tn = toolName(s) + // A Codex subagent's lifecycle span is synthesized from harness events, so + // it carries no `tool.name`. It still reports one child thread, which is + // what the canonical subagent count means. + const subagentLifecycle = isSynthesizedSpan(s.attributes) + && typeof s.attributes['traces.codex.subagent_thread_id'] === 'string' if (tn === 'Skill') { sessionCapabilities.set(group, 'supported') const name = skillNameOf(parseInput(s)) skillInvocations[name] = (skillInvocations[name] ?? 0) + 1 sessionsWithSkill.add(group) - } else if (tn === 'Task' || tn === 'Agent') { + } else if (tn === 'Task' || tn === 'Agent' || subagentLifecycle) { const type = subagentTypeOf(parseInput(s)) subagentSpawns[type] = (subagentSpawns[type] ?? 0) + 1 sessionsWithSubagent.add(group) diff --git a/src/analysis-store.ts b/src/analysis-store.ts new file mode 100644 index 0000000..7d70521 --- /dev/null +++ b/src/analysis-store.ts @@ -0,0 +1,74 @@ +/** + * The trace file and stores every analysis path reads. + * + * `analyze`, `investigate`, `improve`, and `ask` all write the selected spans + * to one OpenInference file and read it back through agent-eval's + * `OtlpFileTraceStore`. Keeping that sequence here means a source-bundle grant, + * the file ceiling, and the containment check cannot differ between them. + */ + +import { OtlpFileTraceStore, type SpanSourceReader } from '@tangle-network/agent-eval/traces' +import { assertOutsideSourceBundle, createBundleSourceReader } from './bundle-source.js' +import type { OtlpSpan } from './otlp.js' +import { writeOtlpFile } from './otlp.js' + +/** + * `viewTrace` and generated-file ceiling. The default 150KB cap exists to + * protect an LLM's context window, and a single coding session is one trace + * whose full span list routinely exceeds 150KB. The fixed ceiling covers large + * sessions without disabling agent-eval's file-size guard. + */ +export const GENERATED_TRACE_FILE_CEILING = 512 * 1024 * 1024 + +export interface AnalysisTraceFile { + readonly otlpPath: string + /** Present only when the caller explicitly granted source reads through a bundle. */ + readonly sourceReader?: SpanSourceReader +} + +/** Write the spans once and bind any explicitly granted source reader. */ +export async function writeAnalysisTraceFile( + spans: readonly OtlpSpan[], + opts: { + sourceBundle?: { path: string; maxRecordBytes?: number } + otlpOutPath?: string + signal?: AbortSignal + } = {}, +): Promise { + if (opts.sourceBundle && opts.otlpOutPath) await assertOutsideSourceBundle(opts.sourceBundle.path, opts.otlpOutPath) + const sourceReader = opts.sourceBundle + ? await createBundleSourceReader(opts.sourceBundle.path, spans, { + signal: opts.signal, + maxRecordBytes: opts.sourceBundle.maxRecordBytes, + }) + : undefined + const otlpPath = await writeOtlpFile(spans, opts.otlpOutPath) + return sourceReader ? { otlpPath, sourceReader } : { otlpPath } +} + +/** + * Store for model-driven analysis. It keeps agent-eval's default per-call byte + * ceiling, so each tool result stays bounded for a model's context; the + * engine drills in with `viewSpans` and `searchTrace` from a summary. + */ +export async function openAgenticTraceStore(file: AnalysisTraceFile): Promise { + const store = new OtlpFileTraceStore({ + path: file.otlpPath, + maxFileBytes: GENERATED_TRACE_FILE_CEILING, + ...(file.sourceReader ? { sourceReader: file.sourceReader } : {}), + }) + await store.ensureIndexed() + return store +} + +/** Store for deterministic analysts, which have no context window to protect. */ +export async function openDeterministicTraceStore(file: AnalysisTraceFile): Promise { + const store = new OtlpFileTraceStore({ + path: file.otlpPath, + maxFileBytes: GENERATED_TRACE_FILE_CEILING, + perCallByteCeiling: GENERATED_TRACE_FILE_CEILING, + ...(file.sourceReader ? { sourceReader: file.sourceReader } : {}), + }) + await store.ensureIndexed() + return store +} diff --git a/src/analyst-model-call.ts b/src/analyst-model-call.ts index a7c7b73..c8e943f 100644 --- a/src/analyst-model-call.ts +++ b/src/analyst-model-call.ts @@ -4,6 +4,20 @@ import { canonicalCandidateDigest, } from '@tangle-network/agent-interface' import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' +import { createDspyRlmTraceEngine, type TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' + +/** Model the CLI's recursive analysts use when neither `--model` nor TRACES_ANALYST_MODEL names one. */ +export const DEFAULT_ANALYST_MODEL = 'gpt-5.6-luna' + +/** + * Provider ceiling for one `ask` question when neither `--question-budget` nor + * a smaller `--budget` names one. It equals the DSPy engine's own default + * investigation ceiling, stated here so `ask` owns the value it applies. + */ +export const DEFAULT_QUESTION_MAX_COST_USD = 1 + +/** Default analysis endpoint: the Tangle router, reached with TANGLE_API_KEY. */ +export const TANGLE_ROUTER_BASE_URL = 'https://router.tangle.tools/v1' export const ANALYST_MAX_OUTPUT_TOKENS = 16_384 export const GPT_5_6_ANALYST_MAX_OUTPUT_TOKENS = 8_192 @@ -55,3 +69,82 @@ export function createAnalystModelOwner(opts: { profile, } } + +export interface AnalysisEngineFromEnvOptions { + model: string + /** + * Provider-side spend ceiling for ONE investigation (one analyst or one + * question). Omitted, the engine keeps its own default. A caller that runs + * several investigations under one shared ledger passes the per-investigation + * cap here and the total to the ledger. + */ + maxCostUsd?: number + /** Receives one line per model call, in the CLI's analyst log format. */ + log?: (msg: string, fields?: Record) => void + /** Environment to read credentials and the Python interpreter from. Default: process.env. */ + env?: NodeJS.ProcessEnv +} + +/** + * The recursive analysis engine behind `--llm` and `ask`. agent-eval's + * model-backed analysts run through DSPy RLM, which drives + * `agent-eval-rpc[dspy]` out of process, so this needs a Python interpreter + * with that extra installed, selectable via TRACES_PYTHON. Every deterministic + * command is unaffected and still needs neither a key nor Python. + */ +export function analysisEngineFromEnv(opts: AnalysisEngineFromEnvOptions): TraceAnalysisEngine { + const env = opts.env ?? process.env + // The router is the default endpoint, so TANGLE_API_KEY alone is enough. + // OPENAI_API_KEY still works and, when it is the only key present, points at + // OpenAI directly; otherwise a plain OpenAI key would be sent to the router. + const tangleKey = env.TANGLE_API_KEY + const openAiKey = env.OPENAI_API_KEY + const apiKey = tangleKey || openAiKey + if (!apiKey) { + throw new Error( + 'model-backed analysis needs a model key: TANGLE_API_KEY for the Tangle router (the default endpoint), or ' + + 'OPENAI_API_KEY for OpenAI. Set OPENAI_BASE_URL to target any other OpenAI-compatible ' + + 'gateway. Deterministic analysis needs no key.', + ) + } + const baseUrl = + env.OPENAI_BASE_URL || + (tangleKey ? TANGLE_ROUTER_BASE_URL : 'https://api.openai.com/v1') + const python = env.TRACES_PYTHON + const owner = createAnalystModelOwner({ + apiKey, + baseUrl, + model: opts.model, + provider: + baseUrl === TANGLE_ROUTER_BASE_URL + ? 'tangle-router' + : baseUrl.startsWith('https://api.openai.com/') + ? 'openai' + : 'openai-compatible', + }) + const log = opts.log + return createDspyRlmTraceEngine({ + call: owner.call, + callRef: owner.callRef, + recordExecution: (observation) => { + log?.( + `[analyst] model call ${observation.sequence} ${observation.succeeded ? 'ok' : 'FAIL'} ${observation.model}`, + observation.succeeded ? undefined : { error: observation.error }, + ) + }, + model: opts.model, + // Model-aware, not defaulted: GPT-5.6 needs less output room than models + // such as GLM, and every recursive call reserves this full amount before + // execution. An oversized reservation can reject useful later calls even + // when the run's measured spend remains well below its limit. + maxOutputTokens: analystMaxOutputTokens(opts.model), + // maxCostUsd defaults to $1 per investigation, a proxy-side ceiling + // separate from any shared ledger. With the larger token cap the per-call + // reservation grows ~4x, so that default can bind before the caller's own + // allocation and kill investigations mid-run. + ...(opts.maxCostUsd !== undefined && Number.isFinite(opts.maxCostUsd) && opts.maxCostUsd > 0 + ? { maxCostUsd: opts.maxCostUsd } + : {}), + ...(python ? { runner: { command: python } } : {}), + }) +} diff --git a/src/analyze.ts b/src/analyze.ts index 41ee982..3a64b65 100644 --- a/src/analyze.ts +++ b/src/analyze.ts @@ -18,14 +18,14 @@ import { type AnalystRegistry, type AnalystRunSummary, buildDefaultAnalystRegistry, + DEFAULT_TRACE_ANALYST_KINDS, type TraceAnalysisEngine, type TraceAnalystDefinition, } from '@tangle-network/agent-eval/analyst' -import { OtlpFileTraceStore } from '@tangle-network/agent-eval/traces' +import { openAgenticTraceStore, openDeterministicTraceStore, writeAnalysisTraceFile } from './analysis-store.js' import { summarizeSpanExecution } from './execution.js' import type { OtlpSpan } from './otlp.js' -import { writeOtlpFile } from './otlp.js' -import { assertOutsideSourceBundle, createBundleSourceReader } from './bundle-source.js' +import { withSessionFactsContext } from './session-facts.js' export interface AnalyzeOptions { /** Explicitly authorize original source reads from this full session bundle. */ @@ -51,6 +51,16 @@ export interface AnalyzeOptions { agenticRegistry?: AnalystRegistry /** Select a subset of agent-eval's maintained trace analyst kinds. */ agenticKinds?: readonly TraceAnalystDefinition[] + /** + * Supply the deterministic session-facts sheet to the agentic kinds as + * prepared context (default true). It costs nothing and removes the guessing + * the bounded trace tools force on a model that needs an exact count. Set + * false to measure an analyst without it. + * + * It applies only to definitions this call builds a registry from; a caller + * who brings `agenticRegistry` owns its own prepared context. + */ + sessionFactsContext?: boolean /** Compact deterministic findings that agents receive before reading spans. */ agenticPriorFindings?: readonly AnalystFinding[] /** Where to write the OTLP-JSONL artifact. Defaults to a temp file. */ @@ -74,16 +84,6 @@ export interface AnalyzeResult { agenticPerAnalyst?: readonly AnalystRunSummary[] } -/** - * `viewTrace` and generated-file ceiling for the deterministic pass. The - * default 150KB cap exists to protect an LLM's context window — the - * deterministic behavioral analyst has none, and a single coding session is one trace whose full - * span list routinely exceeds 150KB (→ oversized summary → zero spans → - * zero findings). The fixed ceiling covers large sessions without disabling - * agent-eval's file-size guard. - */ -const GENERATED_TRACE_FILE_CEILING = 512 * 1024 * 1024 - function mergeCostProvenance( first: RunCostProvenance | undefined, second: RunCostProvenance | undefined, @@ -100,11 +100,12 @@ function mergeCostProvenance( export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOptions = {}): Promise { if (spans.length === 0) throw new Error('analyzeSpans: no spans to analyze') opts.signal?.throwIfAborted() - if (opts.sourceBundle && opts.otlpOutPath) await assertOutsideSourceBundle(opts.sourceBundle.path, opts.otlpOutPath) - const sourceReader = opts.sourceBundle - ? await createBundleSourceReader(opts.sourceBundle.path, spans, { signal: opts.signal, maxRecordBytes: opts.sourceBundle.maxRecordBytes }) - : undefined - const otlpPath = await writeOtlpFile(spans, opts.otlpOutPath) + const traceFile = await writeAnalysisTraceFile(spans, { + sourceBundle: opts.sourceBundle, + otlpOutPath: opts.otlpOutPath, + signal: opts.signal, + }) + const { otlpPath } = traceFile opts.signal?.throwIfAborted() const runId = opts.runId ?? `traces-${Date.now()}` const execution = summarizeSpanExecution(spans, { @@ -114,13 +115,7 @@ export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOpti // Deterministic pass — high ceiling so the behavioral analyst sees the whole // trace. No LLM context to protect here. A caller-supplied registry (custom // analysts / their own agents) runs here instead of the built-in suite. - const detStore = new OtlpFileTraceStore({ - path: otlpPath, - maxFileBytes: GENERATED_TRACE_FILE_CEILING, - perCallByteCeiling: GENERATED_TRACE_FILE_CEILING, - ...(sourceReader ? { sourceReader } : {}), - }) - await detStore.ensureIndexed() + const detStore = await openDeterministicTraceStore(traceFile) opts.signal?.throwIfAborted() const detRegistry = opts.registry ?? buildDefaultAnalystRegistry({ registry: { log: opts.log } }) const result = await detRegistry.run(runId, { traceStore: detStore }, { signal: opts.signal }) @@ -130,14 +125,22 @@ export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOpti // the RLM kinds drill via viewSpans/searchTrace from a summary. let agenticPerAnalyst: readonly AnalystRunSummary[] | undefined if (opts.engine || opts.agenticRegistry) { - const agStore = new OtlpFileTraceStore({ path: otlpPath, maxFileBytes: GENERATED_TRACE_FILE_CEILING, ...(sourceReader ? { sourceReader } : {}) }) - await agStore.ensureIndexed() - const agRegistry = opts.agenticRegistry ?? buildDefaultAnalystRegistry({ - engine: opts.engine!, - ...(opts.agenticKinds ? { definitions: opts.agenticKinds } : {}), - includeBehavioral: false, - registry: { log: opts.log }, - }) + const agStore = await openAgenticTraceStore(traceFile) + // The sheet reaches the model through each definition's `prepareContext`, + // which runs before the first model call. Its facts are exact where the + // bounded trace tools force a guess, and it costs nothing to compute. A + // caller-supplied agentic registry owns its own prepared context, so the + // sheet is built only when this call builds the registry. + const buildRegistry = (): AnalystRegistry => { + const kinds = opts.agenticKinds ?? DEFAULT_TRACE_ANALYST_KINDS + return buildDefaultAnalystRegistry({ + engine: opts.engine!, + definitions: opts.sessionFactsContext === false ? kinds : withSessionFactsContext(kinds, spans), + includeBehavioral: false, + registry: { log: opts.log }, + }) + } + const agRegistry = opts.agenticRegistry ?? buildRegistry() const agResult = await agRegistry.run(runId, { traceStore: agStore }, { budget: opts.budgetUsd != null ? { totalUsd: opts.budgetUsd } : undefined, chainFindings: true, diff --git a/src/answer-schema.ts b/src/answer-schema.ts new file mode 100644 index 0000000..869e727 --- /dev/null +++ b/src/answer-schema.ts @@ -0,0 +1,154 @@ +/** + * Answer schemas for `traces ask`. + * + * A question may fix the shape of its answer with a JSON Schema, so a scorer + * can compare fields exactly instead of reading prose. This package carries no + * JSON Schema library, so it checks a small subset and REJECTS any schema that + * uses a keyword outside it: a schema whose constraint is silently ignored + * would let a wrong answer pass as checked. + * + * Supported keywords: `type`, `properties`, `required`, `additionalProperties` + * (boolean), `items` (one schema), `enum`, `const`, and the annotations + * `title` and `description`. + */ + +export type AnswerSchema = Readonly> + +const SUPPORTED_KEYWORDS = new Set([ + 'type', + 'properties', + 'required', + 'additionalProperties', + 'items', + 'enum', + 'const', + 'title', + 'description', +]) + +const JSON_TYPES = new Set(['string', 'number', 'integer', 'boolean', 'object', 'array', 'null']) + +const MAX_SCHEMA_DEPTH = 32 + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +/** Throw a TypeError naming the first unsupported or malformed part of `schema`. */ +export function assertAnswerSchema(schema: unknown, path = 'answerSchema', depth = 0): asserts schema is AnswerSchema { + if (depth > MAX_SCHEMA_DEPTH) throw new TypeError(`${path}: schema nesting exceeds ${MAX_SCHEMA_DEPTH} levels`) + if (!isRecord(schema)) throw new TypeError(`${path} must be a JSON Schema object`) + for (const key of Object.keys(schema)) { + if (!SUPPORTED_KEYWORDS.has(key)) { + throw new TypeError( + `${path}: unsupported JSON Schema keyword "${key}"; supported: ${[...SUPPORTED_KEYWORDS].join(', ')}`, + ) + } + } + if (schema.type !== undefined) { + const types = Array.isArray(schema.type) ? schema.type : [schema.type] + if (types.length === 0 || !types.every((type) => typeof type === 'string' && JSON_TYPES.has(type))) { + throw new TypeError(`${path}.type must be one of ${[...JSON_TYPES].join(', ')}, or an array of them`) + } + } + if (schema.properties !== undefined) { + if (!isRecord(schema.properties)) throw new TypeError(`${path}.properties must be an object`) + for (const [name, child] of Object.entries(schema.properties)) { + assertAnswerSchema(child, `${path}.properties.${name}`, depth + 1) + } + } + if (schema.required !== undefined) { + if (!Array.isArray(schema.required) || !schema.required.every((name) => typeof name === 'string')) { + throw new TypeError(`${path}.required must be an array of property names`) + } + } + if (schema.additionalProperties !== undefined && typeof schema.additionalProperties !== 'boolean') { + throw new TypeError(`${path}.additionalProperties must be a boolean`) + } + if (schema.items !== undefined) assertAnswerSchema(schema.items, `${path}.items`, depth + 1) + if (schema.enum !== undefined && (!Array.isArray(schema.enum) || schema.enum.length === 0)) { + throw new TypeError(`${path}.enum must be a non-empty array`) + } + for (const key of ['title', 'description'] as const) { + if (schema[key] !== undefined && typeof schema[key] !== 'string') { + throw new TypeError(`${path}.${key} must be a string`) + } + } +} + +function jsonType(value: unknown): string { + if (value === null) return 'null' + if (Array.isArray(value)) return 'array' + if (typeof value === 'number') return Number.isInteger(value) ? 'integer' : 'number' + return typeof value +} + +function typeMatches(value: unknown, type: string): boolean { + const actual = jsonType(value) + return actual === type || (type === 'number' && actual === 'integer') +} + +function sameJson(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right) +} + +/** + * Every way `value` breaks `schema`, as `$.path: problem` strings. Empty means + * the value conforms. The schema must already have passed `assertAnswerSchema`. + */ +export function answerSchemaErrors(value: unknown, schema: AnswerSchema, path = '$'): string[] { + const errors: string[] = [] + if (schema.type !== undefined) { + const types = (Array.isArray(schema.type) ? schema.type : [schema.type]) as string[] + if (!types.some((type) => typeMatches(value, type))) { + errors.push(`${path}: expected ${types.join(' or ')}, got ${jsonType(value)}`) + return errors + } + } + if (schema.const !== undefined && !sameJson(value, schema.const)) { + errors.push(`${path}: expected ${JSON.stringify(schema.const)}`) + } + if (Array.isArray(schema.enum) && !schema.enum.some((option) => sameJson(value, option))) { + errors.push(`${path}: expected one of ${schema.enum.map((option) => JSON.stringify(option)).join(', ')}`) + } + if (isRecord(value)) { + const properties = isRecord(schema.properties) ? schema.properties : {} + for (const name of (schema.required as string[] | undefined) ?? []) { + if (!(name in value)) errors.push(`${path}.${name}: required property is missing`) + } + for (const [name, child] of Object.entries(value)) { + const childSchema = properties[name] + if (childSchema !== undefined) { + errors.push(...answerSchemaErrors(child, childSchema as AnswerSchema, `${path}.${name}`)) + } else if (schema.additionalProperties === false) { + errors.push(`${path}.${name}: property is not allowed`) + } + } + } + if (Array.isArray(value) && schema.items !== undefined) { + value.forEach((entry, index) => { + errors.push(...answerSchemaErrors(entry, schema.items as AnswerSchema, `${path}[${index}]`)) + }) + } + return errors +} + +/** + * Read one JSON value from a model's answer text. The whole text is tried + * first; a single fenced block is accepted because models wrap JSON in one + * even when told not to. Anything else is a parse failure, not a guess. + */ +export function parseJsonAnswer(text: string): { ok: true; value: unknown } | { ok: false; error: string } { + const trimmed = text.trim() + const candidates = [trimmed] + const fenced = /^```(?:json)?\s*\n([\s\S]*?)\n```$/.exec(trimmed) + if (fenced) candidates.push(fenced[1]!.trim()) + for (const candidate of candidates) { + try { + return { ok: true, value: JSON.parse(candidate) as unknown } + } catch { + // Try the next candidate. + } + } + return { ok: false, error: 'answer is not one JSON value' } +} diff --git a/src/ask.ts b/src/ask.ts new file mode 100644 index 0000000..0f85d04 --- /dev/null +++ b/src/ask.ts @@ -0,0 +1,762 @@ +/** + * `traces ask`: free-form questions over trace sessions, answered by a + * recursive trace-analysis engine. + * + * Each question runs through agent-eval's `runTraceAnalyst` directly instead + * of the analyst registry. The registry keeps only findings, so it discards + * the engine's prose answer, and it runs analysts one at a time. Here the + * questions run concurrently under one shared cost ledger, so a single budget + * bounds all of them, and every `trace://` citation in an answer is checked + * against the store before the answer counts as answered. + */ + +import { mkdir, readFile, writeFile } from 'node:fs/promises' +import { join, resolve } from 'node:path' +import { performance } from 'node:perf_hooks' +import { CostCeilingReachedError, CostLedger, type CostProvenance } from '@tangle-network/agent-eval' +import { + type AnalystUsageReceipt, + defineTraceAnalyst, + type RawAnalystFinding, + runTraceAnalyst, + type TraceAnalysisEngine, + type TraceAnalysisEngineResult, + type TraceAnalystDefinition, + type TraceAnalystLimits, +} from '@tangle-network/agent-eval/analyst' +import type { TraceAnalysisStore } from '@tangle-network/agent-eval/traces' +import { openAgenticTraceStore, writeAnalysisTraceFile } from './analysis-store.js' +import { type AnswerSchema, answerSchemaErrors, assertAnswerSchema, parseJsonAnswer } from './answer-schema.js' +import { indexSessionIdsByTrace } from './attributes.js' +import { + createFindingRejectionTally, + type FindingRejectionReasons, + formatFindingRejections, + totalFindingRejections, +} from './finding-rejections.js' +import type { OtlpSpan } from './otlp.js' +import { sessionFactsContext } from './session-facts.js' + +/** One question to ask of the selected traces. */ +export interface TraceQuestion { + /** Stable ID for the report, the JSON result, and cost attribution. Default: `q`. */ + readonly id?: string + readonly question: string + /** Extra guidance for this question only, placed after the shared rules. */ + readonly instructions?: string + /** + * JSON Schema the answer must satisfy (see `answer-schema.ts` for the + * supported subset). With a schema the answer must be one JSON value. + */ + readonly answerSchema?: AnswerSchema +} + +/** Why a question did not produce a checked answer. */ +export type TraceQuestionFailureKind = + | 'error' + | 'aborted' + | 'budget-refused' + | 'no-answer' + | 'invalid-answer' + | 'unresolved-citations' + +export interface TraceQuestionCitation { + readonly uri: string + readonly traceId: string | null + readonly spanId: string | null + /** True only when the store holds the cited span. */ + readonly resolved: boolean +} + +export interface TraceQuestionAnswer { + readonly id: string + readonly question: string + readonly status: 'answered' | 'failed' + readonly failure?: { readonly kind: TraceQuestionFailureKind; readonly message: string } + /** The engine's prose answer, verbatim. Null when the engine returned none. */ + readonly answer: string | null + /** The answer parsed as JSON, present when the question had an answer schema and it parsed. */ + readonly parsedAnswer?: unknown + /** Every `trace://` URI in the answer text, with whether it resolved. */ + readonly citations: readonly TraceQuestionCitation[] + /** Findings the evidence gate accepted. */ + readonly findings: readonly RawAnalystFinding[] + /** Findings the evidence gate refused, by reason. */ + readonly rejectedFindings: FindingRejectionReasons + readonly model: string | null + /** Successful model completions the engine reported; null when the engine failed. */ + readonly modelCalls: number | null + /** Trace-tool requests the engine reported; null when the engine failed. */ + readonly toolCalls: number | null + /** Provider usage and cost for this question alone, from the shared ledger. */ + readonly usage: AnalystUsageReceipt | null + readonly startedAt: string + readonly endedAt: string + readonly latencyMs: number + /** Engine-native steps, retained for audit. */ + readonly trajectory?: readonly unknown[] +} + +/** One trace the questions could read. */ +export interface TraceQuestionTrace { + readonly traceId: string + readonly sessionId: string | null + readonly spanCount: number + readonly startTime: string | null + readonly endTime: string | null + readonly rootSpan: string | null +} + +export interface TraceQuestionsTotals { + readonly questions: number + readonly answered: number + readonly failed: number + /** Null when at least one question's count is unknown. */ + readonly modelCalls: number | null + /** Null when at least one question's count is unknown. */ + readonly toolCalls: number | null + /** Paid provider calls the shared ledger recorded, including failed ones. */ + readonly providerCalls: number + /** Total spend with its provenance; `uncaptured` carries a null amount, never 0. */ + readonly cost: CostProvenance + /** Wall time of the whole run. */ + readonly wallTimeMs: number + /** Sum of the questions' own latencies; above `wallTimeMs` when questions overlapped. */ + readonly questionTimeMs: number + /** Most questions observed running at the same time. */ + readonly peakConcurrency: number +} + +export interface TraceQuestionsResult { + readonly schemaVersion: 1 + readonly kind: 'traces.ask' + readonly generatedAt: string + readonly harness: string + readonly engine: { readonly id: string; readonly version: string; readonly model: string | null } + readonly concurrency: number + /** Shared ceiling across every question; null when uncapped. */ + readonly budgetUsd: number | null + /** The engine's own ceiling for one question, when it declares one. */ + readonly questionBudgetUsd: number | null + readonly spanCount: number + readonly otlpPath: string + readonly traces: readonly TraceQuestionTrace[] + readonly questions: readonly TraceQuestionAnswer[] + readonly totals: TraceQuestionsTotals + readonly warnings: readonly string[] + /** True when every question produced an answer whose citations all resolved. */ + readonly ok: boolean + readonly report: string +} + +export interface TraceQuestionsOptions { + readonly questions: readonly TraceQuestion[] + readonly spans: readonly OtlpSpan[] + readonly engine: TraceAnalysisEngine + readonly harness?: string + /** Questions running at once. Default 4. */ + readonly concurrency?: number + /** Shared USD ceiling across every question. Omit for no shared ceiling. */ + readonly budgetUsd?: number + readonly limits?: Partial + /** Explicit full-bundle source access for `readSpanSource`. */ + readonly sourceBundle?: { path: string; maxRecordBytes?: number } + /** Where to write the OpenInference file the engine reads. Default: a temp file. */ + readonly otlpOutPath?: string + readonly generatedAt?: string + readonly signal?: AbortSignal + readonly log?: (msg: string, fields?: Record) => void +} + +export interface TraceQuestionsArtifacts { + readonly directory: string + readonly result: string + readonly report: string + readonly traces: string +} + +export const DEFAULT_ASK_CONCURRENCY = 4 + +/** Bumped whenever the question layout or rules change. */ +const ASK_DEFINITION_VERSION = '1.0.0' + +/** + * DSPy's RLM shows the model a preview of each input: a value longer than + * 1,000 characters appears as its first and last 500 only. The question + * therefore stays short and whole, and the rules sit in the first 500 + * characters of the instructions, which is all of them the preview keeps. + */ +const DSPY_PREVIEW_HEAD_CHARS = 500 +const QUESTION_FIELD_MAX_CHARS = 900 + +const ASK_QUESTION_HEADER = + 'Answer QUESTION with the trace tools and follow TRACES ASK RULES in analyst_instructions. ' + + 'The trace list is JSON after PREPARED CONTEXT: there. Cite trace:///span/.' + +const QUESTION_LABEL = '\n\nQUESTION: ' + +/** Longest question that keeps the whole question field inside the preview. */ +export const MAX_TRACE_QUESTION_CHARS = QUESTION_FIELD_MAX_CHARS - ASK_QUESTION_HEADER.length - QUESTION_LABEL.length + +const ASK_RULES = [ + 'TRACES ASK RULES', + '1. Answer QUESTION only from trace tool results retrieved in this run.', + '2. PREPARED CONTEXT ends analyst_instructions: the trace list JSON, then a SESSION FACTS sheet. Parse both; copy IDs from them.', + '3. Cite each fact as trace:///span/: span ids the sheet names, never the sheet.', + '4. Quote excerpts from viewSpans output, never searchTrace hits.', + '5. If the trace does not record a fact, say "not in trace".', +].join('\n') + +const ANSWER_SCHEMA_RULE = '6. The answer is one JSON value matching ANSWER SCHEMA below, with no other text.' + +/** Traces listed in the prepared context; the rest are counted, not dropped silently. */ +const MAX_CONTEXT_TRACES = 200 + +const QUESTION_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/ + +const QUESTION_KEYS = new Set(['id', 'question', 'instructions', 'answerSchema']) + +/** + * Validate questions and assign default IDs. Throws on an empty list, a + * duplicate or malformed ID, a question too long for the preview, or an answer + * schema outside the supported subset. + */ +export function normalizeTraceQuestions(questions: readonly TraceQuestion[]): Array { + if (questions.length === 0) throw new Error('ask needs at least one question') + const seen = new Set() + return questions.map((entry, index) => { + const id = entry.id ?? `q${index + 1}` + if (!QUESTION_ID.test(id)) { + throw new Error(`question ID "${id}" must match ${QUESTION_ID} (letters, digits, dot, underscore, hyphen)`) + } + if (seen.has(id)) throw new Error(`duplicate question ID "${id}"`) + seen.add(id) + const question = typeof entry.question === 'string' ? entry.question.trim() : '' + if (!question) throw new Error(`question ${id} is empty`) + if (question.length > MAX_TRACE_QUESTION_CHARS) { + throw new Error( + `question ${id} has ${question.length} characters; the limit is ${MAX_TRACE_QUESTION_CHARS} so the ` + + 'model sees it whole. Move detail into the entry\'s "instructions" field.', + ) + } + if (entry.instructions !== undefined && typeof entry.instructions !== 'string') { + throw new Error(`question ${id}: instructions must be a string`) + } + if (entry.answerSchema !== undefined) assertAnswerSchema(entry.answerSchema, `question ${id} answerSchema`) + return { + id, + question, + ...(entry.instructions?.trim() ? { instructions: entry.instructions.trim() } : {}), + ...(entry.answerSchema !== undefined ? { answerSchema: entry.answerSchema } : {}), + } + }) +} + +/** + * Read questions from a JSON file: an array, or an object with a `questions` + * array. Each entry is a question string or an object with `question` and + * optional `id`, `instructions`, and `answerSchema`. Unknown keys are + * rejected so a misspelled field cannot be ignored silently. + */ +export async function loadTraceQuestionsFile(path: string): Promise { + let parsed: unknown + try { + parsed = JSON.parse(await readFile(path, 'utf8')) + } catch (error) { + throw new Error(`questions file ${path} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`) + } + const list = Array.isArray(parsed) + ? parsed + : typeof parsed === 'object' && parsed !== null && Array.isArray((parsed as { questions?: unknown }).questions) + ? (parsed as { questions: unknown[] }).questions + : undefined + if (!list) throw new Error(`questions file ${path} must hold a JSON array or an object with a "questions" array`) + return list.map((entry, index) => { + if (typeof entry === 'string') return { question: entry } + if (typeof entry !== 'object' || entry === null || Array.isArray(entry)) { + throw new Error(`questions file ${path}: entry ${index + 1} must be a string or an object`) + } + for (const key of Object.keys(entry)) { + if (!QUESTION_KEYS.has(key)) { + throw new Error(`questions file ${path}: entry ${index + 1} has unknown key "${key}"`) + } + } + const record = entry as Record + if (typeof record.question !== 'string') { + throw new Error(`questions file ${path}: entry ${index + 1} needs a "question" string`) + } + if (record.id !== undefined && typeof record.id !== 'string') { + throw new Error(`questions file ${path}: entry ${index + 1} "id" must be a string`) + } + return { + question: record.question, + ...(record.id !== undefined ? { id: record.id as string } : {}), + ...(record.instructions !== undefined ? { instructions: record.instructions as string } : {}), + ...(record.answerSchema !== undefined ? { answerSchema: record.answerSchema as AnswerSchema } : {}), + } + }) +} + +/** The engine's `question` input: a short header, then the question, whole. */ +export function renderTraceQuestionPrompt(question: string): string { + return `${ASK_QUESTION_HEADER}${QUESTION_LABEL}${question}` +} + +/** The definition's instructions: rules first, so the preview keeps them. */ +export function renderTraceQuestionInstructions(question: Pick): string { + const rules = question.answerSchema ? `${ASK_RULES}\n${ANSWER_SCHEMA_RULE}` : ASK_RULES + return [ + rules, + question.instructions ? `QUESTION GUIDANCE:\n${question.instructions}` : '', + question.answerSchema ? `ANSWER SCHEMA:\n${JSON.stringify(question.answerSchema)}` : '', + ].filter(Boolean).join('\n\n') +} + +/** Characters of the instructions the DSPy preview keeps from the start; the rules must fit. */ +export const TRACE_QUESTION_PREVIEW_HEAD_CHARS = DSPY_PREVIEW_HEAD_CHARS + +function describeTraces(spans: readonly OtlpSpan[]): TraceQuestionTrace[] { + const { sessionByTrace } = indexSessionIdsByTrace(spans) + const byTrace = new Map() + for (const span of spans) { + const entry = byTrace.get(span.trace_id) ?? { count: 0, start: Infinity, end: -Infinity, root: null } + entry.count += 1 + const start = Date.parse(span.start_time) + const end = Date.parse(span.end_time) + if (Number.isFinite(start)) entry.start = Math.min(entry.start, start) + if (Number.isFinite(end)) entry.end = Math.max(entry.end, end) + if (span.parent_span_id === null && entry.root === null) entry.root = span.name + byTrace.set(span.trace_id, entry) + } + return [...byTrace].map(([traceId, entry]) => ({ + traceId, + sessionId: sessionByTrace.get(traceId) ?? null, + spanCount: entry.count, + startTime: Number.isFinite(entry.start) ? new Date(entry.start).toISOString() : null, + endTime: Number.isFinite(entry.end) ? new Date(entry.end).toISOString() : null, + rootSpan: entry.root, + })) +} + +/** + * What every question receives before its first model call: the trace list, + * then the deterministic session-facts sheet. + * + * Both are free. The list lets the model copy trace IDs instead of guessing + * them; the sheet answers the counting questions the bounded trace tools cannot + * (`viewTrace` degrades to a 20-entry histogram above 150,000 bytes, and + * `searchTrace` stops at 500 hits). Neither is citable: the sheet names the span + * ids behind every fact, and a citation still has to resolve against the store. + */ +function preparedContext(traces: readonly TraceQuestionTrace[], spans: readonly OtlpSpan[]): string { + const list = JSON.stringify({ + traces: traces.slice(0, MAX_CONTEXT_TRACES).map((trace) => ({ + trace_id: trace.traceId, + session_id: trace.sessionId, + spans: trace.spanCount, + start: trace.startTime, + end: trace.endTime, + root: trace.rootSpan, + })), + omitted_traces: Math.max(0, traces.length - MAX_CONTEXT_TRACES), + }) + const sheet = sessionFactsContext(spans) + return sheet ? `${list}\n\n${sheet}` : list +} + +const TRACE_URI = /trace:\/\/[^\s/"'`<>()[\]{}]+\/span\/[^\s/"'`<>()[\]{},;]+/g + +/** Every distinct `trace:///span/` URI in the text, in order. */ +export function traceCitationsInText(text: string): Array<{ uri: string; traceId: string | null; spanId: string | null }> { + const seen = new Set() + const out: Array<{ uri: string; traceId: string | null; spanId: string | null }> = [] + for (const match of text.matchAll(TRACE_URI)) { + // Sentence punctuation after a URI is prose, not part of the ID. + const uri = match[0].replace(/[.:!?]+$/, '') + if (seen.has(uri)) continue + seen.add(uri) + const parts = /^trace:\/\/([^/]+)\/span\/([^/]+)$/.exec(uri) + let traceId: string | null = null + let spanId: string | null = null + if (parts) { + try { + traceId = decodeURIComponent(parts[1]!) + spanId = decodeURIComponent(parts[2]!) + } catch { + traceId = null + spanId = null + } + } + out.push({ uri, traceId, spanId }) + } + return out +} + +async function verifyCitations( + text: string, + store: TraceAnalysisStore, + signal: AbortSignal | undefined, +): Promise { + const citations = traceCitationsInText(text) + const wanted = new Map>() + for (const citation of citations) { + if (!citation.traceId || !citation.spanId) continue + const spans = wanted.get(citation.traceId) ?? new Set() + spans.add(citation.spanId) + wanted.set(citation.traceId, spans) + } + const found = new Map>() + for (const [traceId, spanIds] of wanted) { + const existing = await store.hasSpans({ trace_id: traceId, span_ids: [...spanIds] }, signal ? { signal } : undefined) + found.set(traceId, new Set(existing)) + } + return citations.map((citation) => ({ + ...citation, + resolved: Boolean(citation.traceId && citation.spanId && found.get(citation.traceId)?.has(citation.spanId)), + })) +} + +function isBudgetRefusal(error: unknown): boolean { + if (error instanceof CostCeilingReachedError) return true + const message = error instanceof Error ? error.message : String(error) + // The DSPy engine crosses a process boundary, so the class does not survive; + // these are the ledger's and the model proxy's own refusal texts. + return /would exceed ceiling|model cost limit reached/.test(message) +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? `${error.constructor.name}: ${error.message}` : String(error) +} + +/** + * Smallest reservation one model call makes before it runs: its full output and + * reasoning allowance at the output rate, before any input tokens. Undefined + * when the engine does not describe its pricing and token caps. + */ +export function engineCallReservationFloorUsd(engine: TraceAnalysisEngine): number | undefined { + const config = engine.executionConfig + const pricing = config.pricing as { outputUsdPerMillion?: unknown } | undefined + const outputRate = pricing?.outputUsdPerMillion + const maxOutput = config.max_output_tokens + const maxReasoning = config.max_reasoning_tokens ?? 0 + if (typeof outputRate !== 'number' || typeof maxOutput !== 'number' || typeof maxReasoning !== 'number') { + return undefined + } + return ((maxOutput + maxReasoning) * outputRate) / 1_000_000 +} + +function formatUsd(value: number): string { + return `$${value < 0.01 && value > 0 ? value.toPrecision(2) : value.toFixed(2)}` +} + +function sumOrNull(values: ReadonlyArray): number | null { + return values.some((value) => value === null) ? null : values.reduce((sum, value) => sum + (value ?? 0), 0) +} + +/** + * Ask every question of the spans, concurrently, under one shared cost + * ledger. Never throws for a failed question: each failure is recorded on its + * answer and `ok` turns false. Throws before any model call for invalid + * questions, options, or a budget below one call's reservation. + */ +export async function runTraceQuestions(opts: TraceQuestionsOptions): Promise { + if (opts.spans.length === 0) throw new Error('runTraceQuestions: no spans to ask about') + const questions = normalizeTraceQuestions(opts.questions) + const concurrency = opts.concurrency ?? DEFAULT_ASK_CONCURRENCY + if (!Number.isSafeInteger(concurrency) || concurrency < 1) throw new RangeError('concurrency must be an integer >= 1') + const budgetUsd = opts.budgetUsd + if (budgetUsd !== undefined && (!Number.isFinite(budgetUsd) || budgetUsd <= 0)) { + throw new RangeError('budgetUsd must be a positive number') + } + const warnings: string[] = [] + const floor = engineCallReservationFloorUsd(opts.engine) + const effectiveConcurrency = Math.min(concurrency, questions.length) + if (budgetUsd !== undefined && floor !== undefined) { + if (budgetUsd < floor) { + throw new RangeError( + `budget ${formatUsd(budgetUsd)} is below one model call's reservation of at least ${formatUsd(floor)} ` + + `for ${opts.engine.model ?? opts.engine.id}; every question would be refused before it ran`, + ) + } + if (budgetUsd < floor * effectiveConcurrency) { + warnings.push( + `budget ${formatUsd(budgetUsd)} covers at most ${Math.floor(budgetUsd / floor)} concurrent model call(s) ` + + `at ${formatUsd(floor)} reserved each; questions wait for reservations, and later calls are refused ` + + 'once settled spend leaves less than one reservation', + ) + } + } + opts.signal?.throwIfAborted() + + const generatedAt = opts.generatedAt ?? new Date().toISOString() + const traceFile = await writeAnalysisTraceFile(opts.spans, { + sourceBundle: opts.sourceBundle, + otlpOutPath: opts.otlpOutPath, + signal: opts.signal, + }) + const store = await openAgenticTraceStore(traceFile) + const traces = describeTraces(opts.spans) + const context = preparedContext(traces, opts.spans) + const runId = `traces-ask-${Date.parse(generatedAt) || Date.now()}` + const ledger = new CostLedger(budgetUsd) + + let active = 0 + let peakConcurrency = 0 + const askOne = async (question: TraceQuestion & { id: string }): Promise => { + const startedAt = new Date() + const started = performance.now() + const rejections = createFindingRejectionTally(question.id) + let usage: AnalystUsageReceipt | null = null + let completed: TraceAnalysisEngineResult | undefined + let failure: TraceQuestionAnswer['failure'] + const log = (msg: string, fields?: Record): void => { + rejections.record(msg, fields) + opts.log?.(`[${question.id}] ${msg}`, fields) + } + const definition: TraceAnalystDefinition = defineTraceAnalyst({ + // Namespaced so a question ID can never select a built-in kind's subject rules. + id: `ask.${question.id}`, + description: `traces ask question ${question.id}`, + area: 'question', + version: ASK_DEFINITION_VERSION, + question: renderTraceQuestionPrompt(question.question), + instructions: renderTraceQuestionInstructions(question), + toolGroup: 'all', + prepareContext: () => context, + ...(opts.limits ? { limits: opts.limits } : {}), + }) + if (opts.signal?.aborted) { + failure = { kind: 'aborted', message: 'run aborted before the question started' } + } else { + active += 1 + peakConcurrency = Math.max(peakConcurrency, active) + try { + completed = await runTraceAnalyst({ + definition, + engine: opts.engine, + store, + context: { + runId, + correlationId: `${runId}:${question.id}`, + costLedger: ledger, + costPhase: 'trace-question', + log, + recordUsage: (receipt) => { + usage = receipt + }, + ...(opts.signal ? { signal: opts.signal } : {}), + }, + }) + } catch (error) { + failure = { + kind: opts.signal?.aborted ? 'aborted' : isBudgetRefusal(error) ? 'budget-refused' : 'error', + message: errorMessage(error), + } + } finally { + active -= 1 + } + } + + const answer = completed && completed.answer.trim() ? completed.answer : null + let parsedAnswer: unknown + let hasParsedAnswer = false + let citations: TraceQuestionCitation[] = [] + if (answer !== null) { + citations = await verifyCitations(answer, store, opts.signal) + if (question.answerSchema) { + const parsed = parseJsonAnswer(answer) + const problems = parsed.ok ? answerSchemaErrors(parsed.value, question.answerSchema) : [parsed.error] + if (parsed.ok) { + parsedAnswer = parsed.value + hasParsedAnswer = true + } + if (problems.length > 0 && !failure) { + failure = { kind: 'invalid-answer', message: problems.slice(0, 5).join('; ') } + } + } + const unresolved = citations.filter((citation) => !citation.resolved) + if (unresolved.length > 0 && !failure) { + failure = { + kind: 'unresolved-citations', + message: `${unresolved.length} cited span(s) do not exist: ${unresolved.slice(0, 3).map((c) => c.uri).join(', ')}`, + } + } + } else if (completed && !failure) { + failure = { kind: 'no-answer', message: 'the engine returned an empty answer' } + } + const endedAt = new Date() + return { + id: question.id, + question: question.question, + status: failure ? 'failed' : 'answered', + ...(failure ? { failure } : {}), + answer, + ...(hasParsedAnswer ? { parsedAnswer } : {}), + citations, + findings: completed?.findings ?? [], + rejectedFindings: rejections.counts()[question.id] ?? {}, + model: opts.engine.model ?? null, + modelCalls: completed?.modelCalls ?? null, + toolCalls: completed?.toolCalls ?? null, + usage, + startedAt: startedAt.toISOString(), + endedAt: endedAt.toISOString(), + latencyMs: Math.round(performance.now() - started), + ...(completed ? { trajectory: completed.trajectory } : {}), + } + } + + // A worker pool, not Promise.all: at most `concurrency` engines (each a + // Python process and a model proxy) exist at once. Unlike the import pool, + // one failed question never stops the others; its failure is its answer. + const answers = new Array(questions.length) + let next = 0 + const runStarted = performance.now() + await Promise.all(Array.from({ length: effectiveConcurrency }, async () => { + while (next < questions.length) { + const index = next + next += 1 + answers[index] = await askOne(questions[index]!) + } + })) + const wallTimeMs = Math.round(performance.now() - runStarted) + + const summary = ledger.summary({ channel: 'analyst' }) + const answered = answers.filter((answer) => answer.status === 'answered').length + const totals: TraceQuestionsTotals = { + questions: answers.length, + answered, + failed: answers.length - answered, + modelCalls: sumOrNull(answers.map((answer) => answer.modelCalls)), + toolCalls: sumOrNull(answers.map((answer) => answer.toolCalls)), + providerCalls: summary.totalCalls + summary.pendingCalls, + cost: summary.costProvenance, + wallTimeMs, + questionTimeMs: answers.reduce((sum, answer) => sum + answer.latencyMs, 0), + peakConcurrency, + } + const partial: Omit = { + schemaVersion: 1, + kind: 'traces.ask', + generatedAt, + harness: opts.harness ?? 'unknown', + engine: { id: opts.engine.id, version: opts.engine.version, model: opts.engine.model ?? null }, + concurrency, + budgetUsd: budgetUsd ?? null, + questionBudgetUsd: typeof opts.engine.executionConfig.max_cost_usd === 'number' + ? opts.engine.executionConfig.max_cost_usd + : null, + spanCount: opts.spans.length, + otlpPath: traceFile.otlpPath, + traces, + questions: answers, + totals, + warnings, + ok: answered === answers.length, + } + return { ...partial, report: renderTraceQuestionsReport(partial) } +} + +function costText(cost: CostProvenance | undefined | null): string { + if (!cost) return 'not captured' + return cost.kind === 'uncaptured' ? 'uncaptured' : `${formatUsd(cost.usd)} ${cost.kind}` +} + +function seconds(ms: number): string { + return `${(ms / 1000).toFixed(1)} s` +} + +function cell(value: string): string { + return value.replace(/\|/g, '\\|').replace(/\s+/g, ' ').trim() +} + +function countText(value: number | null): string { + return value === null ? 'unknown' : String(value) +} + +/** Readable Markdown for a result; the JSON result carries every field. */ +export function renderTraceQuestionsReport(result: Omit): string { + const { totals } = result + const lines = ['# traces ask', ''] + lines.push( + `${totals.questions} question(s) over ${result.traces.length} trace(s) (${result.spanCount} spans, ${result.harness}). ` + + `Engine \`${result.engine.id}\`${result.engine.model ? `, model \`${result.engine.model}\`` : ''}; ` + + `concurrency ${result.concurrency}; budget ${result.budgetUsd === null ? 'uncapped' : `${formatUsd(result.budgetUsd)} shared`}` + + `${result.questionBudgetUsd === null ? '' : `, ${formatUsd(result.questionBudgetUsd)} per question`}.`, + ) + lines.push('') + lines.push( + `**${totals.answered} answered, ${totals.failed} failed.** Wall time ${seconds(totals.wallTimeMs)} ` + + `against ${seconds(totals.questionTimeMs)} of question time (peak ${totals.peakConcurrency} at once). ` + + `Cost ${costText(totals.cost)} over ${totals.providerCalls} provider call(s); ` + + `${countText(totals.modelCalls)} model call(s), ${countText(totals.toolCalls)} tool call(s).`, + ) + lines.push('') + for (const warning of result.warnings) { + lines.push(`> Warning: ${warning}`) + lines.push('') + } + lines.push('| ID | Status | Citations | Rejected findings | Model calls | Tool calls | Cost | Time |') + lines.push('|---|---|---|---|---|---|---|---|') + for (const answer of result.questions) { + const resolved = answer.citations.filter((citation) => citation.resolved).length + lines.push( + `| \`${cell(answer.id)}\` | ${answer.status === 'answered' ? 'answered' : `failed: ${answer.failure?.kind}`} | ` + + `${resolved}/${answer.citations.length} resolved | ${totalFindingRejections(answer.rejectedFindings)} | ` + + `${countText(answer.modelCalls)} | ${countText(answer.toolCalls)} | ${costText(answer.usage?.cost)} | ` + + `${seconds(answer.latencyMs)} |`, + ) + } + lines.push('') + for (const answer of result.questions) { + lines.push(`## ${answer.id}: ${answer.question}`) + lines.push('') + if (answer.failure) { + lines.push(`**Failed (${answer.failure.kind}):** ${answer.failure.message.trim()}`) + lines.push('') + } + if (answer.answer !== null) { + lines.push(answer.answer.trim()) + lines.push('') + } + const notes: string[] = [] + const unresolved = answer.citations.filter((citation) => !citation.resolved) + if (answer.citations.length > 0) { + notes.push( + `- **Citations:** ${answer.citations.length - unresolved.length} of ${answer.citations.length} resolve to a span in the trace.`, + ) + for (const citation of unresolved) notes.push(`- **Unresolved:** ${citation.uri}`) + } else if (answer.answer !== null) { + notes.push('- **Citations:** none in the answer text.') + } + if (answer.findings.length > 0) { + notes.push(`- **Accepted findings:** ${answer.findings.length}`) + for (const finding of answer.findings.slice(0, 5)) notes.push(` - ${finding.severity}: ${finding.claim}`) + } + const rejected = formatFindingRejections(answer.rejectedFindings) + if (rejected) notes.push(`- **Rejected by the evidence gate:** ${rejected}`) + if (notes.length > 0) lines.push(...notes, '') + } + return lines.join('\n') +} + +/** Write `answers.json` and `report.md` next to the trace file the engine read. */ +export async function writeTraceQuestionsArtifacts( + result: TraceQuestionsResult, + outDir: string, +): Promise { + const directory = resolve(outDir) + await mkdir(directory, { recursive: true }) + const paths: TraceQuestionsArtifacts = { + directory, + result: join(directory, 'answers.json'), + report: join(directory, 'report.md'), + traces: result.otlpPath, + } + const { report, ...machine } = result + await Promise.all([ + writeFile(paths.result, `${JSON.stringify(machine, null, 2)}\n`, 'utf8'), + writeFile(paths.report, report, 'utf8'), + ]) + return paths +} diff --git a/src/cli.ts b/src/cli.ts index 6450085..2a5aae9 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -11,6 +11,8 @@ * traces analyze [--format auto] [--out report.md] * traces investigate [input.jsonl] [--format auto] [--out report.md] * traces improve [input.jsonl] [--format auto] --dir .traces/improvement + * traces ask --harness codex --session --question "..." [--questions q.json] [--dir ] + * traces facts [--harness codex] [--last 5] [--format json|text] [--out facts.json] * traces convert [--harness claude-code] [--last 1] --otlp-out spans.jsonl * traces index [--harness claude-code] [--last 20] --out session-index.json * traces bundle --harness claude-code --session --out @@ -42,7 +44,8 @@ import { assertOutsideSourceBundle } from './bundle-source.js' import { readFileSync } from 'node:fs' -import { readdir, readFile, stat, writeFile } from 'node:fs/promises' +import { mkdir, mkdtemp, readdir, readFile, stat, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' import { basename, join, resolve } from 'node:path' import { appendAll } from './arrays.js' import { indexSessionIdsByTrace } from './attributes.js' @@ -58,7 +61,8 @@ import { type VerifyFindingsRun, } from './analyze-verify.js' import { parseCorpusFlag } from './replay-corpus.js' -import { commandAnalyzer, commandRedactor, haloAnalyzer } from './external.js' +import { commandAnalyzer, commandRedactor, externalFailureMessage, haloAnalyzer } from './external.js' +import { findingRejectionDetail } from './finding-rejections.js' import { hodoscopeAnalyzer } from './hodoscope.js' import { primeAnalyzer } from './analyst-engine-prime.js' import { type TraceEvidenceFormatOption, exportTraceEvidenceFile, writeTraceEvidenceExportFile } from './file-export.js' @@ -66,6 +70,7 @@ import { inspectSessionIndex, readSessionIndexFile, renderInspectionReport, writ import { loadTracesConfig, mergeTracesConfig, + isBridgeMismatchError, runTraceImprovement, runTraceInvestigation, saveReport, @@ -89,8 +94,17 @@ import { } from '@tangle-network/agent-eval/supervisor-run' import { fileRunContextSupervisorRunReader, isFileRunContextDir } from './supervisor-run-context.js' import { resolveRunWatchTarget, watchRunTarget } from './run-watch.js' -import { createDspyRlmTraceEngine, type TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' -import { analystMaxOutputTokens, createAnalystModelOwner } from './analyst-model-call.js' +import type { TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' +import { analysisEngineFromEnv, DEFAULT_ANALYST_MODEL, DEFAULT_QUESTION_MAX_COST_USD } from './analyst-model-call.js' +import { + loadTraceQuestionsFile, + MAX_TRACE_QUESTION_CHARS, + normalizeTraceQuestions, + runTraceQuestions, + type TraceQuestion, + type TraceQuestionsResult, + writeTraceQuestionsArtifacts, +} from './ask.js' import type { OtlpSpan } from './otlp.js' import { serializeSpans, writeOtlpFile } from './otlp.js' import type { @@ -101,6 +115,7 @@ import type { UnreadableSourceRows, } from './otlp-input.js' import { readOtlpInput } from './otlp-input.js' +import { buildSessionFactsReport, renderSessionFacts } from './session-facts.js' import { renderValidation, validationExitCode } from './conformance.js' import type { TraceValidation } from '@tangle-network/agent-trace-contract' import { watchSessions } from './observer.js' @@ -179,13 +194,14 @@ interface Args { replayCorpora: string[] /** Receipt root for --verify-findings; defaults next to --out. */ verifyOut?: string + /** ask: repeatable `--question` texts. */ + questions: string[] + /** ask: JSON file of questions, with optional IDs, guidance, and answer schemas. */ + questionsFile?: string + /** ask: provider ceiling for one question; `--budget` bounds all of them together. */ + questionBudget?: number } -const DEFAULT_ANALYST_MODEL = 'gpt-5.6-luna' - -/** Default `--llm` endpoint: the Tangle router, reached with TANGLE_API_KEY. */ -const TANGLE_ROUTER_BASE_URL = 'https://router.tangle.tools/v1' - function packageVersion(): string { const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as { version?: unknown } if (typeof pkg.version !== 'string' || !pkg.version) throw new Error('package.json is missing version') @@ -221,6 +237,7 @@ function parseArgs(argv: string[]): Args { concurrency: 4, verifyFindings: false, replayCorpora: [], + questions: [], } for (let i = 1; i < argv.length; i++) { const arg = argv[i] @@ -269,6 +286,9 @@ function parseArgs(argv: string[]): Args { case '--replay-corpus': { const v = next(); if (v) a.replayCorpora.push(v); break } case '--verify-out': a.verifyOut = next(); break case '--analyzer-prompt': a.analyzerPrompt = next(); break + case '--question': { const v = next(); if (v !== undefined) a.questions.push(v); break } + case '--questions': a.questionsFile = next(); break + case '--question-budget': a.questionBudget = Number(next()); break case '--redactor': a.redactorCmd = next(); break case '--format': a.format = next(); break case '--help': @@ -289,7 +309,7 @@ function parseArgs(argv: string[]): Args { * artifact with `--otlp-out`: one flag, one direction, no command where the * same word means read here and write there. */ -const OTLP_INPUT_COMMANDS = new Set(['analyze', 'investigate', 'improve', 'stream', 'validate']) +const OTLP_INPUT_COMMANDS = new Set(['analyze', 'facts', 'investigate', 'improve', 'ask', 'stream', 'validate']) /** * `--otlp` used to mean "write the artifact here" on every command. It now @@ -327,8 +347,10 @@ function validateOtlpSelection(raw: Args): Args { const CURRENT_SESSION_COMMANDS = new Set([ 'analyze', + 'facts', 'investigate', 'improve', + 'ask', 'convert', 'index', 'evidence', @@ -338,8 +360,10 @@ const CURRENT_SESSION_COMMANDS = new Set([ const WORKFLOW_COMMANDS = new Set([ 'analyze', + 'facts', 'investigate', 'improve', + 'ask', 'convert', 'index', 'evidence', @@ -481,68 +505,16 @@ async function resolveSelectedSession(args: Args): Promise<{ adapter: HarnessTra } /** - * The recursive analysis engine behind `--llm`. agent-eval's model-backed - * analysts run through DSPy RLM, which drives `agent-eval-rpc[dspy]` out of - * process — so `--llm` needs a Python interpreter with that extra installed, - * selectable via TRACES_PYTHON. Every deterministic command is unaffected and - * still needs neither a key nor Python. + * `--llm` engine. It forwards the whole `--budget` as the per-analyst provider + * ceiling: the registry already splits `--budget` across analysts, so this only + * stops the engine's own $1 default from cutting runs short. */ -function buildAnalysisEngine(model: string, budgetUsd?: number): TraceAnalysisEngine { - // The router is the default endpoint, so TANGLE_API_KEY alone is enough. - // OPENAI_API_KEY still works and, when it is the only key present, points at - // OpenAI directly — otherwise a plain OpenAI key would be sent to the router. - const tangleKey = process.env.TANGLE_API_KEY - const openAiKey = process.env.OPENAI_API_KEY - const apiKey = tangleKey || openAiKey - if (!apiKey) { - throw new Error( - '--llm needs a model key: TANGLE_API_KEY for the Tangle router (the default endpoint), or ' + - 'OPENAI_API_KEY for OpenAI. Set OPENAI_BASE_URL to target any other OpenAI-compatible ' + - 'gateway. Deterministic analysis needs no key.', - ) - } - const baseUrl = - process.env.OPENAI_BASE_URL || - (tangleKey ? TANGLE_ROUTER_BASE_URL : 'https://api.openai.com/v1') - const python = process.env.TRACES_PYTHON - const owner = createAnalystModelOwner({ - apiKey, - baseUrl, - model, - provider: - baseUrl === TANGLE_ROUTER_BASE_URL - ? 'tangle-router' - : baseUrl.startsWith('https://api.openai.com/') - ? 'openai' - : 'openai-compatible', - }) - const maxOutputTokens = analystMaxOutputTokens(model) - return createDspyRlmTraceEngine({ - call: owner.call, - callRef: owner.callRef, - recordExecution: (observation) => { - analystLog( - `[analyst] model call ${observation.sequence} ${observation.succeeded ? 'ok' : 'FAIL'} ${observation.model}`, - observation.succeeded ? undefined : { error: observation.error }, - ) - }, - model, - // Model-aware, not defaulted: GPT-5.6 needs less output room than models - // such as GLM, and every recursive call reserves this full amount before - // execution. An oversized reservation can reject useful later calls even - // when the run's measured spend remains well below its limit. - maxOutputTokens, - // maxCostUsd defaults to $1 per analyst — a proxy-side ceiling separate - // from --budget. With the larger token cap the per-call reservation grows - // ~4x, so that default binds before the registry's --budget allocation - // and kills analysts mid-run. --budget is the operator's spend authority - // and the registry still splits it across analysts, so forwarding it here - // only stops the engine's own default from cutting runs short. - ...(budgetUsd !== undefined && Number.isFinite(budgetUsd) && budgetUsd > 0 - ? { maxCostUsd: budgetUsd } - : {}), - ...(python ? { runner: { command: python } } : {}), - }) +function llmAnalysisEngine(args: Args, model: string): TraceAnalysisEngine { + return analysisEngineFromEnv({ model, maxCostUsd: args.budget, log: analystLog }) +} + +function analystModelFor(args: Args): string { + return args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL } /** @@ -564,17 +536,24 @@ function requiredBridgeVersion(): string { } /** - * `--llm` promises agentic findings; delivering a deterministic-only report - * with exit 0 when every agentic analyst died reads as success. Throwing after - * the report is written keeps the deterministic output AND fails loud. The - * bridge-version read happens only once total failure is established, so a - * package.json problem can never turn a successful run into exit 1. + * `--llm` promises agentic findings, and `--analyzer` promises that engine's + * output. Delivering the deterministic report with exit 0 when a requested + * analysis died reads as success. Throwing after the report is written keeps + * the deterministic output AND fails loud. The bridge-version read happens + * only once total agentic failure is established, so a package.json problem + * can never turn a successful run into exit 1. */ -function assertAgenticAnalystsRan(args: Args, agenticPerAnalyst: TraceInvestigationResult['agenticPerAnalyst']): void { - if (!args.llm) return - if (!totalAgenticFailureMessage(agenticPerAnalyst)) return - const message = totalAgenticFailureMessage(agenticPerAnalyst, { requiredBridgeVersion: requiredBridgeVersion() }) - throw new Error(message!) +function assertRequestedAnalysesRan( + args: Args, + result: Pick, +): void { + const messages: string[] = [] + if (args.llm && totalAgenticFailureMessage(result.agenticPerAnalyst)) { + messages.push(totalAgenticFailureMessage(result.agenticPerAnalyst, { requiredBridgeVersion: requiredBridgeVersion() })!) + } + const external = externalFailureMessage(result.external) + if (external) messages.push(external) + if (messages.length > 0) throw new Error(messages.join('\n')) } async function cmdList(args: Args): Promise { @@ -654,7 +633,7 @@ async function collectOtlpSpans(path: string): Promise { async function collectSpans(args: Args): Promise { if (args.sourceBundle) { - if (!['analyze', 'investigate', 'improve'].includes(args.command)) throw new Error('--source-bundle is an analysis option') + if (!['analyze', 'investigate', 'improve', 'ask'].includes(args.command)) throw new Error('--source-bundle is an analysis option') if (args.session || args.otlp || args.input || args.current || args.workflow || args.noContent || args.redactorCmd) { throw new Error('--source-bundle cannot be combined with another input, redaction, or metadata-only selection') } @@ -870,6 +849,49 @@ async function cmdBundleView(args: Args): Promise { ) } +/** + * `traces facts` — the deterministic session-facts sheet, printed. + * + * No model, no engine, no budget: the numbers come out of the spans. JSON by + * default because the sheet's consumers are programs; `--format text` prints + * the short readable form. Every fact names the span ids it came from, so any + * number here can be checked with `traces export` or a trace tool. + * + * Exits non-zero when a selected session cannot be read: `collectSpans` throws, + * and an unreadable session must never be reported as a session with no facts. + */ +async function cmdFacts(args: Args): Promise { + const format = args.format ?? 'json' + if (format !== 'json' && format !== 'text') { + throw new Error(`unknown facts format "${format}" (expected json or text)`) + } + const collected = await collectSpans(args) + if (collected.spans.length === 0) throw new Error('no spans found for the given selection') + warnIncompleteWorkflow(collected.workflow) + const report = buildSessionFactsReport(collected.spans, { harness: collected.harness }) + // A session that yielded no record spans was not read: only its root, and any + // integrity receipt for the bytes that failed to parse. Printing a sheet of + // zeros for it would state, in the sheet's own voice, that the session did + // nothing — which is the one thing the sheet must never do. + const unread = report.sessions.filter((facts) => facts.recordSpans === 0) + if (unread.length > 0) { + throw new Error( + `no records could be read from ${unread.length} selected session(s): ` + + unread + .map((facts) => `${facts.sessionId ?? facts.traceId} (${facts.unreadRecords.value ?? 0} unread record(s))`) + .join(', ') + + '. The facts sheet would state zeros the spans cannot support.', + ) + } + const rendered = format === 'json' ? `${JSON.stringify(report, null, 2)}\n` : renderSessionFacts(report) + if (args.out) { + await writeFile(args.out, rendered, 'utf8') + console.log(`session facts → ${args.out} (${report.sessions.length} session(s), $0, no model call)`) + return + } + process.stdout.write(rendered) +} + async function cmdInspect(args: Args): Promise { if (!args.input) throw new Error('inspect needs an index file; run `traces index --out session-index.json` first') const index = await readSessionIndexFile(args.input) @@ -1011,7 +1033,7 @@ async function cmdAnalyze(args: Args): Promise { } else { console.log(report) } - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) } /** @@ -1156,6 +1178,13 @@ async function collectImportedSpans(args: Args): Promise { * every engine-startup failure into an unactionable one-liner. */ function analystLog(msg: string, fields?: Record): void { + // A gate rejection names its cause only in the fields; without them the + // line says a finding was dropped but not why, which nobody can act on. + const rejection = findingRejectionDetail(msg, fields) + if (rejection) { + process.stderr.write(`${msg} — ${rejection}\n`) + return + } const error = typeof fields?.error === 'string' && fields.error ? fields.error : undefined const errorClass = typeof fields?.error_class === 'string' && fields.error_class ? `${fields.error_class}: ` : '' process.stderr.write(error ? `${msg} — ${errorClass}${error}\n` : `${msg}\n`) @@ -1181,7 +1210,7 @@ async function cmdInvestigate(args: Args): Promise { } else { console.log(result.report) } - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) } /** Conformance fields threaded from the source into the investigation options. */ @@ -1204,8 +1233,8 @@ async function investigate(args: Args, options: { loadDefaultConfig?: boolean } const config = args.config !== undefined || options.loadDefaultConfig !== false ? await loadTracesConfig(args.config) : undefined - const analystModel = args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL - const engine = args.llm ? buildAnalysisEngine(analystModel, args.budget) : undefined + const analystModel = analystModelFor(args) + const engine = args.llm ? llmAnalysisEngine(args, analystModel) : undefined return runTraceInvestigation(mergeTracesConfig({ sourceBundle: args.sourceBundle ? { path: args.sourceBundle } : undefined, spans, @@ -1230,8 +1259,8 @@ async function cmdImprove(args: Args): Promise { const { spans, harness, cwds, sources, workflow } = collected if (spans.length === 0) throw new Error('no spans found for the given selection') const config = await loadTracesConfig(args.config) - const analystModel = args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL - const engine = args.llm ? buildAnalysisEngine(analystModel, args.budget) : undefined + const analystModel = analystModelFor(args) + const engine = args.llm ? llmAnalysisEngine(args, analystModel) : undefined const result = await runTraceImprovement({ ...mergeTracesConfig({ sourceBundle: args.sourceBundle ? { path: args.sourceBundle } : undefined, @@ -1258,7 +1287,86 @@ async function cmdImprove(args: Args): Promise { `improvement artifacts → ${dir} ` + `(${result.findings.length} findings with actions and checks, OTLP: ${result.otlpPath})`, ) - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) +} + +/** + * `traces ask`: free-form questions over the selected sessions. Every question + * is its own recursive investigation; they run concurrently under one shared + * ledger, so `--budget` bounds the whole run and `--question-budget` bounds + * each question. The answers, their citation checks, and per-question cost and + * time are written before the exit code is decided, so a failed question never + * costs the others' answers. + */ +async function cmdAsk(args: Args): Promise { + if (args.out) throw new Error('ask writes a directory of artifacts; pass --dir instead of --out') + if (args.llm) throw new Error('ask always uses the model-backed engine; drop --llm') + const questions: TraceQuestion[] = [ + ...(args.questionsFile ? await loadTraceQuestionsFile(args.questionsFile) : []), + ...args.questions.map((question) => ({ question })), + ] + if (questions.length === 0) { + throw new Error('ask needs --question "" (repeatable) or --questions ') + } + // Reject a bad question before any session is parsed or process started. + normalizeTraceQuestions(questions) + if (args.questionBudget !== undefined && (!Number.isFinite(args.questionBudget) || args.questionBudget <= 0)) { + throw new Error('--question-budget must be a positive number of USD') + } + if (args.budget !== undefined && (!Number.isFinite(args.budget) || args.budget <= 0)) { + throw new Error('--budget must be a positive number of USD') + } + const collected = await collectSpans(args) + if (collected.spans.length === 0) throw new Error('no spans found for the given selection') + warnIncompleteWorkflow(collected.workflow) + const engine = analysisEngineFromEnv({ + model: analystModelFor(args), + maxCostUsd: args.questionBudget ?? Math.min(args.budget ?? Infinity, DEFAULT_QUESTION_MAX_COST_USD), + log: analystLog, + }) + const directory = resolve(args.dir ?? await mkdtemp(join(tmpdir(), 'traces-ask-'))) + await mkdir(directory, { recursive: true }) + const controller = new AbortController() + const interrupt = () => controller.abort(new Error('ask interrupted')) + process.once('SIGINT', interrupt) + let result: TraceQuestionsResult + try { + result = await runTraceQuestions({ + questions, + spans: collected.spans, + engine, + harness: collected.harness, + concurrency: args.concurrency, + ...(args.budget !== undefined ? { budgetUsd: args.budget } : {}), + ...(args.sourceBundle ? { sourceBundle: { path: args.sourceBundle } } : {}), + otlpOutPath: args.otlpOut ?? join(directory, 'traces.otlp.jsonl'), + signal: controller.signal, + log: analystLog, + }) + } finally { + process.removeListener('SIGINT', interrupt) + } + const artifacts = await writeTraceQuestionsArtifacts(result, directory) + process.stdout.write(result.report) + for (const warning of result.warnings) process.stderr.write(`warning: ${warning}\n`) + process.stderr.write( + `ask artifacts → ${artifacts.directory} (${result.totals.answered}/${result.totals.questions} answered, ` + + `wall ${(result.totals.wallTimeMs / 1000).toFixed(1)} s, answers: ${artifacts.result})\n`, + ) + if (!result.ok) { + const failed = result.questions.filter((answer) => answer.status === 'failed') + const lines = [ + `${failed.length} of ${result.questions.length} question(s) failed; the answers file holds every result.`, + ...failed.map((answer) => ` ${answer.id}: ${answer.failure?.kind}: ${answer.failure?.message.slice(0, 300)}`), + ] + if (failed.some((answer) => answer.failure?.kind === 'error' && isBridgeMismatchError(answer.failure.message))) { + lines.push( + 'hint: the DSPy bridge protocol is version-locked — the TRACES_PYTHON interpreter needs ' + + `agent-eval-rpc[dspy]==${requiredBridgeVersion()} (matching this package's @tangle-network/agent-eval).`, + ) + } + throw new Error(lines.join('\n')) + } } function summarizeFindingEvidence(finding: TraceLiveFinding): string { @@ -1615,6 +1723,17 @@ Commands: (--supervisor-run-dir reports a supervision tree instead) investigate Run typed investigation flow, including BYO config + evidence-backed actions improve Write findings, evidence, report, and canonical trace artifacts + ask Answer free-form questions over the selected sessions with the + model-backed engine: questions run concurrently under one budget, + every trace:// citation is checked, and answers.json + report.md + are written to --dir (exit 1 when any question fails) + facts Print the deterministic session-facts sheet for the selected + sessions: tool calls excluding synthesized spans, subagent spawns + with task names, human turns in order, the final message per task, + changed paths, first/last record times, and the harness token total. + No model call, no budget, $0. Every fact names the span ids it came + from; a fact the spans cannot support is null with its reason. + --format json (default) or text (exit 1 when a session cannot be read) convert Emit OTLP-JSONL only, to --otlp-out (HALO: use analyze --analyzer halo) index Emit a reusable session index JSON for later investigation bundle Assemble one session's durable evidence directory: transcript + @@ -1667,19 +1786,20 @@ Options: agent-runtime createFileRunContext journal, and OTLP span files. --since upload: window, 30m / 2h / 7d or an ISO date (default 24h); analyze: ISO cutoff --out Write report to a file - --dir improve: write artifacts to this directory + --dir improve/ask: write artifacts to this directory --otlp READ OTLP-JSONL emitted by any system, skipping the adapters. A directory reads the OTLP files under it — only the otlp/ subdirectory when the producer made one — and names the JSONL that is not OTLP instead of reading it as broken spans. - Supported by: validate, analyze, investigate, improve, stream. + Supported by: validate, analyze, investigate, improve, ask, stream. On a WRITING command it is the deprecated spelling of --otlp-out; it still works, with a warning, until 0.12. --source-bundle Analyze a retained full bundle; explicitly grant source-field reads. - Available for analyze, investigate, and improve. + Available for analyze, investigate, improve, and ask. --otlp-out WRITE the OTLP-JSONL artifact here (also evidence provenance / dry-run upload preview) --format analyze/export: auto | policy-evidence | sandbox-events | openinference | intelligence-spans | chat-trajectory + facts: json (default) | text --metadata analyze/export file: attach JSON object fields as span attributes --attr analyze/export file: attach one span attribute (repeatable) --mode stream: visualizer | findings | agent (default visualizer) @@ -1696,10 +1816,20 @@ Options: agent-eval-rpc[dspy] (TRACES_PYTHON selects the interpreter), version-matched to this package's @tangle-network/agent-eval. Exits 1 when every agentic analyst fails, with each reason. + ask uses the same engine and credentials without --llm. --model Model for --llm, HALO, and Hodoscope (default for --llm: ${DEFAULT_ANALYST_MODEL}) --config investigate/improve/stream: JS config with analysts, liveAnalysts, or external analyzers - --budget USD cap for agentic analysts - --analyzer analyze: also run halo, hodoscope, prime, or an installed command (repeatable) + --budget USD cap for agentic analysts; ask: one ceiling shared by every question + --question ask: a question (repeatable, at most ${MAX_TRACE_QUESTION_CHARS} characters) + --questions ask: JSON array of questions, strings or + { id?, question, instructions?, answerSchema? } + --question-budget + ask: provider ceiling for one question (default: the smaller + of --budget and $${DEFAULT_QUESTION_MAX_COST_USD}) + --concurrency ask: questions running at once (default 4); + import-codetracebench: trajectories imported at once + --analyzer analyze: also run halo, hodoscope, prime, or an installed command (repeatable); + exits 1 after writing the report when any requested analyzer fails prime posts the full span projection to an OpenAI-compatible bridge (TRACES_PRIME_BRIDGE_URL, default http://localhost:4181; TRACES_PRIME_MODEL, default prime/zai/glm-5.2; TRACES_PRIME_TIMEOUT_MS) @@ -1776,6 +1906,8 @@ async function main(): Promise { case 'validate': await cmdValidate(args); break case 'investigate': await cmdInvestigate(args); break case 'improve': await cmdImprove(args); break + case 'ask': await cmdAsk(args); break + case 'facts': await cmdFacts(args); break case 'convert': await cmdConvert(args); break case 'index': await cmdIndex(args); break case 'bundle': await cmdBundle(args); break diff --git a/src/evidence.ts b/src/evidence.ts index 9871226..bffb648 100644 --- a/src/evidence.ts +++ b/src/evidence.ts @@ -6,6 +6,7 @@ import { OPENINFERENCE_SPAN_KIND, TOOL_NAME, } from '@tangle-network/agent-eval/trace-attributes' +import { isInheritedSpan, isSynthesizedSpan } from './adapters/provenance.js' import { ATTR } from './attributes.js' import { summarizeSpanExecution } from './execution.js' import type { OtlpSpan } from './otlp.js' @@ -113,6 +114,11 @@ function spanKind(span: OtlpSpan): string | undefined { return stringAttr(span, OPENINFERENCE_SPAN_KIND) } +/** A tool call the model issued — the only thing "tool call count" may mean. */ +function isModelToolCall(span: OtlpSpan): boolean { + return spanKind(span) === 'TOOL' && !isSynthesizedSpan(span.attributes) +} + function repoFromSpans(spans: readonly OtlpSpan[]): PolicyEvidenceRecord['repo'] { const attrs: { subjectKey?: string @@ -134,8 +140,15 @@ function repoFromSpans(spans: readonly OtlpSpan[]): PolicyEvidenceRecord['repo'] return attrs } +/** + * The window this session ACTED in. An inherited span carries context from + * before the parsed scope (a fork prefix, a compacted history), so counting its + * timestamps here would stretch the session window over the parent's work — + * and `buildSessionBundle` joins external evidence by exactly this window. + */ function timeBounds(spans: readonly OtlpSpan[]): { firstSpanAt: string | null; lastSpanAt: string | null } { const times = spans + .filter((span) => !isInheritedSpan(span.attributes)) .flatMap((span) => [span.start_time, span.end_time]) .filter((value) => value && value !== 'now') .sort() @@ -145,10 +158,15 @@ function timeBounds(spans: readonly OtlpSpan[]): { firstSpanAt: string | null; l } } +/** + * One row per tool the MODEL called. A synthesized span (a subagent's lifecycle + * assembled from harness events) is not a call the model issued, so it is + * excluded here and from every count below. + */ function summarizeTools(spans: readonly OtlpSpan[]): PolicyEvidenceToolSummary[] { const byTool = new Map() for (const span of spans) { - if (spanKind(span) !== 'TOOL') continue + if (!isModelToolCall(span)) continue const name = stringAttr(span, TOOL_NAME) ?? span.name.replace(/^tool\./, '') const current = byTool.get(name) ?? { calls: 0, errors: 0 } current.calls += 1 @@ -168,7 +186,7 @@ export async function buildPolicyEvidenceRecord( if (opts.sourceSha256 && !/^[a-f0-9]{64}$/.test(opts.sourceSha256)) { throw new Error('sourceSha256 must be a lowercase SHA-256 hex digest') } - const toolSpans = spans.filter((span) => spanKind(span) === 'TOOL') + const toolSpans = spans.filter(isModelToolCall) const erroredToolCallCount = toolSpans.filter((span) => span.status.code === 'ERROR').length const pipelines = await runPipelines(spans, { minLoopOccurrences: opts.minLoopOccurrences }) const loopLimit = opts.maxLoopExamples ?? 25 diff --git a/src/external.ts b/src/external.ts index 1bc4472..afeabb1 100644 --- a/src/external.ts +++ b/src/external.ts @@ -655,6 +655,21 @@ export function runExternalAnalyzers( })) } +/** + * Operator-facing message naming every external analyzer that failed, or + * undefined when all succeeded. A caller that requested an engine should + * treat its failure as a failed run: the report still carries the error, but + * exit 0 would read as "the engine ran and found nothing". + */ +export function externalFailureMessage(results: readonly ExternalAnalysisResult[]): string | undefined { + const failed = results.filter((result) => !result.ok) + if (failed.length === 0) return undefined + return [ + `${failed.length} of ${results.length} external analyzer(s) failed; the report contains the other results.`, + ...failed.map((result) => ` ${result.analyzer}: ${(result.error ?? 'failed without an error message').replace(/\s+/g, ' ').trim().slice(0, 400)}`), + ].join('\n') +} + // ──────────────────────────────── redactors ──────────────────────────────── /** An external PII/secret scrubber for free-form text — catches what regex diff --git a/src/finding-rejections.ts b/src/finding-rejections.ts new file mode 100644 index 0000000..60d89bb --- /dev/null +++ b/src/finding-rejections.ts @@ -0,0 +1,104 @@ +/** + * Evidence-gate rejections, counted per analyst and reason. + * + * agent-eval's trace analysts drop a submitted finding when its citations do + * not resolve, and report each drop only as a `finding rejected: ...` log + * event. A report that shows "0 findings" without those events reads as "the + * model found nothing" when the truth may be "the gate refused everything it + * found". This module turns the log events into counts a report can print. + */ + +/** Rejection counts by reason, for one analyst or question. */ +export type FindingRejectionReasons = Readonly> + +/** Rejection counts keyed by analyst ID, then by reason. */ +export type FindingRejectionCounts = Readonly> + +const REJECTION_MESSAGE = /^(?:\[([^\]]+)\] )?finding rejected: (.+)$/ + +/** + * The analyst and reason behind one `finding rejected:` log event, or + * undefined for any other event. The registry prefixes each analyst's log + * message with `[] `; a direct `runTraceAnalyst` caller does not, + * so the analyst ID is optional here. + */ +export function findingRejection( + message: string, + fields?: Readonly>, +): { analystId?: string; reason: string } | undefined { + const match = REJECTION_MESSAGE.exec(message) + if (!match) return undefined + const kind = match[2]!.trim() + // "unresolved evidence" is one message for several causes; the cause is in + // the fields, and it is the part an operator can act on. + const reason = kind === 'unresolved evidence' && typeof fields?.reason === 'string' && fields.reason + ? fields.reason + : kind + return match[1] ? { analystId: match[1], reason } : { reason } +} + +/** One-line detail for a rejection log event, or undefined for any other event. */ +export function findingRejectionDetail( + message: string, + fields?: Readonly>, +): string | undefined { + const rejection = findingRejection(message, fields) + if (!rejection) return undefined + const parts = [rejection.reason] + if (typeof fields?.uri === 'string' && fields.uri) parts.push(`uri ${fields.uri}`) + if (typeof fields?.required === 'number' && typeof fields?.distinct === 'number') { + parts.push(`${fields.distinct} of ${fields.required} required distinct citation(s)`) + } + if (typeof fields?.subject === 'string' && fields.subject) parts.push(`subject ${fields.subject}`) + return parts.join('; ') +} + +export interface FindingRejectionTally { + /** Record one log event; events that are not rejections are ignored. */ + record(message: string, fields?: Readonly>): void + /** A snapshot of the counts so far. */ + counts(): FindingRejectionCounts +} + +/** + * Count rejections from a log stream. `defaultAnalystId` names the analyst + * for unprefixed events, as when one tally observes one question. + */ +export function createFindingRejectionTally(defaultAnalystId = 'unknown'): FindingRejectionTally { + const byAnalyst = new Map>() + return { + record(message, fields) { + const rejection = findingRejection(message, fields) + if (!rejection) return + const analystId = rejection.analystId ?? defaultAnalystId + const reasons = byAnalyst.get(analystId) ?? new Map() + reasons.set(rejection.reason, (reasons.get(rejection.reason) ?? 0) + 1) + byAnalyst.set(analystId, reasons) + }, + counts() { + return Object.fromEntries( + [...byAnalyst].map(([analystId, reasons]) => [analystId, Object.fromEntries(reasons)]), + ) + }, + } +} + +/** Total rejections across every reason. */ +export function totalFindingRejections(reasons: FindingRejectionReasons | undefined): number { + return Object.values(reasons ?? {}).reduce((sum, count) => sum + count, 0) +} + +/** + * "2 finding(s) rejected: excerpt is not present in the cited span content ×2". + * Empty when nothing was rejected. Reasons are ordered by count, then name, so + * the same counts always render the same text. + */ +export function formatFindingRejections(reasons: FindingRejectionReasons | undefined): string { + const total = totalFindingRejections(reasons) + if (total === 0) return '' + const ordered = Object.entries(reasons ?? {}) + .filter(([, count]) => count > 0) + .sort(([a, left], [b, right]) => right - left || a.localeCompare(b)) + .map(([reason, count]) => `${reason} ×${count}`) + return `${total} finding(s) rejected: ${ordered.join('; ')}` +} diff --git a/src/improvement.ts b/src/improvement.ts index 3888ee3..e6c2689 100644 --- a/src/improvement.ts +++ b/src/improvement.ts @@ -38,6 +38,7 @@ import type { UnreadableSourceRows, } from './otlp-input.js' import type { ExternalAnalysisResult, ExternalAnalyzer } from './external.js' +import { createFindingRejectionTally, type FindingRejectionCounts } from './finding-rejections.js' import { spanEvidenceUri } from './external-analysis-validation.js' import { runExternalAnalyzers } from './external.js' import type { TraceLiveAnalyst } from './live.js' @@ -161,6 +162,12 @@ export interface TraceInvestigationResult { * findings as if the requested LLM analysis had happened. */ readonly agenticPerAnalyst?: readonly AnalystRunSummary[] + /** + * Findings the evidence gate refused, by analyst ID and reason. Present + * whenever an agentic pass ran, empty when nothing was refused, so a zero + * finding count can be told apart from "every finding was rejected". + */ + readonly findingRejections?: FindingRejectionCounts readonly external: readonly ExternalAnalysisResult[] readonly report: string } @@ -614,6 +621,7 @@ function renderInvestigationReport( workflow: result.workflow, conformance, unavailableCapabilities: unavailable, + ...(result.findingRejections ? { findingRejections: result.findingRejections } : {}), })}\n${renderAgenticRoute(result.agenticRoute)}` + `${renderPipelines(result.pipelines, unavailable)}\n` + `${renderLoopConvergence(result.loopConvergence, unavailable)}\n` + @@ -693,6 +701,13 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr const agenticRoute = opts.engine && !opts.agenticRegistry ? planTraceAgenticRoute(pipelines, reactions) : undefined + // The evidence gate reports each refused finding only as a log event. + // Counting here keeps those refusals in the report and the JSON result. + const rejections = createFindingRejectionTally() + const log = (msg: string, fields?: Record): void => { + rejections.record(msg, fields) + opts.log?.(msg, fields) + } const analysis = await analyzeSpans(opts.spans, { sourceBundle: opts.sourceBundle, engine: opts.engine, @@ -705,7 +720,7 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr otlpOutPath: opts.otlpOutPath, runId: `traces-investigation-${Date.parse(generatedAt) || Date.now()}`, signal: opts.signal, - log: opts.log, + log, }) const external = opts.externalAnalyzers?.length ? await runExternalAnalyzers(analysis.otlpPath, opts.externalAnalyzers, { @@ -741,7 +756,9 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr loopConvergence: analyzeLoopConvergence(opts.spans), steeringChain: analyzeSteeringChain(opts.spans), ...(agenticRoute ? { agenticRoute } : {}), - ...(analysis.agenticPerAnalyst ? { agenticPerAnalyst: analysis.agenticPerAnalyst } : {}), + ...(analysis.agenticPerAnalyst + ? { agenticPerAnalyst: analysis.agenticPerAnalyst, findingRejections: rejections.counts() } + : {}), external, } const result = { ...partial, report: '' } @@ -760,6 +777,11 @@ const AGENTIC_FAILURE_REASON_MAX_CHARS = 400 */ const BRIDGE_MISMATCH_PATTERN = /must contain exactly|No module named ['"]?agent_eval_rpc|could not start/ +/** True when an engine error looks like bridge version skew or a missing bridge install. */ +export function isBridgeMismatchError(message: string): boolean { + return BRIDGE_MISMATCH_PATTERN.test(message) +} + function condensedReason(summary: AnalystRunSummary): string { const raw = summary.status === 'skipped' ? `skipped — ${summary.reason ?? 'no reason recorded'}` @@ -789,7 +811,7 @@ export function totalAgenticFailureMessage( ] if ( opts.requiredBridgeVersion && - agenticPerAnalyst.some((summary) => BRIDGE_MISMATCH_PATTERN.test(summary.error?.message ?? '')) + agenticPerAnalyst.some((summary) => isBridgeMismatchError(summary.error?.message ?? '')) ) { lines.push( 'hint: the DSPy bridge protocol is version-locked — the TRACES_PYTHON interpreter needs ' + diff --git a/src/index.ts b/src/index.ts index 58bc6a0..54fdef8 100644 --- a/src/index.ts +++ b/src/index.ts @@ -79,6 +79,19 @@ export { textIsSynthetic, } from './adapters/actor.js' export type { Reaction } from './adapters/actor.js' +// Span provenance: which spans an adapter synthesized, and which carry context +// from outside the parsed scope. A count of what the agent DID excludes both. +export { + INHERITED_SOURCE_ATTR, + INHERITED_SPAN_ATTR, + INHERITED_SPAN_COUNT_ATTR, + INHERITED_SPANS_OMITTED_ATTR, + isInheritedSpan, + isSynthesizedSpan, + SYNTHESIZED_SOURCE_ATTR, + SYNTHESIZED_SPAN_ATTR, +} from './adapters/provenance.js' +export type { InheritedSpanSource } from './adapters/provenance.js' // ── Detection / analysis (built-in, or bring your own analysts) ─────────── export * from './failure-followup.js' // classifyFailureFollowUps() — blind vs adapted retry split @@ -97,6 +110,14 @@ export * from './inspect.js' // inspectSessionIndex() — ranked findings from a export * from './file-export.js' // convert evidence/events files to OpenInference JSONL export * from './chat-trajectory.js' // generic chat trajectory to stable step spans export * from './improvement.js' // runTraceInvestigation()/runTraceImprovement() artifact pack +export * from './ask.js' // runTraceQuestions(): concurrent free-form questions, answers kept and citations checked +export * from './session-facts.js' // computeSessionFacts(): the deterministic, free session-facts sheet +export * from './pull-request-facts.js' // readPullRequests(): the pull requests the command spans show +export * from './shell-commands.js' // shellCommands(): the simple commands one script would run +export * from './answer-schema.js' // the JSON Schema subset an ask answer may be held to +export * from './finding-rejections.js' // evidence-gate rejections counted per analyst and reason +export { analysisEngineFromEnv, DEFAULT_ANALYST_MODEL } from './analyst-model-call.js' +export type { AnalysisEngineFromEnvOptions } from './analyst-model-call.js' // ── External engines (NOT bundled — shell out to tools you install) ──────── export * from './external.js' // haloAnalyzer / commandAnalyzer; commandRedactor diff --git a/src/live.ts b/src/live.ts index 5025925..0679d19 100644 --- a/src/live.ts +++ b/src/live.ts @@ -1,4 +1,6 @@ import { createHash } from 'node:crypto' +import { isInheritedSpan, isSynthesizedSpan } from './adapters/provenance.js' +import { isInnerToolCall } from './adapters/tool-io.js' import type { OtlpSpan } from './otlp.js' import type { PipelineReport } from './pipelines.js' import { runPipelines } from './pipelines.js' @@ -237,12 +239,21 @@ function toolSignature(span: OtlpSpan): string { return `${toolName(span)}:${content || span.name}` } +/** + * A tool call the model issued. A synthesized span (a subagent lifecycle the + * adapter assembled from harness events) matches every surface heuristic below, + * so it is rejected first — otherwise this batch's tool count, and the error + * ratio derived from it, run high by one per child thread. + */ function isTool(span: OtlpSpan): boolean { + if (isSynthesizedSpan(span.attributes)) return false return spanKind(span) === 'TOOL' || span.attributes['tool.name'] != null || span.name.startsWith('tool.') } +/** Command and file-change records inside a tool call carry tool I/O, not prose. */ function isTextSpan(span: OtlpSpan): boolean { - return !isTool(span) && spanContent(span).length > 0 + if (isSynthesizedSpan(span.attributes) || isInheritedSpan(span.attributes)) return false + return !isTool(span) && !isInnerToolCall(span.attributes) && spanContent(span).length > 0 } function isVerification(span: OtlpSpan): boolean { diff --git a/src/pull-request-facts.ts b/src/pull-request-facts.ts new file mode 100644 index 0000000..2b5b40e --- /dev/null +++ b/src/pull-request-facts.ts @@ -0,0 +1,491 @@ +/** + * Which pull requests a session created, and which it merged. + * + * The spans already hold the answer and no reader was extracting it. A Codex + * `CommandExecution` item becomes a `command.execution` span carrying the + * script, its exit code and its output; a Claude Code `Bash` call becomes a + * TOOL span with the same three things under the same I/O keys. A `gh pr + * create` is therefore visible, and so is the pull-request URL `gh` printed + * back — which is the only place the new PR's number appears. + * + * Three rules keep the reading honest: + * + * 1. The command is found by scanning the script the way a shell would + * ({@link shellCommands}), not by matching text. A `gh pr create` quoted + * inside a heredoc body or a commit message never ran and is not counted. + * 2. A pull request is named by its number when the command or an output that + * joins to it shows one, and by its head branch when it does not — the + * identity the command itself supplies. A call with neither stays a + * recorded event with a null identifier and a stated reason. + * 3. Every entry names the span its command came from, plus the span whose + * output supplied the number, so the reading can be checked against the + * raw spans rather than trusted. + * + * A command that never reached `gh` is not an event: a failed + * `git push … && gh pr create` exited non-zero without running `gh`, so it + * counts only when the exit code is zero or the output shows `gh` itself + * answering. + */ + +import type { OtlpSpan } from './otlp.js' +import { isInnerToolCall } from './adapters/tool-io.js' +import { commandTextFromInput, shellCommands } from './shell-commands.js' + +/** The `gh` executables an audit recognizes, including the wrapper this project uses. */ +const GH_NAMES: ReadonlySet = new Set(['gh', 'gh-drew']) + +/** `gh pr create` flags that consume the following word. */ +const CREATE_VALUE_FLAGS: ReadonlySet = new Set([ + '-H', '--head', '-B', '--base', '-R', '--repo', '-t', '--title', '-b', '--body', '-F', '--body-file', + '-a', '--assignee', '-l', '--label', '-m', '--milestone', '-p', '--project', '-r', '--reviewer', '-T', '--template', +]) + +/** `gh pr merge` flags that consume the following word. */ +const MERGE_VALUE_FLAGS: ReadonlySet = new Set([ + '-t', '--subject', '-b', '--body', '-F', '--body-file', '--match-head-commit', '-R', '--repo', '-A', '--author-email', +]) + +/** + * Terminal control sequences a command's output may carry: a CSI sequence + * (`ESC [ … letter`) and an OSC string (`ESC ] … BEL`). Written as escapes so + * the source stays free of raw control bytes. + */ +const ANSI = /\u001B\[[0-9;?]*[A-Za-z]|\u001B\][^\u0007]*\u0007/g +const PR_URL_LINE = /^https?:\/\/[^/\s]+\/[^/\s]+\/[^/\s]+\/pull\/(\d+)\/?$/ +const PR_NUMBER = /https?:\/\/[^/\s"']+\/[^/\s"']+\/[^/\s"']+\/pull\/(\d+)|(?:^|\s)#(\d+)\b/g +const MERGED_NUMBER = + /(?:(?:Squashed and merged|Rebased and merged|Merged) pull request|Pull request)\s+(?:[\w.\-]+\/[\w.\-]+)?#(\d+)/ + +/** Output that shows `gh pr create` itself answered, however it answered. */ +const CREATE_REACHED_GH = + /createPullRequest|pull request create failed|must first push the current branch|a pull request for branch|already exists|Creating pull request for|No commits between|Head ref must be a branch|GraphQL:|HTTP 4\d\d/i + +/** Output that shows `gh pr merge` itself answered. */ +const MERGE_REACHED_GH = + /mergePullRequest|pull request|Merged|not mergeable|GraphQL:|failed to run git|base branch policy|approving review is required|HTTP 4\d\d/i + +/** Any pull-request URL, anywhere in a command's output. Host-agnostic: `gh` + * serves GitHub Enterprise hosts under the same `/owner/repo/pull/N` path. */ +const PULL_REQUEST_URL = /https?:\/\/[^/\s"']+\/[^/\s"']+\/[^/\s"']+\/pull\/\d+/ + +/** A branch name a `git push` named in its own output. */ +const PUSH_TRACKING = /branch '([^']+)' set up to track/ +const PUSH_NEW_BRANCH = /\[new branch\]\s+\S+\s+->\s+(\S+)/ + +/** One pull request the agent created or merged, with the evidence that named it. */ +export interface PullRequestFact { + /** The PR number when one is known, else the head branch. Null when neither is. */ + readonly identifier: string | null + readonly number: string | null + readonly headBranch: string | null + /** The `gh` command, as the scanner read it. */ + readonly command: string + /** How `identifier` was arrived at, naming the span that supplied it. */ + readonly evidence: string + /** The command span, plus any span whose output supplied the number. */ + readonly spanIds: readonly string[] + /** Why `identifier` is null. Null when the pull request was identified. */ + readonly unavailable: string | null +} + +export interface PullRequestFacts { + readonly created: readonly PullRequestFact[] + readonly merged: readonly PullRequestFact[] +} + +export interface PullRequestReading { + /** Null exactly when `unavailable` explains why the spans cannot support it. */ + readonly facts: PullRequestFacts | null + readonly spanIds: readonly string[] + readonly unavailable: string | null + readonly partial: string | null + /** Spans carrying an executed command: the denominator of this reading. */ + readonly commandSpans: number +} + +interface CommandRecord { + readonly spanId: string + readonly script: string + readonly output: string + readonly failed: boolean + readonly inputTruncated: boolean + readonly outputTruncated: boolean +} + +interface PullRequestEvent { + action: 'create' | 'merge' + number: string | null + headBranch: string | null + command: string + evidence: string + spanIds: string[] + recordIndex: number + outputTruncated: boolean +} + +function stringAttribute(span: OtlpSpan, key: string): string | null { + const value = span.attributes[key] + return typeof value === 'string' ? value : null +} + +function basename(word: string): string { + const cut = word.lastIndexOf('/') + return cut === -1 ? word : word.slice(cut + 1) +} + +/** + * Commands the spans record, in trace order. `spans` must already be ordered. + * + * A harness that records both levels records one command twice: the call the + * model issued, and the inner span for what that call actually ran. Only the + * inner span carries the exit code and the command's own output, so when both + * are present the outer call is dropped — otherwise a `git push && gh pr + * create` whose push failed would be read once as reaching `gh` (the outer span + * has no exit code) and once as not reaching it. + */ +function commandRecords(spans: readonly OtlpSpan[]): CommandRecord[] { + const scripted = new Map() + for (const span of spans) { + const input = stringAttribute(span, 'input.value') + if (input === null) continue + const script = commandTextFromInput(input) + if (script !== undefined) scripted.set(span, script) + } + const innerParents = new Set() + const innerScripts = new Set() + for (const [span, script] of scripted) { + if (!isInnerToolCall(span.attributes)) continue + if (span.parent_span_id !== null) innerParents.add(span.parent_span_id) + innerScripts.add(script) + } + const records: CommandRecord[] = [] + for (const [span, script] of scripted) { + const outer = !isInnerToolCall(span.attributes) + if (outer && (innerParents.has(span.span_id) || innerScripts.has(script))) continue + const exit = span.attributes['process.exit_code'] + const exitCode = typeof exit === 'number' ? exit : null + records.push({ + spanId: span.span_id, + script, + output: stringAttribute(span, 'output.value') ?? '', + failed: exitCode === null ? span.status.code === 'ERROR' : exitCode !== 0, + inputTruncated: span.attributes['traces.input.truncated'] === true, + outputTruncated: span.attributes['traces.output.truncated'] === true, + }) + } + return records +} + +/** Flags and positional words of one `gh pr …` argument list. */ +function parseFlags( + args: readonly string[], + valueFlags: ReadonlySet, +): { flags: Map; positional: string[] } { + const flags = new Map() + const positional: string[] = [] + for (let index = 0; index < args.length; index += 1) { + const arg = args[index]! + if (arg === '--') { + positional.push(...args.slice(index + 1)) + break + } + if (arg.startsWith('--') && arg.includes('=')) { + const cut = arg.indexOf('=') + flags.set(arg.slice(0, cut), arg.slice(cut + 1)) + } else if (valueFlags.has(arg)) { + flags.set(arg, args[index + 1] ?? true) + index += 1 + } else if (arg.startsWith('-') && arg.length > 1) { + flags.set(arg, true) + } else { + positional.push(arg) + } + } + return { flags, positional } +} + +function flagValue(flags: Map, ...names: readonly string[]): string | null { + for (const name of names) { + const value = flags.get(name) + if (typeof value === 'string' && value.length > 0) return value + } + return null +} + +/** Pull-request URLs `gh` printed on a line of their own, in order. */ +function printedPullRequestNumbers(output: string): string[] { + const numbers: string[] = [] + for (const raw of output.split('\n')) { + const line = raw.replace(ANSI, '').split('\r').pop()!.trim() + const match = PR_URL_LINE.exec(line) + if (match) numbers.push(match[1]!) + } + return numbers +} + +/** Every pull-request number one line of text states, by URL or by `#N`. */ +function numbersInLine(line: string): string[] { + const found: string[] = [] + for (const match of line.matchAll(PR_NUMBER)) found.push(match[1] ?? match[2]!) + return found +} + +/** + * The head branch of a `gh pr create` that did not pass `--head`: the branch the + * same script pushed or created just before it, else the branch the push output + * names. `gh` otherwise defaults to the checked-out branch, which no span carries. + */ +function inferHeadBranch(commands: readonly string[][], upTo: number, output: string): string | null { + for (let index = upTo - 1; index >= 0; index -= 1) { + const words = commands[index]! + if (basename(words[0] ?? '') !== 'git' || words.length < 2) continue + const sub = words[1] + const rest = words.slice(2) + if (sub === 'push') { + const positional = rest.filter((word) => !word.startsWith('-')) + const ref = positional[positional.length - 1] + if (positional.length >= 2 && ref !== undefined && ref !== 'HEAD') { + const branch = (ref.includes(':') ? ref.slice(ref.indexOf(':') + 1) : ref).replace(/^\+/, '') + if (branch.length > 0) return branch + } + } + if (sub === 'checkout' || sub === 'switch') { + for (let position = 0; position < rest.length - 1; position += 1) { + if (['-b', '-B', '-c', '-C', '--create', '--force-create'].includes(rest[position]!)) return rest[position + 1]! + } + } + if (sub === 'worktree' && rest[0] === 'add') { + for (let position = 1; position < rest.length - 1; position += 1) { + if (rest[position] === '-b' || rest[position] === '-B') return rest[position + 1]! + } + } + } + return PUSH_TRACKING.exec(output)?.[1] ?? PUSH_NEW_BRANCH.exec(output)?.[1] ?? null +} + +/** Whether the shell reached `gh` at all, for a command that exited non-zero. */ +function reachedGh(record: CommandRecord, action: 'create' | 'merge', sawNumber: boolean): boolean { + if (!record.failed) return true + if (sawNumber || PULL_REQUEST_URL.test(record.output)) return true + return action === 'create' ? CREATE_REACHED_GH.test(record.output) : MERGE_REACHED_GH.test(record.output) +} + +function createEvent( + record: CommandRecord, + recordIndex: number, + words: readonly string[], + commands: readonly string[][], + commandIndex: number, + printedNumber: string | null, +): PullRequestEvent | null { + const { flags } = parseFlags(words.slice(3), CREATE_VALUE_FLAGS) + if (flags.has('--help') || flags.has('-h')) return null + if (!reachedGh(record, 'create', printedNumber !== null)) return null + const headFlag = flagValue(flags, '--head', '-H') + const headBranch = headFlag ?? inferHeadBranch(commands, commandIndex, record.output) + const evidence = + printedNumber !== null + ? `the pull-request URL this command printed, in span ${record.spanId}` + : headFlag !== null + ? 'the --head branch given to the command' + : headBranch !== null + ? 'the branch this same command pushed or created before calling gh' + : 'none: the command named no head branch and printed no pull-request URL' + return { + action: 'create', + number: printedNumber, + headBranch, + command: words.join(' '), + evidence, + spanIds: [record.spanId], + recordIndex, + outputTruncated: record.outputTruncated, + } +} + +function mergeEvent( + record: CommandRecord, + recordIndex: number, + words: readonly string[], +): PullRequestEvent | null { + const { flags, positional } = parseFlags(words.slice(3), MERGE_VALUE_FLAGS) + if (flags.has('--help') || flags.has('-h')) return null + const target = positional[0] ?? null + const numberInCommand = + target === null ? null : (/^#?(\d+)$/.exec(target)?.[1] ?? numbersInLine(target)[0] ?? null) + const numberInOutput = numberInCommand === null ? (MERGED_NUMBER.exec(record.output)?.[1] ?? null) : null + const number = numberInCommand ?? numberInOutput + if (!reachedGh(record, 'merge', number !== null)) return null + const evidence = + numberInCommand !== null + ? 'the pull request the command names' + : numberInOutput !== null + ? `the merge line this command printed, in span ${record.spanId}` + : target !== null + ? 'the branch the command names; no output that joins to it stated a number' + : 'none: the command named neither a pull request nor a branch' + return { + action: 'merge', + number, + headBranch: number === null ? target : null, + command: words.join(' '), + evidence, + spanIds: [record.spanId], + recordIndex, + outputTruncated: record.outputTruncated, + } +} + +function eventsFromRecord(record: CommandRecord, recordIndex: number): PullRequestEvent[] { + if (!/\bpr\b/.test(record.script)) return [] + const commands = shellCommands(record.script) + const printed = printedPullRequestNumbers(record.output) + const events: PullRequestEvent[] = [] + let createIndex = 0 + for (let index = 0; index < commands.length; index += 1) { + const words = commands[index]! + if (words.length < 3 || !GH_NAMES.has(basename(words[0]!)) || words[1] !== 'pr') continue + const action = words[2] + if (action === 'create') { + const printedNumber = printed[createIndex] ?? null + createIndex += 1 + const event = createEvent(record, recordIndex, words, commands, index, printedNumber) + if (event) events.push(event) + continue + } + if (action !== 'merge') continue + const event = mergeEvent(record, recordIndex, words) + if (event) events.push(event) + } + return events +} + +/** + * A number a later command's output states for this head branch. + * + * `gh pr create` prints the new URL, but a script that redirected or swallowed + * its stdout leaves the branch as the only identity — until a later `gh pr + * view`, `gh pr list` or merge names the same branch beside a number. The join + * demands both on one line and exactly one distinct number there, so a listing + * of many branches identifies none of them. + */ +function numberFromLaterOutput( + records: readonly CommandRecord[], + after: number, + headBranch: string, +): { number: string; spanId: string } | null { + for (let index = after + 1; index < records.length; index += 1) { + const record = records[index]! + const found = new Set() + for (const raw of `${record.script}\n${record.output}`.split('\n')) { + const line = raw.replace(ANSI, '') + if (!line.includes(headBranch)) continue + for (const number of numbersInLine(line)) found.add(number) + } + if (found.size === 1) return { number: [...found][0]!, spanId: record.spanId } + } + return null +} + +function identify(event: PullRequestEvent): PullRequestFact { + const identifier = event.number ?? event.headBranch + return { + identifier, + number: event.number, + headBranch: event.headBranch, + command: event.command, + evidence: event.evidence, + spanIds: [...new Set(event.spanIds)], + unavailable: + identifier === null + ? 'the command named no pull-request number and no head branch, and no output that joins to it stated one' + : null, + } +} + +/** Keep one entry per identity; a failed create and its retry are one pull request. */ +function distinct(facts: readonly PullRequestFact[]): PullRequestFact[] { + const kept = new Map() + for (const fact of facts) { + const key = fact.identifier ?? `command:${fact.command}` + const existing = kept.get(key) + kept.set( + key, + existing ? { ...existing, spanIds: [...new Set([...existing.spanIds, ...fact.spanIds])] } : fact, + ) + } + return [...kept.values()] +} + +/** + * Read every pull request the spans show the agent creating or merging. + * + * `spans` must be in trace order: a number a later command states is joined to + * the create it belongs to by position, and that ordering is the position. + */ +export function readPullRequests(spans: readonly OtlpSpan[]): PullRequestReading { + const records = commandRecords(spans) + if (records.length === 0) { + return { + facts: null, + spanIds: [], + commandSpans: 0, + partial: null, + unavailable: + 'no span in this trace carries an executed command (an input.value with a command or cmd field), ' + + 'so whether the agent created or merged a pull request cannot be read from these spans', + } + } + + const events: PullRequestEvent[] = [] + for (let index = 0; index < records.length; index += 1) events.push(...eventsFromRecord(records[index]!, index)) + + // A create shown only by head branch takes the number another create for the + // same branch produced: a failed first attempt and its retry are one PR. + const numberByBranch = new Map() + for (const event of events) { + if (event.action !== 'create' || event.number === null || event.headBranch === null) continue + if (!numberByBranch.has(event.headBranch)) numberByBranch.set(event.headBranch, event.number) + } + let unjoinedTruncated = 0 + for (const event of events) { + if (event.number !== null || event.headBranch === null) continue + const sibling = numberByBranch.get(event.headBranch) + if (sibling !== undefined) { + event.number = sibling + event.evidence = `${event.evidence}, resolved to the number another create for this branch printed` + continue + } + const later = numberFromLaterOutput(records, event.recordIndex, event.headBranch) + if (later !== null) { + event.number = later.number + event.spanIds.push(later.spanId) + event.evidence = `${event.evidence}, resolved to the number a later output states for this branch, in span ${later.spanId}` + continue + } + if (event.outputTruncated) unjoinedTruncated += 1 + } + + const created = distinct(events.filter((event) => event.action === 'create').map(identify)) + const merged = distinct(events.filter((event) => event.action === 'merge').map(identify)) + const truncatedCommands = records.filter((record) => record.inputTruncated && /\bpr\b/.test(record.script)).length + const notes: string[] = [] + if (truncatedCommands > 0) { + notes.push( + `${truncatedCommands} command span(s) mentioning a pull request had truncated input; ` + + 'a gh command past the cut is not in this list', + ) + } + if (unjoinedTruncated > 0) { + notes.push( + `${unjoinedTruncated} entr(y|ies) kept a branch identity while the output that could have named its number was truncated`, + ) + } + return { + facts: { created, merged }, + spanIds: [...new Set([...created, ...merged].flatMap((fact) => fact.spanIds))], + commandSpans: records.length, + unavailable: null, + partial: notes.length > 0 ? notes.join('; ') : null, + } +} diff --git a/src/reactions.ts b/src/reactions.ts index 4a5ce1e..b6df0ac 100644 --- a/src/reactions.ts +++ b/src/reactions.ts @@ -18,6 +18,7 @@ import { classifyReaction, CORRECTIVE_REACTIONS, type Reaction } from './adapters/actor.js' import { ACTOR_ATTR } from './adapters/conversation.js' +import { isInheritedSpan } from './adapters/provenance.js' import type { OtlpSpan } from './otlp.js' /** Reaction labels in stable render order. */ @@ -80,7 +81,10 @@ function isAssistant(s: OtlpSpan): boolean { return kind === 'LLM' || s.name.startsWith('message.assistant') } +/** An inherited turn was typed into ANOTHER scope (a fork's parent, a + * compacted history), so it has no assistant turn here to react to. */ function isHumanPrompt(s: OtlpSpan): boolean { + if (isInheritedSpan(s.attributes)) return false return s.name === 'user.prompt' && s.attributes[ACTOR_ATTR] === 'human' } diff --git a/src/report.ts b/src/report.ts index 1f35216..506d5f4 100644 --- a/src/report.ts +++ b/src/report.ts @@ -14,8 +14,14 @@ import type { } from '@tangle-network/agent-eval/contract' import type { AdoptionReport } from './adoption.js' import { ACTOR_ATTR } from './adapters/conversation.js' +import { isInheritedSpan } from './adapters/provenance.js' import { ATTR, sessionIdFromAttributes } from './attributes.js' import { incompleteInputsNote, type UnavailableCapabilities } from './conformance.js' +import { + type FindingRejectionCounts, + type FindingRejectionReasons, + formatFindingRejections, +} from './finding-rejections.js' import type { LoopConvergenceReport, SteeringChainReport } from './loop-analysis.js' import type { OtlpSpan } from './otlp.js' import type { PipelineReport } from './pipelines.js' @@ -58,6 +64,8 @@ export interface ReportMeta { * confident totals, is worse than no conformance section at all. */ unavailableCapabilities?: UnavailableCapabilities + /** Findings the evidence gate refused, by analyst ID and reason. */ + findingRejections?: FindingRejectionCounts } export interface ReportSource { @@ -91,9 +99,12 @@ export function sessionReportSource( sessionIdOverride?: string, ): ReportSource { const root = spans.find((item) => item.parent_span_id === null) ?? spans[0] - const prompt = spans.find( + // The subject names what THIS scope was asked to do, so an inherited turn + // (a fork's parent prompt, a compacted history) never supplies it. + const inScope = spans.filter((item) => !isInheritedSpan(item.attributes)) + const prompt = inScope.find( (item) => item.name === 'user.prompt' && item.attributes[ACTOR_ATTR] === 'human', - ) ?? spans.find((item) => item.name === 'user.prompt') ?? spans.find( + ) ?? inScope.find((item) => item.name === 'user.prompt') ?? inScope.find( (item) => item.attributes['span.type'] === 'interaction' && typeof item.attributes.content === 'string', ) const content = typeof prompt?.attributes.content === 'string' ? prompt.attributes.content : '' @@ -221,13 +232,16 @@ export function condenseAnalystError(raw: string, maxChars: number): string { * Failed engine runs die with the whole bridge stderr in the error message; * without this cell the report scores the analyst without saying why. */ -export function analystRunDetail(summary: AnalystRunSummary): string { +export function analystRunDetail(summary: AnalystRunSummary, rejections?: FindingRejectionReasons): string { const raw = summary.status === 'failed' && summary.error ? [summary.error.class, summary.error.message].map((part) => part.trim()).filter(Boolean).join(': ') : summary.status === 'skipped' && summary.reason ? summary.reason : '' - const cell = tableCell(condenseAnalystError(raw, ANALYST_DETAIL_MAX_CHARS)) + const detail = condenseAnalystError(raw, ANALYST_DETAIL_MAX_CHARS) + // The rejection text is short and bounded by the gate's reason vocabulary, + // so it is appended after the condensed error rather than competing for it. + const cell = tableCell([detail, formatFindingRejections(rejections)].filter(Boolean).join('; ')) return cell === '' ? '—' : cell } @@ -567,7 +581,10 @@ export function renderReport(result: AnalystRunResult, meta: ReportMeta): string lines.push('| Analyst | Status | Findings | Latency | Detail |') lines.push('|---|---|---|---|---|') for (const s of result.per_analyst) { - lines.push(`| \`${s.analyst_id}\` | ${s.status} | ${s.findings_count} | ${s.latency_ms}ms | ${analystRunDetail(s)} |`) + lines.push( + `| \`${s.analyst_id}\` | ${s.status} | ${s.findings_count} | ${s.latency_ms}ms | ` + + `${analystRunDetail(s, meta.findingRejections?.[s.analyst_id])} |`, + ) } lines.push('') diff --git a/src/run-span-tree.ts b/src/run-span-tree.ts index f5a8d45..85659ea 100644 --- a/src/run-span-tree.ts +++ b/src/run-span-tree.ts @@ -39,6 +39,7 @@ import { LLM_OUTPUT_TOKEN_ATTR_KEYS, SPAN_KIND_ATTR_KEYS, } from '@tangle-network/agent-eval/trace-attributes' +import { isSynthesizedSpan } from './adapters/provenance.js' import { exportTraceEvidenceFile } from './file-export.js' import type { OtlpSpan } from './otlp.js' import { connectors, forEachTreeNode, int, ms, tokens, usd } from './run-view-format.js' @@ -291,7 +292,10 @@ export function buildSpanRunTree(spans: readonly OtlpSpan[], source: string): Sp llmSpansWithoutTokens += 1 } if (firstNumberAttr(span.attributes, LLM_COST_ATTR_KEYS) === null) llmSpansWithoutCost += 1 - } else if (kind === 'TOOL') host.toolCalls += 1 + } else if (kind === 'TOOL' && !isSynthesizedSpan(span.attributes)) { + // A synthesized lifecycle span is not a call this node made. + host.toolCalls += 1 + } host.startMs = Math.min(host.startMs, epoch(span.start_time)) host.endMs = Math.max(host.endMs, epoch(span.end_time)) if (span.status.code === 'ERROR') { diff --git a/src/session-facts.ts b/src/session-facts.ts new file mode 100644 index 0000000..70950a4 --- /dev/null +++ b/src/session-facts.ts @@ -0,0 +1,1121 @@ +/** + * The deterministic session-facts sheet. + * + * A handful of facts about a session decide most audit questions — how many + * tools ran, which subagents were spawned, which pull requests were opened and + * merged, what the human actually typed, what the agent said last, which files + * changed, when the session started and ended, and how many tokens it burned. + * Every one of them is already carried by the normalized + * spans, but no trace tool returns any of them: `viewTrace` degrades to a + * 20-entry name histogram above its byte ceiling, `countTraces` counts traces + * rather than spans, `viewSpans` needs span ids the reader does not have, and + * `searchTrace` stops at 500 hits. A model asked for a tool-call total therefore + * adds up a capped histogram and decides by eye which names count. + * + * This module computes those facts mechanically instead. No model call, no + * budget, no byte ceiling: it reads the spans directly, so a session far above + * the tool ceiling still yields exact numbers. + * + * Two rules make the sheet auditable: + * + * 1. Every field carries the span ids it was computed from, so a reader can + * open those spans and check the number. + * 2. A field the spans cannot support is `null` with a stated `unavailable` + * reason. It is never guessed, and never silently zero. A value the sheet + * filtered — a `user.prompt` turn that is not a human turn of this session + * — is reported beside the count with the reason, never silently dropped. + * + * The sheet is NOT a span and cannot be cited. It is prepared context, and + * `trace://` citations still resolve against the raw spans — which is why every + * fact names its span ids rather than asking the reader to trust the sheet. + */ + +import type { TraceAnalystDefinition } from '@tangle-network/agent-eval/analyst' +import { OPENINFERENCE_SPAN_KIND, TOOL_NAME } from '@tangle-network/agent-eval/trace-attributes' +import { ACTOR_ATTR } from './adapters/conversation.js' +import { + INHERITED_SOURCE_ATTR, + INHERITED_SPAN_ATTR, + isSynthesizedSpan as isSynthesizedByProvenance, +} from './adapters/provenance.js' +import { indexSessionIdsByTrace } from './attributes.js' +import type { OtlpSpan } from './otlp.js' +import { readPullRequests, type PullRequestFacts } from './pull-request-facts.js' + +/** + * The cumulative harness token total for a whole session, when the adapter + * records one. Codex's `token_count` events carry + * `info.total_token_usage.total_tokens`; the adapter uses that object only as a + * de-duplication signature today, so no span carries the value and + * {@link SessionFacts.tokenTotal} states that rather than summing the per-turn + * deltas, which is a different (and smaller) number. + */ +export const SESSION_TOKEN_TOTAL_ATTR = 'traces.session.total_tokens' + +/** + * The first marker an adapter used for a span it created to describe a + * lifecycle rather than an invocation the agent made. The current marker is + * `traces.span.synthesized` (see `adapters/provenance.ts`); this one is still + * recognized so a trace exported before the rename keeps counting correctly. + */ +export const SPAN_SYNTHESIZED_ATTR = 'traces.codex.span_synthesized' + +/** Records the session reader could not parse, stamped on the session span. */ +const CORRUPTION_COUNT_ATTR = 'traces.session.corruption_count' + +/** Present on a per-record integrity receipt span, which is not a session record. */ +const CORRUPTION_RECEIPT_VERSION_ATTR = 'traces.session.corruption.receipt_version' + +/** Max characters of message or prompt text kept per fact entry. */ +export const FACT_TEXT_CAP = 2000 + +/** Entries kept per list field before the sheet reports the rest as omitted. */ +export const FACT_LIST_CAP = 200 + +/** + * Entries kept for {@link SessionFacts.changedFiles}. + * + * The cap above bounds the sheet's size against lists whose entries carry up to + * {@link FACT_TEXT_CAP} characters of message or prompt text. A changed-file + * entry is a path and two short arrays, two orders of magnitude smaller, and a + * session that edits hundreds of files is the one whose file list a reader most + * needs whole — the rendered context prints only the count either way. So this + * list gets its own ceiling, still finite so a runaway session cannot make the + * sheet unbounded. + */ +export const CHANGED_FILE_LIST_CAP = 5000 + +/** + * One measured fact. `value` is `null` exactly when `unavailable` explains why + * the spans cannot support it; `spanIds` names the spans the value came from, + * in the order they appear in the trace. + */ +export interface SessionFact { + readonly value: T | null + readonly spanIds: readonly string[] + /** Why `value` is null. Null when the fact was measured. */ + readonly unavailable: string | null + /** A measured value that is known to be incomplete says so here. */ + readonly partial?: string +} + +/** One `spawn_agent` call, with the task name the adapter recorded for it. */ +export interface SubagentSpawnFact { + /** `traces.codex.spawn_agent_path`, verbatim. Null when the span carries none. */ + readonly taskName: string | null + /** Why `taskName` is null. Null when it was recorded. */ + readonly taskNameUnavailable: string | null + readonly startedAt: string + /** The spawn span's status: `OK`, `ERROR`, or `UNSET` while still open. */ + readonly status: string + /** The spawn call span, plus the subagent lifecycle span when one joined it. */ + readonly spanIds: readonly string[] +} + +/** One `user.prompt` turn, with the actor the adapter classified it as. */ +export interface SessionTurnFact { + readonly actor: string + readonly at: string + readonly text: string + readonly spanId: string +} + +/** How many `user.prompt` turns each actor produced. */ +export interface TurnActorCount { + readonly actor: string + readonly turns: number + readonly spanIds: readonly string[] +} + +/** + * `user.prompt` turns {@link SessionFacts.humanTurns} did not count, grouped by + * the reason they were not counted. Nothing is dropped silently: every excluded + * span is named here, so a reader who disagrees with a reason can open the span + * and count it back in. + */ +export interface ExcludedTurnFact { + readonly reason: string + readonly turns: number + readonly spanIds: readonly string[] +} + +/** The last thing one agent said, for the main session or for one subagent. */ +export interface FinalMessageFact { + /** + * The subagent's task path, or null for the session's own agent. One session + * interleaves its own `message.assistant` spans with the `message.agent.*` + * traffic of every subagent under the same root, so "the final message" is + * only well defined per task. + */ + readonly task: string | null + readonly at: string + readonly text: string + readonly spanId: string +} + +/** One path a tool call changed, and how. */ +export interface ChangedFileFact { + readonly path: string + /** + * `add`, `update`, `delete`, or `move`, in the order first observed. A kind a + * harness reports under another name is kept verbatim rather than mapped. + */ + readonly operations: readonly string[] + readonly spanIds: readonly string[] +} + +/** Everything the spans of one session state, computed without a model. */ +export interface SessionFacts { + readonly schemaVersion: 1 + readonly kind: 'traces.session_facts' + readonly traceId: string + readonly sessionId: string | null + readonly harness: string | null + /** Spans in this trace, including the synthesized ones. */ + readonly spanCount: number + /** + * Spans the adapter built from the session's own records. The session root + * and the integrity receipts are not among them, so zero means nothing in the + * file was read and every count below would be an unsupported zero. + */ + readonly recordSpans: number + /** Records the adapter could not read, from the session's integrity receipt. */ + readonly unreadRecords: SessionFact + /** TOOL spans the agent actually invoked: synthesized lifecycle spans excluded. */ + readonly toolCalls: SessionFact + /** Every synthesized span excluded from `toolCalls`, so the exclusion is checkable. */ + readonly synthesizedToolSpans: SessionFact + /** Tool-call counts by tool name, over the same spans `toolCalls` counted. */ + readonly toolCallsByName: SessionFact>> + readonly subagents: SessionFact + /** Pull requests this session created and merged, each with its evidence. */ + readonly pullRequests: SessionFact + /** `user.prompt` turns a person typed into THIS session, in order. */ + readonly humanTurns: SessionFact + /** Every `user.prompt` turn `humanTurns` left out, with the reason. */ + readonly excludedTurns: SessionFact + /** Every `user.prompt` turn by actor, so the human filter is checkable. */ + readonly turnsByActor: SessionFact + /** The last message of the session's own agent, and of each subagent task. */ + readonly finalMessages: SessionFact + readonly changedFiles: SessionFact + /** Earliest span start in the trace. Not the session file's first record when + * the adapter selected a task boundary inside the file. */ + readonly firstRecordAt: SessionFact + /** Latest span end in the trace. */ + readonly lastRecordAt: SessionFact + /** The harness's own cumulative token total, when a span carries it. */ + readonly tokenTotal: SessionFact +} + +export interface SessionFactsReport { + readonly schemaVersion: 1 + readonly kind: 'traces.session_facts_report' + readonly generatedAt: string + readonly harness: string | null + readonly spanCount: number + readonly sessions: readonly SessionFacts[] +} + +function attr(span: OtlpSpan, key: string): unknown { + return span.attributes[key] +} + +function stringAttr(span: OtlpSpan, key: string): string | null { + const value = span.attributes[key] + return typeof value === 'string' && value.length > 0 ? value : null +} + +function capText(raw: string): string { + const text = raw.trim() + if (text.length <= FACT_TEXT_CAP) return text + return `${text.slice(0, FACT_TEXT_CAP)}… [+${text.length - FACT_TEXT_CAP} chars]` +} + +/** Trace order: the adapter's `step` when it recorded one, else start time. */ +function orderOf(span: OtlpSpan): [number, number] { + const step = span.attributes.step + const start = Date.parse(span.start_time) + return [typeof step === 'number' ? step : Number.MAX_SAFE_INTEGER, Number.isFinite(start) ? start : 0] +} + +function byTraceOrder(left: OtlpSpan, right: OtlpSpan): number { + const [leftStep, leftStart] = orderOf(left) + const [rightStep, rightStart] = orderOf(right) + if (leftStart !== rightStart) return leftStart - rightStart + return leftStep - rightStep +} + +/** + * A span an adapter created to describe a subagent's lifecycle rather than to + * record a call the agent made. Codex's `ensureSubagentSpan` emits one such + * span per subagent thread with `kind: TOOL` and `tool.name: Agent`, so any + * span-kind count is high by exactly the number of subagents unless it is + * excluded. Two signatures identify it: the explicit tag an adapter may set, + * and the subagent thread/path attribute pair only that span carries. + */ +export function isSynthesizedSpan(span: OtlpSpan): boolean { + if (isSynthesizedByProvenance(span.attributes)) return true + if (attr(span, SPAN_SYNTHESIZED_ATTR) === true) return true + return ( + stringAttr(span, 'traces.codex.subagent_thread_id') !== null && + stringAttr(span, 'traces.codex.subagent_path') !== null + ) +} + +function isToolSpan(span: OtlpSpan): boolean { + return attr(span, OPENINFERENCE_SPAN_KIND) === 'TOOL' +} + +function toolName(span: OtlpSpan): string { + return stringAttr(span, TOOL_NAME) ?? span.name.replace(/^tool\./, '') +} + +function inputValue(span: OtlpSpan): string | null { + const value = span.attributes['input.value'] + return typeof value === 'string' ? value : null +} + +function inputTruncated(span: OtlpSpan): boolean { + return span.attributes['traces.input.truncated'] === true +} + +// The fallback source: `apply_patch` envelopes name every path they touch in a +// header line, and the adapter keeps the patch verbatim in `input.value`, so +// the headers survive unless the value exceeded the adapter's own 16 KiB I/O +// cap. A header is the text the caller wrote, not a path the harness resolved, +// so it is read only for edits no `file.change` span already states. +// +// The patch reaches the span either as raw text or as a JSON-encoded tool +// argument, where the line breaks are the two characters `\` and `n`. The +// header therefore ends at a real newline, at the backslash of an escape, or at +// the closing quote — which is also why a path containing `"` or `\` is not +// recovered, and is reported as a missing path rather than a wrong one. +const PATCH_HEADER = /\*\*\* (Add|Update|Delete|Move to) File: ([^\n"\\]+)/g +const PATCH_OPERATION: Readonly> = { + Add: 'add', + Update: 'update', + Delete: 'delete', + 'Move to': 'move', +} + +/** + * A span the harness wrote for one path its own patch machinery changed. + * + * Codex's adapter emits one `file.change` span per changed path from the + * `FileChange` item the harness records after it applies a patch, carrying the + * path the edit actually reached and the kind of change. That is a better + * source than the patch text below: the harness resolved the path, so a patch a + * script generated is named correctly even when its header still held the + * script's own variable. + */ +const FILE_CHANGE_SPAN = 'file.change' +const FILE_CHANGE_KIND_ATTR = 'traces.codex.file_change_kind' + +/** + * A path that still holds an unexpanded variable: `${path}` from a template + * literal, or `$FILE` from a shell. A script that builds a patch writes its own + * text into the header, so the header can name a variable rather than a file. + * It names no file, so it is dropped and counted rather than emitted as a path + * the session never touched. + */ +const UNRESOLVED_PATH = /(?:^|\/)\$/ + +/** Tools whose arguments name one edited file directly rather than in a patch. */ +const FILE_PATH_TOOLS = new Set(['Edit', 'MultiEdit', 'Write', 'NotebookEdit', 'edit', 'write']) +const FILE_PATH_KEY = /"(?:file_path|filePath|path|notebook_path)"\s*:\s*"((?:[^"\\]|\\.)*)"/g + +function decodeJsonString(raw: string): string | null { + try { + const value: unknown = JSON.parse(`"${raw}"`) + return typeof value === 'string' && value.length > 0 ? value : null + } catch { + return null + } +} + +interface ChangedPath { + operations: string[] + spanIds: string[] +} + +function recordChangedPath( + into: Map, + path: string, + operation: string, + spanId: string, +): void { + const entry = into.get(path) ?? { operations: [], spanIds: [] } + if (!entry.operations.includes(operation)) entry.operations.push(operation) + if (!entry.spanIds.includes(spanId)) entry.spanIds.push(spanId) + into.set(path, entry) +} + +/** The paths one `file.change` span states, with `move_path` as its own path. */ +function harnessChangedPaths(span: OtlpSpan): { path: string; operation: string }[] { + const input = inputValue(span) + if (input === null) return [] + let parsed: unknown + try { + parsed = JSON.parse(input) + } catch { + return [] + } + if (parsed === null || typeof parsed !== 'object') return [] + const record = parsed as Record + const path = typeof record.path === 'string' ? record.path.trim() : '' + if (path.length === 0) return [] + const kind = stringAttr(span, FILE_CHANGE_KIND_ATTR) ?? (typeof record.kind === 'string' ? record.kind : '') + const movePath = typeof record.move_path === 'string' ? record.move_path.trim() : '' + return [ + { path, operation: kind.length > 0 ? kind : 'update' }, + ...(movePath.length > 0 ? [{ path: movePath, operation: 'move' }] : []), + ] +} + +/** + * Record a path recovered from a tool's own arguments, deferring to the + * harness's record of the same edit. + * + * A header that names a file the harness already recorded joins that entry + * rather than opening a second one, so an edit both sources saw is one changed + * file carrying both spans as evidence — including when the header wrote the + * path relative to the directory the harness resolved it against. When several + * recorded paths end that way, the harness has the edit and which of them this + * header names cannot be decided, so no entry claims the span. + * + * Returns false when the path was dropped for holding an unexpanded variable. + */ +function recordRecoveredPath( + into: Map, + resolved: readonly string[], + path: string, + operation: string, + spanId: string, +): boolean { + if (UNRESOLVED_PATH.test(path)) return false + const known = resolved.filter((candidate) => candidate === path || candidate.endsWith(`/${path}`)) + if (known.length > 1) return true + recordChangedPath(into, known[0] ?? path, operation, spanId) + return true +} + +function changedFilesOf(toolSpans: readonly OtlpSpan[], spans: readonly OtlpSpan[]): { + files: ChangedFileFact[] + truncatedInputs: number + unresolvedPaths: number +} { + const paths = new Map() + // The harness's own record first, so the recovery below can defer to it. A + // record the harness marked failed or declined changed no file. + for (const span of spans) { + if (span.name !== FILE_CHANGE_SPAN || span.status.code === 'ERROR') continue + for (const change of harnessChangedPaths(span)) { + recordChangedPath(paths, change.path, change.operation, span.span_id) + } + } + const resolved = [...paths.keys()] + let truncatedInputs = 0 + let unresolvedPaths = 0 + for (const span of toolSpans) { + const input = inputValue(span) + if (input === null) continue + let matched = false + for (const match of input.matchAll(PATCH_HEADER)) { + const operation = PATCH_OPERATION[match[1]!] + const path = match[2]?.trim() + if (!operation || !path) continue + matched = true + if (!recordRecoveredPath(paths, resolved, path, operation, span.span_id)) unresolvedPaths += 1 + } + if (!matched && FILE_PATH_TOOLS.has(toolName(span))) { + for (const match of input.matchAll(FILE_PATH_KEY)) { + const path = decodeJsonString(match[1] ?? '') + if (!path) continue + matched = true + if (!recordRecoveredPath(paths, resolved, path, 'update', span.span_id)) unresolvedPaths += 1 + } + } + if (matched && inputTruncated(span)) truncatedInputs += 1 + } + const files = [...paths] + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .map(([path, entry]) => ({ path, operations: entry.operations, spanIds: entry.spanIds })) + return { files, truncatedInputs, unresolvedPaths } +} + +function subagentsOf(spans: readonly OtlpSpan[]): SubagentSpawnFact[] { + // A lifecycle span records the subagent's own path; joining it to the spawn + // call by that path gives the reader both spans to check the task name with. + const lifecycleByPath = new Map() + for (const span of spans) { + const path = stringAttr(span, 'traces.codex.subagent_path') + if (path === null || !isSynthesizedSpan(span)) continue + lifecycleByPath.set(path, [...(lifecycleByPath.get(path) ?? []), span.span_id]) + } + const spawns: SubagentSpawnFact[] = [] + for (const span of spans) { + const isSpawn = + attr(span, 'traces.codex.agent_operation') === 'spawn_agent' || + stringAttr(span, 'traces.codex.spawn_agent_path') !== null + if (!isSpawn || isSynthesizedSpan(span)) continue + const taskName = stringAttr(span, 'traces.codex.spawn_agent_path') + spawns.push({ + taskName, + taskNameUnavailable: taskName === null + ? 'the spawn span carries no traces.codex.spawn_agent_path; the harness returned no task name for this call' + : null, + startedAt: span.start_time, + status: span.status.code, + spanIds: [span.span_id, ...(taskName ? lifecycleByPath.get(taskName) ?? [] : [])], + }) + } + return spawns +} + +function finalMessagesOf(spans: readonly OtlpSpan[]): FinalMessageFact[] { + const last = new Map() + for (const span of spans) { + let task: string | null | undefined + if (span.name === 'message.assistant') task = null + else if (span.name.startsWith('message.agent.')) { + task = stringAttr(span, 'traces.codex.agent_message_author') ?? 'unattributed subagent' + } + if (task === undefined) continue + const previous = last.get(task) + if (!previous || byTraceOrder(previous, span) <= 0) last.set(task, span) + } + return [...last] + .sort(([left], [right]) => (left === null ? -1 : right === null ? 1 : left.localeCompare(right))) + .map(([task, span]) => ({ + task, + at: span.end_time, + text: capText(typeof span.attributes.content === 'string' ? span.attributes.content : ''), + spanId: span.span_id, + })) +} + +function capList(items: readonly T[], cap = FACT_LIST_CAP): { kept: readonly T[]; partial?: string } { + if (items.length <= cap) return { kept: items } + return { + kept: items.slice(0, cap), + partial: `${items.length - cap} of ${items.length} entries omitted; the count above the list is complete`, + } +} + +function listFact(items: readonly T[], spanIds: readonly string[], cap = FACT_LIST_CAP): SessionFact { + const { kept, partial } = capList(items, cap) + return { value: kept, spanIds: [...new Set(spanIds)], unavailable: null, ...(partial ? { partial } : {}) } +} + +/** The changed-files fact, stating every way the list is known to be short. */ +function changedFilesFact( + files: readonly ChangedFileFact[], + truncatedInputs: number, + unresolvedPaths: number, +): SessionFact { + const fact = listFact(files, files.flatMap((entry) => entry.spanIds), CHANGED_FILE_LIST_CAP) + const gaps = [ + ...(fact.partial ? [fact.partial] : []), + ...(truncatedInputs > 0 + ? [ + `${truncatedInputs} contributing tool span(s) had truncated input; ` + + 'paths named after the cut are not in this list', + ] + : []), + ...(unresolvedPaths > 0 + ? [ + `${unresolvedPaths} recovered path(s) still held an unexpanded variable and were dropped; ` + + 'each names a file only if the harness also recorded that change', + ] + : []), + ] + return { ...fact, ...(gaps.length > 0 ? { partial: gaps.join('; ') } : {}) } +} + +/** + * Why a `user.prompt` turn is not a human turn of this session. + * + * The actors come from the adapters, which classify each turn from the + * harness's own signals; the reasons here say, in the sheet's own voice, what + * each classification means for the count. They match the audit rule a person + * would apply by hand: a user message is one the human typed, and instruction + * files, environment-context blocks, system reminders, tool results, subagent + * notifications, turn-aborted markers and skill or slash-command expansions are + * the harness feeding the model. + */ +const TURN_EXCLUSION_REASONS: Readonly> = { + injected: + 'harness-injected content rather than typed text: an instruction file, an environment-context block, ' + + 'a system reminder, a subagent notification, a turn-aborted marker, or a skill or slash-command expansion', + agent: 'a parent agent sent this prompt to this session; no person typed it', + 'subagent-spawn': 'the brief that spawned this run, written by the agent that spawned it', + 'tool-result': 'a tool result the harness surfaced as a user turn', + unclassified: 'the span carries no actor, so whether a person typed it cannot be read from the spans', +} + +const INHERITED_TURN_REASON = + 'context this session carries but did not receive: history copied from the parent when the session forked, ' + + 'or the turns a compaction retained. The person typed it into the parent thread, not into this session' + +const DUPLICATE_TURN_REASON = + 'a second record of the turn before it: the same text at the same instant, with no model call, tool call ' + + 'or assistant message between them' + +/** Spans that mean the agent acted: a person cannot have typed twice across one. */ +function isAgentActivity(span: OtlpSpan): boolean { + if (span.name === 'user.prompt') return false + const kind = attr(span, OPENINFERENCE_SPAN_KIND) + return kind === 'TOOL' || kind === 'LLM' || span.name === 'message.assistant' || span.name.startsWith('message.agent.') +} + +function turnText(span: OtlpSpan): string { + return capText(typeof span.attributes.content === 'string' ? span.attributes.content : '') +} + +function normalizedTurnText(span: OtlpSpan): string { + return turnText(span).replace(/\s+/g, ' ').trim() +} + +/** + * The human turns of one session, and every `user.prompt` span left out of them. + * + * Three filters, in order: a turn the session inherited rather than received, a + * turn whose actor is not a person, and a second record of the turn before it. + * `spans` is the whole trace in order, because the duplicate test asks what + * happened between two turns. + */ +function humanTurnsOf(spans: readonly OtlpSpan[]): { + turns: SessionTurnFact[] + excluded: ExcludedTurnFact[] +} { + const excluded = new Map() + const drop = (reason: string, spanId: string): void => { + excluded.set(reason, [...(excluded.get(reason) ?? []), spanId]) + } + const kept: OtlpSpan[] = [] + let sinceKept: OtlpSpan[] = [] + for (const span of spans) { + if (span.name !== 'user.prompt') { + sinceKept.push(span) + continue + } + if (span.attributes[INHERITED_SPAN_ATTR] === true) { + const source = stringAttr(span, INHERITED_SOURCE_ATTR) + drop(source === null ? INHERITED_TURN_REASON : `${INHERITED_TURN_REASON} (${source})`, span.span_id) + continue + } + const actor = stringAttr(span, ACTOR_ATTR) ?? 'unclassified' + if (actor !== 'human') { + drop(TURN_EXCLUSION_REASONS[actor] ?? `the adapter classified this turn as "${actor}", not as a person typing`, span.span_id) + continue + } + const previous = kept[kept.length - 1] + // Two records of ONE submission describe one instant, so they carry the + // same start time; a person who sends the same short message twice is + // seconds apart. Pairing records that are merely close is the adapter's + // job, where the records themselves say which log each came from — a sheet + // that guessed from the spans collapsed a measured "continue, continue" + // typed 1.8 s apart into one turn. + const duplicate = + previous !== undefined && + previous.start_time === span.start_time && + normalizedTurnText(previous) === normalizedTurnText(span) && + normalizedTurnText(span).length > 0 && + !sinceKept.some(isAgentActivity) + if (duplicate) { + drop(DUPLICATE_TURN_REASON, span.span_id) + continue + } + kept.push(span) + sinceKept = [] + } + return { + turns: kept.map((span) => ({ actor: 'human', at: span.start_time, text: turnText(span), spanId: span.span_id })), + excluded: [...excluded] + .sort(([left], [right]) => left.localeCompare(right)) + .map(([reason, spanIds]) => ({ reason, turns: spanIds.length, spanIds })), + } +} + +/** + * Compute the facts sheet for every trace in `spans`. One trace is one session. + * Deterministic and free: the same spans always produce the same sheet, and no + * model, network call, or budget is involved. + */ +export function computeSessionFacts(spans: readonly OtlpSpan[]): SessionFacts[] { + const { sessionByTrace } = indexSessionIdsByTrace(spans) + const byTrace = new Map() + for (const span of spans) { + byTrace.set(span.trace_id, [...(byTrace.get(span.trace_id) ?? []), span]) + } + return [...byTrace] + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .map(([traceId, traceSpans]) => sessionFactsForTrace(traceId, sessionByTrace.get(traceId) ?? null, traceSpans)) +} + +function sessionFactsForTrace( + traceId: string, + sessionId: string | null, + unordered: readonly OtlpSpan[], +): SessionFacts { + const spans = [...unordered].sort(byTraceOrder) + const harness = spans.map((span) => stringAttr(span, 'service.name')).find((name) => name !== null) ?? null + + const toolSpans = spans.filter(isToolSpan) + const synthesized = toolSpans.filter(isSynthesizedSpan) + const invoked = toolSpans.filter((span) => !isSynthesizedSpan(span)) + const byName: Record = {} + for (const span of invoked) byName[toolName(span)] = (byName[toolName(span)] ?? 0) + 1 + + const subagents = subagentsOf(spans) + const pullRequests = readPullRequests(spans) + const promptSpans = spans.filter((span) => span.name === 'user.prompt') + const { turns: humanTurns, excluded: excludedTurns } = humanTurnsOf(spans) + const actorCounts = new Map() + for (const span of promptSpans) { + const actor = stringAttr(span, ACTOR_ATTR) ?? 'unclassified' + actorCounts.set(actor, [...(actorCounts.get(actor) ?? []), span.span_id]) + } + const finalMessages = finalMessagesOf(spans) + const { files, truncatedInputs, unresolvedPaths } = changedFilesOf(invoked, spans) + + const starts = spans.filter((span) => Number.isFinite(Date.parse(span.start_time))) + const ends = spans.filter((span) => Number.isFinite(Date.parse(span.end_time))) + const firstSpan = starts.reduce( + (best, span) => (best === null || Date.parse(span.start_time) < Date.parse(best.start_time) ? span : best), + null, + ) + const lastSpan = ends.reduce( + (best, span) => (best === null || Date.parse(span.end_time) > Date.parse(best.end_time) ? span : best), + null, + ) + + const tokenSpan = spans.find((span) => typeof span.attributes[SESSION_TOKEN_TOTAL_ATTR] === 'number') + const receipts = spans.filter((span) => span.attributes[CORRUPTION_RECEIPT_VERSION_ATTR] !== undefined) + const recordSpans = spans.filter( + (span) => span.parent_span_id !== null && span.attributes[CORRUPTION_RECEIPT_VERSION_ATTR] === undefined, + ).length + const corruptionSpan = spans.find((span) => typeof span.attributes[CORRUPTION_COUNT_ATTR] === 'number') + + return { + schemaVersion: 1, + kind: 'traces.session_facts', + traceId, + sessionId, + harness, + spanCount: spans.length, + recordSpans, + unreadRecords: corruptionSpan + ? { + value: corruptionSpan.attributes[CORRUPTION_COUNT_ATTR] as number, + spanIds: [corruptionSpan.span_id, ...receipts.map((span) => span.span_id)], + unavailable: null, + } + : { + value: null, + spanIds: [], + unavailable: + `no span carries ${CORRUPTION_COUNT_ATTR}; this trace was not read through the session ` + + 'integrity path, so whether any record was skipped is unknown', + }, + toolCalls: { + value: invoked.length, + spanIds: invoked.map((span) => span.span_id), + unavailable: null, + }, + synthesizedToolSpans: { + value: synthesized.length, + spanIds: synthesized.map((span) => span.span_id), + unavailable: null, + }, + toolCallsByName: { + value: byName, + spanIds: invoked.map((span) => span.span_id), + unavailable: null, + }, + subagents: listFact(subagents, subagents.flatMap((entry) => entry.spanIds)), + pullRequests: pullRequests.facts + ? { + value: pullRequests.facts, + spanIds: pullRequests.spanIds, + unavailable: null, + ...(pullRequests.partial ? { partial: pullRequests.partial } : {}), + } + : { value: null, spanIds: [], unavailable: pullRequests.unavailable ?? 'unread' }, + humanTurns: listFact(humanTurns, humanTurns.map((turn) => turn.spanId)), + excludedTurns: listFact(excludedTurns, excludedTurns.flatMap((entry) => entry.spanIds)), + turnsByActor: listFact( + [...actorCounts] + .sort(([left], [right]) => left.localeCompare(right)) + .map(([actor, spanIds]) => ({ actor, turns: spanIds.length, spanIds })), + promptSpans.map((span) => span.span_id), + ), + finalMessages: listFact(finalMessages, finalMessages.map((entry) => entry.spanId)), + changedFiles: changedFilesFact(files, truncatedInputs, unresolvedPaths), + firstRecordAt: firstSpan + ? { value: firstSpan.start_time, spanIds: [firstSpan.span_id], unavailable: null } + : { value: null, spanIds: [], unavailable: 'no span in this trace carries a parseable start time' }, + lastRecordAt: lastSpan + ? { value: lastSpan.end_time, spanIds: [lastSpan.span_id], unavailable: null } + : { value: null, spanIds: [], unavailable: 'no span in this trace carries a parseable end time' }, + tokenTotal: tokenSpan + ? { + value: tokenSpan.attributes[SESSION_TOKEN_TOTAL_ATTR] as number, + spanIds: [tokenSpan.span_id], + unavailable: null, + } + : { + value: null, + spanIds: [], + unavailable: + `no span carries ${SESSION_TOKEN_TOTAL_ATTR}; the per-turn llm.turn deltas in this trace ` + + 'are a different quantity and summing them would not be the harness total', + }, + } +} + +/** The facts sheet for a set of spans, as one report. */ +export function buildSessionFactsReport( + spans: readonly OtlpSpan[], + options: { harness?: string | null; generatedAt?: string } = {}, +): SessionFactsReport { + return { + schemaVersion: 1, + kind: 'traces.session_facts_report', + generatedAt: options.generatedAt ?? new Date().toISOString(), + harness: options.harness ?? null, + spanCount: spans.length, + sessions: computeSessionFacts(spans), + } +} + +function factLine(label: string, fact: SessionFact, rendered: string): string { + if (fact.value === null) return ` ${label}: unavailable — ${fact.unavailable}` + const partial = fact.partial ? ` (partial: ${fact.partial})` : '' + return ` ${label}: ${rendered}${partial}` +} + +/** The short readable form: the same facts, one screen, no span-id lists. */ +export function renderSessionFacts(report: SessionFactsReport): string { + const lines: string[] = [ + `session facts — ${report.sessions.length} session(s), ${report.spanCount} span(s), deterministic, $0`, + ] + for (const facts of report.sessions) { + lines.push('', `${facts.sessionId ?? facts.traceId} [${facts.harness ?? 'unknown harness'}] ${facts.spanCount} spans`) + lines.push( + factLine( + 'tool calls', + facts.toolCalls, + `${facts.toolCalls.value} (${facts.synthesizedToolSpans.value ?? 0} synthesized span(s) excluded)`, + ), + ) + lines.push( + factLine( + 'subagents', + facts.subagents, + `${facts.subagents.value?.length ?? 0}${ + facts.subagents.value?.length + ? `: ${facts.subagents.value.map((entry) => entry.taskName ?? '').join(', ')}` + : '' + }`, + ), + ) + lines.push( + factLine( + 'pull requests', + facts.pullRequests, + `${facts.pullRequests.value?.created.length ?? 0} created${ + facts.pullRequests.value?.created.length + ? ` (${facts.pullRequests.value.created.map((entry) => entry.identifier ?? 'unidentified').join(', ')})` + : '' + }, ${facts.pullRequests.value?.merged.length ?? 0} merged${ + facts.pullRequests.value?.merged.length + ? ` (${facts.pullRequests.value.merged.map((entry) => entry.identifier ?? 'unidentified').join(', ')})` + : '' + }`, + ), + ) + lines.push(factLine('human turns', facts.humanTurns, String(facts.humanTurns.value?.length ?? 0))) + for (const entry of facts.excludedTurns.value ?? []) { + lines.push(` turns not counted: ${entry.turns} — ${entry.reason}`) + } + lines.push( + factLine( + 'turns by actor', + facts.turnsByActor, + facts.turnsByActor.value?.length + ? facts.turnsByActor.value.map((entry) => `${entry.actor} ${entry.turns}`).join(', ') + : 'none', + ), + ) + lines.push(factLine('changed files', facts.changedFiles, String(facts.changedFiles.value?.length ?? 0))) + lines.push(factLine('first record', facts.firstRecordAt, String(facts.firstRecordAt.value))) + lines.push(factLine('last record', facts.lastRecordAt, String(facts.lastRecordAt.value))) + lines.push(factLine('token total', facts.tokenTotal, String(facts.tokenTotal.value))) + if ((facts.unreadRecords.value ?? 0) > 0) { + lines.push(` unread records: ${facts.unreadRecords.value} (the facts above are computed from the rest)`) + } + for (const message of facts.finalMessages.value ?? []) { + lines.push(` final message [${message.task ?? 'session agent'}]: ${message.text.split('\n')[0]?.slice(0, 160) ?? ''}`) + } + } + return `${lines.join('\n')}\n` +} + +/** + * The byte ceiling one prepared-context string stays under. + * + * `DEFAULT_TRACE_ANALYST_BUDGETS.perCallByteCeiling` is 150,000 — the size one + * trace-tool result may return. Prepared context is read by the same model in + * the same turn as those results, so the sheet claims a fifth of that ceiling + * and leaves the rest for the tool calls the model still has to make. + */ +export const PREPARED_CONTEXT_BYTE_CEILING = 30_000 + +/** + * Render the sheet as bounded prepared context. + * + * Prepared context is not evidence: it is a head start. The rendering therefore + * keeps every field's span ids so that each fact can be checked against the raw + * spans, and says out loud that the sheet itself is not citable. + * + * The result is guaranteed to be at most `byteCeiling` bytes. Fields are shed + * from the largest list downward, and the shed is reported inside the context + * rather than silently applied. + */ +export function renderSessionFactsContext( + report: SessionFactsReport, + options: { byteCeiling?: number } = {}, +): string { + const ceiling = options.byteCeiling ?? PREPARED_CONTEXT_BYTE_CEILING + const header = + 'SESSION FACTS (deterministic, computed from the spans, no model call). ' + + 'Each fact names the span ids it came from: cite those spans, never this sheet. ' + + 'A null value carries the reason the spans cannot support it — do not replace it with a guess.' + let sessions = report.sessions.map(compactFacts) + const encode = (dropped: string[]): string => + `${header}\n${JSON.stringify({ sessions, ...(dropped.length ? { omitted_fields: dropped } : {}) })}` + + const dropped: string[] = [] + // Shed in reverse usefulness order: text first, then the long id lists, then + // whole list fields. The counts, which answer most questions, are last to go. + const sheds: Array<[string, (facts: CompactFacts) => void]> = [ + ['human_turn_text', (facts) => { for (const turn of facts.human_turns) delete turn.text }], + ['final_message_text', (facts) => { for (const message of facts.final_messages) delete message.text }], + ['pull_request_commands', (facts) => { + for (const entry of [...(facts.pull_requests.created ?? []), ...(facts.pull_requests.merged ?? [])]) { + delete entry.command + delete entry.span_ids + } + }], + ['tool_call_span_ids', (facts) => { facts.tool_calls.span_ids = [] }], + ['tool_calls_by_name', (facts) => { delete facts.tool_calls_by_name }], + ['excluded_turn_span_ids', (facts) => { + facts.excluded_turns = facts.excluded_turns?.map((entry) => { + const { span_ids: _dropped, ...rest } = entry as { span_ids?: readonly string[] } + return rest + }) + }], + ['changed_files', (facts) => { facts.changed_files = { count: facts.changed_files.count } }], + ['turns_by_actor', (facts) => { delete facts.turns_by_actor }], + ['pull_request_evidence', (facts) => { + for (const entry of [...(facts.pull_requests.created ?? []), ...(facts.pull_requests.merged ?? [])]) { + delete entry.evidence + } + }], + ['excluded_turns', (facts) => { delete facts.excluded_turns }], + ['subagents', (facts) => { facts.subagents = { count: facts.subagents.count } }], + ['final_messages', (facts) => { facts.final_messages = [] }], + ['human_turns', (facts) => { facts.human_turns = [] }], + ] + for (const [name, shed] of sheds) { + if (Buffer.byteLength(encode(dropped)) <= ceiling) break + for (const facts of sessions) shed(facts) + dropped.push(name) + } + let text = encode(dropped) + // A trace set too large to describe at all keeps the sessions it can fit. + while (Buffer.byteLength(text) > ceiling && sessions.length > 1) { + sessions = sessions.slice(0, Math.max(1, Math.floor(sessions.length / 2))) + text = encode([...dropped, `sessions_beyond_${sessions.length}`]) + } + if (Buffer.byteLength(text) <= ceiling) return text + // Below one session's smallest record the sheet says so and supplies nothing, + // rather than returning a fragment that reads like a complete sheet. The + // ceiling is a guarantee, so an unusable one produces no context at all. + const refusal = `${header}\n${JSON.stringify({ sessions: [], omitted_fields: ['all: the byte ceiling is below one session record'] })}` + return Buffer.byteLength(refusal) <= ceiling ? refusal : '' +} + +interface CompactPullRequest { + id: string | null + number: string | null + head_branch: string | null + evidence?: string + command?: string + span_ids?: readonly string[] + unavailable?: string +} + +/** Either the two lists, or a null reading with the reason. Never both. */ +interface CompactPullRequests { + created?: CompactPullRequest[] + merged?: CompactPullRequest[] + value?: null + unavailable?: string + partial?: string +} + +interface CompactFacts { + session_id: string | null + trace_id: string + spans: number + tool_calls: { count: number | null; synthesized_excluded: number | null; span_ids: readonly string[] } + tool_calls_by_name?: Readonly> | null + subagents: { count: number; entries?: readonly unknown[] } + pull_requests: CompactPullRequests + human_turns: Array<{ at: string; span_id: string; text?: string }> + excluded_turns?: readonly unknown[] | null + turns_by_actor?: readonly unknown[] | null + final_messages: Array<{ task: string | null; at: string; span_id: string; text?: string }> + changed_files: { count: number; paths?: readonly unknown[]; partial?: string } + first_record_at: unknown + last_record_at: unknown + token_total: unknown +} + +function factJson(fact: SessionFact): unknown { + return fact.value === null + ? { value: null, unavailable: fact.unavailable } + : { value: fact.value, span_ids: fact.spanIds } +} + +function compactPullRequests(fact: SessionFact): CompactPullRequests { + if (fact.value === null) return { value: null, unavailable: fact.unavailable ?? 'unread' } + const entry = (pr: PullRequestFacts['created'][number]): CompactPullRequest => ({ + id: pr.identifier, + number: pr.number, + head_branch: pr.headBranch, + evidence: pr.evidence, + command: pr.command, + span_ids: pr.spanIds, + ...(pr.unavailable ? { unavailable: pr.unavailable } : {}), + }) + return { + created: fact.value.created.map(entry), + merged: fact.value.merged.map(entry), + ...(fact.partial ? { partial: fact.partial } : {}), + } +} + +function compactFacts(facts: SessionFacts): CompactFacts { + return { + session_id: facts.sessionId, + trace_id: facts.traceId, + spans: facts.spanCount, + tool_calls: { + count: facts.toolCalls.value, + synthesized_excluded: facts.synthesizedToolSpans.value, + span_ids: facts.toolCalls.spanIds, + }, + tool_calls_by_name: facts.toolCallsByName.value, + subagents: { + count: facts.subagents.value?.length ?? 0, + entries: facts.subagents.value?.map((entry) => ({ + task_name: entry.taskName, + ...(entry.taskNameUnavailable ? { task_name_unavailable: entry.taskNameUnavailable } : {}), + started_at: entry.startedAt, + status: entry.status, + span_ids: entry.spanIds, + })), + }, + pull_requests: compactPullRequests(facts.pullRequests), + human_turns: (facts.humanTurns.value ?? []).map((turn) => ({ + at: turn.at, + span_id: turn.spanId, + text: turn.text, + })), + excluded_turns: facts.excludedTurns.value?.map((entry) => ({ + reason: entry.reason, + turns: entry.turns, + span_ids: entry.spanIds, + })), + turns_by_actor: facts.turnsByActor.value?.map((entry) => ({ + actor: entry.actor, + turns: entry.turns, + span_ids: entry.spanIds, + })), + final_messages: (facts.finalMessages.value ?? []).map((message) => ({ + task: message.task, + at: message.at, + span_id: message.spanId, + text: message.text, + })), + changed_files: { + count: facts.changedFiles.value?.length ?? 0, + paths: facts.changedFiles.value?.map((file) => ({ + path: file.path, + operations: file.operations, + span_ids: file.spanIds, + })), + ...(facts.changedFiles.partial ? { partial: facts.changedFiles.partial } : {}), + }, + first_record_at: factJson(facts.firstRecordAt), + last_record_at: factJson(facts.lastRecordAt), + token_total: factJson(facts.tokenTotal), + } +} + +/** + * The prepared-context string for one set of spans, or undefined when there is + * nothing to prepare. Analyst definitions install this through + * `TraceAnalystDefinition.prepareContext`, which runs before the model does. + */ +export function sessionFactsContext( + spans: readonly OtlpSpan[], + options: { byteCeiling?: number; generatedAt?: string } = {}, +): string | undefined { + if (spans.length === 0) return undefined + const report = buildSessionFactsReport(spans, { + ...(options.generatedAt ? { generatedAt: options.generatedAt } : {}), + }) + if (report.sessions.length === 0) return undefined + return renderSessionFactsContext(report, options) +} + +/** + * Version suffix stamped on a definition that receives the sheet. + * + * `createTraceAnalyst` records `prepare_context: "version-bound"` in an + * analyst's exact-run identity, which is a contract: a definition whose + * prepared context changed is a different analyst. Wrapping therefore bumps the + * version instead of quietly changing what the same version does. + */ +export const SESSION_FACTS_VERSION_SUFFIX = 'session-facts.1' + +/** + * Supply the sheet to every definition as prepared context, before the model + * runs. A definition that already prepares context keeps it; the sheet is + * appended after it, bounded by its own ceiling, so the pair stays inside the + * documented `perCallByteCeiling` of 150,000 bytes. + * + * The sheet is computed once for the whole set: it is deterministic, so every + * definition receives the identical text. + */ +export function withSessionFactsContext( + definitions: readonly TraceAnalystDefinition[], + spans: readonly OtlpSpan[], + options: { byteCeiling?: number } = {}, +): TraceAnalystDefinition[] { + const sheet = sessionFactsContext(spans, options) + if (sheet === undefined) return [...definitions] + return definitions.map((definition) => ({ + ...definition, + version: `${definition.version}+${SESSION_FACTS_VERSION_SUFFIX}`, + prepareContext: async (store, context) => { + const own = await definition.prepareContext?.(store, context) + return own ? `${own}\n\n${sheet}` : sheet + }, + })) +} diff --git a/src/session-relationship.ts b/src/session-relationship.ts index a71ce7c..49ac061 100644 --- a/src/session-relationship.ts +++ b/src/session-relationship.ts @@ -95,6 +95,8 @@ export function describeSessionRelationship( spans: readonly OtlpSpan[], ): SessionRelationship { const root = sessionRoot(ref, spans) + const sessionId = sessionIdFromAttributes(root?.attributes ?? {}) ?? root?.trace_id ?? ref.sessionId + const parentSessionId = stringAttribute(root, 'traces.parent_session_id') const childSessionIds = new Set() const spawnedChildSessionIds = new Set() const resumedChildSessionIds = new Set() @@ -125,8 +127,14 @@ export function describeSessionRelationship( } } + // A child that messages its parent (`send_message`, `followup_task`) names the + // parent as a target; the session and its parent are never its own children. + for (const ids of [childSessionIds, spawnedChildSessionIds, resumedChildSessionIds]) { + ids.delete(sessionId) + if (parentSessionId) ids.delete(parentSessionId) + } + const role = stringAttribute(root, 'traces.session.role') - const parentSessionId = stringAttribute(root, 'traces.parent_session_id') const depth = numberAttribute(root, 'traces.codex.agent_depth') const agentNickname = stringAttribute(root, 'traces.codex.agent_nickname') const agentRole = stringAttribute(root, 'traces.codex.agent_role') @@ -140,7 +148,7 @@ export function describeSessionRelationship( : undefined const turnId = stringAttribute(root, 'traces.codex.turn_id') return { - sessionId: sessionIdFromAttributes(root?.attributes ?? {}) ?? root?.trace_id ?? ref.sessionId, + sessionId, role: role === 'operator' || role === 'child' ? role : 'unknown', ...(parentSessionId ? { parentSessionId } : {}), childSessionIds: [...childSessionIds].sort(), diff --git a/src/shell-commands.ts b/src/shell-commands.ts new file mode 100644 index 0000000..16f2363 --- /dev/null +++ b/src/shell-commands.ts @@ -0,0 +1,359 @@ +/** + * Split one shell script into the simple commands it would actually run. + * + * A trace records a command as the script the agent handed to `/bin/zsh -lc`, + * and one script routinely runs several commands: `git push && gh pr create`, + * a `cat > file <<'EOF' … EOF` heredoc followed by a `gh` call, a `sh -c` + * wrapper around another script. Asking "did this session create a pull + * request?" with a regular expression over that text answers yes for a heredoc + * body that merely mentions `gh pr create`, and no for a command hidden behind + * a `$( … )` substitution. + * + * This scanner answers the question the way the shell would: it walks the + * script once, tracking quoting, comments, redirections and heredoc bodies, and + * returns the word list of every simple command it finds — recursing into + * command substitutions, subshells, and the `-c` argument of a nested shell. + * + * It is a reader, not an interpreter. Variables are not expanded, globs are not + * matched, and a word built from a variable comes back as the literal text of + * the script. A caller therefore learns which command *names* ran with which + * *literal* arguments, which is exactly what an audit of `gh pr create` needs + * and is never mistaken for a claim about what the shell computed. + */ + +/** Interpreters whose `-c` argument is another script worth scanning. */ +const SHELLS: ReadonlySet = new Set(['sh', 'bash', 'zsh', 'dash', 'ksh', 'ash', 'busybox']) + +/** + * Words that precede a command without being it. `env FOO=1 gh pr create` runs + * `gh`, and an audit that stopped at `env` would miss it. + */ +const COMMAND_PREFIXES: ReadonlySet = new Set([ + 'env', 'command', 'builtin', 'exec', 'nohup', 'time', 'sudo', 'doas', 'nice', 'stdbuf', 'setsid', 'then', 'do', 'else', 'elif', 'if', 'while', 'until', '!', '{', '(', +]) + +/** A leading `NAME=value` word is an environment assignment, not the command. */ +const ASSIGNMENT = /^[A-Za-z_][A-Za-z0-9_]*=/ + +/** Recursion limit for `$( … )`, backticks, subshells and nested `sh -c`. */ +export const MAX_SHELL_DEPTH = 4 + +/** Words a scanned script may grow to before the scanner stops, so a pathological + * input cannot make one span's parse unbounded. */ +const MAX_WORDS = 20_000 + +function basename(word: string): string { + const cut = word.lastIndexOf('/') + return cut === -1 ? word : word.slice(cut + 1) +} + +class ShellScanner { + private readonly text: string + private readonly length: number + private readonly depth: number + private readonly commands: string[][] = [] + private word: string[] = [] + private wordStarted = false + private current: string[] = [] + private words = 0 + private heredocs: Array<{ delimiter: string; stripTabs: boolean }> = [] + + constructor(text: string, depth: number) { + this.text = text + this.length = text.length + this.depth = depth + } + + run(): string[][] { + let index = 0 + while (index < this.length && this.words < MAX_WORDS) { + index = this.step(index) + } + this.endCommand() + return this.commands + } + + private step(index: number): number { + const text = this.text + const char = text[index]! + if (char === '\\') { + if (index + 1 >= this.length) return index + 1 + if (text[index + 1] !== '\n') this.push(text[index + 1]!) + return index + 2 + } + if (char === "'") return this.readSingleQuote(index) + if (char === '"') return this.readDoubleQuote(index) + if (char === '`') return this.readBackTick(index) + if (char === '$' && text[index + 1] === '(' && text[index + 2] === '(') { + // Arithmetic expansion holds no commands; keep it in the word verbatim. + const close = text.indexOf('))', index + 3) + const end = close === -1 ? this.length : close + 2 + this.pushText(text.slice(index, end)) + return end + } + if (char === '$' && text[index + 1] === '(') return this.readSubstitution(index + 2) + if (char === '$' && text[index + 1] === '{') { + const end = this.findBalanced(index + 2, '{', '}') + 1 + this.pushText(text.slice(index, end)) + return end + } + if (char === '#' && !this.wordStarted) { + const newline = text.indexOf('\n', index) + return newline === -1 ? this.length : newline + } + if (char === '<' && text[index + 1] === '<' && text[index + 2] !== '<') return this.readHeredocHeader(index) + if (char === '>' || char === '<') return this.readRedirect(index) + if (char === '\n') { + this.endCommand() + return this.heredocs.length > 0 ? this.skipHeredocBodies(index + 1) : index + 1 + } + if (char === ';' || char === '&' || char === '|' || char === '(' || char === ')') { + this.endCommand() + return index + 1 + } + if (char === ' ' || char === '\t' || char === '\r') { + this.endWord() + return index + 1 + } + // `2>file` and `1>&2`: the digits belong to the redirection, not to a word. + if (char >= '0' && char <= '9' && !this.wordStarted && (text[index + 1] === '>' || text[index + 1] === '<')) { + return this.readRedirect(index + 1) + } + this.push(char) + return index + 1 + } + + private push(char: string): void { + this.word.push(char) + this.wordStarted = true + } + + private pushText(text: string): void { + if (text.length > 0) this.word.push(text) + this.wordStarted = true + } + + private endWord(): void { + if (!this.wordStarted) return + this.current.push(this.word.join('')) + this.words += 1 + this.word = [] + this.wordStarted = false + } + + private endCommand(): void { + this.endWord() + if (this.current.length > 0) this.commands.push(this.normalize(this.current)) + this.current = [] + } + + /** Drop the leading assignments and prefix words so `w[0]` is the command name. */ + private normalize(words: string[]): string[] { + let start = 0 + while (start < words.length) { + const word = words[start]! + if (ASSIGNMENT.test(word) || COMMAND_PREFIXES.has(basename(word))) start += 1 + else break + } + // `env`-style prefixes may be the whole command (`exec`, a bare `{`); keep + // the original words rather than returning an empty command. + return start === 0 || start >= words.length ? words : words.slice(start) + } + + private readSingleQuote(index: number): number { + const end = this.text.indexOf("'", index + 1) + const stop = end === -1 ? this.length : end + this.pushText(this.text.slice(index + 1, stop)) + this.wordStarted = true + return stop + 1 + } + + private readDoubleQuote(index: number): number { + let cursor = index + 1 + this.wordStarted = true + while (cursor < this.length) { + const char = this.text[cursor]! + if (char === '"') return cursor + 1 + if (char === '\\') { + if (cursor + 1 < this.length && this.text[cursor + 1] !== '\n') this.push(this.text[cursor + 1]!) + cursor += 2 + continue + } + if (char === '`') { + cursor = this.readBackTick(cursor) + continue + } + if (char === '$' && this.text[cursor + 1] === '(') { + cursor = this.readSubstitution(cursor + 2) + continue + } + this.push(char) + cursor += 1 + } + return this.length + } + + /** Scan the commands inside `$( … )` and continue after the closing paren. */ + private readSubstitution(index: number): number { + const end = this.findBalanced(index, '(', ')') + this.descend(this.text.slice(index, end)) + this.wordStarted = true + return end + 1 + } + + private readBackTick(index: number): number { + const end = this.text.indexOf('`', index + 1) + const stop = end === -1 ? this.length : end + this.descend(this.text.slice(index + 1, stop)) + this.wordStarted = true + return stop + 1 + } + + private descend(script: string): void { + if (this.depth >= MAX_SHELL_DEPTH || script.length === 0) return + for (const command of shellCommands(script, this.depth + 1)) this.commands.push(command) + } + + private findBalanced(index: number, open: string, close: string): number { + let depth = 1 + let cursor = index + while (cursor < this.length) { + const char = this.text[cursor]! + if (char === '\\') cursor += 2 + else if (char === open) { + depth += 1 + cursor += 1 + } else if (char === close) { + depth -= 1 + if (depth === 0) return cursor + cursor += 1 + } else cursor += 1 + } + return this.length + } + + /** `<()'.includes(char)) break + delimiter.push(char) + cursor += 1 + } + if (delimiter.length > 0) this.heredocs.push({ delimiter: delimiter.join(''), stripTabs }) + return cursor + } + + private skipHeredocBodies(index: number): number { + let cursor = index + for (const { delimiter, stripTabs } of this.heredocs) { + while (cursor < this.length) { + const newline = this.text.indexOf('\n', cursor) + const rawLine = newline === -1 ? this.text.slice(cursor) : this.text.slice(cursor, newline) + const line = (stripTabs ? rawLine.replace(/^\t+/, '') : rawLine).replace(/\r$/, '') + cursor = newline === -1 ? this.length : newline + 1 + if (line === delimiter) break + } + } + this.heredocs = [] + return cursor + } + + /** Consume a redirection operator and its target so the target is not read as an argument. */ + private readRedirect(index: number): number { + this.endWord() + let cursor = index + while (cursor < this.length && '<>&'.includes(this.text[cursor]!)) cursor += 1 + while (cursor < this.length && (this.text[cursor] === ' ' || this.text[cursor] === '\t')) cursor += 1 + while (cursor < this.length && !' \t\n;&|()<>'.includes(this.text[cursor]!)) { + const char = this.text[cursor]! + if (char === "'" || char === '"') { + const close = this.text.indexOf(char, cursor + 1) + cursor = (close === -1 ? this.length : close) + 1 + continue + } + cursor += char === '\\' ? 2 : 1 + } + return cursor + } +} + +/** + * Every simple command in `script`, as word lists, in the order the scanner + * meets them. Commands inside substitutions and nested shells are included. + */ +export function shellCommands(script: string, depth = 0): string[][] { + if (depth > MAX_SHELL_DEPTH || script.length === 0) return [] + const commands = new ShellScanner(script, depth).run() + const out: string[][] = [] + for (const words of commands) { + out.push(words) + const name = basename(words[0] ?? '') + if (!SHELLS.has(name) || depth >= MAX_SHELL_DEPTH) continue + for (let index = 1; index < words.length; index += 1) { + const word = words[index]! + if (!word.startsWith('-') || word.startsWith('--') || !word.slice(1).includes('c')) continue + const nested = words[index + 1] + if (nested !== undefined) out.push(...shellCommands(nested, depth + 1)) + break + } + } + return out +} + +/** + * The script a tool span recorded, from its `input.value`. + * + * Adapters store a command as JSON — `{"command": ["/bin/zsh", "-lc", "…"]}` + * for a Codex `CommandExecution` item, `{"command": "…"}` for a Claude Code + * `Bash` call, `{"cmd": "…"}` for a Codex `exec_command` argument list. An argv + * array whose head is a shell keeps only the script that shell was given; any + * other array is joined back into one command line. A value that is not JSON, + * or JSON without a command field, yields undefined: the span records something + * other than an executed command, and guessing from its text is how a heredoc + * body becomes a pull request. + */ +export function commandTextFromInput(input: string): string | undefined { + const trimmed = input.trim() + if (!trimmed.startsWith('{')) return undefined + let parsed: unknown + try { + parsed = JSON.parse(trimmed) + } catch { + return undefined + } + if (!parsed || typeof parsed !== 'object') return undefined + const record = parsed as Record + const value = record.command ?? record.cmd + if (typeof value === 'string') return value.length > 0 ? value : undefined + if (!Array.isArray(value) || value.length === 0) return undefined + const words = value.filter((entry): entry is string => typeof entry === 'string') + if (words.length !== value.length) return undefined + const head = basename(words[0] ?? '') + if (SHELLS.has(head)) { + const flag = words.findIndex((word, position) => position > 0 && /^-[a-z]*c$/.test(word)) + if (flag !== -1 && words[flag + 1] !== undefined) return words[flag + 1] + } + return words.join(' ') +} diff --git a/tests/adapters.test.ts b/tests/adapters.test.ts index 3634b6b..e2e4bdf 100644 --- a/tests/adapters.test.ts +++ b/tests/adapters.test.ts @@ -1800,8 +1800,16 @@ describe('codex current tool and subagent events', () => { 'traces.codex.task_scope': 'fork-current', 'traces.codex.turn_id': currentTurnId, }) - expect(spans.filter((item) => item.name === 'user.prompt').map((item) => item.attributes.content)) + const prompts = spans.filter((item) => item.name === 'user.prompt') + expect(prompts.filter((item) => item.attributes['traces.session.inherited'] !== true) + .map((item) => item.attributes.content)) .toEqual(['current child prompt']) + // The pre-fork prefix is kept, marked, and left out of this turn's identity. + const inherited = prompts.filter((item) => item.attributes['traces.session.inherited'] === true) + expect(inherited.map((item) => item.attributes.content)).toEqual(['inherited parent prompt']) + expect(inherited[0]?.attributes['traces.session.inherited_source']).toBe('pre-task-prefix') + expect(inherited[0]?.attributes['traces.codex.turn_id']).toBeUndefined() + expect(spans[0]?.attributes['traces.session.inherited_span_count']).toBe(1) }) it('uses task timestamps when older child events omit started_at', async () => { @@ -2365,7 +2373,9 @@ describe('codex current tool and subagent events', () => { const spans = await new CodexAdapter().parse(refFor(path, 'codex')) const tools = spans.filter((item) => item.attributes['openinference.span.kind'] === 'TOOL') - expect(tools).toHaveLength(10) + // The two subagent lifecycles are AGENT spans, not calls the model made. + expect(tools).toHaveLength(8) + expect(tools.every((item) => item.attributes['traces.span.synthesized'] === undefined)).toBe(true) const verifications = tools.filter((item) => item.attributes['tool.name'] === 'exec_command.verify') expect(verifications).toHaveLength(2) const failedVerification = verifications.find((item) => item.status.code === 'ERROR') @@ -2401,8 +2411,12 @@ describe('codex current tool and subagent events', () => { expect(writeStdin?.attributes['traces.expected_blocking']).toBe(true) expect(writeStdin?.status.code).toBe('OK') - const agents = tools.filter((item) => item.attributes['tool.name'] === 'Agent') + const agents = spans.filter((item) => item.attributes['traces.span.synthesized'] === true) expect(agents).toHaveLength(2) + expect(agents.every((item) => item.name === 'subagent.lifecycle')).toBe(true) + expect(agents.every((item) => item.attributes['openinference.span.kind'] === 'AGENT')).toBe(true) + expect(agents.every((item) => item.attributes['tool.name'] === undefined)).toBe(true) + expect(agents.every((item) => item.attributes['traces.span.synthesized_from'] === 'codex.sub_agent_activity')).toBe(true) const agent = agents.find((item) => String(item.attributes['input.value']).includes('paper_audit')) expect(JSON.parse(String(agent?.attributes['input.value']))).toEqual({ subagent_type: 'paper_audit', diff --git a/tests/ask.test.ts b/tests/ask.test.ts new file mode 100644 index 0000000..8d23d30 --- /dev/null +++ b/tests/ask.test.ts @@ -0,0 +1,506 @@ +import { mkdtemp, readFile, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as sleep } from 'node:timers/promises' +import type { + TraceAnalysisEngine, + TraceAnalysisEngineRequest, + TraceAnalysisEngineResult, +} from '@tangle-network/agent-eval/analyst' +import { describe, expect, it } from 'vitest' +import { + loadTraceQuestionsFile, + MAX_TRACE_QUESTION_CHARS, + normalizeTraceQuestions, + renderTraceQuestionInstructions, + runTraceQuestions, + TRACE_QUESTION_PREVIEW_HEAD_CHARS, + traceCitationsInText, + writeTraceQuestionsArtifacts, +} from '../src/ask.js' +import { answerSchemaErrors, assertAnswerSchema, parseJsonAnswer } from '../src/answer-schema.js' +import { type OtlpSpan, span } from '../src/otlp.js' + +const TRACE = 'trace-ask' +const COMMAND = 'git status --short && pnpm test' + +/** A small synthetic session: one prompt, one shell command, one reply. */ +function fixtureSpans(): OtlpSpan[] { + const base = Date.parse('2026-02-01T10:00:00.000Z') + const at = (seconds: number) => new Date(base + seconds * 1000).toISOString() + return [ + span({ + traceId: TRACE, + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: at(0), + endTime: at(30), + service: 'synthetic', + extra: { 'session.id': 'session-ask' }, + }), + span({ + traceId: TRACE, + spanId: 'prompt-1', + parentSpanId: 'root', + name: 'user.prompt', + kind: 'CHAIN', + startTime: at(1), + service: 'synthetic', + content: 'Run the unit tests and tell me whether they pass.', + }), + span({ + traceId: TRACE, + spanId: 'tool-1', + parentSpanId: 'root', + name: 'tool.exec_command', + kind: 'TOOL', + startTime: at(5), + endTime: at(9), + service: 'synthetic', + tool: 'exec_command', + extra: { 'input.value': JSON.stringify({ cmd: COMMAND }), 'output.value': '12 passed' }, + }), + span({ + traceId: TRACE, + spanId: 'reply-1', + parentSpanId: 'root', + name: 'llm.turn', + kind: 'LLM', + startTime: at(10), + service: 'synthetic', + content: 'All 12 unit tests passed.', + }), + ] +} + +type Script = (request: TraceAnalysisEngineRequest) => Promise> + +/** A fake engine that runs a script per request and records every request. */ +function scriptedEngine(script: Script, executionConfig: Record = {}) { + const requests: TraceAnalysisEngineRequest[] = [] + const engine: TraceAnalysisEngine = { + id: 'scripted-test-engine', + description: 'Runs a test script instead of a model.', + model: 'test-model', + version: '1.0.0', + executionConfig, + async analyze(request) { + requests.push(request) + const result = await script(request) + return { + answer: '', + findings: [], + trajectory: [], + modelCalls: 1, + toolCalls: 0, + runtime: {}, + ...result, + } + }, + } + return { engine, requests } +} + +function tool(request: TraceAnalysisEngineRequest, name: string) { + const found = request.tools.find((candidate) => candidate.name === name) + if (!found) throw new Error(`tool ${name} was not offered`) + return found +} + +function questionId(request: TraceAnalysisEngineRequest): string { + return request.analystId.replace(/^ask\./, '') +} + +function deferred() { + let resolve!: () => void + const promise = new Promise((done) => { resolve = done }) + return { promise, resolve } +} + +async function until(condition: () => boolean, timeoutMs = 5_000): Promise { + const deadline = Date.now() + timeoutMs + while (!condition()) { + if (Date.now() > deadline) throw new Error('condition not reached') + await sleep(5) + } +} + +describe('runTraceQuestions', () => { + it('keeps each answer verbatim, verifies its citations, and writes both artifacts', async () => { + const { engine } = scriptedEngine(async (request) => { + // The engine reads through the real tool handler it was given. + const viewed = await tool(request, 'viewSpans').handler({ trace_id: TRACE, span_ids: ['tool-1'] }) as { + spans: Array<{ span_id: string }> + } + expect(viewed.spans.map((entry) => entry.span_id)).toEqual(['tool-1']) + return { + answer: `The session ran \`${COMMAND}\` once (trace://${TRACE}/span/tool-1).`, + findings: [{ + severity: 'info', + claim: 'The unit tests ran once.', + confidence: 0.9, + evidence: [{ uri: `trace://${TRACE}/span/tool-1`, excerpt: 'git status --short' }], + }], + toolCalls: 1, + } + }) + const dir = await mkdtemp(join(tmpdir(), 'traces-ask-test-')) + const result = await runTraceQuestions({ + questions: [{ id: 'commands', question: 'Which shell commands ran?' }], + spans: fixtureSpans(), + engine, + harness: 'synthetic', + otlpOutPath: join(dir, 'traces.otlp.jsonl'), + }) + + expect(result.ok).toBe(true) + const [answer] = result.questions + expect(answer!.status).toBe('answered') + expect(answer!.answer).toBe(`The session ran \`${COMMAND}\` once (trace://${TRACE}/span/tool-1).`) + expect(answer!.citations).toEqual([{ uri: `trace://${TRACE}/span/tool-1`, traceId: TRACE, spanId: 'tool-1', resolved: true }]) + expect(answer!.findings).toHaveLength(1) + expect(answer!.modelCalls).toBe(1) + expect(answer!.toolCalls).toBe(1) + expect(answer!.model).toBe('test-model') + expect(result.traces).toEqual([expect.objectContaining({ traceId: TRACE, sessionId: 'session-ask', spanCount: 4 })]) + + const artifacts = await writeTraceQuestionsArtifacts(result, dir) + const saved = JSON.parse(await readFile(artifacts.result, 'utf8')) as typeof result + expect(saved.questions[0]!.answer).toBe(answer!.answer) + expect(saved).not.toHaveProperty('report') + expect(await readFile(artifacts.report, 'utf8')).toContain(answer!.answer!) + }) + + it('never runs more questions at once than the concurrency limit', async () => { + const gates = new Map>() + const started: string[] = [] + let running = 0 + let peak = 0 + const { engine } = scriptedEngine(async (request) => { + const id = questionId(request) + started.push(id) + running += 1 + peak = Math.max(peak, running) + const gate = deferred() + gates.set(id, gate) + await gate.promise + running -= 1 + return { answer: `answer ${id}` } + }) + const run = runTraceQuestions({ + questions: ['one', 'two', 'three', 'four', 'five'].map((question) => ({ question: `Question ${question}?` })), + spans: fixtureSpans(), + engine, + concurrency: 3, + }) + + await until(() => started.length === 3) + await sleep(50) + // Two questions wait until a running one finishes. + expect(started).toEqual(['q1', 'q2', 'q3']) + gates.get('q2')!.resolve() + await until(() => started.length === 4) + await sleep(50) + expect(started).toHaveLength(4) + for (const id of ['q1', 'q3']) gates.get(id)!.resolve() + await until(() => started.length === 5) + for (const id of ['q4', 'q5']) gates.get(id)!.resolve() + + const result = await run + expect(peak).toBe(3) + expect(result.totals.peakConcurrency).toBe(3) + expect(result.questions.map((answer) => answer.answer)).toEqual(['q1', 'q2', 'q3', 'q4', 'q5'].map((id) => `answer ${id}`)) + }) + + it('overlaps questions, so wall time stays well below the sum of question times', async () => { + const { engine } = scriptedEngine(async (request) => { + await sleep(200) + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 6 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 6, + }) + expect(result.ok).toBe(true) + // Six 200 ms questions: serial would take 1.2 s or more. + expect(result.totals.questionTimeMs).toBeGreaterThanOrEqual(1_200) + expect(result.totals.wallTimeMs).toBeLessThan(result.totals.questionTimeMs / 2) + expect(result.totals.peakConcurrency).toBe(6) + expect(result.report).toMatch(/Wall time \d+\.\d s against \d+\.\d s of question time \(peak 6 at once\)/) + }) + + it('reports questions the shared ledger refused while the others keep their answers', async () => { + const { engine } = scriptedEngine(async (request) => { + // One metered call per question: reserve $0.40, settle at $0.30. + const paid = await request.costLedger.runPaidCall({ + channel: 'analyst', + phase: request.costPhase, + actor: request.analystId, + ...(request.costTags ? { tags: request.costTags } : {}), + maximumCharge: { externallyEnforcedMaximumUsd: 0.4 }, + execute: async () => { + await sleep(20) + return 'ok' + }, + receipt: () => ({ model: 'test-model', inputTokens: 100, outputTokens: 50, actualCostUsd: 0.3 }), + }) + if (!paid.succeeded) throw paid.error + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 5 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 5, + budgetUsd: 1, + }) + + // $1 admits three $0.30 calls; after them, $0.10 is left and a $0.40 reservation is refused. + const answered = result.questions.filter((answer) => answer.status === 'answered') + const refused = result.questions.filter((answer) => answer.failure?.kind === 'budget-refused') + expect(answered).toHaveLength(3) + expect(refused).toHaveLength(2) + for (const answer of refused) { + expect(answer.failure!.message).toContain('would exceed ceiling 1') + expect(answer.answer).toBeNull() + } + for (const answer of answered) expect(answer.usage?.cost).toEqual({ kind: 'observed', usd: 0.3 }) + expect(result.ok).toBe(false) + expect(result.totals.cost.usd).toBeCloseTo(0.9, 10) + expect(result.totals.cost.usd!).toBeLessThanOrEqual(1) + expect(result.report).toContain('failed: budget-refused') + }) + + it('refuses to start when the budget cannot cover one model call', async () => { + const { engine, requests } = scriptedEngine(async () => ({ answer: 'never' }), { + pricing: { inputUsdPerMillion: 1.25, outputUsdPerMillion: 10 }, + max_output_tokens: 8_192, + max_reasoning_tokens: 32_768, + }) + await expect(runTraceQuestions({ + questions: [{ question: 'Anything?' }], + spans: fixtureSpans(), + engine, + budgetUsd: 0.1, + })).rejects.toThrow(/below one model call's reservation of at least \$0\.41/) + expect(requests).toHaveLength(0) + }) + + it('warns when the budget serializes concurrent questions', async () => { + const { engine } = scriptedEngine(async () => ({ answer: 'fine' }), { + pricing: { inputUsdPerMillion: 1.25, outputUsdPerMillion: 10 }, + max_output_tokens: 8_192, + max_reasoning_tokens: 32_768, + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 4 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 4, + budgetUsd: 1, + }) + expect(result.warnings).toEqual([expect.stringContaining('covers at most 2 concurrent model call(s)')]) + expect(result.report).toContain('> Warning: budget $1.00 covers at most 2') + }) + + it('keeps the other answers when one engine call throws, and marks the run failed', async () => { + const { engine } = scriptedEngine(async (request) => { + if (questionId(request) === 'q3') throw new Error('synthetic engine failure') + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 5 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 2, + }) + expect(result.ok).toBe(false) + expect(result.totals).toMatchObject({ answered: 4, failed: 1, modelCalls: null }) + const failed = result.questions.find((answer) => answer.id === 'q3')! + expect(failed.failure).toEqual({ kind: 'error', message: 'Error: synthetic engine failure' }) + expect(failed.modelCalls).toBeNull() + expect(result.questions.filter((answer) => answer.id !== 'q3').map((answer) => answer.answer)) + .toEqual(['answer q1', 'answer q2', 'answer q4', 'answer q5']) + }) + + it('fails an answer that cites a span the trace does not hold', async () => { + const { engine } = scriptedEngine(async () => ({ + answer: `Tests ran at trace://${TRACE}/span/tool-1 and trace://${TRACE}/span/invented-span.`, + })) + const result = await runTraceQuestions({ + questions: [{ question: 'When did the tests run?' }], + spans: fixtureSpans(), + engine, + }) + const [answer] = result.questions + expect(answer!.status).toBe('failed') + expect(answer!.failure?.kind).toBe('unresolved-citations') + expect(answer!.answer).toContain('invented-span') + expect(answer!.citations.map((citation) => [citation.spanId, citation.resolved])).toEqual([ + ['tool-1', true], + ['invented-span', false], + ]) + expect(result.report).toContain(`**Unresolved:** trace://${TRACE}/span/invented-span`) + }) + + it('fails an empty answer', async () => { + const { engine } = scriptedEngine(async () => ({ answer: ' ' })) + const result = await runTraceQuestions({ questions: [{ question: 'Anything?' }], spans: fixtureSpans(), engine }) + expect(result.questions[0]!.failure?.kind).toBe('no-answer') + expect(result.questions[0]!.answer).toBeNull() + }) + + it('shows the evidence gate\'s rejection reasons for each question', async () => { + const { engine } = scriptedEngine(async () => ({ + answer: `One command ran (trace://${TRACE}/span/tool-1).`, + findings: [{ + severity: 'low', + claim: 'The session deleted the build directory.', + confidence: 0.8, + evidence: [{ uri: `trace://${TRACE}/span/tool-1`, excerpt: 'rm -rf build directory' }], + }], + })) + const result = await runTraceQuestions({ questions: [{ question: 'What ran?' }], spans: fixtureSpans(), engine }) + const [answer] = result.questions + expect(answer!.status).toBe('answered') + expect(answer!.findings).toHaveLength(0) + expect(answer!.rejectedFindings).toEqual({ 'excerpt is not present in the cited span content': 1 }) + expect(result.report).toContain( + '**Rejected by the evidence gate:** 1 finding(s) rejected: excerpt is not present in the cited span content ×1', + ) + }) + + it('parses a schema-bound answer and fails one that breaks the schema', async () => { + const schema = { + type: 'object', + properties: { commands: { type: 'integer' }, first: { type: 'string' } }, + required: ['commands', 'first'], + additionalProperties: false, + } + const { engine } = scriptedEngine(async (request) => ({ + answer: questionId(request) === 'good' + ? '```json\n{"commands": 1, "first": "git status --short"}\n```' + : '{"commands": "one"}', + })) + const result = await runTraceQuestions({ + questions: [ + { id: 'good', question: 'How many commands ran, and which was first?', answerSchema: schema }, + { id: 'bad', question: 'How many commands ran, and which was first?', answerSchema: schema }, + ], + spans: fixtureSpans(), + engine, + }) + const [good, bad] = result.questions + expect(good!.status).toBe('answered') + expect(good!.parsedAnswer).toEqual({ commands: 1, first: 'git status --short' }) + expect(bad!.failure?.kind).toBe('invalid-answer') + expect(bad!.failure?.message).toBe('$.first: required property is missing; $.commands: expected integer, got string') + }) +}) + +describe('question layout for the DSPy preview', () => { + it('keeps the question whole and the rules inside the first 500 characters of the instructions', async () => { + const longest = 'x'.repeat(MAX_TRACE_QUESTION_CHARS) + const { engine, requests } = scriptedEngine(async () => ({ answer: 'noted' })) + await runTraceQuestions({ + questions: [ + { id: 'plain', question: longest }, + { + id: 'typed', + question: 'How many commands ran?', + instructions: 'Count only shell commands.', + answerSchema: { type: 'object', properties: { count: { type: 'integer' } } }, + }, + ], + spans: fixtureSpans(), + engine, + }) + expect(requests).toHaveLength(2) + for (const request of requests) { + expect(request.question.length).toBeLessThanOrEqual(1_000) + const head = request.instructions.slice(0, TRACE_QUESTION_PREVIEW_HEAD_CHARS) + expect(head).toContain('TRACES ASK RULES') + expect(head).toContain('5. If the trace does not record a fact, say "not in trace".') + // The trace list reaches the model after the rules, as parseable JSON. + const context = request.instructions.split('PREPARED CONTEXT:\n')[1]!.split('\n\n')[0]! + expect(JSON.parse(context)).toMatchObject({ traces: [{ trace_id: TRACE, session_id: 'session-ask', spans: 4 }], omitted_traces: 0 }) + } + const plain = requests.find((request) => request.analystId === 'ask.plain')! + expect(plain.question.endsWith(`QUESTION: ${longest}`)).toBe(true) + const typed = requests.find((request) => request.analystId === 'ask.typed')! + expect(typed.instructions.slice(0, TRACE_QUESTION_PREVIEW_HEAD_CHARS)).toContain('6. The answer is one JSON value matching ANSWER SCHEMA') + expect(typed.instructions).toContain('QUESTION GUIDANCE:\nCount only shell commands.') + expect(renderTraceQuestionInstructions({})).not.toContain('ANSWER SCHEMA') + }) + + it('rejects a question the preview would cut, before any engine call', () => { + expect(() => normalizeTraceQuestions([{ question: 'y'.repeat(MAX_TRACE_QUESTION_CHARS + 1) }])) + .toThrow(/the limit is \d+ so the model sees it whole/) + }) +}) + +describe('question input', () => { + it('reads strings and objects from a questions file and rejects unknown keys', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-ask-questions-')) + const good = join(dir, 'questions.json') + await writeFile(good, JSON.stringify({ + questions: [ + 'Which commands failed?', + { id: 'last-turn', question: 'What was the last human turn?', answerSchema: { type: 'string' } }, + ], + }), 'utf8') + expect(await loadTraceQuestionsFile(good)).toEqual([ + { question: 'Which commands failed?' }, + { id: 'last-turn', question: 'What was the last human turn?', answerSchema: { type: 'string' } }, + ]) + + const typo = join(dir, 'typo.json') + await writeFile(typo, JSON.stringify([{ question: 'Anything?', schema: { type: 'string' } }]), 'utf8') + await expect(loadTraceQuestionsFile(typo)).rejects.toThrow('entry 1 has unknown key "schema"') + }) + + it('assigns default IDs and rejects duplicates', () => { + expect(normalizeTraceQuestions([{ question: ' a? ' }, { question: 'b?' }]).map((entry) => [entry.id, entry.question])) + .toEqual([['q1', 'a?'], ['q2', 'b?']]) + expect(() => normalizeTraceQuestions([{ id: 'x', question: 'a?' }, { id: 'x', question: 'b?' }])) + .toThrow('duplicate question ID "x"') + expect(() => normalizeTraceQuestions([])).toThrow('at least one question') + }) + + it('extracts trace citations without trailing punctuation', () => { + expect(traceCitationsInText('See trace://t%2F1/span/abc. Also (trace://t2/span/def), trace://t2/span/def.')) + .toEqual([ + { uri: 'trace://t%2F1/span/abc', traceId: 't/1', spanId: 'abc' }, + { uri: 'trace://t2/span/def', traceId: 't2', spanId: 'def' }, + ]) + }) +}) + +describe('answer schemas', () => { + it('rejects keywords it cannot check instead of ignoring them', () => { + expect(() => assertAnswerSchema({ type: 'string', pattern: '^PR' })).toThrow('unsupported JSON Schema keyword "pattern"') + expect(() => assertAnswerSchema({ type: 'object', properties: { n: { minimum: 1 } } })) + .toThrow('answerSchema.properties.n: unsupported JSON Schema keyword "minimum"') + }) + + it('checks types, required properties, enums, and array items', () => { + const schema = { + type: 'array', + items: { + type: 'object', + properties: { pr: { type: 'integer' }, state: { enum: ['merged', 'closed'] } }, + required: ['pr'], + }, + } + expect(answerSchemaErrors([{ pr: 3, state: 'merged' }], schema)).toEqual([]) + expect(answerSchemaErrors([{ pr: 3.5 }, { state: 'open' }], schema)).toEqual([ + '$[0].pr: expected integer, got number', + '$[1].pr: required property is missing', + '$[1].state: expected one of "merged", "closed"', + ]) + expect(parseJsonAnswer('The answer is 3')).toEqual({ ok: false, error: 'answer is not one JSON value' }) + }) +}) diff --git a/tests/cli.test.ts b/tests/cli.test.ts index 2bc8a70..9840ff6 100644 --- a/tests/cli.test.ts +++ b/tests/cli.test.ts @@ -131,7 +131,11 @@ describe('traces CLI', () => { expect(result.spanCount).toBe(2) expect(await readFile(join(improvement, 'traces.otlp.jsonl'), 'utf8')).not.toBe('') expect(await readFile(join(improvement, 'report.md'), 'utf8')).toContain('1 session(s), 2 spans') - }, 15_000) + // Three cold CLI subprocesses, each allowed 30s of its own. A 15s outer + // budget was shorter than the inner ones and failed on a loaded machine + // while every subprocess was still inside its own timeout, which is the + // disagreement `vitest.config.ts` raised the default to fix. + }, 90_000) it('turns deterministic analyze signals into actionable findings', async () => { const dir = await mkdtemp(join(tmpdir(), 'traces-cli-actionable-')) @@ -1231,6 +1235,158 @@ describe('traces analyze --llm failure surfacing', () => { }, 60_000) }) +describe('traces analyze external analyzer failure', () => { + it('writes the report, then exits 1 when a requested analyzer fails', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-external-fail-')) + const input = join(dir, 'spans.openinference.jsonl') + const report = join(dir, 'report.md') + await writeFile(input, serializeSpans([ + span({ + traceId: 'trace-external-fail', + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: '2026-01-01T00:00:00.000Z', + service: 'claude-code', + extra: { 'session.id': 'session-external-fail' }, + }), + ]), 'utf8') + + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'analyze', input, + '--format', 'openinference', + // `false` is an installed command that exits 1 whatever its arguments. + '--analyzer', 'false', + '--out', report, + ], { + cwd: process.cwd(), + env: { ...process.env, NO_COLOR: '1', FORCE_COLOR: '' }, + maxBuffer: 10 * 1024 * 1024, + timeout: 60_000, + }).then( + () => { + throw new Error('analyze exited 0 although its requested analyzer failed') + }, + (error: Error & { code?: number; stdout?: string; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stderr).toContain('1 of 1 external analyzer(s) failed') + expect(failure.stderr).toContain('false: exit 1') + const reportText = await readFile(report, 'utf8') + expect(reportText).toContain('### false (report)') + expect(reportText).toContain('failed: exit 1') + }, 60_000) +}) + +describe('traces ask', () => { + async function writeSessionFile(dir: string): Promise { + const input = join(dir, 'spans.openinference.jsonl') + await writeFile(input, serializeSpans([ + span({ + traceId: 'trace-ask-cli', + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: '2026-01-01T00:00:00.000Z', + service: 'codex', + extra: { 'session.id': 'session-ask-cli' }, + }), + span({ + traceId: 'trace-ask-cli', + spanId: 'turn', + parentSpanId: 'root', + name: 'llm.turn', + kind: 'LLM', + startTime: '2026-01-01T00:00:01.000Z', + service: 'codex', + step: 1, + content: 'Running the checks now.', + }), + ]), 'utf8') + return input + } + + it('needs at least one question', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-ask-none-')) + const input = await writeSessionFile(dir) + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'ask', input, '--format', 'openinference', + ], { cwd: process.cwd(), env: { ...process.env, NO_COLOR: '1' }, timeout: 30_000 }).then( + () => { + throw new Error('ask exited 0 without a question') + }, + (error: Error & { code?: number; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stderr).toContain('ask needs --question') + }, 30_000) + + it('writes every answer and failure, then exits 1 when the engine dies at startup', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-ask-fail-')) + const input = await writeSessionFile(dir) + const out = join(dir, 'ask') + const questions = join(dir, 'questions.json') + await writeFile(questions, JSON.stringify([ + { id: 'checks', question: 'Which checks ran?' }, + { id: 'turns', question: 'How many assistant turns were there?', answerSchema: { type: 'integer' } }, + ]), 'utf8') + const fakeBridge = join(dir, 'failing-bridge.sh') + const bridgeReason = 'DSPY-BRIDGE-FAILURE: ValueError: synthetic startup failure' + await writeFile(fakeBridge, `#!/bin/sh\necho "${bridgeReason}" >&2\nexit 1\n`, { mode: 0o755 }) + + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'ask', input, + '--format', 'openinference', + '--questions', questions, + '--question', 'Did the session finish?', + '--budget', '5', + '--concurrency', '2', + '--dir', out, + ], { + cwd: process.cwd(), + env: { + ...process.env, + NO_COLOR: '1', + FORCE_COLOR: '', + TANGLE_API_KEY: 'test-key-never-sent-upstream', + OPENAI_API_KEY: '', + OPENAI_BASE_URL: '', + TRACES_PYTHON: fakeBridge, + }, + maxBuffer: 10 * 1024 * 1024, + timeout: 90_000, + }).then( + () => { + throw new Error('ask exited 0 although every question failed') + }, + (error: Error & { code?: number; stdout?: string; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stdout).toContain('# traces ask') + expect(failure.stdout).toContain('**0 answered, 3 failed.**') + expect(failure.stderr).toContain('3 of 3 question(s) failed') + + const answers = JSON.parse(await readFile(join(out, 'answers.json'), 'utf8')) as { + kind: string + ok: boolean + budgetUsd: number + questionBudgetUsd: number + questions: Array<{ id: string; status: string; failure?: { kind: string; message: string } }> + } + expect(answers.kind).toBe('traces.ask') + expect(answers.ok).toBe(false) + expect(answers.budgetUsd).toBe(5) + expect(answers.questionBudgetUsd).toBe(1) + expect(answers.questions.map((answer) => answer.id)).toEqual(['checks', 'turns', 'q3']) + for (const answer of answers.questions) { + expect(answer.status).toBe('failed') + expect(answer.failure?.kind).toBe('error') + expect(answer.failure?.message).toContain(bridgeReason) + } + expect(await readFile(join(out, 'report.md'), 'utf8')).toContain('failed: error') + }, 90_000) +}) + describe('traces bundle + bundle-view', () => { it('assembles the full view, then projects the writer view over the same session', async () => { const root = await mkdtemp(join(tmpdir(), 'traces-cli-bundle-')) diff --git a/tests/codex-command-facts.test.ts b/tests/codex-command-facts.test.ts new file mode 100644 index 0000000..b8a2418 --- /dev/null +++ b/tests/codex-command-facts.test.ts @@ -0,0 +1,301 @@ +/** + * Codex rollout facts that audit questions ask about: the commands a code-mode + * script ran, the files a patch changed, and which user turns a person typed. + * Every rollout comes from `tests/codex-facts-fixture.ts` and is synthetic. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { CodexAdapter } from '../src/adapters/codex.js' +import { buildPolicyEvidenceRecord } from '../src/evidence.js' +import type { OtlpSpan } from '../src/otlp.js' +import { runPipelines } from '../src/pipelines.js' +import { describeSessionRelationship } from '../src/session-relationship.js' +import { + at, + command, + commandItem, + ENVIRONMENT_BLOCK, + FIRST_REQUEST, + operatorRollout, + type RecordOrder, + type Row, + script, + scriptOutput, + task, + userEvent, + userItem, + userItemCompleted, + writeRollout as writeRolloutIn, +} from './codex-facts-fixture.js' + +const dir = mkdtempSync(join(tmpdir(), 'traces-codex-facts-')) +afterAll(() => rmSync(dir, { recursive: true, force: true })) + +const rollout = (name: string, order?: RecordOrder) => operatorRollout(dir, name, order) +const writeRollout = (name: string, rows: readonly Row[]) => writeRolloutIn(dir, name, rows) + +const kindOf = (item: OtlpSpan) => item.attributes['openinference.span.kind'] +const byName = (spans: readonly OtlpSpan[], name: string) => spans.filter((item) => item.name === name) +const inner = (spans: readonly OtlpSpan[]) => spans.filter((item) => item.attributes['traces.tool_call.level'] === 'inner') +const humanTurns = (spans: readonly OtlpSpan[]) => + byName(spans, 'user.prompt').filter((item) => item.attributes['tangle.actor'] === 'human') + +/** + * The same fixture parsed by the adapter this branch changes, at origin/main + * (7633ebc): no inner spans, and the two `` blocks plus the + * AGENTS.md block counted as human turns. Measured by running + * `tests/codex-facts-fixture.ts` against that checkout. + */ +const BASELINE = { total: 12, outerTools: 2, inner: 0, userPrompts: 6, humanTurns: 5 } as const + +function spanCounts(spans: readonly OtlpSpan[]) { + return { + total: spans.length, + outerTools: spans.filter((item) => kindOf(item) === 'TOOL').length, + inner: inner(spans).length, + userPrompts: byName(spans, 'user.prompt').length, + humanTurns: humanTurns(spans).length, + } +} + +describe('Codex command and file-change spans', () => { + it('emits each command inside a script with its own times, exit code, and process', async () => { + const spans = await new CodexAdapter().parse(rollout('commands')) + const scriptSpan = spans.find((item) => item.attributes['traces.codex.source_span_id'] === 'tool:call-script')! + const commands = byName(spans, 'command.execution') + + const joined = commands.filter((item) => item.parent_span_id === scriptSpan.span_id) + expect(joined.map((item) => ({ + input: JSON.parse(String(item.attributes['input.value'])).command.at(-1), + start: item.start_time, + end: item.end_time, + exit: item.attributes['process.exit_code'], + pid: item.attributes['traces.codex.process_id'], + status: item.status.code, + join: item.attributes['traces.codex.item_join'], + }))).toEqual([ + { input: 'gh pr create --fill', start: at(4.1), end: at(5), exit: 0, pid: '41001', status: 'OK', join: 'call' }, + { input: 'gh-drew pr merge 3 --squash', start: at(5.1), end: at(6), exit: 1, pid: '41002', status: 'ERROR', join: 'call' }, + { input: 'git status --short', start: at(6.1), end: at(6.5), exit: 0, pid: '41003', status: 'OK', join: 'call' }, + ]) + expect(joined[1]!.attributes['output.value']).toBe('X Pull request #3 is not mergeable\n') + expect(joined[1]!.status.message).toBe('command exited 1') + for (const item of commands) { + expect(kindOf(item)).toBe('CHAIN') + expect(item.attributes['span.type']).toBe('tool.execution') + expect(item.attributes['tool.name']).toBeUndefined() + expect(item.trace_id).toBe(scriptSpan.trace_id) + } + }) + + it('keeps a command that outlives its call under the session root', async () => { + const spans = await new CodexAdapter().parse(rollout('unmatched')) + const root = spans.find((item) => item.parent_span_id === null)! + const watch = byName(spans, 'command.execution') + .find((item) => String(item.attributes['input.value']).includes('pnpm test --watch=false'))! + expect(watch.parent_span_id).toBe(root.span_id) + expect(watch.attributes['traces.codex.item_join']).toBe('unmatched') + expect([watch.start_time, watch.end_time]).toEqual([at(6.8), at(9)]) + }) + + it('leaves a command inside two overlapping calls under the session root', async () => { + const spans = await new CodexAdapter().parse(writeRollout('ambiguous', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + script(2, 'call-a', 'await tools.exec_command({ cmd: "pnpm test" })'), + script(3, 'call-b', 'await tools.exec_command({ cmd: "pnpm build" })'), + command(4, commandItem('item-shared', '42001', 'pnpm test', 0, 'ok\n'), { start: 3.5, end: 4 }), + scriptOutput(5, 'call-a', 'Script completed\nWall time 1.0 seconds\nOutput:\nok'), + scriptOutput(6, 'call-b', 'Script completed\nWall time 1.0 seconds\nOutput:\nok'), + task(7, 'task_complete', 'turn-1'), + ])) + const root = spans.find((item) => item.parent_span_id === null)! + const shared = byName(spans, 'command.execution') + expect(shared).toHaveLength(1) + expect(shared[0]!.attributes['traces.codex.item_join']).toBe('ambiguous') + expect(shared[0]!.parent_span_id).toBe(root.span_id) + }) + + it('emits the paths an apply_patch inside exec changed', async () => { + const spans = await new CodexAdapter().parse(rollout('patch')) + const patchCall = spans.find((item) => item.attributes['traces.codex.source_span_id'] === 'tool:call-patch')! + const files = byName(spans, 'file.change') + expect(files.map((item) => ({ + input: JSON.parse(String(item.attributes['input.value'])), + kind: item.attributes['traces.codex.file_change_kind'], + parent: item.parent_span_id, + status: item.status.code, + }))).toEqual([ + { input: { kind: 'update', path: '/workspace/demo/src/parser.ts' }, kind: 'update', parent: patchCall.span_id, status: 'OK' }, + { input: { kind: 'add', path: '/workspace/demo/tests/parser.test.ts' }, kind: 'add', parent: patchCall.span_id, status: 'OK' }, + ]) + expect(files.every((item) => item.start_time === at(10.2) && item.end_time === at(10.5))).toBe(true) + }) + + it('counts item shapes it cannot represent instead of guessing', async () => { + const spans = await new CodexAdapter().parse(rollout('skipped')) + const root = spans.find((item) => item.parent_span_id === null)! + expect(JSON.parse(String(root.attributes['traces.codex.skipped_item_counts']))).toEqual({ + 'CommandExecution:malformed': 1, + FixtureFutureItem: 1, + }) + expect(spans.some((item) => item.attributes['traces.codex.item_id'] === 'item-broken')).toBe(false) + }) + + it('marks a script whose receipt omits the Wall time colon as successful', async () => { + const spans = await new CodexAdapter().parse(rollout('receipt')) + const scriptSpan = spans.find((item) => item.attributes['traces.codex.source_span_id'] === 'tool:call-script')! + expect(scriptSpan.status.code).toBe('OK') + }) + + it('adds only inner spans: outer tool counts, evidence tool counts, and loop input stay unchanged', async () => { + const spans = await new CodexAdapter().parse(rollout('counts')) + expect(spanCounts(spans)).toEqual({ ...BASELINE, total: BASELINE.total + 6, inner: 6, humanTurns: 2 }) + + const ref = rollout('counts-evidence') + const outerOnly = spans.filter((item) => item.attributes['traces.tool_call.level'] !== 'inner') + const evidence = await buildPolicyEvidenceRecord(ref, spans, { generatedAt: at(0) }) + const outerEvidence = await buildPolicyEvidenceRecord(ref, outerOnly, { generatedAt: at(0) }) + expect(evidence.metrics).toMatchObject({ spanCount: BASELINE.total + 6, toolCallCount: 2, erroredToolCallCount: 0 }) + expect(evidence.metrics.tools).toEqual([ + { name: 'apply_patch', calls: 1, errors: 0 }, + { name: 'exec_command.verify', calls: 1, errors: 0 }, + ]) + const { spanCount: _all, ...toolMetrics } = evidence.metrics + const { spanCount: _outer, ...outerToolMetrics } = outerEvidence.metrics + expect(toolMetrics).toEqual(outerToolMetrics) + expect(evidence.signals).toEqual(outerEvidence.signals) + // agent-eval counts a failed inner command as one more execution error. + expect(evidence.execution.execution.executionErrors.events) + .toBe(outerEvidence.execution.execution.executionErrors.events + 1) + const [pipelines, outerPipelines] = await Promise.all([runPipelines(spans), runPipelines(outerOnly)]) + expect(pipelines.toolUse).toEqual(outerPipelines.toolUse) + expect(pipelines.stuckLoops.findings).toEqual(outerPipelines.stuckLoops.findings) + expect(pipelines.toolUse[0]).toMatchObject({ totalCalls: 2 }) + }) +}) + +describe('Codex human turns', () => { + it.each(['item-first', 'event-first'] as const)( + 'records each human turn once with its text and timestamp (%s)', + async (order) => { + const spans = await new CodexAdapter().parse(rollout(`turns-${order}`, order)) + const turns = humanTurns(spans) + expect(turns.map((item) => [item.attributes.content, item.start_time])).toEqual([ + [FIRST_REQUEST, at(2)], + ['ya?', at(21)], + ]) + expect(turns.every((item) => item.attributes['traces.codex.user_message_event'] === true)).toBe(true) + expect(byName(spans, 'user.prompt').filter((item) => item.attributes.content === FIRST_REQUEST)).toHaveLength(1) + }, + ) + + it('labels injected context blocks as non-human', async () => { + const spans = await new CodexAdapter().parse(rollout('injected')) + const injected = byName(spans, 'user.prompt') + .filter((item) => String(item.attributes.content).startsWith('')) + expect(injected).toHaveLength(3) + expect(injected.every((item) => item.attributes['tangle.actor'] === 'injected')).toBe(true) + expect(humanTurns(spans).at(-1)?.attributes.content).toBe('ya?') + }) + + it('treats a user-role message with no user_message event as injected once the event stream exists', async () => { + const spans = await new CodexAdapter().parse(writeRollout('unknown-wrapper', [ + { t: 0, type: 'session_meta', payload: { id: 'wrapper-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, 'Please add a changelog entry.'), + userEvent(2.001, 'Please add a changelog entry.'), + userItem(3, 'harness text'), + task(4, 'task_complete', 'turn-1'), + ])) + expect(byName(spans, 'user.prompt').map((item) => [item.attributes.content, item.attributes['tangle.actor']])).toEqual([ + ['Please add a changelog entry.', 'human'], + ['harness text', 'injected'], + ]) + }) + + it('records a turn reported as an item_completed UserMessage once', async () => { + const spans = await new CodexAdapter().parse(writeRollout('item-completed-turns', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + userItem(0.5, [{ type: 'input_text', text: ENVIRONMENT_BLOCK }]), + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: 'Rerun the parser tests.' }]), + userItemCompleted(2.001, 'item-turn-1', 'Rerun the parser tests.'), + task(3, 'task_complete', 'turn-1'), + ])) + expect(byName(spans, 'user.prompt').map((item) => [item.attributes.content, item.attributes['tangle.actor']])).toEqual([ + [ENVIRONMENT_BLOCK, 'injected'], + ['Rerun the parser tests.', 'human'], + ]) + expect(humanTurns(spans)[0]!.start_time).toBe(at(2)) + expect(humanTurns(spans)[0]!.attributes['traces.codex.user_message_event']).toBe(true) + const root = spans.find((item) => item.parent_span_id === null)! + expect(root.attributes['traces.codex.skipped_item_counts']).toBeUndefined() + }) + + it('pairs the two records of a turn whose message record carries a context prefix', async () => { + const typed = 'Fix the parser and rerun the tests.' + const prefixed = `${ENVIRONMENT_BLOCK}\n\n## My request for Codex:\n\n${typed}` + const spans = await new CodexAdapter().parse(writeRollout('prefixed-turn', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: prefixed }]), + userEvent(2.001, typed), + task(3, 'task_complete', 'turn-1'), + ])) + expect(byName(spans, 'user.prompt')).toHaveLength(1) + expect(humanTurns(spans).map((item) => [item.attributes.content, item.start_time])).toEqual([[prefixed, at(2)]]) + }) + + it('keeps text heuristics for rollouts that never recorded user_message events', async () => { + const spans = await new CodexAdapter().parse(writeRollout('legacy-turns', [ + { t: 0, type: 'session_meta', payload: { id: 'legacy-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, 'Please add a changelog entry.'), + task(3, 'task_complete', 'turn-1'), + ])) + expect(humanTurns(spans).map((item) => item.attributes.content)).toEqual(['Please add a changelog entry.']) + }) +}) + +describe('Codex child relationships', () => { + it('does not list the parent among the children a forked child messages', async () => { + const parentId = '019f0000-0000-7000-8000-00000000aaaa' + const childId = '019f0000-0000-7000-8000-00000000bbbb' + const siblingId = '019f0000-0000-7000-8000-00000000cccc' + const ref = writeRollout('forked-child', [ + { + t: 0, + type: 'session_meta', + payload: { + id: childId, + parent_thread_id: parentId, + thread_source: 'subagent', + cwd: '/workspace/demo', + source: { subagent: { thread_spawn: { parent_thread_id: parentId, depth: 1, agent_path: '/root/worker' } } }, + }, + }, + task(1, 'task_started', 'child-turn'), + userItem(2, 'Check the parser and report back.'), + { + t: 3, + type: 'response_item', + payload: { type: 'function_call', call_id: 'call-report', name: 'send_message', arguments: JSON.stringify({ target: parentId, message: 'done' }) }, + }, + { t: 4, type: 'response_item', payload: { type: 'function_call_output', call_id: 'call-report', output: '{"ok":true}' } }, + { + t: 5, + type: 'response_item', + payload: { type: 'function_call', call_id: 'call-sibling', name: 'send_message', arguments: JSON.stringify({ target: siblingId, message: 'fyi' }) }, + }, + { t: 6, type: 'response_item', payload: { type: 'function_call_output', call_id: 'call-sibling', output: '{"ok":true}' } }, + task(7, 'task_complete', 'child-turn'), + ]) + const relationship = describeSessionRelationship({ ...ref, sessionId: childId }, await new CodexAdapter().parse(ref)) + expect(relationship.parentSessionId).toBe(parentId) + expect(relationship.childSessionIds).toEqual([siblingId]) + expect(relationship.resumedChildSessionIds).toEqual([siblingId]) + }) +}) diff --git a/tests/codex-facts-fixture.ts b/tests/codex-facts-fixture.ts new file mode 100644 index 0000000..4b2155d --- /dev/null +++ b/tests/codex-facts-fixture.ts @@ -0,0 +1,204 @@ +/** + * Synthetic Codex rollouts for the facts an audit question asks about: the + * commands a code-mode script ran, the files a patch changed, and which user + * turns a person typed. No rollout here comes from a recorded session. + * + * `tests/codex-command-facts.test.ts` asserts against these rollouts. Keeping + * them in their own module also lets a reader parse the same fixture with an + * older checkout of the adapter to compare span counts. + */ +import { writeFileSync } from 'node:fs' +import { join } from 'node:path' +import type { SessionRef } from '../src/types.js' + +export const BASE_MS = Date.UTC(2026, 8, 8, 10, 0, 0) +/** The order in which a rollout recorded the two copies of one submitted turn. */ +export type RecordOrder = 'item-first' | 'event-first' + +export const at = (seconds: number): string => new Date(BASE_MS + seconds * 1000).toISOString() +export const ms = (seconds: number): number => BASE_MS + seconds * 1000 + +export type Row = { readonly t: number } & Record + +export function writeRollout(dir: string, name: string, rows: readonly Row[]): SessionRef { + const path = join(dir, `rollout-${name}.jsonl`) + writeFileSync(path, rows.map(({ t, ...row }) => JSON.stringify({ timestamp: at(t), ...row })).join('\n')) + return { harness: 'codex', sessionId: name, path, cwd: null, mtimeMs: 0 } +} + +export const userItem = (t: number, content: unknown): Row => ({ + t, + type: 'response_item', + payload: { type: 'message', role: 'user', content }, +}) +export const userEvent = (t: number, message: string): Row => ({ + t, + type: 'event_msg', + payload: { type: 'user_message', message, images: [], local_images: [] }, +}) +/** The current rollout shape for a submitted turn: an `item_completed` UserMessage item. */ +export const userItemCompleted = (t: number, id: string, text: string): Row => ({ + t, + type: 'event_msg', + payload: { + type: 'item_completed', + thread_id: 'facts-session', + turn_id: 'turn-1', + started_at_ms: ms(t), + completed_at_ms: ms(t), + item: { type: 'UserMessage', id, content: [{ type: 'text', text }] }, + }, +}) +export const task = (t: number, kind: 'task_started' | 'task_complete', turnId: string): Row => ({ + t, + type: 'event_msg', + payload: { type: kind, turn_id: turnId }, +}) +export const tokens = (t: number, input: number): Row => ({ + t, + type: 'event_msg', + payload: { + type: 'token_count', + info: { + last_token_usage: { input_tokens: input, output_tokens: 20 }, + total_token_usage: { input_tokens: input * 2, output_tokens: 40 }, + }, + }, +}) +export const script = (t: number, callId: string, input: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'custom_tool_call', call_id: callId, name: 'exec', input }, +}) +export const scriptOutput = (t: number, callId: string, output: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'custom_tool_call_output', call_id: callId, output }, +}) +export const command = ( + t: number, + item: Record, + window: { start?: number; end?: number } = {}, +): Row => ({ + t, + type: 'event_msg', + payload: { + type: 'item_completed', + thread_id: 'facts-session', + turn_id: 'turn-1', + ...(window.start === undefined ? {} : { started_at_ms: ms(window.start) }), + ...(window.end === undefined ? {} : { completed_at_ms: ms(window.end) }), + item: { type: 'CommandExecution', ...item }, + }, +}) +export const commandItem = ( + id: string, + processId: string, + script: string, + exitCode: number, + output: string, +): Record => ({ + id, + process_id: processId, + command: ['/bin/zsh', '-lc', script], + cwd: '/workspace/demo', + parsed_cmd: [{ type: 'unknown', cmd: script }], + source: 'agent', + status: exitCode === 0 ? 'completed' : 'failed', + stdout: output, + stderr: '', + aggregated_output: output, + exit_code: exitCode, + duration: { secs: 0, nanos: 800_000_000 }, + formatted_output: output, +}) + +export const AGENTS_BLOCK = '# AGENTS.md instructions for /workspace/demo\n\n\nUse pnpm.\n' +export const ENVIRONMENT_BLOCK = '\n /workspace/demo\n zsh\n' +export const FIRST_REQUEST = 'Open a PR for the parser fix, merge PR 3, then tell me what git status reports.' +export const SCRIPT_INPUT = [ + 'const created = await tools.exec_command({ cmd: "gh pr create --fill" })', + 'const merged = await tools.exec_command({ cmd: "gh-drew pr merge 3 --squash" })', + 'const status = await tools.exec_command({ cmd: "git status --short" })', + 'text([created.output, merged.output, status.output].join("\\n"))', +].join('\n') +export const PATCH_INPUT = [ + 'await tools.apply_patch(`*** Begin Patch', + '*** Update File: src/parser.ts', + '@@', + '-export const mode = "old"', + '+export const mode = "new"', + '*** Add File: tests/parser.test.ts', + '+test("mode", () => {})', + '*** End Patch`)', +].join('\n') + +/** + * One operator session: injected context, a substantive request logged twice, + * a code-mode script around three commands, a command that outlives its call, + * a patch applied inside `exec`, two item shapes the adapter cannot represent, + * and a short last typed turn followed by one more injected block. + */ +export function operatorRollout(dir: string, name: string, order: RecordOrder = 'item-first'): SessionRef { + const request = order === 'item-first' + ? [userItem(2, [{ type: 'input_text', text: FIRST_REQUEST }]), userEvent(2.001, FIRST_REQUEST)] + : [userEvent(2, FIRST_REQUEST), userItem(2.001, [{ type: 'input_text', text: FIRST_REQUEST }])] + const followUp = order === 'item-first' + ? [userItem(21, [{ type: 'input_text', text: 'ya?' }]), userEvent(21.001, 'ya?')] + : [userEvent(21, 'ya?'), userItem(21.001, [{ type: 'input_text', text: 'ya?' }])] + return writeRollout(dir, name, [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + userItem(0.6, [{ type: 'input_text', text: AGENTS_BLOCK }]), + userItem(0.7, [{ type: 'input_text', text: ENVIRONMENT_BLOCK }]), + task(1, 'task_started', 'turn-1'), + ...request, + tokens(3, 1000), + script(4, 'call-script', SCRIPT_INPUT), + command(5, commandItem('item-create', '41001', 'gh pr create --fill', 0, 'https://example.test/demo/pull/7\n'), { start: 4.1, end: 5 }), + command(6, commandItem('item-merge', '41002', 'gh-drew pr merge 3 --squash', 1, 'X Pull request #3 is not mergeable\n'), { start: 5.1, end: 6 }), + command(6.5, commandItem('item-status', '41003', 'git status --short', 0, ' M src/parser.ts\n'), { start: 6.1, end: 6.5 }), + scriptOutput(7, 'call-script', 'Script completed\nWall time 3.0 seconds\nOutput:\nhttps://example.test/demo/pull/7'), + command(9, commandItem('item-watch', '41004', 'pnpm test --watch=false', 0, 'Tests 12 passed\n'), { start: 6.8, end: 9 }), + tokens(9.5, 1400), + script(10, 'call-patch', PATCH_INPUT), + { + t: 10.5, + type: 'event_msg', + payload: { + type: 'item_completed', + thread_id: 'facts-session', + turn_id: 'turn-1', + started_at_ms: ms(10.2), + completed_at_ms: ms(10.5), + item: { + type: 'FileChange', + id: 'item-patch', + changes: { + '/workspace/demo/src/parser.ts': { + type: 'update', + unified_diff: '@@ -1 +1 @@\n-export const mode = "old"\n+export const mode = "new"\n', + move_path: null, + }, + '/workspace/demo/tests/parser.test.ts': { type: 'add', content: 'test("mode", () => {})\n' }, + }, + status: 'completed', + stdout: 'Success. Updated the following files:\nM src/parser.ts\nA tests/parser.test.ts\n', + stderr: '', + }, + }, + }, + scriptOutput(11, 'call-patch', 'Script completed\nWall time 0.3 seconds\nOutput:\nSuccess.'), + command(11.5, { id: 'item-broken', status: 'completed', exit_code: 0 }, { start: 11.2, end: 11.5 }), + { t: 11.6, type: 'event_msg', payload: { type: 'item_completed', turn_id: 'turn-1', completed_at_ms: ms(11.6), item: { type: 'FixtureFutureItem', id: 'item-future' } } }, + task(12, 'task_complete', 'turn-1'), + userItem(19, [{ type: 'input_text', text: ENVIRONMENT_BLOCK.replace('zsh', 'bash') }]), + task(20, 'task_started', 'turn-2'), + ...followUp, + tokens(22, 1500), + // Injected after the last typed turn: a reader that trusts the user role + // reports this block as the session's last human turn. + userItem(22.5, [{ type: 'input_text', text: ENVIRONMENT_BLOCK.replace('zsh', 'fish') }]), + task(23, 'task_complete', 'turn-2'), + ]) +} diff --git a/tests/codex-token-and-provenance.test.ts b/tests/codex-token-and-provenance.test.ts new file mode 100644 index 0000000..e1d5081 --- /dev/null +++ b/tests/codex-token-and-provenance.test.ts @@ -0,0 +1,365 @@ +/** + * Synthetic Codex rollouts for three adapter facts an audit question asks about: + * the harness's own cumulative token total, which spans are calls the model + * made, and the human context a forked or compacted session inherited. + * + * Every rollout here is written inline. None comes from a recorded session. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { CodexAdapter } from '../src/adapters/codex.js' +import { + INHERITED_SPAN_ATTR, + INHERITED_SPAN_COUNT_ATTR, + INHERITED_SPANS_OMITTED_ATTR, + isSynthesizedSpan, + SYNTHESIZED_SPAN_ATTR, +} from '../src/adapters/provenance.js' +import { buildPolicyEvidenceRecord } from '../src/evidence.js' +import { analyzeLiveBatch } from '../src/live.js' +import type { OtlpSpan } from '../src/otlp.js' +import { runPipelines } from '../src/pipelines.js' +import { sessionReportSource } from '../src/report.js' +import { at, ms, type Row, writeRollout } from './codex-facts-fixture.js' + +const dir = mkdtempSync(join(tmpdir(), 'traces-codex-provenance-')) +afterAll(() => rmSync(dir, { recursive: true, force: true })) + +const toolCall = (t: number, callId: string, name: string, cmd: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'function_call', call_id: callId, name, arguments: JSON.stringify({ cmd }) }, +}) +const toolOutput = (t: number, callId: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'function_call_output', call_id: callId, output: { exit_code: 0, output: 'ok' } }, +}) +const subagentActivity = (t: number, threadId: string, agentPath: string, kind: string): Row => ({ + t, + type: 'event_msg', + payload: { + type: 'sub_agent_activity', + event_id: `${threadId}-${kind}`, + occurred_at_ms: ms(t), + agent_thread_id: threadId, + agent_path: agentPath, + kind, + }, +}) +const tokenCount = ( + t: number, + last: Record | undefined, + total: Record, +): Row => ({ + t, + type: 'event_msg', + payload: { + type: 'token_count', + info: { ...(last ? { last_token_usage: last } : {}), total_token_usage: total }, + }, +}) +const userMessage = (t: number, text: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text }] }, +}) +const compacted = (t: number, summary: string, window: number, history: readonly string[]): Row => ({ + t, + type: 'compacted', + payload: { + message: summary, + window_number: window, + window_id: `window-${window}`, + previous_window_id: `window-${window - 1}`, + first_window_id: 'window-0', + replacement_history: [ + { type: 'message', id: `dev-${window}`, role: 'developer', content: [{ type: 'input_text', text: 'developer scaffolding' }] }, + ...history.map((text, index) => ({ + type: 'message', + id: `hist-${window}-${index}`, + role: 'user', + content: [{ type: 'input_text', text }], + })), + ], + }, +}) + +const parse = (ref: Parameters[0]): Promise => + new CodexAdapter().parse(ref) + +const kindOf = (span: OtlpSpan): unknown => span.attributes['openinference.span.kind'] +const inherited = (spans: readonly OtlpSpan[]): OtlpSpan[] => + spans.filter((span) => span.attributes[INHERITED_SPAN_ATTR] === true) + +describe('Codex cumulative token total', () => { + it('carries the harness counter verbatim instead of deriving one', async () => { + const ref = writeRollout(dir, 'token-counter', [ + { t: 0, type: 'session_meta', payload: { id: 'token-session', cwd: '/workspace/demo' } }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-1' } }, + tokenCount(2, { input_tokens: 1_000, output_tokens: 20 }, { input_tokens: 1_000, output_tokens: 20, total_tokens: 1_020 }), + // A repeat of the same cumulative snapshot: one turn, reported twice. + tokenCount(3, { input_tokens: 1_000, output_tokens: 20 }, { input_tokens: 1_000, output_tokens: 20, total_tokens: 1_020 }), + tokenCount( + 4, + { input_tokens: 2_000, output_tokens: 30 }, + { input_tokens: 3_000, output_tokens: 50, reasoning_output_tokens: 10, cached_input_tokens: 500, total_tokens: 3_050 }, + ), + // The counter's last word: the harness advanced the total with no + // per-turn delta to report, so no `llm.turn` span carries this number. + tokenCount( + 5, + undefined, + { input_tokens: 4_000, output_tokens: 60, reasoning_output_tokens: 12, cached_input_tokens: 700, total_tokens: 4_100 }, + ), + { t: 6, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-1' } }, + ]) + const spans = await parse(ref) + const root = spans[0]! + + expect(root.attributes['traces.session.total_tokens']).toBe(4_100) + expect(root.attributes['traces.session.total_tokens_source']).toBe('codex.token_count.info.total_token_usage') + expect(root.attributes['traces.session.total_input_tokens']).toBe(4_000) + expect(root.attributes['traces.session.total_output_tokens']).toBe(60) + expect(root.attributes['traces.session.total_reasoning_tokens']).toBe(12) + expect(root.attributes['traces.session.total_cached_input_tokens']).toBe(700) + + // Neither number a reader could compute from the spans equals the total: + // the deltas sum to 3,050 and the snapshots sum to 9,270. + const turns = spans.filter((span) => span.name === 'llm.turn') + expect(turns).toHaveLength(2) + const deltaSum = turns.reduce( + (total, span) => + total + + Number(span.attributes['llm.token_count.prompt'] ?? 0) + + Number(span.attributes['llm.token_count.completion'] ?? 0), + 0, + ) + expect(deltaSum).toBe(3_050) + expect(root.attributes['traces.session.total_tokens']).not.toBe(deltaSum) + }) + + it('records no total when the harness reported none', async () => { + const ref = writeRollout(dir, 'token-counter-absent', [ + { t: 0, type: 'session_meta', payload: { id: 'token-absent', cwd: '/workspace/demo' } }, + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-1' } }, + { t: 2, type: 'event_msg', payload: { type: 'token_count', info: { last_token_usage: { input_tokens: 10, output_tokens: 2 } } } }, + { t: 3, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-1' } }, + ]) + const spans = await parse(ref) + // Missing stays missing: an unknown total must not become zero. + expect(spans[0]!.attributes['traces.session.total_tokens']).toBeUndefined() + expect(spans[0]!.attributes['traces.session.total_tokens_source']).toBeUndefined() + }) +}) + +describe('Codex synthesized subagent spans', () => { + const subagentRollout = () => + writeRollout(dir, 'synthesized-subagents', [ + { t: 0, type: 'session_meta', payload: { id: 'synth-session', cwd: '/workspace/demo' } }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-1' } }, + userMessage(1.5, 'Audit the parser and report back.'), + tokenCount(2, { input_tokens: 100, output_tokens: 10 }, { input_tokens: 100, output_tokens: 10, total_tokens: 110 }), + toolCall(3, 'call-1', 'exec_command', 'rm -rf build'), + toolOutput(3.5, 'call-1'), + toolCall(4, 'call-2', 'spawn_agent', 'parser_audit'), + toolOutput(4.5, 'call-2'), + toolCall(5, 'call-3', 'exec_command', 'curl -X POST https://example.test/hook'), + toolOutput(5.5, 'call-3'), + subagentActivity(6, 'thread-a', '/root/parser_audit', 'started'), + subagentActivity(7, 'thread-b', '/root/runtime_audit', 'started'), + subagentActivity(8, 'thread-a', '/root/parser_audit', 'completed'), + subagentActivity(9, 'thread-b', '/root/runtime_audit', 'completed'), + { t: 10, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-1' } }, + ]) + + it('keeps the lifecycle span out of every tool-call count', async () => { + const spans = await parse(subagentRollout()) + const synthesized = spans.filter((span) => isSynthesizedSpan(span.attributes)) + const toolSpans = spans.filter((span) => kindOf(span) === 'TOOL') + + expect(synthesized).toHaveLength(2) + expect(synthesized.map((span) => span.name)).toEqual(['subagent.lifecycle', 'subagent.lifecycle']) + expect(synthesized.every((span) => kindOf(span) === 'AGENT')).toBe(true) + expect(synthesized.every((span) => span.attributes['tool.name'] === undefined)).toBe(true) + expect(synthesized.map((span) => span.attributes['traces.codex.subagent_type'])) + .toEqual(['parser_audit', 'runtime_audit']) + + // Before and after, on the same rollout. The previous schema counted a span + // as a tool call when it was TOOL-kind, which included both lifecycle spans. + const countedBefore = spans.filter( + (span) => kindOf(span) === 'TOOL' || isSynthesizedSpan(span.attributes), + ).length + const countedAfter = toolSpans.length + expect(countedBefore - countedAfter).toBe(synthesized.length) + expect(countedAfter).toBe(3) + expect(toolSpans.map((span) => span.attributes['tool.name'])) + .toEqual(['exec_command', 'spawn_agent', 'exec_command']) + }) + + it('reports the model-issued count through the evidence, live, and pipeline paths', async () => { + const ref = subagentRollout() + const spans = await parse(ref) + + const record = await buildPolicyEvidenceRecord(ref, spans) + expect(record.metrics.toolCallCount).toBe(3) + expect(record.metrics.tools.map((tool) => tool.name).sort()).toEqual(['exec_command', 'spawn_agent']) + expect(record.metrics.tools.find((tool) => tool.name === 'Agent')).toBeUndefined() + + expect(analyzeLiveBatch(spans).toolCallCount).toBe(3) + + const pipelines = await runPipelines(spans) + expect(pipelines.toolUse.reduce((total, run) => total + run.totalCalls, 0)).toBe(3) + }) +}) + +describe('Codex inherited context', () => { + const CHILD_ID = 'child-thread-1' + const PARENT_ASK = 'Start with the parser, not the reporter.' + const PARENT_FOLLOW_UP = 'Keep the fixture list short.' + + const forkRollout = () => + writeRollout(dir, 'fork-inherited', [ + { + t: 0, + type: 'session_meta', + payload: { + id: CHILD_ID, + cwd: '/workspace/demo', + thread_source: 'subagent', + source: { subagent: { thread_spawn: { parent_thread_id: 'parent-thread-1', depth: 1, agent_path: '/root/parser_audit' } } }, + }, + }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + // The parent's task, copied into the fork's prefix. + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'parent-turn-1' } }, + userMessage(2, PARENT_ASK), + { t: 3, type: 'event_msg', payload: { type: 'user_message', message: PARENT_FOLLOW_UP } }, + compacted(4, 'Summary of the first window.', 1, [PARENT_ASK, PARENT_FOLLOW_UP]), + // A second compaction repeats the same retained turns. + compacted(5, 'Summary of the second window.', 2, [PARENT_ASK, 'and keep the budget under an hour.']), + toolCall(6, 'parent-call-1', 'exec_command', 'rm -rf parent-build'), + toolOutput(6.5, 'parent-call-1'), + // The fork boundary: everything below is this session's own scope. + { t: 10, type: 'event_msg', payload: { type: 'task_started', turn_id: CHILD_ID } }, + userMessage(11, 'Audit the parser and report back.'), + tokenCount(12, { input_tokens: 100, output_tokens: 10 }, { input_tokens: 100, output_tokens: 10, total_tokens: 110 }), + toolCall(13, 'child-call-1', 'exec_command', 'rm -rf child-build'), + toolOutput(13.5, 'child-call-1'), + { t: 14, type: 'event_msg', payload: { type: 'task_complete', turn_id: CHILD_ID } }, + ]) + + it('keeps the pre-fork prefix and compacted history as marked spans', async () => { + const ref = forkRollout() + const spans = await new CodexAdapter().parse(ref, { captureSources: true }) + const root = spans[0]! + expect(root.attributes['traces.codex.task_scope']).toBe('fork-current') + + const inheritedSpans = inherited(spans) + expect(root.attributes[INHERITED_SPAN_COUNT_ATTR]).toBe(inheritedSpans.length) + + const inheritedPrompts = inheritedSpans.filter((span) => span.name === 'user.prompt') + // The human's words reach a span, once each, however many records repeat them. + expect(inheritedPrompts.map((span) => span.attributes.content)).toEqual([ + PARENT_ASK, + PARENT_FOLLOW_UP, + 'and keep the budget under an hour.', + ]) + expect(inheritedPrompts.every((span) => span.attributes['tangle.actor'] === 'human')).toBe(true) + expect(inheritedPrompts.map((span) => span.attributes['traces.session.inherited_source'])).toEqual([ + 'pre-task-prefix', + 'pre-task-prefix', + 'compacted', + ]) + // Every inherited quote cites the record it came from. + expect(inheritedPrompts.every((span) => typeof span.attributes['traces.source_record.content'] === 'string')).toBe(true) + + const compactions = inheritedSpans.filter((span) => span.name === 'session.compacted') + expect(compactions.map((span) => span.attributes.content)).toEqual([ + 'Summary of the first window.', + 'Summary of the second window.', + ]) + expect(compactions.map((span) => span.attributes['traces.codex.compaction_window_number'])).toEqual([1, 2]) + expect(compactions[0]!.attributes['traces.codex.compaction_window_id']).toBe('window-1') + }) + + it('leaves this scope own counts and identity untouched', async () => { + const ref = forkRollout() + const spans = await parse(ref) + + const ownPrompts = spans.filter( + (span) => span.name === 'user.prompt' && span.attributes[INHERITED_SPAN_ATTR] !== true, + ) + expect(ownPrompts.map((span) => span.attributes.content)).toEqual(['Audit the parser and report back.']) + // A forked child received its brief from its parent agent, not a person. + expect(ownPrompts[0]!.attributes['tangle.actor']).toBe('agent') + + // The prefix's tool call belongs to the parent's turn and is not parsed. + const toolSpans = spans.filter((span) => kindOf(span) === 'TOOL') + expect(toolSpans.map((span) => span.attributes['input.value'])).toEqual([ + JSON.stringify({ cmd: 'rm -rf child-build' }), + ]) + + const record = await buildPolicyEvidenceRecord(ref, spans) + expect(record.metrics.toolCallCount).toBe(1) + // The acted-in window starts at the fork, not at the parent's first record. + expect(record.metrics.firstSpanAt).toBe(at(10)) + + // The report subject names what THIS scope was asked to do. + expect(sessionReportSource(ref, spans).subject).toBe('Audit the parser and report back.') + }) + + it('counts the inherited records its per-session cap dropped', async () => { + const rows: Row[] = [ + { t: 0, type: 'session_meta', payload: { id: 'cap-session', cwd: '/workspace/demo' } }, + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-1' } }, + ] + // 260 distinct inherited turns against a cap of 200. + for (let index = 0; index < 260; index += 1) { + rows.push(userMessage(2 + index * 0.001, `inherited turn ${index}`)) + } + rows.push({ t: 3, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-1' } }) + rows.push({ t: 4, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-2' } }) + rows.push(userMessage(5, 'the turn in scope')) + rows.push({ t: 6, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-2' } }) + const ref = writeRollout(dir, 'inherited-cap', rows) + + const spans = await new CodexAdapter().parse(ref, { taskScope: 'latest' }) + const root = spans[0]! + expect(root.attributes[INHERITED_SPAN_COUNT_ATTR]).toBe(200) + // What the cap turned away is reported, not silently dropped. + expect(root.attributes[INHERITED_SPANS_OMITTED_ATTR]).toBe(60) + expect(inherited(spans)).toHaveLength(200) + }) + + it('keeps a compacted record inside the parsed scope as inherited context', async () => { + const ref = writeRollout(dir, 'compaction-in-scope', [ + { t: 0, type: 'session_meta', payload: { id: 'compaction-session', cwd: '/workspace/demo' } }, + { t: 1, type: 'event_msg', payload: { type: 'task_started', turn_id: 'turn-1' } }, + userMessage(2, 'Ship the parser fix.'), + compacted(3, 'Summary of the first window.', 1, ['Ship the parser fix.', 'and add a regression test.']), + toolCall(4, 'call-1', 'exec_command', 'rm -rf build'), + toolOutput(4.5, 'call-1'), + { t: 5, type: 'event_msg', payload: { type: 'task_complete', turn_id: 'turn-1' } }, + ]) + const spans = await parse(ref) + const inheritedSpans = inherited(spans) + + expect(inheritedSpans.map((span) => span.name)).toEqual([ + 'session.compacted', + 'user.prompt', + 'user.prompt', + ]) + expect(inheritedSpans.every((span) => span.attributes['traces.session.inherited_source'] === 'compacted')).toBe(true) + // The turn this scope actually received keeps its own span. + const ownPrompts = spans.filter( + (span) => span.name === 'user.prompt' && span.attributes[INHERITED_SPAN_ATTR] !== true, + ) + expect(ownPrompts.map((span) => span.attributes.content)).toEqual(['Ship the parser fix.']) + expect(spans.filter((span) => span.attributes[SYNTHESIZED_SPAN_ATTR] === true)).toHaveLength(0) + }) +}) diff --git a/tests/codex-tool-status.test.ts b/tests/codex-tool-status.test.ts index a059aa7..789f412 100644 --- a/tests/codex-tool-status.test.ts +++ b/tests/codex-tool-status.test.ts @@ -57,6 +57,9 @@ describe.each(['function', 'custom'] as const)('Codex %s tool outcomes', (varian { label: 'explicit failure with optimistic stdout', output: { exit_code: 1, output: 'success' }, code: 'ERROR' }, { label: 'initial process header', output: 'Process exited with code 0\nOutput:\nerror: command failed ENOENT', code: 'OK' }, { label: 'initial failed process header', output: 'Process exited with code 1\nOutput:\nsuccess', code: 'ERROR' }, + { label: 'script receipt with wall-time colon', output: 'Script completed\nWall time: 1.2 seconds\nOutput:\nerror: command failed ENOENT', code: 'OK' }, + { label: 'script receipt without wall-time colon', output: 'Script completed\nWall time 1.2 seconds\nOutput:', code: 'OK' }, + { label: 'failed script receipt without wall-time colon', output: 'Script failed\nWall time 1.2 seconds\nOutput:\nsuccess', code: 'ERROR' }, { label: 'initial script error receipt', output: 'Script error:\nExit code: 1\nOutput:\nsuccess', code: 'ERROR' }, { label: 'initial script zero exit receipt', output: 'Script error:\nExit code: 0\nOutput:\nerror: source code', code: 'OK' }, { label: 'script receipt with CRLF', output: 'Script error:\r\nExit code: -1\r\nOutput:\r\nsuccess', code: 'ERROR' }, diff --git a/tests/finding-rejections.test.ts b/tests/finding-rejections.test.ts new file mode 100644 index 0000000..7221f26 --- /dev/null +++ b/tests/finding-rejections.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from 'vitest' +import { + createFindingRejectionTally, + findingRejection, + findingRejectionDetail, + formatFindingRejections, +} from '../src/finding-rejections.js' + +describe('finding rejections', () => { + it('reads the analyst and the actionable reason from registry and direct log events', () => { + expect(findingRejection('[failure-mode] finding rejected: unresolved evidence', { + uri: 'trace://t/span/s', + reason: 'excerpt is not present in the cited span content', + })).toEqual({ analystId: 'failure-mode', reason: 'excerpt is not present in the cited span content' }) + expect(findingRejection('finding rejected: insufficient evidence citations', { required: 2, distinct: 1 })) + .toEqual({ reason: 'insufficient evidence citations' }) + expect(findingRejection('[failure-mode] trace analyst failure-mode completed', {})).toBeUndefined() + }) + + it('formats one stderr detail with the cited URI and citation counts', () => { + expect(findingRejectionDetail('[q1] finding rejected: unresolved evidence', { + uri: 'trace://t/span/s', + reason: 'trace span does not exist', + })).toBe('trace span does not exist; uri trace://t/span/s') + expect(findingRejectionDetail('finding rejected: insufficient evidence citations', { required: 2, distinct: 1 })) + .toBe('insufficient evidence citations; 1 of 2 required distinct citation(s)') + expect(findingRejectionDetail('[analyst] ok failure-mode', {})).toBeUndefined() + }) + + it('counts per analyst, with a default for unprefixed events', () => { + const tally = createFindingRejectionTally('q1') + tally.record('finding rejected: unresolved evidence', { reason: 'trace span does not exist' }) + tally.record('finding rejected: unresolved evidence', { reason: 'trace span does not exist' }) + tally.record('[improvement] finding rejected: schema failure', {}) + tally.record('trace analyst engine started', {}) + expect(tally.counts()).toEqual({ + q1: { 'trace span does not exist': 2 }, + improvement: { 'schema failure': 1 }, + }) + expect(formatFindingRejections(tally.counts().q1)).toBe('2 finding(s) rejected: trace span does not exist ×2') + expect(formatFindingRejections(undefined)).toBe('') + }) +}) diff --git a/tests/improvement.test.ts b/tests/improvement.test.ts index 5ca0d49..5a26556 100644 --- a/tests/improvement.test.ts +++ b/tests/improvement.test.ts @@ -551,6 +551,50 @@ describe('agentic failure surfacing', () => { expect(result.report).toContain('DSPY-BRIDGE-FAILURE: ValueError: analyze input must contain exactly') }) + it('counts evidence-gate rejections per analyst and names the reason in the report', async () => { + const fabricating: TraceAnalysisEngine = { + id: 'fabricating-test-engine', + description: 'Submits one finding whose excerpt the cited span does not contain.', + model: 'test-model', + version: '1.0.0', + executionConfig: {}, + analyze: async () => ({ + answer: 'The agent retried a failing command.', + findings: [{ + severity: 'medium', + claim: 'The agent retried npm test three times without changing anything.', + confidence: 0.8, + evidence: [ + { uri: 'trace://trace-improve/span/tool-0', excerpt: 'this text is not in the span' }, + { uri: 'trace://trace-improve/span/tool-1' }, + ], + }], + trajectory: [], + modelCalls: 1, + toolCalls: 0, + runtime: {}, + }), + } + const logged: string[] = [] + const result = await runTraceInvestigation({ + spans: fixtureSpans(), + harness: 'synthetic', + engine: fabricating, + generatedAt: '2026-01-01T00:00:00.000Z', + log: (msg) => logged.push(msg), + }) + + expect(result.findingRejections?.['failure-mode']).toEqual({ + 'excerpt is not present in the cited span content': 1, + }) + expect(result.findings.some((finding) => finding.analyst_id === 'failure-mode')).toBe(false) + expect(result.report).toMatch( + /\| `failure-mode` \| ok \| 0 \| \d+ms \| 1 finding\(s\) rejected: excerpt is not present in the cited span content ×1 \|/, + ) + // The caller's log still receives every event the tally counted. + expect(logged).toContain('[failure-mode] finding rejected: unresolved evidence') + }) + it('leaves agenticPerAnalyst unset for a deterministic-only run', async () => { const result = await runTraceInvestigation({ spans: fixtureSpans(), @@ -558,6 +602,7 @@ describe('agentic failure surfacing', () => { generatedAt: '2026-01-01T00:00:00.000Z', }) expect(result.agenticPerAnalyst).toBeUndefined() + expect(result.findingRejections).toBeUndefined() expect(totalAgenticFailureMessage(result.agenticPerAnalyst)).toBeUndefined() }) diff --git a/tests/report.test.ts b/tests/report.test.ts index fb0affd..eb57036 100644 --- a/tests/report.test.ts +++ b/tests/report.test.ts @@ -701,4 +701,33 @@ describe('analystRunDetail', () => { usage, })).toBe('—') }) + + it('names the evidence gate\'s rejection reasons for an analyst that returned no findings', () => { + const summary = { + analyst_id: 'failure-mode', + status: 'ok', + findings_count: 0, + latency_ms: 5, + usage, + } as const + expect(analystRunDetail(summary, { + 'excerpt is not present in the cited span content': 2, + 'trace span does not exist': 3, + })).toBe( + '5 finding(s) rejected: trace span does not exist ×3; excerpt is not present in the cited span content ×2', + ) + expect(analystRunDetail(summary, {})).toBe('—') + + const result = emptyResult() + result.per_analyst.push(summary) + const report = renderReport(result, { + harness: 'codex', + sessionCount: 1, + spanCount: 10, + otlpPath: '/tmp/spans.openinference.jsonl', + execution: EMPTY_EXECUTION, + findingRejections: { 'failure-mode': { 'insufficient evidence citations': 1 } }, + }) + expect(report).toContain('| `failure-mode` | ok | 0 | 5ms | 1 finding(s) rejected: insufficient evidence citations ×1 |') + }) }) diff --git a/tests/session-facts-changed-files.test.ts b/tests/session-facts-changed-files.test.ts new file mode 100644 index 0000000..d44ea00 --- /dev/null +++ b/tests/session-facts-changed-files.test.ts @@ -0,0 +1,182 @@ +/** + * Which paths the facts sheet names as changed, and from which source. + * + * The measured failure this replaces: the sheet recovered every path from the + * `apply_patch` headers kept in `input.value`, so a patch a code-mode script + * generated arrived with the script's own `${path}` in the header while the + * `file.change` spans the harness writes — which already carry the path the + * edit reached — were never read. On one private holdout session that cost the + * sheet three of fifteen paths and added one that named no file. + * + * Every record below is invented for this test and parsed through the real + * Codex adapter, so the rule is exercised end to end rather than against spans + * a test built by hand. + */ + +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { CodexAdapter } from '../src/adapters/codex.js' +import { computeSessionFacts, FACT_LIST_CAP, type SessionFacts } from '../src/session-facts.js' + +const dir = mkdtempSync(join(tmpdir(), 'traces-changed-files-')) +afterAll(() => rmSync(dir, { recursive: true, force: true })) + +function at(second: number): string { + return new Date(Date.UTC(2026, 8, 9, 12, 0, second)).toISOString() +} + +/** One `apply_patch` call and its output, as Codex records a function call. */ +function patchCall(callId: string, second: number, patch: string): Record[] { + return [ + { + type: 'response_item', + timestamp: at(second), + payload: { type: 'function_call', call_id: callId, name: 'apply_patch', arguments: JSON.stringify({ input: patch }) }, + }, + { + type: 'response_item', + timestamp: at(second + 1), + payload: { type: 'function_call_output', call_id: callId, output: JSON.stringify({ exit_code: 0 }) }, + }, + ] +} + +/** One shell call whose script writes a patch, as a code-mode session records it. */ +function scriptCall(callId: string, second: number, script: string): Record[] { + return [ + { + type: 'response_item', + timestamp: at(second), + payload: { type: 'function_call', call_id: callId, name: 'exec_command', arguments: JSON.stringify({ cmd: script }) }, + }, + { + type: 'response_item', + timestamp: at(second + 1), + payload: { type: 'function_call_output', call_id: callId, output: JSON.stringify({ exit_code: 0 }) }, + }, + ] +} + +/** The harness's own record of what a patch changed. */ +function fileChangeItem( + itemId: string, + second: number, + changes: Record>, + status = 'completed', +): Record { + return { + type: 'event_msg', + timestamp: at(second), + payload: { type: 'item_completed', item: { id: itemId, type: 'FileChange', status, changes } }, + } +} + +async function facts(name: string, records: readonly Record[]): Promise { + const path = join(dir, `${name}.jsonl`) + writeFileSync(path, `${records.map((record) => JSON.stringify(record)).join('\n')}\n`) + const spans = await new CodexAdapter().parse({ harness: 'codex', sessionId: name, path, cwd: '/repo', mtimeMs: 0 }) + const [session, ...rest] = computeSessionFacts(spans) + expect(rest).toEqual([]) + return session! +} + +const META = { type: 'session_meta', timestamp: at(0), payload: { id: 'changed-files', cwd: '/repo' } } + +function paths(session: SessionFacts): string[] { + return (session.changedFiles.value ?? []).map((file) => file.path) +} + +describe('changed files', () => { + it('takes the harness path for a patch a code-mode script generated', async () => { + // The script interpolates the path, so the header the patch text carries is + // the template literal, not a file. The harness resolved it before applying. + const session = await facts('code-mode', [ + META, + ...scriptCall('call-script', 2, [ + 'node -e "', + 'const target = process.env.TARGET;', + 'writePatch(`*** Begin Patch', + '*** Update File: ${target}', + '*** End Patch`);', + '"', + ].join('\n')), + fileChangeItem('item-script', 4, { '/repo/src/target.ts': { type: 'update' } }), + ]) + + expect(paths(session)).toEqual(['/repo/src/target.ts']) + expect(session.changedFiles.value?.[0]?.operations).toEqual(['update']) + expect(session.changedFiles.partial).toContain('unexpanded variable') + }) + + it('drops a path written through a shell variable and keeps the resolved one', async () => { + const session = await facts('shell-variable', [ + META, + ...scriptCall('call-sh', 2, [ + 'OUT=generated', + 'apply_patch < path.includes('$'))).toBe(false) + }) + + it('counts an edit both sources saw once, with both spans as its evidence', async () => { + // The header names the path relative to the directory the harness resolved + // it against, so the two sources describe one edit under two spellings. + const session = await facts('both-sources', [ + META, + ...patchCall('call-patch', 2, ['*** Begin Patch', '*** Update File: src/upload.ts', '*** End Patch'].join('\n')), + fileChangeItem('item-patch', 4, { '/repo/src/upload.ts': { type: 'update' } }), + ]) + + expect(paths(session)).toEqual(['/repo/src/upload.ts']) + expect(session.changedFiles.value?.[0]?.spanIds.length).toBe(2) + expect(session.changedFiles.partial).toBeUndefined() + }) + + it('still recovers header paths in a session that records no file change', async () => { + const session = await facts('headers-only', [ + META, + ...patchCall('call-legacy', 2, [ + '*** Begin Patch', + '*** Update File: /repo/src/legacy.ts', + '*** Add File: /repo/src/added.ts', + '*** End Patch', + ].join('\n')), + ]) + + expect(paths(session)).toEqual(['/repo/src/added.ts', '/repo/src/legacy.ts']) + }) + + it('lists every path of a session that changed more files than a text list holds', async () => { + // The list cap bounds the sheet against entries carrying whole messages. A + // path costs a fraction of that, and a run that edited hundreds of files is + // the one whose list a reader needs whole. + const many = FACT_LIST_CAP * 2 + const changes = Object.fromEntries( + Array.from({ length: many }, (_unused, index) => [`/repo/src/file-${index}.ts`, { type: 'update' }]), + ) + const session = await facts('over-the-cap', [META, fileChangeItem('item-many', 2, changes)]) + + expect(session.changedFiles.value?.length).toBe(many) + expect(session.changedFiles.partial).toBeUndefined() + }) + + it('names the rename destination and leaves a declined change out', async () => { + const session = await facts('rename-and-declined', [ + META, + fileChangeItem('item-move', 2, { '/repo/src/old.ts': { type: 'update', move_path: '/repo/src/new.ts' } }), + fileChangeItem('item-declined', 4, { '/repo/src/rejected.ts': { type: 'add' } }, 'declined'), + ]) + + expect(paths(session)).toEqual(['/repo/src/new.ts', '/repo/src/old.ts']) + }) +}) diff --git a/tests/session-facts-fixture.ts b/tests/session-facts-fixture.ts new file mode 100644 index 0000000..9e7497e --- /dev/null +++ b/tests/session-facts-fixture.ts @@ -0,0 +1,201 @@ +/** + * A synthetic Codex rollout whose facts are known by hand. + * + * Every record here is invented for this test. The shape follows the rollout + * format the Codex adapter parses — `session_meta`, `turn_context`, + * `response_item` calls and outputs, `event_msg` token counts and subagent + * activity — so the facts sheet is exercised through the real adapter rather + * than against spans a test built directly. + */ + +import { mkdtemp, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { CodexAdapter } from '../src/adapters/codex.js' +import type { OtlpSpan } from '../src/otlp.js' + +export const FIXTURE_SESSION_ID = 'facts-fixture-session' +export const FIXTURE_AGENT_PATH = '/root/uploader_audit' +export const FIXTURE_THREAD_ID = 'facts-fixture-child' + +/** What the fixture's human typed, verbatim, in order. */ +export const FIXTURE_HUMAN_TURNS = [ + 'add a retry to the uploader and keep the tests green', + 'also make the backoff configurable', +] as const + +export const FIXTURE_FINAL_ASSISTANT = 'Backoff is configurable now and the suite is green.' +export const FIXTURE_FINAL_SUBAGENT_TEXT = + 'Message Type: FINAL_ANSWER\nThe uploader retries three times and never swallows a 4xx.' + +/** Paths the fixture's patches touch, and how. */ +export const FIXTURE_CHANGED_FILES = [ + { path: '/fixture/src/retry.ts', operations: ['add'] }, + { path: '/fixture/src/upload.ts', operations: ['update'] }, +] as const + +function at(second: number): string { + return new Date(Date.UTC(2026, 8, 9, 12, 0, second)).toISOString() +} + +export const FIXTURE_FIRST_RECORD_AT = at(0) + +/** The cumulative harness total the fixture's `token_count` event reports. */ +export const FIXTURE_TOKEN_TOTAL = 980 + +/** + * Records of the fixture session. + * + * `extraTools` appends N further `exec_command` calls with padded arguments, to + * build a session whose serialized spans exceed the trace tools' byte ceiling + * while the hand-written facts stay exactly derivable. + */ +export function fixtureRecords(extraTools = 0): unknown[] { + const records: unknown[] = [ + { type: 'session_meta', timestamp: at(0), payload: { id: FIXTURE_SESSION_ID, cwd: '/fixture' } }, + { type: 'turn_context', timestamp: at(1), payload: { model: 'gpt-5.4-codex' } }, + { + type: 'response_item', + timestamp: at(2), + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: FIXTURE_HUMAN_TURNS[0] }] }, + }, + { + type: 'event_msg', + timestamp: at(3), + payload: { + type: 'token_count', + info: { last_token_usage: { input_tokens: 900, output_tokens: 80 }, total_token_usage: { input_tokens: 900, output_tokens: 80, total_tokens: 980 } }, + }, + }, + // A plain shell call. + { + type: 'response_item', + timestamp: at(4), + payload: { type: 'function_call', call_id: 'call-shell', name: 'exec_command', arguments: JSON.stringify({ cmd: 'ls src' }) }, + }, + { type: 'response_item', timestamp: at(5), payload: { type: 'function_call_output', call_id: 'call-shell', output: JSON.stringify({ exit_code: 0, output: 'upload.ts' }) } }, + // A verification call: the adapter names it `exec_command.verify`, which is + // the category the measured analyst kept subtracting by hand. + { + type: 'response_item', + timestamp: at(6), + payload: { type: 'function_call', call_id: 'call-verify', name: 'exec_command', arguments: JSON.stringify({ cmd: 'pnpm test' }) }, + }, + { type: 'response_item', timestamp: at(7), payload: { type: 'function_call_output', call_id: 'call-verify', output: JSON.stringify({ exit_code: 0 }) } }, + // A patch naming one updated and one added file. + { + type: 'response_item', + timestamp: at(8), + payload: { + type: 'function_call', + call_id: 'call-patch', + name: 'apply_patch', + arguments: JSON.stringify({ + input: [ + '*** Begin Patch', + '*** Update File: /fixture/src/upload.ts', + '@@', + '- await send(body)', + '+ await withRetry(() => send(body))', + '*** Add File: /fixture/src/retry.ts', + '+export const withRetry = async () => {}', + '*** End Patch', + ].join('\n'), + }), + }, + }, + { type: 'response_item', timestamp: at(9), payload: { type: 'function_call_output', call_id: 'call-patch', output: JSON.stringify({ exit_code: 0 }) } }, + // A subagent spawn, with the task name the adapter promotes to an attribute. + { + type: 'response_item', + timestamp: at(10), + payload: { + type: 'function_call', + call_id: 'call-spawn', + name: 'spawn_agent', + arguments: JSON.stringify({ task_name: FIXTURE_AGENT_PATH, message: 'audit the uploader' }), + }, + }, + { type: 'response_item', timestamp: at(11), payload: { type: 'function_call_output', call_id: 'call-spawn', output: JSON.stringify({ task_name: FIXTURE_AGENT_PATH }) } }, + // The lifecycle events that make the adapter synthesize a `tool.Agent` span. + { + type: 'event_msg', + timestamp: at(12), + payload: { + type: 'sub_agent_activity', + event_id: 'call-spawn', + kind: 'started', + agent_thread_id: FIXTURE_THREAD_ID, + agent_path: FIXTURE_AGENT_PATH, + occurred_at_ms: Date.parse(at(12)), + }, + }, + { + type: 'response_item', + timestamp: at(13), + payload: { type: 'agent_message', author: FIXTURE_AGENT_PATH, recipient: 'root', content: FIXTURE_FINAL_SUBAGENT_TEXT }, + }, + { + type: 'event_msg', + timestamp: at(14), + payload: { + type: 'sub_agent_activity', + event_id: 'call-spawn', + kind: 'completed', + agent_thread_id: FIXTURE_THREAD_ID, + agent_path: FIXTURE_AGENT_PATH, + occurred_at_ms: Date.parse(at(14)), + }, + }, + { + type: 'response_item', + timestamp: at(15), + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'Retry added; running the suite.' }] }, + }, + { + type: 'response_item', + timestamp: at(16), + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: FIXTURE_HUMAN_TURNS[1] }] }, + }, + ] + for (let index = 0; index < extraTools; index += 1) { + const callId = `call-bulk-${index}` + records.push({ + type: 'response_item', + timestamp: at(17), + payload: { + type: 'function_call', + call_id: callId, + name: 'exec_command', + arguments: JSON.stringify({ cmd: `rg --files-with-matches token ${'padding/'.repeat(40)}${index}` }), + }, + }) + records.push({ + type: 'response_item', + timestamp: at(17), + payload: { type: 'function_call_output', call_id: callId, output: JSON.stringify({ exit_code: 0, output: 'x'.repeat(400) }) }, + }) + } + records.push({ + type: 'response_item', + timestamp: at(18), + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: FIXTURE_FINAL_ASSISTANT }] }, + }) + return records +} + +export const FIXTURE_LAST_RECORD_AT = at(18) + +/** Parse the fixture through the real Codex adapter. */ +export async function fixtureSpans(extraTools = 0): Promise { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-fixture-')) + const path = join(dir, `${FIXTURE_SESSION_ID}.jsonl`) + await writeFile(path, `${fixtureRecords(extraTools).map((record) => JSON.stringify(record)).join('\n')}\n`, 'utf8') + return new CodexAdapter().parse({ + harness: 'codex', + sessionId: FIXTURE_SESSION_ID, + path, + cwd: '/fixture', + mtimeMs: 0, + }) +} diff --git a/tests/session-facts-human-turns.test.ts b/tests/session-facts-human-turns.test.ts new file mode 100644 index 0000000..35d5c6b --- /dev/null +++ b/tests/session-facts-human-turns.test.ts @@ -0,0 +1,232 @@ +/** + * Which `user.prompt` turns are turns a person typed into THIS session. + * + * The measured gap this closes: on a private battery of thirteen Codex + * sessions the free facts sheet counted every span whose actor was `human`, + * including the history a forked session copies from its parent and the turns a + * compaction replays. It scored 0.076 on "how many user messages are there, and + * what do the first and last say?" while the spans already carried the answer. + * + * The rule the tests hold the sheet to is the one an auditor applies by hand: a + * user message is one the human typed. Instruction files, environment-context + * blocks, subagent notifications and skill expansions are the harness feeding + * the model; a turn recorded twice is one turn; and history a session carries + * but did not receive is not a turn of this session. + * + * Every rollout below is written here by hand. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { CodexAdapter } from '../src/adapters/codex.js' +import type { OtlpSpan } from '../src/otlp.js' +import { computeSessionFacts, type SessionFacts } from '../src/session-facts.js' +import { + AGENTS_BLOCK, + ENVIRONMENT_BLOCK, + ms, + type Row, + task, + userItem, + userItemCompleted, + writeRollout as writeRolloutIn, +} from './codex-facts-fixture.js' + +const dir = mkdtempSync(join(tmpdir(), 'traces-facts-turns-')) +afterAll(() => rmSync(dir, { recursive: true, force: true })) + +const writeRollout = (name: string, rows: readonly Row[]) => writeRolloutIn(dir, name, rows) + +/** A user-role response item carrying Codex's own per-item labelling. */ +const labelled = (t: number, texts: readonly string[], kinds: readonly string[]): Row => ({ + t, + type: 'response_item', + payload: { + type: 'message', + role: 'user', + content: texts.map((text) => ({ type: 'input_text', text })), + internal_chat_message_metadata_passthrough: { turn_id: 'turn-1', content_item_kinds: kinds }, + }, +}) + +async function factsFor(name: string, rows: readonly Row[]): Promise { + const [facts, ...rest] = computeSessionFacts(await new CodexAdapter().parse(writeRollout(name, rows))) + expect(rest).toEqual([]) + return facts! +} + +const texts = (facts: SessionFacts) => facts.humanTurns.value!.map((turn) => turn.text) +const excludedFor = (facts: SessionFacts, fragment: string) => + facts.excludedTurns.value!.filter((entry) => entry.reason.includes(fragment)) + +const TYPED = 'rerun the parser tests and tell me which one still fails' +const FOLLOW_UP = 'do it' + +describe('session facts: human turns', () => { + it('counts the typed turns and no injected block, whatever the block is', async () => { + const facts = await factsFor('injected', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + // An AGENTS.md file and an environment block, both recorded as user-role + // messages, both labelled by Codex as what they are. + labelled(0.6, [AGENTS_BLOCK], ['agents_md.instructions']), + labelled(0.7, [ENVIRONMENT_BLOCK], ['environments.environment_context']), + task(1, 'task_started', 'turn-1'), + labelled(2, [TYPED], ['user.text']), + { + t: 3, + type: 'response_item', + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'one fails' }] }, + }, + // A block the harness added after the last typed turn: a reader that + // trusts the user role reports this as the session's last human message. + labelled(4, [ENVIRONMENT_BLOCK.replace('zsh', 'fish')], ['environments.environment_context']), + task(5, 'task_complete', 'turn-1'), + ]) + + expect(texts(facts)).toEqual([TYPED]) + expect(facts.humanTurns.value![0]!.at).toBe(new Date(ms(2)).toISOString()) + expect(excludedFor(facts, 'harness-injected content')).toEqual([ + expect.objectContaining({ turns: 3 }), + ]) + // Every excluded span is named, so the exclusion can be checked or undone. + expect(excludedFor(facts, 'harness-injected content')[0]!.spanIds).toHaveLength(3) + expect(facts.turnsByActor.value).toEqual([ + expect.objectContaining({ actor: 'human', turns: 1 }), + expect.objectContaining({ actor: 'injected', turns: 3 }), + ]) + }) + + it('counts a turn once when the rollout records it twice', async () => { + const facts = await factsFor('twice', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + // Codex logs one submitted turn as a response item and as an + // `item_completed` UserMessage. Both records describe the same typing. + labelled(2, [TYPED], ['user.text']), + userItemCompleted(2.001, 'item-user-1', TYPED), + task(3, 'task_complete', 'turn-1'), + task(4, 'task_started', 'turn-2'), + userItemCompleted(5, 'item-user-2', FOLLOW_UP), + labelled(5.001, [FOLLOW_UP], ['user.text']), + task(6, 'task_complete', 'turn-2'), + ]) + expect(texts(facts)).toEqual([TYPED, FOLLOW_UP]) + }) + + it('collapses a duplicate the adapter did not pair, and says it did', async () => { + const spans = await new CodexAdapter().parse(writeRollout('backstop', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + labelled(2, [TYPED], ['user.text']), + task(3, 'task_complete', 'turn-1'), + ])) + const turn = spans.find((span) => span.name === 'user.prompt')! + // A second record of the same submission, from a harness whose two logs the + // adapter could not pair: one instant, one text, nothing in between. + const echo: OtlpSpan = { + ...turn, + span_id: `${turn.span_id}-echo`, + attributes: { ...turn.attributes, step: Number(turn.attributes.step) + 0.5 }, + } + const [facts] = computeSessionFacts([...spans, echo]) + expect(texts(facts!)).toEqual([TYPED]) + expect(excludedFor(facts!, 'a second record of the turn before it')).toEqual([ + { reason: expect.stringContaining('a second record'), turns: 1, spanIds: [echo.span_id] }, + ]) + }) + + it('keeps a repeated message the person actually sent twice', async () => { + const facts = await factsFor('repeat', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + labelled(2, [FOLLOW_UP], ['user.text']), + { + t: 3, + type: 'response_item', + payload: { type: 'function_call', call_id: 'call-1', name: 'exec_command', arguments: JSON.stringify({ cmd: 'ls' }) }, + }, + { t: 3.5, type: 'response_item', payload: { type: 'function_call_output', call_id: 'call-1', output: 'src\n' } }, + task(4, 'task_complete', 'turn-1'), + task(5, 'task_started', 'turn-2'), + labelled(6, [FOLLOW_UP], ['user.text']), + task(7, 'task_complete', 'turn-2'), + ]) + expect(texts(facts)).toEqual([FOLLOW_UP, FOLLOW_UP]) + expect(excludedFor(facts, 'a second record of the turn before it')).toEqual([]) + }) + + it('keeps a message queued twice while the agent was still working', async () => { + // Measured on a real session: a person typed "continue" twice, 1.8 s apart, + // with nothing recorded in between because the agent was mid-turn. A + // duplicate rule that allowed any short gap counted that as one turn. + const facts = await factsFor('queued', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + labelled(2, ['continue'], ['user.text']), + labelled(3.8, ['continue'], ['user.text']), + task(4, 'task_complete', 'turn-1'), + ]) + expect(texts(facts)).toEqual(['continue', 'continue']) + expect(excludedFor(facts, 'a second record of the turn before it')).toEqual([]) + }) + + it('does not count history a forked session copied from its parent', async () => { + const facts = await factsFor('fork', [ + { + t: 0, + type: 'session_meta', + payload: { + id: 'child-session', + cwd: '/workspace/demo', + parent_thread_id: 'parent-session', + thread_source: 'subagent', + }, + }, + // The parent's history, rewritten into the child's rollout with the fork + // time. A person typed it — into the parent thread, not into this session. + labelled(0.6, ['what is slow about the uploader?'], ['user.text']), + labelled(0.7, ['and what did you try already?'], ['user.text']), + // The child's own task begins here. + task(1, 'task_started', 'child-session'), + { + t: 2, + type: 'response_item', + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'retries dominate' }] }, + }, + task(3, 'task_complete', 'child-session'), + ]) + + expect(facts.humanTurns.value).toEqual([]) + const inherited = excludedFor(facts, 'context this session carries but did not receive') + expect(inherited).toEqual([expect.objectContaining({ turns: 2 })]) + expect(inherited[0]!.reason).toContain('pre-task-prefix') + // The words are still in the trace; only the count excludes them. + expect(facts.turnsByActor.value).toEqual([expect.objectContaining({ actor: 'human', turns: 2 })]) + }) + + it("takes Codex's own item labelling over what the text looks like", async () => { + const brief = `You are the reviewer for this change. ${'Read every file. '.repeat(120)}` + const labelledFacts = await factsFor('brief-labelled', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + labelled(2, [brief], ['user.text']), + task(3, 'task_complete', 'turn-1'), + ]) + // Long, and it opens like an agent brief — but Codex recorded it as the + // person's own text, and that is what the session log says happened. + expect(labelledFacts.humanTurns.value).toHaveLength(1) + + const unlabelledFacts = await factsFor('brief-unlabelled', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: brief }]), + task(3, 'task_complete', 'turn-1'), + ]) + // Without the labelling there is nothing structural to go on, so the text + // heuristic still calls a first-turn agent brief an injected prompt. + expect(unlabelledFacts.humanTurns.value).toEqual([]) + expect(excludedFor(unlabelledFacts, 'harness-injected content')).toHaveLength(1) + }) +}) diff --git a/tests/session-facts-pull-requests.test.ts b/tests/session-facts-pull-requests.test.ts new file mode 100644 index 0000000..fc26808 --- /dev/null +++ b/tests/session-facts-pull-requests.test.ts @@ -0,0 +1,268 @@ +/** + * The pull requests a session created and merged, read from the command spans. + * + * The measured gap this closes: on a private battery of thirteen Codex + * sessions the free facts sheet answered "which pull requests did the agent + * create, and which did it merge?" with nothing at all, scoring 0.31 against a + * subagent fleet's 0.97 — while every command, exit code and printed + * pull-request URL was already in the spans. + * + * Every rollout below is written here by hand. None comes from a recorded + * session, and the URLs point at `example.test`. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { CodexAdapter } from '../src/adapters/codex.js' +import { computeSessionFacts, type SessionFacts } from '../src/session-facts.js' +import { + at, + command, + commandItem, + ms, + type Row, + script, + scriptOutput, + task, + userItem, + writeRollout as writeRolloutIn, +} from './codex-facts-fixture.js' + +const dir = mkdtempSync(join(tmpdir(), 'traces-facts-prs-')) +afterAll(() => rmSync(dir, { recursive: true, force: true })) + +const writeRollout = (name: string, rows: readonly Row[]) => writeRolloutIn(dir, name, rows) + +/** A model-issued `exec_command` call and the output it got back. */ +const execCall = (t: number, callId: string, cmd: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'function_call', call_id: callId, name: 'exec_command', arguments: JSON.stringify({ cmd }) }, +}) +const execOutput = (t: number, callId: string, output: string): Row => ({ + t, + type: 'response_item', + payload: { type: 'function_call_output', call_id: callId, output: JSON.stringify({ exit_code: 0, output }) }, +}) + +async function factsFor(name: string, rows: readonly Row[]): Promise { + const spans = await new CodexAdapter().parse(writeRollout(name, rows)) + const [facts, ...rest] = computeSessionFacts(spans) + expect(rest).toEqual([]) + return facts! +} + +const ids = (entries: readonly { identifier: string | null }[]) => entries.map((entry) => entry.identifier) + +const HEREDOC_NOTE = [ + "cat > /tmp/pr-note.md <<'EOF'", + 'Reviewers: run gh pr create --head feat/never-ran when the branch is ready,', + 'then gh-drew pr merge 999 --squash.', + 'EOF', +].join('\n') + +/** + * One shipping turn: a script that pushes and opens a PR, a second create + * through the `gh-drew` wrapper, a create whose stdout was redirected away, a + * heredoc that only talks about `gh pr create`, and a later `gh pr list` whose + * output is the first place the redirected PR's number appears. + */ +const SHIP_SCRIPT = [ + 'const first = await tools.exec_command({ cmd: "git push -u origin feat/parser && gh pr create --base main --fill" })', + 'const second = await tools.exec_command({ cmd: "gh-drew pr create --head feat/docs --base main --fill" })', + 'const third = await tools.exec_command({ cmd: "gh pr create --head feat/quiet --base main --fill > /tmp/pr.txt" })', + 'const note = await tools.exec_command({ cmd: "cat > /tmp/pr-note.md" })', + 'text([first.output, second.output, third.output, note.output].join("\\n"))', +].join('\n') + +function shipRollout(): Row[] { + return [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + { t: 0.5, type: 'turn_context', payload: { cwd: '/workspace/demo', model: 'gpt-fixture' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: 'ship the parser fix and the docs branch' }]), + script(3, 'call-ship', SHIP_SCRIPT), + command( + 4, + commandItem( + 'item-push-create', + '51001', + 'git push -u origin feat/parser && gh pr create --base main --fill', + 0, + "branch 'feat/parser' set up to track 'origin/feat/parser'.\nhttps://example.test/acme/demo/pull/41\n", + ), + { start: 3.1, end: 4 }, + ), + command( + 5, + commandItem('item-drew', '51002', 'gh-drew pr create --head feat/docs --base main --fill', 0, 'https://example.test/acme/demo/pull/42\n'), + { start: 4.1, end: 5 }, + ), + command( + 6, + commandItem('item-quiet', '51003', 'gh pr create --head feat/quiet --base main --fill > /tmp/pr.txt', 0, ''), + { start: 5.1, end: 6 }, + ), + command(7, commandItem('item-note', '51004', HEREDOC_NOTE, 0, ''), { start: 6.1, end: 7 }), + scriptOutput(8, 'call-ship', 'Script completed\nWall time 4.0 seconds\nOutput:\nok'), + // The only record that ties feat/quiet to a number. + execCall(9, 'call-list', 'gh pr list --state open'), + command( + 10, + commandItem( + 'item-list', + '51005', + 'gh pr list --state open', + 0, + 'feat/parser https://example.test/acme/demo/pull/41 OPEN\nfeat/quiet https://example.test/acme/demo/pull/43 OPEN\n', + ), + { start: 9.1, end: 10 }, + ), + execOutput(11, 'call-list', 'feat/quiet https://example.test/acme/demo/pull/43 OPEN'), + // A verification call: the adapter names it `exec_command.verify`, and the + // merge lives inside the script that call ran. + execCall(12, 'call-verify', 'gh-drew pr checks 41 && gh-drew pr merge 41 --squash --match-head-commit deadbeef'), + command( + 13, + commandItem( + 'item-verify', + '51006', + 'gh-drew pr checks 41 && gh-drew pr merge 41 --squash --match-head-commit deadbeef', + 0, + 'All checks were successful\nSquashed and merged pull request demo#41\n', + ), + { start: 12.1, end: 13 }, + ), + execOutput(14, 'call-verify', 'Squashed and merged pull request demo#41'), + task(15, 'task_complete', 'turn-1'), + ] +} + +describe('session facts: pull requests', () => { + it('names every pull request the commands created, however the number reached the spans', async () => { + const facts = await factsFor('ship', shipRollout()) + const created = facts.pullRequests.value!.created + expect(ids(created)).toEqual(['41', '42', '43']) + + const [pushed, wrapper, redirected] = created + // The command printed the URL itself. + expect(pushed).toMatchObject({ number: '41', headBranch: 'feat/parser' }) + expect(pushed!.evidence).toContain('printed') + // The head branch came from the `git push` in the same script, not a flag. + expect(pushed!.command).toContain('gh pr create --base main --fill') + + // The `gh-drew` wrapper is the same command. + expect(wrapper).toMatchObject({ number: '42', headBranch: 'feat/docs' }) + expect(wrapper!.command.startsWith('gh-drew pr create')).toBe(true) + + // stdout went to a file; the number only appears in a later command's output. + expect(redirected).toMatchObject({ number: '43', headBranch: 'feat/quiet' }) + expect(redirected!.evidence).toContain('a later output states for this branch') + expect(redirected!.spanIds.length).toBeGreaterThan(1) + }) + + it('reads a merge that ran inside a verification script', async () => { + const spans = await new CodexAdapter().parse(writeRollout('ship-merge', shipRollout())) + const verify = spans.find((span) => span.attributes['tool.name'] === 'exec_command.verify') + expect(verify).toBeDefined() + + const [facts] = computeSessionFacts(spans) + expect(ids(facts!.pullRequests.value!.merged)).toEqual(['41']) + expect(facts!.pullRequests.value!.merged[0]!.evidence).toContain('the pull request the command names') + }) + + it('does not count a gh command that only appears inside a heredoc body', async () => { + const facts = await factsFor('heredoc', shipRollout()) + const everything = [...facts.pullRequests.value!.created, ...facts.pullRequests.value!.merged] + expect(everything.map((entry) => entry.identifier)).not.toContain('feat/never-ran') + expect(everything.map((entry) => entry.identifier)).not.toContain('999') + }) + + it('cites a real span for every pull request it names', async () => { + const spans = await new CodexAdapter().parse(writeRollout('cites', shipRollout())) + const known = new Set(spans.map((span) => span.span_id)) + const [facts] = computeSessionFacts(spans) + const cited = [...facts!.pullRequests.value!.created, ...facts!.pullRequests.value!.merged] + .flatMap((entry) => entry.spanIds) + expect(cited.length).toBeGreaterThan(0) + for (const spanId of cited) expect(known.has(spanId)).toBe(true) + expect(facts!.pullRequests.spanIds).toEqual(expect.arrayContaining(cited)) + }) + + it('leaves a create the shell never reached out of the list', async () => { + const facts = await factsFor('unreached', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: 'push and open the PR' }]), + execCall(3, 'call-fail', 'git push -u origin feat/broken && gh pr create --fill'), + command( + 4, + commandItem( + 'item-fail', + '52001', + 'git push -u origin feat/broken && gh pr create --fill', + 1, + "error: failed to push some refs to 'origin'\n", + ), + { start: 3.1, end: 4 }, + ), + task(5, 'task_complete', 'turn-1'), + ]) + expect(facts.pullRequests.value).toEqual({ created: [], merged: [] }) + expect(facts.pullRequests.unavailable).toBeNull() + }) + + it('counts one pull request when a failed create is retried on the same branch', async () => { + const facts = await factsFor('retry', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: 'open it' }]), + execCall(3, 'call-a', 'gh pr create --head feat/retry --base main --fill'), + command( + 4, + commandItem( + 'item-a', + '53001', + 'gh pr create --head feat/retry --base main --fill', + 1, + 'pull request create failed: GraphQL: No commits between main and feat/retry\n', + ), + { start: 3.1, end: 4 }, + ), + execCall(5, 'call-b', 'gh pr create --head feat/retry --base main --fill'), + command( + 6, + commandItem('item-b', '53002', 'gh pr create --head feat/retry --base main --fill', 0, 'https://example.test/acme/demo/pull/44\n'), + { start: 5.1, end: 6 }, + ), + task(7, 'task_complete', 'turn-1'), + ]) + expect(ids(facts.pullRequests.value!.created)).toEqual(['44']) + }) + + it('says the spans cannot answer when none of them carries a command', async () => { + const facts = await factsFor('no-commands', [ + { t: 0, type: 'session_meta', payload: { id: 'facts-session', cwd: '/workspace/demo' } }, + task(1, 'task_started', 'turn-1'), + userItem(2, [{ type: 'input_text', text: 'did you open the PR?' }]), + { + t: 3, + type: 'response_item', + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'I opened PR 91.' }] }, + }, + task(4, 'task_complete', 'turn-1'), + ]) + // Zero pull requests and "the spans cannot say" are different answers, and + // an audit that reports the first for the second is stating a fact it has not read. + expect(facts.pullRequests.value).toBeNull() + expect(facts.pullRequests.unavailable).toContain('no span in this trace carries an executed command') + expect(facts.pullRequests.spanIds).toEqual([]) + }) + + it('keeps its start and end times inside the fixture window', async () => { + const facts = await factsFor('window', shipRollout()) + expect(facts.firstRecordAt.value).toBe(at(0)) + expect(Date.parse(facts.lastRecordAt.value!)).toBeLessThanOrEqual(ms(20)) + }) +}) diff --git a/tests/session-facts.test.ts b/tests/session-facts.test.ts new file mode 100644 index 0000000..33ad4d4 --- /dev/null +++ b/tests/session-facts.test.ts @@ -0,0 +1,382 @@ +/** + * The deterministic session-facts sheet, checked against a fixture whose gold + * is written by hand. + * + * The measured failure this replaces: over twelve private audit sessions, the + * model-backed analyst scored a deterministic mean of 0.389 while the same + * facts, extracted mechanically from the spans the run already wrote, scored + * 0.858. Nothing was missing from the spans — no tool returned an exact count, + * so the model added up a capped name histogram and guessed. These tests hold + * the extraction to exactness on a session it can be checked against by eye. + */ + +import { execFile } from 'node:child_process' +import { mkdtemp, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { promisify } from 'node:util' +import { describe, expect, it } from 'vitest' +import { + DEFAULT_TRACE_ANALYST_KINDS, + defineTraceAnalyst, + type TraceAnalysisEngine, + type TraceAnalysisEngineRequest, + type TraceAnalystDefinition, +} from '@tangle-network/agent-eval/analyst' +import type { TraceAnalysisStore } from '@tangle-network/agent-eval/traces' +import { + buildSessionFactsReport, + computeSessionFacts, + PREPARED_CONTEXT_BYTE_CEILING, + renderSessionFacts, + renderSessionFactsContext, + SESSION_FACTS_VERSION_SUFFIX, + SESSION_TOKEN_TOTAL_ATTR, + sessionFactsContext, + withSessionFactsContext, +} from '../src/session-facts.js' +import { SYNTHESIZED_SPAN_ATTR } from '../src/adapters/provenance.js' +import { analyzeSpans } from '../src/analyze.js' +import { serializeSpans } from '../src/otlp.js' +import { + FIXTURE_AGENT_PATH, + FIXTURE_CHANGED_FILES, + FIXTURE_FINAL_ASSISTANT, + FIXTURE_FINAL_SUBAGENT_TEXT, + FIXTURE_FIRST_RECORD_AT, + FIXTURE_HUMAN_TURNS, + FIXTURE_LAST_RECORD_AT, + FIXTURE_SESSION_ID, + FIXTURE_TOKEN_TOTAL, + fixtureRecords, + fixtureSpans, +} from './session-facts-fixture.js' + +const execFileAsync = promisify(execFile) + +/** The trace-tool byte ceiling every benchmark session exceeded. */ +const VIEW_TRACE_BYTE_CEILING = 150_000 + +describe('session facts', () => { + it('states every field exactly, against a hand-written gold', async () => { + const spans = await fixtureSpans() + const [facts, ...rest] = computeSessionFacts(spans) + expect(rest).toEqual([]) + expect(facts!.sessionId).toBe(FIXTURE_SESSION_ID) + expect(facts!.harness).toBe('codex') + + // Four calls the agent made: two shells, one patch, one spawn. + expect(facts!.toolCalls.value).toBe(4) + expect(facts!.toolCalls.unavailable).toBeNull() + expect(facts!.toolCallsByName.value).toEqual({ + 'apply_patch': 1, + 'exec_command': 1, + 'exec_command.verify': 1, + 'spawn_agent': 1, + }) + + expect(facts!.subagents.value).toHaveLength(1) + expect(facts!.subagents.value?.[0]?.taskName).toBe(FIXTURE_AGENT_PATH) + expect(facts!.subagents.value?.[0]?.taskNameUnavailable).toBeNull() + + expect(facts!.humanTurns.value?.map((turn) => turn.text)).toEqual([...FIXTURE_HUMAN_TURNS]) + expect(facts!.humanTurns.value?.map((turn) => turn.actor)).toEqual(['human', 'human']) + expect(facts!.humanTurns.value?.map((turn) => turn.at)).toEqual([ + '2026-09-09T12:00:02.000Z', + '2026-09-09T12:00:16.000Z', + ]) + expect(facts!.turnsByActor.value).toEqual([ + { actor: 'human', turns: 2, spanIds: facts!.humanTurns.value!.map((turn) => turn.spanId) }, + ]) + + // The session's own last word, and the subagent's, kept apart: both stream + // under one root, and picking "the last message" across both is how the + // measured run returned a subagent's report as the session's answer. + expect(facts!.finalMessages.value).toEqual([ + expect.objectContaining({ task: null, text: FIXTURE_FINAL_ASSISTANT }), + expect.objectContaining({ task: FIXTURE_AGENT_PATH, text: FIXTURE_FINAL_SUBAGENT_TEXT }), + ]) + + expect(facts!.changedFiles.value?.map((file) => ({ path: file.path, operations: file.operations }))) + .toEqual(FIXTURE_CHANGED_FILES.map((file) => ({ path: file.path, operations: [...file.operations] }))) + + expect(facts!.firstRecordAt.value).toBe(FIXTURE_FIRST_RECORD_AT) + expect(facts!.lastRecordAt.value).toBe(FIXTURE_LAST_RECORD_AT) + }) + + it('names a real span for every fact it states', async () => { + const spans = await fixtureSpans() + const known = new Set(spans.map((span) => span.span_id)) + const [facts] = computeSessionFacts(spans) + const cited = [ + ...facts!.toolCalls.spanIds, + ...facts!.subagents.spanIds, + ...facts!.humanTurns.spanIds, + ...facts!.finalMessages.spanIds, + ...facts!.changedFiles.spanIds, + ...facts!.firstRecordAt.spanIds, + ...facts!.lastRecordAt.spanIds, + ] + expect(cited.length).toBeGreaterThan(0) + for (const spanId of cited) expect(known.has(spanId)).toBe(true) + }) + + it('says why a fact the spans cannot support is null, instead of guessing', async () => { + const spans = await fixtureSpans() + // A trace from an adapter that does not record the harness's cumulative + // total. Summing the per-turn deltas would answer a different question, so + // the sheet says so rather than reporting the smaller number. + for (const span of spans) delete span.attributes[SESSION_TOKEN_TOTAL_ATTR] + const [facts] = computeSessionFacts(spans) + expect(facts!.tokenTotal.value).toBeNull() + expect(facts!.tokenTotal.unavailable).toContain(SESSION_TOKEN_TOTAL_ATTR) + expect(facts!.tokenTotal.spanIds).toEqual([]) + }) + + it('reports the harness token total the adapter recorded', async () => { + const spans = await fixtureSpans() + const [facts] = computeSessionFacts(spans) + expect(facts!.tokenTotal.value).toBe(FIXTURE_TOKEN_TOTAL) + expect(facts!.tokenTotal.unavailable).toBeNull() + expect(facts!.tokenTotal.spanIds).toHaveLength(1) + }) + + it('reports the harness token total once a span carries it', async () => { + const spans = await fixtureSpans() + const root = spans.find((span) => span.parent_span_id === null)! + root.attributes[SESSION_TOKEN_TOTAL_ATTR] = 17_025_686 + const [facts] = computeSessionFacts(spans) + expect(facts!.tokenTotal.value).toBe(17_025_686) + expect(facts!.tokenTotal.unavailable).toBeNull() + expect(facts!.tokenTotal.spanIds).toEqual([root.span_id]) + }) + + it('does not let a synthesized subagent span inflate the tool count', async () => { + const spans = await fixtureSpans() + // The adapter now marks a subagent's lifecycle span as synthesized and + // keeps it out of the TOOL kind, so a plain span-kind count is already + // right here: four calls, four TOOL spans. + const toolSpans = spans.filter((span) => span.attributes['openinference.span.kind'] === 'TOOL') + expect(toolSpans).toHaveLength(4) + const lifecycle = spans.find((span) => span.attributes[SYNTHESIZED_SPAN_ATTR] === true)! + expect(lifecycle.attributes['openinference.span.kind']).not.toBe('TOOL') + + const [facts] = computeSessionFacts(spans) + expect(facts!.toolCalls.value).toBe(4) + expect(facts!.synthesizedToolSpans.value).toBe(0) + + // A trace exported before that fix still carries the lifecycle span as a + // TOOL call named `Agent`. The sheet must keep counting it out, which is + // the whole reason the exclusion is a field rather than an adapter detail. + const legacy = [ + ...spans, + { ...lifecycle, span_id: `${lifecycle.span_id}-legacy`, name: 'tool.Agent', + attributes: { ...lifecycle.attributes, 'openinference.span.kind': 'TOOL', 'tool.name': 'Agent' } }, + ] + const [legacyFacts] = computeSessionFacts(legacy) + expect(legacyFacts!.toolCalls.value).toBe(4) + expect(legacyFacts!.synthesizedToolSpans.value).toBe(1) + expect(legacyFacts!.toolCallsByName.value).not.toHaveProperty('Agent') + expect(legacyFacts!.toolCalls.spanIds).not.toContain(legacyFacts!.synthesizedToolSpans.spanIds[0]) + }) + + it('stays exact on a session larger than the trace tools can return', async () => { + const spans = await fixtureSpans(400) + // `viewTrace` degrades to a ≤20-entry name histogram above this many bytes, + // which is the surface the measured analyst had to count from. + expect(Buffer.byteLength(serializeSpans(spans))).toBeGreaterThan(VIEW_TRACE_BYTE_CEILING) + + const [facts] = computeSessionFacts(spans) + expect(facts!.toolCalls.value).toBe(404) + expect(facts!.synthesizedToolSpans.value).toBe(0) + expect(facts!.toolCallsByName.value).toEqual({ + 'apply_patch': 1, + 'exec_command': 401, + 'exec_command.verify': 1, + 'spawn_agent': 1, + }) + expect(facts!.humanTurns.value).toHaveLength(2) + expect(facts!.finalMessages.value?.[0]?.text).toBe(FIXTURE_FINAL_ASSISTANT) + expect(facts!.changedFiles.value?.map((file) => file.path)) + .toEqual(FIXTURE_CHANGED_FILES.map((file) => file.path)) + }) + + it('renders a short readable form and a machine-readable report', async () => { + const spans = await fixtureSpans() + const report = buildSessionFactsReport(spans, { harness: 'codex', generatedAt: '2026-09-09T12:30:00.000Z' }) + expect(report.kind).toBe('traces.session_facts_report') + expect(report.sessions).toHaveLength(1) + expect(JSON.parse(JSON.stringify(report))).toMatchObject({ schemaVersion: 1, harness: 'codex' }) + + const text = renderSessionFacts(report) + expect(text).toContain('tool calls: 4 (0 synthesized span(s) excluded)') + expect(text).toContain(`subagents: 1: ${FIXTURE_AGENT_PATH}`) + expect(text).toContain('human turns: 2') + expect(text).toContain(`token total: ${FIXTURE_TOKEN_TOTAL}`) + expect(text).toContain('pull requests: 0 created, 0 merged') + }) +}) + +describe('session facts as prepared context', () => { + it('stays inside its byte bound and keeps the counts when it must shed', async () => { + const spans = await fixtureSpans(400) + const context = sessionFactsContext(spans)! + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(PREPARED_CONTEXT_BYTE_CEILING) + // Under the documented per-call ceiling the analyst tools work to. + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(VIEW_TRACE_BYTE_CEILING) + const body = JSON.parse(context.slice(context.indexOf('\n') + 1)) as { + sessions: Array<{ tool_calls: { count: number; synthesized_excluded: number } }> + omitted_fields?: string[] + } + expect(body.sessions[0]!.tool_calls.count).toBe(404) + expect(body.sessions[0]!.tool_calls.synthesized_excluded).toBe(0) + }) + + it('holds an arbitrarily tight ceiling, and reports every shed', async () => { + const spans = await fixtureSpans(400) + const report = buildSessionFactsReport(spans) + let previous = Number.POSITIVE_INFINITY + for (const ceiling of [30_000, 8_000, 2_000, 900, 400, 100]) { + const context = renderSessionFactsContext(report, { byteCeiling: ceiling }) + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(ceiling) + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(previous) + previous = Buffer.byteLength(context) + if (context === '') continue + const body = JSON.parse(context.slice(context.indexOf('\n') + 1)) as { + sessions: Array<{ tool_calls?: { count: number } }> + omitted_fields?: string[] + } + // Whatever it shed, it says so, and the tool-call count is the last fact + // to go: it answers the question the bounded tools cannot. + if (ceiling < 8_000) expect(body.omitted_fields?.length).toBeGreaterThan(0) + if (body.sessions.length > 0) expect(body.sessions[0]!.tool_calls?.count).toBe(404) + } + }) + + it('tells the reader the sheet is not citable', async () => { + const spans = await fixtureSpans() + const context = sessionFactsContext(spans)! + expect(context).toContain('cite those spans, never this sheet') + expect(context).toContain('no model call') + }) + + it('supplies the sheet to a built-in kind without dropping its own context', async () => { + const spans = await fixtureSpans() + const base: TraceAnalystDefinition = defineTraceAnalyst({ + id: 'fixture-kind', + description: 'a kind that already prepares context', + area: 'fixture', + version: '2.0.0', + instructions: 'Investigate.', + toolGroup: 'all', + prepareContext: () => 'OWN CONTEXT', + }) + const [wrapped] = withSessionFactsContext([base], spans) + expect(wrapped!.version).toBe(`2.0.0+${SESSION_FACTS_VERSION_SUFFIX}`) + const prepared = await wrapped!.prepareContext!({} as TraceAnalysisStore, { runId: 'r', correlationId: 'r:1' }) + expect(prepared!.startsWith('OWN CONTEXT\n\n')).toBe(true) + expect(prepared).toContain('SESSION FACTS') + expect(Buffer.byteLength(prepared!)).toBeLessThanOrEqual( + PREPARED_CONTEXT_BYTE_CEILING + Buffer.byteLength('OWN CONTEXT\n\n'), + ) + }) + + it('prepares nothing when there are no spans', () => { + expect(sessionFactsContext([])).toBeUndefined() + }) +}) + +describe('analyzeSpans', () => { + it('hands the sheet to the built-in kinds, and can be run without it', async () => { + const spans = await fixtureSpans() + const requests: TraceAnalysisEngineRequest[] = [] + const engine: TraceAnalysisEngine = { + id: 'session-facts-test-engine', + description: 'records the instructions it is given instead of calling a model', + model: 'test-model', + version: '1.0.0', + executionConfig: {}, + async analyze(request) { + requests.push(request) + return { answer: 'noted', findings: [], trajectory: [], modelCalls: 1, toolCalls: 0, runtime: {} } + }, + } + + await analyzeSpans(spans, { engine, agenticKinds: DEFAULT_TRACE_ANALYST_KINDS.slice(0, 1) }) + expect(requests).toHaveLength(1) + const prepared = requests[0]!.instructions.split('PREPARED CONTEXT:\n')[1]! + expect(prepared).toContain('SESSION FACTS') + expect(JSON.parse(prepared.slice(prepared.indexOf('\n') + 1).split('\n\n')[0]!)) + .toMatchObject({ sessions: [{ tool_calls: { count: 4, synthesized_excluded: 0 } }] }) + + requests.length = 0 + await analyzeSpans(spans, { + engine, + agenticKinds: DEFAULT_TRACE_ANALYST_KINDS.slice(0, 1), + sessionFactsContext: false, + }) + expect(requests[0]!.instructions).not.toContain('SESSION FACTS') + }) +}) + +describe('traces facts', () => { + const cli = (args: string[]) => + execFileAsync(process.execPath, ['--import', 'tsx', 'src/cli.ts', ...args], { cwd: process.cwd() }) + + it('prints the sheet as JSON with no model call', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-cli-')) + const otlp = join(dir, 'spans.otlp.jsonl') + await writeFile(otlp, serializeSpans(await fixtureSpans()), 'utf8') + + const { stdout } = await cli(['facts', '--otlp', otlp]) + const report = JSON.parse(stdout) as { + kind: string + sessions: Array<{ toolCalls: { value: number }; subagents: { value: Array<{ taskName: string }> } }> + } + expect(report.kind).toBe('traces.session_facts_report') + expect(report.sessions[0]!.toolCalls.value).toBe(4) + expect(report.sessions[0]!.subagents.value[0]!.taskName).toBe(FIXTURE_AGENT_PATH) + + const readable = await cli(['facts', '--otlp', otlp, '--format', 'text']) + expect(readable.stdout).toContain('tool calls: 4') + expect(readable.stdout).toContain('deterministic, $0') + }) + + it('exits non-zero when a session cannot be read', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-unreadable-')) + // A file that is not a session, and a path that does not exist. Neither may + // be reported as a session with no facts. + const garbage = join(dir, 'not-a-session.jsonl') + await writeFile(garbage, 'this is not a rollout\n', 'utf8') + await expect(cli(['facts', '--harness', 'codex', '--session', garbage])) + .rejects.toMatchObject({ code: 1 }) + await expect(cli(['facts', '--otlp', join(dir, 'absent.otlp.jsonl')])) + .rejects.toMatchObject({ code: 1 }) + }) + + it('keeps the facts and names the unread records when part of a session is corrupt', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-corrupt-')) + const session = join(dir, `${FIXTURE_SESSION_ID}.jsonl`) + const records = fixtureRecords().map((record) => JSON.stringify(record)) + // One unparseable line among readable ones: the facts still hold, and the + // sheet says how many records it could not read. + records.splice(3, 0, '{ not json') + await writeFile(session, `${records.join('\n')}\n`, 'utf8') + + const { stdout } = await cli(['facts', '--harness', 'codex', '--session', session]) + const report = JSON.parse(stdout) as { + sessions: Array<{ toolCalls: { value: number }; unreadRecords: { value: number } }> + } + expect(report.sessions[0]!.toolCalls.value).toBe(4) + expect(report.sessions[0]!.unreadRecords.value).toBe(1) + + const readable = await cli(['facts', '--harness', 'codex', '--session', session, '--format', 'text']) + expect(readable.stdout).toContain('unread records: 1') + }) + + it('rejects an unknown output format before reading anything', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-format-')) + await expect(cli(['facts', '--otlp', join(dir, 'unused.otlp.jsonl'), '--format', 'yaml'])) + .rejects.toMatchObject({ code: 1, stderr: expect.stringContaining('unknown facts format') }) + }) +})