diff --git a/README.md b/README.md index ff5d32c..3fc981d 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ Emitting the contract is the supported way to integrate a new system. The adapte - [Live stream](#live-stream) - [Watch a run tree](#watch-a-run-tree) - [Improvement engine](#improvement-engine) +- [Ask questions](#ask-questions) +- [Session facts](#session-facts) - [Session index](#session-index) - [Session bundle](#session-bundle) · [Two views](#two-views-two-consumers) - [Policy-mining evidence](#policy-mining-evidence) @@ -243,6 +245,8 @@ traces analyze --harness codex --current --latest-turn --workflow # current tur traces analyze --harness claude-code --session --latest-turn # latest task plus its subagents traces investigate --all --last 10 --out report.md # explicit investigation alias traces improve --all --last 10 --dir .traces/improvement +traces ask --harness codex --session --question "Which commands failed?" +traces facts --harness codex --session # the deterministic facts sheet, $0 traces analyze --all --since 2026-06-18 --out report.md traces validate spans.otlp.jsonl # conformance; exit 1 only when it is not a trace traces validate results/sessions --out conformance.md # a whole directory of exports @@ -291,11 +295,15 @@ See [Replay verification](./docs/replay-verify.md) for setup, semantics, and hon | `--cwd ` | Filter by working directory | | `--since ` | `upload`: window, `30m`/`2h`/`7d` or ISO (default 24h); `analyze`: ISO cutoff | | `--out ` | Write the report to a file | -| `--dir ` | `improve`: write the full artifact pack to this directory | +| `--dir ` | `improve`: write the full artifact pack to this directory; `ask`: write `answers.json` + `report.md` there | | `--otlp ` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `analyze`, `investigate`, `improve`, `stream` | | `--otlp-out ` | **WRITE** the OTLP artifact here (also evidence provenance / dry-run upload preview) | | `--format ` | File `analyze`, `export`, or `stream`: `auto`, `policy-evidence`, `sandbox-events`, `openinference`, `intelligence-spans`, or `chat-trajectory` | | `--llm` / `--budget ` | Enable agentic analysts (needs `TANGLE_API_KEY` + Python with `agent-eval-rpc[dspy]`) / cap their spend | +| `--question ` | `ask`: one question, repeatable. Kept short so the engine sees it whole | +| `--questions ` | `ask`: JSON array of questions — strings, or `{ id?, question, instructions?, answerSchema? }` | +| `--question-budget ` | `ask`: provider ceiling for ONE question; `--budget` is the ceiling shared by all of them | +| `--concurrency ` | `ask`: questions running at once (default 4); `import-codetracebench`: trajectories imported at once | | `--config ` | `analyze` / `investigate` / `improve` / `stream`: load BYO analysts, live analysts, and external analyzers | | `--interval ` / `--window ` | `watch` / live `stream`: poll seconds (sessions 5, run tree 2) / active-session window minutes (default 30) | | `--min-loop ` | Identical repeated calls before flagging a loop (default 3) | @@ -443,6 +451,90 @@ The config can export: Traces does not pretend that an action is a measured candidate. Use `agent-eval` to propose and compare candidate changes, `agent-runtime` to package an approved improvement, and `agent-interface` to represent profile edits. +## Ask questions + +`traces ask` answers free-form questions about the selected sessions. +Each question becomes its own recursive investigation over the same trace, and the questions run at the same time under one budget. + +```bash +traces ask --harness codex --session \ + --question "Which shell commands exited non-zero, and what were they?" \ + --question "What was the last thing the human asked for?" \ + --dir .traces/ask +``` + +Ask many questions from a file, and hold an answer to a shape a scorer can compare: + +```bash +traces ask --all --last 3 --questions questions.json --concurrency 6 --budget 2 --dir .traces/ask +``` + +```json +[ + "Which pull requests did this session open?", + { + "id": "merged-prs", + "question": "Which pull requests were merged?", + "instructions": "Count only merges the trace records, not merges the agent said it would do.", + "answerSchema": { "type": "array", "items": { "type": "integer" } } + } +] +``` + +An entry is a question string, or an object with `question` and an optional `id`, `instructions`, and `answerSchema`. +The schema accepts a small JSON Schema subset: `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, and `const`. +Any other keyword is rejected when the run starts, because a constraint that is silently ignored would let a wrong answer pass as checked. + +The command writes two artifacts to `--dir`: + +| File | Contents | +|---|---| +| `answers.json` | per question: the answer text, the parsed answer, every citation with whether it resolves, accepted findings, evidence-gate rejections by reason, model calls, tool calls, cost with provenance, and latency | +| `report.md` | the same run as readable Markdown: a summary line, a per-question table, then each answer with its citation and rejection notes | + +What it checks, and what it costs: + +- **Facts first.** Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context before its first model call, at $0. The sheet is not citable; it names the span ids behind each fact. +- **Citations.** Every `trace:///span/` URI in an answer is looked up in the trace. An answer that cites a span the trace does not hold fails. +- **Budget.** `--budget` is one ceiling shared by every question. `--question-budget` bounds one question. The run refuses to start when the budget cannot cover a single model call, and warns when the budget admits fewer concurrent calls than `--concurrency`. +- **Cost.** Each cost carries its provenance: `observed` from a provider receipt, `estimated` from token counts, or `uncaptured`. An uncaptured cost stays null; it never becomes zero. +- **Exit code.** `ask` writes both artifacts first, then exits 1 when any question failed, returned no answer, broke its schema, or cited a span that does not exist. A failed question never costs the other answers. + +`ask` uses the same engine and credentials as `--llm`, so it needs `TANGLE_API_KEY` and a Python interpreter with `agent-eval-rpc[dspy]`. +Do not pass `--llm`; the command is model-backed by definition. + +## Session facts + +`traces facts` prints the deterministic session-facts sheet: the answers a session audit needs first, computed straight from the spans. +No model call, no engine, no budget — it costs $0 and always returns the same sheet for the same spans. + +```bash +traces facts --harness codex --session # JSON on stdout +traces facts --harness codex --last 5 --format text # the short readable form +traces facts --otlp spans.otlp.jsonl --out facts.json +``` + +| Fact | What it is | +|---|---| +| `toolCalls` | TOOL spans the agent actually invoked. Synthesized subagent lifecycle spans are excluded and counted separately in `synthesizedToolSpans`, so the total is not high by the number of subagents | +| `toolCallsByName` | the same calls by tool name, so a category decision is the reader's, not a guess | +| `subagents` | every `spawn_agent` call with the task name the adapter recorded | +| `humanTurns` | `user.prompt` turns a person typed, in order, with the timestamp; `turnsByActor` shows every turn by actor so the human filter is checkable | +| `finalMessages` | the last message of the session's own agent, and of each subagent task, kept apart | +| `changedFiles` | paths named by patch headers and file-editing tool arguments, with the operation | +| `firstRecordAt` / `lastRecordAt` | the trace's earliest span start and latest span end | +| `unreadRecords` | records the session reader could not parse, from the session's integrity receipt | +| `tokenTotal` | the harness's own cumulative token total, when a span carries `traces.session.total_tokens` | + +Two rules hold for every field: + +- **Every fact names its span ids.** `spanIds` lists the spans the value was computed from, so any number here can be opened and checked. The sheet itself is not a span and cannot be cited. +- **A fact the spans cannot support is `null` with its reason.** It is never guessed, and never a silent zero. `partial` marks a measured value that is known to be incomplete — a truncated patch, or a list above the entry cap. + +`facts` exits non-zero when a selected session cannot be read at all: a session that produced no record spans would otherwise print a sheet of zeros stating, in the sheet's own voice, that the session did nothing. + +The same sheet reaches the model-backed analysts as prepared context, before their first model call — see [Trace analysts](docs/trace-analysts.md#session-facts-as-prepared-context). + ## Session index `traces index` writes one general JSON catalog over the selected sessions. @@ -608,6 +700,8 @@ traces analyze --last 1 --analyzer prime traces analyze --last 1 --analyzer my-installed-command ``` +To ask your own question instead of the built-in kinds, use [`traces ask`](#ask-questions). + HALO returns a diagnosis report. Hodoscope samples distinct behaviors and marks every sample `needs_review`. Prime posts the full span projection to an OpenAI-compatible bridge (`TRACES_PRIME_BRIDGE_URL`, default `http://localhost:4181`) and returns validated findings with span evidence. @@ -630,6 +724,7 @@ See [`examples/external-engines.ts`](./examples/external-engines.ts). > `--llm` also needs a Python interpreter with `agent-eval-rpc[dspy]` installed, because agent-eval's model-backed analysts run through the DSPy RLM engine out of process; set `TRACES_PYTHON` to choose the interpreter. > The bridge protocol is version-locked: install the exact version matching this package's `@tangle-network/agent-eval` dependency (`pip install "agent-eval-rpc[dspy]==$(npm view @tangle-network/traces dependencies.@tangle-network/agent-eval)"`) — a skewed bridge kills every agentic analyst at startup. > When `--llm` was requested and every agentic analyst fails, `analyze`/`investigate`/`improve` still write the deterministic report, then exit 1 with each analyst's underlying error. +> A requested `--analyzer` behaves the same way: its error is written into the report, and `analyze` then exits 1 naming every external analyzer that failed. > Every deterministic command — `list`, `analyze` without `--llm`, `convert`, `index`, `inspect`, `export`, `evidence`, `stream`, `watch`, `analyze --supervisor-run-dir` — needs neither a key nor Python. ## Agent skills @@ -689,6 +784,7 @@ The CLI is a thin consumer of these exports. | `analyzeSpans` | `(spans, { registry?, ai?, budgetUsd? }) → AnalyzeResult` | run built-in analysts, or **your own** via `registry` | | `runTraceInvestigation` | `(TraceInvestigationOptions) → TraceInvestigationResult` | typed findings with actions/checks, execution facts, external analyzer output, and report | | `runTraceImprovement` | `(TraceImprovementOptions) → TraceImprovementResult` | writes the full findings, evidence, report, and trace artifact pack | +| `runTraceQuestions` | `(TraceQuestionsOptions) → TraceQuestionsResult` | ask many free-form questions of one span list, concurrently, under one shared cost ledger; keeps each answer and checks its citations | | `buildTraceFindingPacket` | `({ findings }) → TraceFindingPacket` | render any `AnalystFinding[]` without changing its schema | | `runTraceStoreInvestigation` | `({ traceStore }) → TraceStoreInvestigationResult` | run the same packet layer over a hosted/custom `TraceAnalysisStore` | | `loadTracesConfig` | `(path?) → TracesConfig \| undefined` | load BYO analysts and external analyzers | diff --git a/docs/trace-analysts.md b/docs/trace-analysts.md index 102ff6d..e1a5543 100644 --- a/docs/trace-analysts.md +++ b/docs/trace-analysts.md @@ -8,6 +8,8 @@ Keeping them separate prevents an exploratory model output from being reported a | What happened? | Built-in local checks | Findings from explicit trace facts | | Why might it have happened? | `--llm`, HALO, or a custom analyst | Findings or a diagnosis report with cited spans | | What behavior should we inspect? | Hodoscope | Samples marked `needs_review` | +| What exactly does this session say about X? | `traces ask` | An answer per question, with checked `trace://` citations | +| What does this session state, exactly and for free? | `traces facts` | A deterministic facts sheet, no model call | ## Start here @@ -56,6 +58,188 @@ Returned directories from another resumed Claude session are parsed and included The OpenInference file is the shared input for external engines. The original trace and exact cited span remain available for review. +## Ask free-form questions + +The built-in kinds ask fixed questions. +`traces ask` asks your own. + +```bash +traces ask --harness codex --session \ + --question "Which shell commands exited non-zero?" \ + --question "Which pull requests did the session open, and were they merged?" \ + --dir .traces/ask +``` + +Each question runs through agent-eval's `runTraceAnalyst` directly, not through the analyst registry. +That matters for three reasons. + +- The registry keeps only findings, so it discards the engine's prose answer. `ask` keeps the answer text verbatim. +- The registry runs analysts one at a time. `ask` runs questions concurrently, so wall time falls toward the slowest question instead of the sum. +- One shared `CostLedger` bounds the whole run, so `--budget` means the same thing whatever the number of questions. + +### Questions + +A question stays short on purpose. +The engine shows the model a preview of each long input: it keeps the first 500 and last 500 characters and drops the middle. +A question inside the limit therefore reaches the model whole, and the answer rules sit in the first 500 characters of the instructions, which the preview always keeps. +`ask` rejects an over-long question before it starts a model call and names the limit. +Put the detail in the entry's `instructions` field instead, which the model reads after the rules. + +Read many questions from a file: + +```json +{ + "questions": [ + "What was the last thing the human asked for?", + { + "id": "failed-commands", + "question": "Which shell commands exited non-zero?", + "instructions": "Report the command line and the exit code. Ignore commands the agent only proposed.", + "answerSchema": { + "type": "array", + "items": { + "type": "object", + "properties": { "command": { "type": "string" }, "exit_code": { "type": "integer" } }, + "required": ["command", "exit_code"] + } + } + } + ] +} +``` + +An `answerSchema` makes the answer one JSON value a scorer can compare field by field. +The supported keywords are `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `title`, and `description`. +Any other keyword is rejected when the run starts. +This package carries no JSON Schema library, and a constraint that is quietly ignored would let a wrong answer pass as checked. + +### What the run guarantees + +- Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context, before its first model call. It costs nothing and answers the counting questions the bounded trace tools cannot. +- Every `trace:///span/` URI in an answer is resolved against the store. An unresolvable citation fails that question. +- Findings the answer submits still pass the same evidence gate as the built-in kinds. Refused findings are counted by reason in both artifacts. +- A failed question never stops the others. Its failure is recorded on its own answer, and the remaining answers are written. +- The artifacts are written before the exit code is decided. `ask` exits 1 when any question failed, returned no answer, broke its schema, or cited a missing span. + +### Budget under concurrency + +`--budget` is the ceiling shared by every question; `--question-budget` is the provider ceiling for one question. +The ledger reserves each model call's maximum charge before the call runs, and releases the unused part when the call settles. +Two consequences follow. + +- A budget below one call's reservation refuses the run before any model call, rather than failing every question. +- A budget that admits fewer concurrent reservations than `--concurrency` still runs, and the report carries a warning naming how many concurrent calls it covers. + +A question the ledger refuses is reported as `budget-refused`, and the answers already produced are kept. + +### SDK + +```ts +import { analysisEngineFromEnv, runTraceQuestions, writeTraceQuestionsArtifacts } from '@tangle-network/traces' + +const result = await runTraceQuestions({ + questions: [ + { id: 'failed-commands', question: 'Which shell commands exited non-zero?' }, + { id: 'last-ask', question: 'What was the last thing the human asked for?' }, + ], + spans, + engine: analysisEngineFromEnv({ model: 'gpt-5.6-luna', maxCostUsd: 1 }), + concurrency: 4, + budgetUsd: 2, +}) +await writeTraceQuestionsArtifacts(result, '.traces/ask') +if (!result.ok) process.exitCode = 1 +``` + +`result.questions[i].answer` is the engine's prose, unedited. +`result.totals` carries the wall time, the summed question time, the peak concurrency, and the cost with its provenance. + +## Session facts + +`traces facts` computes a fixed set of session facts straight from the spans. +It runs no model, opens no engine, and spends nothing. +The same spans always produce the same sheet. + +```bash +traces facts --harness codex --session # JSON on stdout +traces facts --harness codex --last 5 --format text # the short readable form +traces facts --otlp spans.otlp.jsonl --out facts.json +``` + +### Why it exists + +The trace tools are bounded, and above their bounds they answer a different question than the one asked. +`viewTrace` returns a `≤20`-entry span-name histogram once a trace exceeds `perCallByteCeiling` (150,000 bytes). +`countTraces` counts traces, not spans. +`viewSpans` needs span ids the reader does not have yet. +`searchTrace` stops at 500 hits. + +A model asked "how many tool calls ran?" therefore adds up a capped histogram and decides by eye which names belong. +Measured over twelve private audit sessions, the model-backed analyst arm scored a deterministic mean of **0.389**. +Extracting the same answers mechanically from the OTLP spans those runs already wrote scored **0.858** — the facts were present and exact the whole time. +The sheet is that extraction, made part of the tool. + +### What it states + +| Field | Value | +| --- | --- | +| `toolCalls` | TOOL spans the agent invoked. Synthesized subagent lifecycle spans are excluded and counted in `synthesizedToolSpans` | +| `toolCallsByName` | the same calls by tool name | +| `subagents` | every `spawn_agent` call with the task name from `traces.codex.spawn_agent_path` | +| `humanTurns` | `user.prompt` turns with `tangle.actor` `human`, in order, with timestamps | +| `turnsByActor` | every `user.prompt` turn by actor, so the human filter is checkable | +| `finalMessages` | the last message of the session's own agent, and of each subagent task, separately | +| `changedFiles` | paths from `*** Add/Update/Delete/Move to File:` patch headers and from file-editing tool arguments | +| `firstRecordAt`, `lastRecordAt` | the trace's earliest span start and latest span end | +| `unreadRecords` | records the session reader could not parse | +| `tokenTotal` | the harness's cumulative total, when a span carries `traces.session.total_tokens` | + +Two rules hold for every field. + +- **Every fact names the span ids it came from.** A reader can open those spans and check the number. The sheet is not a span and cannot be cited. +- **A fact the spans cannot support is `null` with a stated reason.** It is never guessed and never a silent zero. `partial` marks a measured value known to be incomplete. + +`facts` exits non-zero when a selected session produced no record spans, rather than printing a sheet of zeros for a session it could not read. + +### Session facts as prepared context + +The same sheet is supplied to the model-backed analysts before their first model call, through `TraceAnalystDefinition.prepareContext`. +It reaches the built-in kinds run by `analyze --llm`, `investigate`, and `improve`, and it reaches every `traces ask` question inside `PREPARED CONTEXT:`. + +- The sheet is bounded at `PREPARED_CONTEXT_BYTE_CEILING` (30,000 bytes), a fifth of the `perCallByteCeiling` of 150,000 the trace tools work to, so the rest of the budget stays available for the tool calls the model still makes. +- When the sheet does not fit, fields are shed from the largest downward and the shed is listed in `omitted_fields`. The tool-call counts are the last facts to go. +- Receiving the sheet changes an analyst's behavior, so its version carries `+session-facts.1`. `createTraceAnalyst` records `prepare_context` in the exact-run identity, and a changed prepared context must not hide behind an unchanged version. +- The sheet is not evidence. Citations still resolve against the raw spans, which is why every fact names its span ids rather than asking the model to trust the sheet. + +Pass `sessionFactsContext: false` to `analyzeSpans` to run an analyst without it. + +```ts +import { buildSessionFactsReport, computeSessionFacts, renderSessionFacts } from '@tangle-network/traces' + +const [facts] = computeSessionFacts(spans) +console.log(facts.toolCalls.value, facts.toolCalls.spanIds) +console.log(renderSessionFacts(buildSessionFactsReport(spans))) +``` + +## Evidence-gate rejections + +A model-backed analyst can submit a finding the evidence gate then refuses. +Without the reason, a report showing "0 findings" reads as "the model found nothing" when the truth may be "the gate refused everything it found". + +`analyze`, `investigate`, `improve`, and `ask` now carry those refusals: + +- the CLI log prints the reason and the offending URI on each `finding rejected` line; +- the analyst table's Detail cell names the reasons and their counts; +- `result.findingRejections` (investigation and improvement) and `answers.json` (`ask`) hold the counts per analyst and reason. + +The common reasons are an excerpt the cited span does not contain, a span the trace does not hold, and too few distinct citations for the kind. + +## External analyzer failures exit non-zero + +`--analyzer halo|hodoscope|prime|` promises that engine's output. +An analyzer that fails now writes its error into the report as before, and then `analyze` exits 1 naming every analyzer that failed. +Scripts that treated exit 0 as "the analyzer ran" were reading a report that said otherwise. + ## Codex tool outcomes Both function and custom tool outputs use the same status parser. diff --git a/src/analysis-store.ts b/src/analysis-store.ts new file mode 100644 index 0000000..7d70521 --- /dev/null +++ b/src/analysis-store.ts @@ -0,0 +1,74 @@ +/** + * The trace file and stores every analysis path reads. + * + * `analyze`, `investigate`, `improve`, and `ask` all write the selected spans + * to one OpenInference file and read it back through agent-eval's + * `OtlpFileTraceStore`. Keeping that sequence here means a source-bundle grant, + * the file ceiling, and the containment check cannot differ between them. + */ + +import { OtlpFileTraceStore, type SpanSourceReader } from '@tangle-network/agent-eval/traces' +import { assertOutsideSourceBundle, createBundleSourceReader } from './bundle-source.js' +import type { OtlpSpan } from './otlp.js' +import { writeOtlpFile } from './otlp.js' + +/** + * `viewTrace` and generated-file ceiling. The default 150KB cap exists to + * protect an LLM's context window, and a single coding session is one trace + * whose full span list routinely exceeds 150KB. The fixed ceiling covers large + * sessions without disabling agent-eval's file-size guard. + */ +export const GENERATED_TRACE_FILE_CEILING = 512 * 1024 * 1024 + +export interface AnalysisTraceFile { + readonly otlpPath: string + /** Present only when the caller explicitly granted source reads through a bundle. */ + readonly sourceReader?: SpanSourceReader +} + +/** Write the spans once and bind any explicitly granted source reader. */ +export async function writeAnalysisTraceFile( + spans: readonly OtlpSpan[], + opts: { + sourceBundle?: { path: string; maxRecordBytes?: number } + otlpOutPath?: string + signal?: AbortSignal + } = {}, +): Promise { + if (opts.sourceBundle && opts.otlpOutPath) await assertOutsideSourceBundle(opts.sourceBundle.path, opts.otlpOutPath) + const sourceReader = opts.sourceBundle + ? await createBundleSourceReader(opts.sourceBundle.path, spans, { + signal: opts.signal, + maxRecordBytes: opts.sourceBundle.maxRecordBytes, + }) + : undefined + const otlpPath = await writeOtlpFile(spans, opts.otlpOutPath) + return sourceReader ? { otlpPath, sourceReader } : { otlpPath } +} + +/** + * Store for model-driven analysis. It keeps agent-eval's default per-call byte + * ceiling, so each tool result stays bounded for a model's context; the + * engine drills in with `viewSpans` and `searchTrace` from a summary. + */ +export async function openAgenticTraceStore(file: AnalysisTraceFile): Promise { + const store = new OtlpFileTraceStore({ + path: file.otlpPath, + maxFileBytes: GENERATED_TRACE_FILE_CEILING, + ...(file.sourceReader ? { sourceReader: file.sourceReader } : {}), + }) + await store.ensureIndexed() + return store +} + +/** Store for deterministic analysts, which have no context window to protect. */ +export async function openDeterministicTraceStore(file: AnalysisTraceFile): Promise { + const store = new OtlpFileTraceStore({ + path: file.otlpPath, + maxFileBytes: GENERATED_TRACE_FILE_CEILING, + perCallByteCeiling: GENERATED_TRACE_FILE_CEILING, + ...(file.sourceReader ? { sourceReader: file.sourceReader } : {}), + }) + await store.ensureIndexed() + return store +} diff --git a/src/analyst-model-call.ts b/src/analyst-model-call.ts index a7c7b73..c8e943f 100644 --- a/src/analyst-model-call.ts +++ b/src/analyst-model-call.ts @@ -4,6 +4,20 @@ import { canonicalCandidateDigest, } from '@tangle-network/agent-interface' import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' +import { createDspyRlmTraceEngine, type TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' + +/** Model the CLI's recursive analysts use when neither `--model` nor TRACES_ANALYST_MODEL names one. */ +export const DEFAULT_ANALYST_MODEL = 'gpt-5.6-luna' + +/** + * Provider ceiling for one `ask` question when neither `--question-budget` nor + * a smaller `--budget` names one. It equals the DSPy engine's own default + * investigation ceiling, stated here so `ask` owns the value it applies. + */ +export const DEFAULT_QUESTION_MAX_COST_USD = 1 + +/** Default analysis endpoint: the Tangle router, reached with TANGLE_API_KEY. */ +export const TANGLE_ROUTER_BASE_URL = 'https://router.tangle.tools/v1' export const ANALYST_MAX_OUTPUT_TOKENS = 16_384 export const GPT_5_6_ANALYST_MAX_OUTPUT_TOKENS = 8_192 @@ -55,3 +69,82 @@ export function createAnalystModelOwner(opts: { profile, } } + +export interface AnalysisEngineFromEnvOptions { + model: string + /** + * Provider-side spend ceiling for ONE investigation (one analyst or one + * question). Omitted, the engine keeps its own default. A caller that runs + * several investigations under one shared ledger passes the per-investigation + * cap here and the total to the ledger. + */ + maxCostUsd?: number + /** Receives one line per model call, in the CLI's analyst log format. */ + log?: (msg: string, fields?: Record) => void + /** Environment to read credentials and the Python interpreter from. Default: process.env. */ + env?: NodeJS.ProcessEnv +} + +/** + * The recursive analysis engine behind `--llm` and `ask`. agent-eval's + * model-backed analysts run through DSPy RLM, which drives + * `agent-eval-rpc[dspy]` out of process, so this needs a Python interpreter + * with that extra installed, selectable via TRACES_PYTHON. Every deterministic + * command is unaffected and still needs neither a key nor Python. + */ +export function analysisEngineFromEnv(opts: AnalysisEngineFromEnvOptions): TraceAnalysisEngine { + const env = opts.env ?? process.env + // The router is the default endpoint, so TANGLE_API_KEY alone is enough. + // OPENAI_API_KEY still works and, when it is the only key present, points at + // OpenAI directly; otherwise a plain OpenAI key would be sent to the router. + const tangleKey = env.TANGLE_API_KEY + const openAiKey = env.OPENAI_API_KEY + const apiKey = tangleKey || openAiKey + if (!apiKey) { + throw new Error( + 'model-backed analysis needs a model key: TANGLE_API_KEY for the Tangle router (the default endpoint), or ' + + 'OPENAI_API_KEY for OpenAI. Set OPENAI_BASE_URL to target any other OpenAI-compatible ' + + 'gateway. Deterministic analysis needs no key.', + ) + } + const baseUrl = + env.OPENAI_BASE_URL || + (tangleKey ? TANGLE_ROUTER_BASE_URL : 'https://api.openai.com/v1') + const python = env.TRACES_PYTHON + const owner = createAnalystModelOwner({ + apiKey, + baseUrl, + model: opts.model, + provider: + baseUrl === TANGLE_ROUTER_BASE_URL + ? 'tangle-router' + : baseUrl.startsWith('https://api.openai.com/') + ? 'openai' + : 'openai-compatible', + }) + const log = opts.log + return createDspyRlmTraceEngine({ + call: owner.call, + callRef: owner.callRef, + recordExecution: (observation) => { + log?.( + `[analyst] model call ${observation.sequence} ${observation.succeeded ? 'ok' : 'FAIL'} ${observation.model}`, + observation.succeeded ? undefined : { error: observation.error }, + ) + }, + model: opts.model, + // Model-aware, not defaulted: GPT-5.6 needs less output room than models + // such as GLM, and every recursive call reserves this full amount before + // execution. An oversized reservation can reject useful later calls even + // when the run's measured spend remains well below its limit. + maxOutputTokens: analystMaxOutputTokens(opts.model), + // maxCostUsd defaults to $1 per investigation, a proxy-side ceiling + // separate from any shared ledger. With the larger token cap the per-call + // reservation grows ~4x, so that default can bind before the caller's own + // allocation and kill investigations mid-run. + ...(opts.maxCostUsd !== undefined && Number.isFinite(opts.maxCostUsd) && opts.maxCostUsd > 0 + ? { maxCostUsd: opts.maxCostUsd } + : {}), + ...(python ? { runner: { command: python } } : {}), + }) +} diff --git a/src/analyze.ts b/src/analyze.ts index 41ee982..3a64b65 100644 --- a/src/analyze.ts +++ b/src/analyze.ts @@ -18,14 +18,14 @@ import { type AnalystRegistry, type AnalystRunSummary, buildDefaultAnalystRegistry, + DEFAULT_TRACE_ANALYST_KINDS, type TraceAnalysisEngine, type TraceAnalystDefinition, } from '@tangle-network/agent-eval/analyst' -import { OtlpFileTraceStore } from '@tangle-network/agent-eval/traces' +import { openAgenticTraceStore, openDeterministicTraceStore, writeAnalysisTraceFile } from './analysis-store.js' import { summarizeSpanExecution } from './execution.js' import type { OtlpSpan } from './otlp.js' -import { writeOtlpFile } from './otlp.js' -import { assertOutsideSourceBundle, createBundleSourceReader } from './bundle-source.js' +import { withSessionFactsContext } from './session-facts.js' export interface AnalyzeOptions { /** Explicitly authorize original source reads from this full session bundle. */ @@ -51,6 +51,16 @@ export interface AnalyzeOptions { agenticRegistry?: AnalystRegistry /** Select a subset of agent-eval's maintained trace analyst kinds. */ agenticKinds?: readonly TraceAnalystDefinition[] + /** + * Supply the deterministic session-facts sheet to the agentic kinds as + * prepared context (default true). It costs nothing and removes the guessing + * the bounded trace tools force on a model that needs an exact count. Set + * false to measure an analyst without it. + * + * It applies only to definitions this call builds a registry from; a caller + * who brings `agenticRegistry` owns its own prepared context. + */ + sessionFactsContext?: boolean /** Compact deterministic findings that agents receive before reading spans. */ agenticPriorFindings?: readonly AnalystFinding[] /** Where to write the OTLP-JSONL artifact. Defaults to a temp file. */ @@ -74,16 +84,6 @@ export interface AnalyzeResult { agenticPerAnalyst?: readonly AnalystRunSummary[] } -/** - * `viewTrace` and generated-file ceiling for the deterministic pass. The - * default 150KB cap exists to protect an LLM's context window — the - * deterministic behavioral analyst has none, and a single coding session is one trace whose full - * span list routinely exceeds 150KB (→ oversized summary → zero spans → - * zero findings). The fixed ceiling covers large sessions without disabling - * agent-eval's file-size guard. - */ -const GENERATED_TRACE_FILE_CEILING = 512 * 1024 * 1024 - function mergeCostProvenance( first: RunCostProvenance | undefined, second: RunCostProvenance | undefined, @@ -100,11 +100,12 @@ function mergeCostProvenance( export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOptions = {}): Promise { if (spans.length === 0) throw new Error('analyzeSpans: no spans to analyze') opts.signal?.throwIfAborted() - if (opts.sourceBundle && opts.otlpOutPath) await assertOutsideSourceBundle(opts.sourceBundle.path, opts.otlpOutPath) - const sourceReader = opts.sourceBundle - ? await createBundleSourceReader(opts.sourceBundle.path, spans, { signal: opts.signal, maxRecordBytes: opts.sourceBundle.maxRecordBytes }) - : undefined - const otlpPath = await writeOtlpFile(spans, opts.otlpOutPath) + const traceFile = await writeAnalysisTraceFile(spans, { + sourceBundle: opts.sourceBundle, + otlpOutPath: opts.otlpOutPath, + signal: opts.signal, + }) + const { otlpPath } = traceFile opts.signal?.throwIfAborted() const runId = opts.runId ?? `traces-${Date.now()}` const execution = summarizeSpanExecution(spans, { @@ -114,13 +115,7 @@ export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOpti // Deterministic pass — high ceiling so the behavioral analyst sees the whole // trace. No LLM context to protect here. A caller-supplied registry (custom // analysts / their own agents) runs here instead of the built-in suite. - const detStore = new OtlpFileTraceStore({ - path: otlpPath, - maxFileBytes: GENERATED_TRACE_FILE_CEILING, - perCallByteCeiling: GENERATED_TRACE_FILE_CEILING, - ...(sourceReader ? { sourceReader } : {}), - }) - await detStore.ensureIndexed() + const detStore = await openDeterministicTraceStore(traceFile) opts.signal?.throwIfAborted() const detRegistry = opts.registry ?? buildDefaultAnalystRegistry({ registry: { log: opts.log } }) const result = await detRegistry.run(runId, { traceStore: detStore }, { signal: opts.signal }) @@ -130,14 +125,22 @@ export async function analyzeSpans(spans: readonly OtlpSpan[], opts: AnalyzeOpti // the RLM kinds drill via viewSpans/searchTrace from a summary. let agenticPerAnalyst: readonly AnalystRunSummary[] | undefined if (opts.engine || opts.agenticRegistry) { - const agStore = new OtlpFileTraceStore({ path: otlpPath, maxFileBytes: GENERATED_TRACE_FILE_CEILING, ...(sourceReader ? { sourceReader } : {}) }) - await agStore.ensureIndexed() - const agRegistry = opts.agenticRegistry ?? buildDefaultAnalystRegistry({ - engine: opts.engine!, - ...(opts.agenticKinds ? { definitions: opts.agenticKinds } : {}), - includeBehavioral: false, - registry: { log: opts.log }, - }) + const agStore = await openAgenticTraceStore(traceFile) + // The sheet reaches the model through each definition's `prepareContext`, + // which runs before the first model call. Its facts are exact where the + // bounded trace tools force a guess, and it costs nothing to compute. A + // caller-supplied agentic registry owns its own prepared context, so the + // sheet is built only when this call builds the registry. + const buildRegistry = (): AnalystRegistry => { + const kinds = opts.agenticKinds ?? DEFAULT_TRACE_ANALYST_KINDS + return buildDefaultAnalystRegistry({ + engine: opts.engine!, + definitions: opts.sessionFactsContext === false ? kinds : withSessionFactsContext(kinds, spans), + includeBehavioral: false, + registry: { log: opts.log }, + }) + } + const agRegistry = opts.agenticRegistry ?? buildRegistry() const agResult = await agRegistry.run(runId, { traceStore: agStore }, { budget: opts.budgetUsd != null ? { totalUsd: opts.budgetUsd } : undefined, chainFindings: true, diff --git a/src/answer-schema.ts b/src/answer-schema.ts new file mode 100644 index 0000000..869e727 --- /dev/null +++ b/src/answer-schema.ts @@ -0,0 +1,154 @@ +/** + * Answer schemas for `traces ask`. + * + * A question may fix the shape of its answer with a JSON Schema, so a scorer + * can compare fields exactly instead of reading prose. This package carries no + * JSON Schema library, so it checks a small subset and REJECTS any schema that + * uses a keyword outside it: a schema whose constraint is silently ignored + * would let a wrong answer pass as checked. + * + * Supported keywords: `type`, `properties`, `required`, `additionalProperties` + * (boolean), `items` (one schema), `enum`, `const`, and the annotations + * `title` and `description`. + */ + +export type AnswerSchema = Readonly> + +const SUPPORTED_KEYWORDS = new Set([ + 'type', + 'properties', + 'required', + 'additionalProperties', + 'items', + 'enum', + 'const', + 'title', + 'description', +]) + +const JSON_TYPES = new Set(['string', 'number', 'integer', 'boolean', 'object', 'array', 'null']) + +const MAX_SCHEMA_DEPTH = 32 + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +/** Throw a TypeError naming the first unsupported or malformed part of `schema`. */ +export function assertAnswerSchema(schema: unknown, path = 'answerSchema', depth = 0): asserts schema is AnswerSchema { + if (depth > MAX_SCHEMA_DEPTH) throw new TypeError(`${path}: schema nesting exceeds ${MAX_SCHEMA_DEPTH} levels`) + if (!isRecord(schema)) throw new TypeError(`${path} must be a JSON Schema object`) + for (const key of Object.keys(schema)) { + if (!SUPPORTED_KEYWORDS.has(key)) { + throw new TypeError( + `${path}: unsupported JSON Schema keyword "${key}"; supported: ${[...SUPPORTED_KEYWORDS].join(', ')}`, + ) + } + } + if (schema.type !== undefined) { + const types = Array.isArray(schema.type) ? schema.type : [schema.type] + if (types.length === 0 || !types.every((type) => typeof type === 'string' && JSON_TYPES.has(type))) { + throw new TypeError(`${path}.type must be one of ${[...JSON_TYPES].join(', ')}, or an array of them`) + } + } + if (schema.properties !== undefined) { + if (!isRecord(schema.properties)) throw new TypeError(`${path}.properties must be an object`) + for (const [name, child] of Object.entries(schema.properties)) { + assertAnswerSchema(child, `${path}.properties.${name}`, depth + 1) + } + } + if (schema.required !== undefined) { + if (!Array.isArray(schema.required) || !schema.required.every((name) => typeof name === 'string')) { + throw new TypeError(`${path}.required must be an array of property names`) + } + } + if (schema.additionalProperties !== undefined && typeof schema.additionalProperties !== 'boolean') { + throw new TypeError(`${path}.additionalProperties must be a boolean`) + } + if (schema.items !== undefined) assertAnswerSchema(schema.items, `${path}.items`, depth + 1) + if (schema.enum !== undefined && (!Array.isArray(schema.enum) || schema.enum.length === 0)) { + throw new TypeError(`${path}.enum must be a non-empty array`) + } + for (const key of ['title', 'description'] as const) { + if (schema[key] !== undefined && typeof schema[key] !== 'string') { + throw new TypeError(`${path}.${key} must be a string`) + } + } +} + +function jsonType(value: unknown): string { + if (value === null) return 'null' + if (Array.isArray(value)) return 'array' + if (typeof value === 'number') return Number.isInteger(value) ? 'integer' : 'number' + return typeof value +} + +function typeMatches(value: unknown, type: string): boolean { + const actual = jsonType(value) + return actual === type || (type === 'number' && actual === 'integer') +} + +function sameJson(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right) +} + +/** + * Every way `value` breaks `schema`, as `$.path: problem` strings. Empty means + * the value conforms. The schema must already have passed `assertAnswerSchema`. + */ +export function answerSchemaErrors(value: unknown, schema: AnswerSchema, path = '$'): string[] { + const errors: string[] = [] + if (schema.type !== undefined) { + const types = (Array.isArray(schema.type) ? schema.type : [schema.type]) as string[] + if (!types.some((type) => typeMatches(value, type))) { + errors.push(`${path}: expected ${types.join(' or ')}, got ${jsonType(value)}`) + return errors + } + } + if (schema.const !== undefined && !sameJson(value, schema.const)) { + errors.push(`${path}: expected ${JSON.stringify(schema.const)}`) + } + if (Array.isArray(schema.enum) && !schema.enum.some((option) => sameJson(value, option))) { + errors.push(`${path}: expected one of ${schema.enum.map((option) => JSON.stringify(option)).join(', ')}`) + } + if (isRecord(value)) { + const properties = isRecord(schema.properties) ? schema.properties : {} + for (const name of (schema.required as string[] | undefined) ?? []) { + if (!(name in value)) errors.push(`${path}.${name}: required property is missing`) + } + for (const [name, child] of Object.entries(value)) { + const childSchema = properties[name] + if (childSchema !== undefined) { + errors.push(...answerSchemaErrors(child, childSchema as AnswerSchema, `${path}.${name}`)) + } else if (schema.additionalProperties === false) { + errors.push(`${path}.${name}: property is not allowed`) + } + } + } + if (Array.isArray(value) && schema.items !== undefined) { + value.forEach((entry, index) => { + errors.push(...answerSchemaErrors(entry, schema.items as AnswerSchema, `${path}[${index}]`)) + }) + } + return errors +} + +/** + * Read one JSON value from a model's answer text. The whole text is tried + * first; a single fenced block is accepted because models wrap JSON in one + * even when told not to. Anything else is a parse failure, not a guess. + */ +export function parseJsonAnswer(text: string): { ok: true; value: unknown } | { ok: false; error: string } { + const trimmed = text.trim() + const candidates = [trimmed] + const fenced = /^```(?:json)?\s*\n([\s\S]*?)\n```$/.exec(trimmed) + if (fenced) candidates.push(fenced[1]!.trim()) + for (const candidate of candidates) { + try { + return { ok: true, value: JSON.parse(candidate) as unknown } + } catch { + // Try the next candidate. + } + } + return { ok: false, error: 'answer is not one JSON value' } +} diff --git a/src/ask.ts b/src/ask.ts new file mode 100644 index 0000000..0f85d04 --- /dev/null +++ b/src/ask.ts @@ -0,0 +1,762 @@ +/** + * `traces ask`: free-form questions over trace sessions, answered by a + * recursive trace-analysis engine. + * + * Each question runs through agent-eval's `runTraceAnalyst` directly instead + * of the analyst registry. The registry keeps only findings, so it discards + * the engine's prose answer, and it runs analysts one at a time. Here the + * questions run concurrently under one shared cost ledger, so a single budget + * bounds all of them, and every `trace://` citation in an answer is checked + * against the store before the answer counts as answered. + */ + +import { mkdir, readFile, writeFile } from 'node:fs/promises' +import { join, resolve } from 'node:path' +import { performance } from 'node:perf_hooks' +import { CostCeilingReachedError, CostLedger, type CostProvenance } from '@tangle-network/agent-eval' +import { + type AnalystUsageReceipt, + defineTraceAnalyst, + type RawAnalystFinding, + runTraceAnalyst, + type TraceAnalysisEngine, + type TraceAnalysisEngineResult, + type TraceAnalystDefinition, + type TraceAnalystLimits, +} from '@tangle-network/agent-eval/analyst' +import type { TraceAnalysisStore } from '@tangle-network/agent-eval/traces' +import { openAgenticTraceStore, writeAnalysisTraceFile } from './analysis-store.js' +import { type AnswerSchema, answerSchemaErrors, assertAnswerSchema, parseJsonAnswer } from './answer-schema.js' +import { indexSessionIdsByTrace } from './attributes.js' +import { + createFindingRejectionTally, + type FindingRejectionReasons, + formatFindingRejections, + totalFindingRejections, +} from './finding-rejections.js' +import type { OtlpSpan } from './otlp.js' +import { sessionFactsContext } from './session-facts.js' + +/** One question to ask of the selected traces. */ +export interface TraceQuestion { + /** Stable ID for the report, the JSON result, and cost attribution. Default: `q`. */ + readonly id?: string + readonly question: string + /** Extra guidance for this question only, placed after the shared rules. */ + readonly instructions?: string + /** + * JSON Schema the answer must satisfy (see `answer-schema.ts` for the + * supported subset). With a schema the answer must be one JSON value. + */ + readonly answerSchema?: AnswerSchema +} + +/** Why a question did not produce a checked answer. */ +export type TraceQuestionFailureKind = + | 'error' + | 'aborted' + | 'budget-refused' + | 'no-answer' + | 'invalid-answer' + | 'unresolved-citations' + +export interface TraceQuestionCitation { + readonly uri: string + readonly traceId: string | null + readonly spanId: string | null + /** True only when the store holds the cited span. */ + readonly resolved: boolean +} + +export interface TraceQuestionAnswer { + readonly id: string + readonly question: string + readonly status: 'answered' | 'failed' + readonly failure?: { readonly kind: TraceQuestionFailureKind; readonly message: string } + /** The engine's prose answer, verbatim. Null when the engine returned none. */ + readonly answer: string | null + /** The answer parsed as JSON, present when the question had an answer schema and it parsed. */ + readonly parsedAnswer?: unknown + /** Every `trace://` URI in the answer text, with whether it resolved. */ + readonly citations: readonly TraceQuestionCitation[] + /** Findings the evidence gate accepted. */ + readonly findings: readonly RawAnalystFinding[] + /** Findings the evidence gate refused, by reason. */ + readonly rejectedFindings: FindingRejectionReasons + readonly model: string | null + /** Successful model completions the engine reported; null when the engine failed. */ + readonly modelCalls: number | null + /** Trace-tool requests the engine reported; null when the engine failed. */ + readonly toolCalls: number | null + /** Provider usage and cost for this question alone, from the shared ledger. */ + readonly usage: AnalystUsageReceipt | null + readonly startedAt: string + readonly endedAt: string + readonly latencyMs: number + /** Engine-native steps, retained for audit. */ + readonly trajectory?: readonly unknown[] +} + +/** One trace the questions could read. */ +export interface TraceQuestionTrace { + readonly traceId: string + readonly sessionId: string | null + readonly spanCount: number + readonly startTime: string | null + readonly endTime: string | null + readonly rootSpan: string | null +} + +export interface TraceQuestionsTotals { + readonly questions: number + readonly answered: number + readonly failed: number + /** Null when at least one question's count is unknown. */ + readonly modelCalls: number | null + /** Null when at least one question's count is unknown. */ + readonly toolCalls: number | null + /** Paid provider calls the shared ledger recorded, including failed ones. */ + readonly providerCalls: number + /** Total spend with its provenance; `uncaptured` carries a null amount, never 0. */ + readonly cost: CostProvenance + /** Wall time of the whole run. */ + readonly wallTimeMs: number + /** Sum of the questions' own latencies; above `wallTimeMs` when questions overlapped. */ + readonly questionTimeMs: number + /** Most questions observed running at the same time. */ + readonly peakConcurrency: number +} + +export interface TraceQuestionsResult { + readonly schemaVersion: 1 + readonly kind: 'traces.ask' + readonly generatedAt: string + readonly harness: string + readonly engine: { readonly id: string; readonly version: string; readonly model: string | null } + readonly concurrency: number + /** Shared ceiling across every question; null when uncapped. */ + readonly budgetUsd: number | null + /** The engine's own ceiling for one question, when it declares one. */ + readonly questionBudgetUsd: number | null + readonly spanCount: number + readonly otlpPath: string + readonly traces: readonly TraceQuestionTrace[] + readonly questions: readonly TraceQuestionAnswer[] + readonly totals: TraceQuestionsTotals + readonly warnings: readonly string[] + /** True when every question produced an answer whose citations all resolved. */ + readonly ok: boolean + readonly report: string +} + +export interface TraceQuestionsOptions { + readonly questions: readonly TraceQuestion[] + readonly spans: readonly OtlpSpan[] + readonly engine: TraceAnalysisEngine + readonly harness?: string + /** Questions running at once. Default 4. */ + readonly concurrency?: number + /** Shared USD ceiling across every question. Omit for no shared ceiling. */ + readonly budgetUsd?: number + readonly limits?: Partial + /** Explicit full-bundle source access for `readSpanSource`. */ + readonly sourceBundle?: { path: string; maxRecordBytes?: number } + /** Where to write the OpenInference file the engine reads. Default: a temp file. */ + readonly otlpOutPath?: string + readonly generatedAt?: string + readonly signal?: AbortSignal + readonly log?: (msg: string, fields?: Record) => void +} + +export interface TraceQuestionsArtifacts { + readonly directory: string + readonly result: string + readonly report: string + readonly traces: string +} + +export const DEFAULT_ASK_CONCURRENCY = 4 + +/** Bumped whenever the question layout or rules change. */ +const ASK_DEFINITION_VERSION = '1.0.0' + +/** + * DSPy's RLM shows the model a preview of each input: a value longer than + * 1,000 characters appears as its first and last 500 only. The question + * therefore stays short and whole, and the rules sit in the first 500 + * characters of the instructions, which is all of them the preview keeps. + */ +const DSPY_PREVIEW_HEAD_CHARS = 500 +const QUESTION_FIELD_MAX_CHARS = 900 + +const ASK_QUESTION_HEADER = + 'Answer QUESTION with the trace tools and follow TRACES ASK RULES in analyst_instructions. ' + + 'The trace list is JSON after PREPARED CONTEXT: there. Cite trace:///span/.' + +const QUESTION_LABEL = '\n\nQUESTION: ' + +/** Longest question that keeps the whole question field inside the preview. */ +export const MAX_TRACE_QUESTION_CHARS = QUESTION_FIELD_MAX_CHARS - ASK_QUESTION_HEADER.length - QUESTION_LABEL.length + +const ASK_RULES = [ + 'TRACES ASK RULES', + '1. Answer QUESTION only from trace tool results retrieved in this run.', + '2. PREPARED CONTEXT ends analyst_instructions: the trace list JSON, then a SESSION FACTS sheet. Parse both; copy IDs from them.', + '3. Cite each fact as trace:///span/: span ids the sheet names, never the sheet.', + '4. Quote excerpts from viewSpans output, never searchTrace hits.', + '5. If the trace does not record a fact, say "not in trace".', +].join('\n') + +const ANSWER_SCHEMA_RULE = '6. The answer is one JSON value matching ANSWER SCHEMA below, with no other text.' + +/** Traces listed in the prepared context; the rest are counted, not dropped silently. */ +const MAX_CONTEXT_TRACES = 200 + +const QUESTION_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/ + +const QUESTION_KEYS = new Set(['id', 'question', 'instructions', 'answerSchema']) + +/** + * Validate questions and assign default IDs. Throws on an empty list, a + * duplicate or malformed ID, a question too long for the preview, or an answer + * schema outside the supported subset. + */ +export function normalizeTraceQuestions(questions: readonly TraceQuestion[]): Array { + if (questions.length === 0) throw new Error('ask needs at least one question') + const seen = new Set() + return questions.map((entry, index) => { + const id = entry.id ?? `q${index + 1}` + if (!QUESTION_ID.test(id)) { + throw new Error(`question ID "${id}" must match ${QUESTION_ID} (letters, digits, dot, underscore, hyphen)`) + } + if (seen.has(id)) throw new Error(`duplicate question ID "${id}"`) + seen.add(id) + const question = typeof entry.question === 'string' ? entry.question.trim() : '' + if (!question) throw new Error(`question ${id} is empty`) + if (question.length > MAX_TRACE_QUESTION_CHARS) { + throw new Error( + `question ${id} has ${question.length} characters; the limit is ${MAX_TRACE_QUESTION_CHARS} so the ` + + 'model sees it whole. Move detail into the entry\'s "instructions" field.', + ) + } + if (entry.instructions !== undefined && typeof entry.instructions !== 'string') { + throw new Error(`question ${id}: instructions must be a string`) + } + if (entry.answerSchema !== undefined) assertAnswerSchema(entry.answerSchema, `question ${id} answerSchema`) + return { + id, + question, + ...(entry.instructions?.trim() ? { instructions: entry.instructions.trim() } : {}), + ...(entry.answerSchema !== undefined ? { answerSchema: entry.answerSchema } : {}), + } + }) +} + +/** + * Read questions from a JSON file: an array, or an object with a `questions` + * array. Each entry is a question string or an object with `question` and + * optional `id`, `instructions`, and `answerSchema`. Unknown keys are + * rejected so a misspelled field cannot be ignored silently. + */ +export async function loadTraceQuestionsFile(path: string): Promise { + let parsed: unknown + try { + parsed = JSON.parse(await readFile(path, 'utf8')) + } catch (error) { + throw new Error(`questions file ${path} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`) + } + const list = Array.isArray(parsed) + ? parsed + : typeof parsed === 'object' && parsed !== null && Array.isArray((parsed as { questions?: unknown }).questions) + ? (parsed as { questions: unknown[] }).questions + : undefined + if (!list) throw new Error(`questions file ${path} must hold a JSON array or an object with a "questions" array`) + return list.map((entry, index) => { + if (typeof entry === 'string') return { question: entry } + if (typeof entry !== 'object' || entry === null || Array.isArray(entry)) { + throw new Error(`questions file ${path}: entry ${index + 1} must be a string or an object`) + } + for (const key of Object.keys(entry)) { + if (!QUESTION_KEYS.has(key)) { + throw new Error(`questions file ${path}: entry ${index + 1} has unknown key "${key}"`) + } + } + const record = entry as Record + if (typeof record.question !== 'string') { + throw new Error(`questions file ${path}: entry ${index + 1} needs a "question" string`) + } + if (record.id !== undefined && typeof record.id !== 'string') { + throw new Error(`questions file ${path}: entry ${index + 1} "id" must be a string`) + } + return { + question: record.question, + ...(record.id !== undefined ? { id: record.id as string } : {}), + ...(record.instructions !== undefined ? { instructions: record.instructions as string } : {}), + ...(record.answerSchema !== undefined ? { answerSchema: record.answerSchema as AnswerSchema } : {}), + } + }) +} + +/** The engine's `question` input: a short header, then the question, whole. */ +export function renderTraceQuestionPrompt(question: string): string { + return `${ASK_QUESTION_HEADER}${QUESTION_LABEL}${question}` +} + +/** The definition's instructions: rules first, so the preview keeps them. */ +export function renderTraceQuestionInstructions(question: Pick): string { + const rules = question.answerSchema ? `${ASK_RULES}\n${ANSWER_SCHEMA_RULE}` : ASK_RULES + return [ + rules, + question.instructions ? `QUESTION GUIDANCE:\n${question.instructions}` : '', + question.answerSchema ? `ANSWER SCHEMA:\n${JSON.stringify(question.answerSchema)}` : '', + ].filter(Boolean).join('\n\n') +} + +/** Characters of the instructions the DSPy preview keeps from the start; the rules must fit. */ +export const TRACE_QUESTION_PREVIEW_HEAD_CHARS = DSPY_PREVIEW_HEAD_CHARS + +function describeTraces(spans: readonly OtlpSpan[]): TraceQuestionTrace[] { + const { sessionByTrace } = indexSessionIdsByTrace(spans) + const byTrace = new Map() + for (const span of spans) { + const entry = byTrace.get(span.trace_id) ?? { count: 0, start: Infinity, end: -Infinity, root: null } + entry.count += 1 + const start = Date.parse(span.start_time) + const end = Date.parse(span.end_time) + if (Number.isFinite(start)) entry.start = Math.min(entry.start, start) + if (Number.isFinite(end)) entry.end = Math.max(entry.end, end) + if (span.parent_span_id === null && entry.root === null) entry.root = span.name + byTrace.set(span.trace_id, entry) + } + return [...byTrace].map(([traceId, entry]) => ({ + traceId, + sessionId: sessionByTrace.get(traceId) ?? null, + spanCount: entry.count, + startTime: Number.isFinite(entry.start) ? new Date(entry.start).toISOString() : null, + endTime: Number.isFinite(entry.end) ? new Date(entry.end).toISOString() : null, + rootSpan: entry.root, + })) +} + +/** + * What every question receives before its first model call: the trace list, + * then the deterministic session-facts sheet. + * + * Both are free. The list lets the model copy trace IDs instead of guessing + * them; the sheet answers the counting questions the bounded trace tools cannot + * (`viewTrace` degrades to a 20-entry histogram above 150,000 bytes, and + * `searchTrace` stops at 500 hits). Neither is citable: the sheet names the span + * ids behind every fact, and a citation still has to resolve against the store. + */ +function preparedContext(traces: readonly TraceQuestionTrace[], spans: readonly OtlpSpan[]): string { + const list = JSON.stringify({ + traces: traces.slice(0, MAX_CONTEXT_TRACES).map((trace) => ({ + trace_id: trace.traceId, + session_id: trace.sessionId, + spans: trace.spanCount, + start: trace.startTime, + end: trace.endTime, + root: trace.rootSpan, + })), + omitted_traces: Math.max(0, traces.length - MAX_CONTEXT_TRACES), + }) + const sheet = sessionFactsContext(spans) + return sheet ? `${list}\n\n${sheet}` : list +} + +const TRACE_URI = /trace:\/\/[^\s/"'`<>()[\]{}]+\/span\/[^\s/"'`<>()[\]{},;]+/g + +/** Every distinct `trace:///span/` URI in the text, in order. */ +export function traceCitationsInText(text: string): Array<{ uri: string; traceId: string | null; spanId: string | null }> { + const seen = new Set() + const out: Array<{ uri: string; traceId: string | null; spanId: string | null }> = [] + for (const match of text.matchAll(TRACE_URI)) { + // Sentence punctuation after a URI is prose, not part of the ID. + const uri = match[0].replace(/[.:!?]+$/, '') + if (seen.has(uri)) continue + seen.add(uri) + const parts = /^trace:\/\/([^/]+)\/span\/([^/]+)$/.exec(uri) + let traceId: string | null = null + let spanId: string | null = null + if (parts) { + try { + traceId = decodeURIComponent(parts[1]!) + spanId = decodeURIComponent(parts[2]!) + } catch { + traceId = null + spanId = null + } + } + out.push({ uri, traceId, spanId }) + } + return out +} + +async function verifyCitations( + text: string, + store: TraceAnalysisStore, + signal: AbortSignal | undefined, +): Promise { + const citations = traceCitationsInText(text) + const wanted = new Map>() + for (const citation of citations) { + if (!citation.traceId || !citation.spanId) continue + const spans = wanted.get(citation.traceId) ?? new Set() + spans.add(citation.spanId) + wanted.set(citation.traceId, spans) + } + const found = new Map>() + for (const [traceId, spanIds] of wanted) { + const existing = await store.hasSpans({ trace_id: traceId, span_ids: [...spanIds] }, signal ? { signal } : undefined) + found.set(traceId, new Set(existing)) + } + return citations.map((citation) => ({ + ...citation, + resolved: Boolean(citation.traceId && citation.spanId && found.get(citation.traceId)?.has(citation.spanId)), + })) +} + +function isBudgetRefusal(error: unknown): boolean { + if (error instanceof CostCeilingReachedError) return true + const message = error instanceof Error ? error.message : String(error) + // The DSPy engine crosses a process boundary, so the class does not survive; + // these are the ledger's and the model proxy's own refusal texts. + return /would exceed ceiling|model cost limit reached/.test(message) +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? `${error.constructor.name}: ${error.message}` : String(error) +} + +/** + * Smallest reservation one model call makes before it runs: its full output and + * reasoning allowance at the output rate, before any input tokens. Undefined + * when the engine does not describe its pricing and token caps. + */ +export function engineCallReservationFloorUsd(engine: TraceAnalysisEngine): number | undefined { + const config = engine.executionConfig + const pricing = config.pricing as { outputUsdPerMillion?: unknown } | undefined + const outputRate = pricing?.outputUsdPerMillion + const maxOutput = config.max_output_tokens + const maxReasoning = config.max_reasoning_tokens ?? 0 + if (typeof outputRate !== 'number' || typeof maxOutput !== 'number' || typeof maxReasoning !== 'number') { + return undefined + } + return ((maxOutput + maxReasoning) * outputRate) / 1_000_000 +} + +function formatUsd(value: number): string { + return `$${value < 0.01 && value > 0 ? value.toPrecision(2) : value.toFixed(2)}` +} + +function sumOrNull(values: ReadonlyArray): number | null { + return values.some((value) => value === null) ? null : values.reduce((sum, value) => sum + (value ?? 0), 0) +} + +/** + * Ask every question of the spans, concurrently, under one shared cost + * ledger. Never throws for a failed question: each failure is recorded on its + * answer and `ok` turns false. Throws before any model call for invalid + * questions, options, or a budget below one call's reservation. + */ +export async function runTraceQuestions(opts: TraceQuestionsOptions): Promise { + if (opts.spans.length === 0) throw new Error('runTraceQuestions: no spans to ask about') + const questions = normalizeTraceQuestions(opts.questions) + const concurrency = opts.concurrency ?? DEFAULT_ASK_CONCURRENCY + if (!Number.isSafeInteger(concurrency) || concurrency < 1) throw new RangeError('concurrency must be an integer >= 1') + const budgetUsd = opts.budgetUsd + if (budgetUsd !== undefined && (!Number.isFinite(budgetUsd) || budgetUsd <= 0)) { + throw new RangeError('budgetUsd must be a positive number') + } + const warnings: string[] = [] + const floor = engineCallReservationFloorUsd(opts.engine) + const effectiveConcurrency = Math.min(concurrency, questions.length) + if (budgetUsd !== undefined && floor !== undefined) { + if (budgetUsd < floor) { + throw new RangeError( + `budget ${formatUsd(budgetUsd)} is below one model call's reservation of at least ${formatUsd(floor)} ` + + `for ${opts.engine.model ?? opts.engine.id}; every question would be refused before it ran`, + ) + } + if (budgetUsd < floor * effectiveConcurrency) { + warnings.push( + `budget ${formatUsd(budgetUsd)} covers at most ${Math.floor(budgetUsd / floor)} concurrent model call(s) ` + + `at ${formatUsd(floor)} reserved each; questions wait for reservations, and later calls are refused ` + + 'once settled spend leaves less than one reservation', + ) + } + } + opts.signal?.throwIfAborted() + + const generatedAt = opts.generatedAt ?? new Date().toISOString() + const traceFile = await writeAnalysisTraceFile(opts.spans, { + sourceBundle: opts.sourceBundle, + otlpOutPath: opts.otlpOutPath, + signal: opts.signal, + }) + const store = await openAgenticTraceStore(traceFile) + const traces = describeTraces(opts.spans) + const context = preparedContext(traces, opts.spans) + const runId = `traces-ask-${Date.parse(generatedAt) || Date.now()}` + const ledger = new CostLedger(budgetUsd) + + let active = 0 + let peakConcurrency = 0 + const askOne = async (question: TraceQuestion & { id: string }): Promise => { + const startedAt = new Date() + const started = performance.now() + const rejections = createFindingRejectionTally(question.id) + let usage: AnalystUsageReceipt | null = null + let completed: TraceAnalysisEngineResult | undefined + let failure: TraceQuestionAnswer['failure'] + const log = (msg: string, fields?: Record): void => { + rejections.record(msg, fields) + opts.log?.(`[${question.id}] ${msg}`, fields) + } + const definition: TraceAnalystDefinition = defineTraceAnalyst({ + // Namespaced so a question ID can never select a built-in kind's subject rules. + id: `ask.${question.id}`, + description: `traces ask question ${question.id}`, + area: 'question', + version: ASK_DEFINITION_VERSION, + question: renderTraceQuestionPrompt(question.question), + instructions: renderTraceQuestionInstructions(question), + toolGroup: 'all', + prepareContext: () => context, + ...(opts.limits ? { limits: opts.limits } : {}), + }) + if (opts.signal?.aborted) { + failure = { kind: 'aborted', message: 'run aborted before the question started' } + } else { + active += 1 + peakConcurrency = Math.max(peakConcurrency, active) + try { + completed = await runTraceAnalyst({ + definition, + engine: opts.engine, + store, + context: { + runId, + correlationId: `${runId}:${question.id}`, + costLedger: ledger, + costPhase: 'trace-question', + log, + recordUsage: (receipt) => { + usage = receipt + }, + ...(opts.signal ? { signal: opts.signal } : {}), + }, + }) + } catch (error) { + failure = { + kind: opts.signal?.aborted ? 'aborted' : isBudgetRefusal(error) ? 'budget-refused' : 'error', + message: errorMessage(error), + } + } finally { + active -= 1 + } + } + + const answer = completed && completed.answer.trim() ? completed.answer : null + let parsedAnswer: unknown + let hasParsedAnswer = false + let citations: TraceQuestionCitation[] = [] + if (answer !== null) { + citations = await verifyCitations(answer, store, opts.signal) + if (question.answerSchema) { + const parsed = parseJsonAnswer(answer) + const problems = parsed.ok ? answerSchemaErrors(parsed.value, question.answerSchema) : [parsed.error] + if (parsed.ok) { + parsedAnswer = parsed.value + hasParsedAnswer = true + } + if (problems.length > 0 && !failure) { + failure = { kind: 'invalid-answer', message: problems.slice(0, 5).join('; ') } + } + } + const unresolved = citations.filter((citation) => !citation.resolved) + if (unresolved.length > 0 && !failure) { + failure = { + kind: 'unresolved-citations', + message: `${unresolved.length} cited span(s) do not exist: ${unresolved.slice(0, 3).map((c) => c.uri).join(', ')}`, + } + } + } else if (completed && !failure) { + failure = { kind: 'no-answer', message: 'the engine returned an empty answer' } + } + const endedAt = new Date() + return { + id: question.id, + question: question.question, + status: failure ? 'failed' : 'answered', + ...(failure ? { failure } : {}), + answer, + ...(hasParsedAnswer ? { parsedAnswer } : {}), + citations, + findings: completed?.findings ?? [], + rejectedFindings: rejections.counts()[question.id] ?? {}, + model: opts.engine.model ?? null, + modelCalls: completed?.modelCalls ?? null, + toolCalls: completed?.toolCalls ?? null, + usage, + startedAt: startedAt.toISOString(), + endedAt: endedAt.toISOString(), + latencyMs: Math.round(performance.now() - started), + ...(completed ? { trajectory: completed.trajectory } : {}), + } + } + + // A worker pool, not Promise.all: at most `concurrency` engines (each a + // Python process and a model proxy) exist at once. Unlike the import pool, + // one failed question never stops the others; its failure is its answer. + const answers = new Array(questions.length) + let next = 0 + const runStarted = performance.now() + await Promise.all(Array.from({ length: effectiveConcurrency }, async () => { + while (next < questions.length) { + const index = next + next += 1 + answers[index] = await askOne(questions[index]!) + } + })) + const wallTimeMs = Math.round(performance.now() - runStarted) + + const summary = ledger.summary({ channel: 'analyst' }) + const answered = answers.filter((answer) => answer.status === 'answered').length + const totals: TraceQuestionsTotals = { + questions: answers.length, + answered, + failed: answers.length - answered, + modelCalls: sumOrNull(answers.map((answer) => answer.modelCalls)), + toolCalls: sumOrNull(answers.map((answer) => answer.toolCalls)), + providerCalls: summary.totalCalls + summary.pendingCalls, + cost: summary.costProvenance, + wallTimeMs, + questionTimeMs: answers.reduce((sum, answer) => sum + answer.latencyMs, 0), + peakConcurrency, + } + const partial: Omit = { + schemaVersion: 1, + kind: 'traces.ask', + generatedAt, + harness: opts.harness ?? 'unknown', + engine: { id: opts.engine.id, version: opts.engine.version, model: opts.engine.model ?? null }, + concurrency, + budgetUsd: budgetUsd ?? null, + questionBudgetUsd: typeof opts.engine.executionConfig.max_cost_usd === 'number' + ? opts.engine.executionConfig.max_cost_usd + : null, + spanCount: opts.spans.length, + otlpPath: traceFile.otlpPath, + traces, + questions: answers, + totals, + warnings, + ok: answered === answers.length, + } + return { ...partial, report: renderTraceQuestionsReport(partial) } +} + +function costText(cost: CostProvenance | undefined | null): string { + if (!cost) return 'not captured' + return cost.kind === 'uncaptured' ? 'uncaptured' : `${formatUsd(cost.usd)} ${cost.kind}` +} + +function seconds(ms: number): string { + return `${(ms / 1000).toFixed(1)} s` +} + +function cell(value: string): string { + return value.replace(/\|/g, '\\|').replace(/\s+/g, ' ').trim() +} + +function countText(value: number | null): string { + return value === null ? 'unknown' : String(value) +} + +/** Readable Markdown for a result; the JSON result carries every field. */ +export function renderTraceQuestionsReport(result: Omit): string { + const { totals } = result + const lines = ['# traces ask', ''] + lines.push( + `${totals.questions} question(s) over ${result.traces.length} trace(s) (${result.spanCount} spans, ${result.harness}). ` + + `Engine \`${result.engine.id}\`${result.engine.model ? `, model \`${result.engine.model}\`` : ''}; ` + + `concurrency ${result.concurrency}; budget ${result.budgetUsd === null ? 'uncapped' : `${formatUsd(result.budgetUsd)} shared`}` + + `${result.questionBudgetUsd === null ? '' : `, ${formatUsd(result.questionBudgetUsd)} per question`}.`, + ) + lines.push('') + lines.push( + `**${totals.answered} answered, ${totals.failed} failed.** Wall time ${seconds(totals.wallTimeMs)} ` + + `against ${seconds(totals.questionTimeMs)} of question time (peak ${totals.peakConcurrency} at once). ` + + `Cost ${costText(totals.cost)} over ${totals.providerCalls} provider call(s); ` + + `${countText(totals.modelCalls)} model call(s), ${countText(totals.toolCalls)} tool call(s).`, + ) + lines.push('') + for (const warning of result.warnings) { + lines.push(`> Warning: ${warning}`) + lines.push('') + } + lines.push('| ID | Status | Citations | Rejected findings | Model calls | Tool calls | Cost | Time |') + lines.push('|---|---|---|---|---|---|---|---|') + for (const answer of result.questions) { + const resolved = answer.citations.filter((citation) => citation.resolved).length + lines.push( + `| \`${cell(answer.id)}\` | ${answer.status === 'answered' ? 'answered' : `failed: ${answer.failure?.kind}`} | ` + + `${resolved}/${answer.citations.length} resolved | ${totalFindingRejections(answer.rejectedFindings)} | ` + + `${countText(answer.modelCalls)} | ${countText(answer.toolCalls)} | ${costText(answer.usage?.cost)} | ` + + `${seconds(answer.latencyMs)} |`, + ) + } + lines.push('') + for (const answer of result.questions) { + lines.push(`## ${answer.id}: ${answer.question}`) + lines.push('') + if (answer.failure) { + lines.push(`**Failed (${answer.failure.kind}):** ${answer.failure.message.trim()}`) + lines.push('') + } + if (answer.answer !== null) { + lines.push(answer.answer.trim()) + lines.push('') + } + const notes: string[] = [] + const unresolved = answer.citations.filter((citation) => !citation.resolved) + if (answer.citations.length > 0) { + notes.push( + `- **Citations:** ${answer.citations.length - unresolved.length} of ${answer.citations.length} resolve to a span in the trace.`, + ) + for (const citation of unresolved) notes.push(`- **Unresolved:** ${citation.uri}`) + } else if (answer.answer !== null) { + notes.push('- **Citations:** none in the answer text.') + } + if (answer.findings.length > 0) { + notes.push(`- **Accepted findings:** ${answer.findings.length}`) + for (const finding of answer.findings.slice(0, 5)) notes.push(` - ${finding.severity}: ${finding.claim}`) + } + const rejected = formatFindingRejections(answer.rejectedFindings) + if (rejected) notes.push(`- **Rejected by the evidence gate:** ${rejected}`) + if (notes.length > 0) lines.push(...notes, '') + } + return lines.join('\n') +} + +/** Write `answers.json` and `report.md` next to the trace file the engine read. */ +export async function writeTraceQuestionsArtifacts( + result: TraceQuestionsResult, + outDir: string, +): Promise { + const directory = resolve(outDir) + await mkdir(directory, { recursive: true }) + const paths: TraceQuestionsArtifacts = { + directory, + result: join(directory, 'answers.json'), + report: join(directory, 'report.md'), + traces: result.otlpPath, + } + const { report, ...machine } = result + await Promise.all([ + writeFile(paths.result, `${JSON.stringify(machine, null, 2)}\n`, 'utf8'), + writeFile(paths.report, report, 'utf8'), + ]) + return paths +} diff --git a/src/cli.ts b/src/cli.ts index 6450085..2a5aae9 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -11,6 +11,8 @@ * traces analyze [--format auto] [--out report.md] * traces investigate [input.jsonl] [--format auto] [--out report.md] * traces improve [input.jsonl] [--format auto] --dir .traces/improvement + * traces ask --harness codex --session --question "..." [--questions q.json] [--dir ] + * traces facts [--harness codex] [--last 5] [--format json|text] [--out facts.json] * traces convert [--harness claude-code] [--last 1] --otlp-out spans.jsonl * traces index [--harness claude-code] [--last 20] --out session-index.json * traces bundle --harness claude-code --session --out @@ -42,7 +44,8 @@ import { assertOutsideSourceBundle } from './bundle-source.js' import { readFileSync } from 'node:fs' -import { readdir, readFile, stat, writeFile } from 'node:fs/promises' +import { mkdir, mkdtemp, readdir, readFile, stat, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' import { basename, join, resolve } from 'node:path' import { appendAll } from './arrays.js' import { indexSessionIdsByTrace } from './attributes.js' @@ -58,7 +61,8 @@ import { type VerifyFindingsRun, } from './analyze-verify.js' import { parseCorpusFlag } from './replay-corpus.js' -import { commandAnalyzer, commandRedactor, haloAnalyzer } from './external.js' +import { commandAnalyzer, commandRedactor, externalFailureMessage, haloAnalyzer } from './external.js' +import { findingRejectionDetail } from './finding-rejections.js' import { hodoscopeAnalyzer } from './hodoscope.js' import { primeAnalyzer } from './analyst-engine-prime.js' import { type TraceEvidenceFormatOption, exportTraceEvidenceFile, writeTraceEvidenceExportFile } from './file-export.js' @@ -66,6 +70,7 @@ import { inspectSessionIndex, readSessionIndexFile, renderInspectionReport, writ import { loadTracesConfig, mergeTracesConfig, + isBridgeMismatchError, runTraceImprovement, runTraceInvestigation, saveReport, @@ -89,8 +94,17 @@ import { } from '@tangle-network/agent-eval/supervisor-run' import { fileRunContextSupervisorRunReader, isFileRunContextDir } from './supervisor-run-context.js' import { resolveRunWatchTarget, watchRunTarget } from './run-watch.js' -import { createDspyRlmTraceEngine, type TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' -import { analystMaxOutputTokens, createAnalystModelOwner } from './analyst-model-call.js' +import type { TraceAnalysisEngine } from '@tangle-network/agent-eval/analyst' +import { analysisEngineFromEnv, DEFAULT_ANALYST_MODEL, DEFAULT_QUESTION_MAX_COST_USD } from './analyst-model-call.js' +import { + loadTraceQuestionsFile, + MAX_TRACE_QUESTION_CHARS, + normalizeTraceQuestions, + runTraceQuestions, + type TraceQuestion, + type TraceQuestionsResult, + writeTraceQuestionsArtifacts, +} from './ask.js' import type { OtlpSpan } from './otlp.js' import { serializeSpans, writeOtlpFile } from './otlp.js' import type { @@ -101,6 +115,7 @@ import type { UnreadableSourceRows, } from './otlp-input.js' import { readOtlpInput } from './otlp-input.js' +import { buildSessionFactsReport, renderSessionFacts } from './session-facts.js' import { renderValidation, validationExitCode } from './conformance.js' import type { TraceValidation } from '@tangle-network/agent-trace-contract' import { watchSessions } from './observer.js' @@ -179,13 +194,14 @@ interface Args { replayCorpora: string[] /** Receipt root for --verify-findings; defaults next to --out. */ verifyOut?: string + /** ask: repeatable `--question` texts. */ + questions: string[] + /** ask: JSON file of questions, with optional IDs, guidance, and answer schemas. */ + questionsFile?: string + /** ask: provider ceiling for one question; `--budget` bounds all of them together. */ + questionBudget?: number } -const DEFAULT_ANALYST_MODEL = 'gpt-5.6-luna' - -/** Default `--llm` endpoint: the Tangle router, reached with TANGLE_API_KEY. */ -const TANGLE_ROUTER_BASE_URL = 'https://router.tangle.tools/v1' - function packageVersion(): string { const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as { version?: unknown } if (typeof pkg.version !== 'string' || !pkg.version) throw new Error('package.json is missing version') @@ -221,6 +237,7 @@ function parseArgs(argv: string[]): Args { concurrency: 4, verifyFindings: false, replayCorpora: [], + questions: [], } for (let i = 1; i < argv.length; i++) { const arg = argv[i] @@ -269,6 +286,9 @@ function parseArgs(argv: string[]): Args { case '--replay-corpus': { const v = next(); if (v) a.replayCorpora.push(v); break } case '--verify-out': a.verifyOut = next(); break case '--analyzer-prompt': a.analyzerPrompt = next(); break + case '--question': { const v = next(); if (v !== undefined) a.questions.push(v); break } + case '--questions': a.questionsFile = next(); break + case '--question-budget': a.questionBudget = Number(next()); break case '--redactor': a.redactorCmd = next(); break case '--format': a.format = next(); break case '--help': @@ -289,7 +309,7 @@ function parseArgs(argv: string[]): Args { * artifact with `--otlp-out`: one flag, one direction, no command where the * same word means read here and write there. */ -const OTLP_INPUT_COMMANDS = new Set(['analyze', 'investigate', 'improve', 'stream', 'validate']) +const OTLP_INPUT_COMMANDS = new Set(['analyze', 'facts', 'investigate', 'improve', 'ask', 'stream', 'validate']) /** * `--otlp` used to mean "write the artifact here" on every command. It now @@ -327,8 +347,10 @@ function validateOtlpSelection(raw: Args): Args { const CURRENT_SESSION_COMMANDS = new Set([ 'analyze', + 'facts', 'investigate', 'improve', + 'ask', 'convert', 'index', 'evidence', @@ -338,8 +360,10 @@ const CURRENT_SESSION_COMMANDS = new Set([ const WORKFLOW_COMMANDS = new Set([ 'analyze', + 'facts', 'investigate', 'improve', + 'ask', 'convert', 'index', 'evidence', @@ -481,68 +505,16 @@ async function resolveSelectedSession(args: Args): Promise<{ adapter: HarnessTra } /** - * The recursive analysis engine behind `--llm`. agent-eval's model-backed - * analysts run through DSPy RLM, which drives `agent-eval-rpc[dspy]` out of - * process — so `--llm` needs a Python interpreter with that extra installed, - * selectable via TRACES_PYTHON. Every deterministic command is unaffected and - * still needs neither a key nor Python. + * `--llm` engine. It forwards the whole `--budget` as the per-analyst provider + * ceiling: the registry already splits `--budget` across analysts, so this only + * stops the engine's own $1 default from cutting runs short. */ -function buildAnalysisEngine(model: string, budgetUsd?: number): TraceAnalysisEngine { - // The router is the default endpoint, so TANGLE_API_KEY alone is enough. - // OPENAI_API_KEY still works and, when it is the only key present, points at - // OpenAI directly — otherwise a plain OpenAI key would be sent to the router. - const tangleKey = process.env.TANGLE_API_KEY - const openAiKey = process.env.OPENAI_API_KEY - const apiKey = tangleKey || openAiKey - if (!apiKey) { - throw new Error( - '--llm needs a model key: TANGLE_API_KEY for the Tangle router (the default endpoint), or ' + - 'OPENAI_API_KEY for OpenAI. Set OPENAI_BASE_URL to target any other OpenAI-compatible ' + - 'gateway. Deterministic analysis needs no key.', - ) - } - const baseUrl = - process.env.OPENAI_BASE_URL || - (tangleKey ? TANGLE_ROUTER_BASE_URL : 'https://api.openai.com/v1') - const python = process.env.TRACES_PYTHON - const owner = createAnalystModelOwner({ - apiKey, - baseUrl, - model, - provider: - baseUrl === TANGLE_ROUTER_BASE_URL - ? 'tangle-router' - : baseUrl.startsWith('https://api.openai.com/') - ? 'openai' - : 'openai-compatible', - }) - const maxOutputTokens = analystMaxOutputTokens(model) - return createDspyRlmTraceEngine({ - call: owner.call, - callRef: owner.callRef, - recordExecution: (observation) => { - analystLog( - `[analyst] model call ${observation.sequence} ${observation.succeeded ? 'ok' : 'FAIL'} ${observation.model}`, - observation.succeeded ? undefined : { error: observation.error }, - ) - }, - model, - // Model-aware, not defaulted: GPT-5.6 needs less output room than models - // such as GLM, and every recursive call reserves this full amount before - // execution. An oversized reservation can reject useful later calls even - // when the run's measured spend remains well below its limit. - maxOutputTokens, - // maxCostUsd defaults to $1 per analyst — a proxy-side ceiling separate - // from --budget. With the larger token cap the per-call reservation grows - // ~4x, so that default binds before the registry's --budget allocation - // and kills analysts mid-run. --budget is the operator's spend authority - // and the registry still splits it across analysts, so forwarding it here - // only stops the engine's own default from cutting runs short. - ...(budgetUsd !== undefined && Number.isFinite(budgetUsd) && budgetUsd > 0 - ? { maxCostUsd: budgetUsd } - : {}), - ...(python ? { runner: { command: python } } : {}), - }) +function llmAnalysisEngine(args: Args, model: string): TraceAnalysisEngine { + return analysisEngineFromEnv({ model, maxCostUsd: args.budget, log: analystLog }) +} + +function analystModelFor(args: Args): string { + return args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL } /** @@ -564,17 +536,24 @@ function requiredBridgeVersion(): string { } /** - * `--llm` promises agentic findings; delivering a deterministic-only report - * with exit 0 when every agentic analyst died reads as success. Throwing after - * the report is written keeps the deterministic output AND fails loud. The - * bridge-version read happens only once total failure is established, so a - * package.json problem can never turn a successful run into exit 1. + * `--llm` promises agentic findings, and `--analyzer` promises that engine's + * output. Delivering the deterministic report with exit 0 when a requested + * analysis died reads as success. Throwing after the report is written keeps + * the deterministic output AND fails loud. The bridge-version read happens + * only once total agentic failure is established, so a package.json problem + * can never turn a successful run into exit 1. */ -function assertAgenticAnalystsRan(args: Args, agenticPerAnalyst: TraceInvestigationResult['agenticPerAnalyst']): void { - if (!args.llm) return - if (!totalAgenticFailureMessage(agenticPerAnalyst)) return - const message = totalAgenticFailureMessage(agenticPerAnalyst, { requiredBridgeVersion: requiredBridgeVersion() }) - throw new Error(message!) +function assertRequestedAnalysesRan( + args: Args, + result: Pick, +): void { + const messages: string[] = [] + if (args.llm && totalAgenticFailureMessage(result.agenticPerAnalyst)) { + messages.push(totalAgenticFailureMessage(result.agenticPerAnalyst, { requiredBridgeVersion: requiredBridgeVersion() })!) + } + const external = externalFailureMessage(result.external) + if (external) messages.push(external) + if (messages.length > 0) throw new Error(messages.join('\n')) } async function cmdList(args: Args): Promise { @@ -654,7 +633,7 @@ async function collectOtlpSpans(path: string): Promise { async function collectSpans(args: Args): Promise { if (args.sourceBundle) { - if (!['analyze', 'investigate', 'improve'].includes(args.command)) throw new Error('--source-bundle is an analysis option') + if (!['analyze', 'investigate', 'improve', 'ask'].includes(args.command)) throw new Error('--source-bundle is an analysis option') if (args.session || args.otlp || args.input || args.current || args.workflow || args.noContent || args.redactorCmd) { throw new Error('--source-bundle cannot be combined with another input, redaction, or metadata-only selection') } @@ -870,6 +849,49 @@ async function cmdBundleView(args: Args): Promise { ) } +/** + * `traces facts` — the deterministic session-facts sheet, printed. + * + * No model, no engine, no budget: the numbers come out of the spans. JSON by + * default because the sheet's consumers are programs; `--format text` prints + * the short readable form. Every fact names the span ids it came from, so any + * number here can be checked with `traces export` or a trace tool. + * + * Exits non-zero when a selected session cannot be read: `collectSpans` throws, + * and an unreadable session must never be reported as a session with no facts. + */ +async function cmdFacts(args: Args): Promise { + const format = args.format ?? 'json' + if (format !== 'json' && format !== 'text') { + throw new Error(`unknown facts format "${format}" (expected json or text)`) + } + const collected = await collectSpans(args) + if (collected.spans.length === 0) throw new Error('no spans found for the given selection') + warnIncompleteWorkflow(collected.workflow) + const report = buildSessionFactsReport(collected.spans, { harness: collected.harness }) + // A session that yielded no record spans was not read: only its root, and any + // integrity receipt for the bytes that failed to parse. Printing a sheet of + // zeros for it would state, in the sheet's own voice, that the session did + // nothing — which is the one thing the sheet must never do. + const unread = report.sessions.filter((facts) => facts.recordSpans === 0) + if (unread.length > 0) { + throw new Error( + `no records could be read from ${unread.length} selected session(s): ` + + unread + .map((facts) => `${facts.sessionId ?? facts.traceId} (${facts.unreadRecords.value ?? 0} unread record(s))`) + .join(', ') + + '. The facts sheet would state zeros the spans cannot support.', + ) + } + const rendered = format === 'json' ? `${JSON.stringify(report, null, 2)}\n` : renderSessionFacts(report) + if (args.out) { + await writeFile(args.out, rendered, 'utf8') + console.log(`session facts → ${args.out} (${report.sessions.length} session(s), $0, no model call)`) + return + } + process.stdout.write(rendered) +} + async function cmdInspect(args: Args): Promise { if (!args.input) throw new Error('inspect needs an index file; run `traces index --out session-index.json` first') const index = await readSessionIndexFile(args.input) @@ -1011,7 +1033,7 @@ async function cmdAnalyze(args: Args): Promise { } else { console.log(report) } - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) } /** @@ -1156,6 +1178,13 @@ async function collectImportedSpans(args: Args): Promise { * every engine-startup failure into an unactionable one-liner. */ function analystLog(msg: string, fields?: Record): void { + // A gate rejection names its cause only in the fields; without them the + // line says a finding was dropped but not why, which nobody can act on. + const rejection = findingRejectionDetail(msg, fields) + if (rejection) { + process.stderr.write(`${msg} — ${rejection}\n`) + return + } const error = typeof fields?.error === 'string' && fields.error ? fields.error : undefined const errorClass = typeof fields?.error_class === 'string' && fields.error_class ? `${fields.error_class}: ` : '' process.stderr.write(error ? `${msg} — ${errorClass}${error}\n` : `${msg}\n`) @@ -1181,7 +1210,7 @@ async function cmdInvestigate(args: Args): Promise { } else { console.log(result.report) } - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) } /** Conformance fields threaded from the source into the investigation options. */ @@ -1204,8 +1233,8 @@ async function investigate(args: Args, options: { loadDefaultConfig?: boolean } const config = args.config !== undefined || options.loadDefaultConfig !== false ? await loadTracesConfig(args.config) : undefined - const analystModel = args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL - const engine = args.llm ? buildAnalysisEngine(analystModel, args.budget) : undefined + const analystModel = analystModelFor(args) + const engine = args.llm ? llmAnalysisEngine(args, analystModel) : undefined return runTraceInvestigation(mergeTracesConfig({ sourceBundle: args.sourceBundle ? { path: args.sourceBundle } : undefined, spans, @@ -1230,8 +1259,8 @@ async function cmdImprove(args: Args): Promise { const { spans, harness, cwds, sources, workflow } = collected if (spans.length === 0) throw new Error('no spans found for the given selection') const config = await loadTracesConfig(args.config) - const analystModel = args.model ?? process.env.TRACES_ANALYST_MODEL ?? DEFAULT_ANALYST_MODEL - const engine = args.llm ? buildAnalysisEngine(analystModel, args.budget) : undefined + const analystModel = analystModelFor(args) + const engine = args.llm ? llmAnalysisEngine(args, analystModel) : undefined const result = await runTraceImprovement({ ...mergeTracesConfig({ sourceBundle: args.sourceBundle ? { path: args.sourceBundle } : undefined, @@ -1258,7 +1287,86 @@ async function cmdImprove(args: Args): Promise { `improvement artifacts → ${dir} ` + `(${result.findings.length} findings with actions and checks, OTLP: ${result.otlpPath})`, ) - assertAgenticAnalystsRan(args, result.agenticPerAnalyst) + assertRequestedAnalysesRan(args, result) +} + +/** + * `traces ask`: free-form questions over the selected sessions. Every question + * is its own recursive investigation; they run concurrently under one shared + * ledger, so `--budget` bounds the whole run and `--question-budget` bounds + * each question. The answers, their citation checks, and per-question cost and + * time are written before the exit code is decided, so a failed question never + * costs the others' answers. + */ +async function cmdAsk(args: Args): Promise { + if (args.out) throw new Error('ask writes a directory of artifacts; pass --dir instead of --out') + if (args.llm) throw new Error('ask always uses the model-backed engine; drop --llm') + const questions: TraceQuestion[] = [ + ...(args.questionsFile ? await loadTraceQuestionsFile(args.questionsFile) : []), + ...args.questions.map((question) => ({ question })), + ] + if (questions.length === 0) { + throw new Error('ask needs --question "" (repeatable) or --questions ') + } + // Reject a bad question before any session is parsed or process started. + normalizeTraceQuestions(questions) + if (args.questionBudget !== undefined && (!Number.isFinite(args.questionBudget) || args.questionBudget <= 0)) { + throw new Error('--question-budget must be a positive number of USD') + } + if (args.budget !== undefined && (!Number.isFinite(args.budget) || args.budget <= 0)) { + throw new Error('--budget must be a positive number of USD') + } + const collected = await collectSpans(args) + if (collected.spans.length === 0) throw new Error('no spans found for the given selection') + warnIncompleteWorkflow(collected.workflow) + const engine = analysisEngineFromEnv({ + model: analystModelFor(args), + maxCostUsd: args.questionBudget ?? Math.min(args.budget ?? Infinity, DEFAULT_QUESTION_MAX_COST_USD), + log: analystLog, + }) + const directory = resolve(args.dir ?? await mkdtemp(join(tmpdir(), 'traces-ask-'))) + await mkdir(directory, { recursive: true }) + const controller = new AbortController() + const interrupt = () => controller.abort(new Error('ask interrupted')) + process.once('SIGINT', interrupt) + let result: TraceQuestionsResult + try { + result = await runTraceQuestions({ + questions, + spans: collected.spans, + engine, + harness: collected.harness, + concurrency: args.concurrency, + ...(args.budget !== undefined ? { budgetUsd: args.budget } : {}), + ...(args.sourceBundle ? { sourceBundle: { path: args.sourceBundle } } : {}), + otlpOutPath: args.otlpOut ?? join(directory, 'traces.otlp.jsonl'), + signal: controller.signal, + log: analystLog, + }) + } finally { + process.removeListener('SIGINT', interrupt) + } + const artifacts = await writeTraceQuestionsArtifacts(result, directory) + process.stdout.write(result.report) + for (const warning of result.warnings) process.stderr.write(`warning: ${warning}\n`) + process.stderr.write( + `ask artifacts → ${artifacts.directory} (${result.totals.answered}/${result.totals.questions} answered, ` + + `wall ${(result.totals.wallTimeMs / 1000).toFixed(1)} s, answers: ${artifacts.result})\n`, + ) + if (!result.ok) { + const failed = result.questions.filter((answer) => answer.status === 'failed') + const lines = [ + `${failed.length} of ${result.questions.length} question(s) failed; the answers file holds every result.`, + ...failed.map((answer) => ` ${answer.id}: ${answer.failure?.kind}: ${answer.failure?.message.slice(0, 300)}`), + ] + if (failed.some((answer) => answer.failure?.kind === 'error' && isBridgeMismatchError(answer.failure.message))) { + lines.push( + 'hint: the DSPy bridge protocol is version-locked — the TRACES_PYTHON interpreter needs ' + + `agent-eval-rpc[dspy]==${requiredBridgeVersion()} (matching this package's @tangle-network/agent-eval).`, + ) + } + throw new Error(lines.join('\n')) + } } function summarizeFindingEvidence(finding: TraceLiveFinding): string { @@ -1615,6 +1723,17 @@ Commands: (--supervisor-run-dir reports a supervision tree instead) investigate Run typed investigation flow, including BYO config + evidence-backed actions improve Write findings, evidence, report, and canonical trace artifacts + ask Answer free-form questions over the selected sessions with the + model-backed engine: questions run concurrently under one budget, + every trace:// citation is checked, and answers.json + report.md + are written to --dir (exit 1 when any question fails) + facts Print the deterministic session-facts sheet for the selected + sessions: tool calls excluding synthesized spans, subagent spawns + with task names, human turns in order, the final message per task, + changed paths, first/last record times, and the harness token total. + No model call, no budget, $0. Every fact names the span ids it came + from; a fact the spans cannot support is null with its reason. + --format json (default) or text (exit 1 when a session cannot be read) convert Emit OTLP-JSONL only, to --otlp-out (HALO: use analyze --analyzer halo) index Emit a reusable session index JSON for later investigation bundle Assemble one session's durable evidence directory: transcript + @@ -1667,19 +1786,20 @@ Options: agent-runtime createFileRunContext journal, and OTLP span files. --since upload: window, 30m / 2h / 7d or an ISO date (default 24h); analyze: ISO cutoff --out Write report to a file - --dir improve: write artifacts to this directory + --dir improve/ask: write artifacts to this directory --otlp READ OTLP-JSONL emitted by any system, skipping the adapters. A directory reads the OTLP files under it — only the otlp/ subdirectory when the producer made one — and names the JSONL that is not OTLP instead of reading it as broken spans. - Supported by: validate, analyze, investigate, improve, stream. + Supported by: validate, analyze, investigate, improve, ask, stream. On a WRITING command it is the deprecated spelling of --otlp-out; it still works, with a warning, until 0.12. --source-bundle Analyze a retained full bundle; explicitly grant source-field reads. - Available for analyze, investigate, and improve. + Available for analyze, investigate, improve, and ask. --otlp-out WRITE the OTLP-JSONL artifact here (also evidence provenance / dry-run upload preview) --format analyze/export: auto | policy-evidence | sandbox-events | openinference | intelligence-spans | chat-trajectory + facts: json (default) | text --metadata analyze/export file: attach JSON object fields as span attributes --attr analyze/export file: attach one span attribute (repeatable) --mode stream: visualizer | findings | agent (default visualizer) @@ -1696,10 +1816,20 @@ Options: agent-eval-rpc[dspy] (TRACES_PYTHON selects the interpreter), version-matched to this package's @tangle-network/agent-eval. Exits 1 when every agentic analyst fails, with each reason. + ask uses the same engine and credentials without --llm. --model Model for --llm, HALO, and Hodoscope (default for --llm: ${DEFAULT_ANALYST_MODEL}) --config investigate/improve/stream: JS config with analysts, liveAnalysts, or external analyzers - --budget USD cap for agentic analysts - --analyzer analyze: also run halo, hodoscope, prime, or an installed command (repeatable) + --budget USD cap for agentic analysts; ask: one ceiling shared by every question + --question ask: a question (repeatable, at most ${MAX_TRACE_QUESTION_CHARS} characters) + --questions ask: JSON array of questions, strings or + { id?, question, instructions?, answerSchema? } + --question-budget + ask: provider ceiling for one question (default: the smaller + of --budget and $${DEFAULT_QUESTION_MAX_COST_USD}) + --concurrency ask: questions running at once (default 4); + import-codetracebench: trajectories imported at once + --analyzer analyze: also run halo, hodoscope, prime, or an installed command (repeatable); + exits 1 after writing the report when any requested analyzer fails prime posts the full span projection to an OpenAI-compatible bridge (TRACES_PRIME_BRIDGE_URL, default http://localhost:4181; TRACES_PRIME_MODEL, default prime/zai/glm-5.2; TRACES_PRIME_TIMEOUT_MS) @@ -1776,6 +1906,8 @@ async function main(): Promise { case 'validate': await cmdValidate(args); break case 'investigate': await cmdInvestigate(args); break case 'improve': await cmdImprove(args); break + case 'ask': await cmdAsk(args); break + case 'facts': await cmdFacts(args); break case 'convert': await cmdConvert(args); break case 'index': await cmdIndex(args); break case 'bundle': await cmdBundle(args); break diff --git a/src/external.ts b/src/external.ts index 1bc4472..afeabb1 100644 --- a/src/external.ts +++ b/src/external.ts @@ -655,6 +655,21 @@ export function runExternalAnalyzers( })) } +/** + * Operator-facing message naming every external analyzer that failed, or + * undefined when all succeeded. A caller that requested an engine should + * treat its failure as a failed run: the report still carries the error, but + * exit 0 would read as "the engine ran and found nothing". + */ +export function externalFailureMessage(results: readonly ExternalAnalysisResult[]): string | undefined { + const failed = results.filter((result) => !result.ok) + if (failed.length === 0) return undefined + return [ + `${failed.length} of ${results.length} external analyzer(s) failed; the report contains the other results.`, + ...failed.map((result) => ` ${result.analyzer}: ${(result.error ?? 'failed without an error message').replace(/\s+/g, ' ').trim().slice(0, 400)}`), + ].join('\n') +} + // ──────────────────────────────── redactors ──────────────────────────────── /** An external PII/secret scrubber for free-form text — catches what regex diff --git a/src/finding-rejections.ts b/src/finding-rejections.ts new file mode 100644 index 0000000..60d89bb --- /dev/null +++ b/src/finding-rejections.ts @@ -0,0 +1,104 @@ +/** + * Evidence-gate rejections, counted per analyst and reason. + * + * agent-eval's trace analysts drop a submitted finding when its citations do + * not resolve, and report each drop only as a `finding rejected: ...` log + * event. A report that shows "0 findings" without those events reads as "the + * model found nothing" when the truth may be "the gate refused everything it + * found". This module turns the log events into counts a report can print. + */ + +/** Rejection counts by reason, for one analyst or question. */ +export type FindingRejectionReasons = Readonly> + +/** Rejection counts keyed by analyst ID, then by reason. */ +export type FindingRejectionCounts = Readonly> + +const REJECTION_MESSAGE = /^(?:\[([^\]]+)\] )?finding rejected: (.+)$/ + +/** + * The analyst and reason behind one `finding rejected:` log event, or + * undefined for any other event. The registry prefixes each analyst's log + * message with `[] `; a direct `runTraceAnalyst` caller does not, + * so the analyst ID is optional here. + */ +export function findingRejection( + message: string, + fields?: Readonly>, +): { analystId?: string; reason: string } | undefined { + const match = REJECTION_MESSAGE.exec(message) + if (!match) return undefined + const kind = match[2]!.trim() + // "unresolved evidence" is one message for several causes; the cause is in + // the fields, and it is the part an operator can act on. + const reason = kind === 'unresolved evidence' && typeof fields?.reason === 'string' && fields.reason + ? fields.reason + : kind + return match[1] ? { analystId: match[1], reason } : { reason } +} + +/** One-line detail for a rejection log event, or undefined for any other event. */ +export function findingRejectionDetail( + message: string, + fields?: Readonly>, +): string | undefined { + const rejection = findingRejection(message, fields) + if (!rejection) return undefined + const parts = [rejection.reason] + if (typeof fields?.uri === 'string' && fields.uri) parts.push(`uri ${fields.uri}`) + if (typeof fields?.required === 'number' && typeof fields?.distinct === 'number') { + parts.push(`${fields.distinct} of ${fields.required} required distinct citation(s)`) + } + if (typeof fields?.subject === 'string' && fields.subject) parts.push(`subject ${fields.subject}`) + return parts.join('; ') +} + +export interface FindingRejectionTally { + /** Record one log event; events that are not rejections are ignored. */ + record(message: string, fields?: Readonly>): void + /** A snapshot of the counts so far. */ + counts(): FindingRejectionCounts +} + +/** + * Count rejections from a log stream. `defaultAnalystId` names the analyst + * for unprefixed events, as when one tally observes one question. + */ +export function createFindingRejectionTally(defaultAnalystId = 'unknown'): FindingRejectionTally { + const byAnalyst = new Map>() + return { + record(message, fields) { + const rejection = findingRejection(message, fields) + if (!rejection) return + const analystId = rejection.analystId ?? defaultAnalystId + const reasons = byAnalyst.get(analystId) ?? new Map() + reasons.set(rejection.reason, (reasons.get(rejection.reason) ?? 0) + 1) + byAnalyst.set(analystId, reasons) + }, + counts() { + return Object.fromEntries( + [...byAnalyst].map(([analystId, reasons]) => [analystId, Object.fromEntries(reasons)]), + ) + }, + } +} + +/** Total rejections across every reason. */ +export function totalFindingRejections(reasons: FindingRejectionReasons | undefined): number { + return Object.values(reasons ?? {}).reduce((sum, count) => sum + count, 0) +} + +/** + * "2 finding(s) rejected: excerpt is not present in the cited span content ×2". + * Empty when nothing was rejected. Reasons are ordered by count, then name, so + * the same counts always render the same text. + */ +export function formatFindingRejections(reasons: FindingRejectionReasons | undefined): string { + const total = totalFindingRejections(reasons) + if (total === 0) return '' + const ordered = Object.entries(reasons ?? {}) + .filter(([, count]) => count > 0) + .sort(([a, left], [b, right]) => right - left || a.localeCompare(b)) + .map(([reason, count]) => `${reason} ×${count}`) + return `${total} finding(s) rejected: ${ordered.join('; ')}` +} diff --git a/src/improvement.ts b/src/improvement.ts index 3888ee3..e6c2689 100644 --- a/src/improvement.ts +++ b/src/improvement.ts @@ -38,6 +38,7 @@ import type { UnreadableSourceRows, } from './otlp-input.js' import type { ExternalAnalysisResult, ExternalAnalyzer } from './external.js' +import { createFindingRejectionTally, type FindingRejectionCounts } from './finding-rejections.js' import { spanEvidenceUri } from './external-analysis-validation.js' import { runExternalAnalyzers } from './external.js' import type { TraceLiveAnalyst } from './live.js' @@ -161,6 +162,12 @@ export interface TraceInvestigationResult { * findings as if the requested LLM analysis had happened. */ readonly agenticPerAnalyst?: readonly AnalystRunSummary[] + /** + * Findings the evidence gate refused, by analyst ID and reason. Present + * whenever an agentic pass ran, empty when nothing was refused, so a zero + * finding count can be told apart from "every finding was rejected". + */ + readonly findingRejections?: FindingRejectionCounts readonly external: readonly ExternalAnalysisResult[] readonly report: string } @@ -614,6 +621,7 @@ function renderInvestigationReport( workflow: result.workflow, conformance, unavailableCapabilities: unavailable, + ...(result.findingRejections ? { findingRejections: result.findingRejections } : {}), })}\n${renderAgenticRoute(result.agenticRoute)}` + `${renderPipelines(result.pipelines, unavailable)}\n` + `${renderLoopConvergence(result.loopConvergence, unavailable)}\n` + @@ -693,6 +701,13 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr const agenticRoute = opts.engine && !opts.agenticRegistry ? planTraceAgenticRoute(pipelines, reactions) : undefined + // The evidence gate reports each refused finding only as a log event. + // Counting here keeps those refusals in the report and the JSON result. + const rejections = createFindingRejectionTally() + const log = (msg: string, fields?: Record): void => { + rejections.record(msg, fields) + opts.log?.(msg, fields) + } const analysis = await analyzeSpans(opts.spans, { sourceBundle: opts.sourceBundle, engine: opts.engine, @@ -705,7 +720,7 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr otlpOutPath: opts.otlpOutPath, runId: `traces-investigation-${Date.parse(generatedAt) || Date.now()}`, signal: opts.signal, - log: opts.log, + log, }) const external = opts.externalAnalyzers?.length ? await runExternalAnalyzers(analysis.otlpPath, opts.externalAnalyzers, { @@ -741,7 +756,9 @@ export async function runTraceInvestigation(opts: TraceInvestigationOptions): Pr loopConvergence: analyzeLoopConvergence(opts.spans), steeringChain: analyzeSteeringChain(opts.spans), ...(agenticRoute ? { agenticRoute } : {}), - ...(analysis.agenticPerAnalyst ? { agenticPerAnalyst: analysis.agenticPerAnalyst } : {}), + ...(analysis.agenticPerAnalyst + ? { agenticPerAnalyst: analysis.agenticPerAnalyst, findingRejections: rejections.counts() } + : {}), external, } const result = { ...partial, report: '' } @@ -760,6 +777,11 @@ const AGENTIC_FAILURE_REASON_MAX_CHARS = 400 */ const BRIDGE_MISMATCH_PATTERN = /must contain exactly|No module named ['"]?agent_eval_rpc|could not start/ +/** True when an engine error looks like bridge version skew or a missing bridge install. */ +export function isBridgeMismatchError(message: string): boolean { + return BRIDGE_MISMATCH_PATTERN.test(message) +} + function condensedReason(summary: AnalystRunSummary): string { const raw = summary.status === 'skipped' ? `skipped — ${summary.reason ?? 'no reason recorded'}` @@ -789,7 +811,7 @@ export function totalAgenticFailureMessage( ] if ( opts.requiredBridgeVersion && - agenticPerAnalyst.some((summary) => BRIDGE_MISMATCH_PATTERN.test(summary.error?.message ?? '')) + agenticPerAnalyst.some((summary) => isBridgeMismatchError(summary.error?.message ?? '')) ) { lines.push( 'hint: the DSPy bridge protocol is version-locked — the TRACES_PYTHON interpreter needs ' + diff --git a/src/index.ts b/src/index.ts index 58bc6a0..1c7c1b1 100644 --- a/src/index.ts +++ b/src/index.ts @@ -97,6 +97,12 @@ export * from './inspect.js' // inspectSessionIndex() — ranked findings from a export * from './file-export.js' // convert evidence/events files to OpenInference JSONL export * from './chat-trajectory.js' // generic chat trajectory to stable step spans export * from './improvement.js' // runTraceInvestigation()/runTraceImprovement() artifact pack +export * from './ask.js' // runTraceQuestions(): concurrent free-form questions, answers kept and citations checked +export * from './session-facts.js' // computeSessionFacts(): the deterministic, free session-facts sheet +export * from './answer-schema.js' // the JSON Schema subset an ask answer may be held to +export * from './finding-rejections.js' // evidence-gate rejections counted per analyst and reason +export { analysisEngineFromEnv, DEFAULT_ANALYST_MODEL } from './analyst-model-call.js' +export type { AnalysisEngineFromEnvOptions } from './analyst-model-call.js' // ── External engines (NOT bundled — shell out to tools you install) ──────── export * from './external.js' // haloAnalyzer / commandAnalyzer; commandRedactor diff --git a/src/report.ts b/src/report.ts index 1f35216..94425ff 100644 --- a/src/report.ts +++ b/src/report.ts @@ -16,6 +16,11 @@ import type { AdoptionReport } from './adoption.js' import { ACTOR_ATTR } from './adapters/conversation.js' import { ATTR, sessionIdFromAttributes } from './attributes.js' import { incompleteInputsNote, type UnavailableCapabilities } from './conformance.js' +import { + type FindingRejectionCounts, + type FindingRejectionReasons, + formatFindingRejections, +} from './finding-rejections.js' import type { LoopConvergenceReport, SteeringChainReport } from './loop-analysis.js' import type { OtlpSpan } from './otlp.js' import type { PipelineReport } from './pipelines.js' @@ -58,6 +63,8 @@ export interface ReportMeta { * confident totals, is worse than no conformance section at all. */ unavailableCapabilities?: UnavailableCapabilities + /** Findings the evidence gate refused, by analyst ID and reason. */ + findingRejections?: FindingRejectionCounts } export interface ReportSource { @@ -221,13 +228,16 @@ export function condenseAnalystError(raw: string, maxChars: number): string { * Failed engine runs die with the whole bridge stderr in the error message; * without this cell the report scores the analyst without saying why. */ -export function analystRunDetail(summary: AnalystRunSummary): string { +export function analystRunDetail(summary: AnalystRunSummary, rejections?: FindingRejectionReasons): string { const raw = summary.status === 'failed' && summary.error ? [summary.error.class, summary.error.message].map((part) => part.trim()).filter(Boolean).join(': ') : summary.status === 'skipped' && summary.reason ? summary.reason : '' - const cell = tableCell(condenseAnalystError(raw, ANALYST_DETAIL_MAX_CHARS)) + const detail = condenseAnalystError(raw, ANALYST_DETAIL_MAX_CHARS) + // The rejection text is short and bounded by the gate's reason vocabulary, + // so it is appended after the condensed error rather than competing for it. + const cell = tableCell([detail, formatFindingRejections(rejections)].filter(Boolean).join('; ')) return cell === '' ? '—' : cell } @@ -567,7 +577,10 @@ export function renderReport(result: AnalystRunResult, meta: ReportMeta): string lines.push('| Analyst | Status | Findings | Latency | Detail |') lines.push('|---|---|---|---|---|') for (const s of result.per_analyst) { - lines.push(`| \`${s.analyst_id}\` | ${s.status} | ${s.findings_count} | ${s.latency_ms}ms | ${analystRunDetail(s)} |`) + lines.push( + `| \`${s.analyst_id}\` | ${s.status} | ${s.findings_count} | ${s.latency_ms}ms | ` + + `${analystRunDetail(s, meta.findingRejections?.[s.analyst_id])} |`, + ) } lines.push('') diff --git a/src/session-facts.ts b/src/session-facts.ts new file mode 100644 index 0000000..df092f1 --- /dev/null +++ b/src/session-facts.ts @@ -0,0 +1,789 @@ +/** + * The deterministic session-facts sheet. + * + * Seven facts about a session decide most audit questions — how many tools ran, + * which subagents were spawned, what the human actually typed, what the agent + * said last, which files changed, when the session started and ended, and how + * many tokens it burned. Every one of them is already carried by the normalized + * spans, but no trace tool returns any of them: `viewTrace` degrades to a + * 20-entry name histogram above its byte ceiling, `countTraces` counts traces + * rather than spans, `viewSpans` needs span ids the reader does not have, and + * `searchTrace` stops at 500 hits. A model asked for a tool-call total therefore + * adds up a capped histogram and decides by eye which names count. + * + * This module computes those facts mechanically instead. No model call, no + * budget, no byte ceiling: it reads the spans directly, so a session far above + * the tool ceiling still yields exact numbers. + * + * Two rules make the sheet auditable: + * + * 1. Every field carries the span ids it was computed from, so a reader can + * open those spans and check the number. + * 2. A field the spans cannot support is `null` with a stated `unavailable` + * reason. It is never guessed, and never silently zero. + * + * The sheet is NOT a span and cannot be cited. It is prepared context, and + * `trace://` citations still resolve against the raw spans — which is why every + * fact names its span ids rather than asking the reader to trust the sheet. + */ + +import type { TraceAnalystDefinition } from '@tangle-network/agent-eval/analyst' +import { OPENINFERENCE_SPAN_KIND, TOOL_NAME } from '@tangle-network/agent-eval/trace-attributes' +import { ACTOR_ATTR } from './adapters/conversation.js' +import { indexSessionIdsByTrace } from './attributes.js' +import type { OtlpSpan } from './otlp.js' + +/** + * The cumulative harness token total for a whole session, when the adapter + * records one. Codex's `token_count` events carry + * `info.total_token_usage.total_tokens`; the adapter uses that object only as a + * de-duplication signature today, so no span carries the value and + * {@link SessionFacts.tokenTotal} states that rather than summing the per-turn + * deltas, which is a different (and smaller) number. + */ +export const SESSION_TOKEN_TOTAL_ATTR = 'traces.session.total_tokens' + +/** + * Set by an adapter on a span it created to describe a lifecycle, not to record + * an invocation the agent made. A synthesized span never counts as a tool call. + */ +export const SPAN_SYNTHESIZED_ATTR = 'traces.codex.span_synthesized' + +/** Records the session reader could not parse, stamped on the session span. */ +const CORRUPTION_COUNT_ATTR = 'traces.session.corruption_count' + +/** Present on a per-record integrity receipt span, which is not a session record. */ +const CORRUPTION_RECEIPT_VERSION_ATTR = 'traces.session.corruption.receipt_version' + +/** Max characters of message or prompt text kept per fact entry. */ +export const FACT_TEXT_CAP = 2000 + +/** Entries kept per list field before the sheet reports the rest as omitted. */ +export const FACT_LIST_CAP = 200 + +/** + * One measured fact. `value` is `null` exactly when `unavailable` explains why + * the spans cannot support it; `spanIds` names the spans the value came from, + * in the order they appear in the trace. + */ +export interface SessionFact { + readonly value: T | null + readonly spanIds: readonly string[] + /** Why `value` is null. Null when the fact was measured. */ + readonly unavailable: string | null + /** A measured value that is known to be incomplete says so here. */ + readonly partial?: string +} + +/** One `spawn_agent` call, with the task name the adapter recorded for it. */ +export interface SubagentSpawnFact { + /** `traces.codex.spawn_agent_path`, verbatim. Null when the span carries none. */ + readonly taskName: string | null + /** Why `taskName` is null. Null when it was recorded. */ + readonly taskNameUnavailable: string | null + readonly startedAt: string + /** The spawn span's status: `OK`, `ERROR`, or `UNSET` while still open. */ + readonly status: string + /** The spawn call span, plus the subagent lifecycle span when one joined it. */ + readonly spanIds: readonly string[] +} + +/** One `user.prompt` turn, with the actor the adapter classified it as. */ +export interface SessionTurnFact { + readonly actor: string + readonly at: string + readonly text: string + readonly spanId: string +} + +/** How many `user.prompt` turns each actor produced. */ +export interface TurnActorCount { + readonly actor: string + readonly turns: number + readonly spanIds: readonly string[] +} + +/** The last thing one agent said, for the main session or for one subagent. */ +export interface FinalMessageFact { + /** + * The subagent's task path, or null for the session's own agent. One session + * interleaves its own `message.assistant` spans with the `message.agent.*` + * traffic of every subagent under the same root, so "the final message" is + * only well defined per task. + */ + readonly task: string | null + readonly at: string + readonly text: string + readonly spanId: string +} + +/** One path a tool call changed, and how. */ +export interface ChangedFileFact { + readonly path: string + /** `add`, `update`, `delete`, or `move`, in the order first observed. */ + readonly operations: readonly string[] + readonly spanIds: readonly string[] +} + +/** Everything the spans of one session state, computed without a model. */ +export interface SessionFacts { + readonly schemaVersion: 1 + readonly kind: 'traces.session_facts' + readonly traceId: string + readonly sessionId: string | null + readonly harness: string | null + /** Spans in this trace, including the synthesized ones. */ + readonly spanCount: number + /** + * Spans the adapter built from the session's own records. The session root + * and the integrity receipts are not among them, so zero means nothing in the + * file was read and every count below would be an unsupported zero. + */ + readonly recordSpans: number + /** Records the adapter could not read, from the session's integrity receipt. */ + readonly unreadRecords: SessionFact + /** TOOL spans the agent actually invoked: synthesized lifecycle spans excluded. */ + readonly toolCalls: SessionFact + /** Every synthesized span excluded from `toolCalls`, so the exclusion is checkable. */ + readonly synthesizedToolSpans: SessionFact + /** Tool-call counts by tool name, over the same spans `toolCalls` counted. */ + readonly toolCallsByName: SessionFact>> + readonly subagents: SessionFact + /** `user.prompt` turns a person typed, in order. */ + readonly humanTurns: SessionFact + /** Every `user.prompt` turn by actor, so the human filter is checkable. */ + readonly turnsByActor: SessionFact + /** The last message of the session's own agent, and of each subagent task. */ + readonly finalMessages: SessionFact + readonly changedFiles: SessionFact + /** Earliest span start in the trace. Not the session file's first record when + * the adapter selected a task boundary inside the file. */ + readonly firstRecordAt: SessionFact + /** Latest span end in the trace. */ + readonly lastRecordAt: SessionFact + /** The harness's own cumulative token total, when a span carries it. */ + readonly tokenTotal: SessionFact +} + +export interface SessionFactsReport { + readonly schemaVersion: 1 + readonly kind: 'traces.session_facts_report' + readonly generatedAt: string + readonly harness: string | null + readonly spanCount: number + readonly sessions: readonly SessionFacts[] +} + +function attr(span: OtlpSpan, key: string): unknown { + return span.attributes[key] +} + +function stringAttr(span: OtlpSpan, key: string): string | null { + const value = span.attributes[key] + return typeof value === 'string' && value.length > 0 ? value : null +} + +function capText(raw: string): string { + const text = raw.trim() + if (text.length <= FACT_TEXT_CAP) return text + return `${text.slice(0, FACT_TEXT_CAP)}… [+${text.length - FACT_TEXT_CAP} chars]` +} + +/** Trace order: the adapter's `step` when it recorded one, else start time. */ +function orderOf(span: OtlpSpan): [number, number] { + const step = span.attributes.step + const start = Date.parse(span.start_time) + return [typeof step === 'number' ? step : Number.MAX_SAFE_INTEGER, Number.isFinite(start) ? start : 0] +} + +function byTraceOrder(left: OtlpSpan, right: OtlpSpan): number { + const [leftStep, leftStart] = orderOf(left) + const [rightStep, rightStart] = orderOf(right) + if (leftStart !== rightStart) return leftStart - rightStart + return leftStep - rightStep +} + +/** + * A span an adapter created to describe a subagent's lifecycle rather than to + * record a call the agent made. Codex's `ensureSubagentSpan` emits one such + * span per subagent thread with `kind: TOOL` and `tool.name: Agent`, so any + * span-kind count is high by exactly the number of subagents unless it is + * excluded. Two signatures identify it: the explicit tag an adapter may set, + * and the subagent thread/path attribute pair only that span carries. + */ +export function isSynthesizedSpan(span: OtlpSpan): boolean { + if (attr(span, SPAN_SYNTHESIZED_ATTR) === true) return true + return ( + stringAttr(span, 'traces.codex.subagent_thread_id') !== null && + stringAttr(span, 'traces.codex.subagent_path') !== null + ) +} + +function isToolSpan(span: OtlpSpan): boolean { + return attr(span, OPENINFERENCE_SPAN_KIND) === 'TOOL' +} + +function toolName(span: OtlpSpan): string { + return stringAttr(span, TOOL_NAME) ?? span.name.replace(/^tool\./, '') +} + +function inputValue(span: OtlpSpan): string | null { + const value = span.attributes['input.value'] + return typeof value === 'string' ? value : null +} + +function inputTruncated(span: OtlpSpan): boolean { + return span.attributes['traces.input.truncated'] === true +} + +// `apply_patch` envelopes name every path they touch in a header line. The +// adapter keeps the patch verbatim in `input.value`, so the headers survive +// unless the value exceeded the adapter's own 16 KiB I/O cap. +// +// The patch reaches the span either as raw text or as a JSON-encoded tool +// argument, where the line breaks are the two characters `\` and `n`. The +// header therefore ends at a real newline, at the backslash of an escape, or at +// the closing quote — which is also why a path containing `"` or `\` is not +// recovered, and is reported as a missing path rather than a wrong one. +const PATCH_HEADER = /\*\*\* (Add|Update|Delete|Move to) File: ([^\n"\\]+)/g +const PATCH_OPERATION: Readonly> = { + Add: 'add', + Update: 'update', + Delete: 'delete', + 'Move to': 'move', +} + +/** Tools whose arguments name one edited file directly rather than in a patch. */ +const FILE_PATH_TOOLS = new Set(['Edit', 'MultiEdit', 'Write', 'NotebookEdit', 'edit', 'write']) +const FILE_PATH_KEY = /"(?:file_path|filePath|path|notebook_path)"\s*:\s*"((?:[^"\\]|\\.)*)"/g + +function decodeJsonString(raw: string): string | null { + try { + const value: unknown = JSON.parse(`"${raw}"`) + return typeof value === 'string' && value.length > 0 ? value : null + } catch { + return null + } +} + +interface ChangedPath { + operations: string[] + spanIds: string[] +} + +function recordChangedPath( + into: Map, + path: string, + operation: string, + spanId: string, +): void { + const entry = into.get(path) ?? { operations: [], spanIds: [] } + if (!entry.operations.includes(operation)) entry.operations.push(operation) + if (!entry.spanIds.includes(spanId)) entry.spanIds.push(spanId) + into.set(path, entry) +} + +function changedFilesOf(toolSpans: readonly OtlpSpan[]): { + files: ChangedFileFact[] + truncatedInputs: number +} { + const paths = new Map() + let truncatedInputs = 0 + for (const span of toolSpans) { + const input = inputValue(span) + if (input === null) continue + let matched = false + for (const match of input.matchAll(PATCH_HEADER)) { + const operation = PATCH_OPERATION[match[1]!] + const path = match[2]?.trim() + if (!operation || !path) continue + matched = true + recordChangedPath(paths, path, operation, span.span_id) + } + if (!matched && FILE_PATH_TOOLS.has(toolName(span))) { + for (const match of input.matchAll(FILE_PATH_KEY)) { + const path = decodeJsonString(match[1] ?? '') + if (!path) continue + matched = true + recordChangedPath(paths, path, 'update', span.span_id) + } + } + if (matched && inputTruncated(span)) truncatedInputs += 1 + } + const files = [...paths] + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .map(([path, entry]) => ({ path, operations: entry.operations, spanIds: entry.spanIds })) + return { files, truncatedInputs } +} + +function subagentsOf(spans: readonly OtlpSpan[]): SubagentSpawnFact[] { + // A lifecycle span records the subagent's own path; joining it to the spawn + // call by that path gives the reader both spans to check the task name with. + const lifecycleByPath = new Map() + for (const span of spans) { + const path = stringAttr(span, 'traces.codex.subagent_path') + if (path === null || !isSynthesizedSpan(span)) continue + lifecycleByPath.set(path, [...(lifecycleByPath.get(path) ?? []), span.span_id]) + } + const spawns: SubagentSpawnFact[] = [] + for (const span of spans) { + const isSpawn = + attr(span, 'traces.codex.agent_operation') === 'spawn_agent' || + stringAttr(span, 'traces.codex.spawn_agent_path') !== null + if (!isSpawn || isSynthesizedSpan(span)) continue + const taskName = stringAttr(span, 'traces.codex.spawn_agent_path') + spawns.push({ + taskName, + taskNameUnavailable: taskName === null + ? 'the spawn span carries no traces.codex.spawn_agent_path; the harness returned no task name for this call' + : null, + startedAt: span.start_time, + status: span.status.code, + spanIds: [span.span_id, ...(taskName ? lifecycleByPath.get(taskName) ?? [] : [])], + }) + } + return spawns +} + +function finalMessagesOf(spans: readonly OtlpSpan[]): FinalMessageFact[] { + const last = new Map() + for (const span of spans) { + let task: string | null | undefined + if (span.name === 'message.assistant') task = null + else if (span.name.startsWith('message.agent.')) { + task = stringAttr(span, 'traces.codex.agent_message_author') ?? 'unattributed subagent' + } + if (task === undefined) continue + const previous = last.get(task) + if (!previous || byTraceOrder(previous, span) <= 0) last.set(task, span) + } + return [...last] + .sort(([left], [right]) => (left === null ? -1 : right === null ? 1 : left.localeCompare(right))) + .map(([task, span]) => ({ + task, + at: span.end_time, + text: capText(typeof span.attributes.content === 'string' ? span.attributes.content : ''), + spanId: span.span_id, + })) +} + +function capList(items: readonly T[]): { kept: readonly T[]; partial?: string } { + if (items.length <= FACT_LIST_CAP) return { kept: items } + return { + kept: items.slice(0, FACT_LIST_CAP), + partial: `${items.length - FACT_LIST_CAP} of ${items.length} entries omitted; the count above the list is complete`, + } +} + +function listFact(items: readonly T[], spanIds: readonly string[]): SessionFact { + const { kept, partial } = capList(items) + return { value: kept, spanIds: [...new Set(spanIds)], unavailable: null, ...(partial ? { partial } : {}) } +} + +/** + * Compute the facts sheet for every trace in `spans`. One trace is one session. + * Deterministic and free: the same spans always produce the same sheet, and no + * model, network call, or budget is involved. + */ +export function computeSessionFacts(spans: readonly OtlpSpan[]): SessionFacts[] { + const { sessionByTrace } = indexSessionIdsByTrace(spans) + const byTrace = new Map() + for (const span of spans) { + byTrace.set(span.trace_id, [...(byTrace.get(span.trace_id) ?? []), span]) + } + return [...byTrace] + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .map(([traceId, traceSpans]) => sessionFactsForTrace(traceId, sessionByTrace.get(traceId) ?? null, traceSpans)) +} + +function sessionFactsForTrace( + traceId: string, + sessionId: string | null, + unordered: readonly OtlpSpan[], +): SessionFacts { + const spans = [...unordered].sort(byTraceOrder) + const harness = spans.map((span) => stringAttr(span, 'service.name')).find((name) => name !== null) ?? null + + const toolSpans = spans.filter(isToolSpan) + const synthesized = toolSpans.filter(isSynthesizedSpan) + const invoked = toolSpans.filter((span) => !isSynthesizedSpan(span)) + const byName: Record = {} + for (const span of invoked) byName[toolName(span)] = (byName[toolName(span)] ?? 0) + 1 + + const subagents = subagentsOf(spans) + const promptSpans = spans.filter((span) => span.name === 'user.prompt') + const humanTurnSpans = promptSpans.filter((span) => stringAttr(span, ACTOR_ATTR) === 'human') + const actorCounts = new Map() + for (const span of promptSpans) { + const actor = stringAttr(span, ACTOR_ATTR) ?? 'unclassified' + actorCounts.set(actor, [...(actorCounts.get(actor) ?? []), span.span_id]) + } + const finalMessages = finalMessagesOf(spans) + const { files, truncatedInputs } = changedFilesOf(invoked) + + const starts = spans.filter((span) => Number.isFinite(Date.parse(span.start_time))) + const ends = spans.filter((span) => Number.isFinite(Date.parse(span.end_time))) + const firstSpan = starts.reduce( + (best, span) => (best === null || Date.parse(span.start_time) < Date.parse(best.start_time) ? span : best), + null, + ) + const lastSpan = ends.reduce( + (best, span) => (best === null || Date.parse(span.end_time) > Date.parse(best.end_time) ? span : best), + null, + ) + + const tokenSpan = spans.find((span) => typeof span.attributes[SESSION_TOKEN_TOTAL_ATTR] === 'number') + const receipts = spans.filter((span) => span.attributes[CORRUPTION_RECEIPT_VERSION_ATTR] !== undefined) + const recordSpans = spans.filter( + (span) => span.parent_span_id !== null && span.attributes[CORRUPTION_RECEIPT_VERSION_ATTR] === undefined, + ).length + const corruptionSpan = spans.find((span) => typeof span.attributes[CORRUPTION_COUNT_ATTR] === 'number') + + return { + schemaVersion: 1, + kind: 'traces.session_facts', + traceId, + sessionId, + harness, + spanCount: spans.length, + recordSpans, + unreadRecords: corruptionSpan + ? { + value: corruptionSpan.attributes[CORRUPTION_COUNT_ATTR] as number, + spanIds: [corruptionSpan.span_id, ...receipts.map((span) => span.span_id)], + unavailable: null, + } + : { + value: null, + spanIds: [], + unavailable: + `no span carries ${CORRUPTION_COUNT_ATTR}; this trace was not read through the session ` + + 'integrity path, so whether any record was skipped is unknown', + }, + toolCalls: { + value: invoked.length, + spanIds: invoked.map((span) => span.span_id), + unavailable: null, + }, + synthesizedToolSpans: { + value: synthesized.length, + spanIds: synthesized.map((span) => span.span_id), + unavailable: null, + }, + toolCallsByName: { + value: byName, + spanIds: invoked.map((span) => span.span_id), + unavailable: null, + }, + subagents: listFact(subagents, subagents.flatMap((entry) => entry.spanIds)), + humanTurns: listFact( + humanTurnSpans.map((span) => ({ + actor: 'human', + at: span.start_time, + text: capText(typeof span.attributes.content === 'string' ? span.attributes.content : ''), + spanId: span.span_id, + })), + humanTurnSpans.map((span) => span.span_id), + ), + turnsByActor: listFact( + [...actorCounts] + .sort(([left], [right]) => left.localeCompare(right)) + .map(([actor, spanIds]) => ({ actor, turns: spanIds.length, spanIds })), + promptSpans.map((span) => span.span_id), + ), + finalMessages: listFact(finalMessages, finalMessages.map((entry) => entry.spanId)), + changedFiles: { + ...listFact(files, files.flatMap((entry) => entry.spanIds)), + ...(truncatedInputs > 0 + ? { + partial: + `${truncatedInputs} contributing tool span(s) had truncated input; ` + + 'paths named after the cut are not in this list', + } + : {}), + }, + firstRecordAt: firstSpan + ? { value: firstSpan.start_time, spanIds: [firstSpan.span_id], unavailable: null } + : { value: null, spanIds: [], unavailable: 'no span in this trace carries a parseable start time' }, + lastRecordAt: lastSpan + ? { value: lastSpan.end_time, spanIds: [lastSpan.span_id], unavailable: null } + : { value: null, spanIds: [], unavailable: 'no span in this trace carries a parseable end time' }, + tokenTotal: tokenSpan + ? { + value: tokenSpan.attributes[SESSION_TOKEN_TOTAL_ATTR] as number, + spanIds: [tokenSpan.span_id], + unavailable: null, + } + : { + value: null, + spanIds: [], + unavailable: + `no span carries ${SESSION_TOKEN_TOTAL_ATTR}; the per-turn llm.turn deltas in this trace ` + + 'are a different quantity and summing them would not be the harness total', + }, + } +} + +/** The facts sheet for a set of spans, as one report. */ +export function buildSessionFactsReport( + spans: readonly OtlpSpan[], + options: { harness?: string | null; generatedAt?: string } = {}, +): SessionFactsReport { + return { + schemaVersion: 1, + kind: 'traces.session_facts_report', + generatedAt: options.generatedAt ?? new Date().toISOString(), + harness: options.harness ?? null, + spanCount: spans.length, + sessions: computeSessionFacts(spans), + } +} + +function factLine(label: string, fact: SessionFact, rendered: string): string { + if (fact.value === null) return ` ${label}: unavailable — ${fact.unavailable}` + const partial = fact.partial ? ` (partial: ${fact.partial})` : '' + return ` ${label}: ${rendered}${partial}` +} + +/** The short readable form: the same facts, one screen, no span-id lists. */ +export function renderSessionFacts(report: SessionFactsReport): string { + const lines: string[] = [ + `session facts — ${report.sessions.length} session(s), ${report.spanCount} span(s), deterministic, $0`, + ] + for (const facts of report.sessions) { + lines.push('', `${facts.sessionId ?? facts.traceId} [${facts.harness ?? 'unknown harness'}] ${facts.spanCount} spans`) + lines.push( + factLine( + 'tool calls', + facts.toolCalls, + `${facts.toolCalls.value} (${facts.synthesizedToolSpans.value ?? 0} synthesized span(s) excluded)`, + ), + ) + lines.push( + factLine( + 'subagents', + facts.subagents, + `${facts.subagents.value?.length ?? 0}${ + facts.subagents.value?.length + ? `: ${facts.subagents.value.map((entry) => entry.taskName ?? '').join(', ')}` + : '' + }`, + ), + ) + lines.push(factLine('human turns', facts.humanTurns, String(facts.humanTurns.value?.length ?? 0))) + lines.push( + factLine( + 'turns by actor', + facts.turnsByActor, + facts.turnsByActor.value?.length + ? facts.turnsByActor.value.map((entry) => `${entry.actor} ${entry.turns}`).join(', ') + : 'none', + ), + ) + lines.push(factLine('changed files', facts.changedFiles, String(facts.changedFiles.value?.length ?? 0))) + lines.push(factLine('first record', facts.firstRecordAt, String(facts.firstRecordAt.value))) + lines.push(factLine('last record', facts.lastRecordAt, String(facts.lastRecordAt.value))) + lines.push(factLine('token total', facts.tokenTotal, String(facts.tokenTotal.value))) + if ((facts.unreadRecords.value ?? 0) > 0) { + lines.push(` unread records: ${facts.unreadRecords.value} (the facts above are computed from the rest)`) + } + for (const message of facts.finalMessages.value ?? []) { + lines.push(` final message [${message.task ?? 'session agent'}]: ${message.text.split('\n')[0]?.slice(0, 160) ?? ''}`) + } + } + return `${lines.join('\n')}\n` +} + +/** + * The byte ceiling one prepared-context string stays under. + * + * `DEFAULT_TRACE_ANALYST_BUDGETS.perCallByteCeiling` is 150,000 — the size one + * trace-tool result may return. Prepared context is read by the same model in + * the same turn as those results, so the sheet claims a fifth of that ceiling + * and leaves the rest for the tool calls the model still has to make. + */ +export const PREPARED_CONTEXT_BYTE_CEILING = 30_000 + +/** + * Render the sheet as bounded prepared context. + * + * Prepared context is not evidence: it is a head start. The rendering therefore + * keeps every field's span ids so that each fact can be checked against the raw + * spans, and says out loud that the sheet itself is not citable. + * + * The result is guaranteed to be at most `byteCeiling` bytes. Fields are shed + * from the largest list downward, and the shed is reported inside the context + * rather than silently applied. + */ +export function renderSessionFactsContext( + report: SessionFactsReport, + options: { byteCeiling?: number } = {}, +): string { + const ceiling = options.byteCeiling ?? PREPARED_CONTEXT_BYTE_CEILING + const header = + 'SESSION FACTS (deterministic, computed from the spans, no model call). ' + + 'Each fact names the span ids it came from: cite those spans, never this sheet. ' + + 'A null value carries the reason the spans cannot support it — do not replace it with a guess.' + let sessions = report.sessions.map(compactFacts) + const encode = (dropped: string[]): string => + `${header}\n${JSON.stringify({ sessions, ...(dropped.length ? { omitted_fields: dropped } : {}) })}` + + const dropped: string[] = [] + // Shed in reverse usefulness order: text first, then the long id lists, then + // whole list fields. The counts, which answer most questions, are last to go. + const sheds: Array<[string, (facts: CompactFacts) => void]> = [ + ['human_turn_text', (facts) => { for (const turn of facts.human_turns) delete turn.text }], + ['final_message_text', (facts) => { for (const message of facts.final_messages) delete message.text }], + ['tool_call_span_ids', (facts) => { facts.tool_calls.span_ids = [] }], + ['tool_calls_by_name', (facts) => { delete facts.tool_calls_by_name }], + ['changed_files', (facts) => { facts.changed_files = { count: facts.changed_files.count } }], + ['turns_by_actor', (facts) => { delete facts.turns_by_actor }], + ['subagents', (facts) => { facts.subagents = { count: facts.subagents.count } }], + ['final_messages', (facts) => { facts.final_messages = [] }], + ['human_turns', (facts) => { facts.human_turns = [] }], + ] + for (const [name, shed] of sheds) { + if (Buffer.byteLength(encode(dropped)) <= ceiling) break + for (const facts of sessions) shed(facts) + dropped.push(name) + } + let text = encode(dropped) + // A trace set too large to describe at all keeps the sessions it can fit. + while (Buffer.byteLength(text) > ceiling && sessions.length > 1) { + sessions = sessions.slice(0, Math.max(1, Math.floor(sessions.length / 2))) + text = encode([...dropped, `sessions_beyond_${sessions.length}`]) + } + if (Buffer.byteLength(text) <= ceiling) return text + // Below one session's smallest record the sheet says so and supplies nothing, + // rather than returning a fragment that reads like a complete sheet. The + // ceiling is a guarantee, so an unusable one produces no context at all. + const refusal = `${header}\n${JSON.stringify({ sessions: [], omitted_fields: ['all: the byte ceiling is below one session record'] })}` + return Buffer.byteLength(refusal) <= ceiling ? refusal : '' +} + +interface CompactFacts { + session_id: string | null + trace_id: string + spans: number + tool_calls: { count: number | null; synthesized_excluded: number | null; span_ids: readonly string[] } + tool_calls_by_name?: Readonly> | null + subagents: { count: number; entries?: readonly unknown[] } + human_turns: Array<{ at: string; span_id: string; text?: string }> + turns_by_actor?: readonly unknown[] | null + final_messages: Array<{ task: string | null; at: string; span_id: string; text?: string }> + changed_files: { count: number; paths?: readonly unknown[]; partial?: string } + first_record_at: unknown + last_record_at: unknown + token_total: unknown +} + +function factJson(fact: SessionFact): unknown { + return fact.value === null + ? { value: null, unavailable: fact.unavailable } + : { value: fact.value, span_ids: fact.spanIds } +} + +function compactFacts(facts: SessionFacts): CompactFacts { + return { + session_id: facts.sessionId, + trace_id: facts.traceId, + spans: facts.spanCount, + tool_calls: { + count: facts.toolCalls.value, + synthesized_excluded: facts.synthesizedToolSpans.value, + span_ids: facts.toolCalls.spanIds, + }, + tool_calls_by_name: facts.toolCallsByName.value, + subagents: { + count: facts.subagents.value?.length ?? 0, + entries: facts.subagents.value?.map((entry) => ({ + task_name: entry.taskName, + ...(entry.taskNameUnavailable ? { task_name_unavailable: entry.taskNameUnavailable } : {}), + started_at: entry.startedAt, + status: entry.status, + span_ids: entry.spanIds, + })), + }, + human_turns: (facts.humanTurns.value ?? []).map((turn) => ({ + at: turn.at, + span_id: turn.spanId, + text: turn.text, + })), + turns_by_actor: facts.turnsByActor.value?.map((entry) => ({ + actor: entry.actor, + turns: entry.turns, + span_ids: entry.spanIds, + })), + final_messages: (facts.finalMessages.value ?? []).map((message) => ({ + task: message.task, + at: message.at, + span_id: message.spanId, + text: message.text, + })), + changed_files: { + count: facts.changedFiles.value?.length ?? 0, + paths: facts.changedFiles.value?.map((file) => ({ + path: file.path, + operations: file.operations, + span_ids: file.spanIds, + })), + ...(facts.changedFiles.partial ? { partial: facts.changedFiles.partial } : {}), + }, + first_record_at: factJson(facts.firstRecordAt), + last_record_at: factJson(facts.lastRecordAt), + token_total: factJson(facts.tokenTotal), + } +} + +/** + * The prepared-context string for one set of spans, or undefined when there is + * nothing to prepare. Analyst definitions install this through + * `TraceAnalystDefinition.prepareContext`, which runs before the model does. + */ +export function sessionFactsContext( + spans: readonly OtlpSpan[], + options: { byteCeiling?: number; generatedAt?: string } = {}, +): string | undefined { + if (spans.length === 0) return undefined + const report = buildSessionFactsReport(spans, { + ...(options.generatedAt ? { generatedAt: options.generatedAt } : {}), + }) + if (report.sessions.length === 0) return undefined + return renderSessionFactsContext(report, options) +} + +/** + * Version suffix stamped on a definition that receives the sheet. + * + * `createTraceAnalyst` records `prepare_context: "version-bound"` in an + * analyst's exact-run identity, which is a contract: a definition whose + * prepared context changed is a different analyst. Wrapping therefore bumps the + * version instead of quietly changing what the same version does. + */ +export const SESSION_FACTS_VERSION_SUFFIX = 'session-facts.1' + +/** + * Supply the sheet to every definition as prepared context, before the model + * runs. A definition that already prepares context keeps it; the sheet is + * appended after it, bounded by its own ceiling, so the pair stays inside the + * documented `perCallByteCeiling` of 150,000 bytes. + * + * The sheet is computed once for the whole set: it is deterministic, so every + * definition receives the identical text. + */ +export function withSessionFactsContext( + definitions: readonly TraceAnalystDefinition[], + spans: readonly OtlpSpan[], + options: { byteCeiling?: number } = {}, +): TraceAnalystDefinition[] { + const sheet = sessionFactsContext(spans, options) + if (sheet === undefined) return [...definitions] + return definitions.map((definition) => ({ + ...definition, + version: `${definition.version}+${SESSION_FACTS_VERSION_SUFFIX}`, + prepareContext: async (store, context) => { + const own = await definition.prepareContext?.(store, context) + return own ? `${own}\n\n${sheet}` : sheet + }, + })) +} diff --git a/tests/ask.test.ts b/tests/ask.test.ts new file mode 100644 index 0000000..8d23d30 --- /dev/null +++ b/tests/ask.test.ts @@ -0,0 +1,506 @@ +import { mkdtemp, readFile, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as sleep } from 'node:timers/promises' +import type { + TraceAnalysisEngine, + TraceAnalysisEngineRequest, + TraceAnalysisEngineResult, +} from '@tangle-network/agent-eval/analyst' +import { describe, expect, it } from 'vitest' +import { + loadTraceQuestionsFile, + MAX_TRACE_QUESTION_CHARS, + normalizeTraceQuestions, + renderTraceQuestionInstructions, + runTraceQuestions, + TRACE_QUESTION_PREVIEW_HEAD_CHARS, + traceCitationsInText, + writeTraceQuestionsArtifacts, +} from '../src/ask.js' +import { answerSchemaErrors, assertAnswerSchema, parseJsonAnswer } from '../src/answer-schema.js' +import { type OtlpSpan, span } from '../src/otlp.js' + +const TRACE = 'trace-ask' +const COMMAND = 'git status --short && pnpm test' + +/** A small synthetic session: one prompt, one shell command, one reply. */ +function fixtureSpans(): OtlpSpan[] { + const base = Date.parse('2026-02-01T10:00:00.000Z') + const at = (seconds: number) => new Date(base + seconds * 1000).toISOString() + return [ + span({ + traceId: TRACE, + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: at(0), + endTime: at(30), + service: 'synthetic', + extra: { 'session.id': 'session-ask' }, + }), + span({ + traceId: TRACE, + spanId: 'prompt-1', + parentSpanId: 'root', + name: 'user.prompt', + kind: 'CHAIN', + startTime: at(1), + service: 'synthetic', + content: 'Run the unit tests and tell me whether they pass.', + }), + span({ + traceId: TRACE, + spanId: 'tool-1', + parentSpanId: 'root', + name: 'tool.exec_command', + kind: 'TOOL', + startTime: at(5), + endTime: at(9), + service: 'synthetic', + tool: 'exec_command', + extra: { 'input.value': JSON.stringify({ cmd: COMMAND }), 'output.value': '12 passed' }, + }), + span({ + traceId: TRACE, + spanId: 'reply-1', + parentSpanId: 'root', + name: 'llm.turn', + kind: 'LLM', + startTime: at(10), + service: 'synthetic', + content: 'All 12 unit tests passed.', + }), + ] +} + +type Script = (request: TraceAnalysisEngineRequest) => Promise> + +/** A fake engine that runs a script per request and records every request. */ +function scriptedEngine(script: Script, executionConfig: Record = {}) { + const requests: TraceAnalysisEngineRequest[] = [] + const engine: TraceAnalysisEngine = { + id: 'scripted-test-engine', + description: 'Runs a test script instead of a model.', + model: 'test-model', + version: '1.0.0', + executionConfig, + async analyze(request) { + requests.push(request) + const result = await script(request) + return { + answer: '', + findings: [], + trajectory: [], + modelCalls: 1, + toolCalls: 0, + runtime: {}, + ...result, + } + }, + } + return { engine, requests } +} + +function tool(request: TraceAnalysisEngineRequest, name: string) { + const found = request.tools.find((candidate) => candidate.name === name) + if (!found) throw new Error(`tool ${name} was not offered`) + return found +} + +function questionId(request: TraceAnalysisEngineRequest): string { + return request.analystId.replace(/^ask\./, '') +} + +function deferred() { + let resolve!: () => void + const promise = new Promise((done) => { resolve = done }) + return { promise, resolve } +} + +async function until(condition: () => boolean, timeoutMs = 5_000): Promise { + const deadline = Date.now() + timeoutMs + while (!condition()) { + if (Date.now() > deadline) throw new Error('condition not reached') + await sleep(5) + } +} + +describe('runTraceQuestions', () => { + it('keeps each answer verbatim, verifies its citations, and writes both artifacts', async () => { + const { engine } = scriptedEngine(async (request) => { + // The engine reads through the real tool handler it was given. + const viewed = await tool(request, 'viewSpans').handler({ trace_id: TRACE, span_ids: ['tool-1'] }) as { + spans: Array<{ span_id: string }> + } + expect(viewed.spans.map((entry) => entry.span_id)).toEqual(['tool-1']) + return { + answer: `The session ran \`${COMMAND}\` once (trace://${TRACE}/span/tool-1).`, + findings: [{ + severity: 'info', + claim: 'The unit tests ran once.', + confidence: 0.9, + evidence: [{ uri: `trace://${TRACE}/span/tool-1`, excerpt: 'git status --short' }], + }], + toolCalls: 1, + } + }) + const dir = await mkdtemp(join(tmpdir(), 'traces-ask-test-')) + const result = await runTraceQuestions({ + questions: [{ id: 'commands', question: 'Which shell commands ran?' }], + spans: fixtureSpans(), + engine, + harness: 'synthetic', + otlpOutPath: join(dir, 'traces.otlp.jsonl'), + }) + + expect(result.ok).toBe(true) + const [answer] = result.questions + expect(answer!.status).toBe('answered') + expect(answer!.answer).toBe(`The session ran \`${COMMAND}\` once (trace://${TRACE}/span/tool-1).`) + expect(answer!.citations).toEqual([{ uri: `trace://${TRACE}/span/tool-1`, traceId: TRACE, spanId: 'tool-1', resolved: true }]) + expect(answer!.findings).toHaveLength(1) + expect(answer!.modelCalls).toBe(1) + expect(answer!.toolCalls).toBe(1) + expect(answer!.model).toBe('test-model') + expect(result.traces).toEqual([expect.objectContaining({ traceId: TRACE, sessionId: 'session-ask', spanCount: 4 })]) + + const artifacts = await writeTraceQuestionsArtifacts(result, dir) + const saved = JSON.parse(await readFile(artifacts.result, 'utf8')) as typeof result + expect(saved.questions[0]!.answer).toBe(answer!.answer) + expect(saved).not.toHaveProperty('report') + expect(await readFile(artifacts.report, 'utf8')).toContain(answer!.answer!) + }) + + it('never runs more questions at once than the concurrency limit', async () => { + const gates = new Map>() + const started: string[] = [] + let running = 0 + let peak = 0 + const { engine } = scriptedEngine(async (request) => { + const id = questionId(request) + started.push(id) + running += 1 + peak = Math.max(peak, running) + const gate = deferred() + gates.set(id, gate) + await gate.promise + running -= 1 + return { answer: `answer ${id}` } + }) + const run = runTraceQuestions({ + questions: ['one', 'two', 'three', 'four', 'five'].map((question) => ({ question: `Question ${question}?` })), + spans: fixtureSpans(), + engine, + concurrency: 3, + }) + + await until(() => started.length === 3) + await sleep(50) + // Two questions wait until a running one finishes. + expect(started).toEqual(['q1', 'q2', 'q3']) + gates.get('q2')!.resolve() + await until(() => started.length === 4) + await sleep(50) + expect(started).toHaveLength(4) + for (const id of ['q1', 'q3']) gates.get(id)!.resolve() + await until(() => started.length === 5) + for (const id of ['q4', 'q5']) gates.get(id)!.resolve() + + const result = await run + expect(peak).toBe(3) + expect(result.totals.peakConcurrency).toBe(3) + expect(result.questions.map((answer) => answer.answer)).toEqual(['q1', 'q2', 'q3', 'q4', 'q5'].map((id) => `answer ${id}`)) + }) + + it('overlaps questions, so wall time stays well below the sum of question times', async () => { + const { engine } = scriptedEngine(async (request) => { + await sleep(200) + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 6 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 6, + }) + expect(result.ok).toBe(true) + // Six 200 ms questions: serial would take 1.2 s or more. + expect(result.totals.questionTimeMs).toBeGreaterThanOrEqual(1_200) + expect(result.totals.wallTimeMs).toBeLessThan(result.totals.questionTimeMs / 2) + expect(result.totals.peakConcurrency).toBe(6) + expect(result.report).toMatch(/Wall time \d+\.\d s against \d+\.\d s of question time \(peak 6 at once\)/) + }) + + it('reports questions the shared ledger refused while the others keep their answers', async () => { + const { engine } = scriptedEngine(async (request) => { + // One metered call per question: reserve $0.40, settle at $0.30. + const paid = await request.costLedger.runPaidCall({ + channel: 'analyst', + phase: request.costPhase, + actor: request.analystId, + ...(request.costTags ? { tags: request.costTags } : {}), + maximumCharge: { externallyEnforcedMaximumUsd: 0.4 }, + execute: async () => { + await sleep(20) + return 'ok' + }, + receipt: () => ({ model: 'test-model', inputTokens: 100, outputTokens: 50, actualCostUsd: 0.3 }), + }) + if (!paid.succeeded) throw paid.error + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 5 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 5, + budgetUsd: 1, + }) + + // $1 admits three $0.30 calls; after them, $0.10 is left and a $0.40 reservation is refused. + const answered = result.questions.filter((answer) => answer.status === 'answered') + const refused = result.questions.filter((answer) => answer.failure?.kind === 'budget-refused') + expect(answered).toHaveLength(3) + expect(refused).toHaveLength(2) + for (const answer of refused) { + expect(answer.failure!.message).toContain('would exceed ceiling 1') + expect(answer.answer).toBeNull() + } + for (const answer of answered) expect(answer.usage?.cost).toEqual({ kind: 'observed', usd: 0.3 }) + expect(result.ok).toBe(false) + expect(result.totals.cost.usd).toBeCloseTo(0.9, 10) + expect(result.totals.cost.usd!).toBeLessThanOrEqual(1) + expect(result.report).toContain('failed: budget-refused') + }) + + it('refuses to start when the budget cannot cover one model call', async () => { + const { engine, requests } = scriptedEngine(async () => ({ answer: 'never' }), { + pricing: { inputUsdPerMillion: 1.25, outputUsdPerMillion: 10 }, + max_output_tokens: 8_192, + max_reasoning_tokens: 32_768, + }) + await expect(runTraceQuestions({ + questions: [{ question: 'Anything?' }], + spans: fixtureSpans(), + engine, + budgetUsd: 0.1, + })).rejects.toThrow(/below one model call's reservation of at least \$0\.41/) + expect(requests).toHaveLength(0) + }) + + it('warns when the budget serializes concurrent questions', async () => { + const { engine } = scriptedEngine(async () => ({ answer: 'fine' }), { + pricing: { inputUsdPerMillion: 1.25, outputUsdPerMillion: 10 }, + max_output_tokens: 8_192, + max_reasoning_tokens: 32_768, + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 4 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 4, + budgetUsd: 1, + }) + expect(result.warnings).toEqual([expect.stringContaining('covers at most 2 concurrent model call(s)')]) + expect(result.report).toContain('> Warning: budget $1.00 covers at most 2') + }) + + it('keeps the other answers when one engine call throws, and marks the run failed', async () => { + const { engine } = scriptedEngine(async (request) => { + if (questionId(request) === 'q3') throw new Error('synthetic engine failure') + return { answer: `answer ${questionId(request)}` } + }) + const result = await runTraceQuestions({ + questions: Array.from({ length: 5 }, (_, index) => ({ question: `Question ${index + 1}?` })), + spans: fixtureSpans(), + engine, + concurrency: 2, + }) + expect(result.ok).toBe(false) + expect(result.totals).toMatchObject({ answered: 4, failed: 1, modelCalls: null }) + const failed = result.questions.find((answer) => answer.id === 'q3')! + expect(failed.failure).toEqual({ kind: 'error', message: 'Error: synthetic engine failure' }) + expect(failed.modelCalls).toBeNull() + expect(result.questions.filter((answer) => answer.id !== 'q3').map((answer) => answer.answer)) + .toEqual(['answer q1', 'answer q2', 'answer q4', 'answer q5']) + }) + + it('fails an answer that cites a span the trace does not hold', async () => { + const { engine } = scriptedEngine(async () => ({ + answer: `Tests ran at trace://${TRACE}/span/tool-1 and trace://${TRACE}/span/invented-span.`, + })) + const result = await runTraceQuestions({ + questions: [{ question: 'When did the tests run?' }], + spans: fixtureSpans(), + engine, + }) + const [answer] = result.questions + expect(answer!.status).toBe('failed') + expect(answer!.failure?.kind).toBe('unresolved-citations') + expect(answer!.answer).toContain('invented-span') + expect(answer!.citations.map((citation) => [citation.spanId, citation.resolved])).toEqual([ + ['tool-1', true], + ['invented-span', false], + ]) + expect(result.report).toContain(`**Unresolved:** trace://${TRACE}/span/invented-span`) + }) + + it('fails an empty answer', async () => { + const { engine } = scriptedEngine(async () => ({ answer: ' ' })) + const result = await runTraceQuestions({ questions: [{ question: 'Anything?' }], spans: fixtureSpans(), engine }) + expect(result.questions[0]!.failure?.kind).toBe('no-answer') + expect(result.questions[0]!.answer).toBeNull() + }) + + it('shows the evidence gate\'s rejection reasons for each question', async () => { + const { engine } = scriptedEngine(async () => ({ + answer: `One command ran (trace://${TRACE}/span/tool-1).`, + findings: [{ + severity: 'low', + claim: 'The session deleted the build directory.', + confidence: 0.8, + evidence: [{ uri: `trace://${TRACE}/span/tool-1`, excerpt: 'rm -rf build directory' }], + }], + })) + const result = await runTraceQuestions({ questions: [{ question: 'What ran?' }], spans: fixtureSpans(), engine }) + const [answer] = result.questions + expect(answer!.status).toBe('answered') + expect(answer!.findings).toHaveLength(0) + expect(answer!.rejectedFindings).toEqual({ 'excerpt is not present in the cited span content': 1 }) + expect(result.report).toContain( + '**Rejected by the evidence gate:** 1 finding(s) rejected: excerpt is not present in the cited span content ×1', + ) + }) + + it('parses a schema-bound answer and fails one that breaks the schema', async () => { + const schema = { + type: 'object', + properties: { commands: { type: 'integer' }, first: { type: 'string' } }, + required: ['commands', 'first'], + additionalProperties: false, + } + const { engine } = scriptedEngine(async (request) => ({ + answer: questionId(request) === 'good' + ? '```json\n{"commands": 1, "first": "git status --short"}\n```' + : '{"commands": "one"}', + })) + const result = await runTraceQuestions({ + questions: [ + { id: 'good', question: 'How many commands ran, and which was first?', answerSchema: schema }, + { id: 'bad', question: 'How many commands ran, and which was first?', answerSchema: schema }, + ], + spans: fixtureSpans(), + engine, + }) + const [good, bad] = result.questions + expect(good!.status).toBe('answered') + expect(good!.parsedAnswer).toEqual({ commands: 1, first: 'git status --short' }) + expect(bad!.failure?.kind).toBe('invalid-answer') + expect(bad!.failure?.message).toBe('$.first: required property is missing; $.commands: expected integer, got string') + }) +}) + +describe('question layout for the DSPy preview', () => { + it('keeps the question whole and the rules inside the first 500 characters of the instructions', async () => { + const longest = 'x'.repeat(MAX_TRACE_QUESTION_CHARS) + const { engine, requests } = scriptedEngine(async () => ({ answer: 'noted' })) + await runTraceQuestions({ + questions: [ + { id: 'plain', question: longest }, + { + id: 'typed', + question: 'How many commands ran?', + instructions: 'Count only shell commands.', + answerSchema: { type: 'object', properties: { count: { type: 'integer' } } }, + }, + ], + spans: fixtureSpans(), + engine, + }) + expect(requests).toHaveLength(2) + for (const request of requests) { + expect(request.question.length).toBeLessThanOrEqual(1_000) + const head = request.instructions.slice(0, TRACE_QUESTION_PREVIEW_HEAD_CHARS) + expect(head).toContain('TRACES ASK RULES') + expect(head).toContain('5. If the trace does not record a fact, say "not in trace".') + // The trace list reaches the model after the rules, as parseable JSON. + const context = request.instructions.split('PREPARED CONTEXT:\n')[1]!.split('\n\n')[0]! + expect(JSON.parse(context)).toMatchObject({ traces: [{ trace_id: TRACE, session_id: 'session-ask', spans: 4 }], omitted_traces: 0 }) + } + const plain = requests.find((request) => request.analystId === 'ask.plain')! + expect(plain.question.endsWith(`QUESTION: ${longest}`)).toBe(true) + const typed = requests.find((request) => request.analystId === 'ask.typed')! + expect(typed.instructions.slice(0, TRACE_QUESTION_PREVIEW_HEAD_CHARS)).toContain('6. The answer is one JSON value matching ANSWER SCHEMA') + expect(typed.instructions).toContain('QUESTION GUIDANCE:\nCount only shell commands.') + expect(renderTraceQuestionInstructions({})).not.toContain('ANSWER SCHEMA') + }) + + it('rejects a question the preview would cut, before any engine call', () => { + expect(() => normalizeTraceQuestions([{ question: 'y'.repeat(MAX_TRACE_QUESTION_CHARS + 1) }])) + .toThrow(/the limit is \d+ so the model sees it whole/) + }) +}) + +describe('question input', () => { + it('reads strings and objects from a questions file and rejects unknown keys', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-ask-questions-')) + const good = join(dir, 'questions.json') + await writeFile(good, JSON.stringify({ + questions: [ + 'Which commands failed?', + { id: 'last-turn', question: 'What was the last human turn?', answerSchema: { type: 'string' } }, + ], + }), 'utf8') + expect(await loadTraceQuestionsFile(good)).toEqual([ + { question: 'Which commands failed?' }, + { id: 'last-turn', question: 'What was the last human turn?', answerSchema: { type: 'string' } }, + ]) + + const typo = join(dir, 'typo.json') + await writeFile(typo, JSON.stringify([{ question: 'Anything?', schema: { type: 'string' } }]), 'utf8') + await expect(loadTraceQuestionsFile(typo)).rejects.toThrow('entry 1 has unknown key "schema"') + }) + + it('assigns default IDs and rejects duplicates', () => { + expect(normalizeTraceQuestions([{ question: ' a? ' }, { question: 'b?' }]).map((entry) => [entry.id, entry.question])) + .toEqual([['q1', 'a?'], ['q2', 'b?']]) + expect(() => normalizeTraceQuestions([{ id: 'x', question: 'a?' }, { id: 'x', question: 'b?' }])) + .toThrow('duplicate question ID "x"') + expect(() => normalizeTraceQuestions([])).toThrow('at least one question') + }) + + it('extracts trace citations without trailing punctuation', () => { + expect(traceCitationsInText('See trace://t%2F1/span/abc. Also (trace://t2/span/def), trace://t2/span/def.')) + .toEqual([ + { uri: 'trace://t%2F1/span/abc', traceId: 't/1', spanId: 'abc' }, + { uri: 'trace://t2/span/def', traceId: 't2', spanId: 'def' }, + ]) + }) +}) + +describe('answer schemas', () => { + it('rejects keywords it cannot check instead of ignoring them', () => { + expect(() => assertAnswerSchema({ type: 'string', pattern: '^PR' })).toThrow('unsupported JSON Schema keyword "pattern"') + expect(() => assertAnswerSchema({ type: 'object', properties: { n: { minimum: 1 } } })) + .toThrow('answerSchema.properties.n: unsupported JSON Schema keyword "minimum"') + }) + + it('checks types, required properties, enums, and array items', () => { + const schema = { + type: 'array', + items: { + type: 'object', + properties: { pr: { type: 'integer' }, state: { enum: ['merged', 'closed'] } }, + required: ['pr'], + }, + } + expect(answerSchemaErrors([{ pr: 3, state: 'merged' }], schema)).toEqual([]) + expect(answerSchemaErrors([{ pr: 3.5 }, { state: 'open' }], schema)).toEqual([ + '$[0].pr: expected integer, got number', + '$[1].pr: required property is missing', + '$[1].state: expected one of "merged", "closed"', + ]) + expect(parseJsonAnswer('The answer is 3')).toEqual({ ok: false, error: 'answer is not one JSON value' }) + }) +}) diff --git a/tests/cli.test.ts b/tests/cli.test.ts index 2bc8a70..4a0da04 100644 --- a/tests/cli.test.ts +++ b/tests/cli.test.ts @@ -1231,6 +1231,158 @@ describe('traces analyze --llm failure surfacing', () => { }, 60_000) }) +describe('traces analyze external analyzer failure', () => { + it('writes the report, then exits 1 when a requested analyzer fails', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-external-fail-')) + const input = join(dir, 'spans.openinference.jsonl') + const report = join(dir, 'report.md') + await writeFile(input, serializeSpans([ + span({ + traceId: 'trace-external-fail', + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: '2026-01-01T00:00:00.000Z', + service: 'claude-code', + extra: { 'session.id': 'session-external-fail' }, + }), + ]), 'utf8') + + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'analyze', input, + '--format', 'openinference', + // `false` is an installed command that exits 1 whatever its arguments. + '--analyzer', 'false', + '--out', report, + ], { + cwd: process.cwd(), + env: { ...process.env, NO_COLOR: '1', FORCE_COLOR: '' }, + maxBuffer: 10 * 1024 * 1024, + timeout: 60_000, + }).then( + () => { + throw new Error('analyze exited 0 although its requested analyzer failed') + }, + (error: Error & { code?: number; stdout?: string; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stderr).toContain('1 of 1 external analyzer(s) failed') + expect(failure.stderr).toContain('false: exit 1') + const reportText = await readFile(report, 'utf8') + expect(reportText).toContain('### false (report)') + expect(reportText).toContain('failed: exit 1') + }, 60_000) +}) + +describe('traces ask', () => { + async function writeSessionFile(dir: string): Promise { + const input = join(dir, 'spans.openinference.jsonl') + await writeFile(input, serializeSpans([ + span({ + traceId: 'trace-ask-cli', + spanId: 'root', + name: 'session', + kind: 'AGENT', + startTime: '2026-01-01T00:00:00.000Z', + service: 'codex', + extra: { 'session.id': 'session-ask-cli' }, + }), + span({ + traceId: 'trace-ask-cli', + spanId: 'turn', + parentSpanId: 'root', + name: 'llm.turn', + kind: 'LLM', + startTime: '2026-01-01T00:00:01.000Z', + service: 'codex', + step: 1, + content: 'Running the checks now.', + }), + ]), 'utf8') + return input + } + + it('needs at least one question', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-ask-none-')) + const input = await writeSessionFile(dir) + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'ask', input, '--format', 'openinference', + ], { cwd: process.cwd(), env: { ...process.env, NO_COLOR: '1' }, timeout: 30_000 }).then( + () => { + throw new Error('ask exited 0 without a question') + }, + (error: Error & { code?: number; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stderr).toContain('ask needs --question') + }, 30_000) + + it('writes every answer and failure, then exits 1 when the engine dies at startup', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-cli-ask-fail-')) + const input = await writeSessionFile(dir) + const out = join(dir, 'ask') + const questions = join(dir, 'questions.json') + await writeFile(questions, JSON.stringify([ + { id: 'checks', question: 'Which checks ran?' }, + { id: 'turns', question: 'How many assistant turns were there?', answerSchema: { type: 'integer' } }, + ]), 'utf8') + const fakeBridge = join(dir, 'failing-bridge.sh') + const bridgeReason = 'DSPY-BRIDGE-FAILURE: ValueError: synthetic startup failure' + await writeFile(fakeBridge, `#!/bin/sh\necho "${bridgeReason}" >&2\nexit 1\n`, { mode: 0o755 }) + + const failure = await execFileAsync(process.execPath, [ + '--import', 'tsx', 'src/cli.ts', 'ask', input, + '--format', 'openinference', + '--questions', questions, + '--question', 'Did the session finish?', + '--budget', '5', + '--concurrency', '2', + '--dir', out, + ], { + cwd: process.cwd(), + env: { + ...process.env, + NO_COLOR: '1', + FORCE_COLOR: '', + TANGLE_API_KEY: 'test-key-never-sent-upstream', + OPENAI_API_KEY: '', + OPENAI_BASE_URL: '', + TRACES_PYTHON: fakeBridge, + }, + maxBuffer: 10 * 1024 * 1024, + timeout: 90_000, + }).then( + () => { + throw new Error('ask exited 0 although every question failed') + }, + (error: Error & { code?: number; stdout?: string; stderr?: string }) => error, + ) + expect(failure.code).toBe(1) + expect(failure.stdout).toContain('# traces ask') + expect(failure.stdout).toContain('**0 answered, 3 failed.**') + expect(failure.stderr).toContain('3 of 3 question(s) failed') + + const answers = JSON.parse(await readFile(join(out, 'answers.json'), 'utf8')) as { + kind: string + ok: boolean + budgetUsd: number + questionBudgetUsd: number + questions: Array<{ id: string; status: string; failure?: { kind: string; message: string } }> + } + expect(answers.kind).toBe('traces.ask') + expect(answers.ok).toBe(false) + expect(answers.budgetUsd).toBe(5) + expect(answers.questionBudgetUsd).toBe(1) + expect(answers.questions.map((answer) => answer.id)).toEqual(['checks', 'turns', 'q3']) + for (const answer of answers.questions) { + expect(answer.status).toBe('failed') + expect(answer.failure?.kind).toBe('error') + expect(answer.failure?.message).toContain(bridgeReason) + } + expect(await readFile(join(out, 'report.md'), 'utf8')).toContain('failed: error') + }, 90_000) +}) + describe('traces bundle + bundle-view', () => { it('assembles the full view, then projects the writer view over the same session', async () => { const root = await mkdtemp(join(tmpdir(), 'traces-cli-bundle-')) diff --git a/tests/finding-rejections.test.ts b/tests/finding-rejections.test.ts new file mode 100644 index 0000000..7221f26 --- /dev/null +++ b/tests/finding-rejections.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from 'vitest' +import { + createFindingRejectionTally, + findingRejection, + findingRejectionDetail, + formatFindingRejections, +} from '../src/finding-rejections.js' + +describe('finding rejections', () => { + it('reads the analyst and the actionable reason from registry and direct log events', () => { + expect(findingRejection('[failure-mode] finding rejected: unresolved evidence', { + uri: 'trace://t/span/s', + reason: 'excerpt is not present in the cited span content', + })).toEqual({ analystId: 'failure-mode', reason: 'excerpt is not present in the cited span content' }) + expect(findingRejection('finding rejected: insufficient evidence citations', { required: 2, distinct: 1 })) + .toEqual({ reason: 'insufficient evidence citations' }) + expect(findingRejection('[failure-mode] trace analyst failure-mode completed', {})).toBeUndefined() + }) + + it('formats one stderr detail with the cited URI and citation counts', () => { + expect(findingRejectionDetail('[q1] finding rejected: unresolved evidence', { + uri: 'trace://t/span/s', + reason: 'trace span does not exist', + })).toBe('trace span does not exist; uri trace://t/span/s') + expect(findingRejectionDetail('finding rejected: insufficient evidence citations', { required: 2, distinct: 1 })) + .toBe('insufficient evidence citations; 1 of 2 required distinct citation(s)') + expect(findingRejectionDetail('[analyst] ok failure-mode', {})).toBeUndefined() + }) + + it('counts per analyst, with a default for unprefixed events', () => { + const tally = createFindingRejectionTally('q1') + tally.record('finding rejected: unresolved evidence', { reason: 'trace span does not exist' }) + tally.record('finding rejected: unresolved evidence', { reason: 'trace span does not exist' }) + tally.record('[improvement] finding rejected: schema failure', {}) + tally.record('trace analyst engine started', {}) + expect(tally.counts()).toEqual({ + q1: { 'trace span does not exist': 2 }, + improvement: { 'schema failure': 1 }, + }) + expect(formatFindingRejections(tally.counts().q1)).toBe('2 finding(s) rejected: trace span does not exist ×2') + expect(formatFindingRejections(undefined)).toBe('') + }) +}) diff --git a/tests/improvement.test.ts b/tests/improvement.test.ts index 5ca0d49..5a26556 100644 --- a/tests/improvement.test.ts +++ b/tests/improvement.test.ts @@ -551,6 +551,50 @@ describe('agentic failure surfacing', () => { expect(result.report).toContain('DSPY-BRIDGE-FAILURE: ValueError: analyze input must contain exactly') }) + it('counts evidence-gate rejections per analyst and names the reason in the report', async () => { + const fabricating: TraceAnalysisEngine = { + id: 'fabricating-test-engine', + description: 'Submits one finding whose excerpt the cited span does not contain.', + model: 'test-model', + version: '1.0.0', + executionConfig: {}, + analyze: async () => ({ + answer: 'The agent retried a failing command.', + findings: [{ + severity: 'medium', + claim: 'The agent retried npm test three times without changing anything.', + confidence: 0.8, + evidence: [ + { uri: 'trace://trace-improve/span/tool-0', excerpt: 'this text is not in the span' }, + { uri: 'trace://trace-improve/span/tool-1' }, + ], + }], + trajectory: [], + modelCalls: 1, + toolCalls: 0, + runtime: {}, + }), + } + const logged: string[] = [] + const result = await runTraceInvestigation({ + spans: fixtureSpans(), + harness: 'synthetic', + engine: fabricating, + generatedAt: '2026-01-01T00:00:00.000Z', + log: (msg) => logged.push(msg), + }) + + expect(result.findingRejections?.['failure-mode']).toEqual({ + 'excerpt is not present in the cited span content': 1, + }) + expect(result.findings.some((finding) => finding.analyst_id === 'failure-mode')).toBe(false) + expect(result.report).toMatch( + /\| `failure-mode` \| ok \| 0 \| \d+ms \| 1 finding\(s\) rejected: excerpt is not present in the cited span content ×1 \|/, + ) + // The caller's log still receives every event the tally counted. + expect(logged).toContain('[failure-mode] finding rejected: unresolved evidence') + }) + it('leaves agenticPerAnalyst unset for a deterministic-only run', async () => { const result = await runTraceInvestigation({ spans: fixtureSpans(), @@ -558,6 +602,7 @@ describe('agentic failure surfacing', () => { generatedAt: '2026-01-01T00:00:00.000Z', }) expect(result.agenticPerAnalyst).toBeUndefined() + expect(result.findingRejections).toBeUndefined() expect(totalAgenticFailureMessage(result.agenticPerAnalyst)).toBeUndefined() }) diff --git a/tests/report.test.ts b/tests/report.test.ts index fb0affd..eb57036 100644 --- a/tests/report.test.ts +++ b/tests/report.test.ts @@ -701,4 +701,33 @@ describe('analystRunDetail', () => { usage, })).toBe('—') }) + + it('names the evidence gate\'s rejection reasons for an analyst that returned no findings', () => { + const summary = { + analyst_id: 'failure-mode', + status: 'ok', + findings_count: 0, + latency_ms: 5, + usage, + } as const + expect(analystRunDetail(summary, { + 'excerpt is not present in the cited span content': 2, + 'trace span does not exist': 3, + })).toBe( + '5 finding(s) rejected: trace span does not exist ×3; excerpt is not present in the cited span content ×2', + ) + expect(analystRunDetail(summary, {})).toBe('—') + + const result = emptyResult() + result.per_analyst.push(summary) + const report = renderReport(result, { + harness: 'codex', + sessionCount: 1, + spanCount: 10, + otlpPath: '/tmp/spans.openinference.jsonl', + execution: EMPTY_EXECUTION, + findingRejections: { 'failure-mode': { 'insufficient evidence citations': 1 } }, + }) + expect(report).toContain('| `failure-mode` | ok | 0 | 5ms | 1 finding(s) rejected: insufficient evidence citations ×1 |') + }) }) diff --git a/tests/session-facts-fixture.ts b/tests/session-facts-fixture.ts new file mode 100644 index 0000000..419b828 --- /dev/null +++ b/tests/session-facts-fixture.ts @@ -0,0 +1,198 @@ +/** + * A synthetic Codex rollout whose facts are known by hand. + * + * Every record here is invented for this test. The shape follows the rollout + * format the Codex adapter parses — `session_meta`, `turn_context`, + * `response_item` calls and outputs, `event_msg` token counts and subagent + * activity — so the facts sheet is exercised through the real adapter rather + * than against spans a test built directly. + */ + +import { mkdtemp, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { CodexAdapter } from '../src/adapters/codex.js' +import type { OtlpSpan } from '../src/otlp.js' + +export const FIXTURE_SESSION_ID = 'facts-fixture-session' +export const FIXTURE_AGENT_PATH = '/root/uploader_audit' +export const FIXTURE_THREAD_ID = 'facts-fixture-child' + +/** What the fixture's human typed, verbatim, in order. */ +export const FIXTURE_HUMAN_TURNS = [ + 'add a retry to the uploader and keep the tests green', + 'also make the backoff configurable', +] as const + +export const FIXTURE_FINAL_ASSISTANT = 'Backoff is configurable now and the suite is green.' +export const FIXTURE_FINAL_SUBAGENT_TEXT = + 'Message Type: FINAL_ANSWER\nThe uploader retries three times and never swallows a 4xx.' + +/** Paths the fixture's patches touch, and how. */ +export const FIXTURE_CHANGED_FILES = [ + { path: '/fixture/src/retry.ts', operations: ['add'] }, + { path: '/fixture/src/upload.ts', operations: ['update'] }, +] as const + +function at(second: number): string { + return new Date(Date.UTC(2026, 8, 9, 12, 0, second)).toISOString() +} + +export const FIXTURE_FIRST_RECORD_AT = at(0) + +/** + * Records of the fixture session. + * + * `extraTools` appends N further `exec_command` calls with padded arguments, to + * build a session whose serialized spans exceed the trace tools' byte ceiling + * while the hand-written facts stay exactly derivable. + */ +export function fixtureRecords(extraTools = 0): unknown[] { + const records: unknown[] = [ + { type: 'session_meta', timestamp: at(0), payload: { id: FIXTURE_SESSION_ID, cwd: '/fixture' } }, + { type: 'turn_context', timestamp: at(1), payload: { model: 'gpt-5.4-codex' } }, + { + type: 'response_item', + timestamp: at(2), + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: FIXTURE_HUMAN_TURNS[0] }] }, + }, + { + type: 'event_msg', + timestamp: at(3), + payload: { + type: 'token_count', + info: { last_token_usage: { input_tokens: 900, output_tokens: 80 }, total_token_usage: { input_tokens: 900, output_tokens: 80, total_tokens: 980 } }, + }, + }, + // A plain shell call. + { + type: 'response_item', + timestamp: at(4), + payload: { type: 'function_call', call_id: 'call-shell', name: 'exec_command', arguments: JSON.stringify({ cmd: 'ls src' }) }, + }, + { type: 'response_item', timestamp: at(5), payload: { type: 'function_call_output', call_id: 'call-shell', output: JSON.stringify({ exit_code: 0, output: 'upload.ts' }) } }, + // A verification call: the adapter names it `exec_command.verify`, which is + // the category the measured analyst kept subtracting by hand. + { + type: 'response_item', + timestamp: at(6), + payload: { type: 'function_call', call_id: 'call-verify', name: 'exec_command', arguments: JSON.stringify({ cmd: 'pnpm test' }) }, + }, + { type: 'response_item', timestamp: at(7), payload: { type: 'function_call_output', call_id: 'call-verify', output: JSON.stringify({ exit_code: 0 }) } }, + // A patch naming one updated and one added file. + { + type: 'response_item', + timestamp: at(8), + payload: { + type: 'function_call', + call_id: 'call-patch', + name: 'apply_patch', + arguments: JSON.stringify({ + input: [ + '*** Begin Patch', + '*** Update File: /fixture/src/upload.ts', + '@@', + '- await send(body)', + '+ await withRetry(() => send(body))', + '*** Add File: /fixture/src/retry.ts', + '+export const withRetry = async () => {}', + '*** End Patch', + ].join('\n'), + }), + }, + }, + { type: 'response_item', timestamp: at(9), payload: { type: 'function_call_output', call_id: 'call-patch', output: JSON.stringify({ exit_code: 0 }) } }, + // A subagent spawn, with the task name the adapter promotes to an attribute. + { + type: 'response_item', + timestamp: at(10), + payload: { + type: 'function_call', + call_id: 'call-spawn', + name: 'spawn_agent', + arguments: JSON.stringify({ task_name: FIXTURE_AGENT_PATH, message: 'audit the uploader' }), + }, + }, + { type: 'response_item', timestamp: at(11), payload: { type: 'function_call_output', call_id: 'call-spawn', output: JSON.stringify({ task_name: FIXTURE_AGENT_PATH }) } }, + // The lifecycle events that make the adapter synthesize a `tool.Agent` span. + { + type: 'event_msg', + timestamp: at(12), + payload: { + type: 'sub_agent_activity', + event_id: 'call-spawn', + kind: 'started', + agent_thread_id: FIXTURE_THREAD_ID, + agent_path: FIXTURE_AGENT_PATH, + occurred_at_ms: Date.parse(at(12)), + }, + }, + { + type: 'response_item', + timestamp: at(13), + payload: { type: 'agent_message', author: FIXTURE_AGENT_PATH, recipient: 'root', content: FIXTURE_FINAL_SUBAGENT_TEXT }, + }, + { + type: 'event_msg', + timestamp: at(14), + payload: { + type: 'sub_agent_activity', + event_id: 'call-spawn', + kind: 'completed', + agent_thread_id: FIXTURE_THREAD_ID, + agent_path: FIXTURE_AGENT_PATH, + occurred_at_ms: Date.parse(at(14)), + }, + }, + { + type: 'response_item', + timestamp: at(15), + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'Retry added; running the suite.' }] }, + }, + { + type: 'response_item', + timestamp: at(16), + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: FIXTURE_HUMAN_TURNS[1] }] }, + }, + ] + for (let index = 0; index < extraTools; index += 1) { + const callId = `call-bulk-${index}` + records.push({ + type: 'response_item', + timestamp: at(17), + payload: { + type: 'function_call', + call_id: callId, + name: 'exec_command', + arguments: JSON.stringify({ cmd: `rg --files-with-matches token ${'padding/'.repeat(40)}${index}` }), + }, + }) + records.push({ + type: 'response_item', + timestamp: at(17), + payload: { type: 'function_call_output', call_id: callId, output: JSON.stringify({ exit_code: 0, output: 'x'.repeat(400) }) }, + }) + } + records.push({ + type: 'response_item', + timestamp: at(18), + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: FIXTURE_FINAL_ASSISTANT }] }, + }) + return records +} + +export const FIXTURE_LAST_RECORD_AT = at(18) + +/** Parse the fixture through the real Codex adapter. */ +export async function fixtureSpans(extraTools = 0): Promise { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-fixture-')) + const path = join(dir, `${FIXTURE_SESSION_ID}.jsonl`) + await writeFile(path, `${fixtureRecords(extraTools).map((record) => JSON.stringify(record)).join('\n')}\n`, 'utf8') + return new CodexAdapter().parse({ + harness: 'codex', + sessionId: FIXTURE_SESSION_ID, + path, + cwd: '/fixture', + mtimeMs: 0, + }) +} diff --git a/tests/session-facts.test.ts b/tests/session-facts.test.ts new file mode 100644 index 0000000..edeb3e5 --- /dev/null +++ b/tests/session-facts.test.ts @@ -0,0 +1,356 @@ +/** + * The deterministic session-facts sheet, checked against a fixture whose gold + * is written by hand. + * + * The measured failure this replaces: over twelve private audit sessions, the + * model-backed analyst scored a deterministic mean of 0.389 while the same + * facts, extracted mechanically from the spans the run already wrote, scored + * 0.858. Nothing was missing from the spans — no tool returned an exact count, + * so the model added up a capped name histogram and guessed. These tests hold + * the extraction to exactness on a session it can be checked against by eye. + */ + +import { execFile } from 'node:child_process' +import { mkdtemp, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { promisify } from 'node:util' +import { describe, expect, it } from 'vitest' +import { + DEFAULT_TRACE_ANALYST_KINDS, + defineTraceAnalyst, + type TraceAnalysisEngine, + type TraceAnalysisEngineRequest, + type TraceAnalystDefinition, +} from '@tangle-network/agent-eval/analyst' +import type { TraceAnalysisStore } from '@tangle-network/agent-eval/traces' +import { + buildSessionFactsReport, + computeSessionFacts, + PREPARED_CONTEXT_BYTE_CEILING, + renderSessionFacts, + renderSessionFactsContext, + SESSION_FACTS_VERSION_SUFFIX, + SESSION_TOKEN_TOTAL_ATTR, + sessionFactsContext, + withSessionFactsContext, +} from '../src/session-facts.js' +import { analyzeSpans } from '../src/analyze.js' +import { serializeSpans } from '../src/otlp.js' +import { + FIXTURE_AGENT_PATH, + FIXTURE_CHANGED_FILES, + FIXTURE_FINAL_ASSISTANT, + FIXTURE_FINAL_SUBAGENT_TEXT, + FIXTURE_FIRST_RECORD_AT, + FIXTURE_HUMAN_TURNS, + FIXTURE_LAST_RECORD_AT, + FIXTURE_SESSION_ID, + fixtureRecords, + fixtureSpans, +} from './session-facts-fixture.js' + +const execFileAsync = promisify(execFile) + +/** The trace-tool byte ceiling every benchmark session exceeded. */ +const VIEW_TRACE_BYTE_CEILING = 150_000 + +describe('session facts', () => { + it('states every field exactly, against a hand-written gold', async () => { + const spans = await fixtureSpans() + const [facts, ...rest] = computeSessionFacts(spans) + expect(rest).toEqual([]) + expect(facts!.sessionId).toBe(FIXTURE_SESSION_ID) + expect(facts!.harness).toBe('codex') + + // Four calls the agent made: two shells, one patch, one spawn. + expect(facts!.toolCalls.value).toBe(4) + expect(facts!.toolCalls.unavailable).toBeNull() + expect(facts!.toolCallsByName.value).toEqual({ + 'apply_patch': 1, + 'exec_command': 1, + 'exec_command.verify': 1, + 'spawn_agent': 1, + }) + + expect(facts!.subagents.value).toHaveLength(1) + expect(facts!.subagents.value?.[0]?.taskName).toBe(FIXTURE_AGENT_PATH) + expect(facts!.subagents.value?.[0]?.taskNameUnavailable).toBeNull() + + expect(facts!.humanTurns.value?.map((turn) => turn.text)).toEqual([...FIXTURE_HUMAN_TURNS]) + expect(facts!.humanTurns.value?.map((turn) => turn.actor)).toEqual(['human', 'human']) + expect(facts!.humanTurns.value?.map((turn) => turn.at)).toEqual([ + '2026-09-09T12:00:02.000Z', + '2026-09-09T12:00:16.000Z', + ]) + expect(facts!.turnsByActor.value).toEqual([ + { actor: 'human', turns: 2, spanIds: facts!.humanTurns.value!.map((turn) => turn.spanId) }, + ]) + + // The session's own last word, and the subagent's, kept apart: both stream + // under one root, and picking "the last message" across both is how the + // measured run returned a subagent's report as the session's answer. + expect(facts!.finalMessages.value).toEqual([ + expect.objectContaining({ task: null, text: FIXTURE_FINAL_ASSISTANT }), + expect.objectContaining({ task: FIXTURE_AGENT_PATH, text: FIXTURE_FINAL_SUBAGENT_TEXT }), + ]) + + expect(facts!.changedFiles.value?.map((file) => ({ path: file.path, operations: file.operations }))) + .toEqual(FIXTURE_CHANGED_FILES.map((file) => ({ path: file.path, operations: [...file.operations] }))) + + expect(facts!.firstRecordAt.value).toBe(FIXTURE_FIRST_RECORD_AT) + expect(facts!.lastRecordAt.value).toBe(FIXTURE_LAST_RECORD_AT) + }) + + it('names a real span for every fact it states', async () => { + const spans = await fixtureSpans() + const known = new Set(spans.map((span) => span.span_id)) + const [facts] = computeSessionFacts(spans) + const cited = [ + ...facts!.toolCalls.spanIds, + ...facts!.subagents.spanIds, + ...facts!.humanTurns.spanIds, + ...facts!.finalMessages.spanIds, + ...facts!.changedFiles.spanIds, + ...facts!.firstRecordAt.spanIds, + ...facts!.lastRecordAt.spanIds, + ] + expect(cited.length).toBeGreaterThan(0) + for (const spanId of cited) expect(known.has(spanId)).toBe(true) + }) + + it('says why a fact the spans cannot support is null, instead of guessing', async () => { + const spans = await fixtureSpans() + const [facts] = computeSessionFacts(spans) + // The adapter reads Codex's cumulative `total_token_usage` only as a + // de-duplication signature, so no span carries the session total. Summing + // the per-turn deltas would answer a different question. + expect(facts!.tokenTotal.value).toBeNull() + expect(facts!.tokenTotal.unavailable).toContain(SESSION_TOKEN_TOTAL_ATTR) + expect(facts!.tokenTotal.spanIds).toEqual([]) + }) + + it('reports the harness token total once a span carries it', async () => { + const spans = await fixtureSpans() + const root = spans.find((span) => span.parent_span_id === null)! + root.attributes[SESSION_TOKEN_TOTAL_ATTR] = 17_025_686 + const [facts] = computeSessionFacts(spans) + expect(facts!.tokenTotal.value).toBe(17_025_686) + expect(facts!.tokenTotal.unavailable).toBeNull() + expect(facts!.tokenTotal.spanIds).toEqual([root.span_id]) + }) + + it('does not let a synthesized subagent span inflate the tool count', async () => { + const spans = await fixtureSpans() + const toolSpans = spans.filter((span) => span.attributes['openinference.span.kind'] === 'TOOL') + // The adapter gives a subagent's lifecycle span `kind: TOOL` and + // `tool.name: Agent`, so a plain span-kind count is high by exactly one here. + expect(toolSpans).toHaveLength(5) + expect(toolSpans.filter((span) => span.name === 'tool.Agent')).toHaveLength(1) + + const [facts] = computeSessionFacts(spans) + expect(facts!.toolCalls.value).toBe(4) + expect(facts!.synthesizedToolSpans.value).toBe(1) + expect(facts!.toolCallsByName.value).not.toHaveProperty('Agent') + expect(facts!.toolCalls.spanIds).not.toContain(facts!.synthesizedToolSpans.spanIds[0]) + }) + + it('stays exact on a session larger than the trace tools can return', async () => { + const spans = await fixtureSpans(400) + // `viewTrace` degrades to a ≤20-entry name histogram above this many bytes, + // which is the surface the measured analyst had to count from. + expect(Buffer.byteLength(serializeSpans(spans))).toBeGreaterThan(VIEW_TRACE_BYTE_CEILING) + + const [facts] = computeSessionFacts(spans) + expect(facts!.toolCalls.value).toBe(404) + expect(facts!.synthesizedToolSpans.value).toBe(1) + expect(facts!.toolCallsByName.value).toEqual({ + 'apply_patch': 1, + 'exec_command': 401, + 'exec_command.verify': 1, + 'spawn_agent': 1, + }) + expect(facts!.humanTurns.value).toHaveLength(2) + expect(facts!.finalMessages.value?.[0]?.text).toBe(FIXTURE_FINAL_ASSISTANT) + expect(facts!.changedFiles.value?.map((file) => file.path)) + .toEqual(FIXTURE_CHANGED_FILES.map((file) => file.path)) + }) + + it('renders a short readable form and a machine-readable report', async () => { + const spans = await fixtureSpans() + const report = buildSessionFactsReport(spans, { harness: 'codex', generatedAt: '2026-09-09T12:30:00.000Z' }) + expect(report.kind).toBe('traces.session_facts_report') + expect(report.sessions).toHaveLength(1) + expect(JSON.parse(JSON.stringify(report))).toMatchObject({ schemaVersion: 1, harness: 'codex' }) + + const text = renderSessionFacts(report) + expect(text).toContain('tool calls: 4 (1 synthesized span(s) excluded)') + expect(text).toContain(`subagents: 1: ${FIXTURE_AGENT_PATH}`) + expect(text).toContain('human turns: 2') + expect(text).toContain('token total: unavailable —') + }) +}) + +describe('session facts as prepared context', () => { + it('stays inside its byte bound and keeps the counts when it must shed', async () => { + const spans = await fixtureSpans(400) + const context = sessionFactsContext(spans)! + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(PREPARED_CONTEXT_BYTE_CEILING) + // Under the documented per-call ceiling the analyst tools work to. + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(VIEW_TRACE_BYTE_CEILING) + const body = JSON.parse(context.slice(context.indexOf('\n') + 1)) as { + sessions: Array<{ tool_calls: { count: number; synthesized_excluded: number } }> + omitted_fields?: string[] + } + expect(body.sessions[0]!.tool_calls.count).toBe(404) + expect(body.sessions[0]!.tool_calls.synthesized_excluded).toBe(1) + }) + + it('holds an arbitrarily tight ceiling, and reports every shed', async () => { + const spans = await fixtureSpans(400) + const report = buildSessionFactsReport(spans) + let previous = Number.POSITIVE_INFINITY + for (const ceiling of [30_000, 8_000, 2_000, 900, 400, 100]) { + const context = renderSessionFactsContext(report, { byteCeiling: ceiling }) + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(ceiling) + expect(Buffer.byteLength(context)).toBeLessThanOrEqual(previous) + previous = Buffer.byteLength(context) + if (context === '') continue + const body = JSON.parse(context.slice(context.indexOf('\n') + 1)) as { + sessions: Array<{ tool_calls?: { count: number } }> + omitted_fields?: string[] + } + // Whatever it shed, it says so, and the tool-call count is the last fact + // to go: it answers the question the bounded tools cannot. + if (ceiling < 8_000) expect(body.omitted_fields?.length).toBeGreaterThan(0) + if (body.sessions.length > 0) expect(body.sessions[0]!.tool_calls?.count).toBe(404) + } + }) + + it('tells the reader the sheet is not citable', async () => { + const spans = await fixtureSpans() + const context = sessionFactsContext(spans)! + expect(context).toContain('cite those spans, never this sheet') + expect(context).toContain('no model call') + }) + + it('supplies the sheet to a built-in kind without dropping its own context', async () => { + const spans = await fixtureSpans() + const base: TraceAnalystDefinition = defineTraceAnalyst({ + id: 'fixture-kind', + description: 'a kind that already prepares context', + area: 'fixture', + version: '2.0.0', + instructions: 'Investigate.', + toolGroup: 'all', + prepareContext: () => 'OWN CONTEXT', + }) + const [wrapped] = withSessionFactsContext([base], spans) + expect(wrapped!.version).toBe(`2.0.0+${SESSION_FACTS_VERSION_SUFFIX}`) + const prepared = await wrapped!.prepareContext!({} as TraceAnalysisStore, { runId: 'r', correlationId: 'r:1' }) + expect(prepared!.startsWith('OWN CONTEXT\n\n')).toBe(true) + expect(prepared).toContain('SESSION FACTS') + expect(Buffer.byteLength(prepared!)).toBeLessThanOrEqual( + PREPARED_CONTEXT_BYTE_CEILING + Buffer.byteLength('OWN CONTEXT\n\n'), + ) + }) + + it('prepares nothing when there are no spans', () => { + expect(sessionFactsContext([])).toBeUndefined() + }) +}) + +describe('analyzeSpans', () => { + it('hands the sheet to the built-in kinds, and can be run without it', async () => { + const spans = await fixtureSpans() + const requests: TraceAnalysisEngineRequest[] = [] + const engine: TraceAnalysisEngine = { + id: 'session-facts-test-engine', + description: 'records the instructions it is given instead of calling a model', + model: 'test-model', + version: '1.0.0', + executionConfig: {}, + async analyze(request) { + requests.push(request) + return { answer: 'noted', findings: [], trajectory: [], modelCalls: 1, toolCalls: 0, runtime: {} } + }, + } + + await analyzeSpans(spans, { engine, agenticKinds: DEFAULT_TRACE_ANALYST_KINDS.slice(0, 1) }) + expect(requests).toHaveLength(1) + const prepared = requests[0]!.instructions.split('PREPARED CONTEXT:\n')[1]! + expect(prepared).toContain('SESSION FACTS') + expect(JSON.parse(prepared.slice(prepared.indexOf('\n') + 1).split('\n\n')[0]!)) + .toMatchObject({ sessions: [{ tool_calls: { count: 4, synthesized_excluded: 1 } }] }) + + requests.length = 0 + await analyzeSpans(spans, { + engine, + agenticKinds: DEFAULT_TRACE_ANALYST_KINDS.slice(0, 1), + sessionFactsContext: false, + }) + expect(requests[0]!.instructions).not.toContain('SESSION FACTS') + }) +}) + +describe('traces facts', () => { + const cli = (args: string[]) => + execFileAsync(process.execPath, ['--import', 'tsx', 'src/cli.ts', ...args], { cwd: process.cwd() }) + + it('prints the sheet as JSON with no model call', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-cli-')) + const otlp = join(dir, 'spans.otlp.jsonl') + await writeFile(otlp, serializeSpans(await fixtureSpans()), 'utf8') + + const { stdout } = await cli(['facts', '--otlp', otlp]) + const report = JSON.parse(stdout) as { + kind: string + sessions: Array<{ toolCalls: { value: number }; subagents: { value: Array<{ taskName: string }> } }> + } + expect(report.kind).toBe('traces.session_facts_report') + expect(report.sessions[0]!.toolCalls.value).toBe(4) + expect(report.sessions[0]!.subagents.value[0]!.taskName).toBe(FIXTURE_AGENT_PATH) + + const readable = await cli(['facts', '--otlp', otlp, '--format', 'text']) + expect(readable.stdout).toContain('tool calls: 4') + expect(readable.stdout).toContain('deterministic, $0') + }) + + it('exits non-zero when a session cannot be read', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-unreadable-')) + // A file that is not a session, and a path that does not exist. Neither may + // be reported as a session with no facts. + const garbage = join(dir, 'not-a-session.jsonl') + await writeFile(garbage, 'this is not a rollout\n', 'utf8') + await expect(cli(['facts', '--harness', 'codex', '--session', garbage])) + .rejects.toMatchObject({ code: 1 }) + await expect(cli(['facts', '--otlp', join(dir, 'absent.otlp.jsonl')])) + .rejects.toMatchObject({ code: 1 }) + }) + + it('keeps the facts and names the unread records when part of a session is corrupt', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-corrupt-')) + const session = join(dir, `${FIXTURE_SESSION_ID}.jsonl`) + const records = fixtureRecords().map((record) => JSON.stringify(record)) + // One unparseable line among readable ones: the facts still hold, and the + // sheet says how many records it could not read. + records.splice(3, 0, '{ not json') + await writeFile(session, `${records.join('\n')}\n`, 'utf8') + + const { stdout } = await cli(['facts', '--harness', 'codex', '--session', session]) + const report = JSON.parse(stdout) as { + sessions: Array<{ toolCalls: { value: number }; unreadRecords: { value: number } }> + } + expect(report.sessions[0]!.toolCalls.value).toBe(4) + expect(report.sessions[0]!.unreadRecords.value).toBe(1) + + const readable = await cli(['facts', '--harness', 'codex', '--session', session, '--format', 'text']) + expect(readable.stdout).toContain('unread records: 1') + }) + + it('rejects an unknown output format before reading anything', async () => { + const dir = await mkdtemp(join(tmpdir(), 'traces-facts-format-')) + await expect(cli(['facts', '--otlp', join(dir, 'unused.otlp.jsonl'), '--format', 'yaml'])) + .rejects.toMatchObject({ code: 1, stderr: expect.stringContaining('unknown facts format') }) + }) +})