Phase 1: Kitaru as an optional trace source

[dnth]

Phase 1 of #7. Consume a frozen Kitaru cohort as a tracegrad trace batch.

Decisions behind this issue: ADR 0002, 0003, 0004, 0005, 0010. Vocabulary in CONTEXT.md.

Shape

--source kitaru is a fetch-and-map step that writes the JSONL the pipeline already reads (ADR 0004). The deterministic core does not learn Kitaru exists.

tracegrad run \
  --source kitaru \
  --kitaru-cohort support-production \
  --kitaru-evaluation quality \
  --manifest manifest.json

--traces and --source kitaru are mutually exclusive. The mapped Trace objects are written to .tracegrad/ as JSONL alongside the source fingerprint, before ingest. Re-runs read the snapshot; --refresh refetches.

All Kitaru SDK code lives under src/tracegrad/integrations/kitaru/. No TraceSource protocol, no sources/ package, no change to ingest.py (ADR 0010).

Prerequisites

A Kitaru server with imported sessions, and kitaru login. No worker and no registered agent version — those are Phase 2 (#9).

Cohort resolution

Prefer immutable cohort versions. Given only a cohort name: resolve the current version once, persist the immutable cohort_version_id, use it for the whole run, and include it in the source fingerprint. Never re-resolve latest within a run.

Mapping

tracegrad	Kitaru
trace_id	SessionResponse.id (UUID). number carried for display — reports print #4811, the durable key stays the UUID
input	input_text_selector resolved on the first root LLM node
output	output_text_selector resolved on the last root LLM node
judge.score	selected evaluation, mapped per ADR 0003
judge.rationale	EvaluationResponse.explanation, never synthesized
prompt_hash	hash of the extracted system prompt
meta.model	SessionNodeResponse.model on the root LLM nodes

A root LLM node is an llm_call with no subagent_call anywhere in its ancestry, following parent_index and secondary_parent_indexes — the graph is a DAG, so a node reachable from a subagent is not root even when one of its parents is (ADR 0005).

SessionResponse.inputs/.outputs are typed Any with no session-level selector, and are never stringified as a fallback. Tool outputs cannot become the final output, because only llm_call nodes are consulted.

Multi-turn is lossy: Trace.input is a StrictStr, so a session collapses to its first root input and last root output. Report it; do not hide it.

System prompt extraction

Resolve system_prompt_selector against node inputs for each root LLM node, collect unique values:

• exactly one → accept
• zero → drop system-prompt-unavailable
• more than one → drop multiple-system-prompts

Never infer or guess a system prompt.

Prompt lineage

Hash the recorded system prompt; the batch must be single-valued (ADR 0002).

This supports engine = "none" only. A format manifest renders differently per request, so hashing recorded prompts would give N partitions of size one and ingest would keep a batch of one. When the manifest declares engine = "format", refuse the run with a named error rather than proceeding into a batch that collapses.

Evaluation → judge

--kitaru-evaluation quality is required. Mapping per ADR 0003:

Kitaru	tracegrad
bool score, or passed flag	1.0 / 0.0
float in [0, 1]	as-is
float outside [0, 1]	drop judge-score-out-of-range
str / categorical	drop judge-score-unsupported
missing explanation	drop judge-rationale-missing

One evaluation name resolving to more than one evaluator_version across the cohort is ambiguous-evaluation: refuse rather than mix.

judge_fingerprint is derived from the resolved evaluator_name + evaluator_version and overrides the manifest value. A conflicting manifest value is an error, not a tiebreak. Drift detection is only worth having if it reads the thing that actually drifts.

Drop accounting

Two tables, never merged — a source drop means a Session could not become a Trace, a batch drop means a Trace is not part of this Batch. Merging them lets a mapping bug hide behind a legitimate partition.

Sessions selected:              1,240
Traces mapped:                  1,183
  system-prompt-unavailable        21
  judge-rationale-missing          14
  multiple-system-prompts           8
  output-unavailable                6
  judge-score-unavailable           5
  ambiguous-evaluation              3
In batch:                       1,140
  prompt-hash-partition            31
  rationale-below-quality-floor    12

Kebab-case throughout, matching the four reasons ingest.py already uses.

Source fingerprint

{
  "source": "kitaru",
  "cohort_id": "...",
  "cohort_version_id": "...",
  "evaluation_name": "quality",
  "evaluator_id": "...",
  "evaluator_version": 3,
  "agent_id": "...",
  "mapping_version": 1
}

Out of scope

• trace.meta["trajectory"] — TraceMeta is extra="forbid" with one field; the original example raises a ValidationError. Kitaru is the system of record for trajectories.
• Multi-artifact editing.
• Reverse-templating for format prompts (deferred, ADR 0002).
• --kitaru-score-range normalization (deferred, ADR 0003).

Definition of done

Core-only regression

• tracegrad installs and runs with no Kitaru dependency.
• Importing tracegrad does not import or require Kitaru.
• tracegrad run --traces ... works with no Kitaru present.
• Attribution, synthesis, apply, and trends still work standalone.
• A Kitaru path without the extra returns an actionable install message rather than an ImportError.

Mapping

• Kitaru is an optional extra only, pinned >=0.22,<0.23.
• A frozen cohort version maps deterministically to Trace objects, regardless of upstream observability provider.
• Root LLM nodes are identified by DAG traversal, including secondary_parent_indexes.
• A subagent's system prompt can never become the artifact.
• A tool output can never become Trace.output.
• tracegrad never guesses a missing system prompt.
• engine = "format" manifests are refused with a named error.
• Judge scores map per ADR 0003; out-of-range and non-numeric drop by name.
• judge_fingerprint is derived from the evaluator; a conflicting manifest value errors.
• Every dropped session has a named kebab-case reason, in the correct table.

Snapshot

• The mapped batch and source fingerprint are written to .tracegrad/ before ingest.
• A run is reproducible from the snapshot with the server unreachable.
• --refresh refetches and rewrites.
• The cohort version is resolved once per run and never re-resolved.

Unit tests

• session → Trace mapping
• single / missing / multiple system prompts
• selector resolution failures
• root vs subagent node classification, including DAG reachability via a secondary parent
• float / bool / passed-only / out-of-range / categorical evaluator mapping
• missing rationale
• ambiguous evaluator version
• format engine refusal