Kitaru integration: optional trace source + replay verification backend (umbrella)

[dnth]

Summary

Integrate Kitaru as an optional tracegrad backend while keeping tracegrad fully useful as a standalone library.

This is the umbrella issue. It holds the product boundary and the roadmap consequences. The work is split into two children, because they have very different prerequisites:

• Phase 1 — Kitaru as a trace source: Phase 1: Kitaru as an optional trace source #8
• Phase 2 — Kitaru replay verification: Phase 2: Kitaru replay verification backend #9

This issue was rewritten on 2026-08-26 after grilling the original proposal against Kitaru 0.22.3. Four capabilities the original assumed do not exist in the API, and three of its instructions conflict with tracegrad's own contracts. The decisions are recorded as ADRs in docs/adr/ and the shared vocabulary in CONTEXT.md. The original text is in the edit history.

---

Product boundary

Observability providers
        │
        ▼
      Kitaru
 import / normalize / store
 cohorts / evaluators / replay
        │
        ▼
    tracegrad
 distill / attribute / aggregate
 synthesize / evidence-gate / review
        │
        ▼
 candidate artifact
        │
        ▼
      Kitaru
 replay / evaluate / compare
        │
        ▼
 Kitaru UI inspection
        │
        ▼
  tracegrad apply / reject

tracegrad stays the evidence-gated optimization engine. Kitaru owns trace normalization, execution, replay infrastructure, and execution visualization.

tracegrad owns

prompt/text artifact · artifact lineage and hash · deterministic distillation · attribution · failure-theme aggregation · evidence verification · edit synthesis · token-budget discipline · rejection memory · human approval · standalone JSONL source · post-deployment trends · verification summary and decision UX

Kitaru owns

provider-specific trace ingestion · trace normalization · session graph · execution storage · evaluators and judge execution · cohorts · replay · worker runtime · tool-history replay · execution and replay visualization

Do not build in tracegrad

Langfuse/LangSmith/Braintrust importers · a tracegrad-native replay engine · worker infrastructure · tool mocking or history engine · cohort storage · replay experiment orchestration · a full eval runner · a competing trace viewer

---

Product modes

1. Core-only tracegrad

uv tool install tracegrad

No Kitaru package, server, worker, Docker setup, or observability integration. The existing workflow is unchanged:

tracegrad run --traces batch.jsonl --manifest manifest.json
tracegrad apply
tracegrad trends

Core-only users keep JSONL ingestion, deterministic validation and distillation, attribution, aggregation, synthesis, evidence gates, token-budget discipline, rejection memory, human review and apply, and post-deployment trend comparison.

They do not get replay verification or execution inspection. tracegrad verify without a backend prints an actionable message and exits non-zero — a verify that exits 0 having done nothing reads as verified to every downstream step in a CI pipeline. "Does not block other workflows" means run, apply, and trends keep working, not that a command which did nothing should claim success.

2. tracegrad + Kitaru

uv tool install "tracegrad[kitaru]"

The extra installs a client. It does not install verification. See ADR 0001. Kitaru's replay re-executes the user's real agent code: ExperimentRunCreateRequest requires an agent_version_id, and AgentVersionResponse carries a RunSpec with a shell command. Kitaru's README is explicit that replays run "on workers in your environment: your virtualenv, your credentials, your network."

Phase 2 therefore has real prerequisites the user must satisfy:

1. a running Kitaru server (FastAPI + Postgres, via Docker),
2. a worker process running in the virtualenv where their agent code lives,
3. their agent instrumented with a Kitaru adapter and registered as an agent version.

tracegrad is installed via uv tool install, into an isolated virtualenv, which by construction is not the virtualenv the agent runs in. tracegrad cannot host a worker and does not try to.

---

Capability model

Capability	tracegrad core	tracegrad + Kitaru
JSONL traces	✅	✅
failure attribution	✅	✅
evidence-backed edits	✅	✅
human review/apply	✅	✅
post-deployment trends	✅	✅
provider trace normalization	❌	✅
full agent trajectory	limited to supplied data	✅ (in Kitaru, not in the Trace)
historical replay	❌	✅
pre-deployment verification	❌	✅
tool-history replay	❌	✅
execution/replay UI	❌	✅ (navigated by identifier)

---

Dependency boundary

[project.optional-dependencies]
kitaru = ["kitaru>=0.22,<0.23"]

Pinned tight deliberately. Kitaru is 0.22.3+dev, classified Development Status :: 3 - Alpha, default branch develop. Its api_models/v1/ layout signals an intent to version the wire format, but <0.23 is the only honest constraint today. Every breaking bump is a tracegrad release.

The base tracegrad package has no transitive Kitaru requirement. Kitaru SDK imports live under src/tracegrad/integrations/kitaru/. Using a Kitaru path without the extra returns an actionable install message; import tracegrad never fails.

Credentials and server URL come from Kitaru's own config — client/config.py:get_server_url() and CredentialStore, both written by kitaru login. tracegrad stores no Kitaru secrets. .tracegradrc holds only non-secret selection (cohort name, evaluation name), with CLI flags overriding.

---

Trends positioning

Keep trends in core. It serves two roles:

• core-only users — verification after deployment, via the next evaluated batch
• Kitaru users — post-deployment confirmation that replay-verified improvements persist in real traffic

without Kitaru:  proposal → deploy → next batch → trends
with Kitaru:     proposal → replay verify → deploy → trends

trends is demoted from the only pre-ship verification story to the standalone and post-deployment one. It is not removed.

---

Roadmap consequences

• Provider-specific observability importers are removed from the tracegrad roadmap.
• A tracegrad-native replay engine is removed from the roadmap.
• Kitaru remains one optional backend, not a mandatory core dependency.
• README's "Beyond system prompts" section is unaffected: multi-artifact editing stays out of scope here.

---

What was cut from the original proposal, and why

Cut	Reason
tracegrad[kitaru] "gains replay verification"	Replay runs the user's agent on a worker in their venv. The extra is a client. ADR 0001
trace.meta["trajectory"] = [...]	TraceMeta is extra="forbid" with one field. The example raises a ValidationError. Kitaru is the system of record for trajectories; re-fetch by session id when trajectory-aware attribution exists
--open and tracegrad inspect <session-id>	The API exposes no UI URL for an experiment, run, session, or replay — only get_investigation_review_url. The proposal itself forbids reconstructing undocumented routes. ADR 0008
TraceSource protocol and sources/ package	With ADR 0004 there is no second in-process source to abstract over, and ingest_traces already accepts Sequence[Trace]. ADR 0010
Ingestion refactor splitting parsing from validation	That seam already exists in ingest_traces' signature
Arbitrary numeric judge scores	schema.Score is bounded [0, 1]. Out-of-range and non-numeric evaluations drop by name rather than being rescaled against an assumed range. ADR 0003

Deferred, not rejected: reverse-templating for format prompts (ADR 0002), a declared --kitaru-score-range (ADR 0003), deriving a single-agent-version cohort version (ADR 0007), and an approve/apply rename (ADR 0009).

---

Architectural invariant

                    TRACEGRAD CORE
              evaluated static traces
                       │
                       ▼
        attribution / evidence / synthesis
                       │
                       ▼
                candidate artifact
                       │
          ┌────────────┴────────────┐
          │                         │
          │ no backend              │ optional backend
          ▼                         ▼
 deploy + next batch              Kitaru
          │                  replay / evaluate
          ▼                         │
       trends                       ▼
                              Kitaru UI
                         execution inspection
                                   │
                                   ▼
                              tracegrad
                           apply / reject

Do not turn tracegrad into an observability/replay platform.

Do not make Kitaru mandatory for tracegrad.

Do not duplicate Kitaru's execution UI inside tracegrad.

tracegrad alone = lightweight static-trace optimizer.

tracegrad + Kitaru = closed-loop optimizer with pre-deployment replay verification and visual execution inspection.