Phase 2: Kitaru replay verification backend

[dnth]

Phase 2 of #7. Verify an approved candidate against the same frozen cohort before it is written.

Decisions behind this issue: ADR 0001, 0006, 0007, 0008, 0009.

Blocked on #8 — verification reuses the cohort, evaluator, and agent metadata the Phase 1 source persists.

Prerequisites the user must satisfy

tracegrad submits verification; it does not host it (ADR 0001). Required on the user's side:

1. a running Kitaru server,
2. a worker process in the virtualenv where their agent code lives,
3. their agent instrumented with a Kitaru adapter and registered as an agent version.

tracegrad verify preflights before spending anything: probe the server, confirm a live worker claims this agent version (workers report last_seen_at), confirm the agent version and cohort version resolve. A replay experiment is paid and slow; "no worker is polling" should surface in milliseconds.

Preflight cannot check everything. Whether an adapter honours a system-prompt override is not exposed over the wire — AgentCapabilities carries only tools, MCP servers, and skills. That gap is closed by ADR 0006's post-replay assertion, not by prediction.

State machine

tracegrad run → proposal → human approves for testing → tracegrad verify
    → Kitaru replays the cohort → tracegrad summary
    → Kitaru inspection when needed → human decides → apply / reject

CLI

tracegrad verify --backend kitaru --run <tracegrad-run-id>

The happy path reuses the cohort, evaluator, and agent metadata persisted by the originating run.

VerificationBackend goes in ports.py — the orchestrator must hold a backend without becoming backend-aware, which is what that module exists for (ADR 0010). The Kitaru implementation lives under src/tracegrad/integrations/kitaru/.

Replay configuration

Change exactly one variable: the system prompt, passed as ReplayOverride.system_prompt. Hold fixed: baseline inputs, agent code and version, model, model params, cohort, evaluator version, recorded tool history.

Never mutate stored Kitaru sessions or production configuration.

Tool policy — hard invariant

Every tracegrad-created replay explicitly requests recorded history and fails on a miss:

HistoryConfig(scope=HistoryScope.COHORT_VERSION, on_miss=ToolPolicyOnMiss.FAIL)

Never rely on Kitaru's default tool policy, and never allow PASSTHROUGH. A novel call becomes a typed TOOL_HISTORY_MISS divergence rather than reaching a live production system.

Override scope — hard invariant

ReplayOverride.system_prompt is a single unscoped string, and the adapters disagree about where it lands:

• OpenAI Agents scopes it to the starting agent — if data.agent is not starting_agent: return model_data (replay.py:227). Subagents keep their own instructions.
• LangGraph applies it in _model_request with no starting-agent check (langchain.py:101). Every model call gets the candidate, subagents included.

So on a multi-agent LangGraph app, a naive submission moves two variables and reports one. After each replay, fetch the result session's nodes and assert:

• every root LLM node carries the candidate prompt, and
• every non-root LLM node carries what its baseline counterpart carried.

A session failing either is OVERRIDE_SCOPE_DIVERGENCE and is reported incomparable — not improved, not regressed (ADR 0006).

Cohort constraint

Every session in the cohort version must share one agent_version_id. ExperimentRunCreateRequest.agent_version_id applies to the whole run, while evaluate_baselines=True scores the stored baselines under whatever version each recorded — so a mixed cohort silently replays sessions under code they never ran on.

A mixed cohort is refused, with the version breakdown and session count per version (ADR 0007). This will refuse some real cohorts; a report that says "the prompt caused this" when the agent code also moved is worse.

Baseline and candidate always use the same evaluator version. A comparison against stale stored judge output from a different evaluator version is marked incomparable.

Results

Aggregates come from /api/v1/ui/experiment-runs/{id}/evaluation-aggregates (EvaluationStats: count, mean, min, max, pass_rate) so tracegrad's headline numbers match what the Kitaru UI shows for the same run. Per-session detail comes from listing the run's replays and their evaluations — the aggregate endpoint returns only the 50 most recent replays.

/api/v1/ui/ is a UI-support namespace, not an obvious third-party contract. This is the most likely thing to move under us; the <0.23 pin contains it. Say so in a comment at the call site.

Session comparison stays deliberately simple: fail→pass improved, pass→fail regressed, higher score improved, lower regressed, equal unchanged. Do not claim statistical significance.

class VerificationResult(BaseModel):
    status: Literal["completed", "partial", "failed"]
    baseline_count: int
    candidate_count: int
    baseline_mean_score: float | None
    candidate_mean_score: float | None
    baseline_pass_rate: float | None
    candidate_pass_rate: float | None
    improved_sessions: list[str]
    regressed_sessions: list[str]
    unchanged_sessions: list[str]
    diverged_sessions: list[Divergence]   # TOOL_HISTORY_MISS | OVERRIDE_SCOPE_DIVERGENCE
    replay_failures: list[ReplayFailure]
    cohort_version_id: str
    agent_version_id: str
    evaluator_version: str
    baseline_prompt_hash: str
    candidate_prompt_hash: str
    verification_fingerprint: str
    experiment_run_id: str

Report

tracegrad verification
────────────────────────────────
Cohort: support-production/week-34
Sessions: 487

                    Baseline   Candidate
Mean score             0.842       0.901
Pass rate              91.2%       95.7%

Improved                  38
Regressed                  4
Unchanged                441
Diverged                   4

Regressions
#4811   1.00 → 0.00   session 0f3a…c19d
#5102   0.88 → 0.61   session 7b21…4e08

Divergence
#5110   TOOL_HISTORY_MISS  search_account(...)

Experiment run: 41d9…8b7c
Kitaru: http://localhost:8000
Verdict: REVIEW

tracegrad never prints SHIP, and never applies or reverts on the strength of this report.

Inspection

The API exposes no UI URL for an experiment, run, session, or replay — client/dashboard_urls.py has only get_investigation_review_url. So inspection hands off the supported dashboard base plus real identifiers, and --open / tracegrad inspect are not built (ADR 0008). Identifiers are persisted, so wiring a URL helper in later is small.

• File an upstream request for experiment / session / compare URL helpers.

Apply gate

With a backend configured, apply refuses unless a persisted verification exists whose candidate_prompt_hash equals the hash of what is about to be written. --force overrides. Matching on the hash rather than the run id is what makes the gate real: verify, hand-edit, and the gate correctly notices the text was never verified. Core-only users are unaffected (ADR 0009).

Persistence and resume

.tracegrad/verification/<verification-id>.json

Persist the tracegrad run and proposal ids, Kitaru experiment and experiment-run ids, cohort version, evaluator version, agent version, baseline and candidate prompt hashes, tool policy, per-session result ids, summary metrics, and fingerprint.

Persist the Kitaru run id immediately after creation. An interrupted verification resumes and watches the existing experiment run rather than creating a duplicate.

Out of scope

• --open, tracegrad inspect (ADR 0008)
• an approve / apply rename (deferred, ADR 0009)
• deriving a single-agent-version cohort version via remove_session_ids (deferred, ADR 0007)
• any backend other than Kitaru

Definition of done

• tracegrad verify with no backend prints an actionable message and exits non-zero, without blocking run / apply / trends.
• Preflight checks server, live worker for the agent version, agent version, and cohort version before spending.
• A candidate is replayed against the same frozen cohort before it can be applied.
• Only the system prompt changes; model, params, agent version, cohort, and evaluator version are held fixed.
• Every replay explicitly sets HistoryConfig(scope=COHORT_VERSION, on_miss=FAIL).
• No passthrough tool policy is reachable through the tracegrad path.
• Post-replay assertion confirms the override landed on root LLM nodes only; violations are OVERRIDE_SCOPE_DIVERGENCE and incomparable.
• A tool-history miss is TOOL_HISTORY_MISS and incomparable.
• A mixed-agent-version cohort is refused with a per-version breakdown.
• Baseline and candidate use the same evaluator version.
• Improved / regressed / unchanged / diverged sessions are surfaced individually.
• Headline aggregates match the Kitaru UI for the same experiment run.
• Verification state is persisted and an interrupted run resumes without duplicating the experiment.
• apply is gated on a hash-matching verification; --force overrides.
• tracegrad builds no competing execution viewer.
• Proposal approval and application stay in tracegrad, not the Kitaru UI.