feat(feedback): eval sampling — down-thumbs feed AgentCore Evaluations (response-feedback PR-4)

[philmerrell]

docs/specs/response-feedback.md §11 PR-4, off develop (independent of #1146, the implicit-signals PR).

Why

The spec's thesis: feedback is a sampler, not a metric. A down-thumb marks the turn worth spending judge tokens on. The evaluations spike proved EvaluationClient.run() works end to end on dev today; what it lacked was an affordable input queue. This is that queue, and the first standing quality measurement the cost work has ever had.

Opt-in, and why

FEEDBACK_EVAL_SAMPLING_ENABLED defaults off (CDK_FEEDBACK_EVAL_SAMPLING_ENABLED=true to enable). The judge is an AWS-managed evaluator that reads the sampled conversation's spans — the system prompt and every user message. The spike's §2 says to make that decision explicitly per environment rather than let it happen as a side effect, and the feedback spec's §8 puts conversation content behind a scope. Flipping the flag is that decision. Everything else here (the queue keys, the read surfaces, the IAM grant) is wired and inert.

What

• Queue with no new index: thumb rows now carry GSI1PK = FEEDBACK#down|up, GSI1SK = updatedAt on the existing UserTimestampIndex. Recent down-thumbs across the fleet are one query; flipping to up leaves the queue.
• Sampler (apis/shared/feedback_eval/sampler.py): routes by reason code per spec §6 — wrong → Correctness + Faithfulness, instructions → InstructionFollowing, length → Conciseness, other/none → Helpfulness. tool_failed gets no judge: it is ops, corroborated against the call's tool census on the C# row. outdated gets no judge yet (KB-freshness join is a follow-up). Judge is a one-method protocol; AgentCoreJudge is the SDK adapter, lazily imported, sending the runtime session id (sid-<sha256>) the chat proxy already pins, over AGENTCORE_RUNTIME_LOG_GROUP.
• Content-free verdict on the F# row: per-evaluator value, rating, n, tokens, plus toolFailureCorroborated. The evaluator's explanation quotes the conversation, so it is dropped at summarisation, refused by the storage write, and added to the content-policy denylist. (The evaluator's label is stored as rating — label is already a denylisted path on other row families and the denylist is by name.)
• Admin routes (/admin/feedback/…, scope admin.costs): GET /evaluations lists the queue with verdicts, newest first, content-free by projection; POST /evaluations/run?limit= judges up to 50 not-yet-judged thumbs in a background task and returns 202 (the SDK waits on span ingestion — minutes). 404 while the flag is off.
• Profile: feedback.evaluations = {judged, byEvaluator: {name: {n, mean}}, toolFailuresReported, toolFailuresCorroborated}; a line under the Feedback tile.
• CDK: feedbackEvalSampling.enabled (opt-in, only the literal true), AGENTCORE_RUNTIME_LOG_GROUP threaded from the inference construct's runtimeLogGroupName, and three task-role statements: logs:StartQuery on the runtime log group and aws/spans, logs:GetQueryResults/StopQuery (not resource-scopable), bedrock-agentcore:Evaluate/GetEvaluator/ListEvaluators (built-in evaluators are AWS-owned). No Action:*.

What this does not change

• Nothing runs per turn. The batch is admin-triggered; a schedule can follow once a week of verdicts shows the token spend is worth it.
• No SPA button for the run yet — it is an admin API call (Swagger or curl); the queue and verdicts do show on the session profile. The author surfaces (PR-5), the report-dialog hatch, abandonment, and the outdated join remain unbuilt.
• The existing thumbs, retry link, and (in feat(feedback): implicit signals — copy and continue (response-feedback PR-2) #1146) implicit rows are untouched except for the two queue keys.

Tests

• Sampler: routing table incl. unknown codes; summarise keeps numbers/ratings and drops explanation; corroboration from the census; verdict shapes; batch over a real moto table with a fake judge (judged / no-judge / already-judged / one failure, second pass finds only the retry); the adapter refuses a missing log group and sends the runtime session id (SDK module faked at import).
• Storage: down-thumbs land on the queue and leave it on an up-thumb; verdict write refuses prose, converts floats, requires the row; the admin queue reader is content-free and newest first with a seeded stray displayText and a nested explanation both stripped.
• Routes: queue 200 with verdicts and pending; run 404 off / 202 on with the injected judge reaching the batch; limit bounded.
• Profile: judged aggregates per evaluator and corroboration counts; null when nothing judged. Architecture test's admin-scope registry updated for the new router.
• Infra: npm run build clean; 827 passed across all 44 jest suites (config opt-in semantics, security policy).
• SPA: util + anatomy page specs (40 passed); ng build (AOT) clean.
• Full backend suite on this tree: 9036 passed, 1 failed, 3 skipped — the failure was the architecture test registering admin router scopes, fixed in this commit; that file re-ran at 38 passed. Not exercised against dev-ai (the flag is off everywhere until someone opts in).

🤖 Generated with Claude Code