Phase 9: diagnostic-accuracy eval + runbook context in ranking (measured, left off)

[Lushenwar]

Summary

Answers one question with a measurement: does giving Claude the matched runbooks improve its ability to find the commit that caused an incident? At this corpus size, no measurable benefit, so SENTINEL_RAG_RANKING ships off. Full write-up: METRICS.md → Phase 9, ADR.md → ADR-7.

Eval set v2, 24 cases × 3 trials	Baseline	RAG (floor 0.35)
Top-1	90.3% (65/72)	91.7% (66/72)
MRR	0.947	0.958
Paired per case		2 wins / 1 loss / 21 ties

On the 10 cases where runbook text actually reached the prompt, both conditions scored 27/30; the single extra RAG hit came from a case whose prompt was byte-identical, i.e. noise.

What changed

• Eval harness (eval/): builds each case as a throwaway git repo and calls the production get_recent_diffs / find_matching_runbooks / rank_suspect_commits. Scorer (top-1/top-3/MRR, paired table, retrieval + floor table), aborts after 5 consecutive LLM errors, results committed under eval/results/.
• Cases: v1 (29 cases) hit 87/87 at baseline (ceiling) and was replaced by v2 (24 cases, a decoy per case) before any RAG run. Lint test bans answer-revealing words; leakage test asserts no label/id/category reaches the prompt in either condition.
• Runbook corpus: 2 → 10 generic runbooks, written before any case, frozen.
• Core: optional runbook context in ranking behind SENTINEL_RAG_RANKING (default 0), floor SENTINEL_RUNBOOK_FLOOR = 0.35 chosen from baseline retrieval scores only. Golden-string test pins the no-runbook prompt byte-for-byte. Retrieval failure now logs and ranks without runbooks instead of reverting the incident. Runbook text is prompt-only, never persisted.
• LLM client: OpenRouter → Anthropic SDK direct (core/services/llm.py), same model claude-sonnet-5, max_tokens 16000 (Sonnet 5 thinks by default). Env var is now ANTHROPIC_API_KEY; optional ANTHROPIC_WORKSPACE_ID.
• Docs: Phase 5 caveat (those runs used commit messages naming the bug — pipeline verification, not accuracy), METRICS Phase 9, ADR-7, one README line citing baseline accuracy only.

Merge note

Please merge with a merge commit, not squash: the results files record the Sentinel commit each run used (7b4dc90, 2e1eb8d, 530b73f), and squashing would orphan those SHAs. 1dd55ae is the already-squash-merged PR #7 commit; it is a no-op against main.

Test plan

• black --check --line-length 110 core sandbox eval, flake8 core sandbox eval
• pytest core/tests eval/tests -q — 83 passed, all offline (vector store and Claude mocked)
• Dashboard lint + tests
• python -m eval.report eval/results/baseline_v2_2026-10-04.json eval/results/rag_v2_2026-10-04.json reproduces every number in METRICS.md
• CI green on this PR (CI now also covers eval/)

🤖 Generated with Claude Code