Model comparison: Haiku 4.5 vs Sonnet 5, run stability, close-call out of sample

[Lushenwar]

Summary

Reuses the Phase 9 harness (eval set v2, no runbooks, 24 cases × 3 trials).

	Sonnet 5 (re-run)	Haiku 4.5
Top-1	91.7% (66/72)	80.6% (58/72)
Ranking latency (median)	6.5s	4.5s
Cost per ranking call (list price)	~$0.013	~$0.005

• Sonnet 5 stays the default: Haiku is ~2.4x cheaper and ~2s faster but 11 points less accurate (2 case wins / 5 losses).
• Stability: the Sonnet re-run (91.7%) matches the Phase 9 baseline (90.3%).
• Close-call rule holds out of sample on Sonnet (4/6 wrong flagged, 7/66 false alarms) but not on Haiku, whose confidence is identical for right and wrong picks. Documented as model-specific.

Changes

• SENTINEL_MODEL env override in core/services/llm.py (default claude-sonnet-5).
• Harness meters token usage per call (wrapping chat, no production change); report shows tokens and labels model comparisons by model.
• Two results files + METRICS.md section.

Merge note

Results record dfd2797; this will be rebase-merged, so branch feat/model-comparison is kept (noted in eval/results/README.md).

Test plan

• pytest core/tests eval/tests -q (85 passed), black, flake8
• Report reproduces the table; cost recomputed from metered tokens
• CI green

🤖 Generated with Claude Code