Repository navigation

Phase 9: diagnostic-accuracy eval + runbook context in ranking (measured, left off) - #8

Merged

Lushenwar merged 20 commits into

mainfrom

feat/phase9-rag-eval

Oct 6, 2026

Commits on Jul 14, 2026
- style: black-format dedup changes to fix CI
  
  Lushenwar
  and
  claude
  committed
  1dd55ae
Commits on Oct 4, 2026
- feat(core): make Chroma path configurable via SENTINEL_CHROMA_PATH
  
  Lushenwar
  and
  claude
  committed
  24c34e0
- feat(runbooks): expand corpus to 10 generic runbooks, freeze as v1
  
  Lushenwar
  and
  claude
  committed
  1eed3bf
- feat(eval): offline-buildable eval harness, scorer, and leakage tests
  
  Lushenwar
  and
  claude
  committed
  8d593b0
- feat(eval): 29-case fault set on a shared orders-service base, frozen as v1
  
  Lushenwar
  and
  claude
  committed
  d6f6bd1
- fix(eval): ignore untracked files in results git-sha dirty check
  
  Lushenwar
  and
  claude
  committed
  8f07d04
- style(eval): black-format run_eval
  
  Lushenwar
  and
  claude
  committed
  41c8527
- feat(eval): abort runs after 5 consecutive LLM errors; archive the 402-failed baseline
  
  Lushenwar
  and
  claude
  committed
  fd3faba
- chore(eval): move 402-failed baseline under results/aborted
  
  Lushenwar
  and
  claude
  committed
  c2d93bd
- feat(core): call Claude directly via the Anthropic SDK instead of OpenRouter
  
  Lushenwar
  and
  claude
  committed
  5f93cb7
- feat(core): send anthropic-workspace-id header when ANTHROPIC_WORKSPACE_ID is set
  
  Lushenwar
  and
  claude
  committed
  7b4dc90
- feat(eval): baseline run on eval set v1 (87/87 top-1, ceiling)
  
  Lushenwar
  and
  claude
  committed
  b677ac1
- feat(eval): harder eval set v2 (24 cases, decoy per case), replaces v1 after ceiling
  
  Lushenwar
  and
  claude
  committed
  2e1eb8d
- feat(eval): baseline run on eval set v2 (65/72 top-1)
  
  Lushenwar
  and
  claude
  committed
  150d678
- feat(core): optional runbook context in commit ranking behind SENTINEL_RAG_RANKING
  
  Lushenwar
  and
  claude
  committed
  530b73f
- feat(eval): RAG run on eval set v2 (66/72 top-1, floor 0.35)
  
  Lushenwar
  and
  claude
  committed
  4dc3f1f
Commits on Oct 6, 2026
- docs: Phase 9 eval results, ADR-7, Phase 5 accuracy caveat
  
  Lushenwar
  and
  claude
  committed
  b7b7329
- Merge remote-tracking branch 'origin/main' into feat/phase9-rag-eval
  Lushenwar
  committed
  f02dbb8
- docs(core): note the Phase 9 outcome on the RAG ranking flag
  
  Lushenwar
  and
  claude
  committed
  e5cb969
- docs(eval): note where recorded run SHAs live after rebase-merge
  
  Lushenwar
  and
  claude
  committed
  4b5f31d

Navigation Menu

Repository navigation

Phase 9: diagnostic-accuracy eval + runbook context in ranking (measured, left off) - #8

Phase 9: diagnostic-accuracy eval + runbook context in ranking (measured, left off)#8
Lushenwar merged 20 commits into
mainfrom
feat/phase9-rag-eval

Commits on Jul 14, 2026

style: black-format dedup changes to fix CI

Commits on Oct 4, 2026

feat(core): make Chroma path configurable via SENTINEL_CHROMA_PATH

feat(runbooks): expand corpus to 10 generic runbooks, freeze as v1

feat(eval): offline-buildable eval harness, scorer, and leakage tests

feat(eval): 29-case fault set on a shared orders-service base, frozen as v1

fix(eval): ignore untracked files in results git-sha dirty check

style(eval): black-format run_eval

feat(eval): abort runs after 5 consecutive LLM errors; archive the 402-failed baseline

chore(eval): move 402-failed baseline under results/aborted

feat(core): call Claude directly via the Anthropic SDK instead of OpenRouter

feat(core): send anthropic-workspace-id header when ANTHROPIC_WORKSPACE_ID is set

feat(eval): baseline run on eval set v1 (87/87 top-1, ceiling)

feat(eval): harder eval set v2 (24 cases, decoy per case), replaces v1 after ceiling

feat(eval): baseline run on eval set v2 (65/72 top-1)

feat(core): optional runbook context in commit ranking behind SENTINEL_RAG_RANKING

feat(eval): RAG run on eval set v2 (66/72 top-1, floor 0.35)

Commits on Oct 6, 2026

docs: Phase 9 eval results, ADR-7, Phase 5 accuracy caveat

Merge remote-tracking branch 'origin/main' into feat/phase9-rag-eval

docs(core): note the Phase 9 outcome on the RAG ranking flag

docs(eval): note where recorded run SHAs live after rebase-merge