feat(evals): add retrieval-recall metric and the #174 scope regression fixture

[n0nuser]

What this does

Adds the measurable regression target #174 asks for as its suggested first step, plus the metric needed to express it. Measurement only — no retrieval tuning. The deliverable is a failing baseline that a later tuning change can be judged against.

Why a new metric was needed

Nothing in the harness could express "the expected chunk must be retrieved":

• citation_accuracy is set precision over what the answer cited, so retrieval returning topically-similar passages instead of the ones that answer the question scores fine.
• context_recall is an LLM-judged semantic score and needs a live judge.
• The one identity-level predicate, the context_omission failure label, was joined on IDs alone and was wrong in both modes: offline the retrieved IDs were the record's citation list, so the set difference was always empty and the label could never fire; live they are corpus chunk hashes sharing no namespace with dataset citation IDs, so it fired for every record.

Repairing that join is the substantive change here. resolve_retrieved_citations picks the join that means something per mode — exact on ID when overlap with the declared citations proves a shared namespace, else citation text against retrieved context text. When neither join can be made it returns None, meaning undecidable: that is not evidence of omission and is not scored as zero. The new retrieval_recall metric and the context_omission label share this one join, so they cannot disagree.

Offline mode previously could not express retrieval failure at all, since its retrieved IDs were defined as every declared citation. offline_retrieved_citation_ids makes that an explicit, validated declaration; absent, the old all-citations default stands, so existing fixtures keep their meaning.

The fixture

localrag-scope is a synthetic, authored CC0 corpus about a fictional device. It mirrors the acute-vs-chronic shape of the reported failure without redistributing the copyrighted book the failure was observed against. Four records: the motivating failure (0.0), a graded partial (0.5), a boilerplate-outranks-substance case (0.0), and a control that must not be zero (1.0).

Verification

Measured on this branch, rebased on current main:

Fixture	retrieval_recall	Threshold	Result
localrag-scope	0.375	0.8	FAIL — as intended
localrag-core (23 records)	1.000	0.8	unchanged

• uv run pytest -m "not integration" — 496 passed, 1 skipped
• Eval-specific tests — 66 passed
• ruff check / ruff format --check / mypy localrag/ — all clean

Docs updated in the same change per AGENTS.md: docs/eval-datasets.md and docs/evaluation-metrics.md.

Note for reviewers

A full run_evals --offline invocation on a slow local judge can report every metric UNAVAILABLE: the default --case-timeout of 120s kills each case before the four judge metrics finish, discarding the deterministic values computed alongside them. That is a local-environment artifact, not a defect introduced here — raising --case-timeout avoids it. Whether the deterministic metrics should survive a judge timeout is worth considering separately.

The live baseline is still outstanding and needs a machine that can host the stack.

Closes #174

[n0nuser]

Adding the full offline run_evals baseline, which finished after the PR was opened. The PR body reported retrieval_recall computed directly from the metric; this is the same number reached through the complete runner path with the judge actually completing.

exact_match            0.000  FAIL
f1                     0.390  FAIL
hallucination_rate     0.000  PASS
citation_accuracy      nan    UNAVAILABLE
retrieval_recall       0.375  FAIL
faithfulness           1.000  PASS
answer_relevancy       0.000  FAIL
context_precision      0.500  FAIL
context_recall         0.750  PASS

Command: uv run python -m evals.run_evals --offline --dataset localrag-scope --case-timeout 900, exit code 1 — the harness correctly signalling a threshold failure, which is the intended state for this fixture.

Two things this settles:

• retrieval_recall 0.375 is reproducible end-to-end, not just as an isolated metric call. That is the failing baseline a later retrieval-tuning change gets judged against.
• The UNAVAILABLE caveat in the PR body is confirmed as a timeout artifact. With the judge given time to finish, the metrics populate normally; at the default --case-timeout of 120s a slow local judge causes each case to be discarded along with the deterministic values already computed for it. Unchanged conclusion: local-environment artifact, not a defect here, and CI's own offline evaluation job passes.

citation_accuracy stays UNAVAILABLE even on a full run because the scope fixture's records declare no answer_citation_ids. That matches the fixture's purpose — it exercises the retrieval join rather than answer-citation precision, which is exactly the gap retrieval_recall was added to cover.