feat(evals): add retrieval-recall metric and the #174 scope regression fixture

docs/eval-datasets.md

@@ -47,6 +47,15 @@ scoring happens.
 
 - **Answer**: `offline_answer` if set, else `reference_answer`.
 - **Contexts**: `offline_contexts` if set, else each citation's `text`.
+- **Retrieved IDs**: `offline_retrieved_citation_ids` if set, else every
+  declared `citation_id`.
+
+The default for retrieved IDs asserts that retrieval returned everything the
+record declares — perfect recall. That is the right default for a record whose
+citations *are* its context, but it is an assertion, not a measurement, and it
+makes retrieval failure inexpressible. A record whose point is that retrieval
+missed something must list what was actually retrieved in
+`offline_retrieved_citation_ids`; see the `localrag-scope` fixture.
 
 A record with no citations and no `offline_contexts` override has nothing to
 score context-based metrics against — `_build_rows` raises
@@ -74,12 +83,41 @@ enough to tell whether two runs actually evaluated the same inputs.
 | --- | --- | --- | --- |
 | `localrag-core` | binary | 23 | `default` (all), `smoke` (first 3) |
 | `localrag-graded` | graded | 2 | `default` |
+| `localrag-scope` | binary | 4 | `default` |
 
 `localrag-core` is the project's main fixture, migrated from the legacy flat
 `evals/dataset.json` (now removed). `localrag-graded` is a minimal
 second dataset that exists to prove the registry supports more than one
 dataset and judgment type without runner changes.
 
+`localrag-scope` is the regression fixture for
+[#174](https://github.com/n0nuser/LocalRAG/issues/174): questions whose
+temporal qualifier — a single occurrence versus repeated exposure — decides
+which passage answers them. Embedding similarity keys on topic, so a question
+about one occurrence retrieves the passages about cumulative effects; both are
+topically "effects of X".
+
+**It is expected to fail, and that is the point.** `retrieval_recall` scores
+**0.375** against a threshold of 0.8. The fixture exists to make the failure a
+number a later retrieval-tuning change can be judged against, and to catch a
+fix that helps this shape of query while hurting others.
+
+Its corpus is synthetic and authored for the fixture, describing a fictional
+device. The motivating evidence came from a copyrighted book; the bundled
+fixtures are CC0, and shipping excerpts to reproduce a ranking bug is not worth
+the licensing question when an authored corpus reproduces the same shape.
+
+| Record | Shape | `retrieval_recall` |
+| --- | --- | --- |
+| `single-overvoltage-event-effect` | Acute question, only chronic passages retrieved | 0.0 |
+| `repeated-overvoltage-events-effect` | Chronic question, chronic passages retrieved | 1.0 |
+| `clamp-cooldown-interval` | Front matter and legal boilerplate outrank the answer | 0.0 |
+| `single-event-recovery-procedure` | One of two relevant passages retrieved | 0.5 |
+
+The control record matters as much as the failing ones: without it a metric
+that returned zero unconditionally would look like a successful reproduction.
+The partial record proves the metric is graded rather than binary.
+
 ## Metric annotations
 
 Citation IDs are stable within a record and are the only valid join key for

docs/evaluation-metrics.md

@@ -38,6 +38,35 @@ IDs from the #82 dataset judgments. Missing or malformed citation annotations
 are `unavailable`/missing, never zero or perfect. Citation IDs are scoped to a
 record and validated against its declared citations before evaluation.
 
+`retrieval_recall` is the recall counterpart: the fraction of
+annotation-relevant citations that retrieval actually surfaced. Threshold 0.8.
+`citation_accuracy` scores precision over what the *answer* cited, so neither it
+nor the LLM-judged `context_recall` could catch retrieval quietly returning
+topically similar passages instead of the ones that answer the question — this
+metric is identity-level and needs no judge.
+
+It joins the two sides two ways, because the run modes name chunks differently:
+
+| Mode | Retrieved IDs are | Join |
+| --- | --- | --- |
+| Offline | Dataset citation IDs (`offline_retrieved_citation_ids`) | Exact, on ID |
+| Live | Corpus chunk hashes | Citation text against retrieved context text |
+
+The namespace is proven by overlap with the record's declared citation IDs, not
+assumed. The text join accepts normalized containment or 60% token coverage
+(`RETRIEVAL_RECALL_TOKEN_COVERAGE`), because chunk boundaries cut passages and
+an annotated citation is usually a subset of a larger retrieved chunk. When
+neither join is possible the case is `unavailable` — never zero, which would be
+indistinguishable from retrieval genuinely finding nothing.
+
+The `context_omission` failure label uses the same join
+(`evals.metrics.resolve_retrieved_citations`), so the metric and the label can
+never disagree. Before that was shared, the label was joined on IDs alone and
+was wrong in both modes: offline the retrieved IDs *were* the citation list, so
+the difference was always empty and the label could never fire; live they were
+corpus hashes sharing no namespace with citation IDs, so it fired for every
+record.
+
 ## Results and thresholds
 
 Every metric stores an aggregate value, direction, threshold, per-case value,

evals/dataset/fixtures/localrag-scope-1.0.0.json

@@ -0,0 +1,202 @@
+{
+  "schema_version": 1,
+  "dataset_id": "localrag-scope",
+  "dataset_version": "1.0.0",
+  "description": "Regression fixture for issue #174: questions whose temporal qualifier (a single occurrence vs. repeated exposure) decides which passage answers them. The corpus is synthetic and authored for this fixture; the passages describe a fictional device so that no real-world claim is asserted and no copyrighted text is redistributed. Each record's offline artifacts encode the retrieval behaviour actually observed, so the fixture reproduces the failure deterministically without a live stack.",
+  "source": "authored",
+  "license": "CC0-1.0",
+  "judgment_type": "binary",
+  "records": [
+    {
+      "record_id": "single-overvoltage-event-effect",
+      "question": "What happens if a Kestrel-3 sensor is exposed to overvoltage one time?",
+      "reference_answer": "A single overvoltage event trips the input clamp and forces a reset. The sensor reports no readings for about ninety seconds while the clamp cools, after which calibration is unaffected.",
+      "citations": [
+        {
+          "citation_id": "single-overvoltage-event-effect-acute",
+          "source": "kestrel-3-field-manual",
+          "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required."
+        },
+        {
+          "citation_id": "single-overvoltage-event-effect-chronic-drift",
+          "source": "kestrel-3-field-manual",
+          "text": "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none."
+        },
+        {
+          "citation_id": "single-overvoltage-event-effect-chronic-failure",
+          "source": "kestrel-3-field-manual",
+          "text": "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life."
+        }
+      ],
+      "judgments": [
+        {
+          "citation_id": "single-overvoltage-event-effect-acute",
+          "relevant": true
+        },
+        {
+          "citation_id": "single-overvoltage-event-effect-chronic-drift",
+          "relevant": false
+        },
+        {
+          "citation_id": "single-overvoltage-event-effect-chronic-failure",
+          "relevant": false
+        }
+      ],
+      "offline_answer": "Exposure to overvoltage causes a 400% higher rate of permanent calibration drift and is the strongest predictor of eventual sensor failure and shortened service life.",
+      "offline_contexts": [
+        "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none.",
+        "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life."
+      ],
+      "offline_retrieved_citation_ids": [
+        "single-overvoltage-event-effect-chronic-drift",
+        "single-overvoltage-event-effect-chronic-failure"
+      ]
+    },
+    {
+      "record_id": "repeated-overvoltage-events-effect",
+      "question": "What happens to a Kestrel-3 sensor after many overvoltage events across its service life?",
+      "reference_answer": "Overvoltage events accumulate: units that have absorbed more than fifty of them show a 400% higher rate of permanent calibration drift, and sustained exposure across months is the strongest predictor of eventual failure.",
+      "citations": [
+        {
+          "citation_id": "repeated-overvoltage-events-effect-chronic-drift",
+          "source": "kestrel-3-field-manual",
+          "text": "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none."
+        },
+        {
+          "citation_id": "repeated-overvoltage-events-effect-chronic-failure",
+          "source": "kestrel-3-field-manual",
+          "text": "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life."
+        },
+        {
+          "citation_id": "repeated-overvoltage-events-effect-acute",
+          "source": "kestrel-3-field-manual",
+          "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required."
+        }
+      ],
+      "judgments": [
+        {
+          "citation_id": "repeated-overvoltage-events-effect-chronic-drift",
+          "relevant": true
+        },
+        {
+          "citation_id": "repeated-overvoltage-events-effect-chronic-failure",
+          "relevant": true
+        },
+        {
+          "citation_id": "repeated-overvoltage-events-effect-acute",
+          "relevant": false
+        }
+      ],
+      "offline_answer": "Overvoltage events accumulate: units that have absorbed more than fifty of them show a 400% higher rate of permanent calibration drift, and sustained exposure across many months is the strongest predictor of eventual sensor failure.",
+      "offline_contexts": [
+        "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none.",
+        "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life."
+      ],
+      "offline_retrieved_citation_ids": [
+        "repeated-overvoltage-events-effect-chronic-drift",
+        "repeated-overvoltage-events-effect-chronic-failure"
+      ]
+    },
+    {
+      "record_id": "clamp-cooldown-interval",
+      "question": "What is the clamp cooldown interval after a single overvoltage trip?",
+      "reference_answer": "About ninety seconds, during which the sensor reports no readings.",
+      "citations": [
+        {
+          "citation_id": "clamp-cooldown-interval-acute",
+          "source": "kestrel-3-field-manual",
+          "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required."
+        },
+        {
+          "citation_id": "clamp-cooldown-interval-front-matter",
+          "source": "kestrel-3-field-manual",
+          "text": "Kestrel-3 Field Manual. Revision 4. Published by the Instrumentation Documentation Group. All measurements in this manual are given in SI units unless otherwise noted. Printed on acid-free paper."
+        },
+        {
+          "citation_id": "clamp-cooldown-interval-legal",
+          "source": "kestrel-3-field-manual",
+          "text": "Notice. This manual is provided for reference only. Specifications are subject to change without notice. Contact your regional service representative before performing any procedure described in this document."
+        }
+      ],
+      "judgments": [
+        {
+          "citation_id": "clamp-cooldown-interval-acute",
+          "relevant": true
+        },
+        {
+          "citation_id": "clamp-cooldown-interval-front-matter",
+          "relevant": false
+        },
+        {
+          "citation_id": "clamp-cooldown-interval-legal",
+          "relevant": false
+        }
+      ],
+      "offline_answer": "The manual does not state a specific cooldown interval; it advises contacting your regional service representative before performing any procedure.",
+      "offline_contexts": [
+        "Kestrel-3 Field Manual. Revision 4. Published by the Instrumentation Documentation Group. All measurements in this manual are given in SI units unless otherwise noted. Printed on acid-free paper.",
+        "Notice. This manual is provided for reference only. Specifications are subject to change without notice. Contact your regional service representative before performing any procedure described in this document."
+      ],
+      "offline_retrieved_citation_ids": [
+        "clamp-cooldown-interval-front-matter",
+        "clamp-cooldown-interval-legal"
+      ]
+    },
+    {
+      "record_id": "single-event-recovery-procedure",
+      "question": "After one overvoltage event, what must an operator do before the sensor reads normally again?",
+      "reference_answer": "Nothing beyond waiting: the clamp cools in about ninety seconds and calibration is unaffected, so no service action is required. Recalibration is only called for once a unit has accumulated repeated events.",
+      "citations": [
+        {
+          "citation_id": "single-event-recovery-procedure-acute-wait",
+          "source": "kestrel-3-field-manual",
+          "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools."
+        },
+        {
+          "citation_id": "single-event-recovery-procedure-acute-noaction",
+          "source": "kestrel-3-field-manual",
+          "text": "Single-Exposure Response, continued. Calibration is unaffected by an isolated event and no service action is required. Recalibration is indicated only after a unit has accumulated repeated overvoltage events."
+        },
+        {
+          "citation_id": "single-event-recovery-procedure-chronic",
+          "source": "kestrel-3-field-manual",
+          "text": "Cumulative Degradation Over Service Life. Units that have absorbed more than fifty overvoltage events show a 400% higher rate of permanent calibration drift and must be returned for bench recalibration."
+        }
+      ],
+      "judgments": [
+        {
+          "citation_id": "single-event-recovery-procedure-acute-wait",
+          "relevant": true
+        },
+        {
+          "citation_id": "single-event-recovery-procedure-acute-noaction",
+          "relevant": true
+        },
+        {
+          "citation_id": "single-event-recovery-procedure-chronic",
+          "relevant": false
+        }
+      ],
+      "offline_answer": "The unit must be returned for bench recalibration, because overvoltage events cause a 400% higher rate of permanent calibration drift.",
+      "offline_contexts": [
+        "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools.",
+        "Cumulative Degradation Over Service Life. Units that have absorbed more than fifty overvoltage events show a 400% higher rate of permanent calibration drift and must be returned for bench recalibration."
+      ],
+      "offline_retrieved_citation_ids": [
+        "single-event-recovery-procedure-acute-wait",
+        "single-event-recovery-procedure-chronic"
+      ]
+    }
+  ],
+  "splits": [
+    {
+      "name": "default",
+      "record_ids": [
+        "single-overvoltage-event-effect",
+        "repeated-overvoltage-events-effect",
+        "clamp-cooldown-interval",
+        "single-event-recovery-procedure"
+      ]
+    }
+  ]
+}

evals/dataset/schema.py

@@ -83,6 +83,14 @@ class DatasetRecord(BaseModel):
     # falls back to reference_answer as the answer and cited texts as context.
     offline_answer: str | None = None
     offline_contexts: list[str] | None = None
+    offline_retrieved_citation_ids: list[str] | None = Field(
+        default=None,
+        description=(
+            "Citation IDs offline mode should treat as retrieved. Absent means every "
+            "declared citation, which asserts perfect retrieval — state this explicitly "
+            "for records whose point is that retrieval missed something."
+        ),
+    )
 
     @field_validator("record_id")
     @classmethod
@@ -102,6 +110,13 @@ def _judgments_reference_known_citations(self) -> DatasetRecord:
                     f"citation_id {judgment.citation_id!r}"
                 )
                 raise ValueError(message)
+        for citation_id in self.offline_retrieved_citation_ids or []:
+            if citation_id not in known_ids:
+                message = (
+                    f"record {self.record_id!r}: offline_retrieved_citation_ids references "
+                    f"unknown citation_id {citation_id!r}"
+                )
+                raise ValueError(message)
         return self
 
     def offline_context_texts(self) -> list[str]:
@@ -118,6 +133,16 @@ def relevant_citation_ids(self) -> list[str]:
         """Return citation IDs marked relevant by the #82 annotation contract."""
         return [judgment.citation_id for judgment in self.judgments if judgment.relevant]
 
+    def citation_texts(self) -> dict[str, str]:
+        """Map declared citation IDs to their passages, for the retrieval-recall join."""
+        return {citation.citation_id: citation.text for citation in self.citations}
+
+    def offline_retrieved_ids(self) -> list[str]:
+        """IDs offline mode reports as retrieved: explicit override, else every citation."""
+        if self.offline_retrieved_citation_ids is not None:
+            return self.offline_retrieved_citation_ids
+        return [citation.citation_id for citation in self.citations]
+
 
 class DatasetSplit(BaseModel):
     """A named subset of a dataset version's records (e.g. 'default', 'smoke')."""