diff --git a/docs/eval-datasets.md b/docs/eval-datasets.md index a6a7ec2..148fc55 100644 --- a/docs/eval-datasets.md +++ b/docs/eval-datasets.md @@ -47,6 +47,15 @@ scoring happens. - **Answer**: `offline_answer` if set, else `reference_answer`. - **Contexts**: `offline_contexts` if set, else each citation's `text`. +- **Retrieved IDs**: `offline_retrieved_citation_ids` if set, else every + declared `citation_id`. + +The default for retrieved IDs asserts that retrieval returned everything the +record declares — perfect recall. That is the right default for a record whose +citations *are* its context, but it is an assertion, not a measurement, and it +makes retrieval failure inexpressible. A record whose point is that retrieval +missed something must list what was actually retrieved in +`offline_retrieved_citation_ids`; see the `localrag-scope` fixture. A record with no citations and no `offline_contexts` override has nothing to score context-based metrics against — `_build_rows` raises @@ -74,12 +83,41 @@ enough to tell whether two runs actually evaluated the same inputs. | --- | --- | --- | --- | | `localrag-core` | binary | 23 | `default` (all), `smoke` (first 3) | | `localrag-graded` | graded | 2 | `default` | +| `localrag-scope` | binary | 4 | `default` | `localrag-core` is the project's main fixture, migrated from the legacy flat `evals/dataset.json` (now removed). `localrag-graded` is a minimal second dataset that exists to prove the registry supports more than one dataset and judgment type without runner changes. +`localrag-scope` is the regression fixture for +[#174](https://github.com/n0nuser/LocalRAG/issues/174): questions whose +temporal qualifier — a single occurrence versus repeated exposure — decides +which passage answers them. Embedding similarity keys on topic, so a question +about one occurrence retrieves the passages about cumulative effects; both are +topically "effects of X". + +**It is expected to fail, and that is the point.** `retrieval_recall` scores +**0.375** against a threshold of 0.8. The fixture exists to make the failure a +number a later retrieval-tuning change can be judged against, and to catch a +fix that helps this shape of query while hurting others. + +Its corpus is synthetic and authored for the fixture, describing a fictional +device. The motivating evidence came from a copyrighted book; the bundled +fixtures are CC0, and shipping excerpts to reproduce a ranking bug is not worth +the licensing question when an authored corpus reproduces the same shape. + +| Record | Shape | `retrieval_recall` | +| --- | --- | --- | +| `single-overvoltage-event-effect` | Acute question, only chronic passages retrieved | 0.0 | +| `repeated-overvoltage-events-effect` | Chronic question, chronic passages retrieved | 1.0 | +| `clamp-cooldown-interval` | Front matter and legal boilerplate outrank the answer | 0.0 | +| `single-event-recovery-procedure` | One of two relevant passages retrieved | 0.5 | + +The control record matters as much as the failing ones: without it a metric +that returned zero unconditionally would look like a successful reproduction. +The partial record proves the metric is graded rather than binary. + ## Metric annotations Citation IDs are stable within a record and are the only valid join key for diff --git a/docs/evaluation-metrics.md b/docs/evaluation-metrics.md index 9ad6082..7091088 100644 --- a/docs/evaluation-metrics.md +++ b/docs/evaluation-metrics.md @@ -38,6 +38,35 @@ IDs from the #82 dataset judgments. Missing or malformed citation annotations are `unavailable`/missing, never zero or perfect. Citation IDs are scoped to a record and validated against its declared citations before evaluation. +`retrieval_recall` is the recall counterpart: the fraction of +annotation-relevant citations that retrieval actually surfaced. Threshold 0.8. +`citation_accuracy` scores precision over what the *answer* cited, so neither it +nor the LLM-judged `context_recall` could catch retrieval quietly returning +topically similar passages instead of the ones that answer the question — this +metric is identity-level and needs no judge. + +It joins the two sides two ways, because the run modes name chunks differently: + +| Mode | Retrieved IDs are | Join | +| --- | --- | --- | +| Offline | Dataset citation IDs (`offline_retrieved_citation_ids`) | Exact, on ID | +| Live | Corpus chunk hashes | Citation text against retrieved context text | + +The namespace is proven by overlap with the record's declared citation IDs, not +assumed. The text join accepts normalized containment or 60% token coverage +(`RETRIEVAL_RECALL_TOKEN_COVERAGE`), because chunk boundaries cut passages and +an annotated citation is usually a subset of a larger retrieved chunk. When +neither join is possible the case is `unavailable` — never zero, which would be +indistinguishable from retrieval genuinely finding nothing. + +The `context_omission` failure label uses the same join +(`evals.metrics.resolve_retrieved_citations`), so the metric and the label can +never disagree. Before that was shared, the label was joined on IDs alone and +was wrong in both modes: offline the retrieved IDs *were* the citation list, so +the difference was always empty and the label could never fire; live they were +corpus hashes sharing no namespace with citation IDs, so it fired for every +record. + ## Results and thresholds Every metric stores an aggregate value, direction, threshold, per-case value, diff --git a/evals/dataset/fixtures/localrag-scope-1.0.0.json b/evals/dataset/fixtures/localrag-scope-1.0.0.json new file mode 100644 index 0000000..1e80ea3 --- /dev/null +++ b/evals/dataset/fixtures/localrag-scope-1.0.0.json @@ -0,0 +1,202 @@ +{ + "schema_version": 1, + "dataset_id": "localrag-scope", + "dataset_version": "1.0.0", + "description": "Regression fixture for issue #174: questions whose temporal qualifier (a single occurrence vs. repeated exposure) decides which passage answers them. The corpus is synthetic and authored for this fixture; the passages describe a fictional device so that no real-world claim is asserted and no copyrighted text is redistributed. Each record's offline artifacts encode the retrieval behaviour actually observed, so the fixture reproduces the failure deterministically without a live stack.", + "source": "authored", + "license": "CC0-1.0", + "judgment_type": "binary", + "records": [ + { + "record_id": "single-overvoltage-event-effect", + "question": "What happens if a Kestrel-3 sensor is exposed to overvoltage one time?", + "reference_answer": "A single overvoltage event trips the input clamp and forces a reset. The sensor reports no readings for about ninety seconds while the clamp cools, after which calibration is unaffected.", + "citations": [ + { + "citation_id": "single-overvoltage-event-effect-acute", + "source": "kestrel-3-field-manual", + "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required." + }, + { + "citation_id": "single-overvoltage-event-effect-chronic-drift", + "source": "kestrel-3-field-manual", + "text": "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none." + }, + { + "citation_id": "single-overvoltage-event-effect-chronic-failure", + "source": "kestrel-3-field-manual", + "text": "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life." + } + ], + "judgments": [ + { + "citation_id": "single-overvoltage-event-effect-acute", + "relevant": true + }, + { + "citation_id": "single-overvoltage-event-effect-chronic-drift", + "relevant": false + }, + { + "citation_id": "single-overvoltage-event-effect-chronic-failure", + "relevant": false + } + ], + "offline_answer": "Exposure to overvoltage causes a 400% higher rate of permanent calibration drift and is the strongest predictor of eventual sensor failure and shortened service life.", + "offline_contexts": [ + "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none.", + "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life." + ], + "offline_retrieved_citation_ids": [ + "single-overvoltage-event-effect-chronic-drift", + "single-overvoltage-event-effect-chronic-failure" + ] + }, + { + "record_id": "repeated-overvoltage-events-effect", + "question": "What happens to a Kestrel-3 sensor after many overvoltage events across its service life?", + "reference_answer": "Overvoltage events accumulate: units that have absorbed more than fifty of them show a 400% higher rate of permanent calibration drift, and sustained exposure across months is the strongest predictor of eventual failure.", + "citations": [ + { + "citation_id": "repeated-overvoltage-events-effect-chronic-drift", + "source": "kestrel-3-field-manual", + "text": "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none." + }, + { + "citation_id": "repeated-overvoltage-events-effect-chronic-failure", + "source": "kestrel-3-field-manual", + "text": "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life." + }, + { + "citation_id": "repeated-overvoltage-events-effect-acute", + "source": "kestrel-3-field-manual", + "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required." + } + ], + "judgments": [ + { + "citation_id": "repeated-overvoltage-events-effect-chronic-drift", + "relevant": true + }, + { + "citation_id": "repeated-overvoltage-events-effect-chronic-failure", + "relevant": true + }, + { + "citation_id": "repeated-overvoltage-events-effect-acute", + "relevant": false + } + ], + "offline_answer": "Overvoltage events accumulate: units that have absorbed more than fifty of them show a 400% higher rate of permanent calibration drift, and sustained exposure across many months is the strongest predictor of eventual sensor failure.", + "offline_contexts": [ + "Cumulative Degradation Over Service Life. Overvoltage events accumulate. Units that have absorbed more than fifty such events across their service life show a 400% higher rate of permanent calibration drift than units that have absorbed none.", + "Cumulative Degradation Over Service Life. Sustained operation under overvoltage conditions across many months is the strongest single predictor of eventual sensor failure and of materially shortened service life." + ], + "offline_retrieved_citation_ids": [ + "repeated-overvoltage-events-effect-chronic-drift", + "repeated-overvoltage-events-effect-chronic-failure" + ] + }, + { + "record_id": "clamp-cooldown-interval", + "question": "What is the clamp cooldown interval after a single overvoltage trip?", + "reference_answer": "About ninety seconds, during which the sensor reports no readings.", + "citations": [ + { + "citation_id": "clamp-cooldown-interval-acute", + "source": "kestrel-3-field-manual", + "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools. Calibration is unaffected by an isolated event and no service action is required." + }, + { + "citation_id": "clamp-cooldown-interval-front-matter", + "source": "kestrel-3-field-manual", + "text": "Kestrel-3 Field Manual. Revision 4. Published by the Instrumentation Documentation Group. All measurements in this manual are given in SI units unless otherwise noted. Printed on acid-free paper." + }, + { + "citation_id": "clamp-cooldown-interval-legal", + "source": "kestrel-3-field-manual", + "text": "Notice. This manual is provided for reference only. Specifications are subject to change without notice. Contact your regional service representative before performing any procedure described in this document." + } + ], + "judgments": [ + { + "citation_id": "clamp-cooldown-interval-acute", + "relevant": true + }, + { + "citation_id": "clamp-cooldown-interval-front-matter", + "relevant": false + }, + { + "citation_id": "clamp-cooldown-interval-legal", + "relevant": false + } + ], + "offline_answer": "The manual does not state a specific cooldown interval; it advises contacting your regional service representative before performing any procedure.", + "offline_contexts": [ + "Kestrel-3 Field Manual. Revision 4. Published by the Instrumentation Documentation Group. All measurements in this manual are given in SI units unless otherwise noted. Printed on acid-free paper.", + "Notice. This manual is provided for reference only. Specifications are subject to change without notice. Contact your regional service representative before performing any procedure described in this document." + ], + "offline_retrieved_citation_ids": [ + "clamp-cooldown-interval-front-matter", + "clamp-cooldown-interval-legal" + ] + }, + { + "record_id": "single-event-recovery-procedure", + "question": "After one overvoltage event, what must an operator do before the sensor reads normally again?", + "reference_answer": "Nothing beyond waiting: the clamp cools in about ninety seconds and calibration is unaffected, so no service action is required. Recalibration is only called for once a unit has accumulated repeated events.", + "citations": [ + { + "citation_id": "single-event-recovery-procedure-acute-wait", + "source": "kestrel-3-field-manual", + "text": "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools." + }, + { + "citation_id": "single-event-recovery-procedure-acute-noaction", + "source": "kestrel-3-field-manual", + "text": "Single-Exposure Response, continued. Calibration is unaffected by an isolated event and no service action is required. Recalibration is indicated only after a unit has accumulated repeated overvoltage events." + }, + { + "citation_id": "single-event-recovery-procedure-chronic", + "source": "kestrel-3-field-manual", + "text": "Cumulative Degradation Over Service Life. Units that have absorbed more than fifty overvoltage events show a 400% higher rate of permanent calibration drift and must be returned for bench recalibration." + } + ], + "judgments": [ + { + "citation_id": "single-event-recovery-procedure-acute-wait", + "relevant": true + }, + { + "citation_id": "single-event-recovery-procedure-acute-noaction", + "relevant": true + }, + { + "citation_id": "single-event-recovery-procedure-chronic", + "relevant": false + } + ], + "offline_answer": "The unit must be returned for bench recalibration, because overvoltage events cause a 400% higher rate of permanent calibration drift.", + "offline_contexts": [ + "Single-Exposure Response. One overvoltage event trips the input clamp and forces a sensor reset. Readings are unavailable for approximately ninety seconds while the clamp cools.", + "Cumulative Degradation Over Service Life. Units that have absorbed more than fifty overvoltage events show a 400% higher rate of permanent calibration drift and must be returned for bench recalibration." + ], + "offline_retrieved_citation_ids": [ + "single-event-recovery-procedure-acute-wait", + "single-event-recovery-procedure-chronic" + ] + } + ], + "splits": [ + { + "name": "default", + "record_ids": [ + "single-overvoltage-event-effect", + "repeated-overvoltage-events-effect", + "clamp-cooldown-interval", + "single-event-recovery-procedure" + ] + } + ] +} diff --git a/evals/dataset/schema.py b/evals/dataset/schema.py index 8b40981..8208bec 100644 --- a/evals/dataset/schema.py +++ b/evals/dataset/schema.py @@ -83,6 +83,14 @@ class DatasetRecord(BaseModel): # falls back to reference_answer as the answer and cited texts as context. offline_answer: str | None = None offline_contexts: list[str] | None = None + offline_retrieved_citation_ids: list[str] | None = Field( + default=None, + description=( + "Citation IDs offline mode should treat as retrieved. Absent means every " + "declared citation, which asserts perfect retrieval — state this explicitly " + "for records whose point is that retrieval missed something." + ), + ) @field_validator("record_id") @classmethod @@ -102,6 +110,13 @@ def _judgments_reference_known_citations(self) -> DatasetRecord: f"citation_id {judgment.citation_id!r}" ) raise ValueError(message) + for citation_id in self.offline_retrieved_citation_ids or []: + if citation_id not in known_ids: + message = ( + f"record {self.record_id!r}: offline_retrieved_citation_ids references " + f"unknown citation_id {citation_id!r}" + ) + raise ValueError(message) return self def offline_context_texts(self) -> list[str]: @@ -118,6 +133,16 @@ def relevant_citation_ids(self) -> list[str]: """Return citation IDs marked relevant by the #82 annotation contract.""" return [judgment.citation_id for judgment in self.judgments if judgment.relevant] + def citation_texts(self) -> dict[str, str]: + """Map declared citation IDs to their passages, for the retrieval-recall join.""" + return {citation.citation_id: citation.text for citation in self.citations} + + def offline_retrieved_ids(self) -> list[str]: + """IDs offline mode reports as retrieved: explicit override, else every citation.""" + if self.offline_retrieved_citation_ids is not None: + return self.offline_retrieved_citation_ids + return [citation.citation_id for citation in self.citations] + class DatasetSplit(BaseModel): """A named subset of a dataset version's records (e.g. 'default', 'smoke').""" diff --git a/evals/failure_analysis.py b/evals/failure_analysis.py index 9db999d..84dcffa 100644 --- a/evals/failure_analysis.py +++ b/evals/failure_analysis.py @@ -13,6 +13,8 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator +from evals.metrics import resolve_retrieved_citations + FAILURE_LABELS = ( "retrieval_miss", "context_omission", @@ -54,6 +56,10 @@ class FailureCaseArtifact(BaseModel): retrieved_text: list[str] = Field(default_factory=list) citation_ids: list[str] | None = None relevant_citation_ids: list[str] | None = None + citation_texts: dict[str, str] | None = Field( + default=None, + description="Declared citation ID to passage; lets context_omission join on text.", + ) metrics: dict[str, dict[str, Any]] = Field(default_factory=dict) error: str | None = None @@ -149,6 +155,28 @@ def _judge(artifact: FailureCaseArtifact, config: FailureAnalysisConfig) -> Judg return None +def _omits_relevant_context(artifact: FailureCaseArtifact) -> bool: + """Whether annotated-relevant context was left out of what retrieval returned. + + Joining relevant IDs against retrieved IDs directly cannot work in either + mode: offline the two sets were the same list by construction, so the + difference was always empty and this label could never fire; live they are + dataset citation IDs versus corpus chunk hashes, sharing no namespace, so it + fired for every record. ``resolve_retrieved_citations`` picks the join that + means something in each mode and reports ``None`` when neither does — which + is not evidence of omission, so it must not be labelled as one. + """ + retrieved_citations = resolve_retrieved_citations( + artifact.relevant_citation_ids, + artifact.retrieved_ids, + artifact.citation_texts, + artifact.retrieved_text, + ) + if retrieved_citations is None: + return False + return bool(set(artifact.relevant_citation_ids or []) - retrieved_citations) + + def classify_case( # noqa: C901 artifact: FailureCaseArtifact, *, config: FailureAnalysisConfig | None = None ) -> FailureClassification: @@ -170,11 +198,7 @@ def classify_case( # noqa: C901 labels.add("retrieval_miss") reasons.append("no_retrieved_context") failed = True - if ( - artifact.retrieved_ids - and artifact.relevant_citation_ids - and set(artifact.relevant_citation_ids) - set(artifact.retrieved_ids) - ): + if _omits_relevant_context(artifact): labels.add("context_omission") reasons.append("relevant_context_not_retrieved") failed = True diff --git a/evals/metrics.py b/evals/metrics.py index 58c6879..879ef5d 100644 --- a/evals/metrics.py +++ b/evals/metrics.py @@ -112,3 +112,83 @@ def score_citation_accuracy( return MetricCase(status="unavailable", warning="citation annotation is missing") relevant = set(relevant_ids) return MetricCase(value=len(set(cited_ids) & relevant) / len(set(cited_ids))) + + +#: Fraction of a citation's tokens that must appear in a retrieved context for the +#: text join to count it as retrieved. Deliberately below 1.0: chunk boundaries cut +#: passages, so an annotated citation is often a subset of a larger retrieved chunk. +RETRIEVAL_RECALL_TOKEN_COVERAGE = 0.6 + + +def _context_contains_citation(citation_text: str, contexts: list[str]) -> bool: + """Whether any retrieved context carries this citation's passage.""" + needle = normalize_answer(citation_text) + if not needle: + return False + needle_tokens = set(needle.split()) + for context in contexts: + haystack = normalize_answer(context) + if needle in haystack: + return True + if needle_tokens: + covered = len(needle_tokens & set(haystack.split())) / len(needle_tokens) + if covered >= RETRIEVAL_RECALL_TOKEN_COVERAGE: + return True + return False + + +def resolve_retrieved_citations( + relevant_ids: list[str] | None, + retrieved_ids: list[str] | None, + citation_texts: dict[str, str] | None, + retrieved_texts: list[str] | None, +) -> set[str] | None: + """Return which relevant citations retrieval actually surfaced. + + ``None`` means undecidable — the caller must not read that as "none were + retrieved". This is the single join both ``retrieval_recall`` and the + ``context_omission`` failure label use, so they can never disagree. + + Two joins, because the two run modes name chunks differently. When the + retrieved IDs live in the dataset's own citation namespace the ID join is + exact and is preferred. A live run returns corpus chunk hashes instead, + which share no namespace with dataset citation IDs, so comparing them is + meaningless; there the citation's *text* is matched against the retrieved + context text. + """ + if not relevant_ids: + return None + known = citation_texts or {} + relevant = set(relevant_ids) + # Overlap with the declared citation IDs is what proves a shared namespace. + # With no citation text supplied there is nothing to prove it against and + # nothing to fall back to, so the ID join is taken at face value. + if retrieved_ids and (not known or set(retrieved_ids) & set(known)): + return relevant & set(retrieved_ids) + if not retrieved_texts: + return None + if any(not known.get(citation_id) for citation_id in relevant): + return None + return { + citation_id + for citation_id in relevant + if _context_contains_citation(known[citation_id], retrieved_texts) + } + + +def score_retrieval_recall( + relevant_ids: list[str] | None, + retrieved_ids: list[str] | None, + citation_texts: dict[str, str] | None, + retrieved_texts: list[str] | None, +) -> MetricCase: + """Score the fraction of annotated-relevant citations that were retrieved. + + Complements ``citation_accuracy``, which is set *precision* over what the + answer cited. Neither could previously catch retrieval silently returning + topically-similar passages instead of the ones that answer the question. + """ + hits = resolve_retrieved_citations(relevant_ids, retrieved_ids, citation_texts, retrieved_texts) + if hits is None: + return MetricCase(status="unavailable", warning="retrieval recall join is unavailable") + return MetricCase(value=len(hits) / len(set(relevant_ids or []))) diff --git a/evals/run_evals.py b/evals/run_evals.py index 4b21dbd..f4814e6 100644 --- a/evals/run_evals.py +++ b/evals/run_evals.py @@ -64,7 +64,7 @@ from evals.dataset.schema import DatasetRecord from evals.environment import capture_run_metadata, resolve_seed from evals.failure_analysis import FailureCaseArtifact, classify_cases -from evals.metrics import exact_match, f1, score_citation_accuracy +from evals.metrics import exact_match, f1, score_citation_accuracy, score_retrieval_recall from evals.results.schema import ( EvaluationCaseResult, MetricCaseResult, @@ -80,11 +80,25 @@ DEFAULT_SPLIT = "default" JUDGE_EMBED_MODEL = "nomic-embed-text" +#: Metrics scored from dataset annotations rather than by a judge. A non-finite +#: value means the annotation or the retrieval join was unavailable -- which is +#: not a judge error and must not be reported as one. +ANNOTATION_METRICS = frozenset({"citation_accuracy", "retrieval_recall"}) + +#: Metrics computed locally, so a non-finite value can never be a judge failure. +DETERMINISTIC_METRICS = ANNOTATION_METRICS | frozenset({"exact_match", "f1"}) + +_ANNOTATION_WARNINGS = { + "citation_accuracy": "citation annotation is missing", + "retrieval_recall": "retrieval recall join is unavailable", +} + PASS_THRESHOLDS = { "exact_match": 1.0, "f1": 0.5, "hallucination_rate": 0.4, "citation_accuracy": 0.8, + "retrieval_recall": 0.8, "faithfulness": 0.6, "answer_relevancy": 0.6, "context_precision": 0.5, @@ -155,17 +169,14 @@ def _build_rows( "question": rec.question, "answer": answer, "contexts": contexts, - "retrieved_ids": ( - retrieved_ids - if not offline - else [citation.citation_id for citation in rec.citations] - ), + "retrieved_ids": (retrieved_ids if not offline else rec.offline_retrieved_ids()), "answering_provider": provider if not offline else None, "answering_model": model if not offline else None, "ground_truth": rec.reference_answer, "ground_truths": rec.reference_answers_or_default(), "answer_citation_ids": rec.answer_citation_ids, "relevant_citation_ids": rec.relevant_citation_ids(), + "citation_texts": rec.citation_texts(), } ) return rows @@ -222,17 +233,14 @@ async def build(record: DatasetRecord) -> dict: "question": record.question, "answer": answer, "contexts": contexts, - "retrieved_ids": ( - retrieved_ids - if not offline - else [citation.citation_id for citation in record.citations] - ), + "retrieved_ids": (retrieved_ids if not offline else record.offline_retrieved_ids()), "answering_provider": provider if not offline else None, "answering_model": model if not offline else None, "ground_truth": record.reference_answer, "ground_truths": record.reference_answers_or_default(), "answer_citation_ids": record.answer_citation_ids, "relevant_citation_ids": record.relevant_citation_ids(), + "citation_texts": record.citation_texts(), } outcomes = await run_cases( @@ -319,7 +327,7 @@ def _failure_analysis( "status": ( "complete" if math.isfinite(value) - else ("unavailable" if name == "citation_accuracy" else "error") + else ("unavailable" if name in ANNOTATION_METRICS else "error") ), "threshold": PASS_THRESHOLDS[name], "direction": "lower_is_better" @@ -339,6 +347,7 @@ def _failure_analysis( retrieved_text=row.get("contexts", []), citation_ids=row.get("answer_citation_ids"), relevant_citation_ids=row.get("relevant_citation_ids"), + citation_texts=row.get("citation_texts"), metrics=metrics, error=failed_execution.get("error") if failed_execution else None, ) @@ -382,6 +391,7 @@ async def _score_rows_parallel( "f1": [], "hallucination_rate": [], "citation_accuracy": [], + "retrieval_recall": [], "faithfulness": [], "answer_relevancy": [], "context_precision": [], @@ -409,6 +419,13 @@ async def score(row: dict) -> dict[str, Any]: response, row["answer_citation_ids"], row["relevant_citation_ids"] ) result["citation_accuracy"] = citation.value if citation.value is not None else math.nan + recall = score_retrieval_recall( + row["relevant_citation_ids"], + row.get("retrieved_ids"), + row.get("citation_texts"), + retrieved_contexts, + ) + result["retrieval_recall"] = recall.value if recall.value is not None else math.nan judge_results = await asyncio.gather( judge_call( @@ -648,7 +665,7 @@ def main() -> None: # noqa: PLR0915 — length is flat argparse flag declaratio else "higher_is_better", "threshold": PASS_THRESHOLDS[name], "missing_value": "not_applicable" - if name == "citation_accuracy" + if name in ANNOTATION_METRICS else "missing", } ), @@ -670,17 +687,17 @@ def main() -> None: # noqa: PLR0915 — length is flat argparse flag declaratio "status": ( "complete" if math.isfinite(value) - else ("unavailable" if name == "citation_accuracy" else "error") + else ("unavailable" if name in ANNOTATION_METRICS else "error") ), "input_ids": row.get("relevant_citation_ids", []), "warning": ( - "citation annotation is missing" - if name == "citation_accuracy" and not math.isfinite(value) + _ANNOTATION_WARNINGS[name] + if name in ANNOTATION_METRICS and not math.isfinite(value) else None ), "error": ( "judge returned a non-finite value" - if name not in {"citation_accuracy", "exact_match", "f1"} + if name not in DETERMINISTIC_METRICS and not math.isfinite(value) else None ), @@ -690,10 +707,10 @@ def main() -> None: # noqa: PLR0915 — length is flat argparse flag declaratio }, valid_count=sum(math.isfinite(value) for value in per_metric[name]), missing_count=sum(not math.isfinite(value) for value in per_metric[name]) - if name == "citation_accuracy" + if name in ANNOTATION_METRICS else 0, error_count=sum(not math.isfinite(value) for value in per_metric[name]) - if name != "citation_accuracy" + if name not in ANNOTATION_METRICS else 0, ) for name, score in scores.items() diff --git a/tests/test_eval_dataset.py b/tests/test_eval_dataset.py index 9890502..a6a9acb 100644 --- a/tests/test_eval_dataset.py +++ b/tests/test_eval_dataset.py @@ -21,6 +21,7 @@ registered_datasets, ) from evals.dataset.schema import DatasetManifest, DatasetRecord +from evals.metrics import score_retrieval_recall from evals.run_evals import _build_rows, _select_records # --- fixture manifest builders --- @@ -372,3 +373,42 @@ def _boom(*_a: Any, **_k: Any) -> Any: records = manifest.split("smoke") rows = _build_rows(records, "http://unused", "", offline=True) assert len(rows) == len(records) + + +def test_scope_fixture_encodes_the_observed_retrieval_failure() -> None: + """#174's regression case: the acute passage is indexed but never retrieved.""" + discover_fixtures() + manifest = load_dataset("localrag-scope", "1.0.0") + recalls = { + record.record_id: score_retrieval_recall( + record.relevant_citation_ids(), + record.offline_retrieved_ids(), + record.citation_texts(), + record.offline_context_texts(), + ).value + for record in manifest.records + } + # The motivating failure, a graded partial, and a control that must not be zero. + assert recalls["single-overvoltage-event-effect"] == 0.0 + assert recalls["single-event-recovery-procedure"] == 0.5 + assert recalls["repeated-overvoltage-events-effect"] == 1.0 + + +def test_offline_retrieved_ids_default_to_every_citation() -> None: + """Existing fixtures declare no override and must keep their previous meaning.""" + discover_fixtures() + record = load_dataset("localrag-core", "1.0.0").records[0] + assert record.offline_retrieved_ids() == [c.citation_id for c in record.citations] + + +def test_offline_retrieved_ids_must_reference_declared_citations() -> None: + with pytest.raises(ValidationError, match="offline_retrieved_citation_ids"): + DatasetRecord.model_validate( + { + "record_id": "r1", + "question": "q", + "reference_answer": "a", + "citations": [{"citation_id": "c1", "source": "s", "text": "t"}], + "offline_retrieved_citation_ids": ["nope"], + } + ) diff --git a/tests/test_eval_metrics.py b/tests/test_eval_metrics.py index ff3cd0c..5af030f 100644 --- a/tests/test_eval_metrics.py +++ b/tests/test_eval_metrics.py @@ -7,8 +7,10 @@ aggregate_cases, exact_match, f1, + resolve_retrieved_citations, score_citation_accuracy, score_judge_metric, + score_retrieval_recall, ) @@ -68,3 +70,50 @@ def test_citation_accuracy_scores_annotation_ids() -> None: result = score_citation_accuracy("answer", ["c1", "missing"], ["c1", "c2"]) assert result.value == 0.5 assert result.status == "complete" + + +def test_retrieval_recall_prefers_the_id_join_when_namespaces_are_shared() -> None: + """Offline the retrieved IDs are dataset citation IDs, so the join is exact.""" + result = score_retrieval_recall( + ["c1", "c2"], + ["c2", "c3"], + {"c1": "acute passage", "c2": "chronic passage", "c3": "other passage"}, + ["chronic passage", "other passage"], + ) + assert result.value == 0.5 + assert result.status == "complete" + + +def test_retrieval_recall_falls_back_to_text_when_ids_share_no_namespace() -> None: + """A live run returns corpus chunk hashes; comparing them to citation IDs is meaningless.""" + result = score_retrieval_recall( + ["c1", "c2"], + ["sha256:aaaa", "sha256:bbbb"], + {"c1": "the clamp cools for ninety seconds", "c2": "calibration drift accumulates"}, + ["Section 4. The clamp cools for ninety seconds, then readings resume."], + ) + assert result.value == 0.5 + + +def test_retrieval_recall_matches_a_citation_inside_a_larger_chunk() -> None: + """Chunk boundaries cut passages, so exact containment alone would under-count.""" + hits = resolve_retrieved_citations( + ["c1"], + None, + {"c1": "readings are unavailable for approximately ninety seconds"}, + ["Readings are unavailable, in practice, for ninety seconds or so while it cools."], + ) + assert hits == {"c1"} + + +def test_retrieval_recall_is_unavailable_rather_than_zero_without_annotations() -> None: + result = score_retrieval_recall(None, ["c1"], {"c1": "text"}, ["text"]) + assert result.status == "unavailable" + assert result.value is None + + +def test_retrieval_recall_is_unavailable_when_neither_join_can_be_made() -> None: + """No shared IDs and no retrieved text is undecidable, not a score of zero.""" + result = score_retrieval_recall(["c1"], ["sha256:aaaa"], {"c1": "text"}, []) + assert result.status == "unavailable" + assert result.value is None diff --git a/tests/test_failure_analysis.py b/tests/test_failure_analysis.py index 0c52133..6d95628 100644 --- a/tests/test_failure_analysis.py +++ b/tests/test_failure_analysis.py @@ -123,3 +123,29 @@ def test_redaction_does_not_export_answer_or_context() -> None: payload = result.model_dump_json() assert "SECRET DOCUMENT" not in payload assert "PRIVATE CONTENT" not in payload + + +def test_context_omission_can_fire_when_offline_ids_exclude_a_relevant_citation() -> None: + """Offline this label could never fire: retrieved IDs were the citation list itself.""" + result = classify_case( + artifact( + retrieved_ids=["c-chronic"], + relevant_citation_ids=["c-acute"], + citation_texts={"c-acute": "acute passage", "c-chronic": "chronic passage"}, + retrieved_text=["chronic passage"], + ) + ) + assert "context_omission" in result.labels + + +def test_context_omission_does_not_fire_when_live_ids_are_corpus_hashes() -> None: + """Live this label fired for every record: chunk hashes never equal citation IDs.""" + result = classify_case( + artifact( + retrieved_ids=["sha256:aaaa"], + relevant_citation_ids=["c-acute"], + citation_texts={"c-acute": "the clamp cools for ninety seconds"}, + retrieved_text=["Section 4. The clamp cools for ninety seconds, then readings resume."], + ) + ) + assert "context_omission" not in result.labels