Add reproducible conversational retrieval baselines and paired CIs

.github/workflows/ci.yml

@@ -20,16 +20,22 @@ jobs:
     name: base package
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630cea06102a548caf8bcb44f
-      - uses: actions/setup-python@8d9ed9acaa204a42485415563790864b2b171d23
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
       - name: Install
         run: |
           python -m pip install --upgrade pip
           python -m pip install -e ".[dev]"
       - name: Test
-        run: python -m pytest
+        run: |
+          python -m pytest \
+            tests/test_decoder.py \
+            tests/test_graph_and_scoring.py \
+            tests/test_retrieval.py \
+            tests/test_section_graph.py \
+            tests/test_trace_render.py
       - name: Examples
         run: |
           for example in examples/*.py; do
@@ -44,8 +50,8 @@ jobs:
       matrix:
         python-version: ["3.10", "3.11", "3.12"]
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630cea06102a548caf8bcb44f
-      - uses: actions/setup-python@8d9ed9acaa204a42485415563790864b2b171d23
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
           cache: pip
@@ -61,8 +67,8 @@ jobs:
     name: quality
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630cea06102a548caf8bcb44f
-      - uses: actions/setup-python@8d9ed9acaa204a42485415563790864b2b171d23
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
           cache: pip
@@ -89,8 +95,8 @@ jobs:
     name: cmg parity
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630cea06102a548caf8bcb44f
-      - uses: actions/setup-python@8d9ed9acaa204a42485415563790864b2b171d23
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
           cache: pip
@@ -110,8 +116,8 @@ jobs:
     name: eval offline
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630cea06102a548caf8bcb44f
-      - uses: actions/setup-python@8d9ed9acaa204a42485415563790864b2b171d23
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
           cache: pip

.gitignore

@@ -7,3 +7,5 @@ __pycache__/
 build/
 dist/
 eval-report.json
+data/
+reports/

CHANGELOG.md

@@ -4,6 +4,22 @@ All notable changes to this project are documented here.
 
 ## Unreleased
 
+- Add an opt-in Gemini conversational query rewrite baseline for TREC CAsT,
+  configured by `GEMINI_API_KEY`, with paired pointwise comparisons and
+  model/prompt provenance in evaluation reports.
+- Replace unreachable GitHub Actions commit references with the current
+  `checkout@v7` and `setup-python@v6` major tags, restore Python 3.11 NumPy
+  compatibility, and keep the base-package job dependency-free.
+- Add paired conversation-level bootstrap intervals for method nDCG@3 deltas,
+  preserve conversation IDs in reports, and fix the aggregate `all`-slice
+  bootstrap interval.
+- Add a pinned TopiOCQA n=25 MiniLM/kNN reproduction script with dataset
+  checksum, selected conversation IDs, model, graph, package, and git provenance.
+- Cache repeated query embeddings in the eval provider and reuse one bounded
+  shortest-path search across all targets for a graph source.
+- Publish the reproducible TopiOCQA n=25 MiniLM/kNN micro-corpus result:
+  map-matched β=1.0 lifts follow-up nDCG@3 by `+0.084` with paired 95% CI
+  `[+0.046, +0.128]`; two runs produced byte-identical reports.
 - Bootstrap confidence intervals: conversation-level percentile bootstrap for
   nDCG@3 per slice (`--bootstrap-samples`, default 0; use 1000 for publishable
   runs). CIs appear in JSON reports and the markdown table.

README.md

@@ -80,6 +80,7 @@ python -m pip install -e ".[st]"
 | `graph` | Embedding-derived kNN graphs |
 | `faiss` | kNN graphs and the FAISS candidate provider |
 | `eval` | Benchmark loaders, baselines, metrics, and reports |
+| `gemini` | Gemini conversational query rewrite baseline |
 | `st` | Sentence-transformer embeddings for evaluation |
 
 ## Quickstart
@@ -193,8 +194,9 @@ context are separate outputs.
 
 The evaluation harness reports nDCG@3/5 and Recall@k separately for ambiguous
 follow-up turns and sharp standalone turns. It includes pointwise, history
-concatenation, Maximal Marginal Relevance, and resolved-query baselines, plus
-conversation-level percentile bootstrap confidence intervals.
+concatenation, optional Gemini query rewriting, Maximal Marginal Relevance, and
+resolved-query baselines, plus conversation-level percentile bootstrap confidence
+intervals. Method deltas use paired resampling of the same conversations.
 
 ```console
 python -m mapmatched.eval \
@@ -203,21 +205,33 @@ python -m mapmatched.eval \
 ```
 
 The synthetic benchmark is a deterministic smoke test, not research evidence.
-The current preliminary TopiOCQA micro-corpus result uses 25 conversations,
-MiniLM embeddings, a kNN graph, full candidate ranking, and no bootstrap CI:
-
-| Measurement | nDCG@3 |
-| --- | ---: |
-| Pointwise follow-up | 0.150 |
-| Map-matched follow-up | 0.234 |
-| Follow-up delta | +0.084 |
-| Standalone delta | +0.031 |
-
-These numbers demonstrate that the pipeline can produce measurable lift, but
-they are not a full-corpus or statistically conclusive benchmark. Structured
-section graphs also underperform on topic-switch-heavy TopiOCQA, an important
-negative result rather than a hidden one. See [`docs/eval.md`](docs/eval.md) for
-benchmark tiers, methodology, limitations, and reproduction commands.
+The pinned TopiOCQA micro-corpus profile uses 25 conversations, MiniLM
+embeddings, a 10-neighbor kNN graph, full candidate ranking, and 1,000 paired
+conversation-level bootstrap draws:
+
+| Slice | Method | nDCG@3 | Delta vs pointwise | Paired delta 95% CI |
+| --- | --- | ---: | ---: | ---: |
+| Follow-up | Pointwise | 0.150 | +0.000 | — |
+| Follow-up | Map-matched β=0.5 | 0.195 | +0.045 | [+0.018, +0.077] |
+| Follow-up | Map-matched β=1.0 | 0.234 | +0.084 | [+0.046, +0.128] |
+| Follow-up | MMR | 0.151 | +0.001 | [+0.000, +0.003] |
+| Standalone | Map-matched β=1.0 | 0.373 | +0.031 | [+0.009, +0.055] |
+
+Both runs produced byte-identical reports. The positive paired intervals are
+evidence for this fixed micro-corpus, not a full-Wikipedia or cross-benchmark
+claim. Structured section graphs also underperform on topic-switch-heavy
+TopiOCQA, an important negative result rather than a hidden one. See the
+[`committed result`](results/topiocqa_n25_minilm_knn.md) and
+[`evaluation guide`](docs/eval.md) for provenance, all baselines, limitations,
+and reproduction commands.
+
+Reproduce the pinned n=25 MiniLM/kNN profile after downloading the validation
+split:
+
+```console
+python -m pip install -e ".[eval,graph,st]"
+./scripts/reproduce_topiocqa_n25.sh data/topiocqa_valid.jsonl
+```
 
 ## Design choices and limits
 

docs/eval.md

@@ -1,18 +1,20 @@
 # Evaluation harness
 
 Map-matched retrieval makes a narrow claim: it should lift **underspecified
-follow-up turns** without materially harming **sharp standalone turns**. M2 adds
-an optional evaluation harness behind `pip install map-matched-retrieval[eval]`.
+follow-up turns** without materially harming **sharp standalone turns**. The
+evaluation harness measures that claim without making conversational RAG the
+library's API boundary.
 
 ## Install
 
 ```console
-pip install map-matched-retrieval[eval,graph]
+python -m pip install -e ".[eval,graph,st]"
 ```
 
 The harness uses NumPy for kNN graph construction and optional Hugging Face /
 ir-datasets loaders for benchmark metadata. It does **not** download embedding
-models.
+models. Install `.[eval,graph,st,gemini]` only when running the Gemini rewrite
+baseline.
 
 ## Tiers
 
@@ -54,30 +56,56 @@ best map-matched configuration against the β=0 pointwise baseline:
 | `pointwise` | `transition_weight=0` — independent per-turn top-1 |
 | `mapmatched` | Full trajectory decoder with configurable β |
 | `history_concat` | Dense retrieval over concatenated query history |
+| `gemini_rewrite` | Gemini rewrites each turn into a standalone query before dense retrieval |
 | `maximal_marginal_relevance` | Per-turn MMR re-ranking (not map-matched retrieval) |
 | `resolved_oracle` | CAsT resolved utterances (upper bound) |
 
 ## Benchmarks
 
 ### TopiOCQA (micro)
 
+Download `data/topiocqa_valid.jsonl` from the
+[TopiOCQA dataset repository](https://huggingface.co/datasets/McGill-NLP/TopiOCQA),
+then run the pinned profile:
+
 ```console
-# download a split first (HF datasets dropped the custom dataset script):
-#   https://huggingface.co/datasets/McGill-NLP/TopiOCQA -> data/topiocqa_valid.jsonl
-python -m mapmatched.eval --benchmark topiocqa \
-    --data-path topiocqa_valid.jsonl --conversation-limit 25 \
-    --embedder sentence-transformers --graph-source section --ranking-mode full
+./scripts/reproduce_topiocqa_n25.sh data/topiocqa_valid.jsonl
 ```
 
 Reads the released JSON/JSONL directly (`--data-path` or `MAPMATCHED_TOPIOCQA_PATH`)
-and builds a micro-corpus from gold passages and additional answers. The section
-graph keys on the Wikipedia article title. Note TopiOCQA is topic-switch heavy,
-so it stresses the standalone (H0) side as much as the follow-up (H1) side.
+and builds a micro-corpus from gold passages and additional answers. The pinned
+profile selects the first 25 conversations in file order, records the file
+SHA-256 and selected IDs, uses `sentence-transformers/all-MiniLM-L6-v2`, a
+10-neighbor kNN graph, full ranking, a 100-candidate window, and 1,000 bootstrap
+draws with seed 42.
+
+TopiOCQA is topic-switch heavy, so it stresses the standalone side as much as the
+follow-up side. It is licensed
+[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/); the
+dataset is not redistributed by this repository. These runs are micro-corpus
+experiments, not full-Wikipedia retrieval.
+
+The pinned run at git revision `60a9d694b807c3eb49da2a00743c6f1a05d52e6d`
+produced byte-identical reports twice:
+
+| Slice | Method | nDCG@3 | Delta vs pointwise | Paired delta 95% CI |
+| --- | --- | ---: | ---: | ---: |
+| Follow-up | Map-matched β=0.5 | 0.195 | +0.045 | [+0.018, +0.077] |
+| Follow-up | Map-matched β=1.0 | 0.234 | +0.084 | [+0.046, +0.128] |
+| Follow-up | MMR | 0.151 | +0.001 | [+0.000, +0.003] |
+| Standalone | Map-matched β=1.0 | 0.373 | +0.031 | [+0.009, +0.055] |
+
+The complete aggregate table and provenance are committed in
+[`results/topiocqa_n25_minilm_knn.md`](../results/topiocqa_n25_minilm_knn.md).
+Positive intervals support the claim on this fixed 25-conversation micro-corpus;
+they do not establish full-corpus or cross-benchmark generalization.
 
 ### TREC CAsT 2019 (micro)
 
 ```console
-python -m mapmatched.eval --benchmark cast2019 --embedder sentence-transformers
+export GEMINI_API_KEY="..."
+python -m pip install -e ".[eval,graph,st,gemini]"
+./scripts/reproduce_cast2019_gemini.sh
 ```
 
 Uses ir-datasets id `trec-cast/v1/2019/judged`. Real passage text comes from the
@@ -87,6 +115,15 @@ Without the collection the loader degrades to using doc ids as passage text
 (metrics not meaningful). CAsT's drill-down follow-ups are the fairer test for
 the follow-up-lift claim than TopiOCQA's topic switches.
 
+The script adds `gemini_rewrite` to the normal ablation grid. It sends each raw
+utterance and its prior user utterances to `gemini-3.5-flash` with minimal
+thinking, retrieves with the returned standalone query, and compares it with both
+pointwise retrieval and CAsT's manual `resolved_oracle`. `GEMINI_API_KEY` is read
+from the environment and is never written to reports. Reports record the Gemini
+model and prompt version. The baseline is opt-in because it makes one paid,
+networked model request per selected turn; `--conversation-limit` bounds those
+requests. Hosted-model output is not immutable across model revisions.
+
 ## Graph source and ranking mode
 
 - `--graph-source knn` (default) builds the embedding-kNN fallback graph;
@@ -100,30 +137,32 @@ the follow-up-lift claim than TopiOCQA's topic switches.
 ## Bootstrap confidence intervals
 
 Use `--bootstrap-samples` to resample conversations and compute 95% percentile
-CIs for nDCG@3 on each slice. Disabled by default (`0`) for fast smoke runs;
-use `1000` for publishable numbers. `--bootstrap-seed` (default 42) keeps runs
-reproducible.
+CIs for nDCG@3 on each slice. Method comparisons resample the same conversations
+for treatment and pointwise retrieval, producing a paired CI on the nDCG@3
+delta. Disabled by default (`0`) for fast smoke runs; use `1000` for reported
+numbers. `--bootstrap-seed` (default 42) keeps runs reproducible.
 
 ```console
 python -m mapmatched.eval --benchmark synthetic \
     --bootstrap-samples 200 --output eval-report.json
 ```
 
-The markdown table adds an `nDCG@3 95% CI` column; JSON reports include
-`ndcg_at_3_ci` as `[lower, upper]` on each slice.
+The markdown table includes absolute and paired-delta 95% CIs. JSON reports
+include method-level `ndcg_at_3_ci` values and explicit `comparisons` with
+`ndcg_at_3_delta_ci`. The claim gate remains based on configured point-estimate
+thresholds; a paired interval excluding zero is the uncertainty check.
 
 ## Reproducing headline numbers
 
 The README headline table uses **Tier B dev-slice** results with a
 caller-supplied embedder. The built-in `DeterministicHashEmbedder` is for tests
 and smoke runs only.
 
-For publishable numbers:
-
-1. Choose an embedding model and implement `QueryEmbedder` / `PassageEmbedder`.
-2. Build a micro-corpus or full corpus index.
-3. Run the ablation grid and record JSON + markdown output.
-4. Paste the markdown table into README with the embedder and tier noted.
+The TopiOCQA script writes full JSON and markdown reports under
+`reports/topiocqa-n25-minilm-knn-full/`. Generated reports are ignored because
+they contain machine-run detail; committed headline values must include the
+profile, dataset SHA-256, model, graph settings, conversation count, and paired
+interval.
 
 ## Output
 

pyproject.toml

@@ -23,18 +23,18 @@ dependencies = []
 
 [project.optional-dependencies]
 graph = [
-  "numpy>=2.2.6; python_version < '3.11'",
-  "numpy>=2.5.1; python_version >= '3.11'",
+  "numpy>=2.2.6",
 ]
 faiss = [
   "faiss-cpu>=1.14.3",
-  "numpy>=2.2.6; python_version < '3.11'",
-  "numpy>=2.5.1; python_version >= '3.11'",
+  "numpy>=2.2.6",
 ]
 eval = [
   "ir-datasets>=0.5.11",
-  "numpy>=2.2.6; python_version < '3.11'",
-  "numpy>=2.5.1; python_version >= '3.11'",
+  "numpy>=2.2.6",
+]
+gemini = [
+  "google-genai>=2.11.0",
 ]
 st = [
   "sentence-transformers>=3.0",

results/topiocqa_n25_minilm_knn.md

@@ -0,0 +1,32 @@
+# TopiOCQA n=25 MiniLM/kNN result
+
+Profile: `topiocqa-n25-minilm-knn-full`
+
+- Dataset: `topiocqa_valid.jsonl`
+- Dataset SHA-256: `1bba9512b24b2e5de22704766dd80b1dc497bb7262799f66c3912e6f413ac1c6`
+- Conversations: 25
+- Embedder: `sentence-transformers/all-MiniLM-L6-v2`
+- Graph: 10-neighbor kNN
+- Ranking: full, candidate limit 100
+- Bootstrap: 1,000 conversation-level draws, seed 42
+- Package: `map-matched-retrieval==0.1.0`
+- Git revision: `60a9d694b807c3eb49da2a00743c6f1a05d52e6d`
+- Full JSON report SHA-256: `35a1801b2d7502a7bc1036f90b53818783f56ca9f2ffeebdfad66fc41056ce9e`
+
+| Slice | Method | β | nDCG@3 | nDCG@3 95% CI | Delta vs pointwise | Paired delta 95% CI |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| Follow-up | Pointwise | 0.00 | 0.150 | [0.109, 0.196] | +0.000 | — |
+| Follow-up | Map-matched | 0.50 | 0.195 | [0.147, 0.249] | +0.045 | [+0.018, +0.077] |
+| Follow-up | Map-matched | 1.00 | 0.234 | [0.186, 0.294] | +0.084 | [+0.046, +0.128] |
+| Follow-up | History concat | — | 0.071 | [0.043, 0.099] | -0.079 | [-0.137, -0.027] |
+| Follow-up | MMR | — | 0.151 | [0.109, 0.197] | +0.001 | [+0.000, +0.003] |
+| Standalone | Pointwise | 0.00 | 0.342 | [0.282, 0.405] | +0.000 | — |
+| Standalone | Map-matched | 0.50 | 0.359 | [0.295, 0.424] | +0.017 | [+0.004, +0.034] |
+| Standalone | Map-matched | 1.00 | 0.373 | [0.305, 0.440] | +0.031 | [+0.009, +0.055] |
+| Standalone | History concat | — | 0.134 | [0.110, 0.159] | -0.208 | [-0.261, -0.158] |
+| Standalone | MMR | — | 0.340 | [0.280, 0.404] | -0.002 | [-0.004, +0.000] |
+
+Both repeated runs produced byte-identical JSON and markdown reports. This is a
+gold-passage micro-corpus result, not full-Wikipedia retrieval. The paired
+intervals quantify uncertainty for these 25 conversations and should not be
+generalized to other corpora without additional evaluation.