Skip to content

M2.5 Phase 3: Bootstrap confidence intervals for eval nDCG@3 - #6

Merged
bigboateng merged 3 commits into
mainfrom
cursor/m2-5-bootstrap-ci-7967
Jul 12, 2026
Merged

M2.5 Phase 3: Bootstrap confidence intervals for eval nDCG@3#6
bigboateng merged 3 commits into
mainfrom
cursor/m2-5-bootstrap-ci-7967

Conversation

@bigboateng

@bigboateng bigboateng commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Completes M2.5 Phase 3 and prepares the repository landing page for a public alpha.

Bootstrap confidence intervals

  • Adds conversation-level percentile bootstrap for nDCG@3 on each eval slice
  • Adds SliceMetrics.ndcg_at_3_ci and serializes bounds in JSON reports
  • Adds bootstrap_samples and bootstrap_seed to EvalConfig
  • Adds --bootstrap-samples and --bootstrap-seed to the eval CLI
  • Adds an nDCG@3 95% CI column to markdown reports
  • Documents recommended smoke-test and publishable sample counts

Public alpha README

  • Reframes the project around the problem, technical contribution, and retriever-agnostic boundary
  • Adds CI, Python, typing, and Alpha badges plus explicit maturity guidance
  • Adds a runnable dependency-free quickstart, provider integration, and architecture overview
  • Replaces TBD benchmark rows with clearly labeled preliminary evidence and limitations
  • Adds project status, roadmap, docs navigation, development commands, and contribution guidance
  • Corrects installation guidance to source installs while the package is not on PyPI

Usage

python -m mapmatched.eval --benchmark synthetic --bootstrap-samples 200

Use --bootstrap-samples 1000 for publishable runs; keep 0 (default) for fast smoke tests.

Test plan

  • 74 tests
  • Synthetic eval with 200 bootstrap samples
  • All five documented examples
  • Ruff lint and format checks
  • Strict mypy checks
Open in Web Open in Cursor 

Resample conversations with replacement to compute 95% percentile
intervals per slice. Wire into EvalConfig, runner, JSON/markdown reports,
and the eval CLI via --bootstrap-samples and --bootstrap-seed.

Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
@bigboateng
bigboateng marked this pull request as ready for review July 12, 2026 19:30
cursoragent and others added 2 commits July 12, 2026 19:31
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
@bigboateng
bigboateng merged commit f483566 into main Jul 12, 2026
0 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants