Skip to content

Make TREC CAsT evaluation tractable and publish Gemini baseline - #8

Draft
bigboateng wants to merge 15 commits into
mainfrom
cursor/cast-gemini-eval-2551
Draft

Make TREC CAsT evaluation tractable and publish Gemini baseline#8
bigboateng wants to merge 15 commits into
mainfrom
cursor/cast-gemini-eval-2551

Conversation

@bigboateng

@bigboateng bigboateng commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to merged PR #7, driven by execution of the pinned TREC CAsT profile.

  • Make the combined MS MARCO/TREC CAR judged-passage workflow tractable
  • Add durable, paced, retryable Gemini rewrite prefetch
  • Use stable gemini-3.1-flash-lite
  • Omit the resolved oracle when resolved queries are unavailable
  • Publish the completed CAsT result and README headline table
  • Fix CI's unavailable build>=1.5.1 requirement

Result

20 judged topics, 173 turns, 21,726 judged passages, MiniLM, 10-neighbor kNN, candidate limit 100, and 1,000 paired bootstrap draws.

  • Pointwise follow-up nDCG@3: 0.325
  • Map-matched β=1.0: 0.352, delta +0.027, 95% CI [+0.008, +0.049]
  • Gemini Flash-Lite rewrite: 0.516, delta +0.191, 95% CI [+0.078, +0.296]
  • Claim gate: pass

These are judged-passage micro-corpus results, not full-corpus retrieval evidence.

Runtime improvements

  • TREC CAR dependency and one-shot docstore failure handling
  • Blockwise NumPy kNN graph construction
  • SciPy sparse shortest paths with compact bounded caching
  • β=0 graph bypass
  • Vectorized full-corpus scoring
  • Gemini JSON checkpoints, prefetch, pacing, request timeouts, and transient retries

Test plan

  • 88 tests passed, 1 skipped
  • Ruff and format checks
  • Strict mypy
  • Source and wheel build
  • Corrected cached CAsT/Gemini profile completed
  • Result artifact and README table
  • All seven GitHub Actions checks pass
Open in Web Open in Cursor 

cursoragent and others added 15 commits July 13, 2026 19:59
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
Co-authored-by: bigboateng <bigboateng@users.noreply.github.com>
@cursor cursor Bot changed the title Make TREC CAsT evaluation tractable at full micro-corpus scale Make TREC CAsT evaluation tractable and publish Gemini baseline Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants