Skip to content

sampler: align GMM selection with the paper - #17

Merged
huangyz0918 merged 1 commit into
mainfrom
fix/paper-source-selection
Sep 16, 2026
Merged

huangyz0918 merged 1 commit into
mainfrom
fix/paper-source-selection

Conversation

@huangyz0918

Copy link
Copy Markdown
Member

Summary

Correct source-selection differences found by comparing the implementation with Sections 2.2.3 and E.2.2鈥揈.3 of the paper: https://arxiv.org/html/2604.23099v1

  • Cluster PCA-reduced per-question score vectors, retaining 95% variance. Benchmark mean/std summaries discarded which questions models actually failed on.
  • Always exclude the target benchmark from reference features, including when explicitly listed.
  • Include one-component GMMs in the BIC search and reject degenerate or nonconverged fits.
  • Abstain with a clear ValueError when fewer than three source models in the target cluster are available on the target benchmark. Do not silently substitute unrelated models from other clusters.
  • Preserve model-name prefixes correctly and apply the existing DICES score conversion consistently.

Compatibility

The return signature is unchanged. Source choices and estimates can change because the feature construction is corrected. Small clusters now explicitly abstain; this is intentional, not a fallback to all sources. Missing models on a reference benchmark retain mean imputation, now per question; explicitly present score columns must be finite.

Verification

  • Existing repository suite: 123 passed, including PyTorch paths.
  • Local probes cover equal-mean/opposite-error patterns, PCA, missing-model imputation, DICES conversion, target exclusion, singleton clusters, K=1, and nonconvergence.
  • Bundled SVAMP and StrategyQA select three sources for gemini25_flash; GSM8K explicitly abstains for its empty source cluster.
  • Only production bug fixes are included. Additional validation code stays local; no credentials or live API calls are involved.

@huangyz0918
huangyz0918 merged commit 1e971f1 into main Sep 16, 2026
7 checks passed
@huangyz0918
huangyz0918 deleted the fix/paper-source-selection branch September 16, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant