Skip to content

fix: calibrate generated risk estimates - #4

Open
cr-shah wants to merge 1 commit into
mainfrom
codex/calibrate-probabilities
Open

cr-shah wants to merge 1 commit into
mainfrom
codex/calibrate-probabilities

Conversation

@cr-shah

@cr-shah cr-shah commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Finding

The demo and PaySim workflows fed uncalibrated model outputs into expected-loss ranking even though the architecture and model card describe calibrated probabilities. The existing prefit helper also used an API removed by the locked scikit-learn version, and reporting could reselect a model from held-out test performance.

Fix

  • select the base model only from chronological validation average precision
  • fit sigmoid calibration on validation data with current and 1.5-compatible scikit-learn paths
  • require reporting to use the validation-selected calibrated model
  • regenerate all affected report artifacts through the project command and synchronize README metrics
  • add regression checks for prefit calibration and generated-result documentation

Verification

  • Ruff: passed
  • pytest: 60 passed
  • uv lock check offline: passed
  • installed dependency compatibility: passed
  • two fresh 12,000-row demo runs matched all generated JSON, CSV, and PNG artifacts byte-for-byte

This is separate from open PRs #1-#3 and does not touch their files.

@cr-shah cr-shah added the bug Something isn't working label Sep 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant