fix: calibrate generated risk estimates

README.md

@@ -15,8 +15,8 @@ RiskQueue scores transactions, quantifies the dollars exposed, and places the mo
 
 | Rare-event ranking | Operational outcome at 750 reviews | Decision policy | Engineering |
 |---|---|---|---|
-| **0.130 average precision** | **97.1% fraud value captured** with expected-loss ranking | **$9,034 modeled cost** at the validation-selected cost threshold | **10.90 ms p95** for batch 1,000 |
-| 3.49% demo fraud rate | vs. 95.1% using probability alone | vs. $34,205 at threshold 0.50 | In-process local benchmark; 57 tests pass |
+| **0.113 average precision** | **96.9% fraud value captured** with expected-loss ranking | **$8,280 modeled cost** at the validation-selected cost threshold | **10.90 ms p95** for batch 1,000 |
+| 3.49% demo fraud rate | vs. 92.3% using probability alone | vs. $38,906 at threshold 0.50 | In-process local benchmark; 60 tests pass |
 
 ## Results at a glance
 
@@ -28,21 +28,21 @@ Fraud is a small share of transactions, so accuracy is not a useful headline. Pr
 
 | Held-out demo model | Average precision | ROC-AUC | Brier score |
 |---|---:|---:|---:|
-| Logistic Regression | 0.097 | **0.749** | 0.158 |
-| Histogram Gradient Boosting demo fallback | **0.130** | 0.745 | **0.033** |
+| Logistic Regression + sigmoid calibration | 0.097 | **0.749** | 0.033 |
+| Histogram Gradient Boosting demo fallback + sigmoid calibration | **0.113** | 0.734 | **0.033** |
 
 ### 2. What happens when analysts can review only a limited queue?
 
 ![Analyst capacity versus fraud value](artifacts/figures/capacity_value.png)
 
-The chart measures how much fraudulent transaction value appears in the first *k* reviews. At 750 demo reviews, probability ranking captures 95.1% of fraudulent value, while expected-loss ranking captures 97.1%. The comparison matters because a slightly less likely but much larger transaction can deserve earlier review when the objective is dollars protected.
+The chart measures how much fraudulent transaction value appears in the first *k* reviews. At 750 demo reviews, probability ranking captures 92.3% of fraudulent value, while expected-loss ranking captures 96.9%. The comparison matters because a slightly less likely but much larger transaction can deserve earlier review when the objective is dollars protected.
 
 | Reviews | Probability queue | Expected-loss queue | Improvement |
 |---:|---:|---:|---:|
-| 100 | 53.6% | **68.3%** | +14.7 points |
-| 250 | 79.0% | **85.5%** | +6.5 points |
-| 500 | 87.2% | **94.2%** | +7.0 points |
-| 750 | 95.1% | **97.1%** | +2.0 points |
+| 100 | 55.5% | **68.3%** | +12.8 points |
+| 250 | 76.9% | **84.1%** | +7.2 points |
+| 500 | 86.9% | **94.7%** | +7.8 points |
+| 750 | 92.3% | **96.9%** | +4.6 points |
 
 ### 3. Why is 0.50 not automatically the right threshold?
 
@@ -52,9 +52,9 @@ The score estimates risk; it does not decide the operating policy by itself. The
 
 | Policy | Threshold | Reviews | Precision | Recall | Modeled cost |
 |---|---:|---:|---:|---:|---:|
-| Default | 0.50 | 2 | 100.0% | 3.3% | $34,205 |
-| F1-optimal | 0.15 | 78 | 16.7% | 21.3% | $18,613 |
-| Cost-optimized | **0.03** | 291 | 9.3% | **44.3%** | **$9,034** |
+| Default | 0.50 | 0 | 0.0% | 0.0% | $38,906 |
+| F1-optimal | 0.12 | 75 | 16.0% | 19.7% | $18,628 |
+| Cost-optimized | **0.04** | 312 | 8.7% | **44.3%** | **$8,280** |
 
 ### 4. Would the system notice changed transaction behavior?
 
@@ -90,6 +90,8 @@ flowchart LR
 
 The primary split uses whole time steps: earliest 70% for training, next 15% for validation and final 15% for test. History features update state only after a row is transformed. `isFlaggedFraud` and the four balance columns never enter the model feature list.
 
+After validation average precision selects the base model, sigmoid calibration is fitted on that chronological validation period. Thresholds are selected from the calibrated validation probabilities, and the calibrated model is evaluated once on the final test period.
+
 ## Explore in three minutes
 
 ```bash

artifacts/figures/capacity_results.csv

@@ -7,16 +7,16 @@ probability,500,500.0,0.076,0.6229508196721312,33823.16,0.8693496268468743
 probability,750,750.0,0.06,0.7377049180327869,35914.66,0.9231070151142697
 probability,1000,1000.0,0.052,0.8524590163934426,38237.840000000004,0.9828192260992316
 expected_loss,25,25.0,0.24,0.09836065573770492,16538.129999999997,0.4250761059654123
-expected_loss,50,50.0,0.16,0.13114754098360656,20359.809999999998,0.5233039499021752
-expected_loss,100,100.0,0.15,0.2459016393442623,27137.33,0.697505132847448
-expected_loss,250,250.0,0.092,0.3770491803278688,31930.4,0.8207004113474742
+expected_loss,50,50.0,0.18,0.14754098360655737,22220.8,0.5711365877179726
+expected_loss,100,100.0,0.14,0.22950819672131148,26564.61,0.6827846301419721
+expected_loss,250,250.0,0.088,0.36065573770491804,32707.73,0.8406799622066157
 expected_loss,500,500.0,0.072,0.5901639344262295,36863.61,0.9474976790379345
-expected_loss,750,750.0,0.058666666666666666,0.7213114754098361,37817.25,0.972008889053387
-expected_loss,1000,1000.0,0.052,0.8524590163934426,38650.560000000005,0.9934272821765538
+expected_loss,750,750.0,0.056,0.6885245901639344,37719.46,0.9694954130798423
+expected_loss,1000,1000.0,0.049,0.8032786885245902,38517.51,0.9900075257773298
 expected_review_value,25,25.0,0.24,0.09836065573770492,16538.129999999997,0.4250761059654123
-expected_review_value,50,50.0,0.16,0.13114754098360656,20359.809999999998,0.5233039499021752
-expected_review_value,100,100.0,0.15,0.2459016393442623,27137.33,0.697505132847448
-expected_review_value,250,250.0,0.092,0.3770491803278688,31930.4,0.8207004113474742
+expected_review_value,50,50.0,0.18,0.14754098360655737,22220.8,0.5711365877179726
+expected_review_value,100,100.0,0.14,0.22950819672131148,26564.61,0.6827846301419721
+expected_review_value,250,250.0,0.088,0.36065573770491804,32707.73,0.8406799622066157
 expected_review_value,500,500.0,0.072,0.5901639344262295,36863.61,0.9474976790379345
-expected_review_value,750,750.0,0.058666666666666666,0.7213114754098361,37817.25,0.972008889053387
-expected_review_value,1000,1000.0,0.052,0.8524590163934426,38650.560000000005,0.9934272821765538
+expected_review_value,750,750.0,0.056,0.6885245901639344,37719.46,0.9694954130798423
+expected_review_value,1000,1000.0,0.049,0.8032786885245902,38517.51,0.9900075257773298

artifacts/figures/error_analysis.csv

@@ -1,10 +1,10 @@
 dimension,segment,transactions,fraud_cases,recall,false_positives,fraud_value
-type,CASH_IN,186,5,0.2,16,2051.48
-type,CASH_OUT,414,29,0.7241379310344828,255,15869.330000000002
-type,DEBIT,135,1,0.0,4,1198.31
-type,PAYMENT,742,7,0.14285714285714285,101,400.48
-type,TRANSFER,272,19,0.9473684210526315,214,19386.68
-amount_bucket,"(1.0190000000000001, 46.03]",438,11,0.2727272727272727,37,288.7
-amount_bucket,"(113.37, 275.26]",437,12,0.75,157,2225.91
-amount_bucket,"(275.26, 6700.6]",437,27,0.8148148148148148,280,35560.24
-amount_bucket,"(46.03, 113.37]",437,11,0.6363636363636364,116,831.43
+type,CASH_IN,186,5,0.0,0,2051.48
+type,CASH_OUT,414,29,0.41379310344827586,128,15869.330000000002
+type,DEBIT,135,1,0.0,0,1198.31
+type,PAYMENT,742,7,0.14285714285714285,23,400.48
+type,TRANSFER,272,19,0.7368421052631579,134,19386.68
+amount_bucket,"(1.0190000000000001, 46.03]",438,11,0.0,9,288.7
+amount_bucket,"(113.37, 275.26]",437,12,0.3333333333333333,58,2225.91
+amount_bucket,"(275.26, 6700.6]",437,27,0.7407407407407407,194,35560.24
+amount_bucket,"(46.03, 113.37]",437,11,0.2727272727272727,24,831.43