PolicyEngine · juaristi22 · Sep 5, 2026
diff --git a/changelog.d/uk-cgt-open-tail.fixed.md b/changelog.d/uk-cgt-open-tail.fixed.md
@@ -0,0 +1 @@
+Represent the UK CGT open tail with household-weighted Pareto interval means anchored to published cell gains, avoiding random aggregate drift and rounded-count mean errors while preserving taxpayer selection and bounded-band draws.
diff --git a/experiments/receipts/uk-cgt-open-tail.json b/experiments/receipts/uk-cgt-open-tail.json
diff --git a/experiments/uk-cgt-open-tail-quality.md b/experiments/uk-cgt-open-tail-quality.md
@@ -0,0 +1,127 @@
+# UK CGT open-tail source matching
+
+This independent spine experiment starts from
+`bc11803f915fe72afd724515a27a3863035f4efc` (`uk-spine-rebuild-842-850`).
+It addresses open-tail amount instability relevant to #364, #796 and #736.
+The separate SPI income experiment starts from the same commit.
+
+## Plan and mechanism
+
+1. Rebuild the unmodified national spine with pinned source inputs.
+2. Hold taxpayer selection, income/gain bands, household weights, bounded-band
+   draws, losses, zeroes and all non-CGT columns fixed.
+3. In the £5m+ band, use published cell gains divided by the actual carrier
+   mass to define the amount distribution. Apply the existing explicit support
+   shortfall factor and band floor. Suppressed amounts retain the existing mean
+   fallback. Missing carriers or an infeasible floor remain visible residuals.
+4. Partition the fitted Pareto probability distribution into intervals whose
+   masses equal the carriers' relative household weights. Assign each carrier
+   its interval's conditional mean. Held uniforms order intervals and the final
+   unbounded interval is integrated analytically.
+5. Run the same national-only calibration and report both source-level matching
+   and calibrated effects, including UC, income tax and pension pressure.
+
+Independent point draws can make an extremely thin tail a seed lottery. Counts
+rounded to thousands introduce another error: multiplying a cell's inferred
+mean by the actual carrier mass need not recover its published gains. An initial
+experiment integrating the old means increased top-band gains from £24.603bn to
+£29.004bn. That result motivated using the published cell amounts explicitly.
+
+Conditional interval means preserve the specified weighted mean even with one
+carrier. They smooth variation within each represented probability interval;
+they do not identify the true within-cell distribution from aggregate data.
+The Pareto family and existing shape assumption remain modeling choices.
+
+## Comparison contract
+
+Use PolicyEngine-UK 2.92.1, PolicyEngine-Core 3.31.0, the locked environment,
+the full FRS spine and the same declared seeds. Calibrate year 2025 against the
+364 national targets, with 1,500 Adam epochs, `family_equal`, learning rate 0.02,
+seed 0 and maximum weight ratio 10. No target, exclusion or tolerance changes.
+
+The frozen Chronicle commit is `1cab80987a462e00055f259cc56dc6b311c030bf`;
+facts SHA-256 is
+`226358e73e7c449e71a3f6dc91a72e8e3941e0f14265943d81c990edb21c2a6c`.
+The control reproduces the reported v19 loss and pension-band errors.
+
+The CGT workbook remains the pinned 2025 publication, tax year 2023–24,
+SHA-256 `8e75c00bab949348a7238fea6d995f626c85e5d02813b46606dd7fea85e9d0c3`.
+Its live URL now serves different bytes. The exact 11,996-byte file was recovered
+from the [National Archives snapshot](https://webarchive.nationalarchives.gov.uk/ukgwa/20250730164651id_/https://assets.publishing.service.gov.uk/media/6878ac62760bf6cedaf5bd93/Table_3_2025_Size_of_gain_by_income.ods)
+and verified against the existing pin. The source vintage was not changed.
+
+## Findings and limits
+
+The treatment changes only `capital_gains` for 13 of 113,626 person rows.
+Every other person column, all household and benefit-unit tables, weights,
+taxpayer identities and bounded-band gains are held fixed. Pre-calibration
+£5m+ gains fall from £24.603bn to £22.715bn. The latter equals the sum of the
+six published income cells; their £1m difference from the £22.714bn band total
+is retained as source rounding. Top-band taxpayer mass remains 1,934.256.
+
+This is not a solution to #875: tax-year realization spikes, changed tax rates,
+cash-receipt timing and individuals-versus-trusts scope need a separate
+consumer-side alignment decision. It also does not identify age or regional
+joints described in #725, or provide the accrued-gains/realization model in #320.
+The amount correction is an incremental spine improvement; stage gates and
+national calibration do not certify a publishable UK dataset.
+
+## National calibration results
+
+The [full paired receipt](receipts/uk-cgt-open-tail.json) includes every target,
+source and diagnostics hashes, implementation file hashes and terminal failures.
+The runs used the parent commit plus those implementation files; the recorded
+parent code pin alone does not identify an uncommitted experimental treatment.
+
+| Metric | Control | CGT treatment |
+|---|---:|---:|
+| Final loss | 0.0253150 | 0.0253577 |
+| Effective sample size | 5,722.6 | 5,732.0 |
+| Income tax | £290.991bn | £290.897bn |
+| Income-tax error | −12.203% | −12.232% |
+| OBR CGT error | −24.975% | −25.077% |
+| HMRC gains-total error | −0.137% | −0.271% |
+| State pension, £20–30k income band | +26.795% | +26.824% |
+| State pension, £50–70k income band | +32.999% | +33.110% |
+| UC single-with-children error | −33.611% | −33.600% |
+
+The source match improves; national fit does not. The small CGT movement crosses
+the existing 25% target-fit threshold, which is retained as a failure. Both runs
+also fail the inherited pension-band checks and stale-exclusion checks. The
+calibration solve and diagnostics completed, but the terminal battery refused
+to write a calibrated release artifact. No exclusions were added or extended.
+
+The relevant imputation, manifest, source-stage, coverage and graph-parity test
+suites pass, as do repository Ruff and CI test-inventory checks. Unit regressions
+cover unequal weights, carrier permutations, multiple seeds and rounded carrier
+mass. The full spine comparison verifies that only 13 CGT amounts change.
+
+## Reproduction
+
+At the control commit and this branch, use separate output directories and the
+same licensed inputs. `UK_INPUTS` below is the local input root; the builder
+checks the source hashes. The archived CGT file must match the existing pin.
+
+```sh
+uv sync --all-packages --locked --extra uk
+export POPULACE_FIT_N_JOBS=2 POPULACE_FIT_PREDICT_WORKERS=2
+export OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 VECLIB_MAXIMUM_THREADS=2
+uv run python tools/build_uk_frs_spine.py \
+  --frs-raw-dir "$UK_INPUTS/frs_2024_25" \
+  --spi-tab "$UK_INPUTS/spi_2022_23/put2223uk.tab" \
+  --hmrc-ods "$UK_INPUTS/Collated_Tables_3_1_to_3_11_2324.ods" \
+  --cgt-ods "$UK_INPUTS/Table_3_2025_Size_of_gain_by_income.ods" \
+  --was-tab "$UK_INPUTS/was_2006_22/was_round_8_hhold_eul_may_2025_230525.tab" \
+  --lcfs-hh-tab "$UK_INPUTS/lcfs_2023_24/dvhh_ukanon_v2_2023.tab" \
+  --lcfs-person-tab "$UK_INPUTS/lcfs_2023_24/dvper_ukanon_202324_2023.tab" \
+  --etb-tab "$UK_INPUTS/etb_1977_24/householdv2_1977-2024.tab" \
+  --spine-h5 "$UK_RUN/spine.h5"
+```
+
+Build the Chronicle UK bundle and consumer artifact at the stated Chronicle
+commit. Pass its directory and computed facts/manifest SHA-256 values, and the
+spine's computed SHA-256, to `tools/calibrate_uk_national_dataset.py`, with
+`--epochs 1500 --target-weight-rule family_equal` and a `dev-` release ID.
+Use a local developer signing key for this diagnostic run. Preserve the emitted
+diagnostics and gate report even when the command exits nonzero; a terminal
+failure must not be converted into an export or certification success.
diff --git a/packages/microcosm-build/src/microcosm/build/uk/cgt_source_stages.json b/packages/microcosm-build/src/microcosm/build/uk/cgt_source_stages.json
@@ -99,7 +99,7 @@
         {
           "kind": "within_band_draws",
           "bounded_band_family": "truncated exponential matched to the cell's published mean",
-          "open_band_family": "Pareto with alpha = mean / (mean - lower bound)",
+          "open_band_family": "Pareto with alpha = mean / (mean - lower bound); mean from published cell gains / actual carrier mass, scaled for support shortfall and subject to band floor; household-weighted probability-interval means ordered by held uniforms",
           "mean_repair_margin": 0.02,
           "mean_repair_reason": "Published counts round to the nearest thousand and amounts to the nearest million; four cells of the 2023-24 table imply a mean outside their own band, and repaired means clamp just inside the violated boundary.",
           "bottom_band_floor": "annual exempt amount plus one pound, since every allocated person is a taxpayer with a liability",

diff --git a/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json b/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json
@@ -36,7 +36,7 @@
       "status": "required_at_build",
       "stage": "cgt_incidence_clone",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -59,7 +59,7 @@
       "status": "required_at_build",
       "stage": "cgt_band_donors",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -82,7 +82,7 @@
       "status": "required_at_build",
       "stage": "hmrc_cgt_gains_spine",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -106,7 +106,7 @@
       "status": "required_at_build",
       "stage": "salary_sacrifice",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -130,7 +130,7 @@
       "status": "required_at_build",
       "stage": "student_loans",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -152,11 +152,11 @@
       "status": "required_at_build",
       "stage": "hmrc_cgt_gains",
       "source_manifest": "cgt_source_stages.json",
-      "source_manifest_sha256": "71104111b4b9f2da00ce49ad5abec54a35d300196032c9742d62e02aa1730774",
+      "source_manifest_sha256": "4a7234131c245ce6f195e0fe500877505cc91a0d7b9ff68276164c289ad25d10",
       "superseded_by": {
         "stage": "hmrc_cgt_gains_spine",
         "source_manifest": "source_stages.json",
-        "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+        "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
         "reason": "The FRS spine build executes hmrc_cgt_gains_spine, which applies the same HMRC Table 3 amounts redraw directly in source_stages.json before calibration."
       },
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
@@ -182,11 +182,11 @@
       "source_manifest": "hmrc_income_source_stages.json",
       "source_manifest_sha256": "c0341af7166ae3a85a3c1164e7d9e880c4b4aec122f1a8fa90c73b46c596e1ea",
       "canonical_source_manifest": "source_stages.json",
-      "canonical_source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "canonical_source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "superseded_by": {
         "stage": "hmrc_spi_income_spine",
         "source_manifest": "source_stages.json",
-        "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+        "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
         "reason": "The FRS spine build executes hmrc_spi_income_spine, which supersedes the June retained-leaves/hmrc_spi_income pair inside source_stages.json."
       },
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
@@ -272,7 +272,7 @@
       "status": "required_at_build",
       "stage": "was_wealth",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -307,7 +307,7 @@
       "status": "required_at_build",
       "stage": "regional_property_uprating",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -328,7 +328,7 @@
       "status": "required_at_build",
       "stage": "lcfs_consumption",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -366,7 +366,7 @@
       "status": "required_at_build",
       "stage": "etb_vat",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {
@@ -386,7 +386,7 @@
       "status": "required_at_build",
       "stage": "etb_services",
       "source_manifest": "source_stages.json",
-      "source_manifest_sha256": "1379145c702edc275ef14e5bc564f793133e3b70ae985182a389be56a190c129",
+      "source_manifest_sha256": "4c177a6c3535836983e85cae3a0b81fbaa4e1a623ac4f087cd225bd4b1c9e762",
       "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833",
       "base_candidate_tier": "frs",
       "source_vintages": {

diff --git a/packages/microcosm-build/src/microcosm/build/uk/source_stages.json b/packages/microcosm-build/src/microcosm/build/uk/source_stages.json
@@ -2859,7 +2859,7 @@
     {
      "kind": "within_band_draws",
      "bounded_band_family": "truncated exponential matched to the cell's published mean",
-     "open_band_family": "Pareto with alpha = mean / (mean - lower bound)",
+     "open_band_family": "Pareto with alpha = mean / (mean - lower bound); mean from published cell gains / actual carrier mass, scaled for support shortfall and subject to band floor; household-weighted probability-interval means ordered by held uniforms",
      "mean_repair_margin": 0.02,
      "mean_repair_reason": "Published counts round to the nearest thousand and amounts to the nearest million; four cells of the 2023-24 table imply a mean outside their own band, and repaired means clamp just inside the violated boundary.",
      "bottom_band_floor": "annual exempt amount plus one pound",

diff --git a/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml b/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml
@@ -2269,7 +2269,7 @@ stages:
     weights: household_weight mapped to persons; no person splits across bands
   - kind: within_band_draws
     bounded_band_family: truncated exponential matched to the cell's published mean
-    open_band_family: Pareto with alpha = mean / (mean - lower bound)
+    open_band_family: Pareto with alpha = mean / (mean - lower bound); mean from published cell gains / actual carrier mass, scaled for support shortfall and subject to band floor; household-weighted probability-interval means ordered by held uniforms
     mean_repair_margin: 0.02
     mean_repair_reason: Published counts round to the nearest thousand and amounts to the nearest million; four cells of the 2023-24 table imply a mean outside their own band, and repaired means clamp just inside the violated boundary.
     bottom_band_floor: annual exempt amount plus one pound
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Represent the UK CGT open tail with household-weighted Pareto interval means anchored to published cell gains, avoiding random aggregate drift and rounded-count mean errors while preserving taxpayer selection and bounded-band draws.