UK rowwise candidate: solve local and national targets as one matrix on the cloned spine (#762 PR A)

[juaristi22]

First of two PRs for #762 (plan: repos/uk-762-rowwise-candidate-plan.md, approved 2026-09-02) — the joint-solve machinery. The second PR carries the signed registers, the measured doctrine adjudication, the receipts, and the local-area release contract/assembly after the licensed runs.

What this does

One matrix, one weight vector, one solve. tools/build_uk_rowwise_candidate.py now clones the spine through the OA ladder at K, runs the policyengine-uk engine on the cloned frame, and solves the national families (compiled + measure-excluded, 364 activated references) and the local families (compiled references over both grains minus signed deferrals, plus the ladder's census-household rows) as rows of a single sparse constraint matrix in one calibrate() call under the local doctrine. No national-only solve exists on this path; the national line's diagnostics are only a comparison yardstick.

• Surface (local_rowwise.py): a long-format multi-grain builder build_uk_rowwise_local_surface_matrix (constituency + local authority, present cells only, signed deferrals are absent rows, canonical grain/metric/area order, fail-closed on unreachable nonzero targets and per-grain roster coverage). The dense single-grain builder delegates to it and keeps its legacy area-major result bit-for-bit. Local rows are named by the compiled register's row_name, so the scorer joins by identity; area masks are materialized lazily from an int32 index.
• National rows (UKRowwiseNationalRows, solve_uk_rowwise_weights_under_doctrine(..., national_rows=, target_weight_rule=, restore=)): local rows first, national rows after, one TargetSet; row-name alignment asserted over every compiled row; diagnostics and past-cap censuses reported per row kind; prepared slash-named measure columns are restored away before the frame rebuild and refused if any survive. rotated_uk_local_holdout keeps the national rows in every training fold.
• Surface assembly + cross-grain (ledger_targets.uk_local_target_surface): compiled local registry → present cells (family via family_for_metric, compiled source family retained), ladder household rows for both grains, then apply_uk_cross_grain_reconciliation with the national control rows present (the standing country > constituency > la rule and bridges from Declare cross-grain target reconciliation for the UK local surface: national wins, sub-national rescales (#495 increment 6 companion) #802/UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 fire in the joint surface). A declared bridge whose missing national members are all in-force reviewed measure exclusions is classified unbound and listed under cross_grain.unbound_bridges in the plan and manifest instead of refusing the surface; a partial binding with any unreviewed missing member still fails closed by name. On the live surface this is the household-composition partition bridge (three cells are UK spine: relationship-to-head frame column, so the ONS household-composition split is measurable #791's exclusions to 2026-11-26), so the ladder's census household counts bind as published until that is adjudicated (plan item A11). A national target that fans out into several cells at one geography (UC payment-distribution bands, SPI income bands: 15 targets, 213 cells on the live compile) is a distribution, not a control: it hands the engine no control row, is dropped from the bound ids the engine sees, and is receipted as fanout_targets_not_controls (the compile activates only a subset of the DWP award bands, so their sum can never reproduce the caseload); every cell still solves as its own row. Eight declared age-band bridges (local ons.age.{b} at both grains ↔ the single-cell national ons.population.age_*_by_region UK totals; 80–89 has no local counterpart) let the local age bands rescale to the national totals despite the half-open vs inclusive edge encodings that defeat the exact-signature match.
• Doctrine (local_doctrine.py): weighting vocabulary ("uniform", "grain_equal") — grain_equal gives the national rows, the constituency rows, and the LA rows one equal share each, uniform within, derived from row labels through the national family_equal arithmetic; uk_local_doctrine_with_overrides receipts the single overridable field. UK_LOCAL_TARGET_LOSS_CAP 10.0 and UK_LOCAL_MAX_WEIGHT_RATIO 100.0 are unchanged.
• Scale ladder (--sample-fraction {0.01, 0.10, 1.0}, --sample-seed): f100 never samples (sampled: false); below it, sample_uk_spine_frame (new beside the certified-compact policy in national_sampling.py) samples whole source families keyed on the spine's source_household_id, stratified by the raw row's region, through the shared frame sampler with one full-mass renormalization; orphan derivatives and multi-region families are refused. Found on the first licensed dry-run: the compact policy needs a clone_index column the spine does not carry. Below f100 the surface restricts to cells whose area has assigned rows (receipted as rung_surface), the battery runs and persists every entry but enforces only the ladder gate, and the manifest says releasable: false; at f100 every release-blocking entry blocks as before.
• Engine on the clone: one scratch-mode UKMeasureResolver over the cloned frame resolves the national measures and feeds compute_household_metrics for both grains; --engine-blocks K resolves per clone block and rejoins by entity id (refused unless K equals --n-clones). Dry-run compiles both registries, assembles and reconciles the surface, checks binding adjudications, sizes the matrix analytically, and sweeps --candidate-clone-counts — without the engine, the solve, or any write.
• Pins and inputs: --input-sha256, --ladder-sha256, --ledger-facts/-sha256/--ledger-manifest-sha256 (the chronicle consumer artifact, same loader as the national seam), optional --measure-exclusions and --register-json; --release-candidate requires every pin and refuses overrides, --skip-holdout, sampled rungs, and per-block engine runs.
• Area-support reviewed exclusions: area_support_gate(reviewed_exclusions=) records an excluded area's measured shortfalls instead of failing, fails a stale exclusion (area now meets the floor) and an unknown key; the binding resolves exclusions_resource = new schema-2 register uk/local_area_support_exclusions.json on the shared exclusion clock (expired/premature entries fail). The register ships empty; entries are adjudications for the second PR. Gate-battery digests, the spec sha, the parameter-key pin, and docs/gate-battery-contract.md re-cut accordingly.
• Manifest v2: identity chain (spine sha + sidecar provenance via the promoted load_bound_spine_sidecar / spine_provenance_from_sidecar, ladder sha + layer vintages, ledger shas, code pin, runtime), survey_year (2024) and calibration_year (2025; artifact microcosm_uk_2025_local.h5), sampling and rung receipts, geography, weights (kind chain, mass conservation, realized max ratio vs design), solve (rule + override receipt, per-kind censuses, measure resolution, cross-grain, binding adjudications, area-support exclusions), fit by family for local and national rows, support_limited_misses (failing cells vs per-area ESS), holdout, vintages, releasable.
• Vintage receipts: the area authoring path now records uprating_holds like the national path — local_target_references.json gains uprating_from_period/uprating_to_period on the 17,077 cells whose facts predate the 2025 period (UC 2025-05, council-tax stock 2025 carry none; age 2024, SPI tax-year 2023, tenure 2021/22 do), membership gains the hold census; every other byte of the surface is unchanged.
• UC child bands at benefit-unit grain (A1-4 ruling): the four constituency uc_hh_*_children metrics classify each UC benefit unit by its own num_children and sum benefit units into households, matching the national benefit-unit re-bind (Re-bind the DWP UC composition and TCL disability-children targets to the publisher's grain (#808, #809) #813); names and order unchanged, census descriptions regenerated.
• Scoring inputs: tools/score_uk_local_candidate.py pins the merged UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 surface (19,618) and accepts a candidate superset with a rows_outside_register receipt; new tools/extract_uk_local_incumbent_surface.py turns the incumbent's wide weight arrays (one 2025 dataset shaped (650|360, 52846), no ids inside) plus the roster CSVs and the incumbent household table into the sha-pinned CSVs the scorer consumes, pairing by position and refusing length mismatches.

Verification (run independently of the implementation pass, on d2b31496)

• ruff check . clean; ruff format --check on the 28 touched files clean (repo-wide format debt untouched).
• tools/census_uk_local_targets.py --check and tools/census_uk_data_target_parity.py --check current; tools/ci_test_groups.py --verify ok (new test files land in the UK lane); git diff | grep -c "policyengine-uk-data#" → 0.
• Targeted battery (rowwise/doctrine/candidate/scorer/gate-battery/bindings/gates/ledger/ladder/national-calibration/diagnostics/authoring/council-tax/spec-bundle/country-spec/holdout/contract-pins/release-certification/extractor): 562 passed on the first commit; after the rebase over UK spine: benefit-unit-grain SPI UC reporter draw with pre-take-up screen (#832); exact capital sentinel (#833) #835 and the three licensed-touch fixes (spine sampler, Ledger geography spelling, reviewed-exclusion bridges): targeted 443 passed, spec/country/contract-pins/contract 511 passed, cross-grain/ledger/candidate/rowwise 90 passed.
• microcosm-data contract suite: 345 passed, 1 skipped.
• CI groups run in full from tools/ci_test_groups.py --list: rest 6,098 passed, 63 skipped; spine-uk 1,712 passed, 16 skipped, 2 failed; uk engine 1,450 passed, 16 skipped, 2 failed — the 2 failures are test_cached_reference_regeneration_matches_committed_surface and test_cached_candidate_regeneration_matches_committed_evidence, reproduced identically on clean main (installed policyengine-uk 2.92.1 vs committed 2.89.0 licensed-cache evidence; Make the UK eFRS parity re-pin one reviewed command; hold every identity mirror in lockstep (companion to #747) #749's lane).
• Re-cut pins (from real payloads on this base): UK spec sha 0733a81e… → 91eb9472…; gate policy bcbcfd55… → d76f012f…; gates manifest 76c861d4… → df8c9292…; composition fingerprint dfbad2aa… → 0de7f351….
• Regenerated local_target_references.json / membership: 19,618 rows identical apart from the added uprating-hold fields on 17,077 rows; membership gains uprating_holds (17,077) and holds_by_target; every other membership field byte-equal.
• End-to-end on the synthetic fixture: a K=2 f100 joint build (stub resolver, national + local + ladder rows in one matrix, manifest v2, Logbook rung f100, releasable: true) and an f001 run (rung receipt, restricted surface, releasable: false); the licensed spine, ledger artifact, and incumbent weights are not in the PR-CI environment by design.

Review notes

• Implemented by Codex under the reviewed plan's protocol on origin/main @ 90293e0a; deviations were verified and are recorded in the plan doc's execution notes (dense-wrapper order compatibility flag; households stays the ladder-only metric; compiled spec.family is the source family, so surface rows classify by family_for_metric and keep source_family; ledger/pin arguments stay optional on the synthetic path and are mandatory under --release-candidate).
• Nothing here binds a new family in production: the joint path refuses to solve until uk/local_binding_adjudications.json carries in-force entries for the five fenced families (second PR).
• The two UK-lane failures on this base (test_cached_reference_regeneration_matches_committed_surface, test_cached_candidate_regeneration_matches_committed_evidence) are the known policyengine-uk 2.92.1 vs committed 2.89.0 evidence mismatch and fail identically on clean main.

Refs #762, #495, #665. Not for merge without María's explicit go.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review pass at 2e9cde8c (Claude Code, high effort; worktree checkout, Python surface read against the PR body — the cited plan repos/uk-762-rowwise-candidate-plan.md is not in the tree, so this is against body and code — targeted suites run without the licensed engine: 576 passed in microcosm-build across 17 files, 228 in microcosm-data's contract tests, ruff check clean on all 37 changed Python files).

The design holds

"One matrix, one weight vector, one solve" is what the code does. solve_uk_rowwise_weights_under_doctrine builds one TargetSet with local rows first and national rows after and makes a single calibrate() call, spied by test_joint_doctrine_solve_compiles_local_then_national_once_and_restores. Alignment is by name and cannot pass misaligned: TargetSet rejects duplicate (name, period) keys, diagnostics are asserted against declared rows by row_name over every compiled row (local_rowwise.py:1164-1175), national rows against the registry likewise (:1100-1112), and calibrate maps loss weights by key. The long-format surface builder is genuinely fail-closed — unreachable nonzero targets, uncovered assigned areas, off-roster areas and duplicate cells all raise — and row order is a stable rank sort over tuple rosters with no set iteration. Lazy int32 area masks are bounded by roster size and transient; the compile sparsifies per target, so nothing about K=15 threatens memory or indexing.

I also checked the "bit-for-bit" claim for the legacy dense path independently: loading origin/main's local_rowwise.py beside the PR's and running both on the test_uk_local_rowwise fixture gives identical CSR indptr/indices/data, identical targets, area_codes, assigned_areas, household_ids, metric_names and metric_values, and an equal target_frame on all common columns (the new one adds family and target_name). The claim is true. Three things to fix before approving, none of them in the solve itself.

Findings

1. Should-fix — --engine-blocks K can still write releasable: true, and nothing guards the ×K aggregate artefact from your own #736 erratum. build_uk_rowwise_candidate.py:592 exposes --engine-blocks; _validate_cli_args allows any value equal to --n-clones (:2491), and the refusal at :2469-2470 fires only under --release-candidate. The manifest's "releasable": args.sample_fraction == 1.0 and all_gates_passed (:2271) has no engine_blocks term, so a non-release f100 run at blocks=K writes a releasable artifact with the deviation recorded only under solve.measure_resolution. _resolve_candidate_engine_surface (:333-365) rejoins per-block outputs by entity id with duplicate and missing-id checks, but there is no check that a per-block-resolved allocated aggregate sums to the whole-frame value — exactly the mechanism that produced ×15 on corporate_land_value, and the stub resolver in test_candidate_engine_surface_resolves_real_per_clone_blocks returns np.arange, so it can only pin id order. Cheapest fix: make releasable require engine_blocks == 1, or drop the hatch now that single-block is the release posture. The default path is single-block and the end-to-end test pins one resolver construction per run, so the production path is fine; the hole is the flag.

2. Should-fix — the gate partition is fail-open on a missing criticality key. _gate_failures_by_criticality (:1817) buckets a failed gate as diagnostic unless payload.get("criticality") == "release_blocking", and release_gate_rows (:2049-2051) filters the same way, so a failed gate whose persisted payload lacks the key drops out of both the blocking list and all_gates_passed. Today the battery raises GateBatteryBlockedError first and the except branch re-raises when the blocking list is empty, and if every key is missing release_gate_rows is empty and the artifact is unreleasable — so the live exposure is partial schema drift on one entry. Unknown or missing criticality should bucket as blocking; that is the fail-closed reading of "partition by criticality".

3. Should-fix — the delegation test does not pin what its name claims. test_dense_builder_delegates_without_changing_legacy_result (test_uk_local_rowwise.py:376) compares the post-PR dense wrapper against the post-PR long-format builder with assert_allclose — self-consistency, not equality with the pre-PR result. Since the result really is bit-for-bit (above), an exact-equality check against a frozen fixture from main would guard the claim rather than restate the delegation.

Questions

• restore post-condition. Frame ids are asserted equal to problem ids at entry (local_rowwise.py:1057) but not after restore(result.frame) (:1283) before weights are written back by position (:1300). CalibrationFrameAdapter.restore rebuilds from the original tables so order holds today; a reordering restore would misattribute weights silently. One re-assert matches the module's other boundary guards.
• Fan-out keyed on target_id alone. fanout_target_ids drops a target as a control at every geography if it fans out at any non-bridge geography, so a control single-celled at one geography is dropped silently; and a bridge control that fans out is kept as multiple same-(target_id, geography_id) reconciliation rows. Neither is live today; neither is guarded.
• Holdout scale under grain_equal. rotated_uk_local_holdout recomputes held_weights over the two held local grains while training weights spanned three grains at 1/3 each, so the report-only holdout is not on the training scale. Worth stating in the receipt if it stays.

Nits

exclusions_evaluated_on: date.today() in the gate battery (:1726) makes verdicts wall-clock-dependent once local_area_support_exclusions.json gains entries (it is empty on this PR; PR B fills it). ruff format --check flags battery_bindings.py:397, pre-existing and outside this PR's hunks.

Not verifiable here

Live-surface counts (364 national, 19,618 local), the age and household-composition bridges on the real compile, ×K behaviour on the real engine, the incumbent extractor on the real H5, and every fit or holdout number all need the licensed engine and data.

---

Ready to approve once 1 and 2 land; 3 is small and can ride along. The solve, the alignment discipline and the fail-closed surface builder are the substance, and they check out.

[juaristi22]

Thanks, all three land in 8be32dc (main merged in b875d4c, one pin re-cut).

(1) releasable now requires engine_blocks == 1 beside the full rung and the release-blocking gates, and the manifest writes release_posture so the failing leg is visible; the per-block aggregate check you describe would need a whole-frame resolution, so the flag is closed rather than the artefact detected.

(2) _is_release_blocking treats anything but an explicit diagnostic as blocking, in both the partition and all_gates_passed.

(3) tests/golden/uk_local_rowwise_dense_fixture.json is main's result on the fixture, compared exactly; the old test is renamed to the wrapper/long-format agreement it checks.

Questions: restore's post-condition is asserted (the test uses a stand-in because uk_national_frame refuses an unsorted id column); a bridge control that fans out raises, and a single cell dropped by the target-id rule is receipted; the holdout report carries loss_weight_scale: held_local_grains_only.

Nits: exclusions_evaluated_on is the run's started_ts.date(), and the format nit is fixed.

[vahid-ahmadi]

Second pass at b875d4c4 (Claude Code, high effort; worktree checkout, the fix commit 8be32dc9 read against the findings, golden regenerated from origin/main's builder, targeted suites run: 647 passed across test_uk_local_rowwise, test_uk_rowwise_candidate, test_uk_ledger_targets, test_gates, test_country_spec, test_spec_engine_country_bundles and the data contract tests; ruff check and ruff format --check clean on every file differing from main).

All three closed; two small residuals, neither worth holding for.

1. Verified. _release_verdict (build_uk_rowwise_candidate.py:1838-1863) requires single_block_engine, full_rung and release_blocking_gates_passed together and returns the posture; the final manifest writes both releasable and release_posture (:2314-2315), the dry-run plan gained the engine_blocks == 1 term (:1920), and the blocked path stays a literal False. No site can emit true with blocks > 1. Closing the flag rather than detecting the artefact is the right reading — a per-block aggregate check would need a whole-frame resolution to compare against, which is the thing single-block already is. Residual: test_release_verdict_requires_single_block_engine is a unit test of the helper; nothing runs --engine-blocks K on f100 and asserts the manifest's releasable is false and its release_posture names single_block_engine. The helper is small enough that this is a nit, but it is the assertion that would catch a future caller bypassing _release_verdict.

2. Verified. _is_release_blocking (:1825-1835) exempts only an explicit criticality: diagnostic, and both the partition (:1820) and release_gate_rows feeding all_gates_passed (:2086-2092) go through it. test_gate_criticality_reads_fail_closed covers the missing key, an unknown value, and a failed gate with no key landing in the blocking list; it fails on the previous head.

3. Verified. I regenerated tests/golden/uk_local_rowwise_dense_fixture.json from origin/main's local_rowwise.py on the same fixture and every key is equal, so the golden is main's output. test_dense_builder_result_equals_frozen_pre_pr_fixture compares indptr/indices/data, targets, area codes, assigned areas, household ids and metric names and values exactly. Residual: target_frame goes through assert_frame_equal with the default check_exact=False, so that one part is tolerance-based; the matrix data, which is what matters, is exact.

Questions, all answered. The restore post-condition is asserted right after restore(result.frame) and before the write-back (local_rowwise.py:1284-1290); the stand-in test fails on the old head, though for the older slash-column guard's reason rather than a demonstrated misattribution, so it is adequate rather than sharp. A bridge control that fans out now raises and a single-cell drop under the target-id rule is receipted (ledger_targets.py:992-1016), both with tests that fail on the old head. The holdout report carries loss_weight_scale: held_local_grains_only (:1470), unasserted but visible.

Nits. exclusions_evaluated_on is the run's started_ts.date() and the format nit is gone. The merge's one re-cut pin is the UK spec_sha256 in test_spec_engine_country_bundles.py; the contract digests are unchanged and consistent.

---

Nothing outstanding. Ready from my side; the two residuals (an end-to-end releasable assertion under --engine-blocks K, and exact comparison on the golden's target_frame) can ride with PR B.