Add six September 2026 models (GPT-6 Astra #2 of 39), exclude eleven ambiguous outputs from scoring, publish dashboard-data-20260905c - #164
Merged
Conversation
Registry, cards, prices and app metadata for gpt-6-astra, gemini-3.8-flash, gemini-3.5-flash-lite, glm-5.3, deepseek-v4-pro-0813 and deepseek-v4-flash-0731. scripts/run_audit_claude.sh mirrors run_audit_codex.sh through the Claude Code CLI and writes judge provenance beside each verdict.
The failure-audit judge may return a case as prompt_ambiguity, which the finality validator rejects. policybench/adjudications.py applies a committed adjudication record (annotations/<run>/us_adjudications.json) to the row annotations and case notes, keeping the judge's verdict and the reasoning beside the adjudicated class; scripts/apply_adjudications.py rewrites a publish bundle before export so the payload and the frozen copies agree. The freezer verifies every adjudicated case and pins the record. One entry: scenario_064 SSI, judged prompt_ambiguity by Claude Opus 5 on 2026-09-05, adjudicated llm_error (age_disability) as on the 2026-09-01 board.
…026 adds; refreeze the manuscript GPT-6 Astra debuts second (87.4), Gemini 3.8 Flash 11th (82.1), DeepSeek V4 Flash 0731 15th (79.3), DeepSeek V4 Pro 0813 18th (79.0), Gemini 3.5 Flash-Lite 19th (78.6), GLM-5.3 21st (77.7). Incumbent rows are byte-identical to dashboard-data-20260901c. Judge sweep: the 350 cases a new model joined were judged by Claude Opus 5 through the Claude Code CLI; the other 318 keep their GPT-5.6 Sol verdicts. The manifest now tallies judge provenance per case and the developer adjudication record; the paper, benchmark card and docs disclose both. The frozen per-country payload is now stored as a deterministic gzip (data.json.gz, pinned by sha256) read through policybench.snapshot_payload: the plain export passed GitHub's 100 MB file limit at 39 models. Also: Ox Alpha identified by Z.ai as GLM-5.3-Flash after its run (prose, registry comments, provider mapping; listing date corrected to 2026-08-20 per the OpenRouter page); six release dates with sources; GLM-5.3 and the dated DeepSeek checkpoints marked open-weight; supervised-run evidence for the six new rows (ten rows pinned, tool choice for nine); board-count copy, sensitivity-doc ranks and audit-universe counts moved to the 39-model board.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…t ties deterministically Peer review round 1 on #164: - verify_adjudications_applied requires agreement with the complete record (source, subtype, and the exact adjudication sentence with its reasoning); apply_adjudications is case-level and replaces a stale sentence, so a revised record re-applies. Regression tests for revised and mismatched records. - Both audit runners validate verdicts against the audit tree's schema.json (scripts/validate_verdict.py, jsonschema) before publishing a verdict or treating a case as complete; a partial fallback object is rejected. - federal_state_joint_accuracy breaks ties on the joint rate by model id with a stable sort; the exception list and prose order identically everywhere (CI failed on the Sol/Astra tie). Paper re-rendered and re-pinned.
…ring; publish dashboard-data-20260905c A benchmark output is scored only when its reference follows from the facts the prompt states. Eleven outputs in ten households depend on engine inputs the certified household data never carried and the prompt never listed: the SSI disability criterion (false for every person in the June 2026 build) and months of SSDI receipt (never carried; integer-typed and never promptable). Each reference was recomputed with policyengine-us 1.755.4, the version that produced it, under the reading a careful reader could take of the stated 'is disabled' or SSDI-income fact, and moved: three SSI, three SNAP and five Medicare outputs. They are removed from scoring for every model, so no model gains or loses from them; their rows stay in the payload with scored=false. Mechanics: policybench/reference_exclusions.py reads reference_exclusions.json beside the reference CSV; export, the analyze CLI, the audit validators and paper_results all score the remaining 1,973 outputs per model. The freezer copies and pins the record, publishes a manifest block, and refuses to freeze unless it agrees with the adjudication record, whose eleven entries affirm prompt_ambiguity (allowed only with excluded_from_scoring). The 287 annotated rows on excluded outputs carry prompt_ambiguity as description; no scored row does. The app skips unscored rows in its client-side scorer, model page and audit recount. Also: to_pe_household() now builds marital units (head and spouse as the couple, everyone else alone); without them policyengine-core lumped every member into one marital unit and deemed the household's income to any eligible adult. No frozen reference changes (verified against 1.755.4). The sensitivity runs are rescored on the same 1,973 outputs: Fable 5 thinking 87.5 (+7.1), Opus 5 86.2 (+5.9), Sonnet 5 80.8 (+10.9), Fable 5.1 auto 88.2 (+1.2). Paper, benchmark card, docs and tests follow. Headline effect: every model moves by -0.03 to +0.64 points; Sol 89.2, Astra 88.0, Fable 5.1 86.9, Kimi K3 86.7.
MaxGhenis
changed the title
This was referenced Sep 5, 2026
v2 prompt and data: program-specific disability facts, integer inputs, and the take-up boundary
#165
Open
… require jsonschema; analyze export keeps exclusions; disclosures corrected - Scenario explorer: scoring status is excluded/unparsed/correct/off (app/src/lib/predictionStatus.ts) and exclusion outranks exact-match fields. Excluded outputs get a row tag, muted uncolored cells, a neutral "Excluded" pill, a "Score contribution: none" block and an ExclusionNote with the unlisted input, alternative reading, the reference under it, engine version and decision date from the release's referenceExclusions record. prompt_ambiguity labeled. DetailContent exported and covered with an excluded-row test. - Audit runners select AUDIT_PYTHON, then .venv/bin/python, then python3, and refuse to start when jsonschema is missing (test). - export_dashboard_data forwards excluded_reference and reference_exclusions; the analyze CLI passes both (tests/test_cli_analyze.py runs the command end to end and checks the export keeps excluded rows and the record). - Benchmark card: eleven adjudications affirming prompt_ambiguity. Paper: 287 prompt_ambiguity rows plus six parse_contract_failure rows among excluded annotated rows; the SNAP sentence accounts for scenario_057's SSI reference. Re-rendered and re-pinned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Scenario.marital_couple() identifies the head and spouse from the is_tax_unit_head / is_tax_unit_spouse inputs each adult carries, with name and joint-filing fallbacks for manifests that predate those inputs; marital_units() places that couple in one marital unit and everyone else alone. Renaming people no longer changes the household the engine sees (peer review round 3: adult1/adult2 was split into two single-person units, raising a joint-filing couple's SSI). Tests: unit structure is invariant to renaming; flags win over names; every person lands in exactly one unit; legacy fallbacks pinned; a slow engine-backed test reproduces the SSI case. All 1,984 frozen references recomputed with policyengine-us 1.755.4 under the new builder: 0 differ. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…l-unit keys positional - paper/index.qmd: load_snapshot_ground_truth() now returns the scored reference (frozen CSV minus reference_exclusions.json), so the household bootstrap intervals, the weighting comparison and the simple baselines score the same 1,973 outputs as the published board. Table 6's GPT-5.6 Sol interval moves from 85.8–91.2 (the old 88.65 point) to 86.3–91.7 around the published 89.2. Re-rendered and re-pinned. Regression test asserts bootstrap point estimates reproduce every model's published exact score (fails on the unscored reference: 39 of 39 mismatch). - Scenario.marital_units(): unit identifiers are positional (marital_unit_1, ...) rather than name-derived, so a person named "couple" cannot displace the couple's unit. Tests cover the collision and, under the engine, an unchanged SSI when a child is renamed "couple". Frozen references recomputed with policyengine-us 1.755.4 under the positional keys: 0 of 1,984 differ. Peer review round 4 (Astra). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ference Peer review (gate 449e834c round 1) found four readers of the frozen reference that predated the exclusion record; a sweep found four more. All now score the same 1,973 outputs as the published board: - Table 9 alternative views: row flattening moved to policybench/manuscript_views.py (exclusion-aware, tested); the manuscript imports it. GPT-5.5 positive-reference 85.8 -> 87.8. - Freezer legacy impact summary: computed on the scored reference; full re-freeze regenerated impact_summary_by_model.csv and the top-level copy with their pins (GPT-6 Astra 0.945347 -> 0.949644). Regression test compares both to a recomputation. - Sensitivity summary bounded scores: board 95.289 -> 95.891 (frozen modelStats), sensitivity 95.738 -> 96.347 (recomputed with the app's continuous scorer); evidence tests now check all three scores. - Benchmark card judge attribution: Opus judged six of the eleven adjudicated cases (064 SSI and the five Medicare outputs), Sol five. - paper_results: always-zero baseline, zero share, parse-failure and explanation-missing counters on scored rows (parse failures 707 -> 701; contract violations 768 -> 762; 0.9% unchanged); joint-credit table guarded; per-model output counts re-derived (1,973 of 1,984). Paper re-rendered and re-pinned. Headline scores, references, the exclusion record and the published payload are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… per-program assets on the scored reference Peer review (gate 449e834c round 2): - fold_board copies reference_exclusions.json beside the reference files it stages (removing a stale copy on reuse), so a staged export scores the same 1,973 outputs as the board instead of all 1,984. Test. - scripts/sensitivity_by_variable.py recomputes a thinking run's per-program rows exactly as the exporter's heatmap does, on the scored reference, and regenerates every *-by-variable.csv.gz. The doc's Fable 5.1 table (Medicare 96.5/98.3, SSI 99.0/100.0, SNAP 81.4/81.4) and the August Fable 5 table are rewritten from it. The three August runs' predictions are committed under sensitivity/data and pinned, with exact, within-1% and bounded scores recomputed on the scored reference, in claude-thinking-2026-08.json; the Fable 5.1 summary's pins are refreshed. Tests recompute every asset from its predictions, check every pin, and parse both doc tables against the frozen heatmap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Sep 5, 2026
* Bind judge provenance to the verdict it describes Peer review of #164 (gate f4abb02c) found that re-judging a case through the Codex runner could keep the Claude runner's provenance sidecar: prepare_audit removed verdict.json but left verdict.meta.json, the Codex runner never touched the sidecar, and the freezer preferred it over codex.log. A later freeze would have miscounted judges. - prepare_audit unlinks verdict.meta.json together with a stale verdict. - Both runners write verdict.meta.json with the verdict's sha256 and the runner name (the Codex runner now writes one too, with the model from its log header), and remove any leftover sidecar before judging. - The freezer counts a sidecar only when its hash matches the case's current verdict.json; otherwise it reads codex.log, else "unknown". - scripts/backfill_verdict_provenance.py binds existing sidecars (only when the recorded judging time sits within tolerance of the verdict file's mtime) and writes Codex sidecars from codex.log. Run on the audit tree: 350 bound, 318 written; the tally reproduces the frozen manifest (350 Claude Opus 5, 318 GPT-5.6 Sol). - Tests: sidecar binding rules; prepare_audit cleanup; a cross-runner re-judge through both runner scripts with fake CLIs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Require ownership evidence before trusting legacy provenance Gate review of #166 (round 1): the backfill's timestamp tolerance could bind a legacy Claude sidecar to a Codex verdict written minutes later, and the freezer's codex.log fallback could attribute a current hash-less Claude verdict to an older Codex attempt. - Backfill binds a legacy sidecar only when the verdict file dates from its judging time AND no codex.log was written after that time; a verdict with no sidecar at all can only have come from the Codex runner, which is the evidence for writing its Codex sidecar. - The freezer reads codex.log only when the case has no sidecar and the log was written alongside the verdict (600 s); a stale or hash-less sidecar beside an unrelated log counts as unknown. - Tests for both rules; the real audit tree's tally still reproduces the frozen manifest (350 / 318) and the backfill is idempotent (668 already). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds six September 2026 releases to the board, publishes the 39-model payload as
dashboard-data-20260905c, refreezes the manuscript on it, and introduces one scoring-policy change: outputs whose reference depends on an input the household data never carried are excluded from scoring for every model.Incumbent predictions and references are byte-identical to
dashboard-data-20260901c; incumbent scores move by −0.03 to +0.64 points because of the exclusion below (Sol 89.2, Fable 5.1 86.9, Kimi K3 86.7).Scoring exclusion (the substantive change; please review)
The Opus judge flagged scenario_064 SSI as
prompt_ambiguity: the household listsis disabledfor a 27-year-old, the engine's SSI gate is a separate input (meets_ssi_disability_criteria) that is false for every person in the certified June 2026 microdata, and the prompt never lists it. The same applies to Medicare's under-65 route (months_receiving_social_security_disability, never carried, integer-typed and never promptable).Rather than reclassify by which models missed, every one of the 1,984 references was recomputed with policyengine-us 1.755.4 (the version that produced them; the base recompute matches the frozen CSV on every row) under the alternative reading, and every output whose value moved is excluded: three SSI (064, 067, 057), three SNAP (023, 057, 100, where SSI-disabled status switches on the elderly-or-disabled SNAP rules), and five Medicare (007, 073, 074, 079, 084). Exclusion is symmetric: rows that matched the frozen value leave the score along with rows that did not, so every model is scored on 1,973 outputs.
Mechanics:
policybench/reference_exclusions.pyreadsreference_exclusions.jsonbeside the reference CSV (publish bundle and frozen run dir). Export, the analyze CLI, the audit validators andpaper_resultsall score the remaining outputs; excluded rows stay in the payload withscored: falseand the reason, and the payload listsreferenceExclusions.reference_exclusionsmanifest block, and refuses to freeze unless the record agrees with the adjudication record, whose eleven entries affirmprompt_ambiguity(allowed only together withexcluded_from_scoring: true).prompt_ambiguityas description; no scored row does. The scored audit universe is 8,783 rows (8,780 exact misses, three exact hits; 1,605 scored rows below full bounded score unannotated).policybench/manuscript_views.py), the freezer's legacy impact summary, the sensitivity summary's three scores, and the paper's always-zero baseline, zero share and parse-failure counters (paper_results). Regression tests tie each to the frozenmodelStatsor the scored reference.reference_exclusions.jsonbeside the reference files it stages, so staged exports score the same universe. The four thinking-sensitivity runs' per-program assets are regenerated on the scored reference (scripts/sensitivity_by_variable.py), the August runs' predictions are committed and pinned, and tests recompute every asset and both doc tables.scored,excludedReason,excludedInput.Superseded tags:
20260905carried the judge's verdict before adjudication;20260905bscored all 1,984 outputs.Harness bug found on the way (fixed; no frozen reference changes)
Scenario.to_pe_household()defined no marital units, so policyengine-core placed every household member in one marital unit and the engine deemed the whole household's income to any SSI-eligible adult as if the others were a spouse. The head and spouse, identified by theis_tax_unit_head/is_tax_unit_spouseinputs each adult carries (with name and joint-filing fallbacks for manifests that predate those inputs), now form the only couple; everyone else is alone, and renaming people leaves the household unchanged (structural tests plus an engine-backed test). Verified against 1.755.4: no frozen output changes, because no benchmark household has a non-spouse adult the engine finds SSI-eligible under the facts as listed. The Alt-A recompute above used the corrected construction.Judge provenance
The 350 cases a new model joined were judged by Claude Opus 5 through the Claude Code CLI (
scripts/run_audit_claude.sh, new; writes averdict.meta.jsonsidecar per case and validates verdicts against the audit schema). The other 318 keep their GPT-5.6 Sol verdicts. The manifest tallies judge provenance per case; the paper, benchmark card and docs disclose that two judge models, both board rows, produced the verdicts.Frozen payload storage
The frozen per-country export passed GitHub's 100 MB file limit, so it is stored as a deterministic gzip (
runs/<run>/data.json.gz) pinned by sha256; every reader goes throughpolicybench.snapshot_payload.Other changes
Verification
uv run pytest -m "not slow": 798 passed, 5 skipped (plus two slow engine-backed marital-unit tests).cd app && bun run lint && bun run test && bun run build: 116 tests pass, build clean.freeze_snapshot.py --rendered-onlyre-pinned the PDF and web bundle.results/local/v1_1/venv-refs(pe-us 1.755.4, core 3.28.0): base = frozen on all 1,984 rows; corrected marital units = frozen on all rows; alternative readings move exactly the eleven excluded outputs.🤖 Generated with Claude Code