Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Binary file modified app/public/paper/policybench.pdf
Binary file not shown.
Binary file modified app/public/paper/web/figures/positive_zero_scatter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
786 changes: 464 additions & 322 deletions app/public/paper/web/index.html

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions app/src/app/paper/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ import Link from "next/link";

import SiteHeader from "../../components/SiteHeader";

const SNAPSHOT_DATE_LABEL = "Snapshot 2026-09-01";
const SNAPSHOT_DATE_LABEL = "Snapshot 2026-09-05";

const PAPER_DESCRIPTION =
"PolicyBench paper: the 2026-09-01 manuscript snapshot reporting the household-impact-weighted exact-match rate, with model responses collected from June 12 through September 1 and PolicyEngine reference outputs on the US populace microdata.";
"PolicyBench paper: the 2026-09-05 manuscript snapshot reporting the household-impact-weighted exact-match rate, with model responses collected from June 12 through September 1 and PolicyEngine reference outputs on the US populace microdata.";

export const metadata: Metadata = {
title: "Paper",
Expand Down Expand Up @@ -49,7 +49,7 @@ export default function PaperPage() {
<>
<p className="max-w-2xl text-sm leading-relaxed text-text-secondary sm:text-base">
Benchmarking no-tool tax-and-benefit estimation in frontier language
models. This page embeds the 2026-09-01 scored manuscript snapshot: a
models. This page embeds the 2026-09-05 scored manuscript snapshot: a
100-household public preview reporting the household-impact-weighted
exact-match rate against PolicyEngine reference outputs.
</p>
Expand Down
84 changes: 84 additions & 0 deletions app/src/components/ExclusionNote.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
import type { ReferenceExclusion, ScenarioPrediction } from "../types";
import { formatCurrency } from "../format";
import { binaryFlag } from "../lib/scoring";
import { describeExclusionReason } from "../lib/predictionStatus";

function formatValue(
value: number,
isBinary: boolean,
currencySymbol: "$" | "£",
): string {
if (isBinary) {
const flag = binaryFlag(value);
return flag === null ? "Invalid" : flag === 1 ? "Yes" : "No";
}
return formatCurrency(value, currencySymbol);
}

/**
* Why an output contributes to no model's score. Rendered in the prediction
* detail dialog for rows the payload marks `scored: false`; the release's
* exclusion record, when present, supplies the alternative reading and the
* reference under it.
*/
export default function ExclusionNote({
pred,
exclusion,
isBinary,
currencySymbol,
}: {
pred: ScenarioPrediction;
exclusion?: ReferenceExclusion;
isBinary: boolean;
currencySymbol: "$" | "£";
}) {
const unlistedInput = exclusion?.unlistedInput ?? pred.excludedInput;
const reason = describeExclusionReason(
exclusion?.reasonCode ?? pred.excludedReason,
);
return (
<section
className="mt-4 rounded-lg border border-border-subtle bg-surface px-4 py-3"
data-testid="exclusion-note"
>
<div className="text-[10px] uppercase tracking-[0.14em] text-text-muted font-medium">
Excluded from scoring
</div>
<p className="mt-2 text-sm text-text-secondary leading-relaxed">
This output is scored for no model: {reason}
{unlistedInput ? (
<>
{" "}
(
<code className="font-[family-name:var(--font-mono)] text-[12px]">
{unlistedInput}
</code>
)
</>
) : null}
. The prediction above is shown for inspection and counts neither for
nor against this model.
</p>
{exclusion ? (
<>
<p className="mt-2 text-sm text-text-secondary leading-relaxed">
{exclusion.alternativeReading} Under that reading the reference is{" "}
<span className="font-[family-name:var(--font-mono)]">
{formatValue(exclusion.alternativeValue, isBinary, currencySymbol)}
</span>{" "}
rather than the frozen{" "}
<span className="font-[family-name:var(--font-mono)]">
{formatValue(exclusion.frozenValue, isBinary, currencySymbol)}
</span>
.
</p>
<p className="mt-2 text-xs text-text-muted leading-relaxed">
Both values recomputed with {exclusion.engineVersion}; excluded by
the developers on {exclusion.decidedOn}.
{exclusion.note ? ` ${exclusion.note}` : ""}
</p>
</>
) : null}
</section>
);
}
2 changes: 1 addition & 1 deletion app/src/components/Methodology.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ export default function Methodology({
selects JSON for the family (the older Gemini and DeepSeek rows).{" "}
{currentBoard ? (
<>
Ten of the 33 models answer the same facts in subsets of one or
Ten of the 39 models answer the same facts in subsets of one or
three outputs per request, an accommodation that predates the
whole-scenario rule. The per-model transport and request shape are
recorded in the paper&apos;s serving-configuration table and the
Expand Down
14 changes: 8 additions & 6 deletions app/src/components/ModelLeaderboard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,15 @@ import { wouldRank } from "../lib/wouldRank";
import { isCurrentBoard, modelPageHref } from "../lib/boardScope";

// Exact-match scores of the tool_choice: auto sensitivity runs
// (sensitivity/claude-thinking-2026-08.md). Their "would rank" positions are
// derived from the live board rows at render time, never typed by hand.
// (sensitivity/claude-thinking-2026-08.md), scored on the same 1,973 outputs
// the board scores (eleven outputs are excluded for every model). Their
// "would rank" positions are derived from the live board rows at render time,
// never typed by hand.
const SENSITIVITY_EXACT = {
"claude-fable-5": 86.9,
"claude-opus-5": 85.6,
"claude-sonnet-5": 80.2,
"claude-fable-5.1": 87.5,
"claude-fable-5": 87.5,
"claude-opus-5": 86.2,
"claude-sonnet-5": 80.8,
"claude-fable-5.1": 88.2,
} as const;
import {
rankWithFallbackScore,
Expand Down
Loading