export-results --merge carries forward any cell a run did not re-execute, which is deliberate — the -no-skills twins run monthly, so without the merge a weekly snapshot would have holes. The consequence is not deliberate: a snapshot is not a run, and nothing in the file says so.
What the current snapshot contains
results/runs/2026-08-25T10-33-18-000Z.json, 114 rows, by execution date:
| ranAt |
rows |
| 13 Aug |
22 |
| 17 Aug |
44 |
| 18 Aug |
6 |
| 19 Aug |
12 |
| 24 Aug |
18 |
| 25 Aug |
12 |
Six execution dates spanning thirteen days. The oldest rows predate the sandbox CLI 2.5.0 bump (14 Aug), the polling fixes that removed fixed sleeps (17–18 Aug), several scorer corrections, and the prompt addendum change (24 Aug). AGENTS.md already says a CLI bump "changes the product under test" — here it changed it inside a single published snapshot.
The 22 rows from 13 August are all -no-skills arms of the two frontier models, and 21 of 22 pass with full check lists, so these are not the dead rows from that day's outage. They are simply thirteen days old.
Why it matters
1. Anyone counting snapshots overcounts. Across the ten published snapshots each cell has a mean of 2.05 distinct executions, and one 17 August execution of verification-002 appears in seven consecutive snapshots. Reading that as seven runs is how I reported an apparent skills inversion as holding across "8 consecutive runs" in #2 when it was one pair of executions. Loop 1's whole lesson is that a single attempt cannot be separated from variance; republication makes a single attempt look like it already has been.
2. The skills delta compares arms measured under different instruments. For the frontier models the +skills arm is 17–25 August and the -no-skills arm is 13 August, with different scorers and a different CLI underneath. Whatever that difference is, part of it is not skills.
What would fix it
Not the merge itself, which earns its place. Options, roughly increasing in cost:
- Record it. Put the execution-date histogram in the snapshot metadata and print it in
report-results, so a stale arm is visible rather than discoverable.
- Age out. Refuse to carry a row past some age, and leave the cell absent rather than stale — an absent cell is honest, a thirteen-day-old one is not.
- Invalidate on instrument change. A CLI bump or a scorer edit drops carried rows for the affected scenarios, because those rows measured a different thing.
compare-snapshots should probably also refuse to report a delta between two snapshots that share executions, or at least say how many of the rows are the same rows.
Found while chasing #2. Related: #26 (submodule pin drift, same class of problem — the instrument moving without the results saying so).
export-results --mergecarries forward any cell a run did not re-execute, which is deliberate — the-no-skillstwins run monthly, so without the merge a weekly snapshot would have holes. The consequence is not deliberate: a snapshot is not a run, and nothing in the file says so.What the current snapshot contains
results/runs/2026-08-25T10-33-18-000Z.json, 114 rows, by execution date:Six execution dates spanning thirteen days. The oldest rows predate the sandbox CLI 2.5.0 bump (14 Aug), the polling fixes that removed fixed sleeps (17–18 Aug), several scorer corrections, and the prompt addendum change (24 Aug). AGENTS.md already says a CLI bump "changes the product under test" — here it changed it inside a single published snapshot.
The 22 rows from 13 August are all
-no-skillsarms of the two frontier models, and 21 of 22 pass with full check lists, so these are not the dead rows from that day's outage. They are simply thirteen days old.Why it matters
1. Anyone counting snapshots overcounts. Across the ten published snapshots each cell has a mean of 2.05 distinct executions, and one 17 August execution of
verification-002appears in seven consecutive snapshots. Reading that as seven runs is how I reported an apparent skills inversion as holding across "8 consecutive runs" in #2 when it was one pair of executions. Loop 1's whole lesson is that a single attempt cannot be separated from variance; republication makes a single attempt look like it already has been.2. The skills delta compares arms measured under different instruments. For the frontier models the
+skillsarm is 17–25 August and the-no-skillsarm is 13 August, with different scorers and a different CLI underneath. Whatever that difference is, part of it is not skills.What would fix it
Not the merge itself, which earns its place. Options, roughly increasing in cost:
report-results, so a stale arm is visible rather than discoverable.compare-snapshotsshould probably also refuse to report a delta between two snapshots that share executions, or at least say how many of the rows are the same rows.Found while chasing #2. Related: #26 (submodule pin drift, same class of problem — the instrument moving without the results saying so).