Add hot-run-stability.py: run-to-run spread report from published results

[ipezygj]

What

A standalone report script — no changes to the methodology, the site, or any results.

The ranking takes the smaller of the 2nd and 3rd runtime per query (a defensible cache-stability choice), but both hot runs are already stored in every results file. That means the run-to-run timing noise — the thing that decides whether a small gap between adjacent systems is real — is measurable directly from the published data. This script does that:

./hot-run-stability.py                                   # all latest active results
./hot-run-stability.py clickhouse/results/20260808/*.json

Per results file it reports the median and p90 relative spread between the two hot runs (|run2−run3| / min) and the share of queries where the spread exceeds a threshold (default 10%).

What it shows on today's data

Across the 962 active result files, the median per-file hot-run spread is 2.9% — most of the board is tight. But the tail is heavy: on c6a.4xlarge, ClickHouse's own results show a median spread of 5.6% with p90 at 153%, and several systems' hot runs differ by 50%+ on most queries. Where the spread is that large, adjacent-rank gaps on the aggregate are smaller than one system's own timing noise — useful context for anyone citing close rankings, and free to compute since the runs already happen.

File selection mirrors validate-results.py (latest date per system/machine, same skip list). Python stdlib only.

Context

I work on benchmark-integrity tooling (recent public work: a rank-stability audit of 19 Kaggle competition leaderboards, a label-leakage census of a popular ML benchmark). ClickBench storing all three runs in the public data is what makes this kind of check possible at all — most boards discard everything but the winner, so this is genuinely to ClickBench's credit. If a variant of this would be more useful wired into the site (e.g. a per-system stability badge), happy to follow up.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[rschu1ze]

@ipezygj

The ranking takes the smaller of the 2nd and 3rd runtime per query (a defensible cache-stability choice), but both hot runs are already stored in every results file. That means the run-to-run timing noise — the thing that decides whether a small gap between adjacent systems is real — is measurable directly from the published data.

"defensible cache-stability choice"?
"run-to-run timing noise"
"adjacent systems"

Sorry, it is unclear what this means.

Can you please clearly outline: What is the motivation and use case for this PR?

[ipezygj]

Fair — that opening paragraph was written in my own vocabulary, not ClickBench's. Let me restate it plainly.

The use case. People cite ClickBench rankings, including small differences between neighbouring entries. This script lets a reader check whether a difference that small is bigger than the measurement's own repeatability, using data ClickBench already publishes.

Why it costs nothing to compute. Each query is run three times and all three timings are stored in every results file. The ranking uses the smaller of runs 2 and 3. That is a sensible choice — it is what I clumsily called "a defensible cache-stability choice", and I meant it as a compliment, not a complaint: taking the smaller of the two warm runs avoids cold-cache effects. But it also throws away the one thing those two runs tell you together, which is how much the same query varies between two identical warm executions on the same system and machine.

What the script does. For each results file it prints the median and p90 of |run2 − run3| / min(run2, run3) across the queries, plus the share of queries above a threshold. That is all. No change to the methodology, the ranking, the site or any results file; stdlib only; it reads the published JSON.

Why it might interest you. On the current data the board is mostly tight — median spread per file is 2.9% across 962 active result files. The tail is not: ClickHouse's own c6a.4xlarge results have a median spread of 5.6% and p90 of 153%, and several systems' two warm runs differ by more than 50% on most queries. Where that happens, a few per cent of difference in the aggregate between two neighbouring entries is smaller than one of those systems' own variation between its two runs.

The three phrases you flagged, in plain words: "run-to-run timing noise" = the difference between run 2 and run 3 of the same query; "adjacent systems" = two entries next to each other in the results table.

If that use case is worth having, I am happy to rewrite the PR description to lead with it. If it is out of scope for this repo, say so and I will close it — the report runs fine as an external script against the published data.

[rschu1ze]

The use case. People cite ClickBench rankings, including small differences between neighbouring entries. This script lets a reader check whether a difference that small is bigger than the measurement's own repeatability, using data ClickBench already publishes.

That makes no sense, sorry.