npx flakehunter --runs 10 --reports ./test-results -- npm test2 flaky tests across 10 runs
40% checkout › charges the card
failed 4/10 runs, on 1, 4, 5, 9
×3 ECONNREFUSED connecting to payments
×1 timeout of 5000ms exceeded
→ Fails 2 different ways, which usually means a race — an await that is
not awaited, a timer, or a shared fixture being mutated while another
test reads it.
20% checkout › sends a receipt
failed 2/10 runs, on 3, 7
×2 Expected 1 email, received 0
→ Fails the same way each time. That usually means order dependence or
shared state, not timing.
Everyone knows which tests are flaky. Nobody knows why, so they get retried, then skipped, then deleted — and whatever bug they were catching goes with them.
The useful question is not "which tests are flaky" but "what kind of flaky", because the two kinds need completely different fixes and are trivially distinguishable if anyone bothers to look.
Fails several different ways → a race. The timing decides how it breaks: an unawaited promise, a timer, a shared fixture mutated while another test reads it.
Fails the same way every time → order or shared state. Something is either present or it isn't. Run the suite in a different order and it will reproduce.
Fails in consecutive runs → leaked state that survives between runs, rather than a race, which would scatter.
That is most of the diagnosis, and it falls straight out of comparing failure messages across runs.
A test that fails every run is not flaky. It is broken. It gets counted separately and kept out of the list. Conflating the two is exactly what makes flake dashboards useless — the broken tests drown the interesting ones.
A big timing ratio is not evidence. A test varying between 0.05ms and 1.2ms is a twentyfold difference and completely meaningless. The timeout hint only appears when the slow run is slow in absolute terms too.
It reads JUnit XML, which everything emits. flakehunter never needs to know what language you write tests in.
| Vitest | --reporter=junit --outputFile=./test-results/run.xml |
| Jest | --reporters=jest-junit |
| pytest | --junitxml=test-results/run.xml |
| Go | gotestsum --junitfile test-results/run.xml |
| RSpec | rspec_junit_formatter |
--runs <n> How many times to run the suite (default: 5)
--reports <dir> Where your runner writes its JUnit XML (required)
--fail-at <pct> Exit non-zero at this flake rate (default: 1)
--format <pretty|json>
--keep-going Do not stop when a run produces no output
Exit code is 1 when anything flakes at or above --fail-at, so it gates a merge without extra scripting.
import { analyse, parseRun, advise } from 'flakehunter';
const analysis = analyse([
{ run: 1, results: parseRun([xmlFromRunOne]) },
{ run: 2, results: parseRun([xmlFromRunTwo]) },
]);
for (const flake of analysis.flakes) console.log(flake.id, advise(flake));Flakiness is probabilistic. A test that fails one run in fifty will very likely survive ten runs untouched, and a clean report means "nothing surfaced in this many runs" rather than "your suite is reliable". More runs find more; nothing finds everything.
Nothing at runtime. A tool you run in CI should not itself be a supply chain.
MIT.