feat(benches): 📈 track benchmarks with Bencher

[Panadestein]

🤖 AI text below 🤖

First slice of continuous benchmarking with Bencher: every commit on main runs the internal suite and uploads the results, which Bencher graphs over time and alerts on when a metric drifts outside its threshold.

This is deliberately the smallest end-to-end slice — it proves the pipeline before any PR gating is layered on.

What's here

benches/bmf.py	Converts one label's two run artifacts into Bencher Metric Format JSON
just bench-ci LABEL	The tracked benchmark profile
just bench-bmf LABEL	Emits the BMF report on stdout
.github/workflows/bench_main.yml	Runs both on every push to main, then bencher run
tests/test_bench_bmf.py	8 unit tests for the exporter

Why a custom exporter

Bencher accepts one --adapter per upload. The stock python_pytest adapter would keep the timings and silently drop everything else, so bmf.py merges both artifacts — pytest-benchmark's JSON and the /proc memory sampler's — into a single report. Four measures are tracked:

Measure	Unit	Source	Threshold
latency	nanoseconds	pytest-benchmark mean ±1σ	Student's t-test, upper 0.99
peak-memory	bytes	PSS high-water mark	percentage, upper 10%
resting-memory	bytes	PSS after the run settles	percentage, upper 10%
terms	count	operator size after propagation	exact match

terms is deterministic for a fixed seed and problem size, so any change at all is a real change in what the propagator computes — it gets an exact-match threshold rather than a statistical bound, which makes it the sharpest of the four.

Unlike just bench and just bench-smoke, bench-ci does not render REPORT.md; the BMF report is the artifact that matters here.

Benchmark sizing

bench-ci reuses the documented bench default sizes rather than introducing a second set of magic numbers to keep in sync. Measured locally at ~56 s, with the heavier Schrödinger operations landing in the 50 ms – 1 s range — comfortably above the runner noise floor. It bumps --bench-rounds to 5 (CI reports the mean) and drops the slow fixed models, which belong in a nightly job.

Alerts do not fail the build (yet)

--error-on-alert is intentionally absent. The timing measures need history on a shared runner before their noise floor is known, and a red main from GitHub runner variance would train everyone to ignore the signal. Gating belongs on pull requests, where base and head are benchmarked on the same runner and machine-to-machine variance cancels out.

Maintainer setup required

The workflow is a no-op on forks (it checks github.repository) and will fail on main until:

1. A Bencher project exists for the repo.
2. BENCHER_API_KEY is added as a repository secret.
3. Optionally, a BENCHER_PROJECT repository variable if the project slug isn't monoprop.

One caveat worth flagging

Benchmark names are Bencher's history key. Renaming or moving a bench_* test orphans its tracked series — noted in AGENTS.md next to the benches/report.py / benches/bmf.py entry so it isn't rediscovered the hard way.

Follow-ups (out of scope here)

• PR gating: bench_pr.yml (relative CB), bench_pr_closed.yml (bencher archive), and the fork-PR pull_request / workflow_run pair
• Enable --error-on-alert once a noise baseline exists — starting with terms, then latency
• Nightly job for the slow models and an MPI testbed
• Embed the Perf page in the docs benchmarks page

Validation

• just bench-smoke and just bench-ci ci-linux both run clean; BMF output verified against the real artifacts of each
• 585 passed, 8 deselected (non-MPI suite)
• prek run clean, including the workflow schema hook

[github-actions]

Docs preview: https://pr-223.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (bb3d28f) to head (00e2abe).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #223   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud