test(perf): add ingest-ceiling harness for the audit write path

[odedlaz]

The backend perf deep-dive predicts a hard ingest ceiling from the six sequential round trips in AuditService::log, which sits on the OK path. Nothing in this repo could measure it: there are no criterion benches, and perf/ deliberately covers only the Redis fan-out boundary. This adds the missing instrument. No production code changes.

Three pieces:

• scripts/start-perf-ingest-rig.sh — one relay process serving two communities resolved by Host (a.localhost / b.localhost both reach 127.0.0.1, so the URL host is the header), on the isolated buzz-harness Compose project. --skip-relay attaches to a relay someone else supervises.
• crates/buzz-test-client/src/bin/ingest_load.rs — paced generator driving those communities at independently settable rates from one clock.
• perf/relay_ingest_ceiling.py — owns the experiment and the verdict, exits non-zero on a violated contract, --mode model for review without services. Follows relay_bus_scaling.py.

Two measurement choices that carry the result

Pacing. The generator advances a fixed schedule and never rebases it on the response, and it reports latency from the intended slot as well as from the actual send. A generator that paces with a self-correcting timer and times from the send — the shape wamp_bench.rs uses — silently redefines its own offered rate downward when the relay slows, caps throughput at connections / latency, and keeps the queueing delay out of its percentiles. That hides the damage exactly when the damage is the measurement.

Admission limits. The rig raises them, and this is not incidental. At defaults one identity is capped at 50 events per 5s (human_ws_events_per_sec = 10, widened by ws_admission_budget into a 5s window), and a rejected EVENT gets a NOTICE, which carries no event id — so a NIP-01 client waiting for OK blocks for its whole publish timeout instead of learning it was rejected. Measured here: 50 events land in 2.4s, the next send stalls 30s, the run self-truncates, and the surviving numbers read as a textbook saturation knee at ~1.5/s. A sweep at defaults measures the limiter. The runner therefore invalidates any run where buzz_admission_rejections_total{reason="quota"} moves, and prints the configured limits in the run metadata.

What it measured, and what has since been withdrawn

Standing: the arm separation. With the audit log enabled, delivered/offered collapses
to 0.62 at 800/s and 0.25 at 1600/s. With BUZZ_AUDIT_ENABLED=false, the same rates
deliver 1.0000, and audit_log stayed flat at 63,723 rows through that sweep, so the
control is verified positive rather than assumed. Round 1 also recorded zero quota
rejections, zero admission-unavailable warnings across 302,531 relay log lines, and
zero rejected events or transport errors across all 20 runs. Splitting the same offer
across two communities did not raise the combined ceiling.

Withdrawn: every absolute ceiling figure, including the "~400–494/s plateau" and the
round-trip time back-solved from it. Review found that an initially-empty 1000-slot audit
queue lends up to 1000/duration free accepts to every fixed-duration run — +100/s at
10s — so the numbers are biased upward by an unmeasured amount, consistently, in a way no
number of repeats corrects. The repeats were also run back-to-back inside the queue's
drain window, so they inherited each other's backlog rather than being independent draws.

That was a defect in this harness, and it is fixed in ac64b0bf0 along with the rest of two review panels' findings.

What changed after review

The pass predicate was the load-bearing problem. It derived a threshold from the spread of repeated runs, which is ~0 in the region where accepted/offered is pinned at 1.0 and is the system's own throughput variability in the region where it is not — so no placement of that control made it mean anything. The contract is now that the two arms separate: repeats give each arm an interval and the verdict asks whether the difference between audit-off and audit-on excludes zero at some rate. Nothing gates on the knee, and no comparison reads overlapping intervals as evidence of equality.

Also closed, each of them a route to passing without establishing anything:

• a single-arm dataset returning ok with attribution never tested, and an audit-off null indistinguishable from a control that never ran — the verdict now carries an explicit ran/skipped marker;
• --combine accepting halves from different durations, rate grids, limiter settings or source revisions;
• only quota rejections invalidating a cell, when unavailable rejections take the same NOTICE-without-OK path and cost Redis round trips against the rig the sweep is loading, so they are load-correlated and can forge a knee that survives repeats;
• the generator failing with its error context captured into an exception instead of reaching the operator, and a null percentile aborting the sweep before any report was written;
• the rig killing a pid from a stale file without checking it still belongs to this harness.

Cells now carry outstanding_delta and audit_busy_fraction, and the worker-rate estimate is drawn only from cells where the worker was demonstrably busy in steady state. The steady-state criterion is stability, not emptiness — a saturating cell settles with the channel full, and a gate demanding an empty queue would have made every cell that matters unmeasurable.

Scope and limits, stated in the doc

The harness characterizes the mechanism, not deployability at scale — a raised-limit sweep cannot show that a real community reaches these rates, since production defaults would need hundreds of concurrent identities each paying its own admission round trip. Sensitivity to round-trip latency is unmeasured; a sweep of injected delays should predeclare six added exchange delays plus a durability intercept, since disabling synchronous_commit removes the flush wait but not the COMMIT exchange. And the per-community lock ceiling is structurally blind in this round: the worker ceiling is lower and masks it, so a passing run is not evidence the lock is fine.

Verification

cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all --check clean; cargo test -p buzz-test-client green; python3 -m unittest discover -s perf -p 'test_*.py' 51 tests green; shellcheck clean. The unit tests pair every passing case with a mutant that must fail — a lone dip that must not count as a knee, a control that did not run, an arm separation in the wrong direction, a cell that banked the whole channel, a --combine across mismatched durations — because a contract that cannot go red is decoration.

Larger than the usual single-pass PR, and I'd rather not split it: the generator, the rig, and the verdict logic are not independently useful, and the doc and the mutant tests are the parts that make the numbers trustworthy.