[Bug]: `streaming_resource_bench` process RSS column is multimodal and invites false perf claims

[mohamedmansour]

Before filing

• I searched existing issues and did not find a duplicate.
• This is not a security vulnerability.

Area

Performance

Summary

The process RSS column in streaming_resource_bench is multimodal across runs of an identical binary, with a spread wider than the effect sizes it gets quoted for. It has already caused two incorrect claims in PR descriptions. It measures fixture construction more than the render path it sits next to.

Minimal reproduction

Build the benchmark once and run it repeatedly without rebuilding:

cargo build --release --example streaming_resource_bench -p microsoft-webui
for i in $(seq 1 8); do
  ./target/release/examples/streaming_resource_bench | grep "POOLED/1000" | awk '{print $(NF-2)}'
done

Steps to reproduce

1. Run the loop above on an unmodified checkout.
2. Compare the process RSS values across the 8 runs of the same binary.
3. Observe that they land in several discrete states rather than clustering.

Expected behavior

A memory column in a benchmark table should be stable enough across runs that a reader can attribute a difference between two builds to the code change. Either the column is reproducible, or the table makes clear it is not comparable build-to-build.

Actual behavior

Eight runs of one unchanged binary produced:

12.66  12.73  12.94  13.20  13.70  13.97  13.98  14.02   (MiB)
min 12.66 | median 13.45 | mean 13.40 | max 14.02 | spread 1.36

A 1.36 MiB spread with no code change at all. For comparison, PR #511 originally quoted a 1.29 MiB RSS "improvement" as a headline metric - smaller than the noise floor of the measurement.

Two independent problems compound here:

1. It is multimodal, so small n cannot distinguish a mode from a trend. Values cluster in a few discrete states spanning ~1 MiB rather than varying smoothly, so n=1 or n=3 samples can easily catch the low mode for one build and the high mode for the other and read as a large regression or improvement. This is not a hypothetical: it produced a bogus -9.1% improvement claim in #511 and a bogus +1.5 MiB regression claim in #513, in opposite directions, from the same column.

2. It is dominated by fixture construction, not rendering. The peak is reached while the ~6.8 MiB scale-1000 state is being built, before any render executes. Instrumenting the benchmark's GlobalAlloc to track live heap directly shows peak live heap and live-heap-at-exit are byte-identical between builds that the RSS column reported as differing by 1.5 MiB. So the column largely reflects where the allocator happened to place one large fixture object, not anything about the render path.

The header note already says the column is "only meaningful as a peak" and "cumulative across all phases", which is accurate but evidently not a strong enough warning - it does not convey that the value is not comparable between two runs, which is exactly what a reader wants to do with it.

Suggested fix

Some combination of:

• Add a live-heap high-water metric from the existing counting allocator. The benchmark already wraps GlobalAlloc; tracking live and peak_live alongside ALLOC_COUNT and ALLOC_BYTES is a small change and measures retention exactly, with no allocator slack or OS paging in the way. This is the metric people actually reach for the RSS column hoping to get.
• Relabel or drop process RSS. If it stays, rename it to something that signals non-comparability and note the observed run-to-run spread inline, so nobody quotes a delta from it.
• Optionally build the fixture states outside the measured region, or measure each scale in a separate process, so the column stops being dominated by fixture construction.

Also worth updating BENCHMARKS.md, which currently lists this example under "exact alloc count + bytes + getrusage CPU + RSS" for "proving zero-alloc claims"; the alloc/bytes half of that is exact and trustworthy, the RSS half is not, and the doc does not distinguish them.

Environment

• OS: macOS (Darwin), Apple Silicon
• Rust version: workspace toolchain
• Node.js version: n/a
• WebUI package or CLI version: reproduces on main at 78b7617
• Command: cargo build --release --example streaming_resource_bench -p microsoft-webui then repeated direct invocation

Logs or terminal output

built. sampling RSS at scale 1000, n=8 on IDENTICAL binary:
13.70 12.94 12.73 13.98 13.97 13.20 12.66 14.02
n=8 min=12.66 median=13.45 mean=13.40 max=14.02 spread=1.36 MiB

---

Found while reviewing #511 and #513. Both PR descriptions have been corrected; allocs/run and bytes/run from the counting allocator are exact and unaffected by this, and remain the right metrics for those changes.