routercap: capacity benchmark harness for atenet-router

[Chuang Wang (chuangw6)]

Part of #665.

A one-command capacity sweep for atenet-router: each arm sets one Envoy CPU size and walks a rising load ladder against a dedicated cluster; every run produces a self-contained report.html.

• Open-loop pacing with latency timed from scheduled send, so coordinated omission is impossible and a struggling generator cannot flatter the router.
• Measures throughput/latency, a per-hop breakdown of the mean request (generator / Envoy / ext_proc sidecar / worker), per-thread Envoy CPU, and container CPU/memory on cAdvisor-aligned windows.
• Six guards distinguish "the rig ran out" from "the router ran out" and mark arms rig-limited.

Methodology and reproduction steps are in benchmarking/routercap/README.md. Findings from the first full study land in #808.

🤖 Generated with Claude Code

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Max Smythe (maxsmythe)]

Thank you for the PR! Have some generalized thoughts/feedback

[Max Smythe (maxsmythe)]

I am not sure this is 100% accurate, especially with Boomer.

Is there anything preventing a Locust user from spinning off a goroutine per-iteration? That would ensure any outbound requests are not blocking, removing the "closed-loop" effect.

Not implying this means all benchmarks must use Locust/boomer, but we should be accurate in our reasoning.

Reasons for using Locust/Boomer (may not be comprehensive):

• Unified entry point for running benchmark tests and scraping results, simplifying any automation of benchmarking testing.
  • This may or may not have an impact on how easy it is to bootstrap necessary infrastructure (procuring a cluster to run the test on, spinning up Substrate with the necessary config, etc.), since the current work around that is built around Locust but is not actually part of Locust.
• Built-in protocol for aggregating client-level observations across multiple worker pods, allowing scaling traffic beyond one machine and gathering things like percentiles that require non-trivial aggregation.
• Ability to easily customize traffic shape, especially across distributed workers (e.g. the laddering you do)

Reasons for not using Locust (again, may not be comprehensive):

• Needing to work around framework, making the test brittle or overly complex
• Lack of need for any of the above benefits

Is this benchmark something we expect to run regularly?

Also, I have not had a chance to look at this PR as it is very large (12k lines of code).

It appears that large chunks of this are dedicated to data visualization (e.g. charts.py), bootstrapping infrastructure, and scraping data sources.

Data vis -- any chance this can be delegated to an external tool? Generally speaking, benchmarks (especially if they are regularly run), export data in a standardized format that let users choose how they slice, dice and visualize the data. Particularly if these are regularly-running benchmarks.

Bootstrapping infrastructure -- we should try to avoid idiosyncratic bootstrapping where possible. I have not dug too deeply into this PR, so some of this could be justified, but that would be the exception, not the rule.

Scraping data sources -- The ability to scrape different data points and gather statistics on them is super useful. We should commoditize and share these capabilities as much as possible -- example, I would love to sample router stats as part of Locust.

In general, it seems like benchmarking/routercap could use purpose-built sub-packaging. It is hard to tell what class/function is defined where. More modules can help make that clearer via imports. It is also a step towards making the stats scraping shareable.