Establish the on-CPU profiling workflow: adopt cargo-flamegraph and add a dedicated profiling build profile

[membphis]

Background

When the gateway shows high CPU but lower-than-expected throughput, we need to see exactly where the cycles go. The standard diagnostic for this is an on-CPU flame graph produced by a sampling profiler — the same flame-graph-first workflow that is battle-tested in the Nginx/OpenResty world.

We surveyed the Rust ecosystem (The Rust Performance Book recommendations, plus the profiling setups documented by TiKV, Databend, GreptimeDB, RisingWave, Quickwit, DataFusion, and the rustc team). The landscape is well converged, and we are adopting cargo-flamegraph as the standard tool for this scenario. This issue pins that decision, sketches the intended usage, and tracks the repo changes needed to make flame graphs actually readable for aisix builds.

Scenario → tool mapping

Scenario	Tool	Status
On-CPU hotspots — CPU is high, QPS is low; "what is burning CPU?"	cargo-flamegraph (wraps perf on Linux / xctrace on macOS; renders an interactive SVG via inferno)	this issue
Interactive drill-down — per-thread timelines, call trees, inverted stacks	samply (Firefox Profiler UI); benefits from the same build profile	optional companion; no repo change needed
On-demand profiling of a live production process over HTTP	pprof-rs embedded endpoint (the /debug/pprof/profile pattern used across the Rust infra ecosystem)	out of scope — to be filed separately
Async stalls — tasks parked on .await, scheduling problems; CPU is not hot	tokio-console / off-CPU profiling	out of scope

Boundary to keep in mind: an on-CPU flame graph answers "what is burning CPU". Tasks parked on .await consume no CPU and are invisible to it; that class of problem needs the async tooling instead.

Usage sketch

A dedicated internal guide with the full walkthrough will follow; this is the shape of it:

cargo install flamegraph

# Linux: perf needs privileges — either run with sudo,
# or lower kernel.perf_event_paranoid for the session:
sudo sysctl kernel.perf_event_paranoid=1

# Build, run the gateway, sample until exit, render the SVG — release is
# cargo flamegraph's default profile, and release binaries are profileable
# by design (the bin target is named `aisix`; the package is `aisix-server`):
cargo flamegraph --bin aisix -- <args>

# Or attach to any already-running aisix process — including a user's
# production deployment — while live traffic or a load test drives it:
flamegraph --pid <aisix-pid>   # Ctrl-C to stop sampling and render

Notes that belong in the guide:

• Sampling uses DWARF-based unwinding by default; Rust always emits .eh_frame, so no frame-pointer rebuild is required.
• Pre-session sanity gate on the target binary (built or deployed): nm /path/to/aisix | grep -c ' [tT] ' should report hundreds of text symbols; a near-zero count means a stripped binary sneaked in.
• Consider --remap-path-prefix later if embedding builder paths in shipped binaries becomes a concern; decide in the guide.
• Expect tokio's physical stacks: the bottom of every stack is the runtime worker loop, business logic appears as nested poll() frames, and spawned tasks are rooted at the scheduler rather than at their spawn site.
• Read the graph for wide frames (cumulative CPU), not tall ones.

Required repo changes

1. Make release binaries themselves profileable (the actual blocker today).

aisix ships as a single binary, and performance problems surface on the build users actually run — asking a user to reproduce on a special diagnostic build is not a workable support model. The shipped artifact must be the profileable one. [profile.release] currently sets strip = "symbols", so shipped binaries carry no symbol table and a flame graph of them degrades to bare addresses. Change release itself:

[profile.release]
lto = "thin"
codegen-units = 1
strip = "debuginfo"  # keep .symtab (function-level flame graphs); all DWARF stays out

Function-level frames are the deliverable. Full line tables were evaluated and rejected: with codegen-units = 1 + thin LTO they measure ~142 MB of DWARF (~4x the shipped binary) — not worth file:line attribution in every image. Codegen is unchanged (lto/codegen-units untouched), so this trades ~15% binary size (46.7 → 53.8 MiB, measured on the PR) for field diagnosability. When a deep dive needs file:line/inline attribution, build ad hoc:

CARGO_PROFILE_RELEASE_DEBUG=line-tables-only CARGO_PROFILE_RELEASE_STRIP=none cargo build --release

2. Verify the rust-lld --no-rosegment caveat.

The toolchain pins 1.93.1; since Rust 1.90, x86_64 Linux links with rust-lld by default, and perf is known to mis-unwind lld-linked binaries unless the linker gets --no-rosegment (see the cargo-flamegraph README). Verify on a profiling build; if stacks come out broken, bake RUSTFLAGS="-C link-arg=-Wl,--no-rosegment" into the documented profiling invocation.

3. Optional, only if needed later. If DWARF-mode sampling overhead or truncated deep stacks ever become a problem, the fallback is building with -C force-frame-pointers=yes and sampling in frame-pointer mode. Not in scope for this issue; decide in the guide if it comes up.

Acceptance / deliverables

• [profile.release] keeps the symbol table (strip = "debuginfo"), with the binary-size delta measured and recorded on the PR, and a CI guard asserting the shipped image binary stays symbolized
• One real profiling session against the aisix binary under load (e2e traffic or a load generator), with the rendered SVG attached to this issue — acceptance bar: frames show crate::module::fn names with file:line, tokio runtime frames visible, no address-only towers
• rust-lld --no-rosegment behavior verified on 1.93.1 and the outcome recorded in the guide
• Internal profiling guide written (scenario → tool mapping, full command walkthrough, how to read async/poll stacks); kept as internal engineering material and linked from here when ready
• Follow-up issue filed for the production-facing option: a pprof-rs /debug/pprof/profile endpoint on the admin/status surface

References

• https://github.com/flamegraph-rs/flamegraph
• https://nnethercote.github.io/perf-book/profiling.html
• https://doc.rust-lang.org/cargo/reference/profiles.html
• https://github.com/mstange/samply
• https://github.com/tikv/pprof-rs

[coderabbitai]

🔗 Related PRs

#849 - build: keep symbols and line tables in release binaries for field profiling [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!