feat(proxy): serve from thread-per-core workers with per-worker upstream pools

[membphis]

What this changes

The proxy hands a request between threads about twice per request: once when a work-stealing worker picks up the task, again when the upstream response lands on whichever thread happens to own that connection. Each handoff costs a wakeup and a context switch. On a 4-core saturation run that is where most of the CPU goes — 89% of it is kernel time, at 232k context switches per second, 11.9 syscalls per request.

This serves instead from N independent workers, each with its own runtime, its own SO_REUSEPORT listener on proxy.addr, and its own upstream connection pool, so a request is accepted, dispatched, answered, and its upstream call polled all on one thread. Syscalls per request drop to 5.0 — 4 network send/recv plus the request-id getrandom — and context switches to 222/s.

Two bootstrap knobs, applied at startup (a restart is required to change either):

knob	default	meaning
proxy.thread_per_core	on for Linux, off elsewhere	the kernel spreading this relies on is a Linux behavior. Set false to serve from one shared runtime on any platform.
proxy.workers	parallelism available to the process	follows a cgroup CPU limit or a taskset affinity mask. Rejected at load when 0.

Both are reachable through the environment as AISIX_PROXY__THREAD_PER_CORE / AISIX_PROXY__WORKERS (note the double underscore), which is how a managed deployment sets them.

Which mode a process is running is visible without any new endpoint: the startup log prints one line per worker, and worker threads are named tpc-N (ps -T -p <pid>) versus tokio's tokio-rt-worker on the shared runtime.

Measurements

4 pinned vCPUs, local mock upstream, 25s windows, fail=0 everywhere. The only difference between legs is proxy.thread_per_core in the config file — no build flags, no environment tuning. Rig floor (mock direct, c=128) is 461,691 rps, ten times the loaded figures.

c	thread-per-core	work-stealing	delta
8	17,128 rps · 199% CPU · p99 0.87ms	23,398 rps · 363% CPU · p99 0.58ms	-26.8%
32	41,214 rps · 398% CPU · p99 1.38ms	23,965 rps · 365% CPU · p99 2.16ms	+72%
128	45,447 rps · 398% CPU · p99 4.38ms	24,297 rps · 365% CPU · p99 8.96ms	+87%
768, 10ms TTFT mock	39,873 rps · 398% CPU · p99 24.9ms	21,897 rps · 358% CPU · p99 55.4ms	+82%

c=128 was repeated four times: 44.0k / 44.4k / 44.7k / 46.1k against 24.0k / 24.1k / 24.2k / 24.7k. CPU per request falls from 150µs to 88µs.

The c=8 row is the documented cost, not a regression to fix: below about four client connections per worker the kernel's per-connection spreading leaves workers uneven, and CPU only reaches 199%. It is called out on the knob's own documentation. Real gateway traffic sits far above that connection count.

p99 at c=128 lands at 48-52% of the baseline across reps — halved, but sitting exactly at half rather than comfortably under it.

Verification

• Full e2e suite green in both modes: 175 files, 463 tests, 3 pre-existing skips. CI now runs it as a two-leg matrix (thread-per-core, work-stealing) so the fallback stays a real option.
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, git diff --check: clean.
• New unit tests: platform default resolution, explicit override beating the platform default, workers: 0 rejection naming the field, and a source scan holding every dispatch client to one user agent.
• Behaviour checked by hand against the built binary: per-worker startup logs and tpc-N thread names; available_parallelism honouring taskset -c 0-3 (4 workers, not 12); the fallback binding exactly one listener; graceful shutdown draining and exiting 0 in 1.01s; a second gateway on the same address failing loudly; workers: 0 refused at load.
• HTTPS is covered: the TLS listener path serves through from_tcp_rustls per worker, and listener-tls-e2e runs it in whichever mode the suite leg selects.

Reliability details worth reviewing

• Co-binding listeners would hide a port conflict. SO_REUSEPORT lets a second gateway bind the same address and silently take half the traffic. The mode probes the address with a non-reuseport bind first, so a conflict stays the loud startup failure it has always been.
• A worker leaving is fatal. Accept-loop error or panic unwinding, the first worker to exit decides the outcome for the process, rather than leaving it serving on fewer listeners than it reported binding.
• One RustlsConfig, cloned. Per-worker TLS configs would cut the session-resumption hit rate to 1/N, since rustls keeps that cache per ServerConfig.
• socket2 now declares features = ["all"] — set_reuse_port resolved before only because other dependencies happened to enable it.

Deliberate behaviour changes

• Upstream pools are per worker in thread-per-core mode. Idle upstream connections scale with the worker count, and pool_max_idle_per_host now applies per worker. One pool per worker serves every dispatch client, which is safe only while those clients agree on their user agent — a scan test enforces that and will fail anyone who gives one bridge its own.
• upstream-pool-idle-e2e is pinned to thread_per_core: false. It asserts that two sequential requests reuse exactly one upstream connection, which under per-worker pools becomes a question about which worker the kernel picked. Pinning keeps it measuring the idle deadline that is its actual subject. Flagging this for a reviewer rather than treating it as a test to relax.

Known gaps — please do not let these close with the PR

• api7/docs configuration page is not written. proxy.thread_per_core and proxy.workers are undocumented for users outside this repo's config.example.yaml comments. Needs a separate PR in api7/docs: both knobs, restart-to-apply, the platform default, the AISIX_PROXY__* double-underscore form, ps -T for checking the live mode, and the low-connection-count caveat.
• Graviton validation pending. Every number above is from a 12-vCPU VM, where context-switch cost is amplified; the gain should be smaller on bare metal. An m7g.4xlarge run against this branch is the release gate. If thread-per-core does not win there, flipping the Linux platform default back is a one-line change with no other code impact.
• Exporter pipelines pin themselves to one worker. ExporterPipelines::get_or_create spawns its delivery task from the request path, so in this mode it lands on whichever worker first emits a usage event for that exporter and stays there, carrying the whole process's export delivery on one worker. Not a correctness problem; worth its own issue.

No control-plane counterpart is required. These are bootstrap-config knobs read from the local config file and the environment, not etcd resources validated against cp-admin.yaml, so nothing here is gated on the closed schema.

Not in scope

The request-id getrandom per request and the metrics macro registration hot path are both real and both measured, but they are user-space work — 11% of total CPU on this rig — and land separately.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 47e9f6c6-26e0-4a1b-80b4-287a61a37d9b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}