From e271d00f3e6e72f519ac4aa1647eacd9d93cdc68 Mon Sep 17 00:00:00 2001 From: Shreemaan Abhishek Date: Tue, 4 Aug 2026 11:04:31 +0000 Subject: [PATCH 1/2] bench: generate both columns of every comparison from one shape table The old harness hand-wrote one target location per client, so the two sides drifted into different API shapes and the published ratio credited that drift to the C-vs-Lua difference. Every case is now a driver plus a shape, and both columns of a pair are built from the same shape table, so they send the same headers, size the same pool and consume the body the same way by construction rather than by review. A fairness audit diffs the raw request each client actually put on the wire and fails the run if the caller headers disagree. benchmark/cases.txt is the single source of truth: run.sh generates one location per name and the smoke check diffs it against the table loaded in the worker, so the two cannot drift. The matrix covers read mode, connection lifetime, method, transport, peer address, response framing and header count. Also adds repeats with median reporting, a saturation gate that discards any window where the target worker was not CPU-bound, CSV output, fold.sh to render it, a separate fixed-rate latency phase (latency measured at saturation is queue depth, not latency), and connection-reuse accounting taken from the upstream so it stays off the measured worker's hot path. --- .gitignore | 2 +- README.md | 62 ++- benchmark/README.md | 492 ++++++++++++++----- benchmark/cases.txt | 59 +++ benchmark/conf/target.nginx.conf | 229 +++------ benchmark/conf/upstream.nginx.conf | 185 ++++++- benchmark/fold.sh | 225 +++++++++ benchmark/lua/bench.lua | 555 +++++++++++++++++++++ benchmark/results-full.md | 199 ++++++++ benchmark/run.sh | 756 +++++++++++++++++++++++++---- docs/ai-proxy-integration.md | 163 +++++++ t/003-benchmark-runner.t | 118 ++++- 12 files changed, 2637 insertions(+), 408 deletions(-) create mode 100644 benchmark/cases.txt create mode 100755 benchmark/fold.sh create mode 100644 benchmark/lua/bench.lua create mode 100644 benchmark/results-full.md create mode 100644 docs/ai-proxy-integration.md diff --git a/.gitignore b/.gitignore index a9e5ca0..0ef3281 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ docs/superpowers/ -benchmark/.run/ +benchmark/.run*/ t/servroot/ objs/ diff --git a/README.md b/README.md index 6cd5b6e..67abf9a 100644 --- a/README.md +++ b/README.md @@ -201,20 +201,52 @@ protocol work is intentionally deferred. ## Benchmark The local benchmark pins the target OpenResty worker to one CPU core and -compares three request paths: a no-upstream baseline, the C FFI client, and -`resty.http`. The latest 15-second local runs used wrk2 with a saturated target -worker. - -| wrk2 connections | no-upstream baseline | C FFI client | `resty.http` | C FFI / `resty.http` QPS | outbound cost ratio (`resty.http` / C FFI) | -| --- | ---: | ---: | ---: | ---: | ---: | -| `10` | `83998.04` QPS | `32831.68` QPS | `16123.19` QPS | `2.04x` | `2.70x` | -| `100` | `96155.73` QPS | `38582.72` QPS | `14665.62` QPS | `2.63x` | `3.72x` | - -The C FFI path delivers substantially higher end-to-end throughput than -`resty.http`, and the baseline-derived estimate shows `resty.http` spending -about `2.70x` to `3.72x` as much outbound client CPU time as the C FFI path in -these local runs. See -[benchmark/README.md](benchmark/README.md) for the benchmark topology, -reproduction notes, and detailed measurements. +compares a no-upstream baseline against both clients across a matrix of request +shapes: one-shot and stateful, buffered and streaming, plaintext and TLS, IP +literal and hostname, content-length and chunked framing. + +Median of 5 repeats, 30s measured each, target worker saturated on every row, +no non-2xx responses, `llhttp` parser, 1KB response body, 100 wrk2 connections: + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| one-shot (`request_uri`) | `30154` QPS | `14785` QPS | `2.04x` | `2.64x` | +| stateful object | `27316` QPS | `15382` QPS | `1.78x` | `2.16x` | + +Baseline without an upstream call: `82185` QPS. The outbound cost ratio +subtracts that baseline from both client paths and compares what is left, which +is the closest this gets to isolating the client itself. + +Per-request Lua heap churn, which is the direct test of the claim that response +handling stays in C: + +| shape | C FFI | `resty.http` | +| --- | ---: | ---: | +| one-shot | `2.49` KB | `7.45` KB | +| stateful | `3.33` KB | `6.49` KB | + +**These supersede the previously published `2.04x`-`2.63x` QPS and +`2.70x`-`3.72x` outbound figures, which were withdrawn.** Those compared the C +one-shot fast path against a stateful `lua-resty-http` object, so the ratio +contained the API-shape difference as well as the C-versus-Lua difference and +credited all of it to the latter. They also predate per-request token +validation, the case-insensitive header table with repeated headers folded into +arrays, `llhttp` as the default parser, interim `1xx` consumption, and trailer +parsing, all of which add hot-path work. Matching the shapes and re-measuring +the current library moves the honest one-shot figure from `2.63x` to `2.04x`. + +Across the full case matrix the ratio holds between `1.68x` and `1.96x` for +streaming, chunked framing, hostname resolution, POST bodies, and header-heavy +responses, and reaches `4.41x` on responses with trailers. + +**Where this client loses.** The advantage is in per-request work, so it shrinks +as connection setup takes over the request and eventually reverses: `1.49x` on +pooled TLS, `1.19x` on short-lived plaintext connections, and `0.79x` on a fresh +TLS handshake per request, where `lua-resty-http` is about 21% faster. Anything +that cannot hold a keepalive pool, TLS to many short-lived peers most of all, is +a case for the other client today. + +See [benchmark/README.md](benchmark/README.md) for the topology, the full case +matrix, the fairness audit, and how to reproduce. Design notes and implementation plans used during development live under `docs/superpowers/` and are intentionally ignored by git. diff --git a/benchmark/README.md b/benchmark/README.md index fc2674a..0d43dde 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,138 +1,416 @@ # Benchmark -This directory contains the local benchmark for comparing -`resty.ngx_http_ffi_client` with the Lua cosocket baseline implemented with -`lua-resty-http` / `resty.http`. +This directory holds the local benchmark comparing +`resty.ngx_http_ffi_client` with `lua-resty-http` / `resty.http`. The benchmark is scoped to the current HTTP/1.1 client work. It validates the C/Nginx module path and the llhttp-based HTTP/1 direction; it does not evaluate HTTP/2, HTTP/3, or alternative protocol APIs. -The benchmark measures maximum QPS when the target OpenResty worker is pinned to -one CPU core and that core is saturated. It is not a fixed-rate latency test. +Two things are measured, in two separate passes: -The runner measures a baseline first. The baseline endpoint does not contact the -upstream; it only returns a response body with the same size as the upstream -response. The C FFI and `resty.http` cases are then compared against this -baseline to estimate the outbound HTTP client cost. +- **Throughput**: maximum QPS with the target OpenResty worker pinned to one + core and that core saturated. +- **Latency**: p50/p99/p99.9 at a fixed rate below saturation. Latency measured + at saturation is queue depth, not client cost, so it gets its own pass. -## Current Result Summary +A baseline endpoint that returns the same response body without contacting the +upstream runs alongside both, so the downstream leg can be subtracted out. -The latest local runs compare three paths: +## Results -- no-upstream baseline: target OpenResty returns a fixed response body directly -- C FFI client: target OpenResty sends one upstream HTTP request with - `resty.ngx_http_ffi_client` -- `resty.http`: target OpenResty sends one upstream HTTP request with - `lua-resty-http` +Target worker pinned to CPU `0`, upstream on `1-5` with 5 workers, wrk2 on +`6-7`. Median of 5 repeats, 10s discarded warmup and 30s measured per repeat, +`llhttp` parser, `RESPONSE_SIZE=1024`, `WRK_CONNECTIONS=100`. Every row below +was saturated (target worker CPU >= 95%) and reported no non-2xx responses; no +rows were discarded. -Both load points used `WRK_DURATION=15s`, `WRK_RATE=100000`, -`WRK_THREADS=2`, `KEEPALIVE_POOL_SIZE=120`, `KEEPALIVE_IDLE_TIMEOUT=60000`, -and `RESPONSE_SIZE=1024`. The target worker was pinned to CPU `0`, the -upstream workers used CPUs `1-5`, wrk2 used CPUs `6-7`, and the upstream used -`UPSTREAM_WORKERS=5`. +### Headline -| wrk2 connections | no-upstream baseline | C FFI client | `resty.http` | C FFI / `resty.http` QPS | outbound cost ratio (`resty.http` / C FFI) | -| --- | ---: | ---: | ---: | ---: | ---: | -| `10` | `83998.04` QPS | `32831.68` QPS | `16123.19` QPS | `2.04x` | `2.70x` | -| `100` | `96155.73` QPS | `38582.72` QPS | `14665.62` QPS | `2.63x` | `3.72x` | +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `oneshot` | `30154.14` | `14785.18` | `2.04x` | `2.64x` | +| `stateful` | `27316.07` | `15381.85` | `1.78x` | `2.16x` | -The QPS ratio is the direct end-to-end throughput comparison between the C FFI -client and `resty.http`. The outbound cost ratio subtracts the no-upstream -baseline from both client paths and compares the estimated upstream HTTP client -CPU cost as `resty.http / C FFI`. In these runs, the C FFI client is about -`2.04x` to `2.63x` faster end-to-end, and `resty.http` spends about `2.70x` to -`3.72x` as much estimated outbound client CPU time. +Baseline (no upstream call): `82184.57` QPS, `12.168` us/request. -## Detailed Local Measurements +### Detail -All target-worker CPU averages are above the default `95%` saturation threshold, -and no `Non-2xx` responses were reported in these runs. - -| wrk2 connections | case | QPS | target worker CPU avg | saturation | -| --- | --- | ---: | ---: | --- | -| `10` | no-upstream baseline | `83998.04` | `108.6%` | valid | -| `10` | C FFI client | `32831.68` | `109.1%` | valid | -| `10` | `resty.http` | `16123.19` | `105.6%` | valid | -| `100` | no-upstream baseline | `96155.73` | `103.9%` | valid | -| `100` | C FFI client | `38582.72` | `110.1%` | valid | -| `100` | `resty.http` | `14665.62` | `106.4%` | valid | - -The baseline-derived per-request estimates are: - -| wrk2 connections | baseline downstream + handler | C FFI total | C FFI estimated outbound | `resty.http` total | `resty.http` estimated outbound | outbound cost ratio | +| case | median QPS | min | max | repeats used | worker CPU avg | worker RSS | | --- | ---: | ---: | ---: | ---: | ---: | ---: | -| `10` | `11.905 us` | `30.458 us` | `18.553 us` | `62.022 us` | `50.117 us` | `2.70x` | -| `100` | `10.400 us` | `25.918 us` | `15.519 us` | `68.187 us` | `57.787 us` | `3.72x` | - -The main project advantage shown by this benchmark is that the C FFI client -keeps outbound HTTP request handling in Nginx C module code while preserving -OpenResty worker nonblocking behavior and keepalive reuse. Under a saturated -single target worker, this reduces Lua cosocket overhead on the upstream client -path and leaves more of the worker CPU budget for application logic. +| `baseline` | `82184.57` | `81513.32` | `85599.33` | 5 | `109.3%` | `37300` KB | +| `ffi.oneshot` | `30154.14` | `29789.00` | `30489.65` | 5 | `116.7%` | `37300` KB | +| `ffi.stateful` | `27316.07` | `27143.61` | `27724.92` | 5 | `115.9%` | `37300` KB | +| `resty.oneshot` | `14785.18` | `13465.86` | `14855.69` | 5 | `113.5%` | `37300` KB | +| `resty.stateful` | `15381.85` | `15275.92` | `15617.69` | 5 | `114.3%` | `37332` KB | + +### Latency, fixed rate at 60% of the slower path + +| case | rate | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | ---: | +| `ffi.oneshot` | `8871` req/s | `1.060` ms | `2.450` ms | `5.430` ms | +| `resty.oneshot` | `8871` req/s | `1.370` ms | `3.920` ms | `10.550` ms | +| `ffi.stateful` | `9229` req/s | `1.160` ms | `3.360` ms | `20.940` ms | +| `resty.stateful` | `9229` req/s | `1.400` ms | `3.740` ms | `6.800` ms | + +At those fixed rates the worker sat at `45.6%` CPU on `ffi.oneshot` against +`72.4%` on `resty.oneshot`, which is the same comparison measured from the +other side. + +### Per-request Lua heap delta + +| case | KB/request | +| --- | ---: | +| `ffi.oneshot` | `2.494` | +| `ffi.stateful` | `3.331` | +| `resty.stateful` | `6.492` | +| `resty.oneshot` | `7.448` | + +### Matrix + +The full case matrix, median of 2 repeats (`BENCH_CASES=all`, same host and +topology, `matrix.csv`). Every row was saturated with no non-2xx responses and +nothing was discarded; per-case spread was under 1% on most cases. + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `trailers` | `27919.64` | `6325.35` | `4.41x` | `6.10x` | +| `oneshot` | `28052.14` | `14314.11` | `1.96x` | `2.44x` | +| `hdr40` | `17337.44` | `9235.26` | `1.88x` | `2.10x` | +| `post` | `24121.31` | `12826.85` | `1.88x` | `2.23x` | +| `req30` | `23370.36` | `13014.28` | `1.80x` | `2.10x` | +| `cookies` | `24343.96` | `13719.88` | `1.77x` | `2.09x` | +| `chunked` | `26844.53` | `15521.60` | `1.73x` | `2.07x` | +| `dns` | `26937.76` | `15538.12` | `1.73x` | `2.08x` | +| `stateful` | `26365.78` | `15289.51` | `1.72x` | `2.05x` | +| `stream` | `26490.00` | `15686.39` | `1.69x` | `2.00x` | +| `readbody` | `26051.07` | `15462.52` | `1.68x` | `1.99x` | +| `tls` | `17732.35` | `11861.95` | `1.49x` | `1.63x` | +| `short` | `8275.94` | `6973.64` | `1.19x` | `1.21x` | +| `tlsshort` | `872.96` | `1108.38` | `0.79x` | `0.79x` | + +Baseline: `84504.18` QPS. + +### Where the FFI client does not win + +**`tlsshort`, a fresh TLS handshake per request: `0.79x`.** `872.96` QPS against +`resty.http`'s `1108.38`, so this client is about 21% slower. The bottom of the +table reads as one story rather than three: `tls` (`1.49x`), `short` (`1.19x`) +and `tlsshort` (`0.79x`) are exactly the cases where connection setup dominates +the request. The advantage this client has is in per-request work, and it +shrinks as setup takes over, then reverses once setup is nearly all of it. +Anything that cannot hold a keepalive pool, and TLS to many short-lived peers +most of all, is a case where `lua-resty-http` is currently the better choice. + +**`ffi.stateful` tail latency.** `p99.9` of `20.940` ms against +`resty.stateful`'s `6.800` ms, a roughly 3x worse tail, even though `p50` and +`p99` are both better. `ffi.oneshot` shows no such tail at `5.430` ms, so +something on the stateful object's path stalls occasionally in a way the +one-shot path does not. + +This one is **not yet evidence**: the latency phase ran a single 30s window per +case, and tail percentiles are the noisiest thing measured here. Re-run with +`BENCH_LATENCY_REPEATS=3` before acting on it, and attribute with `perf` (A7) if +it survives. + +### Reading the matrix numbers + +Two caveats on the matrix table specifically, neither of which applies to the +five-case headline run above: + +- **Two repeats, not five.** Enough for the ranking, which is stable, and the + min/max spread is under 1% on most cases. Not enough to argue about a + difference of a few percent between adjacent rows. +- **Worker RSS is cumulative in an `all` run and should be ignored there.** It + climbs from about `527` MB on the early cases to `1.5` GB on the late ones, + which is 28 separate keepalive pools of up to `120` connections each + accumulating over the run, not per-case memory. The headline run, with five + cases and four pools, held steady at about `37` MB. Read RSS from a run that + measures the cases you care about, not from the sweep. + +## What the previous numbers said, and why they were withdrawn + +The tables that used to sit here reported the C FFI client as `2.04x`-`2.63x` +faster end-to-end and spending `2.70x`-`3.72x` less outbound CPU. Measured +properly against the current library, the matched one-shot pair is `2.04x` on +throughput and `2.64x` on outbound cost: the same ballpark, but the old `2.63x` +headline was not comparing what it claimed to. Two independent problems: + +**The two cases compared different API shapes.** `/bench/ffi` called +`client.request_uri`, the one-shot C fast path that resumes the coroutine once. +`/bench/resty-http` drove a stateful object: `new` → `connect` → `request` → +`read_body` → `set_keepalive`. When that config was written this library only +had `request_uri`; `_M.new()` now exists with the same shape as +`lua-resty-http`. The old ratio therefore contained the API-shape difference and +attributed all of it to C versus Lua. + +**They predate most of the current hot path.** Landed since those runs: +per-request token validation of the method and header names with +control-character checks on values, the case-insensitive header metatable with +repeated headers folded into arrays, `llhttp` as the default parser, interim +`1xx` consumption, and trailer parsing. Each adds per-request work. + +The harness below fixes both, and the Results section above is what it produces. + +## Topology requirement + +The measurement assumes three things that a small host cannot provide at once: a +target worker that is the only thing on its core, an upstream that cannot become +the bottleneck, and a load generator competing with neither. + +| role | CPUs | why | +| --- | --- | --- | +| target OpenResty worker | 1, exclusive | this is the thing being measured | +| upstream mock | 5 | must never be the bottleneck | +| wrk2 | 2 | must be able to saturate the target | + +That means **8 or more dedicated cores**. `run.sh` warns when fewer are online +and still runs, because the saturation gate will mark the rows invalid anyway. +Below eight cores this is a regression smoke test, not a source of headline +numbers. GitHub Actions runners (4 shared vCPUs) fall in that category. + +`run.sh` requires `taskset` and `/proc`, so it does not run on macOS. + +## The comparison is a 2x2 + +The headline ratio comes from reading one shape across both columns: + +| | one-shot | stateful | +| -------------- | --------------- | ------------------------------------------------------- | +| C FFI | `ffi.oneshot` | `ffi.stateful` | +| `resty.http` | `resty.oneshot` | `resty.stateful` | + +`ffi.oneshot` and `resty.oneshot` both call `request_uri`. `ffi.stateful` and +`resty.stateful` both drive `new`/`connect`/`request`/read/`set_keepalive`. The +gap between `ffi.oneshot` and `ffi.stateful` separately prices the C fast path +against this library's own Lua object layer, which is a different question from +the C-versus-Lua one and should not be mixed into it. + +## Case matrix + +Cases are named `.` and listed in +[`cases.txt`](cases.txt), which is the single source of truth: `run.sh` +generates one nginx location per name from it, and the smoke check diffs it +against the case table the worker actually loaded, so the two cannot drift. + +| shape | axis it prices | +| --- | --- | +| `oneshot`, `stateful` | API shape; the headline pair | +| `readbody` | buffering through the Lua chunk loop instead of the C preread fast path | +| `stream` | `res.body_reader`, chunk by chunk | +| `short` | connection setup, with no keepalive pool | +| `post` | a 4KB request body | +| `tls` | TLS with a pooled connection | +| `tlsshort` | a fresh TLS handshake per request | +| `dns` | resolving the peer by hostname instead of an IP literal | +| `chunked` | chunked response framing | +| `trailers` | chunked framing with a trailer section | +| `hdr40` | 40 response headers | +| `cookies` | ten repeated `Set-Cookie` headers, which fold into an array | +| `req30` | 30 request headers, which prices per-request header validation | + +Axes that are not cases because they are swept by environment instead: + +- **Body size** via `RESPONSE_SIZE` (the plan sweeps 0, 1KB, 16KB, 256KB, 1MB). +- **Concurrency** via `WRK_CONNECTIONS` (1, 10, 50, 100, 500). +- **Parser backend** by building the module twice, with + `NGX_HTTP_FFI_CLIENT_USE_LLHTTP=1` and `=0`, and setting `BENCH_PARSER` so the + rows are labelled. + +The full cross-product is too large to run. Sweep body size against concurrency +on the plaintext keepalive GET, then run one point per remaining axis against +that same reference configuration. + +Not every case works on every build. The hand-written parser has no chunked +framing, so `ffi.chunked` and `ffi.trailers` fail their smoke check there; they +are excluded from the run, named in the summary, and the rest proceeds. +`BENCH_STRICT=1` aborts instead. + +## Fairness audit + +An unfair configuration makes every number worthless, so the runner proves +fairness before applying any load rather than asserting it in prose. + +Both columns of a pair are generated from one shape table in +[`lua/bench.lua`](lua/bench.lua), so identical pool size, idle timeout, +timeouts, request headers and body consumption are structural rather than +reviewed. On top of that, the upstream exposes `/mock/echo`, which replies with +the raw request header block it received. The runner fetches it through both +clients and diffs: + +- **The request line and every caller-supplied header must match.** A difference + means the pair is not measuring the same request, and the audit fails. +- **Headers a client generates for itself are reported, not failed.** They are + the client's own contract. + +That second category is not empty. The FFI client emits two headers on every +request that `lua-resty-http` does not: -## Dependencies +``` +Connection: keep-alive +Content-Length: 0 +``` -- OpenResty or Nginx binary built with `ngx_http_ffi_client_module` -- `lua-resty-http` available in `TEST_NGINX_LUA_PACKAGE_PATH` -- `wrk2` -- `curl` -- `taskset` from util-linux -- Linux `/proc` filesystem for CPU sampling +`lua-resty-http` omits both on a bodyless GET; it only sets `Content-Length` for +methods that expect a body. The C path always writes `Content-Length` from +`req->body.len`. This costs the FFI client roughly 45 extra bytes per request to +format and send, so it works against the FFI side of the ratio rather than +inflating it, but it is a real difference in the request being measured and a +`Content-Length: 0` on a GET is the kind of thing a strict peer can reject. It +is worth deciding deliberately whether the C path should keep emitting it. + +Connection reuse is verified from the upstream side rather than the target's, +which keeps the check off the measured worker entirely. The upstream counts +connections and requests in a shared dict; the counters are reset after the +warmup, so they describe the steady state. **Zero new connections against a +non-zero request count is perfect pool reuse** and is what the keepalive cases +should report. A ratio near one request per connection means the pool is not +reusing anything, and the runner warns. + +Any row reporting non-2xx responses is invalid, and so is any throughput row +where the target worker averaged below `CPU_SATURATION_THRESHOLD`. Both are +excluded from every median and counted in the summary. + +## Metrics + +| metric | how | +| --- | --- | +| throughput | saturated QPS, gated on >=95% target-worker CPU | +| latency | p50/p99/p99.9 from a separate fixed-rate pass at 60% of the slower path's median saturated QPS | +| outbound cost | baseline-subtracted `1/QPS`, compared as `resty.http / C FFI` | +| worker RSS | `VmRSS` of the pinned worker at the end of the measured window | +| Lua heap per request | `collectgarbage("count")` delta over 20 requests, via `/gcdelta/` | + +The Lua heap delta is reported rather than left as a diagnostic. The library's +claim is that response handling stays in C, and the header arrays plus the +case-insensitive metatable are exactly where that could regress. + +wrk2 is run with `-L -U`, which emits both a coordinated-omission-corrected +distribution and an uncorrected one. The corrected figures are reported by +default, since correcting for delayed starts is the reason to run wrk2 at a +fixed rate at all. `BENCH_LATENCY_MODE=uncorrected` selects the other block. + +## Protocol + +Per data point: `BENCH_REPEATS` repeats (default 5), each a discarded +`WRK_WARMUP` warmup followed by a measured `WRK_DURATION` window. Cases rotate +within each repeat rather than running back to back, so drift and background +load land on every case equally. Results are reported as the median with the +min/max spread. ## Command ```bash export TEST_NGINX_BINARY=/path/to/openresty/nginx/sbin/nginx -export TEST_NGINX_LUA_PACKAGE_PATH="$PWD/lib/?.lua;$PWD/lib/?/init.lua;/path/to/lua-resty-http/lib/?.lua;;" +export TEST_NGINX_LUA_PACKAGE_PATH="$PWD/lib/?.lua;$PWD/lib/?/init.lua;/path/to/lua-resty-http/lib/?.lua" +export WRK2_BIN=/path/to/wrk2/wrk + +# the headline 2x2 make bench + +# the whole matrix +BENCH_CASES=all make bench + +# one axis point +BENCH_CASES="baseline ffi.stream resty.stream" RESPONSE_SIZE=262144 make bench ``` -## Topology +The module has to be compiled into the nginx binary: -- upstream instance: mock HTTP server, multiple workers, non-target CPU cores -- target instance: one OpenResty worker, pinned to `TARGET_CPU` -- wrk2: load generator, pinned to `WRK_CPUS` +```bash +NGX_HTTP_FFI_CLIENT_USE_LLHTTP=1 \ +NGX_HTTP_LUA_MODULE_DIR=/path/to/openresty/bundle/ngx_lua-VERSION \ + ./configure --prefix=/tmp/openresty-llhttp --with-http_ssl_module \ + --with-pcre-jit --add-module=/path/to/ngx_http_ffi_client +``` + +## Results + +Every run writes one CSV row per wrk2 invocation to +`benchmark/.run/results/.csv`, with the run parameters on each row so +rows from different runs can be concatenated: + +``` +run_id,phase,repeat,case,qps,rate,non2xx,socket_errors,p50_ms,p99_ms,p999_ms, +cpu_avg,cpu_max,cpu_samples,saturated,rss_kb,upstream_connections, +upstream_requests,requests_per_connection,response_size,connections,threads, +duration,parser +``` -`wrk2` sends requests to the target instance. The target instance sends one -outbound HTTP/1.1 request to the upstream per inbound request. +[`fold.sh`](fold.sh) folds those rows into the markdown tables this file +carries. `run.sh` calls it at the end of a run; it also runs standalone: -The benchmark cases run in this order: +```bash +benchmark/fold.sh benchmark/.run/results/.csv +``` + +The tables it emits are the headline ratio per shape, a per-case detail table +with medians and spread, the latency table, and a list of everything discarded. +Paste them in rather than hand-maintaining them, which is what the previous +tables were. + +## Dependencies + +- OpenResty or Nginx built with `ngx_http_ffi_client_module` +- `lua-resty-http` on `TEST_NGINX_LUA_PACKAGE_PATH` +- `wrk2` +- `curl`, `openssl`, `taskset` from util-linux +- Linux `/proc` for CPU and RSS sampling + +## Topology detail + +- upstream instance: mock HTTP server on multiple workers, non-target cores, + plus a TLS listener and a raw TCP `stream` mock for the trailer case, since + trailers cannot be emitted from the http subsystem +- target instance: one OpenResty worker, pinned to `TARGET_CPU` +- wrk2: load generator, pinned to `WRK_CPUS` -1. `/bench/baseline`: target worker returns a fixed body without upstream IO. -2. `/bench/ffi`: target worker calls `resty.ngx_http_ffi_client`. -3. `/bench/resty-http`: target worker calls `resty.http`. +Both instances raise `keepalive_requests` far above the default. At tens of +thousands of requests per second the stock limit would recycle connections +hundreds of times per run: on the downstream leg that charges accept cost to +whichever case is running, and on the upstream leg it would force the client +pools to reconnect and stop the keepalive cases from measuring keepalive. ## Configuration | Variable | Default | Meaning | | --- | --- | --- | +| `BENCH_CASES` | the 2x2 plus baseline | space-separated case names, or `all` | +| `BENCH_REPEATS` | `5` | repeats per data point | +| `BENCH_LATENCY` | `1` | run the fixed-rate latency phase | +| `BENCH_LATENCY_FRACTION` | `0.6` | fraction of the slower path's QPS for that phase | +| `BENCH_LATENCY_MODE` | `corrected` | `corrected` or `uncorrected` percentiles | +| `BENCH_GC` | `1` | report the per-request Lua heap delta | +| `BENCH_AUDIT` | `1` | run the fairness audit before load | +| `BENCH_STRICT` | `0` | abort instead of excluding a case that fails its smoke check | +| `BENCH_PARSER` | `llhttp` | label recorded in the CSV rows | +| `BENCH_RESULTS_DIR` | `.run/results` | where CSV rows are written | | `TARGET_CPU` | `0` | CPU core used by the target OpenResty worker | | `UPSTREAM_CPUS` | non-target cores | CPU cores used by the upstream instance | | `WRK_CPUS` | `UPSTREAM_CPUS` | CPU cores used by wrk2 | | `TARGET_PORT` | `19840` | target listen port | | `UPSTREAM_PORT` | `19841` | upstream listen port | +| `UPSTREAM_TLS_PORT` | `19842` | upstream TLS listen port | +| `UPSTREAM_TRAILER_PORT` | `19843` | raw TCP trailer mock port | | `UPSTREAM_WORKERS` | count of `UPSTREAM_CPUS` | upstream worker count | +| `DNS_HOST` | `localhost` | hostname used by the `dns` cases | +| `BENCH_RESOLVER` | first `/etc/resolv.conf` nameserver | resolver for those cases | +| `TLS_SERVER_NAME` | `bench.local` | SNI both clients send for the TLS cases | | `RESPONSE_SIZE` | `1024` | upstream response body size in bytes | | `KEEPALIVE_POOL_SIZE` | `120` | keepalive pool size for both client paths | | `KEEPALIVE_IDLE_TIMEOUT` | `60000` | keepalive idle timeout in milliseconds | +| `CLIENT_TIMEOUT` | `60000` | connect/send/read timeout for both clients | | `WRK_THREADS` | `2` | wrk2 thread count | | `WRK_CONNECTIONS` | `100` | wrk2 connection count | -| `WRK_DURATION` | `30s` | official benchmark duration | -| `WRK_RATE` | `100000` | wrk2 target request rate, set high enough to saturate the target worker | -| `WRK2_BIN` | `wrk` | wrk2 executable path | -| `CPU_SATURATION_THRESHOLD` | `95` | minimum average target worker CPU for a valid run | +| `WRK_WARMUP` | `10s` | discarded warmup before each measured window | +| `WRK_DURATION` | `30s` | measured window | +| `WRK_RATE` | `100000` | wrk2 target rate, set above the expected max QPS | +| `CPU_SATURATION_THRESHOLD` | `95` | minimum average target worker CPU for a valid throughput row | -The runner invokes wrk2 with `-R "$WRK_RATE" -L -U`. The `-L` output reports -wrk2's latency distribution and `-U` reports the uncorrected latency -distribution. For this benchmark, set `WRK_RATE` above the expected maximum QPS -so the single target worker remains CPU-bound. +## Baseline-derived estimate -## Baseline-Derived Estimate - -The QPS results include downstream client-to-OpenResty overhead. To estimate the -outbound HTTP client cost, the runner treats single-core CPU time per request as -approximately `1 / QPS`: +QPS includes the downstream leg. To estimate outbound client cost, the runner +treats single-core CPU time per request as approximately `1 / QPS`: ```text baseline_us = 1_000_000 / baseline_qps @@ -141,44 +419,18 @@ resty_outbound_us = 1_000_000 / resty_http_qps - baseline_us ratio = resty_outbound_us / ffi_outbound_us ``` -This is still an estimate, but it removes the common downstream and simple -response path from the comparison. - -## Valid Runs - -A result is valid only when the target worker CPU average is at least -`CPU_SATURATION_THRESHOLD`. - -If the result is marked suspicious: - -- increase `WRK_THREADS` -- increase `WRK_CONNECTIONS` -- increase `WRK_RATE` -- move wrk2 to non-target CPU cores -- increase `UPSTREAM_WORKERS` -- assign more non-target cores to `UPSTREAM_CPUS` +Every case, the baseline included, writes the same fixed downstream body, so +that subtraction removes an identical cost from each. It is still an estimate. -Do not compare C FFI and `resty.http` QPS from runs where the target worker was -not saturated. - -## Manual RSS and GC Checks - -The benchmark output focuses on QPS, latency, and target worker CPU saturation. -Use these commands for supporting observations: - -```bash -cat benchmark/.run/target/logs/nginx.pid -ps -o pid,ppid,pcpu,rss,comm --ppid "$(cat benchmark/.run/target/logs/nginx.pid)" -curl -fsS "http://127.0.0.1:${TARGET_PORT:-19840}/gc" -``` +## If a run is marked unsaturated -The GC endpoint prints `collectgarbage("count")` in KB for the target worker. -Treat RSS and GC values as diagnostics, not as the primary comparison metric. +Do not compare QPS across cases from runs where the target worker was not the +bottleneck. Increase `WRK_THREADS`, `WRK_CONNECTIONS` or `WRK_RATE`; move wrk2 +to non-target cores; or give the upstream more workers and cores. -## Dry Run +## Dry run -Use dry-run mode to verify template rendering without starting OpenResty or -`wrk2`: +Renders the configs and generates the locations without starting anything: ```bash BENCH_DRY_RUN=1 make bench diff --git a/benchmark/cases.txt b/benchmark/cases.txt new file mode 100644 index 0000000..44e09da --- /dev/null +++ b/benchmark/cases.txt @@ -0,0 +1,59 @@ +# Canonical bench case list. One name per line, "#" starts a comment. +# +# This is the single source of truth: run.sh generates one target location per +# name from this file, and the smoke check diffs it against the case table in +# benchmark/lua/bench.lua so the two cannot drift apart. +# +# Names are .. A headline comparison is one shape read across the +# ffi.* and resty.* columns; the ffi.oneshot-vs-ffi.stateful gap prices the C +# fast path against this library's own Lua object layer. + +# --- the 2x2 --- +ffi.oneshot +resty.oneshot +ffi.stateful +resty.stateful + +# --- read mode --- +ffi.readbody +resty.readbody +ffi.stream +resty.stream + +# --- connection lifetime --- +ffi.short +resty.short + +# --- method --- +ffi.post +resty.post + +# --- transport --- +ffi.tls +resty.tls +ffi.tlsshort +resty.tlsshort +ffi.tlsverify +resty.tlsverify +ffi.tlsverifyshort +resty.tlsverifyshort + +# --- peer address --- +ffi.dns +resty.dns + +# --- response framing --- +ffi.chunked +resty.chunked +ffi.trailers +resty.trailers + +# --- response header count --- +ffi.hdr40 +resty.hdr40 +ffi.cookies +resty.cookies + +# --- request header count --- +ffi.req30 +resty.req30 diff --git a/benchmark/conf/target.nginx.conf b/benchmark/conf/target.nginx.conf index 6e2100c..a39984b 100644 --- a/benchmark/conf/target.nginx.conf +++ b/benchmark/conf/target.nginx.conf @@ -1,7 +1,7 @@ worker_processes 1; daemon on; master_process on; -error_log logs/error.log info; +error_log logs/error.log error; pid logs/nginx.pid; events { @@ -12,12 +12,40 @@ http { lua_package_path "{{LUA_PACKAGE_PATH}};;"; access_log off; + # The downstream leg is common to every case including the baseline, so it + # must not introduce churn of its own: at tens of thousands of requests per + # second the stock keepalive_requests would recycle wrk's connections + # hundreds of times per run and charge the accept path to whichever case was + # running. + keepalive_requests 1000000; + keepalive_timeout 300s; + + resolver {{RESOLVER}} ipv6=off; + resolver_timeout 5s; + + # resty.http verifies against the cosocket trust store, which is per-worker + # config rather than per-request. The FFI client is handed the same file as + # ssl_trusted_certificate, so the verify-on cases ask both clients to check + # the identical chain against the identical root. + lua_ssl_trusted_certificate {{CA_FILE}}; + lua_ssl_verify_depth 5; + init_by_lua_block { - benchmark_upstream_host = "127.0.0.1" - benchmark_upstream_port = {{UPSTREAM_PORT}} - benchmark_keepalive_pool_size = {{KEEPALIVE_POOL_SIZE}} - benchmark_keepalive_idle_timeout = {{KEEPALIVE_IDLE_TIMEOUT}} - benchmark_response_body = string.rep("x", {{RESPONSE_SIZE}}) + local bench = require "bench" + + bench.init({ + host = "{{UPSTREAM_HOST}}", + port = {{UPSTREAM_PORT}}, + tls_port = {{UPSTREAM_TLS_PORT}}, + trailer_port = {{UPSTREAM_TRAILER_PORT}}, + dns_host = "{{DNS_HOST}}", + tls_server_name = "{{TLS_SERVER_NAME}}", + ca_file = "{{CA_FILE}}", + pool_size = {{KEEPALIVE_POOL_SIZE}}, + idle_timeout = {{KEEPALIVE_IDLE_TIMEOUT}}, + timeout = {{CLIENT_TIMEOUT}}, + response_body = string.rep("x", {{RESPONSE_SIZE}}), + }) } server { @@ -27,6 +55,7 @@ http { content_by_lua_block { local ok_ffi, ffi_err = pcall(require, "resty.ngx_http_ffi_client") local ok_http, http_err = pcall(require, "resty.http") + local ok_bench, bench_err = pcall(require, "bench") if not ok_ffi then ngx.status = 500 @@ -40,183 +69,75 @@ http { return end + if not ok_bench then + ngx.status = 500 + ngx.say("bench: " .. tostring(bench_err)) + return + end + ngx.say("ok") } } - location = /gc { + # The case table as the worker actually sees it. The runner diffs this + # against benchmark/cases.txt, so a case added to one and not the other + # fails the smoke check instead of silently going unmeasured. + location = /cases { content_by_lua_block { - ngx.say(collectgarbage("count")) + local bench = require "bench" + ngx.say(table.concat(bench.case_names(), "\n")) } } - location = /bench/baseline { + location = /gc { content_by_lua_block { - local body = benchmark_response_body - ngx.header["Content-Length"] = #body - ngx.print(body) + ngx.say(collectgarbage("count")) } } - location = /smoke/ffi-short { + # Per-request Lua heap delta: allocate-free the case once, then measure + # the next call against a settled heap. Diagnostic on its own, but the + # library's claim is that response handling stays in C, and the header + # arrays and the case-insensitive metatable are exactly where that could + # regress, so the runner records it as a reported metric. + location ~ ^/gcdelta/(?[a-z0-9._-]+)$ { content_by_lua_block { - local client = require "resty.ngx_http_ffi_client" - local res, err = client.request_uri({ - scheme = "http", - host = benchmark_upstream_host, - port = benchmark_upstream_port, - method = "GET", - path = "/mock", - headers = { - ["X-Benchmark-Case"] = "ffi-short", - }, - }) - - if not res then - ngx.status = 502 - ngx.say(err) - return - end + local bench = require "bench" + local name = ngx.var.bench_case - ngx.status = res.status - ngx.print(res.body) - } - } + bench.audit(name) - location = /smoke/resty-http-short { - content_by_lua_block { - local http = require "resty.http" - local httpc, err = http.new() - if not httpc then - ngx.status = 502 - ngx.say(err) - return - end + collectgarbage("collect") + collectgarbage("collect") - httpc:set_timeout(60000) + local before = collectgarbage("count") + local n = 20 - local ok - ok, err = httpc:connect(benchmark_upstream_host, benchmark_upstream_port) - if not ok then - ngx.status = 502 - ngx.say(err) - return + for _ = 1, n do + bench.audit(name) end - local res - res, err = httpc:request({ - method = "GET", - path = "/mock", - headers = { - ["Host"] = benchmark_upstream_host, - ["X-Benchmark-Case"] = "resty-http-short", - }, - }) - if not res then - ngx.status = 502 - ngx.say(err) - return - end + local after = collectgarbage("count") - local body - body, err = res:read_body() - httpc:close() - if not body then - ngx.status = 502 - ngx.say(err) - return - end - - ngx.status = res.status - ngx.print(body) + ngx.say(string.format("%.3f", (after - before) / n)) } } - location = /bench/ffi { + location ~ ^/audit/(?[a-z0-9._-]+)$ { content_by_lua_block { - local client = require "resty.ngx_http_ffi_client" - local res, err = client.request_uri({ - scheme = "http", - host = benchmark_upstream_host, - port = benchmark_upstream_port, - method = "GET", - path = "/mock", - headers = { - ["X-Benchmark-Case"] = "ffi-keepalive", - }, - keepalive = { - pool = "benchmark-ffi", - idle_timeout = benchmark_keepalive_idle_timeout, - pool_size = benchmark_keepalive_pool_size, - }, - }) - - if not res then - ngx.status = 502 - ngx.say(err) - return - end - - ngx.status = res.status - ngx.header["X-Benchmark-Reused"] = tostring(res.reused) - ngx.print(res.body) + require("bench").audit(ngx.var.bench_case) } } - location = /bench/resty-http { + location = /bench/baseline { content_by_lua_block { - local http = require "resty.http" - local httpc, err = http.new() - if not httpc then - ngx.status = 502 - ngx.say(err) - return - end - - httpc:set_timeout(60000) - - local ok - ok, err = httpc:connect(benchmark_upstream_host, benchmark_upstream_port) - if not ok then - ngx.status = 502 - ngx.say(err) - return - end - - local res - res, err = httpc:request({ - method = "GET", - path = "/mock", - headers = { - ["Host"] = benchmark_upstream_host, - ["X-Benchmark-Case"] = "resty-http-keepalive", - }, - }) - if not res then - ngx.status = 502 - ngx.say(err) - return - end - - local body - body, err = res:read_body() - if not body then - ngx.status = 502 - ngx.say(err) - return - end - - local ok_keepalive, keepalive_err = - httpc:set_keepalive(benchmark_keepalive_idle_timeout, - benchmark_keepalive_pool_size) - if not ok_keepalive then - ngx.log(ngx.ERR, "resty.http set_keepalive failed: ", - keepalive_err) - end - - ngx.status = res.status - ngx.print(body) + require("bench").baseline() } } + + # One exact location per case, generated from benchmark/cases.txt. They + # are exact rather than a single regex location on purpose: a regex match + # would land on the measured worker's hot path in every case. + include bench_locations.conf; } } diff --git a/benchmark/conf/upstream.nginx.conf b/benchmark/conf/upstream.nginx.conf index c1fb60c..98edf76 100644 --- a/benchmark/conf/upstream.nginx.conf +++ b/benchmark/conf/upstream.nginx.conf @@ -1,25 +1,78 @@ worker_processes {{UPSTREAM_WORKERS}}; daemon on; master_process on; -error_log logs/error.log info; +error_log logs/error.log error; pid logs/nginx.pid; events { worker_connections 65535; } +# The mock peer. It is deliberately over-provisioned relative to the single +# pinned target worker: any measurement where this side is the bottleneck is +# measuring this file rather than the client under test. http { access_log off; + # A client keepalive pool is only worth measuring if the peer keeps its end + # of the connection open. At the stock keepalive_requests the upstream would + # close every connection after 1000 requests and the client pools would + # spend the run reconnecting. + keepalive_requests 1000000; + keepalive_timeout 300s; + + lua_shared_dict bench_stats 1m; + init_by_lua_block { benchmark_response_body = string.rep("x", {{RESPONSE_SIZE}}) + + -- 40 response headers, for the response-header-count axis + benchmark_headers40 = {} + for i = 1, 40 do + benchmark_headers40[i] = { string.format("X-H-%02d", i), + string.rep("h", 24) } + end + + -- a Set-Cookie-heavy response: one name arriving ten times, which is + -- the shape that makes the client fold repeated headers into an array + benchmark_cookies = {} + for i = 1, 10 do + benchmark_cookies[i] = string.format( + "bench%02d=%s; Path=/; HttpOnly", i, string.rep("c", 24)) + end + } + + # Connection and request counters. The runner reads these before and after + # each case: requests-per-connection at or near 1 means the client pool is + # not reusing anything, which invalidates every keepalive case. Counting + # here rather than on the target keeps the check off the measured worker's + # hot path entirely. + init_worker_by_lua_block { + local stats = ngx.shared.bench_stats + + function benchmark_count() + stats:incr("requests", 1, 0) + + if ngx.var.connection_requests == "1" then + stats:incr("connections", 1, 0) + end + end } server { listen 127.0.0.1:{{UPSTREAM_PORT}}; + listen 127.0.0.1:{{UPSTREAM_TLS_PORT}} ssl; + + ssl_certificate tls.crt; + ssl_certificate_key tls.key; + ssl_session_cache shared:bench_ssl:10m; + ssl_session_timeout 10m; + # Content-Length framed, the reference response shape. location = /mock { content_by_lua_block { + benchmark_count() + local body = benchmark_response_body ngx.header["Content-Length"] = #body ngx.header["X-Benchmark-Upstream"] = "mock" @@ -27,8 +80,138 @@ http { } } + # Same response, but reads a request body first: the POST axis. + location = /mock/sink { + content_by_lua_block { + benchmark_count() + + ngx.req.read_body() + + local body = benchmark_response_body + ngx.header["Content-Length"] = #body + ngx.header["X-Benchmark-Upstream"] = "sink" + ngx.print(body) + } + } + + # No Content-Length, so nginx frames the body with chunked encoding. + location = /mock/chunked { + content_by_lua_block { + benchmark_count() + + ngx.header["X-Benchmark-Upstream"] = "chunked" + ngx.print(benchmark_response_body) + } + } + + location = /mock/headers40 { + content_by_lua_block { + benchmark_count() + + local h = ngx.header + for i = 1, #benchmark_headers40 do + local pair = benchmark_headers40[i] + h[pair[1]] = pair[2] + end + + local body = benchmark_response_body + h["Content-Length"] = #body + ngx.print(body) + } + } + + location = /mock/cookies { + content_by_lua_block { + benchmark_count() + + ngx.header["Set-Cookie"] = benchmark_cookies + + local body = benchmark_response_body + ngx.header["Content-Length"] = #body + ngx.print(body) + } + } + + # Echoes back the raw request header block exactly as received. The + # runner's fairness audit diffs the two columns of a pair against this: + # if the bytes differ, the pair is not measuring the same request. + location = /mock/echo { + content_by_lua_block { + local raw = ngx.req.raw_header() + ngx.header["Content-Length"] = #raw + ngx.header["Content-Type"] = "text/plain" + ngx.print(raw) + } + } + + location = /stats { + content_by_lua_block { + local stats = ngx.shared.bench_stats + ngx.say("connections ", stats:get("connections") or 0) + ngx.say("requests ", stats:get("requests") or 0) + } + } + + location = /stats/reset { + content_by_lua_block { + local stats = ngx.shared.bench_stats + stats:set("connections", 0) + stats:set("requests", 0) + ngx.say("ok") + } + } + location = /ready { return 200 "ok\n"; } } } + +# Trailers cannot be emitted from the http subsystem, so the trailer case gets a +# raw TCP mock that writes exact response bytes: chunked framing, then a trailer +# section behind the terminating chunk. It answers on a keepalive loop so the +# client pool behaves the same as it does against the http mock. +stream { + init_by_lua_block { + local body = string.rep("x", {{RESPONSE_SIZE}}) + + benchmark_trailer_response = table.concat({ + "HTTP/1.1 200 OK\r\n", + "X-Benchmark-Upstream: trailers\r\n", + "Trailer: X-Checksum\r\n", + "Transfer-Encoding: chunked\r\n", + "\r\n", + string.format("%x\r\n", #body), body, "\r\n", + "0\r\n", + "X-Checksum: abc123\r\n", + "\r\n", + }) + } + + server { + listen 127.0.0.1:{{UPSTREAM_TRAILER_PORT}}; + + content_by_lua_block { + local sock = ngx.req.socket() + sock:settimeout(300000) + + local resp = benchmark_trailer_response + -- one receive per request rather than one per header line: this mock + -- has to keep up with a saturated target worker + local read_head = sock:receiveuntil("\r\n\r\n") + + while true do + local head, err = read_head() + if not head then + return + end + + local ok + ok, err = sock:send(resp) + if not ok then + return + end + end + } + } +} diff --git a/benchmark/fold.sh b/benchmark/fold.sh new file mode 100755 index 0000000..5a40c3c --- /dev/null +++ b/benchmark/fold.sh @@ -0,0 +1,225 @@ +#!/usr/bin/env bash +# +# Fold raw benchmark rows into the markdown tables the READMEs carry. +# +# benchmark/fold.sh benchmark/.run/results/.csv +# +# Reads the CSV that run.sh emits, one row per wrk2 run, and prints the summary +# tables. Rows that failed the saturation gate or reported non-2xx responses are +# excluded from every median and counted separately, because a run where the +# target worker was not the bottleneck is not measuring the client. + +set -euo pipefail + +CSV="${1:-}" + +if [[ -z "$CSV" ]]; then + latest="$(ls -1t "$(dirname "${BASH_SOURCE[0]}")/.run/results"/*.csv \ + 2>/dev/null | head -n 1 || true)" + CSV="$latest" +fi + +[[ -n "$CSV" && -f "$CSV" ]] || { + printf 'usage: %s \n' "${BASH_SOURCE[0]}" >&2 + exit 1 +} + +awk -F, ' +function median(arr, n, i, j, t) { + if (n == 0) return "" + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (arr[j] < arr[i]) { t = arr[i]; arr[i] = arr[j]; arr[j] = t } + if (n % 2) return arr[(n - 1) / 2] + return (arr[n / 2 - 1] + arr[n / 2]) / 2 +} + +function collect(key, value, n) { + n = count[key]++ + vals[key SUBSEP n] = value +} + +function med_of(key, i, n, a) { + n = count[key] + if (n == 0) return "" + for (i = 0; i < n; i++) a[i] = vals[key SUBSEP i] + return median(a, n) +} + +function min_of(key, i, n, m) { + n = count[key] + if (n == 0) return "" + m = vals[key SUBSEP 0] + for (i = 1; i < n; i++) if (vals[key SUBSEP i] < m) m = vals[key SUBSEP i] + return m +} + +function max_of(key, i, n, m) { + n = count[key] + if (n == 0) return "" + m = vals[key SUBSEP 0] + for (i = 1; i < n; i++) if (vals[key SUBSEP i] > m) m = vals[key SUBSEP i] + return m +} + +NR == 1 { next } + +{ + phase = $2; name = $4; qps = $5 + 0; non2xx = $7 + 0 + cpu = $12 + 0; sat = $15; rss = $16 + 0; rpc = $19 + p50 = $9; p99 = $10; p999 = $11; rate = $6 + parser = $24; conns = $21; rsize = $20; dur = $23 + + meta_parser = parser; meta_conns = conns; meta_rsize = rsize; meta_dur = dur + + if (phase == "throughput") { + seen_tp[name] = 1 + total[name]++ + + if (sat == "yes" && non2xx == 0) { + collect("qps:" name, qps) + collect("cpu:" name, cpu) + collect("rss:" name, rss) + # "inf" is not a number but it is the best possible reuse result: + # no new connections at all during the measured window + if (rpc == "inf") reuse_inf[name]++ + else if (rpc != "") collect("rpc:" name, rpc + 0) + } else { + discarded[name]++ + } + } + + if (phase == "latency") { + seen_lat[name] = 1 + lat_rate[name] = rate + if (non2xx == 0) { + if (p50 != "") collect("p50:" name, p50 + 0) + if (p99 != "") collect("p99:" name, p99 + 0) + if (p999 != "") collect("p999:" name, p999 + 0) + } + } +} + +END { + base = med_of("qps:baseline") + + printf "\n### Run parameters\n\n" + printf "parser `%s`, response size `%s` bytes, wrk2 connections `%s`, ", + meta_parser, meta_rsize, meta_conns + printf "duration `%s` per repeat\n", meta_dur + + # --- headline: one row per shape --- + printf "\n### Headline (median QPS per shape)\n\n" + printf "| shape | C FFI | `resty.http` | C FFI / `resty.http` |" + if (base != "") printf " outbound cost ratio |" + printf "\n| --- | ---: | ---: | ---: |" + if (base != "") printf " ---: |" + printf "\n" + + for (name in seen_tp) { + if (name == "baseline") continue + # not split(name, parts, ".") -- awk reads the separator as a regex, so + # "." matches every character and every field comes back empty + if (substr(name, 1, 4) != "ffi.") continue + shape = substr(name, 5) + + f = med_of("qps:ffi." shape) + r = med_of("qps:resty." shape) + if (f == "" || r == "") continue + + printf "| `%s` | `%.2f` | `%.2f` | `%.2fx` |", shape, f, r, f / r + + if (base != "" && f > 0 && r > 0) { + bus = 1000000 / base + fus = 1000000 / f - bus + rus = 1000000 / r - bus + if (fus > 0 && rus > 0) printf " `%.2fx` |", rus / fus + else printf " n/a |" + } else if (base != "") { + printf " n/a |" + } + printf "\n" + } + + if (base != "") { + printf "\nBaseline (no upstream call): `%.2f` QPS, `%.3f` us/request.\n", + base, 1000000 / base + printf "The outbound cost ratio subtracts that baseline from both ", + "" + printf "client paths and compares what is left.\n" + } + + # --- detail --- + printf "\n### Detail (throughput phase)\n\n" + printf "| case | median QPS | min | max | repeats used | discarded |" + printf " worker CPU avg | worker RSS | requests/connection |\n" + printf "| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n" + + n = 0 + for (name in seen_tp) ordered[n++] = name + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (ordered[j] < ordered[i]) { t = ordered[i]; ordered[i] = ordered[j]; ordered[j] = t } + + for (i = 0; i < n; i++) { + name = ordered[i] + m = med_of("qps:" name) + used = count["qps:" name] + drop = discarded[name] + 0 + rssv = med_of("rss:" name) + rpcv = med_of("rpc:" name) + + printf "| `%s` |", name + if (m == "") printf " n/a | n/a | n/a |" + else printf " `%.2f` | `%.2f` | `%.2f` |", m, min_of("qps:" name), max_of("qps:" name) + printf " %d | %d |", used, drop + if (used > 0) printf " `%.1f%%` | `%d` KB |", med_of("cpu:" name), rssv + else printf " n/a | n/a |" + if (rpcv != "") printf " `%.1f` |", rpcv + else if (reuse_inf[name] > 0) printf " no new connections |" + else printf " n/a |" + printf "\n" + } + + # --- latency --- + haslat = 0 + for (name in seen_lat) haslat = 1 + + if (haslat) { + printf "\n### Latency (fixed rate, corrected for coordinated omission)\n\n" + printf "| case | rate | p50 | p99 | p99.9 |\n" + printf "| --- | ---: | ---: | ---: | ---: |\n" + + n = 0 + delete ordered + for (name in seen_lat) ordered[n++] = name + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (ordered[j] < ordered[i]) { t = ordered[i]; ordered[i] = ordered[j]; ordered[j] = t } + + for (i = 0; i < n; i++) { + name = ordered[i] + printf "| `%s` | `%s` req/s |", name, lat_rate[name] + v = med_of("p50:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + v = med_of("p99:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + v = med_of("p999:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + printf "\n" + } + } + + # --- what got thrown away --- + anydrop = 0 + for (name in discarded) if (discarded[name] > 0) anydrop = 1 + + if (anydrop) { + printf "\n### Discarded rows\n\n" + printf "Rows failing the saturation gate or reporting non-2xx responses," + printf " excluded from every median above:\n\n" + for (i = 0; i < n; i++) { } + for (name in discarded) + if (discarded[name] > 0) + printf "- `%s`: %d of %d\n", name, discarded[name], total[name] + } + printf "\n" +} +' "$CSV" diff --git a/benchmark/lua/bench.lua b/benchmark/lua/bench.lua new file mode 100644 index 0000000..17a4b5d --- /dev/null +++ b/benchmark/lua/bench.lua @@ -0,0 +1,555 @@ +-- Bench case drivers. +-- +-- Every case is a driver plus a shape, never a hand-written handler. That is +-- deliberate: the old harness hand-wrote one location per client and the two +-- drifted into different API shapes, so the published ratio credited the +-- API-shape difference to the C-vs-Lua difference. Here both columns of a pair +-- are built from one shape table, so they send the same headers, carry the same +-- timeouts, size the same pool, and consume the body the same way by +-- construction rather than by review. +-- +-- A driver returns (bytes_read, status) or (nil, nil, err). The dispatcher +-- writes the same fixed-size downstream body for every case, the baseline +-- included, so the downstream write cost cancels out of the baseline +-- subtraction and only the outbound client work differs between cases. + +local ffi_client = require "resty.ngx_http_ffi_client" +local resty_http = require "resty.http" + +local ngx = ngx +local ngx_log = ngx.log +local ngx_ERR = ngx.ERR +local pairs = pairs +local tostring = tostring +local table_sort = table.sort +local str_format = string.format +local str_rep = string.rep + + +local _M = {} + + +local cfg = { + host = "127.0.0.1", + port = 80, + tls_port = 443, + trailer_port = 8443, + dns_host = "localhost", + tls_server_name = "bench.local", + -- the upstream's self-signed cert, trusted by both clients for the + -- verify-on cases. resty.http has no per-request trust store, so its side + -- comes from lua_ssl_trusted_certificate in the target conf; this is the + -- same file, so both are asked to verify the identical chain. + ca_file = "", + pool_size = 120, + idle_timeout = 60000, + timeout = 60000, + response_body = "", +} + + +-- Request header sets. Both columns of a pair always send the identical table, +-- so the 3-vs-30 axis prices per-request header validation rather than a +-- difference in what the two clients were asked to send. The tables are built +-- once: constructing them per request would measure Lua table construction. +local HEADERS_3 = { + ["X-Bench-A"] = "aaaaaaaa", + ["X-Bench-B"] = "bbbbbbbb", + ["X-Bench-C"] = "cccccccc", +} + +local HEADERS_30 = {} +do + for i = 1, 30 do + HEADERS_30[str_format("X-Bench-%02d", i)] = str_rep("v", 16) + end +end + +local POST_BODY = str_rep("p", 4096) + + +function _M.init(opts) + for k, v in pairs(opts) do + cfg[k] = v + end +end + + +-- ===== shape ===== +-- +-- scheme "http" or "https" +-- dns resolve the peer by hostname instead of using the IP literal +-- trailers talk to the raw TCP mock, the only peer here that sends trailers +-- path upstream path, which selects the response framing and header count +-- method defaults to GET +-- body request body, for the POST axis +-- headers request header table, for the request-header-count axis +-- keepalive false for the short-lived-connection axis +-- stream drain res.body_reader instead of buffering the body +-- read_body buffer the body, but through the Lua chunk loop rather than the +-- C preread fast path (FFI only; resty.http has no other mode) +-- echo return the body itself instead of its length, for the audit +-- pool keepalive pool name, assigned per case by pair_case + +local function peer(s) + local scheme = s.scheme or "http" + local https = scheme == "https" + local port + + if https then + port = cfg.tls_port + elseif s.trailers then + port = cfg.trailer_port + else + port = cfg.port + end + + return scheme, + s.dns and cfg.dns_host or cfg.host, + port, + s.path or "/mock", + s.method or "GET", + s.headers or HEADERS_3, + s.keepalive ~= false, + -- both clients send the same SNI for the TLS cases; left to their own + -- devices the FFI client defaults it to the host and resty.http omits + -- it, which would not be the same handshake + https and cfg.tls_server_name or nil, + -- verification is off unless a shape asks for it, so the plain tls + -- cases keep pricing the handshake rather than the trust store + (https and s.verify) and true or false +end + + +-- ===== FFI client ===== + +local function ffi_oneshot(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local res, err = ffi_client.request_uri({ + scheme = scheme, + host = host, + port = port, + method = method, + path = path, + headers = headers, + body = s.body, + timeout = cfg.timeout, + ssl_verify = verify, + ssl_trusted_certificate = verify and cfg.ca_file or nil, + ssl_server_name = sni, + keepalive = keepalive and { + pool = s.pool, + pool_size = cfg.pool_size, + idle_timeout = cfg.idle_timeout, + } or nil, + }) + + if not res then + return nil, nil, err + end + + if s.echo then + return res.body, res.status + end + + return #res.body, res.status +end + + +local function ffi_stateful(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = ffi_client.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local ok + ok, err = httpc:connect({ + scheme = scheme, + host = host, + port = port, + pool = s.pool, + pool_size = cfg.pool_size, + ssl_verify = verify, + ssl_trusted_certificate = verify and cfg.ca_file or nil, + ssl_server_name = sni, + }) + if not ok then + return nil, nil, err + end + + local res + res, err = httpc:request({ + method = method, + path = path, + headers = headers, + body = s.body, + -- the buffered fast path: C reads the whole body and resumes once. + -- read_body sends the same request but takes the Lua chunk loop back. + preread_body = not (s.stream or s.read_body), + }) + if not res then + httpc:close() + return nil, nil, err + end + + local result + + if s.stream then + local n = 0 + local reader = res.body_reader + + while true do + local chunk, rerr = reader() + if rerr then + httpc:close() + return nil, nil, rerr + end + if not chunk then + break + end + n = n + #chunk + end + + result = n + + else + local body + body, err = res:read_body() + if not body then + httpc:close() + return nil, nil, err + end + + result = s.echo and body or #body + end + + if keepalive then + ok, err = httpc:set_keepalive(cfg.idle_timeout, cfg.pool_size) + if not ok then + ngx_log(ngx_ERR, "ffi set_keepalive failed: ", tostring(err)) + end + + else + httpc:close() + end + + return result, res.status +end + + +-- ===== lua-resty-http ===== + +local function resty_oneshot(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = resty_http.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local res + res, err = httpc:request_uri( + str_format("%s://%s:%d%s", scheme, host, port, path), { + method = method, + headers = headers, + body = s.body, + -- the FFI client sends no User-Agent, so neither does this one: + -- otherwise resty.http pays for ~55 extra bytes on every request + use_default_user_agent = false, + ssl_verify = verify, + ssl_server_name = sni, + pool = s.pool, + pool_size = cfg.pool_size, + keepalive = keepalive, + keepalive_timeout = cfg.idle_timeout, + keepalive_pool = cfg.pool_size, + }) + + if not res then + return nil, nil, err + end + + if s.echo then + return res.body, res.status + end + + return #res.body, res.status +end + + +local function resty_stateful(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = resty_http.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local ok + ok, err = httpc:connect({ + scheme = scheme, + host = host, + port = port, + pool = s.pool, + pool_size = cfg.pool_size, + ssl_verify = verify, + ssl_server_name = sni, + }) + if not ok then + return nil, nil, err + end + + local res + res, err = httpc:request({ + method = method, + path = path, + headers = headers, + body = s.body, + use_default_user_agent = false, + }) + if not res then + httpc:close() + return nil, nil, err + end + + local result + + if s.stream then + local n = 0 + local reader = res.body_reader + + while true do + local chunk, rerr = reader() + if rerr then + httpc:close() + return nil, nil, rerr + end + if not chunk then + break + end + n = n + #chunk + end + + result = n + + else + local body + body, err = res:read_body() + if not body then + httpc:close() + return nil, nil, err + end + + result = s.echo and body or #body + end + + if keepalive then + ok, err = httpc:set_keepalive(cfg.idle_timeout, cfg.pool_size) + if not ok then + ngx_log(ngx_ERR, "resty.http set_keepalive failed: ", tostring(err)) + end + + else + httpc:close() + end + + return result, res.status +end + + +-- ===== case table ===== +-- +-- Names are .. The headline ratio compares the same shape across +-- the ffi.* and resty.* columns; the gap between ffi.oneshot and ffi.stateful +-- separately prices the C fast path against this library's own Lua object +-- layer. + +local cases = {} + + +local function pair_case(name, shape) + for _, driver in ipairs({ "ffi", "resty" }) do + local copy = {} + + for k, v in pairs(shape) do + copy[k] = v + end + + -- separate pools, so one client can never be handed a connection the + -- other opened and look free for whichever case ran second + copy.pool = "bench-" .. driver .. "-" .. name + + local fn + if driver == "ffi" then + fn = shape.oneshot and ffi_oneshot or ffi_stateful + else + fn = shape.oneshot and resty_oneshot or resty_stateful + end + + cases[driver .. "." .. name] = { fn = fn, shape = copy } + end +end + + +-- the 2x2: one-shot and stateful, on both clients, against the same upstream +pair_case("oneshot", { oneshot = true }) +pair_case("stateful", {}) + +-- read mode +pair_case("readbody", { read_body = true }) +pair_case("stream", { stream = true }) + +-- connection lifetime +pair_case("short", { keepalive = false }) + +-- method +pair_case("post", { method = "POST", body = POST_BODY, path = "/mock/sink" }) + +-- transport: pooled connections amortise the handshake away, so the +-- short-lived TLS case is the one that prices a fresh handshake +pair_case("tls", { scheme = "https" }) +pair_case("tlsshort", { scheme = "https", keepalive = false }) +-- verify on: these price the trust store, which the two cases above +-- deliberately do not. Both clients trust the same self-signed upstream cert. +pair_case("tlsverify", { scheme = "https", verify = true }) +pair_case("tlsverifyshort", + { scheme = "https", verify = true, keepalive = false }) + +-- peer address +pair_case("dns", { dns = true }) + +-- response framing +pair_case("chunked", { path = "/mock/chunked" }) +pair_case("trailers", { trailers = true, path = "/" }) + +-- response header count +pair_case("hdr40", { path = "/mock/headers40" }) +pair_case("cookies", { path = "/mock/cookies" }) + +-- request header count +pair_case("req30", { headers = HEADERS_30 }) + + +function _M.run(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.print("unknown bench case: ", tostring(name)) + return + end + + local n, status, err = case.fn(case.shape) + + if not n then + ngx.status = 502 + ngx_log(ngx_ERR, "bench case ", name, " failed: ", tostring(err)) + ngx.print(tostring(err)) + return + end + + ngx.status = status + + -- every case, baseline included, writes the same downstream bytes, so the + -- baseline subtraction removes an identical downstream cost from each + local body = cfg.response_body + ngx.header["Content-Length"] = #body + ngx.print(body) +end + + +function _M.baseline() + local body = cfg.response_body + ngx.header["Content-Length"] = #body + ngx.print(body) +end + + +-- Report what a case actually did: status and bytes read. The runner calls this +-- during the fairness audit, before any load, so nothing here costs anything on +-- the measured path. +function _M.audit(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.say("unknown bench case: ", tostring(name)) + return + end + + local n, status, err = case.fn(case.shape) + + if not n then + ngx.status = 502 + ngx.say("case=", name, " error=", tostring(err)) + return + end + + ngx.say("case=", name, " status=", status, " bytes=", n) +end + + +-- Run a case against the upstream's echo endpoint, which replies with the raw +-- request header block it received. The runner diffs the two columns of a pair: +-- if the bytes differ, the pair is not measuring the same request, and that is +-- a fairness failure before any load is applied. +function _M.echo(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.say("unknown bench case: ", tostring(name)) + return + end + + local shape = {} + for k, v in pairs(case.shape) do + shape[k] = v + end + + shape.path = "/mock/echo" + -- the raw TCP trailer mock has no echo endpoint, so this pair is audited + -- against the http mock instead + shape.trailers = nil + -- echoing is a buffered read whatever the case's own read mode is + shape.stream = nil + shape.read_body = nil + shape.echo = true + shape.pool = shape.pool .. "-echo" + + local body, _, err = case.fn(shape) + + if not body then + ngx.status = 502 + ngx.say("error: ", tostring(err)) + return + end + + ngx.print(body) +end + + +function _M.case_names() + local names = {} + + for name in pairs(cases) do + names[#names + 1] = name + end + + table_sort(names) + + return names +end + + +return _M diff --git a/benchmark/results-full.md b/benchmark/results-full.md new file mode 100644 index 0000000..e8e52b1 --- /dev/null +++ b/benchmark/results-full.md @@ -0,0 +1,199 @@ +# Full-matrix benchmark run (`full`) + +Run finished 2026-08-04. This file records one complete run of +`benchmark/run.sh` comparing this repo's C/FFI HTTP client +(`resty.ngx_http_ffi_client`) against `lua-resty-http` 0.16.1 across the full +matrix of request shapes. It is a results record only; it does not replace the +methodology write-up in `benchmark/README.md`. + +## What was run + +- Run parameters: `BENCH_RUN_ID=full`, `BENCH_CASES=all`, `BENCH_REPEATS=5`, + `BENCH_LATENCY_REPEATS=3`. +- Hardware: 8-core VM. The target OpenResty worker (openresty/1.29.2.4) was + pinned to CPU 0, the upstream to CPUs 1-5, and `wrk2` to CPUs 6-7, so the + measured client always runs on a single dedicated core and neither the load + generator nor the upstream competes with it. +- The run exited 0. The fairness audit passed for all 14 shapes: every shape + produced the same request line and the same caller-supplied headers from both + clients (all 14 `echo-.diff` artifacts are empty). The only reported + difference is the known, non-failing one — the FFI client emits + `Connection: keep-alive` and `Content-Length: 0` on a bodyless GET where + `lua-resty-http` omits them (`Connection: keep-alive` only, for `post`). +- Raw rows: `benchmark/.run/results/full.csv`, 229 data rows — 145 throughput + rows (29 cases x 5 repeats) and 84 latency rows (28 cases x 3 repeats). +- Exactly 1 row was discarded: `resty.dns` throughput repeat 1, which reported + 4 non-2xx responses and 13 socket errors. It passed the saturation gate but + the non-2xx count excludes it. No other row in the file has a non-zero + `non2xx` or `socket_errors` value, and no throughput row failed the saturation + gate (`saturated` is `yes` on all 145). + +## Folded results + +The tables below are the verbatim output of +`./benchmark/fold.sh benchmark/.run/results/full.csv`. Every figure is a median +over the kept repeats. + +### Run parameters + +parser `llhttp`, response size `1024` bytes, wrk2 connections `100`, duration `30s` per repeat + +### Headline (median QPS per shape) + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `stream` | `27506.05` | `16236.65` | `1.69x` | `2.02x` | +| `stateful` | `26919.60` | `15604.60` | `1.73x` | `2.05x` | +| `hdr40` | `17724.68` | `9218.37` | `1.92x` | `2.16x` | +| `req30` | `23339.15` | `13283.15` | `1.76x` | `2.04x` | +| `cookies` | `25440.85` | `13603.27` | `1.87x` | `2.23x` | +| `trailers` | `28005.24` | `6622.05` | `4.23x` | `5.77x` | +| `readbody` | `26965.62` | `15710.28` | `1.72x` | `2.04x` | +| `oneshot` | `30925.84` | `14510.37` | `2.13x` | `2.76x` | +| `tlsshort` | `932.16` | `1149.50` | `0.81x` | `0.81x` | +| `post` | `24173.16` | `12780.46` | `1.89x` | `2.24x` | +| `chunked` | `26663.11` | `15366.94` | `1.74x` | `2.06x` | +| `short` | `8721.09` | `7152.20` | `1.22x` | `1.24x` | +| `dns` | `26814.07` | `15634.98` | `1.72x` | `2.04x` | +| `tls` | `18655.46` | `11866.50` | `1.57x` | `1.73x` | + +Baseline (no upstream call): `86589.91` QPS, `11.549` us/request. +The outbound cost ratio subtracts that baseline from both client paths and compares what is left. + +### Detail (throughput phase) + +| case | median QPS | min | max | repeats used | discarded | worker CPU avg | worker RSS | requests/connection | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `baseline` | `86589.91` | `85202.57` | `86833.79` | 5 | 0 | `106.6%` | `2035128` KB | n/a | +| `ffi.chunked` | `26663.11` | `25049.33` | `26744.73` | 5 | 0 | `116.2%` | `3062632` KB | `751598.0` | +| `ffi.cookies` | `25440.85` | `25385.00` | `25508.05` | 5 | 0 | `116.1%` | `3062632` KB | `764350.5` | +| `ffi.dns` | `26814.07` | `25315.59` | `26863.08` | 5 | 0 | `116.2%` | `3062632` KB | `804524.0` | +| `ffi.hdr40` | `17724.68` | `17643.17` | `17737.64` | 5 | 0 | `107.8%` | `3062632` KB | `531768.5` | +| `ffi.oneshot` | `30925.84` | `30751.13` | `31065.85` | 5 | 0 | `117.7%` | `2035128` KB | no new connections | +| `ffi.post` | `24173.16` | `22771.48` | `24269.40` | 5 | 0 | `115.3%` | `2082632` KB | `702825.5` | +| `ffi.readbody` | `26965.62` | `26790.39` | `27007.54` | 5 | 0 | `116.2%` | `2035128` KB | no new connections | +| `ffi.req30` | `23339.15` | `23306.86` | `23466.05` | 5 | 0 | `114.8%` | `3062632` KB | `700270.0` | +| `ffi.short` | `8721.09` | `8389.79` | `8743.68` | 5 | 0 | `105.9%` | `2082104` KB | `1.0` | +| `ffi.stateful` | `26919.60` | `26402.32` | `26949.99` | 5 | 0 | `116.3%` | `2035128` KB | `807690.0` | +| `ffi.stream` | `27506.05` | `27498.37` | `27630.86` | 5 | 0 | `116.7%` | `2035128` KB | `825047.0` | +| `ffi.tls` | `18655.46` | `18438.79` | `18782.34` | 5 | 0 | `114.9%` | `2083824` KB | `563467.5` | +| `ffi.tlsshort` | `932.16` | `861.20` | `933.17` | 5 | 0 | `102.8%` | `3059584` KB | `1.0` | +| `ffi.trailers` | `28005.24` | `27915.10` | `28128.38` | 5 | 0 | `116.2%` | `3062632` KB | n/a | +| `resty.chunked` | `15366.94` | `15164.85` | `15764.74` | 5 | 0 | `114.1%` | `3062632` KB | `53208.0` | +| `resty.cookies` | `13603.27` | `13582.52` | `13898.90` | 5 | 0 | `113.2%` | `3062632` KB | `51824.9` | +| `resty.dns` | `15634.98` | `15489.53` | `15936.46` | 4 | 1 | `114.2%` | `3566436` KB | `46329.1` | +| `resty.hdr40` | `9218.37` | `9200.90` | `9442.03` | 5 | 0 | `111.2%` | `3062632` KB | `30854.3` | +| `resty.oneshot` | `14510.37` | `14299.32` | `14629.47` | 5 | 0 | `113.4%` | `2035128` KB | `89883.6` | +| `resty.post` | `12780.46` | `12371.84` | `13177.75` | 5 | 0 | `113.8%` | `2082632` KB | `94728.2` | +| `resty.readbody` | `15710.28` | `15562.79` | `15942.52` | 5 | 0 | `114.3%` | `2035128` KB | `86699.9` | +| `resty.req30` | `13283.15` | `13208.27` | `13354.04` | 5 | 0 | `112.8%` | `3062632` KB | `22262.1` | +| `resty.short` | `7152.20` | `7099.21` | `7266.89` | 5 | 0 | `105.5%` | `2082228` KB | `1.0` | +| `resty.stateful` | `15604.60` | `15493.86` | `15795.75` | 5 | 0 | `114.4%` | `2035128` KB | `31003.5` | +| `resty.stream` | `16236.65` | `16021.80` | `17296.03` | 5 | 0 | `114.4%` | `2035128` KB | `61189.6` | +| `resty.tls` | `11866.50` | `11503.14` | `12022.41` | 5 | 0 | `113.6%` | `2085452` KB | `36355.9` | +| `resty.tlsshort` | `1149.50` | `1105.48` | `1152.59` | 5 | 0 | `102.9%` | `3063980` KB | `1.0` | +| `resty.trailers` | `6622.05` | `6543.31` | `6782.34` | 5 | 0 | `105.3%` | `3062632` KB | n/a | + +### Latency (fixed rate, corrected for coordinated omission) + +| case | rate | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | ---: | +| `ffi.chunked` | `9220` req/s | `1.060` ms | `2.260` ms | `2.710` ms | +| `ffi.cookies` | `8161` req/s | `1.040` ms | `2.200` ms | `2.670` ms | +| `ffi.dns` | `9380` req/s | `1.070` ms | `2.300` ms | `2.760` ms | +| `ffi.hdr40` | `5531` req/s | `1.030` ms | `2.110` ms | `2.540` ms | +| `ffi.oneshot` | `8706` req/s | `1.020` ms | `2.210` ms | `2.770` ms | +| `ffi.post` | `7668` req/s | `1.050` ms | `2.220` ms | `2.670` ms | +| `ffi.readbody` | `9426` req/s | `1.060` ms | `2.270` ms | `2.770` ms | +| `ffi.req30` | `7969` req/s | `1.150` ms | `2.370` ms | `2.850` ms | +| `ffi.short` | `4291` req/s | `1.180` ms | `2.490` ms | `3.180` ms | +| `ffi.stateful` | `9362` req/s | `1.070` ms | `2.270` ms | `2.730` ms | +| `ffi.stream` | `9741` req/s | `1.090` ms | `2.370` ms | `2.880` ms | +| `ffi.tls` | `7119` req/s | `1.110` ms | `2.340` ms | `2.790` ms | +| `ffi.tlsshort` | `559` req/s | `4.260` ms | `9.010` ms | `11.840` ms | +| `ffi.trailers` | `3973` req/s | `1.000` ms | `2.100` ms | `2.600` ms | +| `resty.chunked` | `9220` req/s | `1.300` ms | `2.800` ms | `3.400` ms | +| `resty.cookies` | `8161` req/s | `1.290` ms | `2.820` ms | `3.480` ms | +| `resty.dns` | `9380` req/s | `1.300` ms | `2.810` ms | `3.380` ms | +| `resty.hdr40` | `5531` req/s | `1.200` ms | `2.540` ms | `3.080` ms | +| `resty.oneshot` | `8706` req/s | `1.290` ms | `2.830` ms | `3.410` ms | +| `resty.post` | `7668` req/s | `1.320` ms | `2.800` ms | `3.350` ms | +| `resty.readbody` | `9426` req/s | `1.270` ms | `2.760` ms | `3.350` ms | +| `resty.req30` | `7969` req/s | `1.500` ms | `3.020` ms | `3.940` ms | +| `resty.short` | `4291` req/s | `1.300` ms | `2.740` ms | `3.310` ms | +| `resty.stateful` | `9362` req/s | `1.270` ms | `2.750` ms | `3.320` ms | +| `resty.stream` | `9741` req/s | `2.040` ms | `4.650` ms | `5.630` ms | +| `resty.tls` | `7119` req/s | `2.330` ms | `5.410` ms | `6.580` ms | +| `resty.tlsshort` | `559` req/s | `3.330` ms | `5.610` ms | `8.030` ms | +| `resty.trailers` | `3973` req/s | `1.890` ms | `43.520` ms | `109.570` ms | + +### Discarded rows + +Rows failing the saturation gate or reporting non-2xx responses, excluded from every median above: + +- `resty.dns`: 1 of 5 + +## What this shows + +- The FFI client wins throughput on 13 of the 14 shapes. Excluding `trailers`, + the ratios span `1.22x` (`short`, 8721.09 vs 7152.20 QPS) to `2.13x` + (`oneshot`, 30925.84 vs 14510.37 QPS); the common keepalive shapes + (`stream`, `stateful`, `readbody`, `chunked`, `dns`) cluster at `1.69x`-`1.74x`. +- `trailers` is the largest win at `4.23x` (28005.24 vs 6622.05 QPS), and + `5.77x` once the baseline is subtracted from both paths. +- `tlsshort` is the one shape the FFI client LOSES: `0.81x` (932.16 vs 1149.50 + QPS). That shape opens a fresh TLS connection and performs a full handshake + for every request, so the measurement is dominated by handshake cost rather + than by client-side request/response handling. Use `lua-resty-http` if your + workload really does handshake per request. +- Latency: the FFI client wins p50, p99 and p99.9 on every shape except + `tlsshort`. Typical keepalive p50 is `1.02`-`1.15` ms for FFI against + `1.27`-`1.50` ms for `resty.http`, and p99.9 is `2.54`-`3.18` ms against + `3.08`-`3.94` ms. +- On `tlsshort` the FFI client loses the latency picture too: p50 `4.260` vs + `3.330` ms, p99 `9.010` vs `5.610` ms, p99.9 `11.840` vs `8.030` ms. +- Connection reuse differs sharply. On keepalive shapes the FFI client either + opened no new upstream connections at all during the measured window + (`oneshot`, `readbody`) or reached 5.3e5-8.3e5 requests per new connection, + while `resty.http` sat at 2.2e4-9.5e4. Both clients are at `1.0` on the + deliberately non-keepalive shapes (`short`, `tlsshort`). +- Repeat-to-repeat spread was small. Across the 5 throughput repeats, 20 of the + 29 cases had a max-min spread under 4% of the maximum; the widest were + `ffi.tlsshort` at 7.71% (861.20-933.17), `resty.stream` at 7.37% + (16021.80-17296.03) and `ffi.chunked` at 6.34% (25049.33-26744.73). The + baseline varied by 1.88% (85202.57-86833.79). +- The `trailers` rows carry no requests-per-connection figure because the + upstream connection counters read 0/0 for both clients on that shape; the + QPS and latency numbers for `trailers` are unaffected. + +## Correction: the `ffi.stateful` p99.9 tail claim is disproven + +An earlier run suggested that `ffi.stateful` had a bad p99.9 tail of 20.94 ms +against `resty.stateful` at 6.80 ms. That was a single-sample artifact. This run +used 3 latency repeats and the result does not reproduce: + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.stateful` | `1.070` ms | `2.270` ms | `2.730` ms | +| `resty.stateful` | `1.270` ms | `2.750` ms | `3.320` ms | + +The FFI client wins the tail on `stateful`, it does not lose it. The three +individual `ffi.stateful` p99.9 samples were 2.720, 2.730 and 2.730 ms — tightly +clustered, with nothing near 20 ms — against 3.320, 3.340 and 3.280 ms for +`resty.stateful`. Do not act on the old 20.94 ms claim; it is superseded by +this run. + +## Reverse finding: `resty.trailers` tail latency + +The one genuinely large tail in the whole run belongs to `lua-resty-http`, not +to the FFI client. On the `trailers` shape at 3973 req/s: + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.trailers` | `1.000` ms | `2.100` ms | `2.600` ms | +| `resty.trailers` | `1.890` ms | `43.520` ms | `109.570` ms | + +That is a 42x gap at p99.9. It is also unstable across repeats: the three +`resty.trailers` p99.9 samples were 109.570, 196.610 and 11.970 ms (p99: 43.520, +126.850 and 3.600 ms), while `ffi.trailers` produced 2.690, 2.600 and 2.560 ms. +The median reported above is the middle sample in each case. diff --git a/benchmark/run.sh b/benchmark/run.sh index ef944bd..c580f6e 100755 --- a/benchmark/run.sh +++ b/benchmark/run.sh @@ -6,32 +6,65 @@ ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" RUN_DIR="${RUN_DIR:-$ROOT_DIR/benchmark/.run}" TARGET_PREFIX="$RUN_DIR/target" UPSTREAM_PREFIX="$RUN_DIR/upstream" +RESULTS_DIR="${BENCH_RESULTS_DIR:-$RUN_DIR/results}" +CASES_FILE="${BENCH_CASES_FILE:-$ROOT_DIR/benchmark/cases.txt}" TARGET_PORT="${TARGET_PORT:-19840}" UPSTREAM_PORT="${UPSTREAM_PORT:-19841}" +UPSTREAM_TLS_PORT="${UPSTREAM_TLS_PORT:-19842}" +UPSTREAM_TRAILER_PORT="${UPSTREAM_TRAILER_PORT:-19843}" +UPSTREAM_HOST="${UPSTREAM_HOST:-127.0.0.1}" TARGET_CPU="${TARGET_CPU:-0}" RESPONSE_SIZE="${RESPONSE_SIZE:-1024}" KEEPALIVE_POOL_SIZE="${KEEPALIVE_POOL_SIZE:-120}" KEEPALIVE_IDLE_TIMEOUT="${KEEPALIVE_IDLE_TIMEOUT:-60000}" +CLIENT_TIMEOUT="${CLIENT_TIMEOUT:-60000}" +DNS_HOST="${DNS_HOST:-localhost}" +TLS_SERVER_NAME="${TLS_SERVER_NAME:-bench.local}" +CA_FILE="${CA_FILE:-$RUN_DIR/target/conf/ca.crt}" + WRK_THREADS="${WRK_THREADS:-2}" WRK_CONNECTIONS="${WRK_CONNECTIONS:-100}" WRK_DURATION="${WRK_DURATION:-30s}" +WRK_WARMUP="${WRK_WARMUP:-10s}" WRK_RATE="${WRK_RATE:-100000}" CPU_SATURATION_THRESHOLD="${CPU_SATURATION_THRESHOLD:-95}" + +BENCH_REPEATS="${BENCH_REPEATS:-5}" +BENCH_LATENCY="${BENCH_LATENCY:-1}" +BENCH_LATENCY_FRACTION="${BENCH_LATENCY_FRACTION:-0.6}" +# Tail percentiles are the noisiest thing here, so one window is not evidence. +BENCH_LATENCY_REPEATS="${BENCH_LATENCY_REPEATS:-3}" +BENCH_GC="${BENCH_GC:-1}" +BENCH_AUDIT="${BENCH_AUDIT:-1}" +BENCH_STRICT="${BENCH_STRICT:-0}" +DROPPED=() BENCH_DRY_RUN="${BENCH_DRY_RUN:-0}" +BENCH_PARSER="${BENCH_PARSER:-${NGX_HTTP_FFI_CLIENT_USE_LLHTTP:-llhttp}}" + +# The default set is the 2x2 that produces the headline ratio. BENCH_CASES=all +# runs every case in benchmark/cases.txt. +BENCH_CASES="${BENCH_CASES:-baseline ffi.oneshot resty.oneshot ffi.stateful resty.stateful}" TEST_NGINX_BINARY="${TEST_NGINX_BINARY:-}" +BENCH_LUA_DIR="$ROOT_DIR/benchmark/lua" TEST_NGINX_LUA_PACKAGE_PATH="${TEST_NGINX_LUA_PACKAGE_PATH:-$ROOT_DIR/lib/?.lua;$ROOT_DIR/lib/?/init.lua}" +LUA_PACKAGE_PATH="$TEST_NGINX_LUA_PACKAGE_PATH;$BENCH_LUA_DIR/?.lua" WRK2_BIN="${WRK2_BIN:-${WRK_BIN:-wrk}}" CURL_BIN="${CURL_BIN:-curl}" TASKSET_BIN="${TASKSET_BIN:-taskset}" +OPENSSL_BIN="${OPENSSL_BIN:-openssl}" die() { printf 'error: %s\n' "$*" >&2 exit 1 } +warn() { + printf 'warning: %s\n' "$*" >&2 +} + need_cmd() { command -v "$1" >/dev/null 2>&1 || die "$1 is required" } @@ -100,6 +133,57 @@ cpu_lists_overlap "$TARGET_CPU" "$UPSTREAM_CPUS" && cpu_lists_overlap "$TARGET_CPU" "$WRK_CPUS" && die "TARGET_CPU overlaps WRK_CPUS" +# The topology this benchmark assumes is a pinned target worker, an upstream +# that cannot be the bottleneck, and a load generator that competes with +# neither. Below eight cores that is not available, and the numbers describe the +# host rather than the client. +if (( NCPU < 8 )); then + warn "only $NCPU cores online; the plan's topology wants 8+ (target on 1," \ + "upstream on 5, wrk2 on 2). Results here are regression smoke, not" \ + "headline numbers." +fi + +default_resolver() { + awk '/^nameserver/ { print $2; exit }' /etc/resolv.conf 2>/dev/null || + printf '127.0.0.1\n' +} + +RESOLVER="${BENCH_RESOLVER:-$(default_resolver)}" +[[ -n "$RESOLVER" ]] || RESOLVER="127.0.0.1" + +# ===== case list ===== + +read_case_file() { + sed -e 's/#.*//' -e 's/[[:space:]]//g' "$CASES_FILE" | grep -v '^$' +} + +resolve_cases() { + local requested="$1" + + if [[ "$requested" == "all" ]]; then + printf 'baseline\n' + read_case_file + return + fi + + printf '%s\n' $requested +} + +mapfile -t CASES < <(resolve_cases "$BENCH_CASES") +(( ${#CASES[@]} > 0 )) || die "BENCH_CASES resolved to nothing" + +case_path() { + local name="$1" + + if [[ "$name" == "baseline" ]]; then + printf '/bench/baseline' + else + printf '/bench/%s' "$name" + fi +} + +# ===== template rendering ===== + sed_escape() { printf '%s' "$1" | sed -e 's/[&|]/\\&/g' } @@ -109,16 +193,57 @@ render_template() { local output="$2" sed \ - -e "s|{{LUA_PACKAGE_PATH}}|$(sed_escape "$TEST_NGINX_LUA_PACKAGE_PATH")|g" \ + -e "s|{{LUA_PACKAGE_PATH}}|$(sed_escape "$LUA_PACKAGE_PATH")|g" \ -e "s|{{TARGET_PORT}}|$(sed_escape "$TARGET_PORT")|g" \ + -e "s|{{UPSTREAM_HOST}}|$(sed_escape "$UPSTREAM_HOST")|g" \ -e "s|{{UPSTREAM_PORT}}|$(sed_escape "$UPSTREAM_PORT")|g" \ + -e "s|{{UPSTREAM_TLS_PORT}}|$(sed_escape "$UPSTREAM_TLS_PORT")|g" \ + -e "s|{{UPSTREAM_TRAILER_PORT}}|$(sed_escape "$UPSTREAM_TRAILER_PORT")|g" \ -e "s|{{UPSTREAM_WORKERS}}|$(sed_escape "$UPSTREAM_WORKERS")|g" \ -e "s|{{RESPONSE_SIZE}}|$(sed_escape "$RESPONSE_SIZE")|g" \ -e "s|{{KEEPALIVE_POOL_SIZE}}|$(sed_escape "$KEEPALIVE_POOL_SIZE")|g" \ -e "s|{{KEEPALIVE_IDLE_TIMEOUT}}|$(sed_escape "$KEEPALIVE_IDLE_TIMEOUT")|g" \ + -e "s|{{CLIENT_TIMEOUT}}|$(sed_escape "$CLIENT_TIMEOUT")|g" \ + -e "s|{{DNS_HOST}}|$(sed_escape "$DNS_HOST")|g" \ + -e "s|{{TLS_SERVER_NAME}}|$(sed_escape "$TLS_SERVER_NAME")|g" \ + -e "s|{{CA_FILE}}|$(sed_escape "$CA_FILE")|g" \ + -e "s|{{RESOLVER}}|$(sed_escape "$RESOLVER")|g" \ "$template" > "$output" } +# One exact location per case, generated from cases.txt. Exact rather than one +# regex location on purpose: a regex match would land on the measured worker's +# hot path for every request of every case. +generate_locations() { + local output="$1" + local name + + : > "$output" + while read -r name; do + cat >> "$output" </dev/null 2>&1 || + die "failed to generate a TLS certificate with $OPENSSL_BIN" +} + prepare_prefix() { local prefix="$1" @@ -136,31 +261,54 @@ prepare_prefix() { prepare_configs() { prepare_prefix "$UPSTREAM_PREFIX" prepare_prefix "$TARGET_PREFIX" + mkdir -p "$RESULTS_DIR" render_template "$ROOT_DIR/benchmark/conf/upstream.nginx.conf" \ "$UPSTREAM_PREFIX/conf/nginx.conf" render_template "$ROOT_DIR/benchmark/conf/target.nginx.conf" \ "$TARGET_PREFIX/conf/nginx.conf" + + generate_locations "$TARGET_PREFIX/conf/bench_locations.conf" + # ssl_certificate resolves relative paths against the conf prefix + generate_cert "$UPSTREAM_PREFIX/conf" + + # The cert is self-signed with CA:TRUE, so it is its own root: handing the + # target a copy is all the verify-on cases need to trust the upstream. + # lua_ssl_trusted_certificate is read at config load, so it must be in place + # before nginx starts. + cp "$UPSTREAM_PREFIX/conf/tls.crt" "$CA_FILE" || + die "failed to install the trusted CA at $CA_FILE" } print_config() { cat </dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/baseline" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/smoke/ffi-short" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/smoke/resty-http-short" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/ffi" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/resty-http" >/dev/null - printf 'Smoke checks passed\n' -} - target_worker_pid() { local master_pid @@ -230,10 +370,10 @@ target_worker_pid() { ps -o pid= --ppid "$master_pid" | awk 'NR == 1 { print $1 }' } -read_proc_ticks() { - local pid="$1" +# ===== CPU sampling ===== - awk '{ print $14 + $15 }' "/proc/$pid/stat" +read_proc_ticks() { + awk '{ print $14 + $15 }' "/proc/$1/stat" } read_total_ticks() { @@ -246,6 +386,10 @@ read_total_ticks() { }' /proc/stat } +read_rss_kb() { + awk '/^VmRSS:/ { print $2 }' "/proc/$1/status" 2>/dev/null || printf '0\n' +} + sample_cpu_once() { local pid="$1" local proc_before proc_after total_before total_after proc_delta total_delta @@ -281,141 +425,538 @@ sample_cpu_loop() { done } -summarize_cpu() { - local samples="$1" - - awk ' +cpu_stat() { + awk -v want="$2" ' NF { sum += $1 - if ($1 > max) { - max = $1 - } + if ($1 > max) { max = $1 } n += 1 } END { - if (n == 0) { - printf "avg=0.0 max=0.0 samples=0" - } else { - printf "avg=%.1f max=%.1f samples=%d", sum / n, max, n - } + if (n == 0) { print "0"; exit } + if (want == "avg") { printf "%.1f\n", sum / n } + else if (want == "max") { printf "%.1f\n", max } + else { print n } } - ' "$samples" + ' "$1" } -cpu_saturated() { - local samples="$1" +# ===== wrk2 output parsing ===== - awk -v threshold="$CPU_SATURATION_THRESHOLD" ' - NF { - sum += $1 - n += 1 - } - END { - if (n == 0) { - exit 1 +extract_qps() { + awk '/Requests\/sec:/ { qps = $2 } END { print (qps == "" ? "0" : qps) }' "$1" +} + +extract_non2xx() { + awk '/Non-2xx or 3xx responses:/ { n = $NF } END { print (n == "" ? "0" : n) }' "$1" +} + +extract_socket_errors() { + awk ' + /Socket errors:/ { + total = 0 + for (i = 1; i <= NF; i++) { + if ($i ~ /^[0-9]+,?$/) { + v = $i + sub(",", "", v) + total += v + } } - exit !((sum / n) >= threshold) + print total + found = 1 } - ' "$samples" + END { if (!found) print "0" } + ' "$1" } -case_stem() { - printf '%s' "$1" | sed -e 's/[^A-Za-z0-9_-]/_/g' +# wrk2 with -L -U prints two distributions: the recorded one, which corrects for +# coordinated omission by accounting for delayed request starts, and the +# uncorrected one. The corrected figures are the point of running wrk2 at a +# fixed rate at all, so they are the default; BENCH_LATENCY_MODE=uncorrected +# selects the other block. Percentile values come in whatever unit fits, so they +# are normalised to milliseconds before anything compares them. +extract_percentile() { + local output="$1" + local pct="$2" + local mode="${BENCH_LATENCY_MODE:-corrected}" + + awk -v pct="$pct" -v mode="$mode" ' + function to_ms(v) { + if (v ~ /us$/) { sub(/us$/, "", v); return v / 1000 } + if (v ~ /ms$/) { sub(/ms$/, "", v); return v + 0 } + if (v ~ /m$/) { sub(/m$/, "", v); return v * 60000 } + if (v ~ /s$/) { sub(/s$/, "", v); return v * 1000 } + return v + 0 + } + /Latency Distribution/ { + in_block = (($0 ~ /Uncorrected/) == (mode == "uncorrected")) + next + } + /Detailed Percentile spectrum/ { in_block = 0 } + in_block && $1 ~ /%$/ { + p = $1 + sub(/%$/, "", p) + if (p + 0 == pct + 0 && !seen) { + printf "%.3f\n", to_ms($2) + seen = 1 + } + } + END { if (!seen) print "" } + ' "$output" } -qps_file_for_label() { - local label="$1" +# ===== upstream reuse accounting ===== - printf '%s/%s.qps' "$RUN_DIR" "$(case_stem "$label")" +upstream_stats_reset() { + "$CURL_BIN" -fsS "http://127.0.0.1:$UPSTREAM_PORT/stats/reset" \ + >/dev/null 2>&1 || true } -extract_qps() { - local output="$1" +upstream_stat() { + "$CURL_BIN" -fsS "http://127.0.0.1:$UPSTREAM_PORT/stats" 2>/dev/null | + awk -v key="$1" '$1 == key { print $2 }' +} + +# ===== results ===== - awk '/Requests\/sec:/ { qps = $2 } END { if (qps == "") exit 1; print qps }' \ - "$output" +RUN_ID="${BENCH_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}" +RESULTS_CSV="$RESULTS_DIR/$RUN_ID.csv" + +CSV_HEADER='run_id,phase,repeat,case,qps,rate,non2xx,socket_errors,p50_ms,p99_ms,p999_ms,cpu_avg,cpu_max,cpu_samples,saturated,rss_kb,upstream_connections,upstream_requests,requests_per_connection,response_size,connections,threads,duration,parser' + +init_results() { + mkdir -p "$RESULTS_DIR" + printf '%s\n' "$CSV_HEADER" > "$RESULTS_CSV" + printf 'Writing rows to %s\n' "$RESULTS_CSV" } +# ===== one measured run ===== + run_wrk_case() { - local label="$1" - local path="$2" - local worker_pid stem samples output qps sampler_pid + local phase="$1" + local repeat="$2" + local name="$3" + local duration="$4" + local rate="$5" + + local path stem samples output worker_pid sampler_pid + local qps non2xx sockerr cpu_avg cpu_max cpu_n saturated rss + local up_conn up_req rpc p50 p99 p999 + + path="$(case_path "$name")" + stem="$(printf '%s' "${phase}_${repeat}_${name}" | tr -c 'A-Za-z0-9_-' '_')" + samples="$RUN_DIR/$stem.cpu" + output="$RUN_DIR/$stem.wrk" worker_pid="$(target_worker_pid)" [[ -n "$worker_pid" ]] || die "failed to find target worker pid" - stem="$(case_stem "$label")" - samples="$RUN_DIR/$stem.cpu" - output="$RUN_DIR/$stem.wrk" + printf '\n== %s repeat %s: %s (%s, rate %s) ==\n' \ + "$phase" "$repeat" "$name" "$duration" "$rate" + + # warmup, discarded: fills the keepalive pools and lets the JIT settle so + # the measured window is not paying for either + if [[ "$WRK_WARMUP" != "0" && "$WRK_WARMUP" != "0s" ]]; then + "$TASKSET_BIN" -c "$WRK_CPUS" "$WRK2_BIN" \ + -t "$WRK_THREADS" -c "$WRK_CONNECTIONS" -d "$WRK_WARMUP" \ + -R "$rate" \ + -s "$ROOT_DIR/benchmark/wrk/headers.lua" \ + "http://127.0.0.1:$TARGET_PORT$path" >/dev/null 2>&1 || true + fi + + upstream_stats_reset - printf '\n== %s ==\n' "$label" sample_cpu_loop "$worker_pid" "$samples" & sampler_pid="$!" "$TASKSET_BIN" -c "$WRK_CPUS" "$WRK2_BIN" \ - -t "$WRK_THREADS" \ - -c "$WRK_CONNECTIONS" \ - -d "$WRK_DURATION" \ - -R "$WRK_RATE" \ - -L \ - -U \ + -t "$WRK_THREADS" -c "$WRK_CONNECTIONS" -d "$duration" \ + -R "$rate" -L -U \ -s "$ROOT_DIR/benchmark/wrk/headers.lua" \ "http://127.0.0.1:$TARGET_PORT$path" | tee "$output" + rss="$(read_rss_kb "$worker_pid")" + kill "$sampler_pid" >/dev/null 2>&1 || true wait "$sampler_pid" >/dev/null 2>&1 || true - qps="$(extract_qps "$output")" || - die "failed to parse Requests/sec from $output" - printf '%s\n' "$qps" > "$(qps_file_for_label "$label")" - printf 'actual throughput: %s requests/sec\n' "$qps" + qps="$(extract_qps "$output")" + non2xx="$(extract_non2xx "$output")" + sockerr="$(extract_socket_errors "$output")" + cpu_avg="$(cpu_stat "$samples" avg)" + cpu_max="$(cpu_stat "$samples" max)" + cpu_n="$(cpu_stat "$samples" n)" + p50="$(extract_percentile "$output" 50)" + p99="$(extract_percentile "$output" 99)" + p999="$(extract_percentile "$output" 99.9)" + + up_conn="$(upstream_stat connections)" + up_req="$(upstream_stat requests)" + up_conn="${up_conn:-0}" + up_req="${up_req:-0}" + # The counters are reset after the warmup, so they describe the steady + # state: connections opened during the measured window. Zero new + # connections against a non-zero request count is perfect pool reuse, which + # is the result the keepalive cases want; a ratio near 1 means the pool is + # not reusing anything and the case is measuring connection setup. + rpc="$(awk -v c="$up_conn" -v r="$up_req" 'BEGIN { + if (c + 0 > 0) printf "%.1f", r / c + else if (r + 0 > 0) print "inf" + else print "" + }')" + + saturated=no + if awk -v t="$CPU_SATURATION_THRESHOLD" -v a="$cpu_avg" \ + 'BEGIN { exit !(a + 0 >= t + 0) }'; then + saturated=yes + fi + + printf 'qps=%s non2xx=%s socket_errors=%s cpu_avg=%s%% rss=%sKB\n' \ + "$qps" "$non2xx" "$sockerr" "$cpu_avg" "$rss" + printf 'latency p50=%sms p99=%sms p99.9=%sms\n' "$p50" "$p99" "$p999" + + if [[ "$name" != "baseline" ]]; then + printf 'upstream: %s new connections, %s requests' "$up_conn" "$up_req" + if [[ -n "$rpc" ]]; then + printf ', %s requests/connection\n' "$rpc" + else + printf '\n' + fi + + # a keepalive case that opens roughly one connection per request is not + # exercising the pool, whatever else the row says + case "$name" in + *.short|*.tlsshort) ;; + *) + if [[ "$rpc" != "inf" && -n "$rpc" ]] && + awk -v v="$rpc" 'BEGIN { exit !(v + 0 < 2) }' + then + warn "$name reused almost nothing (${rpc} requests per" \ + "connection); this is not a keepalive result" + fi + ;; + esac + fi - printf 'target worker CPU: %s\n' "$(summarize_cpu "$samples")" - if cpu_saturated "$samples"; then - printf 'saturation: valid (average CPU >= %s%%)\n' \ - "$CPU_SATURATION_THRESHOLD" + if [[ "$non2xx" != "0" ]]; then + warn "$name reported $non2xx non-2xx responses; this row is invalid" + fi + + if [[ "$phase" == "throughput" && "$saturated" == "no" ]]; then + warn "$name average target worker CPU ${cpu_avg}% is below" \ + "${CPU_SATURATION_THRESHOLD}%; this row is not a saturated result" + fi + + printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n' \ + "$RUN_ID" "$phase" "$repeat" "$name" "$qps" "$rate" "$non2xx" \ + "$sockerr" "$p50" "$p99" "$p999" "$cpu_avg" "$cpu_max" "$cpu_n" \ + "$saturated" "$rss" "$up_conn" "$up_req" "$rpc" "$RESPONSE_SIZE" \ + "$WRK_CONNECTIONS" "$WRK_THREADS" "$duration" "$BENCH_PARSER" \ + >> "$RESULTS_CSV" +} + +# ===== checks before any load ===== + +smoke_check() { + printf 'Running smoke checks\n' + + "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/ready" >/dev/null || + die "target /ready failed" + + # the case table the worker actually has, against the file the runner drives + local worker_cases file_cases + worker_cases="$("$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/cases" | + sort)" + file_cases="$(read_case_file | sort)" + + if [[ "$worker_cases" != "$file_cases" ]]; then + printf 'case table drift between benchmark/cases.txt and bench.lua:\n' >&2 + diff <(printf '%s\n' "$file_cases") <(printf '%s\n' "$worker_cases") >&2 || + true + die "case list mismatch" + fi + + "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/baseline" >/dev/null || + die "baseline failed" + + # A case that cannot complete once is not going to produce a meaningful + # number under load. Not every case works on every build, though: the + # hand-written parser has no chunked framing, so its chunked and trailer + # cases fail here by design. Those are dropped and named in the summary + # rather than aborting the whole matrix. BENCH_STRICT=1 aborts instead. + local name kept=() + for name in "${CASES[@]}"; do + if [[ "$name" == "baseline" ]]; then + kept+=("$name") + continue + fi + + if ! "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/audit/$name" \ + > "$RUN_DIR/audit-$name.txt" 2>&1 + then + if [[ "$BENCH_STRICT" == "1" ]]; then + printf 'audit output: %s\n' \ + "$(cat "$RUN_DIR/audit-$name.txt")" >&2 + die "case $name failed its smoke check" + fi + + warn "case $name failed its smoke check and is excluded from this" \ + "run; it is not supported by this build or configuration" + DROPPED+=("$name") + continue + fi + + printf ' %s\n' "$(cat "$RUN_DIR/audit-$name.txt")" + kept+=("$name") + done + + CASES=("${kept[@]}") + + local remaining=0 + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] || remaining=$((remaining + 1)) + done + + (( remaining > 0 )) || die "every client case failed its smoke check" + + if (( ${#DROPPED[@]} )); then + printf 'Smoke checks passed for %d cases, %d excluded\n' \ + "$remaining" "${#DROPPED[@]}" else - printf 'saturation: suspicious (average CPU < %s%%; increase wrk load or check upstream capacity)\n' \ - "$CPU_SATURATION_THRESHOLD" + printf 'Smoke checks passed\n' fi } -print_outbound_estimate() { - local baseline_qps ffi_qps resty_qps +# Headers each client writes for itself rather than on the caller's behalf. A +# difference here is the client's own contract and is reported, not failed: the +# FFI client always emits Connection and Content-Length, lua-resty-http omits +# both on a bodyless GET. +GENERATED_HEADERS='^(host|connection|content-length|user-agent|transfer-encoding|accept|accept-encoding):' - baseline_qps="$(cat "$(qps_file_for_label "baseline")")" - ffi_qps="$(cat "$(qps_file_for_label "C FFI keepalive")")" - resty_qps="$(cat "$(qps_file_for_label "resty.http keepalive")")" +request_line() { + head -n 1 "$1" +} - awk -v baseline_qps="$baseline_qps" \ - -v ffi_qps="$ffi_qps" \ - -v resty_qps="$resty_qps" ' - BEGIN { - baseline_total_us = 1000000 / baseline_qps - ffi_total_us = 1000000 / ffi_qps - resty_total_us = 1000000 / resty_qps - ffi_outbound_us = ffi_total_us - baseline_total_us - resty_outbound_us = resty_total_us - baseline_total_us +caller_headers() { + tail -n +2 "$1" | grep -Ev -i "$GENERATED_HEADERS" | grep -v '^$' | sort +} - print "" - print "== Baseline-derived outbound estimate ==" - printf "baseline downstream+handler: %.3f us/request\n", baseline_total_us - printf "C FFI total: %.3f us/request, estimated outbound: %.3f us/request\n", ffi_total_us, ffi_outbound_us - printf "resty.http total: %.3f us/request, estimated outbound: %.3f us/request\n", resty_total_us, resty_outbound_us +generated_headers() { + tail -n +2 "$1" | grep -E -i "$GENERATED_HEADERS" | sort +} - if (ffi_outbound_us <= 0 || resty_outbound_us <= 0) { - print "estimated outbound ratio: unavailable because baseline throughput is not above both client cases" - } else { - printf "estimated outbound ratio (resty.http / C FFI): %.2fx\n", resty_outbound_us / ffi_outbound_us +# A1: prove the two columns of a pair send the same request. The upstream echoes +# the raw header block it received. The request line and every caller-supplied +# header must match, or the pair is not measuring the same request; headers the +# clients generate themselves are reported separately. +fairness_audit() { + printf '\n== Fairness audit ==\n' + + local shapes=() name shape seen ffi_out resty_out failures=0 + + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + shape="${name#*.}" + + seen=0 + local s + for s in "${shapes[@]:-}"; do + [[ "$s" == "$shape" ]] && seen=1 + done + (( seen )) || shapes+=("$shape") + done + + for shape in "${shapes[@]:-}"; do + ffi_out="$RUN_DIR/echo-ffi.$shape.txt" + resty_out="$RUN_DIR/echo-resty.$shape.txt" + + if ! "$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/echo/ffi.$shape" > "$ffi_out" 2>&1 + then + warn "echo for ffi.$shape failed: $(cat "$ffi_out")" + failures=$((failures + 1)) + continue + fi + + if ! "$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/echo/resty.$shape" > "$resty_out" 2>&1 + then + warn "echo for resty.$shape failed: $(cat "$resty_out")" + failures=$((failures + 1)) + continue + fi + + local ok=1 + + if [[ "$(request_line "$ffi_out")" != "$(request_line "$resty_out")" ]] + then + printf ' %-10s REQUEST LINE DIFFERS:\n' "$shape" + printf ' ffi: %s\n' "$(request_line "$ffi_out")" + printf ' resty: %s\n' "$(request_line "$resty_out")" + ok=0 + fi + + # header order is not part of the contract, so compare as sets + if ! diff <(caller_headers "$ffi_out") <(caller_headers "$resty_out") \ + > "$RUN_DIR/echo-$shape.diff" + then + printf ' %-10s CALLER HEADERS DIFFER:\n' "$shape" + sed -e 's/^/ /' "$RUN_DIR/echo-$shape.diff" + ok=0 + fi + + if (( ok )); then + printf ' %-10s same request line, same caller headers\n' "$shape" + else + failures=$((failures + 1)) + fi + + if ! diff <(generated_headers "$ffi_out") \ + <(generated_headers "$resty_out") > "$RUN_DIR/echo-$shape.gen.diff" + then + printf ' %-10s client-generated headers differ (reported, not a failure):\n' \ + "" + sed -e 's/^/ /' "$RUN_DIR/echo-$shape.gen.diff" + fi + done + + if (( failures )); then + warn "$failures shape(s) failed the fairness audit; the affected pairs" \ + "are not comparing like for like" + fi +} + +# A5: per-request Lua heap delta. The library's claim is that response handling +# stays in C, and the header arrays and the case-insensitive metatable are where +# that could regress, so this is reported rather than left as a diagnostic. +gc_report() { + printf '\n== Per-request Lua heap delta (KB/request) ==\n' + + local name delta + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + + delta="$("$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/gcdelta/$name" 2>/dev/null | + tail -n 1)" + printf ' %-18s %s\n' "$name" "${delta:-unavailable}" + done +} + +# ===== phases ===== + +median_qps() { + local name="$1" + + awk -F, -v want="$name" ' + NR > 1 && $2 == "throughput" && $4 == want && $15 == "yes" && $7 == 0 { + v[n++] = $5 + 0 + } + END { + if (n == 0) { print ""; exit } + for (i = 0; i < n; i++) { + for (j = i + 1; j < n; j++) { + if (v[j] < v[i]) { t = v[i]; v[i] = v[j]; v[j] = t } + } } - }' + if (n % 2) { printf "%.2f\n", v[(n - 1) / 2] } + else { printf "%.2f\n", (v[n / 2 - 1] + v[n / 2]) / 2 } + } + ' "$RESULTS_CSV" +} + +throughput_phase() { + local repeat name + + for ((repeat = 1; repeat <= BENCH_REPEATS; repeat++)); do + # cases rotate inside each repeat rather than running back to back, so + # drift and background load land on every case equally + for name in "${CASES[@]}"; do + run_wrk_case throughput "$repeat" "$name" "$WRK_DURATION" "$WRK_RATE" + done + done +} + +# A5: latency measured at saturation is queue depth. This pass re-runs each pair +# at a fixed fraction of the slower side's saturated throughput, which is the +# only rate at which the two are answering the same question. +latency_phase() { + printf '\n== Latency phase ==\n' + + local shapes=() name shape seen ffi_med resty_med rate + + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + shape="${name#*.}" + + seen=0 + local s + for s in "${shapes[@]:-}"; do + [[ "$s" == "$shape" ]] && seen=1 + done + (( seen )) || shapes+=("$shape") + done + + for shape in "${shapes[@]:-}"; do + ffi_med="$(median_qps "ffi.$shape")" + resty_med="$(median_qps "resty.$shape")" + + if [[ -z "$ffi_med" || -z "$resty_med" ]]; then + warn "no valid saturated throughput for both sides of $shape;" \ + "skipping its latency pass" + continue + fi + + rate="$(awk -v a="$ffi_med" -v b="$resty_med" \ + -v f="$BENCH_LATENCY_FRACTION" \ + 'BEGIN { m = (a < b ? a : b); printf "%d", m * f }')" + + (( rate > 0 )) || continue + + printf 'shape %s: fixed rate %s req/s (%.0f%% of the slower path)\n' \ + "$shape" "$rate" \ + "$(awk -v f="$BENCH_LATENCY_FRACTION" 'BEGIN { print f * 100 }')" + + local lrepeat + for ((lrepeat = 1; lrepeat <= BENCH_LATENCY_REPEATS; lrepeat++)); do + run_wrk_case latency "$lrepeat" "ffi.$shape" "$WRK_DURATION" "$rate" + run_wrk_case latency "$lrepeat" "resty.$shape" "$WRK_DURATION" "$rate" + done + done } +summarize() { + printf '\n== Summary (median of %s repeats, saturated rows only) ==\n' \ + "$BENCH_REPEATS" + + "$ROOT_DIR/benchmark/fold.sh" "$RESULTS_CSV" || true + + if (( ${#DROPPED[@]} )); then + printf '### Cases excluded before the run\n\n' + printf 'These failed their smoke check on this build and were not' + printf ' measured:\n\n' + local name + for name in "${DROPPED[@]}"; do + printf -- '- `%s`\n' "$name" + done + printf '\n' + fi + + printf 'Raw rows: %s\n' "$RESULTS_CSV" +} + +# ===== main ===== + prepare_configs print_config if [[ "$BENCH_DRY_RUN" == "1" ]]; then - printf 'BENCH_DRY_RUN=1, not starting OpenResty or wrk2\n' + printf '\nBENCH_DRY_RUN=1, not starting OpenResty or wrk2\n' + printf 'rendered: %s\n' "$TARGET_PREFIX/conf/nginx.conf" + printf 'rendered: %s\n' "$UPSTREAM_PREFIX/conf/nginx.conf" + printf 'rendered: %s (%s locations)\n' \ + "$TARGET_PREFIX/conf/bench_locations.conf" \ + "$(grep -c '^location' "$TARGET_PREFIX/conf/bench_locations.conf")" exit 0 fi @@ -430,7 +971,20 @@ wait_url "http://127.0.0.1:$TARGET_PORT/ready" smoke_check -run_wrk_case "baseline" "/bench/baseline" -run_wrk_case "C FFI keepalive" "/bench/ffi" -run_wrk_case "resty.http keepalive" "/bench/resty-http" -print_outbound_estimate +init_results + +if [[ "$BENCH_AUDIT" == "1" ]]; then + fairness_audit +fi + +throughput_phase + +if [[ "$BENCH_LATENCY" == "1" ]]; then + latency_phase +fi + +if [[ "$BENCH_GC" == "1" ]]; then + gc_report +fi + +summarize diff --git a/docs/ai-proxy-integration.md b/docs/ai-proxy-integration.md new file mode 100644 index 0000000..f373494 --- /dev/null +++ b/docs/ai-proxy-integration.md @@ -0,0 +1,163 @@ +# Track B: APISIX ai-proxy integration + +API contract findings from checking `resty.ngx_http_ffi_client` against what +APISIX's ai-proxy path actually needs from `lua-resty-http`. + +Checked against APISIX at `636100ec` and this library at `f13fcfa`. + +## The seam described in the plan does not exist + +The plan assumes `apisix/apisix/plugins/ai-transport/http.lua`, "already a seam" +exposing `_M.request(params, timeout)` and `_M.set_keepalive(res, ...)`, behind +which an alternate transport is "roughly 30 lines". + +There is no `ai-transport` directory in APISIX at `636100ec`. The outbound call +lives inline in `apisix/plugins/ai-drivers/openai-base.lua`, in `_M.request()`, +interleaved with everything else that function does: + +``` +_M.request(self, ctx, conf, request_table, extra_opts) + http.new() / set_timeout + fetch_gcp_access_token -- auth, may return early + url.parse(endpoint) -- scheme/host/port/path/query derivation + construct_forward_headers + self.request_filter(...) -- per-driver hook that mutates params + httpc:connect(params) + core.json.encode(params.body) + httpc:request(params) + if status == 429 or 5xx: return -- retry-relevant, before the body is read + read_response(...) -- SSE loop or buffered read + httpc:set_keepalive(conf.keepalive_timeout, conf.keepalive_pool) +``` + +`read_response` is a separate local function that receives `res` and drives +either the SSE `body_reader` loop or a buffered read. + +So B3.1 is not "write a 30-line module". It is "extract a transport interface +from `openai-base.lua`, then write the second implementation behind it". The +extraction is the larger and more contentious half, and it is a change to +APISIX, not to this library. That reordering should be agreed before the work +starts, because it changes who reviews it and where the risk sits. + +The rest of this document is B3.2 done statically: the contract gaps that the +swap would hit, found by reading both sides rather than by running the swap. + +## The surface ai-proxy needs + +| call | used for | +| --- | --- | +| `http.new()` | one client per request | +| `httpc:set_timeout(ms)` | single-argument form | +| `httpc:connect(params)` | `scheme`, `host`, `port`, `ssl_verify`, `ssl_server_name`; the whole params table is passed and unknown keys are expected to be ignored | +| `httpc:request(params)` | `method`, `path`, `query` (table), `headers`, `body` (string) | +| `res.status` | retry decision, before the body is read | +| `res.headers["Content-Type"]` | picks the SSE path | +| `res.body_reader` | SSE, chunk by chunk | +| `res:read_body()` | buffered responses | +| `httpc:set_keepalive(timeout, pool_size)` | `conf.keepalive_timeout`, `conf.keepalive_pool` | +| error strings | `handle_error` maps any error containing `timeout` to 504, everything else to 500 | + +## Already satisfied + +These were checked and need no work: + +- **`new`, `set_timeout`, `connect`, `request`, `set_keepalive`, `close`, + `get_reused_times`** all exist with matching signatures. +- **`connect` ignores unknown keys.** ai-proxy passes the whole `params` table, + including `method`, `headers`, `body` and `query`, into `connect`. This + library reads only the keys it knows. +- **`query` as a table** is encoded with `ngx.encode_args`, as `lua-resty-http` + does. +- **`res.headers["Content-Type"]`** resolves through the case-insensitive + metatable regardless of the casing the provider used. +- **`res.body_reader` is the default.** `request{}` without `preread_body` sets + up the streaming reader, which is what the SSE path needs. The buffered fast + path is opt-in, so a naive swap gets streaming behaviour, not a surprise + full-body buffer. +- **Timeout error strings.** `handle_error` looks for the substring `timeout`; + the C path emits exactly `"timeout"` for read, connect and TLS-handshake + timeouts, deliberately, with a comment saying it matches the cosocket. A 504 + stays a 504. +- **Keepalive pool isolation.** ai-proxy passes no explicit pool name, so the + key is derived. The derivation is a tagged, length-prefixed encoding of + `[ssl][verify][host:port][ssl_server_name][ssl_trusted_certificate]`, so + connections cannot cross between TLS and plaintext, between verify modes, + between SNI names, or between trusted CAs. With one gateway fronting several + providers on different TLS settings this is the property that matters most, + and it holds. + +## Gaps + +### 1. `request_uri` is a module function, not a method + +`lua-resty-http` exposes `httpc:request_uri(uri, params)`. This library exposes +`client.request_uri(opts)` at module level, taking a table rather than a URI +string, and has no method of that name on the client object. + +ai-proxy does not call it, so it does not block the swap. Every other APISIX +plugin that reaches for `resty.http` does, though, so it blocks the "drop-in" +claim in general. Either add a `request_uri` method with the `lua-resty-http` +signature, or state plainly in the README that the one-shot entry point is +deliberately different. + +### 2. `res.reason` and `res.has_body` are missing + +`lua-resty-http` sets both on the response table. This library sets neither. + +ai-proxy does not use them, so this is not a blocker for Track B. It is a +drop-in gap for anything that logs `res.reason`, and `has_body` is the idiomatic +way to check for a bodyless response before touching the reader. + +### 3. No proxy support + +`lua-resty-http` has `set_proxy_options` and honours `proxy_opts` in `connect` +and `request_uri`. This library has none. + +ai-proxy's `openai-base.lua` does not use it, but `request_filter` hooks are +free to set it and APISIX deployments behind a corporate proxy will want it. +This is the largest single missing feature relative to `lua-resty-http`. + +### 4. Request bodies must be strings + +`lua-resty-http` accepts a string, a table of strings, or a function for the +request body, and supports chunked request framing. This library takes a string +only, and refuses `Transfer-Encoding` outright because it always frames with +`Content-Length`. + +ai-proxy JSON-encodes to a string first, so it is unaffected. Any caller +streaming a request body is not portable to this client. + +### 5. `Connection` and `Content-Length` are sent on every request + +Found by the benchmark's fairness audit: this library emits + +``` +Connection: keep-alive +Content-Length: 0 +``` + +on requests where `lua-resty-http` emits neither. `lua-resty-http` only sets +`Content-Length` for methods that expect a body. + +For ai-proxy this is low risk: the requests are POSTs with a body, so +`Content-Length` is correct and present either way, and the only difference on +the wire is the explicit `Connection: keep-alive`. It is worth a deliberate +decision rather than an accident, because a `Content-Length: 0` on a GET is the +kind of thing a strict provider or CDN in front of one can reject, and this +would show up as provider-specific failures rather than as an obvious bug. + +## What B3.3 to B3.5 still need + +- **B3.3, compatibility.** Running APISIX's ai-proxy suite under both transports + needs the transport interface from B3.1 to exist first, plus an APISIX test + environment. Not attempted here. +- **B3.4, end-to-end performance.** Needs a mock openai-compatible SSE upstream + on localhost. Against a real provider, network and provider latency dominate + and the client CPU difference disappears into variance, so a real-provider run + is a sanity check only, as the plan already says. +- **Both** need the same 8-core topology Track A needs, for the same reason. + +The Track A harness already covers the three surfaces ai-proxy leans on most +(`body_reader` streaming, TLS, and resolving a peer by hostname) as the +`stream`, `tls` and `dns` cases, so a regression in any of them is visible +without standing up APISIX at all. diff --git a/t/003-benchmark-runner.t b/t/003-benchmark-runner.t index 0069ac3..662fb84 100644 --- a/t/003-benchmark-runner.t +++ b/t/003-benchmark-runner.t @@ -19,6 +19,9 @@ sub slurp { my $makefile = File::Spec->catfile($root, 'Makefile'); my $runner = File::Spec->catfile($root, 'benchmark', 'run.sh'); +my $fold = File::Spec->catfile($root, 'benchmark', 'fold.sh'); +my $cases_file = File::Spec->catfile($root, 'benchmark', 'cases.txt'); +my $bench_lua = File::Spec->catfile($root, 'benchmark', 'lua', 'bench.lua'); my $target_template = File::Spec->catfile($root, 'benchmark', 'conf', 'target.nginx.conf'); my $upstream_template = File::Spec->catfile($root, 'benchmark', 'conf', 'upstream.nginx.conf'); my $wrk_headers = File::Spec->catfile($root, 'benchmark', 'wrk', 'headers.lua'); @@ -29,20 +32,75 @@ like(slurp($makefile), qr/^bench:\s*$/m, 'Makefile defines bench target'); like(slurp($makefile), qr{\./benchmark/run\.sh}, 'bench target delegates to benchmark/run.sh'); ok(-x $runner, 'benchmark/run.sh exists and is executable'); -like(slurp($runner), qr/BENCH_DRY_RUN/, 'runner supports dry-run mode'); -like(slurp($runner), qr/CPU_SATURATION_THRESHOLD/, 'runner checks CPU saturation'); -like(slurp($runner), qr/WRK2_BIN/, 'runner uses wrk2 binary configuration'); -like(slurp($runner), qr/WRK_RATE/, 'runner supports wrk2 target rate'); -like(slurp($runner), qr/WRK_CONNECTIONS="\$\{WRK_CONNECTIONS:-100\}"/, +ok(-x $fold, 'benchmark/fold.sh exists and is executable'); + +my $runner_src = slurp($runner); + +like($runner_src, qr/BENCH_DRY_RUN/, 'runner supports dry-run mode'); +like($runner_src, qr/CPU_SATURATION_THRESHOLD/, 'runner checks CPU saturation'); +like($runner_src, qr/WRK2_BIN/, 'runner uses wrk2 binary configuration'); +like($runner_src, qr/WRK_RATE/, 'runner supports wrk2 target rate'); +like($runner_src, qr/WRK_CONNECTIONS="\$\{WRK_CONNECTIONS:-100\}"/, 'runner defaults wrk2 connections to 100'); -like(slurp($runner), qr/KEEPALIVE_POOL_SIZE="\$\{KEEPALIVE_POOL_SIZE:-120\}"/, +like($runner_src, qr/KEEPALIVE_POOL_SIZE="\$\{KEEPALIVE_POOL_SIZE:-120\}"/, 'runner defaults keepalive pool size above wrk2 connections'); -like(slurp($runner), qr/\s-U\s/, 'runner enables wrk2 uncorrected latency output'); -like(slurp($runner), qr/\s-R\s+"\$WRK_RATE"/, 'runner passes wrk2 target rate'); -like(slurp($runner), qr/run_wrk_case "baseline" "\/bench\/baseline"/, - 'runner runs baseline case first'); -like(slurp($runner), qr/print_outbound_estimate/, - 'runner prints baseline-derived outbound estimate'); +like($runner_src, qr/\s-U\s/, 'runner requests wrk2 uncorrected latency output'); +like($runner_src, qr/-R "\$rate"/, 'runner passes a wrk2 target rate'); + +# the case table replaces the hardcoded case list +like($runner_src, qr/BENCH_CASES=/, 'runner drives cases from BENCH_CASES'); +like($runner_src, qr/BENCH_REPEATS="\$\{BENCH_REPEATS:-5\}"/, + 'runner defaults to five repeats'); +like($runner_src, qr/WRK_WARMUP/, 'runner runs a discarded warmup'); +like($runner_src, qr/RESULTS_CSV/, 'runner writes one CSV row per run'); +like($runner_src, qr/fairness_audit/, 'runner audits request fairness'); +like($runner_src, qr/latency_phase/, 'runner has a separate fixed-rate latency phase'); +like($runner_src, qr/throughput_phase/, 'runner has a throughput phase'); + +ok(-f $cases_file, 'benchmark/cases.txt exists'); +ok(-f $bench_lua, 'benchmark/lua/bench.lua exists'); + +# Every shape must exist on both sides, or the comparison is not a comparison. +my @cases = grep { length } map { + my $line = $_; + $line =~ s/#.*//; + $line =~ s/\s+//g; + $line; +} split /\n/, slurp($cases_file); + +ok(scalar(@cases) > 0, 'cases.txt lists cases'); + +my %by_shape; +for my $name (@cases) { + my ($driver, $shape) = split /\./, $name, 2; + $by_shape{$shape}{$driver} = 1; +} + +my @unpaired = grep { !($by_shape{$_}{ffi} && $by_shape{$_}{resty}) } + sort keys %by_shape; +is(scalar(@unpaired), 0, 'every shape has both an ffi and a resty case') + or diag('unpaired shapes: ' . join(', ', @unpaired)); + +# cases.txt is the source of truth the runner generates locations from; bench.lua +# must define the same shapes. The running worker is diffed against this file at +# smoke-check time too, but a static check fails faster. +my $bench_src = slurp($bench_lua); +my @declared = $bench_src =~ /pair_case\("([a-z0-9_]+)"/g; +my %declared = map { $_ => 1 } @declared; + +my @missing = grep { !$declared{$_} } sort keys %by_shape; +is(scalar(@missing), 0, 'bench.lua defines every shape listed in cases.txt') + or diag('missing from bench.lua: ' . join(', ', @missing)); + +my @extra = grep { !$by_shape{$_} } sort @declared; +is(scalar(@extra), 0, 'cases.txt lists every shape bench.lua defines') + or diag('missing from cases.txt: ' . join(', ', @extra)); + +# the axes the plan asks the matrix to cover +for my $shape (qw(oneshot stateful stream short post tls dns chunked trailers + hdr40 cookies req30)) { + ok($by_shape{$shape}, "matrix covers the $shape axis"); +} ok(-f $target_template, 'target nginx template exists'); ok(-f $upstream_template, 'upstream nginx template exists'); @@ -52,8 +110,15 @@ like(slurp($readme), qr/\| `WRK_CONNECTIONS` \| `100` \|/, like(slurp($readme), qr/\| `KEEPALIVE_POOL_SIZE` \| `120` \|/, 'README documents keepalive pool default'); +# A dedicated RUN_DIR, because prepare_configs starts by rm -rf'ing it. Sharing +# the default with a real run would let this test delete a live benchmark's +# prefix out from under it, taking the pid files with it so the runner could no +# longer even stop its own nginx. +my $dry_run_dir = File::Spec->catdir($root, 'benchmark', '.run-dryrun'); + my $cmd = join ' ', 'cd', quotemeta($root), '&&', + "RUN_DIR=" . quotemeta($dry_run_dir), 'BENCH_DRY_RUN=1', 'TARGET_CPU=0', 'UPSTREAM_CPUS=1', @@ -73,25 +138,46 @@ like($output, qr/UPSTREAM_CPUS=1/, 'dry-run prints upstream CPUs'); like($output, qr/WRK_CPUS=1/, 'dry-run prints wrk CPUs'); like($output, qr/KEEPALIVE_POOL_SIZE=120/, 'dry-run prints keepalive pool default'); like($output, qr/WRK_RATE=1000/, 'dry-run prints wrk2 target rate'); +like($output, qr/BENCH_REPEATS=5/, 'dry-run prints the repeat count'); my $rendered_target = File::Spec->catfile( - $root, 'benchmark', '.run', 'target', 'conf', 'nginx.conf' + $dry_run_dir, 'target', 'conf', 'nginx.conf' ); my $rendered_upstream = File::Spec->catfile( - $root, 'benchmark', '.run', 'upstream', 'conf', 'nginx.conf' + $dry_run_dir, 'upstream', 'conf', 'nginx.conf' +); +my $rendered_locations = File::Spec->catfile( + $dry_run_dir, 'target', 'conf', 'bench_locations.conf' ); ok(-f $rendered_target, 'dry-run renders target nginx config'); ok(-f $rendered_upstream, 'dry-run renders upstream nginx config'); +ok(-f $rendered_locations, 'dry-run generates the bench locations'); my $target_conf = slurp($rendered_target); like($target_conf, qr/location = \/bench\/baseline/, 'target config has baseline benchmark endpoint'); -like($target_conf, qr/location = \/bench\/ffi/, 'target config has C FFI benchmark endpoint'); -like($target_conf, qr/location = \/bench\/resty-http/, 'target config has resty.http benchmark endpoint'); +like($target_conf, qr/include bench_locations\.conf;/, 'target config includes the generated locations'); like($target_conf, qr/worker_processes 1;/, 'target config uses one worker'); +unlike($target_conf, qr/\{\{[A-Z_]+\}\}/, 'target config has no unrendered placeholders'); + +my $locations = slurp($rendered_locations); +for my $name (@cases) { + like($locations, qr/location = \/bench\/\Q$name\E \{/, + "generated locations include /bench/$name"); +} +like($locations, qr/location = \/echo\/ffi\.stateful \{/, + 'generated locations include the fairness-audit echo endpoints'); my $upstream_conf = slurp($rendered_upstream); like($upstream_conf, qr/Content-Length/, 'upstream config returns content-length response'); like($upstream_conf, qr/benchmark_response_body/, 'upstream config prebuilds response body'); +like($upstream_conf, qr/location = \/mock\/chunked/, 'upstream serves a chunked response'); +like($upstream_conf, qr/location = \/mock\/echo/, 'upstream echoes the raw request headers'); +like($upstream_conf, qr/location = \/mock\/sink/, 'upstream accepts a request body'); +like($upstream_conf, qr/location = \/mock\/headers40/, 'upstream serves a header-heavy response'); +like($upstream_conf, qr/location = \/mock\/cookies/, 'upstream serves repeated Set-Cookie headers'); +like($upstream_conf, qr/^stream \{/m, 'upstream runs a raw TCP mock for trailers'); +like($upstream_conf, qr/X-Checksum/, 'raw TCP mock sends a trailer'); +unlike($upstream_conf, qr/\{\{[A-Z_]+\}\}/, 'upstream config has no unrendered placeholders'); done_testing(); From 63771541c5a229da8a840ab6008ba3771a22fb71 Mon Sep 17 00:00:00 2001 From: Shreemaan Abhishek Date: Tue, 4 Aug 2026 11:06:23 +0000 Subject: [PATCH 2/2] docs: record the TLS re-measurement after the trust-store fix --- benchmark/results-full.md | 147 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/benchmark/results-full.md b/benchmark/results-full.md index e8e52b1..6438f15 100644 --- a/benchmark/results-full.md +++ b/benchmark/results-full.md @@ -197,3 +197,150 @@ That is a 42x gap at p99.9. It is also unstable across repeats: the three `resty.trailers` p99.9 samples were 109.570, 196.610 and 11.970 ms (p99: 43.520, 126.850 and 3.600 ms), while `ffi.trailers` produced 2.690, 2.600 and 2.560 ms. The median reported above is the middle sample in each case. + +--- + +# TLS re-measurement after the trust-store fix (`fix`) + +Run finished 2026-08-04, after the run recorded above. This section records a +targeted re-measurement of the TLS shapes following the fix in +`src/ngx_http_ffi_client_request.c` that stopped the client rescanning the CA +directory on every handshake. It supersedes the `tls` and `tlsshort` numbers in +the `full` run above; every other shape in this file is unaffected and still +stands. + +## What was run + +- Run parameters: `BENCH_RUN_ID=fix`, `BENCH_REPEATS=5`, + `BENCH_LATENCY_REPEATS=3`, `BENCH_CASES="ffi.tls resty.tls ffi.tlsshort + resty.tlsshort ffi.tlsverify resty.tlsverify ffi.tlsverifyshort + resty.tlsverifyshort"`. +- Same topology as the `full` run: target worker pinned to CPU 0, upstream on + CPUs 1-5, `wrk2` on CPUs 6-7. +- The run exited 0. The fairness audit passed for all four shapes. All 40 + throughput windows passed the saturation gate with zero non-2xx and zero + socket errors; nothing was discarded. +- `requests_per_connection` is `1.0` for both clients on all four short-lived + cases, so those windows really are measuring a fresh connection and handshake + per request rather than a partly-pooled result. +- Raw rows: `benchmark/.run-fix/results/fix.csv`, 64 data rows. + +## The defect + +`ngx_http_ffi_client_ssl_ctx()` called `SSL_CTX_set_default_verify_paths()` +unconditionally. That registers the hashed CA directory as +`X509_FILETYPE_DEFAULT`, and under `DEFAULT` OpenSSL enumerates the entire +directory on *every* handshake instead of opening the single `.N` file the +lookup needs. + +Found with `perf`, sampling the pinned target worker for 40 s during each case +(~40000 samples each). Every top entry in the `perf diff` is kernel +directory-reading code, present only on the FFI side: + +``` +Baseline Delta Abs Symbol + +2.04% [k] filldir64 + +2.01% [k] ext4_htree_store_dirent + +1.82% [k] str2hashbuf_signed + +1.76% [k] half_md4_transform + +1.05% [k] htree_dirblock_to_tree +``` + +| | C FFI | `lua-resty-http` | +| --- | ---: | ---: | +| CPU in directory-lookup code | `11.83%` | `0.42%` | +| CPU in the kernel overall | `36.7%` | `17.9%` | + +Confirmed at the syscall level under load: 2132 `openat` plus 4264 `getdents64` +for ~2168 requests, one full sweep of a 245-entry directory per handshake. The +same trace during `resty.tlsshort` caught none. After the fix, tracing both +`ffi.tlsshort` and `ffi.tlsverifyshort` caught none either. + +The cost is the filetype, not hash-directory lookup and not lookup failure. +Reproduced independently of this module with `openssl s_client` against the +benchmark upstream: + +| trust store setup | `getdents64` calls | verify result | +| --- | ---: | --- | +| `set_default_verify_paths()` | `4` | 18 (self-signed) | +| explicit `-CAfile` system bundle | `0` | 18 | +| explicit `-CApath` system dir | `0` | 18 | +| explicit `-CApath`, issuer present | `0` | 0 (ok) | + +## Throughput (median of 5 saturated repeats) + +| shape | C FFI | `lua-resty-http` | ratio | +| --- | ---: | ---: | ---: | +| `tls` | `18276.36` req/s | `11926.11` req/s | `1.53x` | +| `tlsshort` | `1256.86` req/s | `1142.44` req/s | `1.10x` | +| `tlsverify` | `17542.73` req/s | `11861.35` req/s | `1.48x` | +| `tlsverifyshort` | `1228.67` req/s | `1136.91` req/s | `1.08x` | + +Against the `full` run above: + +| shape | before | after | ratio before | ratio after | +| --- | ---: | ---: | ---: | ---: | +| `ffi.tlsshort` | `932.16` req/s | `1256.86` req/s | `0.81x` | `1.10x` | +| `ffi.tls` | `18655.46` req/s | `18276.36` req/s | `1.57x` | `1.53x` | + +`tlsshort` gains 35% and turns the only losing shape in the matrix into a win. +Pooled `tls` is unchanged within run-to-run spread, as expected: it amortises +one handshake over many requests, so the sweep was never a large share of its +cost. + +`tlsverify` and `tlsverifyshort` are new. The matrix had never exercised +`ssl_verify = true` — all four call sites in `benchmark/lua/bench.lua` hardcoded +`false` — so the verifying path was previously unmeasured. Verification costs +the FFI client about 4% on pooled TLS (`18276` to `17543`) and roughly 2% on the +short-lived shape, which is real chain-checking work rather than the pathological +sweep. + +## Latency (fixed rate, corrected for coordinated omission) + +Median of 3 repeats. **These numbers are less trustworthy than the throughput +figures above and should not be quoted for the short-lived shapes** — see the +caveat below. + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.tls` | `1.650` ms | `4.130` ms | `4.880` ms | +| `resty.tls` | `1.260` ms | `2.700` ms | `3.270` ms | +| `ffi.tlsshort` | `3.500` ms | `5.810` ms | `7.710` ms | +| `resty.tlsshort` | `3.620` ms | `6.280` ms | `8.330` ms | +| `ffi.tlsverify` | `1.170` ms | `2.460` ms | `2.930` ms | +| `resty.tlsverify` | `2.330` ms | `5.460` ms | `6.420` ms | +| `ffi.tlsverifyshort` | `23.220` ms | `46.720` ms | `48.380` ms | +| `resty.tlsverifyshort` | `3.430` ms | `5.660` ms | `7.000` ms | + +### Caveat: rate attainment, not latency + +The latency phase drives each pair at a fixed rate — 60% of the slower side's +saturated throughput. Several windows failed to reach that rate by roughly 2%, +and wrk2's coordinated-omission correction converts a small rate shortfall into +a large apparent latency, so those rows measure rate attainment rather than +service time: + +| case | repeat | achieved / target req/s | p50 | worker CPU | +| --- | ---: | ---: | ---: | ---: | +| `ffi.tlsshort` | 1 | `683.85` / 685 | `3.500` ms | `57.0%` | +| `ffi.tlsshort` | 2 | `672.02` / 685 | `24.400` ms | `55.8%` | +| `ffi.tlsshort` | 3 | `683.89` / 685 | `3.490` ms | `57.5%` | +| `ffi.tlsverifyshort` | 1 | `669.37` / 682 | `23.220` ms | `56.0%` | +| `ffi.tlsverifyshort` | 2 | `669.38` / 682 | `23.500` ms | `56.4%` | +| `ffi.tlsverifyshort` | 3 | `680.99` / 682 | `3.300` ms | `58.0%` | + +Every window that hit its target rate produced a ~3.3-3.5 ms p50; every window +that missed produced ~23-24 ms. The worker had roughly 42% CPU headroom +throughout, so this is not a CPU limit. + +This is not specific to the FFI client, though it hit it more often here. +`resty.tlsverify` missed the rate on repeats 2 and 3 (`6982` against a `7088` +target) and its p50 went from `1.270` ms to `2.330`/`2.410` ms; `ffi.tls` +repeats 1 and 2 missed (`7020` against `7127`) with the same effect. Where both +sides hit their target — `tlsverify` repeat 1, `tlsverifyshort` repeat 3 — the +FFI client is faster at every percentile. + +The underlying cause is not yet understood. Candidates: connection-teardown +asymmetry over which side holds `TIME_WAIT`, ephemeral port pressure at ~683 new +connections per second, or a stall in the connect path. Until it is diagnosed, +no latency claim should be published for the short-lived TLS shapes.