diff --git a/.gitignore b/.gitignore index a9e5ca0..0ef3281 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ docs/superpowers/ -benchmark/.run/ +benchmark/.run*/ t/servroot/ objs/ diff --git a/README.md b/README.md index 6cd5b6e..67abf9a 100644 --- a/README.md +++ b/README.md @@ -201,20 +201,52 @@ protocol work is intentionally deferred. ## Benchmark The local benchmark pins the target OpenResty worker to one CPU core and -compares three request paths: a no-upstream baseline, the C FFI client, and -`resty.http`. The latest 15-second local runs used wrk2 with a saturated target -worker. - -| wrk2 connections | no-upstream baseline | C FFI client | `resty.http` | C FFI / `resty.http` QPS | outbound cost ratio (`resty.http` / C FFI) | -| --- | ---: | ---: | ---: | ---: | ---: | -| `10` | `83998.04` QPS | `32831.68` QPS | `16123.19` QPS | `2.04x` | `2.70x` | -| `100` | `96155.73` QPS | `38582.72` QPS | `14665.62` QPS | `2.63x` | `3.72x` | - -The C FFI path delivers substantially higher end-to-end throughput than -`resty.http`, and the baseline-derived estimate shows `resty.http` spending -about `2.70x` to `3.72x` as much outbound client CPU time as the C FFI path in -these local runs. See -[benchmark/README.md](benchmark/README.md) for the benchmark topology, -reproduction notes, and detailed measurements. +compares a no-upstream baseline against both clients across a matrix of request +shapes: one-shot and stateful, buffered and streaming, plaintext and TLS, IP +literal and hostname, content-length and chunked framing. + +Median of 5 repeats, 30s measured each, target worker saturated on every row, +no non-2xx responses, `llhttp` parser, 1KB response body, 100 wrk2 connections: + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| one-shot (`request_uri`) | `30154` QPS | `14785` QPS | `2.04x` | `2.64x` | +| stateful object | `27316` QPS | `15382` QPS | `1.78x` | `2.16x` | + +Baseline without an upstream call: `82185` QPS. The outbound cost ratio +subtracts that baseline from both client paths and compares what is left, which +is the closest this gets to isolating the client itself. + +Per-request Lua heap churn, which is the direct test of the claim that response +handling stays in C: + +| shape | C FFI | `resty.http` | +| --- | ---: | ---: | +| one-shot | `2.49` KB | `7.45` KB | +| stateful | `3.33` KB | `6.49` KB | + +**These supersede the previously published `2.04x`-`2.63x` QPS and +`2.70x`-`3.72x` outbound figures, which were withdrawn.** Those compared the C +one-shot fast path against a stateful `lua-resty-http` object, so the ratio +contained the API-shape difference as well as the C-versus-Lua difference and +credited all of it to the latter. They also predate per-request token +validation, the case-insensitive header table with repeated headers folded into +arrays, `llhttp` as the default parser, interim `1xx` consumption, and trailer +parsing, all of which add hot-path work. Matching the shapes and re-measuring +the current library moves the honest one-shot figure from `2.63x` to `2.04x`. + +Across the full case matrix the ratio holds between `1.68x` and `1.96x` for +streaming, chunked framing, hostname resolution, POST bodies, and header-heavy +responses, and reaches `4.41x` on responses with trailers. + +**Where this client loses.** The advantage is in per-request work, so it shrinks +as connection setup takes over the request and eventually reverses: `1.49x` on +pooled TLS, `1.19x` on short-lived plaintext connections, and `0.79x` on a fresh +TLS handshake per request, where `lua-resty-http` is about 21% faster. Anything +that cannot hold a keepalive pool, TLS to many short-lived peers most of all, is +a case for the other client today. + +See [benchmark/README.md](benchmark/README.md) for the topology, the full case +matrix, the fairness audit, and how to reproduce. Design notes and implementation plans used during development live under `docs/superpowers/` and are intentionally ignored by git. diff --git a/benchmark/README.md b/benchmark/README.md index fc2674a..0d43dde 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,138 +1,416 @@ # Benchmark -This directory contains the local benchmark for comparing -`resty.ngx_http_ffi_client` with the Lua cosocket baseline implemented with -`lua-resty-http` / `resty.http`. +This directory holds the local benchmark comparing +`resty.ngx_http_ffi_client` with `lua-resty-http` / `resty.http`. The benchmark is scoped to the current HTTP/1.1 client work. It validates the C/Nginx module path and the llhttp-based HTTP/1 direction; it does not evaluate HTTP/2, HTTP/3, or alternative protocol APIs. -The benchmark measures maximum QPS when the target OpenResty worker is pinned to -one CPU core and that core is saturated. It is not a fixed-rate latency test. +Two things are measured, in two separate passes: -The runner measures a baseline first. The baseline endpoint does not contact the -upstream; it only returns a response body with the same size as the upstream -response. The C FFI and `resty.http` cases are then compared against this -baseline to estimate the outbound HTTP client cost. +- **Throughput**: maximum QPS with the target OpenResty worker pinned to one + core and that core saturated. +- **Latency**: p50/p99/p99.9 at a fixed rate below saturation. Latency measured + at saturation is queue depth, not client cost, so it gets its own pass. -## Current Result Summary +A baseline endpoint that returns the same response body without contacting the +upstream runs alongside both, so the downstream leg can be subtracted out. -The latest local runs compare three paths: +## Results -- no-upstream baseline: target OpenResty returns a fixed response body directly -- C FFI client: target OpenResty sends one upstream HTTP request with - `resty.ngx_http_ffi_client` -- `resty.http`: target OpenResty sends one upstream HTTP request with - `lua-resty-http` +Target worker pinned to CPU `0`, upstream on `1-5` with 5 workers, wrk2 on +`6-7`. Median of 5 repeats, 10s discarded warmup and 30s measured per repeat, +`llhttp` parser, `RESPONSE_SIZE=1024`, `WRK_CONNECTIONS=100`. Every row below +was saturated (target worker CPU >= 95%) and reported no non-2xx responses; no +rows were discarded. -Both load points used `WRK_DURATION=15s`, `WRK_RATE=100000`, -`WRK_THREADS=2`, `KEEPALIVE_POOL_SIZE=120`, `KEEPALIVE_IDLE_TIMEOUT=60000`, -and `RESPONSE_SIZE=1024`. The target worker was pinned to CPU `0`, the -upstream workers used CPUs `1-5`, wrk2 used CPUs `6-7`, and the upstream used -`UPSTREAM_WORKERS=5`. +### Headline -| wrk2 connections | no-upstream baseline | C FFI client | `resty.http` | C FFI / `resty.http` QPS | outbound cost ratio (`resty.http` / C FFI) | -| --- | ---: | ---: | ---: | ---: | ---: | -| `10` | `83998.04` QPS | `32831.68` QPS | `16123.19` QPS | `2.04x` | `2.70x` | -| `100` | `96155.73` QPS | `38582.72` QPS | `14665.62` QPS | `2.63x` | `3.72x` | +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `oneshot` | `30154.14` | `14785.18` | `2.04x` | `2.64x` | +| `stateful` | `27316.07` | `15381.85` | `1.78x` | `2.16x` | -The QPS ratio is the direct end-to-end throughput comparison between the C FFI -client and `resty.http`. The outbound cost ratio subtracts the no-upstream -baseline from both client paths and compares the estimated upstream HTTP client -CPU cost as `resty.http / C FFI`. In these runs, the C FFI client is about -`2.04x` to `2.63x` faster end-to-end, and `resty.http` spends about `2.70x` to -`3.72x` as much estimated outbound client CPU time. +Baseline (no upstream call): `82184.57` QPS, `12.168` us/request. -## Detailed Local Measurements +### Detail -All target-worker CPU averages are above the default `95%` saturation threshold, -and no `Non-2xx` responses were reported in these runs. - -| wrk2 connections | case | QPS | target worker CPU avg | saturation | -| --- | --- | ---: | ---: | --- | -| `10` | no-upstream baseline | `83998.04` | `108.6%` | valid | -| `10` | C FFI client | `32831.68` | `109.1%` | valid | -| `10` | `resty.http` | `16123.19` | `105.6%` | valid | -| `100` | no-upstream baseline | `96155.73` | `103.9%` | valid | -| `100` | C FFI client | `38582.72` | `110.1%` | valid | -| `100` | `resty.http` | `14665.62` | `106.4%` | valid | - -The baseline-derived per-request estimates are: - -| wrk2 connections | baseline downstream + handler | C FFI total | C FFI estimated outbound | `resty.http` total | `resty.http` estimated outbound | outbound cost ratio | +| case | median QPS | min | max | repeats used | worker CPU avg | worker RSS | | --- | ---: | ---: | ---: | ---: | ---: | ---: | -| `10` | `11.905 us` | `30.458 us` | `18.553 us` | `62.022 us` | `50.117 us` | `2.70x` | -| `100` | `10.400 us` | `25.918 us` | `15.519 us` | `68.187 us` | `57.787 us` | `3.72x` | - -The main project advantage shown by this benchmark is that the C FFI client -keeps outbound HTTP request handling in Nginx C module code while preserving -OpenResty worker nonblocking behavior and keepalive reuse. Under a saturated -single target worker, this reduces Lua cosocket overhead on the upstream client -path and leaves more of the worker CPU budget for application logic. +| `baseline` | `82184.57` | `81513.32` | `85599.33` | 5 | `109.3%` | `37300` KB | +| `ffi.oneshot` | `30154.14` | `29789.00` | `30489.65` | 5 | `116.7%` | `37300` KB | +| `ffi.stateful` | `27316.07` | `27143.61` | `27724.92` | 5 | `115.9%` | `37300` KB | +| `resty.oneshot` | `14785.18` | `13465.86` | `14855.69` | 5 | `113.5%` | `37300` KB | +| `resty.stateful` | `15381.85` | `15275.92` | `15617.69` | 5 | `114.3%` | `37332` KB | + +### Latency, fixed rate at 60% of the slower path + +| case | rate | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | ---: | +| `ffi.oneshot` | `8871` req/s | `1.060` ms | `2.450` ms | `5.430` ms | +| `resty.oneshot` | `8871` req/s | `1.370` ms | `3.920` ms | `10.550` ms | +| `ffi.stateful` | `9229` req/s | `1.160` ms | `3.360` ms | `20.940` ms | +| `resty.stateful` | `9229` req/s | `1.400` ms | `3.740` ms | `6.800` ms | + +At those fixed rates the worker sat at `45.6%` CPU on `ffi.oneshot` against +`72.4%` on `resty.oneshot`, which is the same comparison measured from the +other side. + +### Per-request Lua heap delta + +| case | KB/request | +| --- | ---: | +| `ffi.oneshot` | `2.494` | +| `ffi.stateful` | `3.331` | +| `resty.stateful` | `6.492` | +| `resty.oneshot` | `7.448` | + +### Matrix + +The full case matrix, median of 2 repeats (`BENCH_CASES=all`, same host and +topology, `matrix.csv`). Every row was saturated with no non-2xx responses and +nothing was discarded; per-case spread was under 1% on most cases. + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `trailers` | `27919.64` | `6325.35` | `4.41x` | `6.10x` | +| `oneshot` | `28052.14` | `14314.11` | `1.96x` | `2.44x` | +| `hdr40` | `17337.44` | `9235.26` | `1.88x` | `2.10x` | +| `post` | `24121.31` | `12826.85` | `1.88x` | `2.23x` | +| `req30` | `23370.36` | `13014.28` | `1.80x` | `2.10x` | +| `cookies` | `24343.96` | `13719.88` | `1.77x` | `2.09x` | +| `chunked` | `26844.53` | `15521.60` | `1.73x` | `2.07x` | +| `dns` | `26937.76` | `15538.12` | `1.73x` | `2.08x` | +| `stateful` | `26365.78` | `15289.51` | `1.72x` | `2.05x` | +| `stream` | `26490.00` | `15686.39` | `1.69x` | `2.00x` | +| `readbody` | `26051.07` | `15462.52` | `1.68x` | `1.99x` | +| `tls` | `17732.35` | `11861.95` | `1.49x` | `1.63x` | +| `short` | `8275.94` | `6973.64` | `1.19x` | `1.21x` | +| `tlsshort` | `872.96` | `1108.38` | `0.79x` | `0.79x` | + +Baseline: `84504.18` QPS. + +### Where the FFI client does not win + +**`tlsshort`, a fresh TLS handshake per request: `0.79x`.** `872.96` QPS against +`resty.http`'s `1108.38`, so this client is about 21% slower. The bottom of the +table reads as one story rather than three: `tls` (`1.49x`), `short` (`1.19x`) +and `tlsshort` (`0.79x`) are exactly the cases where connection setup dominates +the request. The advantage this client has is in per-request work, and it +shrinks as setup takes over, then reverses once setup is nearly all of it. +Anything that cannot hold a keepalive pool, and TLS to many short-lived peers +most of all, is a case where `lua-resty-http` is currently the better choice. + +**`ffi.stateful` tail latency.** `p99.9` of `20.940` ms against +`resty.stateful`'s `6.800` ms, a roughly 3x worse tail, even though `p50` and +`p99` are both better. `ffi.oneshot` shows no such tail at `5.430` ms, so +something on the stateful object's path stalls occasionally in a way the +one-shot path does not. + +This one is **not yet evidence**: the latency phase ran a single 30s window per +case, and tail percentiles are the noisiest thing measured here. Re-run with +`BENCH_LATENCY_REPEATS=3` before acting on it, and attribute with `perf` (A7) if +it survives. + +### Reading the matrix numbers + +Two caveats on the matrix table specifically, neither of which applies to the +five-case headline run above: + +- **Two repeats, not five.** Enough for the ranking, which is stable, and the + min/max spread is under 1% on most cases. Not enough to argue about a + difference of a few percent between adjacent rows. +- **Worker RSS is cumulative in an `all` run and should be ignored there.** It + climbs from about `527` MB on the early cases to `1.5` GB on the late ones, + which is 28 separate keepalive pools of up to `120` connections each + accumulating over the run, not per-case memory. The headline run, with five + cases and four pools, held steady at about `37` MB. Read RSS from a run that + measures the cases you care about, not from the sweep. + +## What the previous numbers said, and why they were withdrawn + +The tables that used to sit here reported the C FFI client as `2.04x`-`2.63x` +faster end-to-end and spending `2.70x`-`3.72x` less outbound CPU. Measured +properly against the current library, the matched one-shot pair is `2.04x` on +throughput and `2.64x` on outbound cost: the same ballpark, but the old `2.63x` +headline was not comparing what it claimed to. Two independent problems: + +**The two cases compared different API shapes.** `/bench/ffi` called +`client.request_uri`, the one-shot C fast path that resumes the coroutine once. +`/bench/resty-http` drove a stateful object: `new` → `connect` → `request` → +`read_body` → `set_keepalive`. When that config was written this library only +had `request_uri`; `_M.new()` now exists with the same shape as +`lua-resty-http`. The old ratio therefore contained the API-shape difference and +attributed all of it to C versus Lua. + +**They predate most of the current hot path.** Landed since those runs: +per-request token validation of the method and header names with +control-character checks on values, the case-insensitive header metatable with +repeated headers folded into arrays, `llhttp` as the default parser, interim +`1xx` consumption, and trailer parsing. Each adds per-request work. + +The harness below fixes both, and the Results section above is what it produces. + +## Topology requirement + +The measurement assumes three things that a small host cannot provide at once: a +target worker that is the only thing on its core, an upstream that cannot become +the bottleneck, and a load generator competing with neither. + +| role | CPUs | why | +| --- | --- | --- | +| target OpenResty worker | 1, exclusive | this is the thing being measured | +| upstream mock | 5 | must never be the bottleneck | +| wrk2 | 2 | must be able to saturate the target | + +That means **8 or more dedicated cores**. `run.sh` warns when fewer are online +and still runs, because the saturation gate will mark the rows invalid anyway. +Below eight cores this is a regression smoke test, not a source of headline +numbers. GitHub Actions runners (4 shared vCPUs) fall in that category. + +`run.sh` requires `taskset` and `/proc`, so it does not run on macOS. + +## The comparison is a 2x2 + +The headline ratio comes from reading one shape across both columns: + +| | one-shot | stateful | +| -------------- | --------------- | ------------------------------------------------------- | +| C FFI | `ffi.oneshot` | `ffi.stateful` | +| `resty.http` | `resty.oneshot` | `resty.stateful` | + +`ffi.oneshot` and `resty.oneshot` both call `request_uri`. `ffi.stateful` and +`resty.stateful` both drive `new`/`connect`/`request`/read/`set_keepalive`. The +gap between `ffi.oneshot` and `ffi.stateful` separately prices the C fast path +against this library's own Lua object layer, which is a different question from +the C-versus-Lua one and should not be mixed into it. + +## Case matrix + +Cases are named `.` and listed in +[`cases.txt`](cases.txt), which is the single source of truth: `run.sh` +generates one nginx location per name from it, and the smoke check diffs it +against the case table the worker actually loaded, so the two cannot drift. + +| shape | axis it prices | +| --- | --- | +| `oneshot`, `stateful` | API shape; the headline pair | +| `readbody` | buffering through the Lua chunk loop instead of the C preread fast path | +| `stream` | `res.body_reader`, chunk by chunk | +| `short` | connection setup, with no keepalive pool | +| `post` | a 4KB request body | +| `tls` | TLS with a pooled connection | +| `tlsshort` | a fresh TLS handshake per request | +| `dns` | resolving the peer by hostname instead of an IP literal | +| `chunked` | chunked response framing | +| `trailers` | chunked framing with a trailer section | +| `hdr40` | 40 response headers | +| `cookies` | ten repeated `Set-Cookie` headers, which fold into an array | +| `req30` | 30 request headers, which prices per-request header validation | + +Axes that are not cases because they are swept by environment instead: + +- **Body size** via `RESPONSE_SIZE` (the plan sweeps 0, 1KB, 16KB, 256KB, 1MB). +- **Concurrency** via `WRK_CONNECTIONS` (1, 10, 50, 100, 500). +- **Parser backend** by building the module twice, with + `NGX_HTTP_FFI_CLIENT_USE_LLHTTP=1` and `=0`, and setting `BENCH_PARSER` so the + rows are labelled. + +The full cross-product is too large to run. Sweep body size against concurrency +on the plaintext keepalive GET, then run one point per remaining axis against +that same reference configuration. + +Not every case works on every build. The hand-written parser has no chunked +framing, so `ffi.chunked` and `ffi.trailers` fail their smoke check there; they +are excluded from the run, named in the summary, and the rest proceeds. +`BENCH_STRICT=1` aborts instead. + +## Fairness audit + +An unfair configuration makes every number worthless, so the runner proves +fairness before applying any load rather than asserting it in prose. + +Both columns of a pair are generated from one shape table in +[`lua/bench.lua`](lua/bench.lua), so identical pool size, idle timeout, +timeouts, request headers and body consumption are structural rather than +reviewed. On top of that, the upstream exposes `/mock/echo`, which replies with +the raw request header block it received. The runner fetches it through both +clients and diffs: + +- **The request line and every caller-supplied header must match.** A difference + means the pair is not measuring the same request, and the audit fails. +- **Headers a client generates for itself are reported, not failed.** They are + the client's own contract. + +That second category is not empty. The FFI client emits two headers on every +request that `lua-resty-http` does not: -## Dependencies +``` +Connection: keep-alive +Content-Length: 0 +``` -- OpenResty or Nginx binary built with `ngx_http_ffi_client_module` -- `lua-resty-http` available in `TEST_NGINX_LUA_PACKAGE_PATH` -- `wrk2` -- `curl` -- `taskset` from util-linux -- Linux `/proc` filesystem for CPU sampling +`lua-resty-http` omits both on a bodyless GET; it only sets `Content-Length` for +methods that expect a body. The C path always writes `Content-Length` from +`req->body.len`. This costs the FFI client roughly 45 extra bytes per request to +format and send, so it works against the FFI side of the ratio rather than +inflating it, but it is a real difference in the request being measured and a +`Content-Length: 0` on a GET is the kind of thing a strict peer can reject. It +is worth deciding deliberately whether the C path should keep emitting it. + +Connection reuse is verified from the upstream side rather than the target's, +which keeps the check off the measured worker entirely. The upstream counts +connections and requests in a shared dict; the counters are reset after the +warmup, so they describe the steady state. **Zero new connections against a +non-zero request count is perfect pool reuse** and is what the keepalive cases +should report. A ratio near one request per connection means the pool is not +reusing anything, and the runner warns. + +Any row reporting non-2xx responses is invalid, and so is any throughput row +where the target worker averaged below `CPU_SATURATION_THRESHOLD`. Both are +excluded from every median and counted in the summary. + +## Metrics + +| metric | how | +| --- | --- | +| throughput | saturated QPS, gated on >=95% target-worker CPU | +| latency | p50/p99/p99.9 from a separate fixed-rate pass at 60% of the slower path's median saturated QPS | +| outbound cost | baseline-subtracted `1/QPS`, compared as `resty.http / C FFI` | +| worker RSS | `VmRSS` of the pinned worker at the end of the measured window | +| Lua heap per request | `collectgarbage("count")` delta over 20 requests, via `/gcdelta/` | + +The Lua heap delta is reported rather than left as a diagnostic. The library's +claim is that response handling stays in C, and the header arrays plus the +case-insensitive metatable are exactly where that could regress. + +wrk2 is run with `-L -U`, which emits both a coordinated-omission-corrected +distribution and an uncorrected one. The corrected figures are reported by +default, since correcting for delayed starts is the reason to run wrk2 at a +fixed rate at all. `BENCH_LATENCY_MODE=uncorrected` selects the other block. + +## Protocol + +Per data point: `BENCH_REPEATS` repeats (default 5), each a discarded +`WRK_WARMUP` warmup followed by a measured `WRK_DURATION` window. Cases rotate +within each repeat rather than running back to back, so drift and background +load land on every case equally. Results are reported as the median with the +min/max spread. ## Command ```bash export TEST_NGINX_BINARY=/path/to/openresty/nginx/sbin/nginx -export TEST_NGINX_LUA_PACKAGE_PATH="$PWD/lib/?.lua;$PWD/lib/?/init.lua;/path/to/lua-resty-http/lib/?.lua;;" +export TEST_NGINX_LUA_PACKAGE_PATH="$PWD/lib/?.lua;$PWD/lib/?/init.lua;/path/to/lua-resty-http/lib/?.lua" +export WRK2_BIN=/path/to/wrk2/wrk + +# the headline 2x2 make bench + +# the whole matrix +BENCH_CASES=all make bench + +# one axis point +BENCH_CASES="baseline ffi.stream resty.stream" RESPONSE_SIZE=262144 make bench ``` -## Topology +The module has to be compiled into the nginx binary: -- upstream instance: mock HTTP server, multiple workers, non-target CPU cores -- target instance: one OpenResty worker, pinned to `TARGET_CPU` -- wrk2: load generator, pinned to `WRK_CPUS` +```bash +NGX_HTTP_FFI_CLIENT_USE_LLHTTP=1 \ +NGX_HTTP_LUA_MODULE_DIR=/path/to/openresty/bundle/ngx_lua-VERSION \ + ./configure --prefix=/tmp/openresty-llhttp --with-http_ssl_module \ + --with-pcre-jit --add-module=/path/to/ngx_http_ffi_client +``` + +## Results + +Every run writes one CSV row per wrk2 invocation to +`benchmark/.run/results/.csv`, with the run parameters on each row so +rows from different runs can be concatenated: + +``` +run_id,phase,repeat,case,qps,rate,non2xx,socket_errors,p50_ms,p99_ms,p999_ms, +cpu_avg,cpu_max,cpu_samples,saturated,rss_kb,upstream_connections, +upstream_requests,requests_per_connection,response_size,connections,threads, +duration,parser +``` -`wrk2` sends requests to the target instance. The target instance sends one -outbound HTTP/1.1 request to the upstream per inbound request. +[`fold.sh`](fold.sh) folds those rows into the markdown tables this file +carries. `run.sh` calls it at the end of a run; it also runs standalone: -The benchmark cases run in this order: +```bash +benchmark/fold.sh benchmark/.run/results/.csv +``` + +The tables it emits are the headline ratio per shape, a per-case detail table +with medians and spread, the latency table, and a list of everything discarded. +Paste them in rather than hand-maintaining them, which is what the previous +tables were. + +## Dependencies + +- OpenResty or Nginx built with `ngx_http_ffi_client_module` +- `lua-resty-http` on `TEST_NGINX_LUA_PACKAGE_PATH` +- `wrk2` +- `curl`, `openssl`, `taskset` from util-linux +- Linux `/proc` for CPU and RSS sampling + +## Topology detail + +- upstream instance: mock HTTP server on multiple workers, non-target cores, + plus a TLS listener and a raw TCP `stream` mock for the trailer case, since + trailers cannot be emitted from the http subsystem +- target instance: one OpenResty worker, pinned to `TARGET_CPU` +- wrk2: load generator, pinned to `WRK_CPUS` -1. `/bench/baseline`: target worker returns a fixed body without upstream IO. -2. `/bench/ffi`: target worker calls `resty.ngx_http_ffi_client`. -3. `/bench/resty-http`: target worker calls `resty.http`. +Both instances raise `keepalive_requests` far above the default. At tens of +thousands of requests per second the stock limit would recycle connections +hundreds of times per run: on the downstream leg that charges accept cost to +whichever case is running, and on the upstream leg it would force the client +pools to reconnect and stop the keepalive cases from measuring keepalive. ## Configuration | Variable | Default | Meaning | | --- | --- | --- | +| `BENCH_CASES` | the 2x2 plus baseline | space-separated case names, or `all` | +| `BENCH_REPEATS` | `5` | repeats per data point | +| `BENCH_LATENCY` | `1` | run the fixed-rate latency phase | +| `BENCH_LATENCY_FRACTION` | `0.6` | fraction of the slower path's QPS for that phase | +| `BENCH_LATENCY_MODE` | `corrected` | `corrected` or `uncorrected` percentiles | +| `BENCH_GC` | `1` | report the per-request Lua heap delta | +| `BENCH_AUDIT` | `1` | run the fairness audit before load | +| `BENCH_STRICT` | `0` | abort instead of excluding a case that fails its smoke check | +| `BENCH_PARSER` | `llhttp` | label recorded in the CSV rows | +| `BENCH_RESULTS_DIR` | `.run/results` | where CSV rows are written | | `TARGET_CPU` | `0` | CPU core used by the target OpenResty worker | | `UPSTREAM_CPUS` | non-target cores | CPU cores used by the upstream instance | | `WRK_CPUS` | `UPSTREAM_CPUS` | CPU cores used by wrk2 | | `TARGET_PORT` | `19840` | target listen port | | `UPSTREAM_PORT` | `19841` | upstream listen port | +| `UPSTREAM_TLS_PORT` | `19842` | upstream TLS listen port | +| `UPSTREAM_TRAILER_PORT` | `19843` | raw TCP trailer mock port | | `UPSTREAM_WORKERS` | count of `UPSTREAM_CPUS` | upstream worker count | +| `DNS_HOST` | `localhost` | hostname used by the `dns` cases | +| `BENCH_RESOLVER` | first `/etc/resolv.conf` nameserver | resolver for those cases | +| `TLS_SERVER_NAME` | `bench.local` | SNI both clients send for the TLS cases | | `RESPONSE_SIZE` | `1024` | upstream response body size in bytes | | `KEEPALIVE_POOL_SIZE` | `120` | keepalive pool size for both client paths | | `KEEPALIVE_IDLE_TIMEOUT` | `60000` | keepalive idle timeout in milliseconds | +| `CLIENT_TIMEOUT` | `60000` | connect/send/read timeout for both clients | | `WRK_THREADS` | `2` | wrk2 thread count | | `WRK_CONNECTIONS` | `100` | wrk2 connection count | -| `WRK_DURATION` | `30s` | official benchmark duration | -| `WRK_RATE` | `100000` | wrk2 target request rate, set high enough to saturate the target worker | -| `WRK2_BIN` | `wrk` | wrk2 executable path | -| `CPU_SATURATION_THRESHOLD` | `95` | minimum average target worker CPU for a valid run | +| `WRK_WARMUP` | `10s` | discarded warmup before each measured window | +| `WRK_DURATION` | `30s` | measured window | +| `WRK_RATE` | `100000` | wrk2 target rate, set above the expected max QPS | +| `CPU_SATURATION_THRESHOLD` | `95` | minimum average target worker CPU for a valid throughput row | -The runner invokes wrk2 with `-R "$WRK_RATE" -L -U`. The `-L` output reports -wrk2's latency distribution and `-U` reports the uncorrected latency -distribution. For this benchmark, set `WRK_RATE` above the expected maximum QPS -so the single target worker remains CPU-bound. +## Baseline-derived estimate -## Baseline-Derived Estimate - -The QPS results include downstream client-to-OpenResty overhead. To estimate the -outbound HTTP client cost, the runner treats single-core CPU time per request as -approximately `1 / QPS`: +QPS includes the downstream leg. To estimate outbound client cost, the runner +treats single-core CPU time per request as approximately `1 / QPS`: ```text baseline_us = 1_000_000 / baseline_qps @@ -141,44 +419,18 @@ resty_outbound_us = 1_000_000 / resty_http_qps - baseline_us ratio = resty_outbound_us / ffi_outbound_us ``` -This is still an estimate, but it removes the common downstream and simple -response path from the comparison. - -## Valid Runs - -A result is valid only when the target worker CPU average is at least -`CPU_SATURATION_THRESHOLD`. - -If the result is marked suspicious: - -- increase `WRK_THREADS` -- increase `WRK_CONNECTIONS` -- increase `WRK_RATE` -- move wrk2 to non-target CPU cores -- increase `UPSTREAM_WORKERS` -- assign more non-target cores to `UPSTREAM_CPUS` +Every case, the baseline included, writes the same fixed downstream body, so +that subtraction removes an identical cost from each. It is still an estimate. -Do not compare C FFI and `resty.http` QPS from runs where the target worker was -not saturated. - -## Manual RSS and GC Checks - -The benchmark output focuses on QPS, latency, and target worker CPU saturation. -Use these commands for supporting observations: - -```bash -cat benchmark/.run/target/logs/nginx.pid -ps -o pid,ppid,pcpu,rss,comm --ppid "$(cat benchmark/.run/target/logs/nginx.pid)" -curl -fsS "http://127.0.0.1:${TARGET_PORT:-19840}/gc" -``` +## If a run is marked unsaturated -The GC endpoint prints `collectgarbage("count")` in KB for the target worker. -Treat RSS and GC values as diagnostics, not as the primary comparison metric. +Do not compare QPS across cases from runs where the target worker was not the +bottleneck. Increase `WRK_THREADS`, `WRK_CONNECTIONS` or `WRK_RATE`; move wrk2 +to non-target cores; or give the upstream more workers and cores. -## Dry Run +## Dry run -Use dry-run mode to verify template rendering without starting OpenResty or -`wrk2`: +Renders the configs and generates the locations without starting anything: ```bash BENCH_DRY_RUN=1 make bench diff --git a/benchmark/cases.txt b/benchmark/cases.txt new file mode 100644 index 0000000..44e09da --- /dev/null +++ b/benchmark/cases.txt @@ -0,0 +1,59 @@ +# Canonical bench case list. One name per line, "#" starts a comment. +# +# This is the single source of truth: run.sh generates one target location per +# name from this file, and the smoke check diffs it against the case table in +# benchmark/lua/bench.lua so the two cannot drift apart. +# +# Names are .. A headline comparison is one shape read across the +# ffi.* and resty.* columns; the ffi.oneshot-vs-ffi.stateful gap prices the C +# fast path against this library's own Lua object layer. + +# --- the 2x2 --- +ffi.oneshot +resty.oneshot +ffi.stateful +resty.stateful + +# --- read mode --- +ffi.readbody +resty.readbody +ffi.stream +resty.stream + +# --- connection lifetime --- +ffi.short +resty.short + +# --- method --- +ffi.post +resty.post + +# --- transport --- +ffi.tls +resty.tls +ffi.tlsshort +resty.tlsshort +ffi.tlsverify +resty.tlsverify +ffi.tlsverifyshort +resty.tlsverifyshort + +# --- peer address --- +ffi.dns +resty.dns + +# --- response framing --- +ffi.chunked +resty.chunked +ffi.trailers +resty.trailers + +# --- response header count --- +ffi.hdr40 +resty.hdr40 +ffi.cookies +resty.cookies + +# --- request header count --- +ffi.req30 +resty.req30 diff --git a/benchmark/conf/target.nginx.conf b/benchmark/conf/target.nginx.conf index 6e2100c..a39984b 100644 --- a/benchmark/conf/target.nginx.conf +++ b/benchmark/conf/target.nginx.conf @@ -1,7 +1,7 @@ worker_processes 1; daemon on; master_process on; -error_log logs/error.log info; +error_log logs/error.log error; pid logs/nginx.pid; events { @@ -12,12 +12,40 @@ http { lua_package_path "{{LUA_PACKAGE_PATH}};;"; access_log off; + # The downstream leg is common to every case including the baseline, so it + # must not introduce churn of its own: at tens of thousands of requests per + # second the stock keepalive_requests would recycle wrk's connections + # hundreds of times per run and charge the accept path to whichever case was + # running. + keepalive_requests 1000000; + keepalive_timeout 300s; + + resolver {{RESOLVER}} ipv6=off; + resolver_timeout 5s; + + # resty.http verifies against the cosocket trust store, which is per-worker + # config rather than per-request. The FFI client is handed the same file as + # ssl_trusted_certificate, so the verify-on cases ask both clients to check + # the identical chain against the identical root. + lua_ssl_trusted_certificate {{CA_FILE}}; + lua_ssl_verify_depth 5; + init_by_lua_block { - benchmark_upstream_host = "127.0.0.1" - benchmark_upstream_port = {{UPSTREAM_PORT}} - benchmark_keepalive_pool_size = {{KEEPALIVE_POOL_SIZE}} - benchmark_keepalive_idle_timeout = {{KEEPALIVE_IDLE_TIMEOUT}} - benchmark_response_body = string.rep("x", {{RESPONSE_SIZE}}) + local bench = require "bench" + + bench.init({ + host = "{{UPSTREAM_HOST}}", + port = {{UPSTREAM_PORT}}, + tls_port = {{UPSTREAM_TLS_PORT}}, + trailer_port = {{UPSTREAM_TRAILER_PORT}}, + dns_host = "{{DNS_HOST}}", + tls_server_name = "{{TLS_SERVER_NAME}}", + ca_file = "{{CA_FILE}}", + pool_size = {{KEEPALIVE_POOL_SIZE}}, + idle_timeout = {{KEEPALIVE_IDLE_TIMEOUT}}, + timeout = {{CLIENT_TIMEOUT}}, + response_body = string.rep("x", {{RESPONSE_SIZE}}), + }) } server { @@ -27,6 +55,7 @@ http { content_by_lua_block { local ok_ffi, ffi_err = pcall(require, "resty.ngx_http_ffi_client") local ok_http, http_err = pcall(require, "resty.http") + local ok_bench, bench_err = pcall(require, "bench") if not ok_ffi then ngx.status = 500 @@ -40,183 +69,75 @@ http { return end + if not ok_bench then + ngx.status = 500 + ngx.say("bench: " .. tostring(bench_err)) + return + end + ngx.say("ok") } } - location = /gc { + # The case table as the worker actually sees it. The runner diffs this + # against benchmark/cases.txt, so a case added to one and not the other + # fails the smoke check instead of silently going unmeasured. + location = /cases { content_by_lua_block { - ngx.say(collectgarbage("count")) + local bench = require "bench" + ngx.say(table.concat(bench.case_names(), "\n")) } } - location = /bench/baseline { + location = /gc { content_by_lua_block { - local body = benchmark_response_body - ngx.header["Content-Length"] = #body - ngx.print(body) + ngx.say(collectgarbage("count")) } } - location = /smoke/ffi-short { + # Per-request Lua heap delta: allocate-free the case once, then measure + # the next call against a settled heap. Diagnostic on its own, but the + # library's claim is that response handling stays in C, and the header + # arrays and the case-insensitive metatable are exactly where that could + # regress, so the runner records it as a reported metric. + location ~ ^/gcdelta/(?[a-z0-9._-]+)$ { content_by_lua_block { - local client = require "resty.ngx_http_ffi_client" - local res, err = client.request_uri({ - scheme = "http", - host = benchmark_upstream_host, - port = benchmark_upstream_port, - method = "GET", - path = "/mock", - headers = { - ["X-Benchmark-Case"] = "ffi-short", - }, - }) - - if not res then - ngx.status = 502 - ngx.say(err) - return - end + local bench = require "bench" + local name = ngx.var.bench_case - ngx.status = res.status - ngx.print(res.body) - } - } + bench.audit(name) - location = /smoke/resty-http-short { - content_by_lua_block { - local http = require "resty.http" - local httpc, err = http.new() - if not httpc then - ngx.status = 502 - ngx.say(err) - return - end + collectgarbage("collect") + collectgarbage("collect") - httpc:set_timeout(60000) + local before = collectgarbage("count") + local n = 20 - local ok - ok, err = httpc:connect(benchmark_upstream_host, benchmark_upstream_port) - if not ok then - ngx.status = 502 - ngx.say(err) - return + for _ = 1, n do + bench.audit(name) end - local res - res, err = httpc:request({ - method = "GET", - path = "/mock", - headers = { - ["Host"] = benchmark_upstream_host, - ["X-Benchmark-Case"] = "resty-http-short", - }, - }) - if not res then - ngx.status = 502 - ngx.say(err) - return - end + local after = collectgarbage("count") - local body - body, err = res:read_body() - httpc:close() - if not body then - ngx.status = 502 - ngx.say(err) - return - end - - ngx.status = res.status - ngx.print(body) + ngx.say(string.format("%.3f", (after - before) / n)) } } - location = /bench/ffi { + location ~ ^/audit/(?[a-z0-9._-]+)$ { content_by_lua_block { - local client = require "resty.ngx_http_ffi_client" - local res, err = client.request_uri({ - scheme = "http", - host = benchmark_upstream_host, - port = benchmark_upstream_port, - method = "GET", - path = "/mock", - headers = { - ["X-Benchmark-Case"] = "ffi-keepalive", - }, - keepalive = { - pool = "benchmark-ffi", - idle_timeout = benchmark_keepalive_idle_timeout, - pool_size = benchmark_keepalive_pool_size, - }, - }) - - if not res then - ngx.status = 502 - ngx.say(err) - return - end - - ngx.status = res.status - ngx.header["X-Benchmark-Reused"] = tostring(res.reused) - ngx.print(res.body) + require("bench").audit(ngx.var.bench_case) } } - location = /bench/resty-http { + location = /bench/baseline { content_by_lua_block { - local http = require "resty.http" - local httpc, err = http.new() - if not httpc then - ngx.status = 502 - ngx.say(err) - return - end - - httpc:set_timeout(60000) - - local ok - ok, err = httpc:connect(benchmark_upstream_host, benchmark_upstream_port) - if not ok then - ngx.status = 502 - ngx.say(err) - return - end - - local res - res, err = httpc:request({ - method = "GET", - path = "/mock", - headers = { - ["Host"] = benchmark_upstream_host, - ["X-Benchmark-Case"] = "resty-http-keepalive", - }, - }) - if not res then - ngx.status = 502 - ngx.say(err) - return - end - - local body - body, err = res:read_body() - if not body then - ngx.status = 502 - ngx.say(err) - return - end - - local ok_keepalive, keepalive_err = - httpc:set_keepalive(benchmark_keepalive_idle_timeout, - benchmark_keepalive_pool_size) - if not ok_keepalive then - ngx.log(ngx.ERR, "resty.http set_keepalive failed: ", - keepalive_err) - end - - ngx.status = res.status - ngx.print(body) + require("bench").baseline() } } + + # One exact location per case, generated from benchmark/cases.txt. They + # are exact rather than a single regex location on purpose: a regex match + # would land on the measured worker's hot path in every case. + include bench_locations.conf; } } diff --git a/benchmark/conf/upstream.nginx.conf b/benchmark/conf/upstream.nginx.conf index c1fb60c..98edf76 100644 --- a/benchmark/conf/upstream.nginx.conf +++ b/benchmark/conf/upstream.nginx.conf @@ -1,25 +1,78 @@ worker_processes {{UPSTREAM_WORKERS}}; daemon on; master_process on; -error_log logs/error.log info; +error_log logs/error.log error; pid logs/nginx.pid; events { worker_connections 65535; } +# The mock peer. It is deliberately over-provisioned relative to the single +# pinned target worker: any measurement where this side is the bottleneck is +# measuring this file rather than the client under test. http { access_log off; + # A client keepalive pool is only worth measuring if the peer keeps its end + # of the connection open. At the stock keepalive_requests the upstream would + # close every connection after 1000 requests and the client pools would + # spend the run reconnecting. + keepalive_requests 1000000; + keepalive_timeout 300s; + + lua_shared_dict bench_stats 1m; + init_by_lua_block { benchmark_response_body = string.rep("x", {{RESPONSE_SIZE}}) + + -- 40 response headers, for the response-header-count axis + benchmark_headers40 = {} + for i = 1, 40 do + benchmark_headers40[i] = { string.format("X-H-%02d", i), + string.rep("h", 24) } + end + + -- a Set-Cookie-heavy response: one name arriving ten times, which is + -- the shape that makes the client fold repeated headers into an array + benchmark_cookies = {} + for i = 1, 10 do + benchmark_cookies[i] = string.format( + "bench%02d=%s; Path=/; HttpOnly", i, string.rep("c", 24)) + end + } + + # Connection and request counters. The runner reads these before and after + # each case: requests-per-connection at or near 1 means the client pool is + # not reusing anything, which invalidates every keepalive case. Counting + # here rather than on the target keeps the check off the measured worker's + # hot path entirely. + init_worker_by_lua_block { + local stats = ngx.shared.bench_stats + + function benchmark_count() + stats:incr("requests", 1, 0) + + if ngx.var.connection_requests == "1" then + stats:incr("connections", 1, 0) + end + end } server { listen 127.0.0.1:{{UPSTREAM_PORT}}; + listen 127.0.0.1:{{UPSTREAM_TLS_PORT}} ssl; + + ssl_certificate tls.crt; + ssl_certificate_key tls.key; + ssl_session_cache shared:bench_ssl:10m; + ssl_session_timeout 10m; + # Content-Length framed, the reference response shape. location = /mock { content_by_lua_block { + benchmark_count() + local body = benchmark_response_body ngx.header["Content-Length"] = #body ngx.header["X-Benchmark-Upstream"] = "mock" @@ -27,8 +80,138 @@ http { } } + # Same response, but reads a request body first: the POST axis. + location = /mock/sink { + content_by_lua_block { + benchmark_count() + + ngx.req.read_body() + + local body = benchmark_response_body + ngx.header["Content-Length"] = #body + ngx.header["X-Benchmark-Upstream"] = "sink" + ngx.print(body) + } + } + + # No Content-Length, so nginx frames the body with chunked encoding. + location = /mock/chunked { + content_by_lua_block { + benchmark_count() + + ngx.header["X-Benchmark-Upstream"] = "chunked" + ngx.print(benchmark_response_body) + } + } + + location = /mock/headers40 { + content_by_lua_block { + benchmark_count() + + local h = ngx.header + for i = 1, #benchmark_headers40 do + local pair = benchmark_headers40[i] + h[pair[1]] = pair[2] + end + + local body = benchmark_response_body + h["Content-Length"] = #body + ngx.print(body) + } + } + + location = /mock/cookies { + content_by_lua_block { + benchmark_count() + + ngx.header["Set-Cookie"] = benchmark_cookies + + local body = benchmark_response_body + ngx.header["Content-Length"] = #body + ngx.print(body) + } + } + + # Echoes back the raw request header block exactly as received. The + # runner's fairness audit diffs the two columns of a pair against this: + # if the bytes differ, the pair is not measuring the same request. + location = /mock/echo { + content_by_lua_block { + local raw = ngx.req.raw_header() + ngx.header["Content-Length"] = #raw + ngx.header["Content-Type"] = "text/plain" + ngx.print(raw) + } + } + + location = /stats { + content_by_lua_block { + local stats = ngx.shared.bench_stats + ngx.say("connections ", stats:get("connections") or 0) + ngx.say("requests ", stats:get("requests") or 0) + } + } + + location = /stats/reset { + content_by_lua_block { + local stats = ngx.shared.bench_stats + stats:set("connections", 0) + stats:set("requests", 0) + ngx.say("ok") + } + } + location = /ready { return 200 "ok\n"; } } } + +# Trailers cannot be emitted from the http subsystem, so the trailer case gets a +# raw TCP mock that writes exact response bytes: chunked framing, then a trailer +# section behind the terminating chunk. It answers on a keepalive loop so the +# client pool behaves the same as it does against the http mock. +stream { + init_by_lua_block { + local body = string.rep("x", {{RESPONSE_SIZE}}) + + benchmark_trailer_response = table.concat({ + "HTTP/1.1 200 OK\r\n", + "X-Benchmark-Upstream: trailers\r\n", + "Trailer: X-Checksum\r\n", + "Transfer-Encoding: chunked\r\n", + "\r\n", + string.format("%x\r\n", #body), body, "\r\n", + "0\r\n", + "X-Checksum: abc123\r\n", + "\r\n", + }) + } + + server { + listen 127.0.0.1:{{UPSTREAM_TRAILER_PORT}}; + + content_by_lua_block { + local sock = ngx.req.socket() + sock:settimeout(300000) + + local resp = benchmark_trailer_response + -- one receive per request rather than one per header line: this mock + -- has to keep up with a saturated target worker + local read_head = sock:receiveuntil("\r\n\r\n") + + while true do + local head, err = read_head() + if not head then + return + end + + local ok + ok, err = sock:send(resp) + if not ok then + return + end + end + } + } +} diff --git a/benchmark/fold.sh b/benchmark/fold.sh new file mode 100755 index 0000000..5a40c3c --- /dev/null +++ b/benchmark/fold.sh @@ -0,0 +1,225 @@ +#!/usr/bin/env bash +# +# Fold raw benchmark rows into the markdown tables the READMEs carry. +# +# benchmark/fold.sh benchmark/.run/results/.csv +# +# Reads the CSV that run.sh emits, one row per wrk2 run, and prints the summary +# tables. Rows that failed the saturation gate or reported non-2xx responses are +# excluded from every median and counted separately, because a run where the +# target worker was not the bottleneck is not measuring the client. + +set -euo pipefail + +CSV="${1:-}" + +if [[ -z "$CSV" ]]; then + latest="$(ls -1t "$(dirname "${BASH_SOURCE[0]}")/.run/results"/*.csv \ + 2>/dev/null | head -n 1 || true)" + CSV="$latest" +fi + +[[ -n "$CSV" && -f "$CSV" ]] || { + printf 'usage: %s \n' "${BASH_SOURCE[0]}" >&2 + exit 1 +} + +awk -F, ' +function median(arr, n, i, j, t) { + if (n == 0) return "" + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (arr[j] < arr[i]) { t = arr[i]; arr[i] = arr[j]; arr[j] = t } + if (n % 2) return arr[(n - 1) / 2] + return (arr[n / 2 - 1] + arr[n / 2]) / 2 +} + +function collect(key, value, n) { + n = count[key]++ + vals[key SUBSEP n] = value +} + +function med_of(key, i, n, a) { + n = count[key] + if (n == 0) return "" + for (i = 0; i < n; i++) a[i] = vals[key SUBSEP i] + return median(a, n) +} + +function min_of(key, i, n, m) { + n = count[key] + if (n == 0) return "" + m = vals[key SUBSEP 0] + for (i = 1; i < n; i++) if (vals[key SUBSEP i] < m) m = vals[key SUBSEP i] + return m +} + +function max_of(key, i, n, m) { + n = count[key] + if (n == 0) return "" + m = vals[key SUBSEP 0] + for (i = 1; i < n; i++) if (vals[key SUBSEP i] > m) m = vals[key SUBSEP i] + return m +} + +NR == 1 { next } + +{ + phase = $2; name = $4; qps = $5 + 0; non2xx = $7 + 0 + cpu = $12 + 0; sat = $15; rss = $16 + 0; rpc = $19 + p50 = $9; p99 = $10; p999 = $11; rate = $6 + parser = $24; conns = $21; rsize = $20; dur = $23 + + meta_parser = parser; meta_conns = conns; meta_rsize = rsize; meta_dur = dur + + if (phase == "throughput") { + seen_tp[name] = 1 + total[name]++ + + if (sat == "yes" && non2xx == 0) { + collect("qps:" name, qps) + collect("cpu:" name, cpu) + collect("rss:" name, rss) + # "inf" is not a number but it is the best possible reuse result: + # no new connections at all during the measured window + if (rpc == "inf") reuse_inf[name]++ + else if (rpc != "") collect("rpc:" name, rpc + 0) + } else { + discarded[name]++ + } + } + + if (phase == "latency") { + seen_lat[name] = 1 + lat_rate[name] = rate + if (non2xx == 0) { + if (p50 != "") collect("p50:" name, p50 + 0) + if (p99 != "") collect("p99:" name, p99 + 0) + if (p999 != "") collect("p999:" name, p999 + 0) + } + } +} + +END { + base = med_of("qps:baseline") + + printf "\n### Run parameters\n\n" + printf "parser `%s`, response size `%s` bytes, wrk2 connections `%s`, ", + meta_parser, meta_rsize, meta_conns + printf "duration `%s` per repeat\n", meta_dur + + # --- headline: one row per shape --- + printf "\n### Headline (median QPS per shape)\n\n" + printf "| shape | C FFI | `resty.http` | C FFI / `resty.http` |" + if (base != "") printf " outbound cost ratio |" + printf "\n| --- | ---: | ---: | ---: |" + if (base != "") printf " ---: |" + printf "\n" + + for (name in seen_tp) { + if (name == "baseline") continue + # not split(name, parts, ".") -- awk reads the separator as a regex, so + # "." matches every character and every field comes back empty + if (substr(name, 1, 4) != "ffi.") continue + shape = substr(name, 5) + + f = med_of("qps:ffi." shape) + r = med_of("qps:resty." shape) + if (f == "" || r == "") continue + + printf "| `%s` | `%.2f` | `%.2f` | `%.2fx` |", shape, f, r, f / r + + if (base != "" && f > 0 && r > 0) { + bus = 1000000 / base + fus = 1000000 / f - bus + rus = 1000000 / r - bus + if (fus > 0 && rus > 0) printf " `%.2fx` |", rus / fus + else printf " n/a |" + } else if (base != "") { + printf " n/a |" + } + printf "\n" + } + + if (base != "") { + printf "\nBaseline (no upstream call): `%.2f` QPS, `%.3f` us/request.\n", + base, 1000000 / base + printf "The outbound cost ratio subtracts that baseline from both ", + "" + printf "client paths and compares what is left.\n" + } + + # --- detail --- + printf "\n### Detail (throughput phase)\n\n" + printf "| case | median QPS | min | max | repeats used | discarded |" + printf " worker CPU avg | worker RSS | requests/connection |\n" + printf "| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n" + + n = 0 + for (name in seen_tp) ordered[n++] = name + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (ordered[j] < ordered[i]) { t = ordered[i]; ordered[i] = ordered[j]; ordered[j] = t } + + for (i = 0; i < n; i++) { + name = ordered[i] + m = med_of("qps:" name) + used = count["qps:" name] + drop = discarded[name] + 0 + rssv = med_of("rss:" name) + rpcv = med_of("rpc:" name) + + printf "| `%s` |", name + if (m == "") printf " n/a | n/a | n/a |" + else printf " `%.2f` | `%.2f` | `%.2f` |", m, min_of("qps:" name), max_of("qps:" name) + printf " %d | %d |", used, drop + if (used > 0) printf " `%.1f%%` | `%d` KB |", med_of("cpu:" name), rssv + else printf " n/a | n/a |" + if (rpcv != "") printf " `%.1f` |", rpcv + else if (reuse_inf[name] > 0) printf " no new connections |" + else printf " n/a |" + printf "\n" + } + + # --- latency --- + haslat = 0 + for (name in seen_lat) haslat = 1 + + if (haslat) { + printf "\n### Latency (fixed rate, corrected for coordinated omission)\n\n" + printf "| case | rate | p50 | p99 | p99.9 |\n" + printf "| --- | ---: | ---: | ---: | ---: |\n" + + n = 0 + delete ordered + for (name in seen_lat) ordered[n++] = name + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + if (ordered[j] < ordered[i]) { t = ordered[i]; ordered[i] = ordered[j]; ordered[j] = t } + + for (i = 0; i < n; i++) { + name = ordered[i] + printf "| `%s` | `%s` req/s |", name, lat_rate[name] + v = med_of("p50:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + v = med_of("p99:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + v = med_of("p999:" name); printf (v == "" ? " n/a |" : " `" sprintf("%.3f", v) "` ms |") + printf "\n" + } + } + + # --- what got thrown away --- + anydrop = 0 + for (name in discarded) if (discarded[name] > 0) anydrop = 1 + + if (anydrop) { + printf "\n### Discarded rows\n\n" + printf "Rows failing the saturation gate or reporting non-2xx responses," + printf " excluded from every median above:\n\n" + for (i = 0; i < n; i++) { } + for (name in discarded) + if (discarded[name] > 0) + printf "- `%s`: %d of %d\n", name, discarded[name], total[name] + } + printf "\n" +} +' "$CSV" diff --git a/benchmark/lua/bench.lua b/benchmark/lua/bench.lua new file mode 100644 index 0000000..17a4b5d --- /dev/null +++ b/benchmark/lua/bench.lua @@ -0,0 +1,555 @@ +-- Bench case drivers. +-- +-- Every case is a driver plus a shape, never a hand-written handler. That is +-- deliberate: the old harness hand-wrote one location per client and the two +-- drifted into different API shapes, so the published ratio credited the +-- API-shape difference to the C-vs-Lua difference. Here both columns of a pair +-- are built from one shape table, so they send the same headers, carry the same +-- timeouts, size the same pool, and consume the body the same way by +-- construction rather than by review. +-- +-- A driver returns (bytes_read, status) or (nil, nil, err). The dispatcher +-- writes the same fixed-size downstream body for every case, the baseline +-- included, so the downstream write cost cancels out of the baseline +-- subtraction and only the outbound client work differs between cases. + +local ffi_client = require "resty.ngx_http_ffi_client" +local resty_http = require "resty.http" + +local ngx = ngx +local ngx_log = ngx.log +local ngx_ERR = ngx.ERR +local pairs = pairs +local tostring = tostring +local table_sort = table.sort +local str_format = string.format +local str_rep = string.rep + + +local _M = {} + + +local cfg = { + host = "127.0.0.1", + port = 80, + tls_port = 443, + trailer_port = 8443, + dns_host = "localhost", + tls_server_name = "bench.local", + -- the upstream's self-signed cert, trusted by both clients for the + -- verify-on cases. resty.http has no per-request trust store, so its side + -- comes from lua_ssl_trusted_certificate in the target conf; this is the + -- same file, so both are asked to verify the identical chain. + ca_file = "", + pool_size = 120, + idle_timeout = 60000, + timeout = 60000, + response_body = "", +} + + +-- Request header sets. Both columns of a pair always send the identical table, +-- so the 3-vs-30 axis prices per-request header validation rather than a +-- difference in what the two clients were asked to send. The tables are built +-- once: constructing them per request would measure Lua table construction. +local HEADERS_3 = { + ["X-Bench-A"] = "aaaaaaaa", + ["X-Bench-B"] = "bbbbbbbb", + ["X-Bench-C"] = "cccccccc", +} + +local HEADERS_30 = {} +do + for i = 1, 30 do + HEADERS_30[str_format("X-Bench-%02d", i)] = str_rep("v", 16) + end +end + +local POST_BODY = str_rep("p", 4096) + + +function _M.init(opts) + for k, v in pairs(opts) do + cfg[k] = v + end +end + + +-- ===== shape ===== +-- +-- scheme "http" or "https" +-- dns resolve the peer by hostname instead of using the IP literal +-- trailers talk to the raw TCP mock, the only peer here that sends trailers +-- path upstream path, which selects the response framing and header count +-- method defaults to GET +-- body request body, for the POST axis +-- headers request header table, for the request-header-count axis +-- keepalive false for the short-lived-connection axis +-- stream drain res.body_reader instead of buffering the body +-- read_body buffer the body, but through the Lua chunk loop rather than the +-- C preread fast path (FFI only; resty.http has no other mode) +-- echo return the body itself instead of its length, for the audit +-- pool keepalive pool name, assigned per case by pair_case + +local function peer(s) + local scheme = s.scheme or "http" + local https = scheme == "https" + local port + + if https then + port = cfg.tls_port + elseif s.trailers then + port = cfg.trailer_port + else + port = cfg.port + end + + return scheme, + s.dns and cfg.dns_host or cfg.host, + port, + s.path or "/mock", + s.method or "GET", + s.headers or HEADERS_3, + s.keepalive ~= false, + -- both clients send the same SNI for the TLS cases; left to their own + -- devices the FFI client defaults it to the host and resty.http omits + -- it, which would not be the same handshake + https and cfg.tls_server_name or nil, + -- verification is off unless a shape asks for it, so the plain tls + -- cases keep pricing the handshake rather than the trust store + (https and s.verify) and true or false +end + + +-- ===== FFI client ===== + +local function ffi_oneshot(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local res, err = ffi_client.request_uri({ + scheme = scheme, + host = host, + port = port, + method = method, + path = path, + headers = headers, + body = s.body, + timeout = cfg.timeout, + ssl_verify = verify, + ssl_trusted_certificate = verify and cfg.ca_file or nil, + ssl_server_name = sni, + keepalive = keepalive and { + pool = s.pool, + pool_size = cfg.pool_size, + idle_timeout = cfg.idle_timeout, + } or nil, + }) + + if not res then + return nil, nil, err + end + + if s.echo then + return res.body, res.status + end + + return #res.body, res.status +end + + +local function ffi_stateful(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = ffi_client.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local ok + ok, err = httpc:connect({ + scheme = scheme, + host = host, + port = port, + pool = s.pool, + pool_size = cfg.pool_size, + ssl_verify = verify, + ssl_trusted_certificate = verify and cfg.ca_file or nil, + ssl_server_name = sni, + }) + if not ok then + return nil, nil, err + end + + local res + res, err = httpc:request({ + method = method, + path = path, + headers = headers, + body = s.body, + -- the buffered fast path: C reads the whole body and resumes once. + -- read_body sends the same request but takes the Lua chunk loop back. + preread_body = not (s.stream or s.read_body), + }) + if not res then + httpc:close() + return nil, nil, err + end + + local result + + if s.stream then + local n = 0 + local reader = res.body_reader + + while true do + local chunk, rerr = reader() + if rerr then + httpc:close() + return nil, nil, rerr + end + if not chunk then + break + end + n = n + #chunk + end + + result = n + + else + local body + body, err = res:read_body() + if not body then + httpc:close() + return nil, nil, err + end + + result = s.echo and body or #body + end + + if keepalive then + ok, err = httpc:set_keepalive(cfg.idle_timeout, cfg.pool_size) + if not ok then + ngx_log(ngx_ERR, "ffi set_keepalive failed: ", tostring(err)) + end + + else + httpc:close() + end + + return result, res.status +end + + +-- ===== lua-resty-http ===== + +local function resty_oneshot(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = resty_http.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local res + res, err = httpc:request_uri( + str_format("%s://%s:%d%s", scheme, host, port, path), { + method = method, + headers = headers, + body = s.body, + -- the FFI client sends no User-Agent, so neither does this one: + -- otherwise resty.http pays for ~55 extra bytes on every request + use_default_user_agent = false, + ssl_verify = verify, + ssl_server_name = sni, + pool = s.pool, + pool_size = cfg.pool_size, + keepalive = keepalive, + keepalive_timeout = cfg.idle_timeout, + keepalive_pool = cfg.pool_size, + }) + + if not res then + return nil, nil, err + end + + if s.echo then + return res.body, res.status + end + + return #res.body, res.status +end + + +local function resty_stateful(s) + local scheme, host, port, path, method, headers, keepalive, sni, + verify = peer(s) + + local httpc, err = resty_http.new() + if not httpc then + return nil, nil, err + end + + httpc:set_timeouts(cfg.timeout, cfg.timeout, cfg.timeout) + + local ok + ok, err = httpc:connect({ + scheme = scheme, + host = host, + port = port, + pool = s.pool, + pool_size = cfg.pool_size, + ssl_verify = verify, + ssl_server_name = sni, + }) + if not ok then + return nil, nil, err + end + + local res + res, err = httpc:request({ + method = method, + path = path, + headers = headers, + body = s.body, + use_default_user_agent = false, + }) + if not res then + httpc:close() + return nil, nil, err + end + + local result + + if s.stream then + local n = 0 + local reader = res.body_reader + + while true do + local chunk, rerr = reader() + if rerr then + httpc:close() + return nil, nil, rerr + end + if not chunk then + break + end + n = n + #chunk + end + + result = n + + else + local body + body, err = res:read_body() + if not body then + httpc:close() + return nil, nil, err + end + + result = s.echo and body or #body + end + + if keepalive then + ok, err = httpc:set_keepalive(cfg.idle_timeout, cfg.pool_size) + if not ok then + ngx_log(ngx_ERR, "resty.http set_keepalive failed: ", tostring(err)) + end + + else + httpc:close() + end + + return result, res.status +end + + +-- ===== case table ===== +-- +-- Names are .. The headline ratio compares the same shape across +-- the ffi.* and resty.* columns; the gap between ffi.oneshot and ffi.stateful +-- separately prices the C fast path against this library's own Lua object +-- layer. + +local cases = {} + + +local function pair_case(name, shape) + for _, driver in ipairs({ "ffi", "resty" }) do + local copy = {} + + for k, v in pairs(shape) do + copy[k] = v + end + + -- separate pools, so one client can never be handed a connection the + -- other opened and look free for whichever case ran second + copy.pool = "bench-" .. driver .. "-" .. name + + local fn + if driver == "ffi" then + fn = shape.oneshot and ffi_oneshot or ffi_stateful + else + fn = shape.oneshot and resty_oneshot or resty_stateful + end + + cases[driver .. "." .. name] = { fn = fn, shape = copy } + end +end + + +-- the 2x2: one-shot and stateful, on both clients, against the same upstream +pair_case("oneshot", { oneshot = true }) +pair_case("stateful", {}) + +-- read mode +pair_case("readbody", { read_body = true }) +pair_case("stream", { stream = true }) + +-- connection lifetime +pair_case("short", { keepalive = false }) + +-- method +pair_case("post", { method = "POST", body = POST_BODY, path = "/mock/sink" }) + +-- transport: pooled connections amortise the handshake away, so the +-- short-lived TLS case is the one that prices a fresh handshake +pair_case("tls", { scheme = "https" }) +pair_case("tlsshort", { scheme = "https", keepalive = false }) +-- verify on: these price the trust store, which the two cases above +-- deliberately do not. Both clients trust the same self-signed upstream cert. +pair_case("tlsverify", { scheme = "https", verify = true }) +pair_case("tlsverifyshort", + { scheme = "https", verify = true, keepalive = false }) + +-- peer address +pair_case("dns", { dns = true }) + +-- response framing +pair_case("chunked", { path = "/mock/chunked" }) +pair_case("trailers", { trailers = true, path = "/" }) + +-- response header count +pair_case("hdr40", { path = "/mock/headers40" }) +pair_case("cookies", { path = "/mock/cookies" }) + +-- request header count +pair_case("req30", { headers = HEADERS_30 }) + + +function _M.run(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.print("unknown bench case: ", tostring(name)) + return + end + + local n, status, err = case.fn(case.shape) + + if not n then + ngx.status = 502 + ngx_log(ngx_ERR, "bench case ", name, " failed: ", tostring(err)) + ngx.print(tostring(err)) + return + end + + ngx.status = status + + -- every case, baseline included, writes the same downstream bytes, so the + -- baseline subtraction removes an identical downstream cost from each + local body = cfg.response_body + ngx.header["Content-Length"] = #body + ngx.print(body) +end + + +function _M.baseline() + local body = cfg.response_body + ngx.header["Content-Length"] = #body + ngx.print(body) +end + + +-- Report what a case actually did: status and bytes read. The runner calls this +-- during the fairness audit, before any load, so nothing here costs anything on +-- the measured path. +function _M.audit(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.say("unknown bench case: ", tostring(name)) + return + end + + local n, status, err = case.fn(case.shape) + + if not n then + ngx.status = 502 + ngx.say("case=", name, " error=", tostring(err)) + return + end + + ngx.say("case=", name, " status=", status, " bytes=", n) +end + + +-- Run a case against the upstream's echo endpoint, which replies with the raw +-- request header block it received. The runner diffs the two columns of a pair: +-- if the bytes differ, the pair is not measuring the same request, and that is +-- a fairness failure before any load is applied. +function _M.echo(name) + local case = cases[name] + if not case then + ngx.status = 500 + ngx.say("unknown bench case: ", tostring(name)) + return + end + + local shape = {} + for k, v in pairs(case.shape) do + shape[k] = v + end + + shape.path = "/mock/echo" + -- the raw TCP trailer mock has no echo endpoint, so this pair is audited + -- against the http mock instead + shape.trailers = nil + -- echoing is a buffered read whatever the case's own read mode is + shape.stream = nil + shape.read_body = nil + shape.echo = true + shape.pool = shape.pool .. "-echo" + + local body, _, err = case.fn(shape) + + if not body then + ngx.status = 502 + ngx.say("error: ", tostring(err)) + return + end + + ngx.print(body) +end + + +function _M.case_names() + local names = {} + + for name in pairs(cases) do + names[#names + 1] = name + end + + table_sort(names) + + return names +end + + +return _M diff --git a/benchmark/results-full.md b/benchmark/results-full.md new file mode 100644 index 0000000..6438f15 --- /dev/null +++ b/benchmark/results-full.md @@ -0,0 +1,346 @@ +# Full-matrix benchmark run (`full`) + +Run finished 2026-08-04. This file records one complete run of +`benchmark/run.sh` comparing this repo's C/FFI HTTP client +(`resty.ngx_http_ffi_client`) against `lua-resty-http` 0.16.1 across the full +matrix of request shapes. It is a results record only; it does not replace the +methodology write-up in `benchmark/README.md`. + +## What was run + +- Run parameters: `BENCH_RUN_ID=full`, `BENCH_CASES=all`, `BENCH_REPEATS=5`, + `BENCH_LATENCY_REPEATS=3`. +- Hardware: 8-core VM. The target OpenResty worker (openresty/1.29.2.4) was + pinned to CPU 0, the upstream to CPUs 1-5, and `wrk2` to CPUs 6-7, so the + measured client always runs on a single dedicated core and neither the load + generator nor the upstream competes with it. +- The run exited 0. The fairness audit passed for all 14 shapes: every shape + produced the same request line and the same caller-supplied headers from both + clients (all 14 `echo-.diff` artifacts are empty). The only reported + difference is the known, non-failing one — the FFI client emits + `Connection: keep-alive` and `Content-Length: 0` on a bodyless GET where + `lua-resty-http` omits them (`Connection: keep-alive` only, for `post`). +- Raw rows: `benchmark/.run/results/full.csv`, 229 data rows — 145 throughput + rows (29 cases x 5 repeats) and 84 latency rows (28 cases x 3 repeats). +- Exactly 1 row was discarded: `resty.dns` throughput repeat 1, which reported + 4 non-2xx responses and 13 socket errors. It passed the saturation gate but + the non-2xx count excludes it. No other row in the file has a non-zero + `non2xx` or `socket_errors` value, and no throughput row failed the saturation + gate (`saturated` is `yes` on all 145). + +## Folded results + +The tables below are the verbatim output of +`./benchmark/fold.sh benchmark/.run/results/full.csv`. Every figure is a median +over the kept repeats. + +### Run parameters + +parser `llhttp`, response size `1024` bytes, wrk2 connections `100`, duration `30s` per repeat + +### Headline (median QPS per shape) + +| shape | C FFI | `resty.http` | C FFI / `resty.http` | outbound cost ratio | +| --- | ---: | ---: | ---: | ---: | +| `stream` | `27506.05` | `16236.65` | `1.69x` | `2.02x` | +| `stateful` | `26919.60` | `15604.60` | `1.73x` | `2.05x` | +| `hdr40` | `17724.68` | `9218.37` | `1.92x` | `2.16x` | +| `req30` | `23339.15` | `13283.15` | `1.76x` | `2.04x` | +| `cookies` | `25440.85` | `13603.27` | `1.87x` | `2.23x` | +| `trailers` | `28005.24` | `6622.05` | `4.23x` | `5.77x` | +| `readbody` | `26965.62` | `15710.28` | `1.72x` | `2.04x` | +| `oneshot` | `30925.84` | `14510.37` | `2.13x` | `2.76x` | +| `tlsshort` | `932.16` | `1149.50` | `0.81x` | `0.81x` | +| `post` | `24173.16` | `12780.46` | `1.89x` | `2.24x` | +| `chunked` | `26663.11` | `15366.94` | `1.74x` | `2.06x` | +| `short` | `8721.09` | `7152.20` | `1.22x` | `1.24x` | +| `dns` | `26814.07` | `15634.98` | `1.72x` | `2.04x` | +| `tls` | `18655.46` | `11866.50` | `1.57x` | `1.73x` | + +Baseline (no upstream call): `86589.91` QPS, `11.549` us/request. +The outbound cost ratio subtracts that baseline from both client paths and compares what is left. + +### Detail (throughput phase) + +| case | median QPS | min | max | repeats used | discarded | worker CPU avg | worker RSS | requests/connection | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `baseline` | `86589.91` | `85202.57` | `86833.79` | 5 | 0 | `106.6%` | `2035128` KB | n/a | +| `ffi.chunked` | `26663.11` | `25049.33` | `26744.73` | 5 | 0 | `116.2%` | `3062632` KB | `751598.0` | +| `ffi.cookies` | `25440.85` | `25385.00` | `25508.05` | 5 | 0 | `116.1%` | `3062632` KB | `764350.5` | +| `ffi.dns` | `26814.07` | `25315.59` | `26863.08` | 5 | 0 | `116.2%` | `3062632` KB | `804524.0` | +| `ffi.hdr40` | `17724.68` | `17643.17` | `17737.64` | 5 | 0 | `107.8%` | `3062632` KB | `531768.5` | +| `ffi.oneshot` | `30925.84` | `30751.13` | `31065.85` | 5 | 0 | `117.7%` | `2035128` KB | no new connections | +| `ffi.post` | `24173.16` | `22771.48` | `24269.40` | 5 | 0 | `115.3%` | `2082632` KB | `702825.5` | +| `ffi.readbody` | `26965.62` | `26790.39` | `27007.54` | 5 | 0 | `116.2%` | `2035128` KB | no new connections | +| `ffi.req30` | `23339.15` | `23306.86` | `23466.05` | 5 | 0 | `114.8%` | `3062632` KB | `700270.0` | +| `ffi.short` | `8721.09` | `8389.79` | `8743.68` | 5 | 0 | `105.9%` | `2082104` KB | `1.0` | +| `ffi.stateful` | `26919.60` | `26402.32` | `26949.99` | 5 | 0 | `116.3%` | `2035128` KB | `807690.0` | +| `ffi.stream` | `27506.05` | `27498.37` | `27630.86` | 5 | 0 | `116.7%` | `2035128` KB | `825047.0` | +| `ffi.tls` | `18655.46` | `18438.79` | `18782.34` | 5 | 0 | `114.9%` | `2083824` KB | `563467.5` | +| `ffi.tlsshort` | `932.16` | `861.20` | `933.17` | 5 | 0 | `102.8%` | `3059584` KB | `1.0` | +| `ffi.trailers` | `28005.24` | `27915.10` | `28128.38` | 5 | 0 | `116.2%` | `3062632` KB | n/a | +| `resty.chunked` | `15366.94` | `15164.85` | `15764.74` | 5 | 0 | `114.1%` | `3062632` KB | `53208.0` | +| `resty.cookies` | `13603.27` | `13582.52` | `13898.90` | 5 | 0 | `113.2%` | `3062632` KB | `51824.9` | +| `resty.dns` | `15634.98` | `15489.53` | `15936.46` | 4 | 1 | `114.2%` | `3566436` KB | `46329.1` | +| `resty.hdr40` | `9218.37` | `9200.90` | `9442.03` | 5 | 0 | `111.2%` | `3062632` KB | `30854.3` | +| `resty.oneshot` | `14510.37` | `14299.32` | `14629.47` | 5 | 0 | `113.4%` | `2035128` KB | `89883.6` | +| `resty.post` | `12780.46` | `12371.84` | `13177.75` | 5 | 0 | `113.8%` | `2082632` KB | `94728.2` | +| `resty.readbody` | `15710.28` | `15562.79` | `15942.52` | 5 | 0 | `114.3%` | `2035128` KB | `86699.9` | +| `resty.req30` | `13283.15` | `13208.27` | `13354.04` | 5 | 0 | `112.8%` | `3062632` KB | `22262.1` | +| `resty.short` | `7152.20` | `7099.21` | `7266.89` | 5 | 0 | `105.5%` | `2082228` KB | `1.0` | +| `resty.stateful` | `15604.60` | `15493.86` | `15795.75` | 5 | 0 | `114.4%` | `2035128` KB | `31003.5` | +| `resty.stream` | `16236.65` | `16021.80` | `17296.03` | 5 | 0 | `114.4%` | `2035128` KB | `61189.6` | +| `resty.tls` | `11866.50` | `11503.14` | `12022.41` | 5 | 0 | `113.6%` | `2085452` KB | `36355.9` | +| `resty.tlsshort` | `1149.50` | `1105.48` | `1152.59` | 5 | 0 | `102.9%` | `3063980` KB | `1.0` | +| `resty.trailers` | `6622.05` | `6543.31` | `6782.34` | 5 | 0 | `105.3%` | `3062632` KB | n/a | + +### Latency (fixed rate, corrected for coordinated omission) + +| case | rate | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | ---: | +| `ffi.chunked` | `9220` req/s | `1.060` ms | `2.260` ms | `2.710` ms | +| `ffi.cookies` | `8161` req/s | `1.040` ms | `2.200` ms | `2.670` ms | +| `ffi.dns` | `9380` req/s | `1.070` ms | `2.300` ms | `2.760` ms | +| `ffi.hdr40` | `5531` req/s | `1.030` ms | `2.110` ms | `2.540` ms | +| `ffi.oneshot` | `8706` req/s | `1.020` ms | `2.210` ms | `2.770` ms | +| `ffi.post` | `7668` req/s | `1.050` ms | `2.220` ms | `2.670` ms | +| `ffi.readbody` | `9426` req/s | `1.060` ms | `2.270` ms | `2.770` ms | +| `ffi.req30` | `7969` req/s | `1.150` ms | `2.370` ms | `2.850` ms | +| `ffi.short` | `4291` req/s | `1.180` ms | `2.490` ms | `3.180` ms | +| `ffi.stateful` | `9362` req/s | `1.070` ms | `2.270` ms | `2.730` ms | +| `ffi.stream` | `9741` req/s | `1.090` ms | `2.370` ms | `2.880` ms | +| `ffi.tls` | `7119` req/s | `1.110` ms | `2.340` ms | `2.790` ms | +| `ffi.tlsshort` | `559` req/s | `4.260` ms | `9.010` ms | `11.840` ms | +| `ffi.trailers` | `3973` req/s | `1.000` ms | `2.100` ms | `2.600` ms | +| `resty.chunked` | `9220` req/s | `1.300` ms | `2.800` ms | `3.400` ms | +| `resty.cookies` | `8161` req/s | `1.290` ms | `2.820` ms | `3.480` ms | +| `resty.dns` | `9380` req/s | `1.300` ms | `2.810` ms | `3.380` ms | +| `resty.hdr40` | `5531` req/s | `1.200` ms | `2.540` ms | `3.080` ms | +| `resty.oneshot` | `8706` req/s | `1.290` ms | `2.830` ms | `3.410` ms | +| `resty.post` | `7668` req/s | `1.320` ms | `2.800` ms | `3.350` ms | +| `resty.readbody` | `9426` req/s | `1.270` ms | `2.760` ms | `3.350` ms | +| `resty.req30` | `7969` req/s | `1.500` ms | `3.020` ms | `3.940` ms | +| `resty.short` | `4291` req/s | `1.300` ms | `2.740` ms | `3.310` ms | +| `resty.stateful` | `9362` req/s | `1.270` ms | `2.750` ms | `3.320` ms | +| `resty.stream` | `9741` req/s | `2.040` ms | `4.650` ms | `5.630` ms | +| `resty.tls` | `7119` req/s | `2.330` ms | `5.410` ms | `6.580` ms | +| `resty.tlsshort` | `559` req/s | `3.330` ms | `5.610` ms | `8.030` ms | +| `resty.trailers` | `3973` req/s | `1.890` ms | `43.520` ms | `109.570` ms | + +### Discarded rows + +Rows failing the saturation gate or reporting non-2xx responses, excluded from every median above: + +- `resty.dns`: 1 of 5 + +## What this shows + +- The FFI client wins throughput on 13 of the 14 shapes. Excluding `trailers`, + the ratios span `1.22x` (`short`, 8721.09 vs 7152.20 QPS) to `2.13x` + (`oneshot`, 30925.84 vs 14510.37 QPS); the common keepalive shapes + (`stream`, `stateful`, `readbody`, `chunked`, `dns`) cluster at `1.69x`-`1.74x`. +- `trailers` is the largest win at `4.23x` (28005.24 vs 6622.05 QPS), and + `5.77x` once the baseline is subtracted from both paths. +- `tlsshort` is the one shape the FFI client LOSES: `0.81x` (932.16 vs 1149.50 + QPS). That shape opens a fresh TLS connection and performs a full handshake + for every request, so the measurement is dominated by handshake cost rather + than by client-side request/response handling. Use `lua-resty-http` if your + workload really does handshake per request. +- Latency: the FFI client wins p50, p99 and p99.9 on every shape except + `tlsshort`. Typical keepalive p50 is `1.02`-`1.15` ms for FFI against + `1.27`-`1.50` ms for `resty.http`, and p99.9 is `2.54`-`3.18` ms against + `3.08`-`3.94` ms. +- On `tlsshort` the FFI client loses the latency picture too: p50 `4.260` vs + `3.330` ms, p99 `9.010` vs `5.610` ms, p99.9 `11.840` vs `8.030` ms. +- Connection reuse differs sharply. On keepalive shapes the FFI client either + opened no new upstream connections at all during the measured window + (`oneshot`, `readbody`) or reached 5.3e5-8.3e5 requests per new connection, + while `resty.http` sat at 2.2e4-9.5e4. Both clients are at `1.0` on the + deliberately non-keepalive shapes (`short`, `tlsshort`). +- Repeat-to-repeat spread was small. Across the 5 throughput repeats, 20 of the + 29 cases had a max-min spread under 4% of the maximum; the widest were + `ffi.tlsshort` at 7.71% (861.20-933.17), `resty.stream` at 7.37% + (16021.80-17296.03) and `ffi.chunked` at 6.34% (25049.33-26744.73). The + baseline varied by 1.88% (85202.57-86833.79). +- The `trailers` rows carry no requests-per-connection figure because the + upstream connection counters read 0/0 for both clients on that shape; the + QPS and latency numbers for `trailers` are unaffected. + +## Correction: the `ffi.stateful` p99.9 tail claim is disproven + +An earlier run suggested that `ffi.stateful` had a bad p99.9 tail of 20.94 ms +against `resty.stateful` at 6.80 ms. That was a single-sample artifact. This run +used 3 latency repeats and the result does not reproduce: + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.stateful` | `1.070` ms | `2.270` ms | `2.730` ms | +| `resty.stateful` | `1.270` ms | `2.750` ms | `3.320` ms | + +The FFI client wins the tail on `stateful`, it does not lose it. The three +individual `ffi.stateful` p99.9 samples were 2.720, 2.730 and 2.730 ms — tightly +clustered, with nothing near 20 ms — against 3.320, 3.340 and 3.280 ms for +`resty.stateful`. Do not act on the old 20.94 ms claim; it is superseded by +this run. + +## Reverse finding: `resty.trailers` tail latency + +The one genuinely large tail in the whole run belongs to `lua-resty-http`, not +to the FFI client. On the `trailers` shape at 3973 req/s: + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.trailers` | `1.000` ms | `2.100` ms | `2.600` ms | +| `resty.trailers` | `1.890` ms | `43.520` ms | `109.570` ms | + +That is a 42x gap at p99.9. It is also unstable across repeats: the three +`resty.trailers` p99.9 samples were 109.570, 196.610 and 11.970 ms (p99: 43.520, +126.850 and 3.600 ms), while `ffi.trailers` produced 2.690, 2.600 and 2.560 ms. +The median reported above is the middle sample in each case. + +--- + +# TLS re-measurement after the trust-store fix (`fix`) + +Run finished 2026-08-04, after the run recorded above. This section records a +targeted re-measurement of the TLS shapes following the fix in +`src/ngx_http_ffi_client_request.c` that stopped the client rescanning the CA +directory on every handshake. It supersedes the `tls` and `tlsshort` numbers in +the `full` run above; every other shape in this file is unaffected and still +stands. + +## What was run + +- Run parameters: `BENCH_RUN_ID=fix`, `BENCH_REPEATS=5`, + `BENCH_LATENCY_REPEATS=3`, `BENCH_CASES="ffi.tls resty.tls ffi.tlsshort + resty.tlsshort ffi.tlsverify resty.tlsverify ffi.tlsverifyshort + resty.tlsverifyshort"`. +- Same topology as the `full` run: target worker pinned to CPU 0, upstream on + CPUs 1-5, `wrk2` on CPUs 6-7. +- The run exited 0. The fairness audit passed for all four shapes. All 40 + throughput windows passed the saturation gate with zero non-2xx and zero + socket errors; nothing was discarded. +- `requests_per_connection` is `1.0` for both clients on all four short-lived + cases, so those windows really are measuring a fresh connection and handshake + per request rather than a partly-pooled result. +- Raw rows: `benchmark/.run-fix/results/fix.csv`, 64 data rows. + +## The defect + +`ngx_http_ffi_client_ssl_ctx()` called `SSL_CTX_set_default_verify_paths()` +unconditionally. That registers the hashed CA directory as +`X509_FILETYPE_DEFAULT`, and under `DEFAULT` OpenSSL enumerates the entire +directory on *every* handshake instead of opening the single `.N` file the +lookup needs. + +Found with `perf`, sampling the pinned target worker for 40 s during each case +(~40000 samples each). Every top entry in the `perf diff` is kernel +directory-reading code, present only on the FFI side: + +``` +Baseline Delta Abs Symbol + +2.04% [k] filldir64 + +2.01% [k] ext4_htree_store_dirent + +1.82% [k] str2hashbuf_signed + +1.76% [k] half_md4_transform + +1.05% [k] htree_dirblock_to_tree +``` + +| | C FFI | `lua-resty-http` | +| --- | ---: | ---: | +| CPU in directory-lookup code | `11.83%` | `0.42%` | +| CPU in the kernel overall | `36.7%` | `17.9%` | + +Confirmed at the syscall level under load: 2132 `openat` plus 4264 `getdents64` +for ~2168 requests, one full sweep of a 245-entry directory per handshake. The +same trace during `resty.tlsshort` caught none. After the fix, tracing both +`ffi.tlsshort` and `ffi.tlsverifyshort` caught none either. + +The cost is the filetype, not hash-directory lookup and not lookup failure. +Reproduced independently of this module with `openssl s_client` against the +benchmark upstream: + +| trust store setup | `getdents64` calls | verify result | +| --- | ---: | --- | +| `set_default_verify_paths()` | `4` | 18 (self-signed) | +| explicit `-CAfile` system bundle | `0` | 18 | +| explicit `-CApath` system dir | `0` | 18 | +| explicit `-CApath`, issuer present | `0` | 0 (ok) | + +## Throughput (median of 5 saturated repeats) + +| shape | C FFI | `lua-resty-http` | ratio | +| --- | ---: | ---: | ---: | +| `tls` | `18276.36` req/s | `11926.11` req/s | `1.53x` | +| `tlsshort` | `1256.86` req/s | `1142.44` req/s | `1.10x` | +| `tlsverify` | `17542.73` req/s | `11861.35` req/s | `1.48x` | +| `tlsverifyshort` | `1228.67` req/s | `1136.91` req/s | `1.08x` | + +Against the `full` run above: + +| shape | before | after | ratio before | ratio after | +| --- | ---: | ---: | ---: | ---: | +| `ffi.tlsshort` | `932.16` req/s | `1256.86` req/s | `0.81x` | `1.10x` | +| `ffi.tls` | `18655.46` req/s | `18276.36` req/s | `1.57x` | `1.53x` | + +`tlsshort` gains 35% and turns the only losing shape in the matrix into a win. +Pooled `tls` is unchanged within run-to-run spread, as expected: it amortises +one handshake over many requests, so the sweep was never a large share of its +cost. + +`tlsverify` and `tlsverifyshort` are new. The matrix had never exercised +`ssl_verify = true` — all four call sites in `benchmark/lua/bench.lua` hardcoded +`false` — so the verifying path was previously unmeasured. Verification costs +the FFI client about 4% on pooled TLS (`18276` to `17543`) and roughly 2% on the +short-lived shape, which is real chain-checking work rather than the pathological +sweep. + +## Latency (fixed rate, corrected for coordinated omission) + +Median of 3 repeats. **These numbers are less trustworthy than the throughput +figures above and should not be quoted for the short-lived shapes** — see the +caveat below. + +| case | p50 | p99 | p99.9 | +| --- | ---: | ---: | ---: | +| `ffi.tls` | `1.650` ms | `4.130` ms | `4.880` ms | +| `resty.tls` | `1.260` ms | `2.700` ms | `3.270` ms | +| `ffi.tlsshort` | `3.500` ms | `5.810` ms | `7.710` ms | +| `resty.tlsshort` | `3.620` ms | `6.280` ms | `8.330` ms | +| `ffi.tlsverify` | `1.170` ms | `2.460` ms | `2.930` ms | +| `resty.tlsverify` | `2.330` ms | `5.460` ms | `6.420` ms | +| `ffi.tlsverifyshort` | `23.220` ms | `46.720` ms | `48.380` ms | +| `resty.tlsverifyshort` | `3.430` ms | `5.660` ms | `7.000` ms | + +### Caveat: rate attainment, not latency + +The latency phase drives each pair at a fixed rate — 60% of the slower side's +saturated throughput. Several windows failed to reach that rate by roughly 2%, +and wrk2's coordinated-omission correction converts a small rate shortfall into +a large apparent latency, so those rows measure rate attainment rather than +service time: + +| case | repeat | achieved / target req/s | p50 | worker CPU | +| --- | ---: | ---: | ---: | ---: | +| `ffi.tlsshort` | 1 | `683.85` / 685 | `3.500` ms | `57.0%` | +| `ffi.tlsshort` | 2 | `672.02` / 685 | `24.400` ms | `55.8%` | +| `ffi.tlsshort` | 3 | `683.89` / 685 | `3.490` ms | `57.5%` | +| `ffi.tlsverifyshort` | 1 | `669.37` / 682 | `23.220` ms | `56.0%` | +| `ffi.tlsverifyshort` | 2 | `669.38` / 682 | `23.500` ms | `56.4%` | +| `ffi.tlsverifyshort` | 3 | `680.99` / 682 | `3.300` ms | `58.0%` | + +Every window that hit its target rate produced a ~3.3-3.5 ms p50; every window +that missed produced ~23-24 ms. The worker had roughly 42% CPU headroom +throughout, so this is not a CPU limit. + +This is not specific to the FFI client, though it hit it more often here. +`resty.tlsverify` missed the rate on repeats 2 and 3 (`6982` against a `7088` +target) and its p50 went from `1.270` ms to `2.330`/`2.410` ms; `ffi.tls` +repeats 1 and 2 missed (`7020` against `7127`) with the same effect. Where both +sides hit their target — `tlsverify` repeat 1, `tlsverifyshort` repeat 3 — the +FFI client is faster at every percentile. + +The underlying cause is not yet understood. Candidates: connection-teardown +asymmetry over which side holds `TIME_WAIT`, ephemeral port pressure at ~683 new +connections per second, or a stall in the connect path. Until it is diagnosed, +no latency claim should be published for the short-lived TLS shapes. diff --git a/benchmark/run.sh b/benchmark/run.sh index ef944bd..c580f6e 100755 --- a/benchmark/run.sh +++ b/benchmark/run.sh @@ -6,32 +6,65 @@ ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" RUN_DIR="${RUN_DIR:-$ROOT_DIR/benchmark/.run}" TARGET_PREFIX="$RUN_DIR/target" UPSTREAM_PREFIX="$RUN_DIR/upstream" +RESULTS_DIR="${BENCH_RESULTS_DIR:-$RUN_DIR/results}" +CASES_FILE="${BENCH_CASES_FILE:-$ROOT_DIR/benchmark/cases.txt}" TARGET_PORT="${TARGET_PORT:-19840}" UPSTREAM_PORT="${UPSTREAM_PORT:-19841}" +UPSTREAM_TLS_PORT="${UPSTREAM_TLS_PORT:-19842}" +UPSTREAM_TRAILER_PORT="${UPSTREAM_TRAILER_PORT:-19843}" +UPSTREAM_HOST="${UPSTREAM_HOST:-127.0.0.1}" TARGET_CPU="${TARGET_CPU:-0}" RESPONSE_SIZE="${RESPONSE_SIZE:-1024}" KEEPALIVE_POOL_SIZE="${KEEPALIVE_POOL_SIZE:-120}" KEEPALIVE_IDLE_TIMEOUT="${KEEPALIVE_IDLE_TIMEOUT:-60000}" +CLIENT_TIMEOUT="${CLIENT_TIMEOUT:-60000}" +DNS_HOST="${DNS_HOST:-localhost}" +TLS_SERVER_NAME="${TLS_SERVER_NAME:-bench.local}" +CA_FILE="${CA_FILE:-$RUN_DIR/target/conf/ca.crt}" + WRK_THREADS="${WRK_THREADS:-2}" WRK_CONNECTIONS="${WRK_CONNECTIONS:-100}" WRK_DURATION="${WRK_DURATION:-30s}" +WRK_WARMUP="${WRK_WARMUP:-10s}" WRK_RATE="${WRK_RATE:-100000}" CPU_SATURATION_THRESHOLD="${CPU_SATURATION_THRESHOLD:-95}" + +BENCH_REPEATS="${BENCH_REPEATS:-5}" +BENCH_LATENCY="${BENCH_LATENCY:-1}" +BENCH_LATENCY_FRACTION="${BENCH_LATENCY_FRACTION:-0.6}" +# Tail percentiles are the noisiest thing here, so one window is not evidence. +BENCH_LATENCY_REPEATS="${BENCH_LATENCY_REPEATS:-3}" +BENCH_GC="${BENCH_GC:-1}" +BENCH_AUDIT="${BENCH_AUDIT:-1}" +BENCH_STRICT="${BENCH_STRICT:-0}" +DROPPED=() BENCH_DRY_RUN="${BENCH_DRY_RUN:-0}" +BENCH_PARSER="${BENCH_PARSER:-${NGX_HTTP_FFI_CLIENT_USE_LLHTTP:-llhttp}}" + +# The default set is the 2x2 that produces the headline ratio. BENCH_CASES=all +# runs every case in benchmark/cases.txt. +BENCH_CASES="${BENCH_CASES:-baseline ffi.oneshot resty.oneshot ffi.stateful resty.stateful}" TEST_NGINX_BINARY="${TEST_NGINX_BINARY:-}" +BENCH_LUA_DIR="$ROOT_DIR/benchmark/lua" TEST_NGINX_LUA_PACKAGE_PATH="${TEST_NGINX_LUA_PACKAGE_PATH:-$ROOT_DIR/lib/?.lua;$ROOT_DIR/lib/?/init.lua}" +LUA_PACKAGE_PATH="$TEST_NGINX_LUA_PACKAGE_PATH;$BENCH_LUA_DIR/?.lua" WRK2_BIN="${WRK2_BIN:-${WRK_BIN:-wrk}}" CURL_BIN="${CURL_BIN:-curl}" TASKSET_BIN="${TASKSET_BIN:-taskset}" +OPENSSL_BIN="${OPENSSL_BIN:-openssl}" die() { printf 'error: %s\n' "$*" >&2 exit 1 } +warn() { + printf 'warning: %s\n' "$*" >&2 +} + need_cmd() { command -v "$1" >/dev/null 2>&1 || die "$1 is required" } @@ -100,6 +133,57 @@ cpu_lists_overlap "$TARGET_CPU" "$UPSTREAM_CPUS" && cpu_lists_overlap "$TARGET_CPU" "$WRK_CPUS" && die "TARGET_CPU overlaps WRK_CPUS" +# The topology this benchmark assumes is a pinned target worker, an upstream +# that cannot be the bottleneck, and a load generator that competes with +# neither. Below eight cores that is not available, and the numbers describe the +# host rather than the client. +if (( NCPU < 8 )); then + warn "only $NCPU cores online; the plan's topology wants 8+ (target on 1," \ + "upstream on 5, wrk2 on 2). Results here are regression smoke, not" \ + "headline numbers." +fi + +default_resolver() { + awk '/^nameserver/ { print $2; exit }' /etc/resolv.conf 2>/dev/null || + printf '127.0.0.1\n' +} + +RESOLVER="${BENCH_RESOLVER:-$(default_resolver)}" +[[ -n "$RESOLVER" ]] || RESOLVER="127.0.0.1" + +# ===== case list ===== + +read_case_file() { + sed -e 's/#.*//' -e 's/[[:space:]]//g' "$CASES_FILE" | grep -v '^$' +} + +resolve_cases() { + local requested="$1" + + if [[ "$requested" == "all" ]]; then + printf 'baseline\n' + read_case_file + return + fi + + printf '%s\n' $requested +} + +mapfile -t CASES < <(resolve_cases "$BENCH_CASES") +(( ${#CASES[@]} > 0 )) || die "BENCH_CASES resolved to nothing" + +case_path() { + local name="$1" + + if [[ "$name" == "baseline" ]]; then + printf '/bench/baseline' + else + printf '/bench/%s' "$name" + fi +} + +# ===== template rendering ===== + sed_escape() { printf '%s' "$1" | sed -e 's/[&|]/\\&/g' } @@ -109,16 +193,57 @@ render_template() { local output="$2" sed \ - -e "s|{{LUA_PACKAGE_PATH}}|$(sed_escape "$TEST_NGINX_LUA_PACKAGE_PATH")|g" \ + -e "s|{{LUA_PACKAGE_PATH}}|$(sed_escape "$LUA_PACKAGE_PATH")|g" \ -e "s|{{TARGET_PORT}}|$(sed_escape "$TARGET_PORT")|g" \ + -e "s|{{UPSTREAM_HOST}}|$(sed_escape "$UPSTREAM_HOST")|g" \ -e "s|{{UPSTREAM_PORT}}|$(sed_escape "$UPSTREAM_PORT")|g" \ + -e "s|{{UPSTREAM_TLS_PORT}}|$(sed_escape "$UPSTREAM_TLS_PORT")|g" \ + -e "s|{{UPSTREAM_TRAILER_PORT}}|$(sed_escape "$UPSTREAM_TRAILER_PORT")|g" \ -e "s|{{UPSTREAM_WORKERS}}|$(sed_escape "$UPSTREAM_WORKERS")|g" \ -e "s|{{RESPONSE_SIZE}}|$(sed_escape "$RESPONSE_SIZE")|g" \ -e "s|{{KEEPALIVE_POOL_SIZE}}|$(sed_escape "$KEEPALIVE_POOL_SIZE")|g" \ -e "s|{{KEEPALIVE_IDLE_TIMEOUT}}|$(sed_escape "$KEEPALIVE_IDLE_TIMEOUT")|g" \ + -e "s|{{CLIENT_TIMEOUT}}|$(sed_escape "$CLIENT_TIMEOUT")|g" \ + -e "s|{{DNS_HOST}}|$(sed_escape "$DNS_HOST")|g" \ + -e "s|{{TLS_SERVER_NAME}}|$(sed_escape "$TLS_SERVER_NAME")|g" \ + -e "s|{{CA_FILE}}|$(sed_escape "$CA_FILE")|g" \ + -e "s|{{RESOLVER}}|$(sed_escape "$RESOLVER")|g" \ "$template" > "$output" } +# One exact location per case, generated from cases.txt. Exact rather than one +# regex location on purpose: a regex match would land on the measured worker's +# hot path for every request of every case. +generate_locations() { + local output="$1" + local name + + : > "$output" + while read -r name; do + cat >> "$output" </dev/null 2>&1 || + die "failed to generate a TLS certificate with $OPENSSL_BIN" +} + prepare_prefix() { local prefix="$1" @@ -136,31 +261,54 @@ prepare_prefix() { prepare_configs() { prepare_prefix "$UPSTREAM_PREFIX" prepare_prefix "$TARGET_PREFIX" + mkdir -p "$RESULTS_DIR" render_template "$ROOT_DIR/benchmark/conf/upstream.nginx.conf" \ "$UPSTREAM_PREFIX/conf/nginx.conf" render_template "$ROOT_DIR/benchmark/conf/target.nginx.conf" \ "$TARGET_PREFIX/conf/nginx.conf" + + generate_locations "$TARGET_PREFIX/conf/bench_locations.conf" + # ssl_certificate resolves relative paths against the conf prefix + generate_cert "$UPSTREAM_PREFIX/conf" + + # The cert is self-signed with CA:TRUE, so it is its own root: handing the + # target a copy is all the verify-on cases need to trust the upstream. + # lua_ssl_trusted_certificate is read at config load, so it must be in place + # before nginx starts. + cp "$UPSTREAM_PREFIX/conf/tls.crt" "$CA_FILE" || + die "failed to install the trusted CA at $CA_FILE" } print_config() { cat </dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/baseline" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/smoke/ffi-short" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/smoke/resty-http-short" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/ffi" >/dev/null - "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/resty-http" >/dev/null - printf 'Smoke checks passed\n' -} - target_worker_pid() { local master_pid @@ -230,10 +370,10 @@ target_worker_pid() { ps -o pid= --ppid "$master_pid" | awk 'NR == 1 { print $1 }' } -read_proc_ticks() { - local pid="$1" +# ===== CPU sampling ===== - awk '{ print $14 + $15 }' "/proc/$pid/stat" +read_proc_ticks() { + awk '{ print $14 + $15 }' "/proc/$1/stat" } read_total_ticks() { @@ -246,6 +386,10 @@ read_total_ticks() { }' /proc/stat } +read_rss_kb() { + awk '/^VmRSS:/ { print $2 }' "/proc/$1/status" 2>/dev/null || printf '0\n' +} + sample_cpu_once() { local pid="$1" local proc_before proc_after total_before total_after proc_delta total_delta @@ -281,141 +425,538 @@ sample_cpu_loop() { done } -summarize_cpu() { - local samples="$1" - - awk ' +cpu_stat() { + awk -v want="$2" ' NF { sum += $1 - if ($1 > max) { - max = $1 - } + if ($1 > max) { max = $1 } n += 1 } END { - if (n == 0) { - printf "avg=0.0 max=0.0 samples=0" - } else { - printf "avg=%.1f max=%.1f samples=%d", sum / n, max, n - } + if (n == 0) { print "0"; exit } + if (want == "avg") { printf "%.1f\n", sum / n } + else if (want == "max") { printf "%.1f\n", max } + else { print n } } - ' "$samples" + ' "$1" } -cpu_saturated() { - local samples="$1" +# ===== wrk2 output parsing ===== - awk -v threshold="$CPU_SATURATION_THRESHOLD" ' - NF { - sum += $1 - n += 1 - } - END { - if (n == 0) { - exit 1 +extract_qps() { + awk '/Requests\/sec:/ { qps = $2 } END { print (qps == "" ? "0" : qps) }' "$1" +} + +extract_non2xx() { + awk '/Non-2xx or 3xx responses:/ { n = $NF } END { print (n == "" ? "0" : n) }' "$1" +} + +extract_socket_errors() { + awk ' + /Socket errors:/ { + total = 0 + for (i = 1; i <= NF; i++) { + if ($i ~ /^[0-9]+,?$/) { + v = $i + sub(",", "", v) + total += v + } } - exit !((sum / n) >= threshold) + print total + found = 1 } - ' "$samples" + END { if (!found) print "0" } + ' "$1" } -case_stem() { - printf '%s' "$1" | sed -e 's/[^A-Za-z0-9_-]/_/g' +# wrk2 with -L -U prints two distributions: the recorded one, which corrects for +# coordinated omission by accounting for delayed request starts, and the +# uncorrected one. The corrected figures are the point of running wrk2 at a +# fixed rate at all, so they are the default; BENCH_LATENCY_MODE=uncorrected +# selects the other block. Percentile values come in whatever unit fits, so they +# are normalised to milliseconds before anything compares them. +extract_percentile() { + local output="$1" + local pct="$2" + local mode="${BENCH_LATENCY_MODE:-corrected}" + + awk -v pct="$pct" -v mode="$mode" ' + function to_ms(v) { + if (v ~ /us$/) { sub(/us$/, "", v); return v / 1000 } + if (v ~ /ms$/) { sub(/ms$/, "", v); return v + 0 } + if (v ~ /m$/) { sub(/m$/, "", v); return v * 60000 } + if (v ~ /s$/) { sub(/s$/, "", v); return v * 1000 } + return v + 0 + } + /Latency Distribution/ { + in_block = (($0 ~ /Uncorrected/) == (mode == "uncorrected")) + next + } + /Detailed Percentile spectrum/ { in_block = 0 } + in_block && $1 ~ /%$/ { + p = $1 + sub(/%$/, "", p) + if (p + 0 == pct + 0 && !seen) { + printf "%.3f\n", to_ms($2) + seen = 1 + } + } + END { if (!seen) print "" } + ' "$output" } -qps_file_for_label() { - local label="$1" +# ===== upstream reuse accounting ===== - printf '%s/%s.qps' "$RUN_DIR" "$(case_stem "$label")" +upstream_stats_reset() { + "$CURL_BIN" -fsS "http://127.0.0.1:$UPSTREAM_PORT/stats/reset" \ + >/dev/null 2>&1 || true } -extract_qps() { - local output="$1" +upstream_stat() { + "$CURL_BIN" -fsS "http://127.0.0.1:$UPSTREAM_PORT/stats" 2>/dev/null | + awk -v key="$1" '$1 == key { print $2 }' +} + +# ===== results ===== - awk '/Requests\/sec:/ { qps = $2 } END { if (qps == "") exit 1; print qps }' \ - "$output" +RUN_ID="${BENCH_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}" +RESULTS_CSV="$RESULTS_DIR/$RUN_ID.csv" + +CSV_HEADER='run_id,phase,repeat,case,qps,rate,non2xx,socket_errors,p50_ms,p99_ms,p999_ms,cpu_avg,cpu_max,cpu_samples,saturated,rss_kb,upstream_connections,upstream_requests,requests_per_connection,response_size,connections,threads,duration,parser' + +init_results() { + mkdir -p "$RESULTS_DIR" + printf '%s\n' "$CSV_HEADER" > "$RESULTS_CSV" + printf 'Writing rows to %s\n' "$RESULTS_CSV" } +# ===== one measured run ===== + run_wrk_case() { - local label="$1" - local path="$2" - local worker_pid stem samples output qps sampler_pid + local phase="$1" + local repeat="$2" + local name="$3" + local duration="$4" + local rate="$5" + + local path stem samples output worker_pid sampler_pid + local qps non2xx sockerr cpu_avg cpu_max cpu_n saturated rss + local up_conn up_req rpc p50 p99 p999 + + path="$(case_path "$name")" + stem="$(printf '%s' "${phase}_${repeat}_${name}" | tr -c 'A-Za-z0-9_-' '_')" + samples="$RUN_DIR/$stem.cpu" + output="$RUN_DIR/$stem.wrk" worker_pid="$(target_worker_pid)" [[ -n "$worker_pid" ]] || die "failed to find target worker pid" - stem="$(case_stem "$label")" - samples="$RUN_DIR/$stem.cpu" - output="$RUN_DIR/$stem.wrk" + printf '\n== %s repeat %s: %s (%s, rate %s) ==\n' \ + "$phase" "$repeat" "$name" "$duration" "$rate" + + # warmup, discarded: fills the keepalive pools and lets the JIT settle so + # the measured window is not paying for either + if [[ "$WRK_WARMUP" != "0" && "$WRK_WARMUP" != "0s" ]]; then + "$TASKSET_BIN" -c "$WRK_CPUS" "$WRK2_BIN" \ + -t "$WRK_THREADS" -c "$WRK_CONNECTIONS" -d "$WRK_WARMUP" \ + -R "$rate" \ + -s "$ROOT_DIR/benchmark/wrk/headers.lua" \ + "http://127.0.0.1:$TARGET_PORT$path" >/dev/null 2>&1 || true + fi + + upstream_stats_reset - printf '\n== %s ==\n' "$label" sample_cpu_loop "$worker_pid" "$samples" & sampler_pid="$!" "$TASKSET_BIN" -c "$WRK_CPUS" "$WRK2_BIN" \ - -t "$WRK_THREADS" \ - -c "$WRK_CONNECTIONS" \ - -d "$WRK_DURATION" \ - -R "$WRK_RATE" \ - -L \ - -U \ + -t "$WRK_THREADS" -c "$WRK_CONNECTIONS" -d "$duration" \ + -R "$rate" -L -U \ -s "$ROOT_DIR/benchmark/wrk/headers.lua" \ "http://127.0.0.1:$TARGET_PORT$path" | tee "$output" + rss="$(read_rss_kb "$worker_pid")" + kill "$sampler_pid" >/dev/null 2>&1 || true wait "$sampler_pid" >/dev/null 2>&1 || true - qps="$(extract_qps "$output")" || - die "failed to parse Requests/sec from $output" - printf '%s\n' "$qps" > "$(qps_file_for_label "$label")" - printf 'actual throughput: %s requests/sec\n' "$qps" + qps="$(extract_qps "$output")" + non2xx="$(extract_non2xx "$output")" + sockerr="$(extract_socket_errors "$output")" + cpu_avg="$(cpu_stat "$samples" avg)" + cpu_max="$(cpu_stat "$samples" max)" + cpu_n="$(cpu_stat "$samples" n)" + p50="$(extract_percentile "$output" 50)" + p99="$(extract_percentile "$output" 99)" + p999="$(extract_percentile "$output" 99.9)" + + up_conn="$(upstream_stat connections)" + up_req="$(upstream_stat requests)" + up_conn="${up_conn:-0}" + up_req="${up_req:-0}" + # The counters are reset after the warmup, so they describe the steady + # state: connections opened during the measured window. Zero new + # connections against a non-zero request count is perfect pool reuse, which + # is the result the keepalive cases want; a ratio near 1 means the pool is + # not reusing anything and the case is measuring connection setup. + rpc="$(awk -v c="$up_conn" -v r="$up_req" 'BEGIN { + if (c + 0 > 0) printf "%.1f", r / c + else if (r + 0 > 0) print "inf" + else print "" + }')" + + saturated=no + if awk -v t="$CPU_SATURATION_THRESHOLD" -v a="$cpu_avg" \ + 'BEGIN { exit !(a + 0 >= t + 0) }'; then + saturated=yes + fi + + printf 'qps=%s non2xx=%s socket_errors=%s cpu_avg=%s%% rss=%sKB\n' \ + "$qps" "$non2xx" "$sockerr" "$cpu_avg" "$rss" + printf 'latency p50=%sms p99=%sms p99.9=%sms\n' "$p50" "$p99" "$p999" + + if [[ "$name" != "baseline" ]]; then + printf 'upstream: %s new connections, %s requests' "$up_conn" "$up_req" + if [[ -n "$rpc" ]]; then + printf ', %s requests/connection\n' "$rpc" + else + printf '\n' + fi + + # a keepalive case that opens roughly one connection per request is not + # exercising the pool, whatever else the row says + case "$name" in + *.short|*.tlsshort) ;; + *) + if [[ "$rpc" != "inf" && -n "$rpc" ]] && + awk -v v="$rpc" 'BEGIN { exit !(v + 0 < 2) }' + then + warn "$name reused almost nothing (${rpc} requests per" \ + "connection); this is not a keepalive result" + fi + ;; + esac + fi - printf 'target worker CPU: %s\n' "$(summarize_cpu "$samples")" - if cpu_saturated "$samples"; then - printf 'saturation: valid (average CPU >= %s%%)\n' \ - "$CPU_SATURATION_THRESHOLD" + if [[ "$non2xx" != "0" ]]; then + warn "$name reported $non2xx non-2xx responses; this row is invalid" + fi + + if [[ "$phase" == "throughput" && "$saturated" == "no" ]]; then + warn "$name average target worker CPU ${cpu_avg}% is below" \ + "${CPU_SATURATION_THRESHOLD}%; this row is not a saturated result" + fi + + printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n' \ + "$RUN_ID" "$phase" "$repeat" "$name" "$qps" "$rate" "$non2xx" \ + "$sockerr" "$p50" "$p99" "$p999" "$cpu_avg" "$cpu_max" "$cpu_n" \ + "$saturated" "$rss" "$up_conn" "$up_req" "$rpc" "$RESPONSE_SIZE" \ + "$WRK_CONNECTIONS" "$WRK_THREADS" "$duration" "$BENCH_PARSER" \ + >> "$RESULTS_CSV" +} + +# ===== checks before any load ===== + +smoke_check() { + printf 'Running smoke checks\n' + + "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/ready" >/dev/null || + die "target /ready failed" + + # the case table the worker actually has, against the file the runner drives + local worker_cases file_cases + worker_cases="$("$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/cases" | + sort)" + file_cases="$(read_case_file | sort)" + + if [[ "$worker_cases" != "$file_cases" ]]; then + printf 'case table drift between benchmark/cases.txt and bench.lua:\n' >&2 + diff <(printf '%s\n' "$file_cases") <(printf '%s\n' "$worker_cases") >&2 || + true + die "case list mismatch" + fi + + "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/bench/baseline" >/dev/null || + die "baseline failed" + + # A case that cannot complete once is not going to produce a meaningful + # number under load. Not every case works on every build, though: the + # hand-written parser has no chunked framing, so its chunked and trailer + # cases fail here by design. Those are dropped and named in the summary + # rather than aborting the whole matrix. BENCH_STRICT=1 aborts instead. + local name kept=() + for name in "${CASES[@]}"; do + if [[ "$name" == "baseline" ]]; then + kept+=("$name") + continue + fi + + if ! "$CURL_BIN" -fsS "http://127.0.0.1:$TARGET_PORT/audit/$name" \ + > "$RUN_DIR/audit-$name.txt" 2>&1 + then + if [[ "$BENCH_STRICT" == "1" ]]; then + printf 'audit output: %s\n' \ + "$(cat "$RUN_DIR/audit-$name.txt")" >&2 + die "case $name failed its smoke check" + fi + + warn "case $name failed its smoke check and is excluded from this" \ + "run; it is not supported by this build or configuration" + DROPPED+=("$name") + continue + fi + + printf ' %s\n' "$(cat "$RUN_DIR/audit-$name.txt")" + kept+=("$name") + done + + CASES=("${kept[@]}") + + local remaining=0 + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] || remaining=$((remaining + 1)) + done + + (( remaining > 0 )) || die "every client case failed its smoke check" + + if (( ${#DROPPED[@]} )); then + printf 'Smoke checks passed for %d cases, %d excluded\n' \ + "$remaining" "${#DROPPED[@]}" else - printf 'saturation: suspicious (average CPU < %s%%; increase wrk load or check upstream capacity)\n' \ - "$CPU_SATURATION_THRESHOLD" + printf 'Smoke checks passed\n' fi } -print_outbound_estimate() { - local baseline_qps ffi_qps resty_qps +# Headers each client writes for itself rather than on the caller's behalf. A +# difference here is the client's own contract and is reported, not failed: the +# FFI client always emits Connection and Content-Length, lua-resty-http omits +# both on a bodyless GET. +GENERATED_HEADERS='^(host|connection|content-length|user-agent|transfer-encoding|accept|accept-encoding):' - baseline_qps="$(cat "$(qps_file_for_label "baseline")")" - ffi_qps="$(cat "$(qps_file_for_label "C FFI keepalive")")" - resty_qps="$(cat "$(qps_file_for_label "resty.http keepalive")")" +request_line() { + head -n 1 "$1" +} - awk -v baseline_qps="$baseline_qps" \ - -v ffi_qps="$ffi_qps" \ - -v resty_qps="$resty_qps" ' - BEGIN { - baseline_total_us = 1000000 / baseline_qps - ffi_total_us = 1000000 / ffi_qps - resty_total_us = 1000000 / resty_qps - ffi_outbound_us = ffi_total_us - baseline_total_us - resty_outbound_us = resty_total_us - baseline_total_us +caller_headers() { + tail -n +2 "$1" | grep -Ev -i "$GENERATED_HEADERS" | grep -v '^$' | sort +} - print "" - print "== Baseline-derived outbound estimate ==" - printf "baseline downstream+handler: %.3f us/request\n", baseline_total_us - printf "C FFI total: %.3f us/request, estimated outbound: %.3f us/request\n", ffi_total_us, ffi_outbound_us - printf "resty.http total: %.3f us/request, estimated outbound: %.3f us/request\n", resty_total_us, resty_outbound_us +generated_headers() { + tail -n +2 "$1" | grep -E -i "$GENERATED_HEADERS" | sort +} - if (ffi_outbound_us <= 0 || resty_outbound_us <= 0) { - print "estimated outbound ratio: unavailable because baseline throughput is not above both client cases" - } else { - printf "estimated outbound ratio (resty.http / C FFI): %.2fx\n", resty_outbound_us / ffi_outbound_us +# A1: prove the two columns of a pair send the same request. The upstream echoes +# the raw header block it received. The request line and every caller-supplied +# header must match, or the pair is not measuring the same request; headers the +# clients generate themselves are reported separately. +fairness_audit() { + printf '\n== Fairness audit ==\n' + + local shapes=() name shape seen ffi_out resty_out failures=0 + + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + shape="${name#*.}" + + seen=0 + local s + for s in "${shapes[@]:-}"; do + [[ "$s" == "$shape" ]] && seen=1 + done + (( seen )) || shapes+=("$shape") + done + + for shape in "${shapes[@]:-}"; do + ffi_out="$RUN_DIR/echo-ffi.$shape.txt" + resty_out="$RUN_DIR/echo-resty.$shape.txt" + + if ! "$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/echo/ffi.$shape" > "$ffi_out" 2>&1 + then + warn "echo for ffi.$shape failed: $(cat "$ffi_out")" + failures=$((failures + 1)) + continue + fi + + if ! "$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/echo/resty.$shape" > "$resty_out" 2>&1 + then + warn "echo for resty.$shape failed: $(cat "$resty_out")" + failures=$((failures + 1)) + continue + fi + + local ok=1 + + if [[ "$(request_line "$ffi_out")" != "$(request_line "$resty_out")" ]] + then + printf ' %-10s REQUEST LINE DIFFERS:\n' "$shape" + printf ' ffi: %s\n' "$(request_line "$ffi_out")" + printf ' resty: %s\n' "$(request_line "$resty_out")" + ok=0 + fi + + # header order is not part of the contract, so compare as sets + if ! diff <(caller_headers "$ffi_out") <(caller_headers "$resty_out") \ + > "$RUN_DIR/echo-$shape.diff" + then + printf ' %-10s CALLER HEADERS DIFFER:\n' "$shape" + sed -e 's/^/ /' "$RUN_DIR/echo-$shape.diff" + ok=0 + fi + + if (( ok )); then + printf ' %-10s same request line, same caller headers\n' "$shape" + else + failures=$((failures + 1)) + fi + + if ! diff <(generated_headers "$ffi_out") \ + <(generated_headers "$resty_out") > "$RUN_DIR/echo-$shape.gen.diff" + then + printf ' %-10s client-generated headers differ (reported, not a failure):\n' \ + "" + sed -e 's/^/ /' "$RUN_DIR/echo-$shape.gen.diff" + fi + done + + if (( failures )); then + warn "$failures shape(s) failed the fairness audit; the affected pairs" \ + "are not comparing like for like" + fi +} + +# A5: per-request Lua heap delta. The library's claim is that response handling +# stays in C, and the header arrays and the case-insensitive metatable are where +# that could regress, so this is reported rather than left as a diagnostic. +gc_report() { + printf '\n== Per-request Lua heap delta (KB/request) ==\n' + + local name delta + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + + delta="$("$CURL_BIN" -fsS \ + "http://127.0.0.1:$TARGET_PORT/gcdelta/$name" 2>/dev/null | + tail -n 1)" + printf ' %-18s %s\n' "$name" "${delta:-unavailable}" + done +} + +# ===== phases ===== + +median_qps() { + local name="$1" + + awk -F, -v want="$name" ' + NR > 1 && $2 == "throughput" && $4 == want && $15 == "yes" && $7 == 0 { + v[n++] = $5 + 0 + } + END { + if (n == 0) { print ""; exit } + for (i = 0; i < n; i++) { + for (j = i + 1; j < n; j++) { + if (v[j] < v[i]) { t = v[i]; v[i] = v[j]; v[j] = t } + } } - }' + if (n % 2) { printf "%.2f\n", v[(n - 1) / 2] } + else { printf "%.2f\n", (v[n / 2 - 1] + v[n / 2]) / 2 } + } + ' "$RESULTS_CSV" +} + +throughput_phase() { + local repeat name + + for ((repeat = 1; repeat <= BENCH_REPEATS; repeat++)); do + # cases rotate inside each repeat rather than running back to back, so + # drift and background load land on every case equally + for name in "${CASES[@]}"; do + run_wrk_case throughput "$repeat" "$name" "$WRK_DURATION" "$WRK_RATE" + done + done +} + +# A5: latency measured at saturation is queue depth. This pass re-runs each pair +# at a fixed fraction of the slower side's saturated throughput, which is the +# only rate at which the two are answering the same question. +latency_phase() { + printf '\n== Latency phase ==\n' + + local shapes=() name shape seen ffi_med resty_med rate + + for name in "${CASES[@]}"; do + [[ "$name" == "baseline" ]] && continue + shape="${name#*.}" + + seen=0 + local s + for s in "${shapes[@]:-}"; do + [[ "$s" == "$shape" ]] && seen=1 + done + (( seen )) || shapes+=("$shape") + done + + for shape in "${shapes[@]:-}"; do + ffi_med="$(median_qps "ffi.$shape")" + resty_med="$(median_qps "resty.$shape")" + + if [[ -z "$ffi_med" || -z "$resty_med" ]]; then + warn "no valid saturated throughput for both sides of $shape;" \ + "skipping its latency pass" + continue + fi + + rate="$(awk -v a="$ffi_med" -v b="$resty_med" \ + -v f="$BENCH_LATENCY_FRACTION" \ + 'BEGIN { m = (a < b ? a : b); printf "%d", m * f }')" + + (( rate > 0 )) || continue + + printf 'shape %s: fixed rate %s req/s (%.0f%% of the slower path)\n' \ + "$shape" "$rate" \ + "$(awk -v f="$BENCH_LATENCY_FRACTION" 'BEGIN { print f * 100 }')" + + local lrepeat + for ((lrepeat = 1; lrepeat <= BENCH_LATENCY_REPEATS; lrepeat++)); do + run_wrk_case latency "$lrepeat" "ffi.$shape" "$WRK_DURATION" "$rate" + run_wrk_case latency "$lrepeat" "resty.$shape" "$WRK_DURATION" "$rate" + done + done } +summarize() { + printf '\n== Summary (median of %s repeats, saturated rows only) ==\n' \ + "$BENCH_REPEATS" + + "$ROOT_DIR/benchmark/fold.sh" "$RESULTS_CSV" || true + + if (( ${#DROPPED[@]} )); then + printf '### Cases excluded before the run\n\n' + printf 'These failed their smoke check on this build and were not' + printf ' measured:\n\n' + local name + for name in "${DROPPED[@]}"; do + printf -- '- `%s`\n' "$name" + done + printf '\n' + fi + + printf 'Raw rows: %s\n' "$RESULTS_CSV" +} + +# ===== main ===== + prepare_configs print_config if [[ "$BENCH_DRY_RUN" == "1" ]]; then - printf 'BENCH_DRY_RUN=1, not starting OpenResty or wrk2\n' + printf '\nBENCH_DRY_RUN=1, not starting OpenResty or wrk2\n' + printf 'rendered: %s\n' "$TARGET_PREFIX/conf/nginx.conf" + printf 'rendered: %s\n' "$UPSTREAM_PREFIX/conf/nginx.conf" + printf 'rendered: %s (%s locations)\n' \ + "$TARGET_PREFIX/conf/bench_locations.conf" \ + "$(grep -c '^location' "$TARGET_PREFIX/conf/bench_locations.conf")" exit 0 fi @@ -430,7 +971,20 @@ wait_url "http://127.0.0.1:$TARGET_PORT/ready" smoke_check -run_wrk_case "baseline" "/bench/baseline" -run_wrk_case "C FFI keepalive" "/bench/ffi" -run_wrk_case "resty.http keepalive" "/bench/resty-http" -print_outbound_estimate +init_results + +if [[ "$BENCH_AUDIT" == "1" ]]; then + fairness_audit +fi + +throughput_phase + +if [[ "$BENCH_LATENCY" == "1" ]]; then + latency_phase +fi + +if [[ "$BENCH_GC" == "1" ]]; then + gc_report +fi + +summarize diff --git a/docs/ai-proxy-integration.md b/docs/ai-proxy-integration.md new file mode 100644 index 0000000..f373494 --- /dev/null +++ b/docs/ai-proxy-integration.md @@ -0,0 +1,163 @@ +# Track B: APISIX ai-proxy integration + +API contract findings from checking `resty.ngx_http_ffi_client` against what +APISIX's ai-proxy path actually needs from `lua-resty-http`. + +Checked against APISIX at `636100ec` and this library at `f13fcfa`. + +## The seam described in the plan does not exist + +The plan assumes `apisix/apisix/plugins/ai-transport/http.lua`, "already a seam" +exposing `_M.request(params, timeout)` and `_M.set_keepalive(res, ...)`, behind +which an alternate transport is "roughly 30 lines". + +There is no `ai-transport` directory in APISIX at `636100ec`. The outbound call +lives inline in `apisix/plugins/ai-drivers/openai-base.lua`, in `_M.request()`, +interleaved with everything else that function does: + +``` +_M.request(self, ctx, conf, request_table, extra_opts) + http.new() / set_timeout + fetch_gcp_access_token -- auth, may return early + url.parse(endpoint) -- scheme/host/port/path/query derivation + construct_forward_headers + self.request_filter(...) -- per-driver hook that mutates params + httpc:connect(params) + core.json.encode(params.body) + httpc:request(params) + if status == 429 or 5xx: return -- retry-relevant, before the body is read + read_response(...) -- SSE loop or buffered read + httpc:set_keepalive(conf.keepalive_timeout, conf.keepalive_pool) +``` + +`read_response` is a separate local function that receives `res` and drives +either the SSE `body_reader` loop or a buffered read. + +So B3.1 is not "write a 30-line module". It is "extract a transport interface +from `openai-base.lua`, then write the second implementation behind it". The +extraction is the larger and more contentious half, and it is a change to +APISIX, not to this library. That reordering should be agreed before the work +starts, because it changes who reviews it and where the risk sits. + +The rest of this document is B3.2 done statically: the contract gaps that the +swap would hit, found by reading both sides rather than by running the swap. + +## The surface ai-proxy needs + +| call | used for | +| --- | --- | +| `http.new()` | one client per request | +| `httpc:set_timeout(ms)` | single-argument form | +| `httpc:connect(params)` | `scheme`, `host`, `port`, `ssl_verify`, `ssl_server_name`; the whole params table is passed and unknown keys are expected to be ignored | +| `httpc:request(params)` | `method`, `path`, `query` (table), `headers`, `body` (string) | +| `res.status` | retry decision, before the body is read | +| `res.headers["Content-Type"]` | picks the SSE path | +| `res.body_reader` | SSE, chunk by chunk | +| `res:read_body()` | buffered responses | +| `httpc:set_keepalive(timeout, pool_size)` | `conf.keepalive_timeout`, `conf.keepalive_pool` | +| error strings | `handle_error` maps any error containing `timeout` to 504, everything else to 500 | + +## Already satisfied + +These were checked and need no work: + +- **`new`, `set_timeout`, `connect`, `request`, `set_keepalive`, `close`, + `get_reused_times`** all exist with matching signatures. +- **`connect` ignores unknown keys.** ai-proxy passes the whole `params` table, + including `method`, `headers`, `body` and `query`, into `connect`. This + library reads only the keys it knows. +- **`query` as a table** is encoded with `ngx.encode_args`, as `lua-resty-http` + does. +- **`res.headers["Content-Type"]`** resolves through the case-insensitive + metatable regardless of the casing the provider used. +- **`res.body_reader` is the default.** `request{}` without `preread_body` sets + up the streaming reader, which is what the SSE path needs. The buffered fast + path is opt-in, so a naive swap gets streaming behaviour, not a surprise + full-body buffer. +- **Timeout error strings.** `handle_error` looks for the substring `timeout`; + the C path emits exactly `"timeout"` for read, connect and TLS-handshake + timeouts, deliberately, with a comment saying it matches the cosocket. A 504 + stays a 504. +- **Keepalive pool isolation.** ai-proxy passes no explicit pool name, so the + key is derived. The derivation is a tagged, length-prefixed encoding of + `[ssl][verify][host:port][ssl_server_name][ssl_trusted_certificate]`, so + connections cannot cross between TLS and plaintext, between verify modes, + between SNI names, or between trusted CAs. With one gateway fronting several + providers on different TLS settings this is the property that matters most, + and it holds. + +## Gaps + +### 1. `request_uri` is a module function, not a method + +`lua-resty-http` exposes `httpc:request_uri(uri, params)`. This library exposes +`client.request_uri(opts)` at module level, taking a table rather than a URI +string, and has no method of that name on the client object. + +ai-proxy does not call it, so it does not block the swap. Every other APISIX +plugin that reaches for `resty.http` does, though, so it blocks the "drop-in" +claim in general. Either add a `request_uri` method with the `lua-resty-http` +signature, or state plainly in the README that the one-shot entry point is +deliberately different. + +### 2. `res.reason` and `res.has_body` are missing + +`lua-resty-http` sets both on the response table. This library sets neither. + +ai-proxy does not use them, so this is not a blocker for Track B. It is a +drop-in gap for anything that logs `res.reason`, and `has_body` is the idiomatic +way to check for a bodyless response before touching the reader. + +### 3. No proxy support + +`lua-resty-http` has `set_proxy_options` and honours `proxy_opts` in `connect` +and `request_uri`. This library has none. + +ai-proxy's `openai-base.lua` does not use it, but `request_filter` hooks are +free to set it and APISIX deployments behind a corporate proxy will want it. +This is the largest single missing feature relative to `lua-resty-http`. + +### 4. Request bodies must be strings + +`lua-resty-http` accepts a string, a table of strings, or a function for the +request body, and supports chunked request framing. This library takes a string +only, and refuses `Transfer-Encoding` outright because it always frames with +`Content-Length`. + +ai-proxy JSON-encodes to a string first, so it is unaffected. Any caller +streaming a request body is not portable to this client. + +### 5. `Connection` and `Content-Length` are sent on every request + +Found by the benchmark's fairness audit: this library emits + +``` +Connection: keep-alive +Content-Length: 0 +``` + +on requests where `lua-resty-http` emits neither. `lua-resty-http` only sets +`Content-Length` for methods that expect a body. + +For ai-proxy this is low risk: the requests are POSTs with a body, so +`Content-Length` is correct and present either way, and the only difference on +the wire is the explicit `Connection: keep-alive`. It is worth a deliberate +decision rather than an accident, because a `Content-Length: 0` on a GET is the +kind of thing a strict provider or CDN in front of one can reject, and this +would show up as provider-specific failures rather than as an obvious bug. + +## What B3.3 to B3.5 still need + +- **B3.3, compatibility.** Running APISIX's ai-proxy suite under both transports + needs the transport interface from B3.1 to exist first, plus an APISIX test + environment. Not attempted here. +- **B3.4, end-to-end performance.** Needs a mock openai-compatible SSE upstream + on localhost. Against a real provider, network and provider latency dominate + and the client CPU difference disappears into variance, so a real-provider run + is a sanity check only, as the plan already says. +- **Both** need the same 8-core topology Track A needs, for the same reason. + +The Track A harness already covers the three surfaces ai-proxy leans on most +(`body_reader` streaming, TLS, and resolving a peer by hostname) as the +`stream`, `tls` and `dns` cases, so a regression in any of them is visible +without standing up APISIX at all. diff --git a/t/003-benchmark-runner.t b/t/003-benchmark-runner.t index 0069ac3..662fb84 100644 --- a/t/003-benchmark-runner.t +++ b/t/003-benchmark-runner.t @@ -19,6 +19,9 @@ sub slurp { my $makefile = File::Spec->catfile($root, 'Makefile'); my $runner = File::Spec->catfile($root, 'benchmark', 'run.sh'); +my $fold = File::Spec->catfile($root, 'benchmark', 'fold.sh'); +my $cases_file = File::Spec->catfile($root, 'benchmark', 'cases.txt'); +my $bench_lua = File::Spec->catfile($root, 'benchmark', 'lua', 'bench.lua'); my $target_template = File::Spec->catfile($root, 'benchmark', 'conf', 'target.nginx.conf'); my $upstream_template = File::Spec->catfile($root, 'benchmark', 'conf', 'upstream.nginx.conf'); my $wrk_headers = File::Spec->catfile($root, 'benchmark', 'wrk', 'headers.lua'); @@ -29,20 +32,75 @@ like(slurp($makefile), qr/^bench:\s*$/m, 'Makefile defines bench target'); like(slurp($makefile), qr{\./benchmark/run\.sh}, 'bench target delegates to benchmark/run.sh'); ok(-x $runner, 'benchmark/run.sh exists and is executable'); -like(slurp($runner), qr/BENCH_DRY_RUN/, 'runner supports dry-run mode'); -like(slurp($runner), qr/CPU_SATURATION_THRESHOLD/, 'runner checks CPU saturation'); -like(slurp($runner), qr/WRK2_BIN/, 'runner uses wrk2 binary configuration'); -like(slurp($runner), qr/WRK_RATE/, 'runner supports wrk2 target rate'); -like(slurp($runner), qr/WRK_CONNECTIONS="\$\{WRK_CONNECTIONS:-100\}"/, +ok(-x $fold, 'benchmark/fold.sh exists and is executable'); + +my $runner_src = slurp($runner); + +like($runner_src, qr/BENCH_DRY_RUN/, 'runner supports dry-run mode'); +like($runner_src, qr/CPU_SATURATION_THRESHOLD/, 'runner checks CPU saturation'); +like($runner_src, qr/WRK2_BIN/, 'runner uses wrk2 binary configuration'); +like($runner_src, qr/WRK_RATE/, 'runner supports wrk2 target rate'); +like($runner_src, qr/WRK_CONNECTIONS="\$\{WRK_CONNECTIONS:-100\}"/, 'runner defaults wrk2 connections to 100'); -like(slurp($runner), qr/KEEPALIVE_POOL_SIZE="\$\{KEEPALIVE_POOL_SIZE:-120\}"/, +like($runner_src, qr/KEEPALIVE_POOL_SIZE="\$\{KEEPALIVE_POOL_SIZE:-120\}"/, 'runner defaults keepalive pool size above wrk2 connections'); -like(slurp($runner), qr/\s-U\s/, 'runner enables wrk2 uncorrected latency output'); -like(slurp($runner), qr/\s-R\s+"\$WRK_RATE"/, 'runner passes wrk2 target rate'); -like(slurp($runner), qr/run_wrk_case "baseline" "\/bench\/baseline"/, - 'runner runs baseline case first'); -like(slurp($runner), qr/print_outbound_estimate/, - 'runner prints baseline-derived outbound estimate'); +like($runner_src, qr/\s-U\s/, 'runner requests wrk2 uncorrected latency output'); +like($runner_src, qr/-R "\$rate"/, 'runner passes a wrk2 target rate'); + +# the case table replaces the hardcoded case list +like($runner_src, qr/BENCH_CASES=/, 'runner drives cases from BENCH_CASES'); +like($runner_src, qr/BENCH_REPEATS="\$\{BENCH_REPEATS:-5\}"/, + 'runner defaults to five repeats'); +like($runner_src, qr/WRK_WARMUP/, 'runner runs a discarded warmup'); +like($runner_src, qr/RESULTS_CSV/, 'runner writes one CSV row per run'); +like($runner_src, qr/fairness_audit/, 'runner audits request fairness'); +like($runner_src, qr/latency_phase/, 'runner has a separate fixed-rate latency phase'); +like($runner_src, qr/throughput_phase/, 'runner has a throughput phase'); + +ok(-f $cases_file, 'benchmark/cases.txt exists'); +ok(-f $bench_lua, 'benchmark/lua/bench.lua exists'); + +# Every shape must exist on both sides, or the comparison is not a comparison. +my @cases = grep { length } map { + my $line = $_; + $line =~ s/#.*//; + $line =~ s/\s+//g; + $line; +} split /\n/, slurp($cases_file); + +ok(scalar(@cases) > 0, 'cases.txt lists cases'); + +my %by_shape; +for my $name (@cases) { + my ($driver, $shape) = split /\./, $name, 2; + $by_shape{$shape}{$driver} = 1; +} + +my @unpaired = grep { !($by_shape{$_}{ffi} && $by_shape{$_}{resty}) } + sort keys %by_shape; +is(scalar(@unpaired), 0, 'every shape has both an ffi and a resty case') + or diag('unpaired shapes: ' . join(', ', @unpaired)); + +# cases.txt is the source of truth the runner generates locations from; bench.lua +# must define the same shapes. The running worker is diffed against this file at +# smoke-check time too, but a static check fails faster. +my $bench_src = slurp($bench_lua); +my @declared = $bench_src =~ /pair_case\("([a-z0-9_]+)"/g; +my %declared = map { $_ => 1 } @declared; + +my @missing = grep { !$declared{$_} } sort keys %by_shape; +is(scalar(@missing), 0, 'bench.lua defines every shape listed in cases.txt') + or diag('missing from bench.lua: ' . join(', ', @missing)); + +my @extra = grep { !$by_shape{$_} } sort @declared; +is(scalar(@extra), 0, 'cases.txt lists every shape bench.lua defines') + or diag('missing from cases.txt: ' . join(', ', @extra)); + +# the axes the plan asks the matrix to cover +for my $shape (qw(oneshot stateful stream short post tls dns chunked trailers + hdr40 cookies req30)) { + ok($by_shape{$shape}, "matrix covers the $shape axis"); +} ok(-f $target_template, 'target nginx template exists'); ok(-f $upstream_template, 'upstream nginx template exists'); @@ -52,8 +110,15 @@ like(slurp($readme), qr/\| `WRK_CONNECTIONS` \| `100` \|/, like(slurp($readme), qr/\| `KEEPALIVE_POOL_SIZE` \| `120` \|/, 'README documents keepalive pool default'); +# A dedicated RUN_DIR, because prepare_configs starts by rm -rf'ing it. Sharing +# the default with a real run would let this test delete a live benchmark's +# prefix out from under it, taking the pid files with it so the runner could no +# longer even stop its own nginx. +my $dry_run_dir = File::Spec->catdir($root, 'benchmark', '.run-dryrun'); + my $cmd = join ' ', 'cd', quotemeta($root), '&&', + "RUN_DIR=" . quotemeta($dry_run_dir), 'BENCH_DRY_RUN=1', 'TARGET_CPU=0', 'UPSTREAM_CPUS=1', @@ -73,25 +138,46 @@ like($output, qr/UPSTREAM_CPUS=1/, 'dry-run prints upstream CPUs'); like($output, qr/WRK_CPUS=1/, 'dry-run prints wrk CPUs'); like($output, qr/KEEPALIVE_POOL_SIZE=120/, 'dry-run prints keepalive pool default'); like($output, qr/WRK_RATE=1000/, 'dry-run prints wrk2 target rate'); +like($output, qr/BENCH_REPEATS=5/, 'dry-run prints the repeat count'); my $rendered_target = File::Spec->catfile( - $root, 'benchmark', '.run', 'target', 'conf', 'nginx.conf' + $dry_run_dir, 'target', 'conf', 'nginx.conf' ); my $rendered_upstream = File::Spec->catfile( - $root, 'benchmark', '.run', 'upstream', 'conf', 'nginx.conf' + $dry_run_dir, 'upstream', 'conf', 'nginx.conf' +); +my $rendered_locations = File::Spec->catfile( + $dry_run_dir, 'target', 'conf', 'bench_locations.conf' ); ok(-f $rendered_target, 'dry-run renders target nginx config'); ok(-f $rendered_upstream, 'dry-run renders upstream nginx config'); +ok(-f $rendered_locations, 'dry-run generates the bench locations'); my $target_conf = slurp($rendered_target); like($target_conf, qr/location = \/bench\/baseline/, 'target config has baseline benchmark endpoint'); -like($target_conf, qr/location = \/bench\/ffi/, 'target config has C FFI benchmark endpoint'); -like($target_conf, qr/location = \/bench\/resty-http/, 'target config has resty.http benchmark endpoint'); +like($target_conf, qr/include bench_locations\.conf;/, 'target config includes the generated locations'); like($target_conf, qr/worker_processes 1;/, 'target config uses one worker'); +unlike($target_conf, qr/\{\{[A-Z_]+\}\}/, 'target config has no unrendered placeholders'); + +my $locations = slurp($rendered_locations); +for my $name (@cases) { + like($locations, qr/location = \/bench\/\Q$name\E \{/, + "generated locations include /bench/$name"); +} +like($locations, qr/location = \/echo\/ffi\.stateful \{/, + 'generated locations include the fairness-audit echo endpoints'); my $upstream_conf = slurp($rendered_upstream); like($upstream_conf, qr/Content-Length/, 'upstream config returns content-length response'); like($upstream_conf, qr/benchmark_response_body/, 'upstream config prebuilds response body'); +like($upstream_conf, qr/location = \/mock\/chunked/, 'upstream serves a chunked response'); +like($upstream_conf, qr/location = \/mock\/echo/, 'upstream echoes the raw request headers'); +like($upstream_conf, qr/location = \/mock\/sink/, 'upstream accepts a request body'); +like($upstream_conf, qr/location = \/mock\/headers40/, 'upstream serves a header-heavy response'); +like($upstream_conf, qr/location = \/mock\/cookies/, 'upstream serves repeated Set-Cookie headers'); +like($upstream_conf, qr/^stream \{/m, 'upstream runs a raw TCP mock for trailers'); +like($upstream_conf, qr/X-Checksum/, 'raw TCP mock sends a trailer'); +unlike($upstream_conf, qr/\{\{[A-Z_]+\}\}/, 'upstream config has no unrendered placeholders'); done_testing();