Skip to content

fulmine: add the production-stack entry - #1100

Merged
MDA2AV merged 4 commits into
MDA2AV:mainfrom
nigrosimone:fulmine-production-stack
Aug 11, 2026
Merged

fulmine: add the production-stack entry#1100
MDA2AV merged 4 commits into
MDA2AV:mainfrom
nigrosimone:fulmine-production-stack

Conversation

@nigrosimone

Copy link
Copy Markdown
Contributor

Subscribes the fulmine entry to production-stack. Four services as the rules ask: a stock Caddy edge, Redis, the shared authsvc built as-is, and the server.

The edge terminates TLS and h2, serves /static/* off disk, sends /public/* straight through, and puts /api/* behind forward_auth against the authsvc, copying X-User-Id on. Nothing in the server looks at a token.

On the server side most of it was already there for crud: the cache-aside read is now registered under both /crud/items/:id and /api/items/:id, and the JSON route under /public/json/:count. New are /public/baseline, the POST /api/items/{id} that answers 204 and invalidates, and /api/me, which reads the user by X-User-Id with a 30 second TTL.

I ran the stack locally against the repo certs and seed, and walked the validation checks by hand: h2 negotiates, /static comes from the edge, /public/baseline gives 55, /public/json/25 has 25 items with the full schema and correct totals, /api/* is 401 without a token and with a bogus one on both GET and POST, /api/items/1 returns id 1, cache-aside reads MISS then HIT, POST answers 204 and the next read is MISS again, and /api/me returns user 42.

/benchmark -f fulmine -t production-stack --save

@nigrosimone
nigrosimone marked this pull request as ready for review August 11, 2026 09:47
@nigrosimone

Copy link
Copy Markdown
Contributor Author

/benchmark -f fulmine --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: fulmine | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 1,314,024 6826.9% 2.6GiB -1.6% ~0%
baseline 4096 1,277,971 6785.8% 2.7GiB -1.5% +3.8%
pipelined 512 7,896,918 6572.1% 2.5GiB -0.3% ~0%
pipelined 4096 7,877,142 6570.8% 2.6GiB +0.1% ~0%
limited-conn 512 1,079,317 6208.4% 2.6GiB -1.6% ~0%
limited-conn 4096 1,157,560 6458.8% 2.7GiB -1.5% ~0%
json 4096 1,135,231 6462.4% 2.6GiB -0.5% ~0%
json-comp 512 341,959 6239.2% 8.1GiB ~0% +1.2%
json-comp 4096 356,281 6376.5% 9.0GiB ~0% ~0%
json-comp 16384 372,409 6319.7% 10.4GiB -0.6% +1.0%
json-tls 4096 1,069,770 6587.9% 2.6GiB -0.6% ~0%
upload 32 2,031 3141.1% 6.8GiB -0.3% +3.0%
upload 256 2,125 6345.8% 7.4GiB -1.7% -1.3%
api-4 256 35,988 393.8% 426MiB -1.1% +6.0%
api-16 1024 125,628 1495.3% 1.2GiB -4.2% -7.7%
static 1024 532,953 6502.3% 4.2GiB -1.4% +7.7%
static 4096 578,692 6520.8% 7.6GiB -0.7% +1.3%
static 6800 558,769 6476.2% 7.6GiB -1.5% ~0%
static-tls 1024 430,985 6510.8% 4.5GiB -0.4% +21.6%
static-tls 4096 456,896 6451.5% 7.5GiB -1.5% +4.2%
static-tls 6800 447,603 6344.8% 7.6GiB -1.1% ~0%
async-db 1024 225,500 5633.7% 4.1GiB +0.4% +13.9%
crud 4096 358,280 4259.9% 8.3GiB +0.7% +2.5%
gateway-64 512 89,194 4301.6% 4.6GiB +0.3% +2.2%
gateway-64 1024 81,019 4139.8% 6.0GiB -1.6% -10.4%
gateway-h3 64 91,841 3599.2% 2.6GiB +0.8% ~0%
gateway-h3 256 63,011 2649.1% 3.2GiB +0.4% -13.5%
production-stack 256 19,587 1537.6% 2.2GiB NEW NEW
production-stack 1024 19,333 1577.3% 3.0GiB NEW NEW
echo-ws 512 3,728,636 6443.7% 2.4GiB +0.6% +4.3%
echo-ws 4096 3,791,854 6554.3% 2.4GiB +0.2% ~0%
echo-ws 16384 3,550,300 6499.4% 2.5GiB -0.4% ~0%
echo-ws-pipeline 512 24,928,620 6783.7% 2.4GiB +1.4% +4.3%
echo-ws-pipeline 4096 24,879,625 6586.0% 2.4GiB +0.4% ~0%
echo-ws-pipeline 16384 23,721,667 6353.5% 2.6GiB +0.8% ~0%
echo-ws-limited 512 2,084,867 5601.9% 2.4GiB ~0% ~0%
echo-ws-limited 4096 2,558,787 5956.0% 2.4GiB ~0% ~0%
Full log
  Bandwidth:  154.29MB/s
  WS upgrades: 16384
  WS frames:   115199488
  Latency samples: 115199488 / 115199488 responses (100.0%)
[info] CPU 6763.1% | Mem 2.6GiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   11.39ms   11.00ms   13.30ms   17.30ms   21.30ms

  114544176 frames sent in 5.00s, 114282032 frames received
  Throughput: 22.85M req/s
  Bandwidth:  153.06MB/s
  WS upgrades: 16384
  WS frames:   114282032
  Latency samples: 114282032 / 114282032 responses (100.0%)
[info] CPU 6474.8% | Mem 2.6GiB

=== Best: 23721667 req/s (CPU: 6353.5%, Mem: 2.6GiB) ===
[info] saved results/echo-ws-pipeline/16384/fulmine.json
httparena-bench-fulmine
httparena-bench-fulmine

==============================================
=== fulmine / echo-ws-limited / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    148us    120us    261us    548us   1.63ms

  10383843 frames sent in 5.00s, 10383913 frames received
  Throughput: 2.08M req/s
  Bandwidth:  46.73MB/s
  WS upgrades: 1050999
  WS frames:   10383913
  Latency samples: 10383858 / 10383913 responses (100.0%)
  Reconnects: 1038417
[info] CPU 5448.8% | Mem 2.4GiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    147us    121us    260us    533us   1.34ms

  10424257 frames sent in 5.00s, 10424337 frames received
  Throughput: 2.08M req/s
  Bandwidth:  46.91MB/s
  WS upgrades: 1055115
  WS frames:   10424337
  Latency samples: 10424292 / 10424337 responses (100.0%)
  Reconnects: 1042470
[info] CPU 5601.9% | Mem 2.4GiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    146us    120us    260us    528us   1.21ms

  10395213 frames sent in 5.00s, 10395302 frames received
  Throughput: 2.08M req/s
  Bandwidth:  46.78MB/s
  WS upgrades: 1054141
  WS frames:   10395302
  Latency samples: 10395256 / 10395302 responses (100.0%)
  Reconnects: 1039580
[info] CPU 5492.0% | Mem 2.4GiB

=== Best: 2084867 req/s (CPU: 5601.9%, Mem: 2.4GiB) ===
[info] saved results/echo-ws-limited/512/fulmine.json
httparena-bench-fulmine
httparena-bench-fulmine

==============================================
=== fulmine / echo-ws-limited / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.06ms    863us   1.79ms   3.28ms   6.00ms

  12796400 frames sent in 5.00s, 12793936 frames received
  Throughput: 2.56M req/s
  Bandwidth:  57.64MB/s
  WS upgrades: 1283776
  WS frames:   12793936
  Latency samples: 12793917 / 12793936 responses (100.0%)
  Reconnects: 1278357
[info] CPU 5956.0% | Mem 2.4GiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.08ms    968us   1.78ms   3.02ms   4.61ms

  12672847 frames sent in 5.00s, 12671774 frames received
  Throughput: 2.53M req/s
  Bandwidth:  57.09MB/s
  WS upgrades: 1270838
  WS frames:   12671774
  Latency samples: 12671773 / 12671774 responses (100.0%)
  Reconnects: 1267334
[info] CPU 6236.9% | Mem 2.4GiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.05ms    934us   1.75ms   2.89ms   3.98ms

  12793957 frames sent in 5.00s, 12791408 frames received
  Throughput: 2.56M req/s
  Bandwidth:  57.63MB/s
  WS upgrades: 1281606
  WS frames:   12791408
  Latency samples: 12791408 / 12791408 responses (100.0%)
  Reconnects: 1278097
[info] CPU 6016.8% | Mem 2.4GiB

=== Best: 2558787 req/s (CPU: 5956.0%, Mem: 2.4GiB) ===
[info] saved results/echo-ws-limited/4096/fulmine.json
httparena-bench-fulmine
httparena-bench-fulmine
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/fulmine.json - 37 new, 37 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
httparena-redis
[info] restoring loopback MTU to 65536

@nigrosimone

Copy link
Copy Markdown
Contributor Author

The production-stack rows landed at 19.5k, which is well off the 211k the leading entry does, and the CPU column says why: 1537% and 1577% against the 1600% my edge's sixteen logical CPUs allow. The edge was pegged and the other three services sat idle.

I had the split backwards. The nginx entry gives its proxy 30 logical CPUs and its server 24; I had given the edge 16 and the server 38, on a profile where the edge terminates TLS and h2 for every request and serves /static off disk on top. The edge now takes 32 and the server 22.

I also suspected the forward_auth hop was dialling the verifier per request and added a keepalive transport for it, then measured before and after: 200 /api requests leave two established connections either way, so Caddy was already pooling and the change bought nothing. Dropped it rather than carry a fix for a problem that wasn't there.

Happy to sweep EDGE_CPUSET/SERVER_CPUSET further if the next run still leaves headroom.

/benchmark -f fulmine -t production-stack --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: fulmine | Test: production-stack

Test Conn RPS CPU Mem Δ RPS Δ Mem
production-stack 256 59,123 4058.8% 3.4GiB NEW NEW
production-stack 1024 50,374 3916.2% 5.5GiB NEW NEW
Full log
#38 unpacking to docker.io/library/httparena-fulmine-production-stack-authsvc:latest 0.0s done
#38 DONE 0.4s

#40 [edge] resolving provenance for metadata file
#40 DONE 0.0s

#41 [server] resolving provenance for metadata file
#41 DONE 0.0s

#42 [authsvc] resolving provenance for metadata file
#42 DONE 0.0s
 Image httparena-fulmine-production-stack-authsvc Built 
 Image httparena-fulmine-production-stack-server Built 
 Image httparena-fulmine-production-stack-edge Built 
 Container httparena-fulmine-production-stack-cache-1 Creating 
 Container httparena-fulmine-production-stack-authsvc-1 Creating 
 Container httparena-fulmine-production-stack-authsvc-1 Created 
 Container httparena-fulmine-production-stack-cache-1 Created 
 Container httparena-fulmine-production-stack-server-1 Creating 
 Container httparena-fulmine-production-stack-server-1 Created 
 Container httparena-fulmine-production-stack-edge-1 Creating 
 Container httparena-fulmine-production-stack-edge-1 Created 
 Container httparena-fulmine-production-stack-authsvc-1 Starting 
 Container httparena-fulmine-production-stack-cache-1 Starting 
 Container httparena-fulmine-production-stack-authsvc-1 Started 
 Container httparena-fulmine-production-stack-cache-1 Started 
 Container httparena-fulmine-production-stack-server-1 Starting 
 Container httparena-fulmine-production-stack-server-1 Started 
 Container httparena-fulmine-production-stack-edge-1 Starting 
 Container httparena-fulmine-production-stack-edge-1 Started 
[info] gateway containers: 3 (dce55f98031d 5fd66d5c9cc7 27ad433d3967 )
[warn] production-stack expects exactly 4 containers, found 3 — stats may not sum correctly
[info] waiting for server...
[info] server ready

[run 1/3]
=== h2load-reads ===
starting benchmark...
23.

56.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
21.
28.

18. Stopping all clients.

17. Stopping all clients.

finished in 5.07s, 38726.20 req/s, 426.79MB/s
requests: 193631 total, 210015 started, 193631 done, 193285 succeeded, 346 failed, 0 errored, 0 timeout
status codes: 193478 2xx, 0 3xx, 347 4xx, 0 5xx
traffic: 2.08GB (2237614475) total, 2.14MB (2246104) headers (space savings 92.96%), 2.08GB (2230553118) data
                     min         max         mean         sd        +/- sd
time for request:      155us       2.61s    364.28ms    281.89ms    63.14%
time for connect:    26.95ms       1.14s    274.90ms    189.49ms    80.86%
time to 1st byte:    51.41ms       1.23s    377.04ms    252.94ms    77.15%
req/s           :      28.80       47.99       37.81        2.95    78.22%
=== h2load-writes ===
starting benchmark...
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2

finished in 5.00s, 1736.20 req/s, 29.88KB/s
requests: 8681 total, 9705 started, 8681 done, 8618 succeeded, 63 failed, 0 errored, 0 timeout
status codes: 8618 2xx, 0 3xx, 63 4xx, 0 5xx
traffic: 149.39KB (152974) total, 55.28KB (56604) headers (space savings 90.52%), 8.86KB (9072) data
                     min         max         mean         sd        +/- sd
time for request:     3.14ms       2.59s    521.70ms    246.55ms    73.08%
time for connect:     2.83ms    429.93ms     73.68ms     61.01ms    82.03%
time to 1st byte:    81.58ms    964.74ms    326.22ms    283.04ms    73.44%
req/s           :      10.40       16.80       13.56        1.19    71.09%
[info] CPU 3921.6% | Mem 4.5GiB
[info]   edge: 2895% 2.7GiB | authsvc: 457% 287MiB | server: 570% 1.5GiB

[run 2/3]
=== h2load-reads ===
starting benchmark...
41.
.9progress: 87.

% of clients started
38.
11.
57.
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
. Stopping all clients.Stopped all clients for thread #31

41
. Stopping all clients.
45Stopped all clients for thread #51
. Stopping all clients.

finished in 5.05s, 48421.40 req/s, 513.01MB/s
requests: 242107 total, 258491 started, 242107 done, 242040 succeeded, 67 failed, 0 errored, 0 timeout
status codes: 242145 2xx, 0 3xx, 67 4xx, 0 5xx
traffic: 2.50GB (2689674377) total, 2.57MB (2699665) headers (space savings 93.18%), 2.50GB (2681035053) data
                     min         max         mean         sd        +/- sd
time for request:      166us       2.71s    316.60ms    210.22ms    62.08%
time for connect:     3.28ms    150.98ms     61.17ms     36.61ms    58.98%
time to 1st byte:    10.64ms    375.10ms    160.55ms     81.16ms    61.33%
req/s           :      41.60       54.39       47.28        2.07    66.21%
=== h2load-writes ===
starting benchmark...
6.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
1. Stopping all clients.

finished in 5.00s, 2427.60 req/s, 37.48KB/s
requests: 12138 total, 13162 started, 12138 done, 12118 succeeded, 20 failed, 0 errored, 0 timeout
status codes: 12118 2xx, 0 3xx, 20 4xx, 0 5xx
traffic: 187.40KB (191899) total, 67.73KB (69357) headers (space savings 91.62%), 2.81KB (2880) data
                     min         max         mean         sd        +/- sd
time for request:     1.69ms       1.72s    399.67ms    194.24ms    70.04%
time for connect:     3.43ms     34.70ms     19.69ms      9.15ms    58.59%
time to 1st byte:     6.96ms    137.19ms     47.51ms     22.11ms    85.94%
req/s           :      16.00       21.60       18.96        1.24    69.53%
[info] CPU 3997.4% | Mem 4.9GiB
[info]   edge: 2992% 2.6GiB | authsvc: 461% 301MiB | server: 545% 2.1GiB

[run 3/3]
=== h2load-reads ===
starting benchmark...
29.
41.
.Main benchmark duration is started for thread #12.

62% of clients started
progress: 75Warm-up phase is over for thread #59.% of clients started

progress: 87Warm-up phase is over for thread #22% of clients startedWarm-up phase is over for thread #36.

progress: Warm-up phase is over for thread #9346.
% of clients startedMain benchmark duration is started for thread #17.

34.
% of clients started
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
.
24. Stopping all clients.

28
. Stopping all clients.
. Stopping all clients.

finished in 5.06s, 48421.60 req/s, 516.07MB/s
requests: 242108 total, 258492 started, 242108 done, 241977 succeeded, 131 failed, 0 errored, 0 timeout
status codes: 242551 2xx, 0 3xx, 131 4xx, 0 5xx
traffic: 2.52GB (2705694073) total, 2.60MB (2729945) headers (space savings 93.13%), 2.51GB (2697010726) data
                     min         max         mean         sd        +/- sd
time for request:      152us       1.80s    317.25ms    217.28ms    67.83%
time for connect:     3.05ms    748.77ms     86.77ms     92.51ms    90.23%
time to 1st byte:    13.23ms    897.05ms    212.57ms    159.08ms    80.76%
req/s           :      39.59       55.80       47.28        2.55    73.24%
=== h2load-writes ===
starting benchmark...
7.
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2

finished in 5.00s, 2472.20 req/s, 37.85KB/s
requests: 12361 total, 13385 started, 12361 done, 12345 succeeded, 16 failed, 0 errored, 0 timeout
status codes: 12345 2xx, 0 3xx, 16 4xx, 0 5xx
traffic: 189.27KB (193808) total, 68.25KB (69884) headers (space savings 91.70%), 2.25KB (2304) data
                     min         max         mean         sd        +/- sd
time for request:      403us       1.56s    392.71ms    212.78ms    72.33%
time for connect:     3.39ms     30.59ms     18.00ms      8.03ms    57.81%
time to 1st byte:     6.05ms    335.27ms     56.05ms     50.94ms    87.50%
req/s           :      16.00       24.20       19.31        1.60    69.53%
[info] CPU 3916.2% | Mem 5.5GiB
[info]   edge: 2820% 3.1GiB | authsvc: 466% 216MiB | server: 630% 2.4GiB

=== Best: 50374 req/s (CPU: 3916.2%, Mem: 5.5GiB) ===
[info] saved results/production-stack/1024/fulmine.json
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/fulmine.json - 2 new, 37 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
httparena-redis
[info] restoring loopback MTU to 65536

@nigrosimone

Copy link
Copy Markdown
Contributor Author

It's good for me

@MDA2AV
MDA2AV merged commit 391f316 into MDA2AV:main Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants