Skip to content

Add Varnish as a benchmarked framework - #1110

Open
gquintard wants to merge 2 commits into
MDA2AV:mainfrom
gquintard:add-varnish-framework
Open

Add Varnish as a benchmarked framework#1110
gquintard wants to merge 2 commits into
MDA2AV:mainfrom
gquintard:add-varnish-framework

Conversation

@gquintard

@gquintard gquintard commented Aug 13, 2026

Copy link
Copy Markdown

Summary

  • Adds frameworks/varnish/ as an infrastructure entry: Varnish Cache with native TLS/H2 termination (varnishd -A), vmod-fileserver serving /static/* from Varnish's own object cache, and a custom vmod (vmod_httparena, built with varnish-rs) computing /baseline11//baseline2 in-process — no separate origin/backend process.
  • Subscribes to baseline, pipelined, limited-conn, static, baseline-h2, static-h2 (matches nginx's test list minus HTTP/3 — no QUIC support in OSS Varnish).

Test plan

  • docker build succeeds
  • Manual docker run + curl matrix: /pipeline → ok; GET/POST/chunked-POST /baseline11 → correct sums; POST anti-cheat (two different bodies, same URL) → two different correct sums; /static/* → correct Content-Type per extension + exact byte size + 404 for missing files; /baseline2 and /static over HTTP/2+TLS → HTTP/2 200
  • /validate -f varnish (requesting via PR bot)
  • /benchmark -f varnish --save (requesting via PR bot)

🤖 Generated with Claude Code

@gquintard
gquintard force-pushed the add-varnish-framework branch from cd51cfa to 9e8bebc Compare August 13, 2026 17:44
@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/validate -f varnish

@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

server busy with a large run, Ill trigger validation and benchmark here when it is finished.

Repository owner deleted a comment from github-actions Bot Aug 13, 2026
Repository owner deleted a comment from github-actions Bot Aug 13, 2026
@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

/benchmark -f varnish

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 182,712 6488.7% 204MiB NEW NEW
baseline 4096 191,117 6490.9% 433MiB NEW NEW
pipelined 512 287,949 6521.5% 156MiB NEW NEW
pipelined 4096 298,493 6323.6% 287MiB NEW NEW
limited-conn 512 161,534 6460.8% 228MiB NEW NEW
limited-conn 4096 161,095 6508.3% 440MiB NEW NEW
static 1024 108,649 6516.9% 2.4GiB NEW NEW
static 4096 110,386 6430.3% 3.7GiB NEW NEW
static 6800 109,369 6467.6% 2.9GiB NEW NEW
baseline-h2 256 207,409 6281.2% 2.5GiB NEW NEW
baseline-h2 1024 203,568 6275.1% 2.8GiB NEW NEW
static-h2 256 77,556 6003.4% 6.8GiB NEW NEW
static-h2 1024 81,482 6606.5% 7.1GiB NEW NEW
Full log 
Process Request Failure:1479
Process Request Failure:1492
Process Request Failure:928
Process Request Failure:1521
Process Request Failure:1674
Process Request Failure:1961
Process Request Failure:1589
Process Request Failure:1837
Process Request Failure:1368
Process Request Failure:1680
Process Request Failure:960
Process Request Failure:1845
Process Request Failure:1029
Process Request Failure:1806
Process Request Failure:1655
Process Request Failure:1853
Process Request Failure:1377
Process Request Failure:1683
Process Request Failure:964
Process Request Failure:2005
Process Request Failure:1224
Process Request Failure:1306
Process Request Failure:1228
Process Request Failure:2008
Process Request Failure:706
Process Request Failure:1122
Process Request Failure:1651
Process Request Failure:999
Process Request Failure:1719
Process Request Failure:1720
Process Request Failure:998
Process Request Failure:1559
Process Request Failure:1887
Process Request Failure:1570
Process Request Failure:2076
Process Request Failure:1514
Process Request Failure:1404
Process Request Failure:2009
Process Request Failure:965
Process Request Failure:1927
Process Request Failure:777
Process Request Failure:1701
Process Request Failure:1071
Process Request Failure:1373
Process Request Failure:1181
Process Request Failure:786
Process Request Failure:1252
Process Request Failure:1809
Process Request Failure:1693
Process Request Failure:1427
Process Request Failure:1895
Process Request Failure:1694
Process Request Failure:1931
Process Request Failure:1086
Process Request Failure:1746
Process Request Failure:1468
Process Request Failure:1110
Process Request Failure:2044
Process Request Failure:716
Process Request Failure:1176
Process Request Failure:725
Process Request Failure:1830
Process Request Failure:1406
Process Request Failure:1798
Process Request Failure:1130
Process Request Failure:2083
Process Request Failure:1829
Process Request Failure:1489
Process Request Failure:1744
Process Request Failure:2119
Process Request Failure:1669
Process Request Failure:1522
Process Request Failure:1234
Process Request Failure:1207
Process Request Failure:761
Process Request Failure:1466
Process Request Failure:1382
Process Request Failure:1985
Process Request Failure:1956
Process Request Failure:1318
Process Request Failure:1791
Process Request Failure:1474
Process Request Failure:1451
Process Request Failure:1909
Process Request Failure:1567
Process Request Failure:1595
Process Request Failure:1438
Process Request Failure:1692
Process Request Failure:1566
Process Request Failure:1853
Process Request Failure:1165
Process Request Failure:1401
Process Request Failure:1870
Process Request Failure:1822
Process Request Failure:1236
Process Request Failure:1668
Process Request Failure:1918
Process Request Failure:1383
Process Request Failure:2119
Process Request Failure:1839
Process Request Failure:1952
Process Request Failure:1911
Process Request Failure:1965
Process Request Failure:1329
Process Request Failure:1774
Process Request Failure:2040
Process Request Failure:1504
Process Request Failure:1832
Process Request Failure:1749
Process Request Failure:1946
Process Request Failure:1598
Process Request Failure:1925
Process Request Failure:1118
Process Request Failure:1406
Process Request Failure:1919
Process Request Failure:2038
Process Request Failure:2041
Process Request Failure:1915
Process Request Failure:1527
Process Request Failure:1534
Process Request Failure:1615
Process Request Failure:1589
Process Request Failure:1857
Process Request Failure:1775
Process Request Failure:1588
Process Request Failure:1241
Process Request Failure:1579
Process Request Failure:1778
Process Request Failure:1846
Process Request Failure:1660
Process Request Failure:1584
Process Request Failure:1409
Process Request Failure:1909
Process Request Failure:1115
Process Request Failure:2033
Process Request Failure:1728
Process Request Failure:1973
Process Request Failure:1823
Process Request Failure:1957
Process Request Failure:1967
Process Request Failure:1486
Process Request Failure:1690
. Stopping all clients.Stopped all clients for thread #58
56
. Stopping all clients.61
. Stopping all clients.

. Stopping all clients.
56
Process Request Failure:1629

62Stopped all clients for thread #41


finished in 5.07s, 79801.20 req/s, 4.75GB/s
requests: 495445 total, 526347 started, 495445 done, 399006 succeeded, 96439 failed, 96439 errored, 0 timeout
status codes: 413118 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 23.74GB (25491741887) total, 63.86MB (66961981) headers (space savings 18.99%), 23.66GB (25403968184) data
                     min         max         mean         sd        +/- sd
time for request:     3.18ms       1.06s    110.35ms     39.98ms    79.31%
time for connect:     3.17ms       1.59s    676.92ms    549.06ms    50.88%
time to 1st byte:    26.57ms       1.68s    739.32ms    551.20ms    51.97%
req/s           :       0.00      367.08      161.60       76.66    69.53%
[info] CPU 6606.5% | Mem 7.1GiB

=== Best: 81482 req/s (CPU: 6606.5%, Mem: 7.1GiB) ===
[info] saved results/static-h2/1024/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to baseline-h2c
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to stream-grpc
[info] skip: varnish does not subscribe to stream-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

hum, this looks quite bad. I'm bumping the default number of threads

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/benchmark -f varnish

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@MDA2AV

MDA2AV commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

I can trigger a run that posts docker logs if you rebase this to main, if you need. Could be printing errors which slows down quite a lot.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 201,666 6494.4% 205MiB NEW NEW
baseline 4096 182,374 6402.4% 198MiB NEW NEW
pipelined 512 270,442 6510.2% 163MiB NEW NEW
pipelined 4096 272,895 6360.9% 269MiB NEW NEW
limited-conn 512 164,990 6497.3% 214MiB NEW NEW
limited-conn 4096 160,048 6494.9% 416MiB NEW NEW
static 1024 107,400 6513.5% 2.4GiB NEW NEW
static 4096 108,761 6438.1% 3.4GiB NEW NEW
static 6800 109,244 6418.1% 3.9GiB NEW NEW
baseline-h2 256 203,856 6241.5% 2.5GiB NEW NEW
baseline-h2 1024 204,456 6248.1% 2.8GiB NEW NEW
static-h2 256 86,946 6634.2% 5.5GiB NEW NEW
static-h2 1024 81,703 6599.6% 7.0GiB NEW NEW
Full log 
Process Request Failure:1366
Process Request Failure:1234
Process Request Failure:1547
Process Request Failure:1525
Process Request Failure:1303
Process Request Failure:1364
Process Request Failure:1984
Process Request Failure:1429
Process Request Failure:1595
Process Request Failure:1408
Process Request Failure:1359
Process Request Failure:1705
Process Request Failure:1320
Process Request Failure:1594
Process Request Failure:1594
Process Request Failure:1420
Process Request Failure:1406
Process Request Failure:1177
Process Request Failure:1194
Process Request Failure:1709
Process Request Failure:1600
Process Request Failure:1966
Process Request Failure:1318
Process Request Failure:1134
Process Request Failure:1756
Process Request Failure:1287
Process Request Failure:1979
Process Request Failure:1341
Process Request Failure:1671
Process Request Failure:1633
Process Request Failure:1851
Process Request Failure:1501
Process Request Failure:1982
Process Request Failure:1545
Process Request Failure:1168
Process Request Failure:2021
Process Request Failure:1579
Process Request Failure:1705
Process Request Failure:1419
Process Request Failure:1411
Process Request Failure:1744
Process Request Failure:1593
Process Request Failure:1205
Process Request Failure:1742
Process Request Failure:1745
Process Request Failure:1189
Process Request Failure:1637
Process Request Failure:1846
Process Request Failure:1895
Process Request Failure:1442
Process Request Failure:1786
Process Request Failure:1642
Process Request Failure:1501
Process Request Failure:1681
Process Request Failure:1797
Process Request Failure:1432
Process Request Failure:1984
Process Request Failure:1846
Process Request Failure:1718
Process Request Failure:1725
Process Request Failure:1561
Process Request Failure:1471
Process Request Failure:1808
Process Request Failure:1380
Process Request Failure:1460
Process Request Failure:1893
Process Request Failure:577
Process Request Failure:1715
Process Request Failure:1542
Process Request Failure:1627
Process Request Failure:664
Process Request Failure:1532
Process Request Failure:2013
Process Request Failure:1806
Process Request Failure:1312
Process Request Failure:1274
Process Request Failure:1669
Process Request Failure:1757
Process Request Failure:1944
Process Request Failure:1014
Process Request Failure:1670
Process Request Failure:1695
Process Request Failure:1921
Process Request Failure:1784
Process Request Failure:1681
Process Request Failure:2117
Process Request Failure:1549
Process Request Failure:1612
Process Request Failure:1006
Process Request Failure:1526
Process Request Failure:1805
Process Request Failure:1595
Process Request Failure:1560
Process Request Failure:1959
Process Request Failure:1920
Process Request Failure:1484
Process Request Failure:1572
Process Request Failure:1570
Process Request Failure:1808
Process Request Failure:1651
Process Request Failure:1765
Process Request Failure:1560
Process Request Failure:1841
Process Request Failure:1585
Process Request Failure:1571
Process Request Failure:1844
Process Request Failure:1592
Process Request Failure:1550
Process Request Failure:1731
Process Request Failure:1813
Process Request Failure:1817
Process Request Failure:1667
Process Request Failure:1616
Process Request Failure:776
Process Request Failure:1898
Process Request Failure:1775
Process Request Failure:1904
Process Request Failure:1688
Process Request Failure:1494
Process Request Failure:1353
Process Request Failure:2036
Process Request Failure:1827
Process Request Failure:1633
Process Request Failure:1403
Process Request Failure:1941
Process Request Failure:1904
Process Request Failure:1689
Process Request Failure:1677
Process Request Failure:1906
Process Request Failure:1909
Process Request Failure:1812
Process Request Failure:1723
Process Request Failure:711
Process Request Failure:1479
Process Request Failure:1733
Process Request Failure:1851
Process Request Failure:1481
Process Request Failure:1982
Process Request Failure:1719
Process Request Failure:1839
Process Request Failure:1927
Process Request Failure:1633
Process Request Failure:1908

3. Stopping all clients.
48. Stopping all clients.
. Stopping all clients.
. Stopping all clients.
42
46. Stopping all clients.
. Stopping all clients.
15. Stopping all clients.
. Stopping all clients.

finished in 5.06s, 79870.00 req/s, 4.75GB/s
requests: 492306 total, 523084 started, 492306 done, 399350 succeeded, 92956 failed, 92956 errored, 0 timeout
status codes: 413422 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 23.77GB (25528115894) total, 63.92MB (67028824) headers (space savings 18.99%), 23.69GB (25440298128) data
                     min         max         mean         sd        +/- sd
time for request:     2.26ms       1.42s    109.62ms     43.07ms    81.06%
time for connect:     3.22ms       1.53s    564.76ms    475.87ms    53.22%
time to 1st byte:    33.97ms       1.57s    630.80ms    473.32ms    53.73%
req/s           :       0.00      507.28      157.04       80.58    69.04%
[info] CPU 6599.6% | Mem 7.0GiB

=== Best: 81703 req/s (CPU: 6599.6%, Mem: 7.0GiB) ===
[info] saved results/static-h2/1024/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to baseline-h2c
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to stream-grpc
[info] skip: varnish does not subscribe to stream-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

I can trigger a run that posts docker logs if you rebase this to main, if you need. Could be printing errors which slows down quite a lot.

that would be great!

@gquintard
gquintard force-pushed the add-varnish-framework branch from 9e8bebc to c62fddb Compare August 13, 2026 19:23
@MDA2AV
MDA2AV force-pushed the add-varnish-framework branch from c62fddb to 0a5f40d Compare August 13, 2026 19:36
@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

Repository owner deleted a comment from github-actions Bot Aug 13, 2026
Repository owner deleted a comment from github-actions Bot Aug 13, 2026
@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

─── varnish — status=exited exit=2 oom=false error= ─── varnish — last 120 log lines ─── | Error: | Must be no more than 1000 (thread_pool_max) | | (attempting to set param 'thread_pool_min' to '5000') ─── varnish — end of logs ───

servers run with a budget of 32 cores (64 SMT CPU threads) so typically the optimal worker/thread count is around 64 for these non async workloads

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

No results captured

Full log 
[warn] varnish did not come up for pipelined; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / pipelined / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for pipelined; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / limited-conn / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for limited-conn; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / limited-conn / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for limited-conn; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to json
[info] skip: varnish does not subscribe to json-comp
[info] skip: varnish does not subscribe to json-tls
[info] skip: varnish does not subscribe to upload
[info] skip: varnish does not subscribe to api-4
[info] skip: varnish does not subscribe to api-16

==============================================
=== varnish / static / 1024c (tool=wrk) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for static; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / static / 4096c (tool=wrk) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for static; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for static; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to static-tls
[info] skip: varnish does not subscribe to async-db
[info] skip: varnish does not subscribe to crud
[info] skip: varnish does not subscribe to fortunes

==============================================
=== varnish / baseline-h2 / 256c (tool=h2load) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for baseline-h2; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / baseline-h2 / 1024c (tool=h2load) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for baseline-h2; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / static-h2 / 256c (tool=h2load) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for static-h2; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / static-h2 / 1024c (tool=h2load) ===
==============================================
[info] waiting for server...
[warn] varnish did not come up for static-h2; skipping

─── varnish — status=exited exit=2 oom=false error=
─── varnish — last 120 log lines ───
  | Error:
  | Must be no more than 1000 (thread_pool_max)
  | 
  | (attempting to set param 'thread_pool_min' to '5000')
─── varnish — end of logs ───
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to baseline-h2c
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to stream-grpc
[info] skip: varnish does not subscribe to stream-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] skip: varnish does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

Sorry, that one is my fault, I need to up the max thread before the min one. I've fixed it now.

Thank you for the information on the machine. Varnish operates with a slightly different model than other server, with one thread per tasks, so we aren't going to map as cleanly to the CPU core counts, hence the rather large numbers you've seen.

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/benchmark -f varnish --save

@MDA2AV
MDA2AV force-pushed the add-varnish-framework branch from cfa465b to 59a7f93 Compare August 13, 2026 20:22
@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

Repository owner deleted a comment from github-actions Bot Aug 13, 2026
Repository owner deleted a comment from github-actions Bot Aug 13, 2026
@gquintard
gquintard force-pushed the add-varnish-framework branch from 59a7f93 to 683d50f Compare August 13, 2026 20:23
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

⚠️ /benchmark --save cannot start: main has diverged and cannot be auto-merged into this branch. Please merge or rebase main manually, push, and re-run /benchmark --save.

@gquintard
gquintard force-pushed the add-varnish-framework branch from 683d50f to a0a986b Compare August 13, 2026 20:59
@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

I ran directly on the server to see logs with min/max thread combinations of 100/500 and 1000/5000
logs are

Debug: Version: varnish-9.0.3 revision 0a625649cd40af4b6c10be5e58a2e89a5e275baa Debug: Platform: Linux,6.17.0-22-generic,x86_64,-jnone,-smalloc,-sdefault,-hcritbit Debug: Child (20) Started Child launched OK Info: Child (20) said Child starts

on both cases

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

cheers, do we have logs on the client side? Are they timing out, getting refused connections?

@MDA2AV

MDA2AV commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

`gcannon v0.5.3
Target: localhost:8080/
Threads: 64
Conns: 4096 (64/thread)
Pipeline: 1
Req/conn: unlimited (keep-alive)
Templates: 3
Expected: 200
Duration: 5s

Thread Stats Avg p50 p90 p99 p99.9
Latency 13.38ms 11.50ms 26.10ms 44.50ms 81.10ms

1021464 requests in 5.00s, 1018844 responses
Throughput: 203.65K req/s
Bandwidth: 32.02MB/s
Status codes: 2xx=1018844, 3xx=0, 4xx=0, 5xx=0
Latency samples: 1018841 / 1018844 responses (100.0%)
Per-template: 489798,265638,263405
Per-template-ok: 489798,265638,263405
[info] CPU 6518.1% | Mem 671MiB`

load generator looks quite healthy all 2xx , cpu usage too, expected from 64 cpu threads available

I'll try with a different load generator just to make sure it isn't a gcannon issue

@gquintard
gquintard force-pushed the add-varnish-framework branch from a0a986b to 6fe84c6 Compare August 13, 2026 21:05
@gquintard @claude
Add Varnish as a benchmarked framework
6fe84c6 
Varnish Cache accelerator with native TLS/H2 termination (varnishd -A),
vmod-fileserver serving /static/* from Varnish's own object cache, and a
custom vmod (vmod_httparena, built with varnish-rs) computing the
/baseline11 and /baseline2 sum in-process. No separate origin process.

Subscribes to baseline, pipelined, limited-conn, static, baseline-h2, and
static-h2 (HTTP/3 is out of scope: no QUIC support in OSS Varnish).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

well, that's fun, claude is really confused about the whole thread configuration, I've fix it manually. Sorry for all the back-end-forth

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

thanks, just so we are talking about the same thing, I was looking at #1110 (comment):

Process Request Failure:1479
Process Request Failure:1492
Process Request Failure:928
Process Request Failure:1521
Process Request Failure:1674
Process Request Failure:1961

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Nice, running now. When results are posted you can also see logs at the commited files

@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

thanks, just so we are talking about the same thing, I was looking at #1110 (comment):

Process Request Failure:1479
Process Request Failure:1492
Process Request Failure:928
Process Request Failure:1521
Process Request Failure:1674
Process Request Failure:1961

That is on the h2 load generator actually, h2load. I was testing on h1 locally only

@MDA2AV

MDA2AV commented Aug 13, 2026

Copy link
Copy Markdown
Owner

you can see the load gen output here https://github.com/MDA2AV/HttpArena/actions/runs/31744329441

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 204,942 6472.2% 592MiB ~0% ~0%
baseline 4096 186,085 6430.6% 852MiB ~0% ~0%
pipelined 512 285,972 6497.9% 538MiB ~0% ~0%
pipelined 4096 271,412 6494.2% 1.2GiB ~0% ~0%
limited-conn 512 168,691 6502.7% 596MiB ~0% ~0%
limited-conn 4096 172,852 6508.4% 932MiB ~0% ~0%
static 1024 109,610 6528.5% 2.9GiB ~0% ~0%
static 4096 110,957 6490.1% 8.2GiB ~0% ~0%
static 6800 109,069 6508.2% 9.5GiB ~0% ~0%
baseline-h2 256 201,583 6458.9% 4.6GiB ~0% ~0%
baseline-h2 1024 190,607 6491.7% 4.8GiB ~0% ~0%
static-h2 256 86,061 6631.4% 8.0GiB ~0% ~0%
static-h2 1024 82,428 6545.4% 9.0GiB ~0% ~0%
Full log 
Process Request Failure:1681
Process Request Failure:1391
Process Request Failure:1677
Process Request Failure:1754
Process Request Failure:1061
Process Request Failure:1684
Process Request Failure:1741
Process Request Failure:1336
Process Request Failure:1470
Process Request Failure:1646
Process Request Failure:1729
Process Request Failure:1474
Process Request Failure:1830
Process Request Failure:1661
Process Request Failure:1429
Process Request Failure:1483
Process Request Failure:1576
Process Request Failure:1164
Process Request Failure:1874
Process Request Failure:1448
Process Request Failure:1284
Process Request Failure:1483
Process Request Failure:1671
Process Request Failure:1279
Process Request Failure:1485
Process Request Failure:1673
Process Request Failure:1958
Process Request Failure:1292
Process Request Failure:1550
Process Request Failure:1378
Process Request Failure:1095
Process Request Failure:1396
Process Request Failure:1792
Process Request Failure:1657
Process Request Failure:1349
Process Request Failure:1440
Process Request Failure:1583
Process Request Failure:1799
Process Request Failure:1775
Process Request Failure:1785
Process Request Failure:1390
Process Request Failure:1542
Process Request Failure:1349
Process Request Failure:1796
Process Request Failure:1547
Process Request Failure:1158
Process Request Failure:1873
Process Request Failure:1552
Process Request Failure:1194
Process Request Failure:1679
Process Request Failure:1682
Process Request Failure:1792
Process Request Failure:1771
Process Request Failure:1657
Process Request Failure:1643
Process Request Failure:1626
Process Request Failure:1662
Process Request Failure:1550
Process Request Failure:1555
Process Request Failure:1751
Process Request Failure:1371
Process Request Failure:1418
Process Request Failure:1804
Process Request Failure:1367
Process Request Failure:1370
Process Request Failure:1779
Process Request Failure:1382
Process Request Failure:1495
Process Request Failure:1387
Process Request Failure:1803
Process Request Failure:1390
Process Request Failure:1569
Process Request Failure:1895
Process Request Failure:1721
Process Request Failure:2027
Process Request Failure:1434
Process Request Failure:1613
Process Request Failure:1900
Process Request Failure:1312
Process Request Failure:1851
Process Request Failure:1762
Process Request Failure:1904
Process Request Failure:1454
Process Request Failure:1738
Process Request Failure:1744
Process Request Failure:1872
Process Request Failure:1457
Process Request Failure:1231
Process Request Failure:1839
Process Request Failure:1505
Process Request Failure:1614
Process Request Failure:1776
Process Request Failure:1828
Process Request Failure:1780
Process Request Failure:1544
Process Request Failure:1657
Process Request Failure:1616
Process Request Failure:1752
Process Request Failure:1338
Process Request Failure:1254
Process Request Failure:1396
Process Request Failure:1879
Process Request Failure:1906
Process Request Failure:1518
Process Request Failure:1607
Process Request Failure:1868
Process Request Failure:1870
Process Request Failure:1445
Process Request Failure:1239
Process Request Failure:1912
Process Request Failure:1753
Process Request Failure:1336
Process Request Failure:1529
Process Request Failure:496
Process Request Failure:1751
Process Request Failure:1555
Process Request Failure:1484
Process Request Failure:1758
Process Request Failure:1613
Process Request Failure:1754
Process Request Failure:1463
Process Request Failure:1245
Process Request Failure:1768
Process Request Failure:1875
Process Request Failure:1378
Process Request Failure:1749
Process Request Failure:1759
Process Request Failure:1770
Process Request Failure:1614
Process Request Failure:1421
Process Request Failure:1883
Process Request Failure:1777
Process Request Failure:1492
Process Request Failure:1753
Process Request Failure:1684
Process Request Failure:792
Process Request Failure:1686
Process Request Failure:1495
Process Request Failure:2040
Process Request Failure:1358
Process Request Failure:1867
Process Request Failure:1622
Process Request Failure:1618
Process Request Failure:1550
Process Request Failure:1569
Process Request Failure:1890
Process Request Failure:1890
Process Request Failure:1814
Process Request Failure:1694
Process Request Failure:1852
Process Request Failure:1685
Process Request Failure:1554
Process Request Failure:1608
Process Request Failure:1723
Process Request Failure:1917
Process Request Failure:1305
Process Request Failure:1528
43. Stopping all clients.
39. Stopping all clients.
21. Stopping all clients.
21
59

53

finished in 5.06s, 81782.40 req/s, 4.88GB/s
requests: 484711 total, 516063 started, 484711 done, 408912 succeeded, 75799 failed, 75799 errored, 0 timeout
status codes: 417088 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 24.38GB (26174821935) total, 64.46MB (67586800) headers (space savings 19.00%), 24.29GB (26086229286) data
                     min         max         mean         sd        +/- sd
time for request:     3.99ms       2.74s    192.72ms     91.14ms    78.55%
time for connect:     2.74ms       2.51s       1.09s    818.94ms    54.88%
time to 1st byte:    13.73ms       2.65s       1.19s    873.65ms    54.11%
req/s           :       0.00      257.78      109.43       57.02    67.58%
[info] CPU 6545.4% | Mem 9.0GiB

=== Best: 82428 req/s (CPU: 6545.4%, Mem: 9.0GiB) ===
[info] saved results/static-h2/1024/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to baseline-h2c
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to stream-grpc
[info] skip: varnish does not subscribe to stream-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] skip: varnish does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/varnish.json - 13 new, 13 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard
gquintard marked this pull request as ready for review August 13, 2026 21:17
@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

thanks, I think the thread default configuration was pretty good, but I'll keep the changes as they are since they won't hurt.

Is there anything I should check?

@MDA2AV

MDA2AV commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

I see there are some errors in h2 tests, if you want to run locally to try to replicate I can point you to the scripts used but if you run claude I am sure he will find them.

Ill wait a bit (1-2 days) to merge because I am doing a few changes in the infrastructure category which is currently not showing so you wouldn't see results on the site.

Thanks for the contribution

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

Ah! I think found the issue: the framework is probably just triggering the Rapid Reset defense mechanism. I'll try a run without it.

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 206,211 6490.9% 576MiB +0.6% -2.7%
baseline 4096 201,636 6388.6% 859MiB +8.4% +0.8%
pipelined 512 289,792 6525.6% 528MiB +1.3% -1.9%
pipelined 4096 304,036 6539.2% 1.2GiB +12.0% ~0%
limited-conn 512 169,985 6493.4% 601MiB +0.8% +0.8%
limited-conn 4096 170,658 6320.0% 785MiB -1.3% -15.8%
static 1024 108,479 6511.7% 2.8GiB -1.0% -3.4%
static 4096 112,633 6529.7% 7.6GiB +1.5% -7.3%
static 6800 110,376 6464.9% 10.0GiB +1.2% +5.3%
baseline-h2 256 11,535 6354.2% 3.9GiB -94.3% -15.2%
baseline-h2 1024 4,196 6726.2% 5.4GiB -97.8% +12.5%
static-h2 256 86,126 6646.4% 7.7GiB ~0% -3.7%
static-h2 1024 15,046 6433.0% 7.4GiB -81.7% -17.8%
Full log 
Application protocol: h2
57Main benchmark duration is started for thread #59.
.
55.
60.
43.
20. Stopping all clients.
0. Stopping all clients.
9. Stopping all clients.
. Stopping all clients.

finished in 5.02s, 72747.60 req/s, 4.33GB/s
requests: 363738 total, 371930 started, 363738 done, 363738 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 367635 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 21.64GB (23235993651) total, 56.58MB (59329073) headers (space savings 19.06%), 21.57GB (23158889215) data
                     min         max         mean         sd        +/- sd
time for request:      178us       1.39s    109.64ms    110.46ms    95.40%
time for connect:     4.18ms    360.87ms     80.60ms     53.42ms    68.75%
time to 1st byte:    31.75ms    493.66ms    149.61ms     84.83ms    66.80%
req/s           :     250.79      331.19      284.14       14.28    68.36%
[info] CPU 6139.2% | Mem 7.4GiB

[run 2/3]
starting benchmark...
2.

TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
. Stopping all clients.Stopped all clients for thread #14
25. Stopping all clients.

39. Stopping all clients.
58

finished in 5.02s, 85582.00 req/s, 5.09GB/s
requests: 427910 total, 436102 started, 427910 done, 427910 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 431712 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 25.45GB (27324239794) total, 66.86MB (70103538) headers (space savings 18.96%), 25.36GB (27233283055) data
                     min         max         mean         sd        +/- sd
time for request:     1.40ms    329.00ms     94.06ms     26.90ms    70.92%
time for connect:     4.53ms    177.38ms     26.76ms     27.56ms    89.84%
time to 1st byte:     5.94ms    219.64ms     73.47ms     52.00ms    74.22%
req/s           :     292.93      373.54      334.26       13.56    70.31%
[info] CPU 6629.8% | Mem 8.1GiB

[run 3/3]
starting benchmark...

55.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
24.
16.
3.

4. Stopping all clients.
35. Stopping all clients.

finished in 5.03s, 85840.20 req/s, 5.10GB/s
requests: 429201 total, 437393 started, 429201 done, 429201 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 433218 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 25.52GB (27407046243) total, 67.11MB (70367373) headers (space savings 18.96%), 25.44GB (27315760383) data
                     min         max         mean         sd        +/- sd
time for request:     1.13ms    342.33ms     93.33ms     26.01ms    70.33%
time for connect:     3.52ms    175.87ms     56.21ms     53.18ms    78.13%
time to 1st byte:     7.97ms    226.10ms     83.16ms     58.93ms    62.50%
req/s           :     305.78      381.76      335.28       13.33    69.53%
[info] CPU 6646.4% | Mem 7.7GiB

=== Best: 86126 req/s (CPU: 6646.4%, Mem: 7.7GiB) ===
[info] saved results/static-h2/256/varnish.json
httparena-bench-varnish
httparena-bench-varnish

==============================================
=== varnish / static-h2 / 1024c (tool=h2load) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
starting benchmark...
progress: Main benchmark duration is started for thread #1242.
% of clients started
progress: Main benchmark duration is started for thread #8122% of clients started.Warm-up phase is over for thread #
59.

.
56Main benchmark duration is started for thread #33.
.
55.

.

.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
37. Stopping all clients.



[info] CPU 1929.9% | Mem 5.0GiB

[run 2/3]
starting benchmark...

.

.
25Warm-up phase is over for thread #Warm-up phase is over for thread #48.
34.
93% of clients started
100% of clients startedMain benchmark duration is started for thread #49.
14.51
.
52.

27.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2


7. Stopping all clients.

finished in 5.05s, 14119.20 req/s, 846.54MB/s
requests: 1931497 total, 1964265 started, 1931497 done, 70596 succeeded, 1860901 failed, 1860901 errored, 0 timeout
status codes: 75987 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 4.13GB (4438325590) total, 11.75MB (12323012) headers (space savings 18.98%), 4.10GB (4398340948) data
                     min         max         mean         sd        +/- sd
time for request:      490us       4.98s    199.18ms    465.15ms    90.73%
time for connect:     3.01ms       2.94s       1.03s    825.78ms    58.20%
time to 1st byte:    12.82ms       4.42s    967.40ms    698.98ms    68.32%
req/s           :       0.00      143.57       13.79       26.90    84.96%
[info] CPU 6433.0% | Mem 7.4GiB

[run 3/3]
starting benchmark...
8Warm-up phase is over for thread #.
13.
progress: Warm-up phase is over for thread #100% of clients started34.Main benchmark duration is started for thread #

0.
.Main benchmark duration is started for thread #21.

.
62.
10.
28.
22.
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2

30. Stopping all clients.

finished in 5.07s, 12605.20 req/s, 775.89MB/s
requests: 1965255 total, 1998023 started, 1965255 done, 63026 succeeded, 1902229 failed, 1902229 errored, 0 timeout
status codes: 69064 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 3.79GB (4067885844) total, 10.69MB (11214102) headers (space savings 18.96%), 3.75GB (4028775896) data
                     min         max         mean         sd        +/- sd
time for request:     3.44ms       4.87s    234.92ms    509.74ms    95.54%
time for connect:     3.06ms       2.37s    747.97ms    694.93ms    57.71%
time to 1st byte:    14.68ms       4.87s    573.88ms    753.39ms    94.73%
req/s           :       0.00       52.78       12.31       15.54    75.59%
[info] CPU 6522.4% | Mem 9.3GiB

=== Best: 15046 req/s (CPU: 6433.0%, Mem: 7.4GiB) ===
[info] saved results/static-h2/1024/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to baseline-h2c
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to stream-grpc
[info] skip: varnish does not subscribe to stream-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] skip: varnish does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/varnish.json - 13 new, 13 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard

gquintard commented Aug 13, 2026

Copy link
Copy Markdown
Author

well, that's fun: no more errors, but it destroys the numbers

@gquintard
gquintard force-pushed the add-varnish-framework branch from 3cde798 to 4bf4f5f Compare August 13, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MDA2AV MDA2AV Awaiting requested review from MDA2AV MDA2AV is a code owner
@Kaliumhexacyanoferrat Kaliumhexacyanoferrat Awaiting requested review from Kaliumhexacyanoferrat Kaliumhexacyanoferrat is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gquintard @MDA2AV