feat(quic)!: shard the relay's QUIC endpoint per core by default

.github/workflows/nightly.yml

@@ -68,6 +68,8 @@ jobs:
             artifact-path: target/bench/ffi
           - name: Audio grouping relay benchmark
             recipe: bench-audio
+          - name: Relay shard sweep
+            recipe: bench-shard
           # Smoke-tests the capture path: exact off-CPU waits, resolved worker stacks and CPU.
           - name: Relay lock-wait profile
             recipe: bench-lock fanout-heavy 4

bench/README.md

@@ -69,6 +69,21 @@ Runtime comparison requires Linux because io\_uring and relay process metrics
 come from Linux interfaces. The default worker count is the number of online
 logical CPUs.
 
+Sweep the relay's QUIC shard count against the connection count, for the
+room (`video`: every connection publishes and subscribes) and `fanout` (one
+publisher, the rest subscribers) workloads:
+
+```bash
+nix develop --command just bench-shard
+nix develop --command just bench-shard "0 1 8 24" "64 512"
+```
+
+Zero shards is the shared runtime with one Tokio thread per logical CPU; any
+other count is that many unpinned Tokio workers. The table reports CPU per
+connection, so a steering, dispatch, or per-shard cost that grows with either
+axis reads as a slope. Linux only, since nothing else steers a reuseport
+group. It runs nightly.
+
 Attribute relay worker lock waits to the stacks that blocked, as a share of
 each worker's CPU:
 

bench/run.sh

@@ -8,6 +8,7 @@ set -euo pipefail
 # on the current tree. The current load generator drives every relay so the
 # workloads stay identical. `--audio` isolates the audio grouping/load-shape matrix.
 # `--lock` records relay worker off-CPU stacks to attribute std mutex waits.
+# `--shard` sweeps the QUIC shard count against the connection count.
 
 MODE=${1:-}
 ROUNDS=${MOQ_BENCH_RUNTIME_ROUNDS:-3}
@@ -27,6 +28,17 @@ case $MODE in
         # Without frame pointers, perf's DWARF unwinder stops at the first relay frame.
         export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-C force-frame-pointers=yes"
         ;;
+    --shard)
+        BASE=
+        SHARD_COUNTS=${MOQ_BENCH_SHARD_COUNTS:-0 1 4 16}
+        SHARD_CONNECTIONS=${MOQ_BENCH_SHARD_CONNECTIONS:-16 64 256}
+        for value in $SHARD_COUNTS $SHARD_CONNECTIONS; do
+            if [[ ! $value =~ ^[0-9]+$ ]]; then
+                printf 'shard benchmark counts must be integers, got %q\n' "$value" >&2
+                exit 1
+            fi
+        done
+        ;;
     --runtime)
         BASE=
         RUNTIME_WORKERS=${MOQ_BENCH_RUNTIME_WORKERS:-}
@@ -492,6 +504,61 @@ run_runtime_comparison() {
     print_runtime_comparison
 }
 
+# Each workload's load axis is its connection count: every room connection
+# publishes and subscribes, while fanout adds subscribers to one publisher. A
+# cost that grows with the shard count or the load shows up as a slope across
+# the shard counts or the connection counts, which one default-versus-single-
+# socket run cannot show.
+run_shard_sweep() {
+    if [[ $(uname -s) != Linux ]]; then
+        printf 'shard sweep needs Linux: only it steers a reuseport group\n' >&2
+        return 1
+    fi
+    generate_certificate
+
+    printf 'Building relay and load generator...\n'
+    CARGO_TARGET_DIR=$CURRENT_TARGET cargo build --locked --release -p moq-relay -p moq-bench
+    LOAD_BIN=$CURRENT_TARGET/release/moq-bench
+    HOST_BIN=$CURRENT_TARGET/release/moq-bench-host
+
+    local cpus workload connections shards label
+    cpus=$(getconf _NPROCESSORS_ONLN)
+    for workload in video fanout; do
+        for connections in $SHARD_CONNECTIONS; do
+            for shards in $SHARD_COUNTS; do
+                label=shard-$shards-$connections
+                printf '  %s: %d connections, %d shards\n' "$workload" "$connections" "$shards"
+                # Zero shards is the shared runtime, given every CPU.
+                if ((shards == 0)); then
+                    run_workload "$label" "$CURRENT_TARGET/release/moq-relay" "$workload" tokio-shared "$cpus" \
+                        --connections "$connections"
+                else
+                    run_workload "$label" "$CURRENT_TARGET/release/moq-relay" "$workload" tokio-workers "$shards" \
+                        --connections "$connections"
+                fi
+            done
+        done
+    done
+
+    printf '\nShard sweep (0 shards is the shared runtime on %d threads)\n' "$cpus"
+    printf '%-8s %11s %7s %10s %10s %8s %7s %8s %9s %9s\n' \
+        workload connections shards recv-Mbps recv-fps p99-ms loss-% CPU CPU/conn RSS-MiB
+    for workload in video fanout; do
+        for connections in $SHARD_CONNECTIONS; do
+            for shards in $SHARD_COUNTS; do
+                jq -r --arg connections "$connections" --arg shards "$shards" '[
+                    .workload, $connections, $shards, .recv_mbps, .recv_fps, .latency_p99_ms, .loss_pct,
+                    .cpu_cores, (.cpu_cores / ($connections | tonumber)), (.rss_bytes / 1048576)
+                ] | @tsv' "$RUN/relay/shard-$shards-$connections/$workload/summary.json" |
+                    while IFS=$'\t' read -r name conns count recv fps p99 loss cpu per rss; do
+                        printf '%-8s %11s %7s %10.3f %10.1f %8.1f %7.3f %8.3f %9.5f %9.1f\n' \
+                            "$name" "$conns" "$count" "$recv" "$fps" "$p99" "$loss" "$cpu" "$per" "$rss"
+                    done
+            done
+        done
+    done
+}
+
 run_lock_profile() {
     if [[ $(uname -s) != Linux ]]; then
         printf 'lock profiling needs Linux for perf and /proc schedstat\n' >&2
@@ -543,6 +610,9 @@ elif [[ $MODE == --runtime ]]; then
 elif [[ $MODE == --lock ]]; then
     run_lock_profile
     exit 0
+elif [[ $MODE == --shard ]]; then
+    run_shard_sweep
+    exit 0
 elif [[ -n $BASE ]]; then
     BASE_COMMIT=$(git rev-parse --verify "$BASE^{commit}")
     WORKTREE=$RUN/base

doc/bin/relay/config.md

@@ -110,14 +110,15 @@ window, so very small values can prevent other requests from progressing.
 ## \[runtime]
 
 By default one work-stealing runtime serves every connection off one UDP
-socket. On Linux, QUIC can instead run on pinned single-threaded workers, each
-with its own socket on the listen address (`SO_REUSEPORT`), so a connection is
-handled start to finish by one thread and its packets never cross cores.
+socket. On Linux, QUIC can instead run on single-threaded workers, each with
+its own socket on the listen address (`SO_REUSEPORT`), so a connection is
+handled start to finish by one worker thread and its packets never cross
+threads. `pin = true` also keeps that thread on one core.
 
 ```toml
 [runtime]
 workers = 8                          # Single-threaded QUIC workers. Omit for the shared runtime.
-pin = true                           # Pin each worker to a core. Default.
+pin = false                          # Pin each worker to a core. Off by default.
 io_uring = false                     # Drive them with io_uring instead of tokio.
 ```
 
@@ -126,6 +127,10 @@ worker. `workers` needs the `quic` feature. Every worker and the TLS stream
 listener serve one shared identity, generated or file-backed, so a certificate
 rotation reaches them together and `/certificate.sha256` stays accurate.
 
+Pinning is off by default because every relay pins its workers to the same
+first cores, so two relays on one host, or one limited to fewer CPUs than the
+host has, would crowd onto them. Turn it on for a host the relay has to itself.
+
 `io_uring` additionally needs Linux 6.12+ and the `io-uring` cargo feature; it
 shares the same reloadable TLS identity, serves moq-lite only, and refuses
 `mtu_discovery`. A setting the build or host cannot deliver refuses to

justfile

@@ -60,6 +60,11 @@ bench-audio:
 bench-runtime $ROUNDS="3" $WORKERS="":
     MOQ_BENCH_RUNTIME_ROUNDS="$ROUNDS" MOQ_BENCH_RUNTIME_WORKERS="$WORKERS" bench/run.sh --runtime
 
+# Sweep the relay's QUIC shard count against room and fanout connection counts;
+# 0 shards is the shared runtime. Example: `just bench-shard "0 1 8" "64 512"`.
+bench-shard $SHARDS="0 1 4 16" $CONNECTIONS="16 64 256":
+    MOQ_BENCH_SHARD_COUNTS="$SHARDS" MOQ_BENCH_SHARD_CONNECTIONS="$CONNECTIONS" bench/run.sh --shard
+
 # Attribute relay worker std mutex waits by stack, for each worker count.
 bench-lock $WORKLOAD="fanout" $WORKERS="1 4 16":
     MOQ_BENCH_LOCK_WORKLOAD="$WORKLOAD" MOQ_BENCH_LOCK_WORKERS="$WORKERS" bench/run.sh --lock

quest/m1/quic/README.md

@@ -72,8 +72,6 @@ cache limits remain separate; bitrate pacing composes with this memory cap.
   datagrams join their subscription's send group instead of preempting every stream
 - [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - MAX_STREAMS
   and MAX_DATA are raised after SETUP identifies a cluster peer
-- [Shard the endpoint](/quest/m1/quic/shard.md) - the library shards a
-  server endpoint across cores by default, replacing the `--workers` group
 
 ## Related
 

quest/m2/perf/README.md

@@ -77,6 +77,7 @@ optimization.
 - [Run to quiescence](/quest/m2/perf/uring-quiescence.md) - a received packet's reply is staged in the same turn, under a pass and train budget that keeps the fairness rule
 - [Ingest batch](/quest/m2/perf/ingest-batch.md) - relay ingest pays one lock, wake, and clock read per burst of whole frames instead of per frame
 - [Remove moq-uring copies](/quest/m2/perf/uring-copies.md) - egress, stream send and receive, and datagram receive stop copying where the QUIC core already allows it, with before/after measurements on `moq-quic`
+- [Shard by default](/quest/m2/perf/shard-default.md) - a saturating shard sweep decides whether an unset `runtime.workers` runs one worker per core on Linux
 
 ## Related
 

quest/m2/perf/kio-channel-contention/README.md

@@ -135,7 +135,7 @@ Decided (maintainer, 2026-10-14, per-thread follow-up):
 
 - Per-worker sharding is no longer ruled out; two spikes decide. Both run on
   the thread-per-core workers (`--runtime-workers`, the default once
-  [Shard](/quest/m1/quic/shard.md) lands) and link workers over an
+  [Shard by default](/quest/m2/perf/shard-default.md) lands) and link workers over an
   in-memory transport (`moq_net::transport::memory`),
   never real QUIC. The
   [replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md)

quest/m2/perf/shard-default.md

@@ -0,0 +1,37 @@
+# [M] Shard the relay by default
+
+## Goal
+
+A saturating benchmark shows whether thread-per-core QUIC beats the shared
+runtime, and the relay's default follows the measurement. If sharding wins,
+an unset `runtime.workers` runs one worker per core on Linux, and
+`workers = 0` is the explicit way back to the shared runtime. Off Linux, the
+shared runtime stays the default.
+
+## Plan
+
+Sharding is opt-in: `runtime.workers` is unset by default, and
+`runtime.pin` defaults off. The maintainer decided on #5688 to keep it that
+way until a measurement justifies the flip.
+
+- Extend `just bench-shard` to load that saturates the shared runtime, for
+  example 1024+ connections or more than one load host. At 16 to 256
+  connections, #5688's sweep never used one full core, so it showed no
+  regression from sharding and no win either. The 16-shard groups spent about
+  15 to 25% more CPU than the shared runtime at low load, from idle per-core
+  runtimes.
+- Flip the default only if sharding raises the throughput ceiling or cuts
+  CPU per connection at saturation. A measured no-win deletes this quest.
+- Shard only where a group serves the same listener the shared runtime would
+  open. A stream-only relay, or one with a QUIC-LB `lb_id`, stays shared
+  unless a count is set. A count sized from `available_parallelism`, capped at
+  `moq_sock::shard::MAX_SHARDS`, follows cgroup quotas.
+- With the default flipped, `runtime.io_uring` no longer needs an explicit
+  count.
+
+Update `doc/bin/relay/` for the default change.
+
+## Related
+
+- [kio channel contention](/quest/m2/perf/kio-channel-contention/README.md) - its per-thread spikes run on the workers
+- [UDP demux](/quest/m2/p2p/udp-demux.md) - demuxes each shard's socket