Skip to content

docker: jemalloc in published relay image + TLS-terminated stats stack#480

Open
gmarzot wants to merge 36 commits into
mainfrom
feature/docker-jemalloc-stats
Open

docker: jemalloc in published relay image + TLS-terminated stats stack#480
gmarzot wants to merge 36 commits into
mainfrom
feature/docker-jemalloc-stats

Conversation

@gmarzot

@gmarzot gmarzot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Two related additions to the docker deployment, plus load-test sizing for the CI relay.

1. jemalloc in the published relay image

  • docker/Dockerfile: install libjemalloc2 in the runtime stage.
  • docker/entrypoint.sh: LD_PRELOAD jemalloc by default (~10% relay speedup), probing the multiarch paths so it works on both amd64 and arm64 publish variants. Opt out with MOQX_JEMALLOC in {off,false,0,no}; an explicit path forces a specific lib. Graceful fallback (system allocator + warning) when the lib is absent.
  • The interop-client image is intentionally left unchanged — short-lived test client, direct ENTRYPOINT (no wrapper to inject LD_PRELOAD), relay-specific benefit. Easy to add later if we want parity.

2. Stats stack (docker-compose stats profile)

nginx (nginx:stable) terminates TLS reusing the relay's Let's Encrypt cert and reverse-proxies behind HTTP basic auth:

Route Target
/ tools/metrics-dashboard.html — afrind's live dashboard (#479)
/metrics relay admin, same-origin (sidesteps the dashboard's cross-origin/CORS caveat)
/admin/ relay admin JSON (/info, /state, /config)
/grafana/ Grafana (subpath)
/prometheus/ Prometheus UI/API
/healthz unauthenticated liveness
  • Basic-auth .htpasswd generated at container start from STATS_USER/STATS_PASSWORD (openssl apr1) or a verbatim STATS_HTPASSWD.
  • Prometheus scrapes the relay /metrics; Grafana is provisioned with the Prometheus datasource.
  • All three services are behind the stats profile — the base relay compose is unchanged for other users. Bring up with docker compose --profile stats up -d.

3. Load-test sizing

  • Relay: env-overridable cpus/mem_limit (defaults 8 cores / 8g) + MOQX_THREADS passthrough. deploy-relay.yml sets MOQX_CPUS/MOQX_THREADS=$(nproc) on the runner so the relay uses all cores for sub-maximal load testing.
  • Prometheus/Grafana get modest memory bounds.

4. Deploy wiring

  • deploy-relay.yml gains an opt-in enable_stats dispatch input. When set, it brings up the stats profile and passes STATS_* / GRAFANA_ADMIN_PASSWORD secrets. Default off — existing relay deploys are unchanged.

Dependency

Required repo secrets (only to enable stats on deploy)

STATS_USER, STATS_PASSWORD, and optionally GRAFANA_ADMIN_PASSWORD.

Testing (local)

  • jemalloc: bookworm-slim harness — lib installs to /usr/lib/x86_64-linux-gnu/libjemalloc.so.2; default preloads it (inherited by the exec'd process); MOQX_JEMALLOC=off disables. ✓
  • Stats stack end-to-end against the CI-published ghcr.io/openmoq/moqx image:
    • TLS + basic auth: no-auth → 401, wrong pw → 401, good auth → 200 serving the live dashboard. ✓
    • /metrics proxied same-origin returns the real relay metrics. ✓
    • Prometheus target moqx-relay …/metrics → up (scraping the real relay). ✓
    • /admin/info → relay JSON; Grafana /grafana/ subpath → 200. ✓
    • Relay resource limits applied (NanoCpus, Mem); jemalloc graceful-fallback path confirmed on the pre-change published image.

This change is Reviewable

Two related additions to the docker deployment.

jemalloc (published relay image, amd64 + arm64):
- Dockerfile installs libjemalloc2 in the runtime stage.
- entrypoint.sh LD_PRELOADs it by default (~10% relay speedup), probing the
  multiarch paths so it works on both arches. Opt out with MOQX_JEMALLOC in
  {off,false,0,no}; an explicit path forces a specific lib. Graceful fallback
  to the system allocator with a warning if the lib is absent.
- The interop-client image is intentionally NOT changed (short-lived test
  client, direct ENTRYPOINT with no wrapper, relay-specific benefit).

Stats stack (docker-compose 'stats' profile):
- nginx (nginx:stable) terminates TLS reusing the relay's Let's Encrypt cert
  and reverse-proxies behind HTTP basic auth:
    /            → tools/metrics-dashboard.html (live dashboard, see #479)
    /metrics     → relay admin, SAME-ORIGIN (avoids the dashboard's CORS caveat)
    /admin/      → relay admin JSON (/info, /state, /config)
    /grafana/    → Grafana (subpath)
    /prometheus/ → Prometheus UI/API
  /healthz is unauthenticated. Basic-auth .htpasswd is generated at start from
  STATS_USER/STATS_PASSWORD (openssl apr1) or a verbatim STATS_HTPASSWD.
- prometheus scrapes the relay /metrics; grafana is provisioned with the
  Prometheus datasource.

Load-test sizing:
- Relay gets env-overridable cpus/mem (default 8 cores / 8g); MOQX_THREADS is
  passthrough. deploy-relay sets MOQX_CPUS/MOQX_THREADS=$(nproc) on the runner.
- Prometheus/Grafana get modest memory bounds.

deploy-relay.yml gains an opt-in enable_stats input; when set it brings up the
stats profile and passes STATS_*/GRAFANA secrets. Default off — existing relay
deploys are unchanged.

Depends on #479 for tools/metrics-dashboard.html (served at /).
@gmarzot
gmarzot requested review from afrind and peterchave July 2, 2026 11:26
gmarzot added 2 commits July 2, 2026 07:59
…oc-stats

# Conflicts:
#	docker/entrypoint.sh
The stats nginx is the only host-exposed service (grafana/prometheus are
reached only via its proxy on the internal network). Default-bind it to
localhost so a locked-down host only serves it over an SSH tunnel; STATS_BIND
overrides to 0.0.0.0 for public exposure. Refresh the routes comment.

@afrind afrind left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@afrind reviewed 9 files and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on peterchave).

gmarzot and others added 23 commits July 2, 2026 19:51
…retention

The stats profile is now relay-resident only: nginx (renamed moqx-proxy) serves
afrind's live dashboard + proxies /metrics and /admin, with Prometheus for
historical browsing. Grafana and any broader analytics stack (ClickHouse, etc.)
are decoupled — they can scrape the relay /metrics from wherever they live and
are intentionally not bundled into the moqx deploy.

- docker-compose.yml: rename nginx container moqx-stats-proxy → moqx-proxy;
  remove the grafana service, grafana-data volume, and grafana from nginx
  depends_on. Prometheus retention.time 30d → 365d and add a hard
  retention.size cap (10GB, env PROMETHEUS_RETENTION_SIZE) as a safety valve.
- nginx template: remove the /grafana/ location and the now-unused
  Grafana-Live websocket map; keep /, /metrics, /admin/, /prometheus/, /healthz.
- prometheus.yml: scrape the relay at 3s (fine-grained perf history; metrics are
  fixed ~450-series cardinality so 3s×365d stays well under 10GB) and pin the
  Prometheus self-scrape to 60s to keep the total sample rate down.
- deploy-relay.yml: drop the GRAFANA_ADMIN_PASSWORD passthrough and the input
  description mention.
- .env.example: remove Grafana creds; document PROMETHEUS_RETENTION(_SIZE).
- Delete docker/grafana/ provisioning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grafana visualizes the Prometheus data (auto-provisioned datasource) — saved
dashboards + ad-hoc PromQL over the full retention window, complementing
afrind's transient live dashboard. Reinstated the grafana service, grafana-data
volume, /grafana/ nginx route + websocket map, provisioning datasource, and the
GRAFANA_ADMIN_PASSWORD deploy passthrough.

Kept from the prior commit: moqx-proxy rename, 3s relay scrape (60s self-scrape),
and Prometheus 365d + 10GB retention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defaults to the existing named volume (unchanged behavior — persists across
redeploys). Set PROMETHEUS_DATA_DIR to an absolute host path to pin the store
on a dedicated/larger disk without editing compose.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rigin)

afrind's metrics-dashboard.html ships with a hardcoded default endpoint
(18.236.159.20:19702). nginx sub_filter rewrites it to /metrics as the page is
served, so opening https://<domain>/ scrapes THIS relay same-origin on first
load (CORS-free, basic-auth reused) — turnkey, with no IP to type. The source
file is left untouched (stays correct for standalone/Pages use).

Verified: served HTML defaults to value="/metrics"; foreign IP no longer present.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An explicit domain=moqx-main was used verbatim, so Route53 tried to create the
RRSet "moqx-main." at the root and rejected it (not in zone ci.openmoq.org).
Append .ci.openmoq.org when the input has no dot, so a bare label works the same
as the full FQDN. Also shorten the enable_stats input description.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provision a "moqx relay overview" Grafana dashboard (as code) that reproduces
afrind's tools/metrics-dashboard.html panels via PromQL over Prometheus — the
same metrics, but over the full retention window instead of transiently.

- node-exporter sidecar (stats profile): host CPU / NIC-throughput / memory for
  the dashboard's host panels (the relay /metrics has none). Reads /proc+/sys
  read-only; only cpu/netdev/meminfo/loadavg collectors enabled to keep series
  (and disk) low. Scraped by a new `node` job at 5s so host saturation lines up
  with the 3s relay data during load tests.
- prometheus.yml: add the `node` job.
- grafana datasource: pin uid=prometheus so the dashboard references it
  deterministically.
- grafana dashboards provider + moqx-overview.json (12 panels): sessions,
  active conns/streams, host CPU, NIC throughput, QUIC goodput/packets,
  loss+rtx ratios, subgroup errors by code, object-ACK & event-loop-busy
  percentiles, QUIC RTT, host memory.

Validated end-to-end locally: all Prometheus targets up, datasource + dashboard
auto-provisioned, panel query returns data through Grafana's datasource proxy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Relay at 3s + host at 5s is ~12 GB/yr, above the compose default 10G cap. The
CI runner has the disk, so write PROMETHEUS_RETENTION_SIZE=20GB into the stats
.env — the 365d time limit becomes the effective bound and a full year is
retained with headroom. Generic compose default stays 10G for other users.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…irects

- Grafana "Subgroup reset errors by code" panel showed "No data" when there
  were zero resets (the per-code series only exist after an error occurs).
  Add an `or label_replace(vector(0), ...)` fallback so it renders a flat 0
  baseline ("no errors") instead of the empty/warning state.
- nginx: redirect the slashless /grafana and /prometheus to their trailing-
  slash forms (location /grafana/ doesn't match bare /grafana, which otherwise
  falls through to the dashboard/404).

Both validated: dashboard JSON parses; nginx -t passes and bare paths 301.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_exporter

Reworks the Grafana dashboard into three sections and adds the per-track
subscriber view, which the aggregate /metrics can't provide.

- json_exporter sidecar (stats profile) scrapes the relay admin /state and
  emits per-track series (moqx_track_subscribers{track=...}); Prometheus probes
  it via a new moqx-state job at 15s. Per-track cardinality scales with distinct
  track names — hence the modest interval.
- node_exporter: enable the uname collector for hostname/arch (host-info panel).
- Dashboard (moqx-overview.json), now 21 panels in 3 rows:
    Platform — host facts (hostname/arch/cores/mem), CPU & memory gauges +
      history, network throughput in/out, active sessions/tracks stats.
    MoQ/tracks — subscribers-per-track over time, QUIC active conns/streams.
    Details — goodput, packets/s, loss & rtx ratio, object-ACK & event-loop
      latency percentiles, QUIC RTT, subgroup errors by code, aggregate sessions.
  Host IP shown as the relay domain (node_exporter doesn't expose IP).

Validated end-to-end locally: all 4 Prometheus targets up (relay, moqx-state,
node, self), uname/cores/mem/NIC queries return data, per-track query yields the
zero-baseline fallback when idle, dashboard auto-provisions and queries resolve
through Grafana.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The logging redesign moved the whole stack to folly XLOG (docs/logging.md), so
the old GLOG numeric verbose input was the wrong knob. Replace it with a
`logging` choice (default/WARN/DBG1..DBG4) that sets FOLLY_LOGGING, which the
relay's folly::Init reads directly — verified end-to-end (WARN quiets, DBG1
verbose) against the existing image, so no rebuild is needed. Also add
FOLLY_LOGGING passthrough to the compose moqx env (MOQX_LOG_LEVEL/MOQX_VERBOSE
kept as quiet legacy glog-stack defaults), and shorten the enable_stats dialog
to one line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…LOGGING)

FOLLY_LOGGING is folly's own env var (kLoggingEnvVarName in folly/init/Init.h);
we shouldn't make users set a folly-namespaced var directly. Introduce a
MOQX_-namespaced knob, MOQX_XLOG, and have compose drive FOLLY_LOGGING from it
(FOLLY_LOGGING: ${MOQX_XLOG:-}). deploy-relay writes MOQX_XLOG; .env.example
documents it as the primary log knob. Verified end-to-end: MOQX_XLOG=WARN quiets
the relay, empty = baseline INFO — still no image rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make MOQX_LOGGING a first-class knob honored by the binary itself: main.cpp
promotes it to folly's FOLLY_LOGGING (before folly::Init) unless FOLLY_LOGGING
is already set — so it works for any launch method with no wrapper. Name mirrors
both the --logging flag and folly's FOLLY_LOGGING.

Precedence (verified end-to-end, low→high):
  compiled default < MOQX_LOGGING < FOLLY_LOGGING < --logging flag

- src/main.cpp: getenv/setenv shim + <cstdlib>.
- entrypoint.sh: drop the inert MOQX_LOG_LEVEL/MOQX_VERBOSE→GLOG_* mappings
  (glog is only a fatal-assert backstop now; keep GLOG_logtostderr so a CHECK
  still shows in `docker logs`).
- docker-compose.yml: pass MOQX_LOGGING through; remove the glog knobs.
- .env.example: document MOQX_LOGGING; drop the dead glog vars.
- docs/logging.md: new "Environment variables" section with the precedence.

Requires a rebuilt image (relay binary changed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…only landing

Dashboard reorg into a Platform section: a Node info cluster (host facts /
cores / memory / endpoint), three ring gauges (CPU / RAM / HD), current
ingress/egress BW stat panels, and two timeseries below — combined CPU/RAM/HD
utilization and network in/out. MoQ and Details sections retained.

- node-exporter: enable the filesystem collector + host rootfs (ro) mount for
  node_filesystem_* → the HD gauge / disk line (verified: root fs 14.6% used).
- grafana: land on the relay dashboard (GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH).
- provider.yml: allowUiUpdates=false → provisioned dashboards are READ-ONLY
  (the default landing page); edit via the import/export dev workflow (next).

Validated end-to-end: 23 panels provision, disk + BW queries resolve through
Grafana, dashboard reports provisioned=true.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provisioned dashboards are read-only (the landing page). To edit:
  import-dashboards.sh  -> editable "<name> (dev)" copies (uid -dev, "moqx (dev)"
                           folder); the read-only originals stay put.
  ...edit in the UI...
  export-dashboards.sh  -> strips -dev, writes clean JSON back to
                           provisioning/dashboards/ (archives prior); --dry-run diffs.

Scripts run on the box (docker + python3) and reach Grafana via `docker exec`
(admin pw from the container) — no published port or basic-auth needed.
Modeled on the red5 export/import flow. Verified: import creates the dev copy,
export round-trips it back. (First export normalizes the hand-authored JSON to
Grafana's canonical expanded form — expected.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Was a checked-in weak default (admin/admin if unset). Require it from the
deploy secret / docker/.env instead. Login stays gated by nginx regardless.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A tokenized, unauthenticated, read-only Grafana public-dashboard view, served on
a separate public port (STATS_PUBLIC_PORT, default 8443) — OFF until STATS_PUBLIC_BIND=0.0.0.0.

- nginx: a second server block on the public port proxies ONLY the Grafana
  public-dashboard routes (/grafana/public-dashboards/, /api/public/, /public/);
  login, /metrics, /prometheus, and everything else return 403. Per-IP limit_req
  (5r/s burst 20) + limit_conn (10), and an includable denylist.conf for banning.
  The private 443 (tunnel, basic-auth) surface is unchanged.
- grafana: GF_PUBLIC_DASHBOARDS_ENABLED=true; anonymous org auth explicitly OFF.
- publish-public-dashboard.sh: publish / status / pause / resume / unpublish
  (kill switch) via the Grafana API.
- README: enable/link/ban/rate-limit/fail2ban runbook.

Verified end-to-end: public port serves only the token URL (200, no auth) and
403s everything else; private 443 still 401s without auth; rate limiter fires
(429s under burst); pause/unpublish revoke access. Nothing until you opt in.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-out)

New public_dashboard input (default true): when stats is enabled it binds the
public port to 0.0.0.0, opens the ufw port, and publishes the read-only public
dashboard after Grafana is healthy — printing the URL. Uncheck to keep private:
bind stays localhost, the dashboard is paused, and the ufw rule is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enable_stats now governs the whole stats surface incl. the public read-only
dashboard: on = stack up + public port bound + ufw opened + dashboard published;
off = none of it. Dropped the separate public_dashboard input.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TCP 4533 is inside the 4433-4533 range the Linode cloud firewall already permits
(verified reachable externally), so the public dashboard needs no cloud-firewall
change — just the host ufw (which already covers the range). Switch the
STATS_PUBLIC_PORT default 8443 -> 4533 across compose, deploy-relay, the publish
script, and the README.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gmarzot and others added 10 commits July 9, 2026 10:18
…to localhost

- nginx public port: a Grafana dashboard fires one query per panel plus assets
  on load (tens of requests at once), tripping the old 5r/s burst=20 with a 429.
  Raise to rate=50r/s burst=150, limit_conn 30 — absorbs a full page load while
  a sustained flood still 429s.
- relay admin :8000 now binds 127.0.0.1 (was 0.0.0.0, relying on the firewall).
  The health check uses localhost and nginx reaches it over the internal network,
  so it never needs a public bind — /state /config no longer face the internet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ashboard

Factor the relay deploy (write .env → pull → up [+ stats + public dashboard] →
health check → publish) into docker/relay-deploy.sh, and call it from both
ci-main's auto-deploy (on main) and deploy-relay (manual) so they never drift.

- ci-main deploy: was a bare `docker compose up` (no stats), so any push to main
  would `--remove-orphans` the manually-deployed stats stack. Now runs the shared
  script with ENABLE_STATS=true → stats + public read-only dashboard survive.
  Also drops the dead MOQX_LOG_LEVEL/MOQX_VERBOSE glog vars from its .env.
- deploy-relay: keeps its restart-only + cert + image-tag logic, delegates the
  rest to the same script (PULL_IMAGE + MOQX_LOGGING passed in).

So once #480 merges, every main deploy brings up the relay + stats + published
public dashboard (4533) — no manual redeploy after the nightly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rate({__name__=~'moqx_(pub|sub)SubgroupReset_total'}) drops __name__, so pub+sub
series with the same code collapsed to identical labelsets -> 'vector cannot
contain metrics with the same labelset'. Tag each side with a kind label before
the union so sum by(code) aggregates cleanly. Codes are always present (relay
inits them at 0), so drop the redundant vector(0) fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ping

Fixes three dashboard gaps observed against the live relay:

- BW read flat because node-exporter ran in a bridged netns and only saw
  its own veth (/proc/net/dev is netns-scoped). Give it network_mode: host
  (+ uts: host so uname reports the real host name); it now sees eth0 — the
  true platform ingress/egress. Prometheus reaches it via host.docker.internal.
  BW panels drive off eth0 NIC counters, with moqx quicBytes rate overlaid as
  app-level context.

- Per-track/subscriber series were fine (json_exporter emits them) but lacked
  a namespace dimension. Add a slash-joined namespace label + a $namespace
  template var scoping the per-track and active-tracks panels.

- Replace the chunky Node/Cores/Memory/Endpoint tiles with one full-width
  Business Text (marcusolsson-dynamictext-panel) host card: hostname, relay
  id, endpoint, OS/kernel/arch, cores, RAM, disk. New moqx_relay{relay_id}
  info metric feeds the id.

Relay itself still uses bridge networking; moving it to host mode (better NIC
path for the UDP relay) is a separate follow-up.
…URLs, persist :9100 ufw

- Subscriber counts live on the is_publish=true /state entry (publisher-side
  fan-out), and $namespace is uninterpolated on public dashboards (Grafana
  public views don't support template variables). Both filters made the
  active-tracks + per-track panels empty under load. Drop them: active-tracks
  counts all tracks; per-track shows topk(10, max by namespace,track).

- Host card: relay_id now query-driven (label_replace instance-rewrite so it
  joins the host row) instead of an uninterpolated ${relay_id} var; list the
  real MoQT (moqt://...:4433) and WebTransport (https://...:4433/moq-relay)
  URLs plus an OS/arch/cores synopsis.

- relay-deploy.sh: persist the docker-subnet -> host :9100 ufw allow so the
  host-networked node-exporter stays scrapable across deploys.
Public dashboards can't use template variables, so make the MoQ/tracks section
query-driven instead of selector-driven:
- counters row: active sessions / tracks / subscriptions + aggregate timeseries
- namespaces-by-subscribers bar gauge (sum by namespace) as the namespace list
- tracks-by-subscribers table (sortable, color-graded)
- full-width top-10 subscribers-per-track timeseries (legend-click to isolate)
- relocate QUIC connections/streams into Details

Note: per-track counts come from /state forwarder->subscriberCount() which lags
and undercounts the aggregate under load; leaderboard is indicative, panel 21
(aggregate) is authoritative. Real fix is per-track gauges in the relay /metrics.
…query error)

After the query-driven MOQT redesign no panel references $namespace or $relay_id.
Public dashboards issue a panelId-less query per template variable refresh, which
Grafana rejects (panels/undefined/query -> 'invalid panel id'). Remove both vars;
the interactive $namespace view belongs on a separate authed dashboard.
…t ufw

The relay sizes its UDP socket buffer to net.core.wmem_max (entrypoint.sh), but
the box was at the stock 208 KB default — a hard throttle on a high-fanout QUIC
relay (compounded by the userland UDP docker-proxy on 4433). relay-deploy.sh now
writes /etc/sysctl.d/99-moqx-quic.conf and applies it: rmem_max/wmem_max=16 MiB,
netdev_max_backlog=10000, optmem_max=65536. net.core.* are host-global so they
must live on the host; the relay picks up the larger buffer on its next restart.

Also add a nofile=1048576 ulimit to the relay, and drop relay-deploy's explicit
ufw allow for the pub port since 4533 sits inside the open 4433-4533 range.
Move the moqx relay into the host network namespace so QUIC traffic binds eth0
directly: eliminates docker bridge DNAT and the userland UDP docker-proxy on the
hot 4433/4434 path (the measured throughput ceiling), gives native source IPs,
and makes platform-NIC BW == relay BW. Combined with this restart the relay also
picks up the 16 MiB UDP buffers + nofile ulimit.

Because the relay leaves the compose bridge, the bridged stats containers reach
its admin :8000 via the host gateway:
- prometheus.yml: moqx:8000 -> host.docker.internal:8000 (relay + state jobs)
- nginx template: proxy_pass moqx:8000 -> host.docker.internal:8000 (/metrics,/admin)
- json-exporter + nginx services: add extra_hosts host-gateway
- relay-deploy.sh: ufw allow docker subnet -> host :8000 (external stays denied)

Admin :8000 moves from a 127.0.0.1-only publish to eth0-bound but ufw-gated
(only 22/443/4433:4533 open externally); metrics are non-secret on this open
test relay.
…y path)

Kernel tuning + firewall rules are host configuration, not app bring-up, and were
being re-applied on every redeploy. Move them into an idempotent, run-once
docker/setup-host.sh (sysctl UDP tuning + the docker-subnet :8000/:9100 allows +
the public 4433:4533 range). relay-deploy.sh now only verifies provisioning and
warns (non-fatal) if net.core.wmem_max looks unset, pointing at setup-host.sh.

setup-host.sh applies just its own drop-in (sysctl -p <file>, not --system) to
avoid surfacing unrelated sysctl errors from other host drop-ins.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants