Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions adr/003-mojo-default.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,13 +161,11 @@ Each follow-up is a one-page issue. None blocks this ADR.

3. ~~**Rate limiting.**~~ **Shipped (per-process):** `RateLimit[Inner]` token bucket tuned by `RATE_LIMIT_RPS` / `RATE_LIMIT_BURST` (0 = disabled). Approximately global across workers via a shared atomic cell. Distributed rate limiting still needs Redis and stays open.

4. **TLS termination.** **Two upstream artefacts filed; local stand-in kept as reference.** (a) Reactor-native TLS serving in flare is a deferred upstream follow-up — ``STATE_TLS_HANDSHAKE`` is gated on parametric-trait method specialisation in a future Mojo nightly, and the server-side FFI has no ``SSL_read``/``SSL_write`` data path because the reactor integration was never wired. **Design doc filed at [ehsanmok/flare#7](https://github.com/ehsanmok/flare/issues/7)** describing the work for a ``serve_tls`` entry point (essentially: bind the existing ``flare_ssl_read``/``flare_ssl_write`` C exports on the server side and hand the wrapped stream to the unified reactor). (b) An in-binary stand-in via ``opengateway.mojo_bridge.tls_proxy`` is verified by ``tests/test_tls_proxy.py`` — three tests, real TLS round-trip, cleartext rejection, multi-chunk forwarding — but does not compose with the Mojo runtime's daemon-thread + ``GILReleased`` interaction: the proxy threads bind and the TLS handshake completes inside the binary, yet bytes do not flow from the TLS socket to the upstream socket when the reactor also calls into the Python bridge. **Diagnostic filed at [modular/modular#6865](https://github.com/modular/modular/issues/6865)** with a minimal reproducer (``feat/tls-thread-repro`` branch, ``opengateway/mojo/repro_tls_thread.mojo`` + ``opengateway/mojo_bridge/_repro_tls.py``). Production terminates TLS at the edge LB (ALB / NLB / nginx / Caddy), which is the standard gateway deployment shape anyway.

The validated pure-Python proxy (``opengateway/mojo_bridge/tls_proxy.py`` + ``tests/test_tls_proxy.py``) is kept in-tree as a reference implementation and a building block for when flare#7 lands; the wiring in ``main.mojo`` was reverted.
4. **TLS termination.** **Shipped via [PR #36](https://github.com/echohello-dev/opengateway/pull/36) on the flare#7 commit (`edf3f22`).** Bumped `pixi.toml` from `v0.9.0` to `edf3f22`; rebuilt `libflare_tls.so` against the new flare source (one-line `-Wl,-u,_flare_ssl_read_ex` linker flag because clang treats unreferenced extern C as dead-code; gcc includes them by default). `main.mojo` binds via `HttpServer.bind_tls(addr, cert, key, alpn=["http/1.1"])` + `serve_tls(stack)` when `TLS_CERT_FILE`/`TLS_KEY_FILE` are set. e2e: `openssl s_client` handshake completes, ALPN negotiates `http/1.1`, non-streaming + streaming `/v1/chat/completions` round-trip works over HTTPS, structured logger emits JSON per line. The flare reactor-side TLS reactor is real and working. Self-signed cert hostname gotcha documented in README ("Self-signed cert gotcha") — curl needs `--resolve localhost:port:127.0.0.1` when the cert is `CN=localhost`. Production terminates TLS at the edge LB; the in-binary path is for the edge-binary deployment story (Fly.io machines, bare metal, Lambda-ish). The stdlib-ssl proxy (`opengateway/mojo_bridge/tls_proxy.py` + `tests/test_tls_proxy.py` + `opengateway/mojo/repro_tls_thread.mojo`) stays in-tree as a vendored fallback for the v0.9.0 pin and as a reproduction of the early daemon-thread diagnostic; the wrapper usage in `main.mojo` was reverted. The flare issue I filed earlier (#8) was closed because the only "stall" I observed was curl's cert hostname verification failing silently under `-s`, not a reactor bug.

5. ~~**DB-backed virtual keys.**~~ **Shipped via the Python bridge** (no Mojo Postgres driver required): `opengateway/mojo_bridge/db.py` defines a `VirtualKeyStore` protocol with an asyncpg-backed implementation, and `authenticate_authorization` consults it after the root-key short-circuit with a 60 s in-process TTL cache. One connection per lookup — the bridge's one-shot `asyncio.run` model means a pool cannot outlive its event loop; at gateway latencies the connect cost is noise. `database_url` is now unset-by-default; when unset the store seam returns `None` and auth is root-key-only. Schema is created by `PostgresVirtualKeyStore.ensure_schema` (`virtual_keys` table with models / budget / tpm / rpm columns). **Spend recording shipped with it:** the bridge increments `budget_used` by `usage.total_tokens` after each unary completion, and the streaming pump parses the terminal usage chunk (the bridge injects `stream_options: {"include_usage": true}` so upstreams always send one). Budgets are token-denominated; a per-model dollar pricing table is a deliberate product decision left open.

6. **HTTP/3 enablement.** **Blocked on two prerequisites:** (a) server-side TLS (follow-up 4, which is itself blocked on flare's reactor TLS wiring), since QUIC is always encrypted; and (b) the rustls QUIC cdylib (`libflare_rustls_quic.so`), which flare's activation script builds via cargo — not present in the default toolchain image. When both land, `HttpServer.bind_with_http3` + `QuicServerConfig` is the wiring point, gated behind an env flag.
6. **HTTP/3 enablement.** **Partially landed, then rolled back.** The rustls QUIC cdylib now builds in the pixi env and the Docker image (#35), and the flare pin bump (#36) pulls in `HttpServer.bind_with_http3` + `QuicServerConfig`. **The reactor-side driver is not integrated at the pinned commit** — `serve()` does not call `tick_http3_once` / `pump_http3_handler_once`, which the source itself labels "Test-only entry point". Binding via `bind_with_http3` accepts the UDP socket but never processes datagrams, so a working h3 path requires a custom reactor loop (select on the UDP fd, feed into `recv_datagram`, dispatch via `pump_http3_handler_once`, drain egress via `send_to`) which is real upstream flare work, not an opengateway concern. As a safety measure, `main.mojo` raises a clear `Error` if `HTTP3_ENABLED=1` is set rather than binding a UDP port that silently accepts nothing. Once flare integrates h3 into `serve()` (or another PR adds the loop in opengateway and upstreams it), the guard can be replaced with the wiring.

7. **CI parallelism for the Mojo build.** **Shipped:** the mojo CI job caches `~/.cache/modular` (the `.mojo_cache` compile cache) keyed on the Mojo sources + `pixi.lock`, turning the ~6 min cold compile into an incremental rebuild. The pixi version is pinned to v0.70.2 to match flare v0.9.0's build backend.

Expand Down
25 changes: 25 additions & 0 deletions opengateway/mojo/main.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,32 @@ def serve(
- RequestId: inject / propagate ``X-Request-Id``; close to the
router so the id propagates through every header + log line.
- Router: the application dispatcher.

``HTTP3_ENABLED=1`` was planned to bind a UDP/QUIC listener in
addition to the TCP/TLS one, but the flare reactor-side h3 driver
(the one ``tick_http3_once`` / ``pump_http3_handler_once`` /
``emit_http3_response`` test helpers expose) is not yet integrated
into ``serve()`` at the pinned commit, so binding without driving
the socket would silently accept no packets. We surface a clear
error instead. See ADR-003 follow-up 6.
"""
# Guard for ADR-003 #6: HTTP3_ENABLED was wired to bind_with_http3
# in an earlier iteration, but at the pinned flare commit the
# reactor's main ``serve()`` does not drive the h3 UDP loop
# (``tick_http3_once`` / ``pump_http3_handler_once`` are test-only
# entry points). The HTTP/3 listener exists on ``bind`` but never
# processes datagrams. Refuse loudly rather than bind a port that
# silently accepts nothing -- a flapping h3 path is worse than a
# documented "pending upstream integration".
if getenv("HTTP3_ENABLED").byte_length() > 0:
raise Error(
"opengateway: HTTP3_ENABLED is set but flare's reactor-"
"side h3 driver is not yet integrated into serve() at the "
"pinned commit. ADR-003 follow-up 6. Disable HTTP3_ENABLED "
"(TLS over TCP still works) or upgrade opengateway's flare "
"pin once the reactor integration lands upstream."
)

var router = Router()
router.get("/health", health)
router.post("/v1/chat/completions", chat_completions)
Expand Down
Loading