UN-3996 [FIX] Stop WebSockets exhausting the Gunicorn thread pool - #2241
UN-3996 [FIX] Stop WebSockets exhausting the Gunicorn thread pool#2241kirtimanmishrazipstack wants to merge 1 commit into
Conversation
…thread pool The Socket.IO WSGI app runs with async_mode="threading", so every open WebSocket holds one gthread pool thread for that connection's lifetime rather than for a single request. With --threads 2, two browser tabs landing on the same worker left it with no threads for HTTP, and the worker kept accepting connections it could never read. Make workers, threads and worker-connections environment-configurable and raise the default thread ceiling to 512, matching the value already used by the cloud chart. The pool spawns threads lazily, so the higher ceiling costs nothing at idle. Drive --log-level from DEFAULT_LOG_LEVEL instead of hardcoding debug, and state --worker-class gthread explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
kirtimanmishrazipstack
changed the title
|
| Filename | Overview |
|---|---|
| backend/entrypoint.sh | Configures Gunicorn capacity and logging through optional environment variables while preserving the intended keep-alive budget. |
| backend/sample.env | Documents defaults that match the Gunicorn arguments configured by the entrypoint. |
Reviews (2): Last reviewed commit: "UN-4001 [FIX] Stop Socket.IO WebSockets ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
kirtimanmishrazipstack
changed the title
Unstract test resultsPer-group results
Critical paths
|
|
@greptile-apps please re-review. The one open finding ( |
kirtimanmishrazipstack
changed the title
| --threads 2 | ||
| --log-level debug | ||
| --workers "${GUNICORN_WORKERS:-2}" | ||
| --threads "${GUNICORN_THREADS:-512}" |
There was a problem hiding this comment.
@kirtimanmishrazipstack a default 512 threads might eatup your local machine memory and cpu. I believe that is why it was left 2 deliberately in the beginning. @muhammad-ali-e would be better person to answer.
What I would suggest is a bit more conservative default value something like 8 or 16.
Also if any other similar services need such a configurability please do include that as well.
| --threads "${GUNICORN_THREADS:-512}" | ||
| --worker-class gthread | ||
| --worker-connections "${GUNICORN_WORKER_CONNECTIONS:-1000}" | ||
| --log-level "${DEFAULT_LOG_LEVEL:-INFO}" |
There was a problem hiding this comment.
@kirtimanmishrazipstack I will suggest using a different env for this log level. There might be places when we need gunicorn at debug log level but not the application
2 participants
What
backend/entrypoint.sh: Gunicorn workers/threads/worker-connections are now env-driven, with the thread default raised from2to512.backend/entrypoint.sh:--log-levelreadsDEFAULT_LOG_LEVELinstead of being hardcoded todebug;--worker-class gthreadis stated explicitly.backend/sample.env: documents the three new optional variables.Why
async_mode="threading", so every open WebSocket holds one gthread pool thread for that connection's lifetime rather than for a single request.--threads 2, two logged-in browser tabs landing on the same worker left it with zero threads for HTTP, and it kept accepting connections it could never read — so requests stalled silently instead of failing over.How
frontend/src/index.jsx:64mounts oneSocketProviderper tab →backend/backend/wsgi.py:42wraps Django in the Socket.IO app (async_mode="threading",utils/log_events.py:29-31) →engineio/socket.py:222runs the socket loop inline on a pool thread sized by--threads(gunicorn/workers/gthread.py:98).512matches the value already running in the cloud chart and is free at idle, sinceThreadPoolExecutorspawns threads lazily on submit.worker-connectionsstays at1000: the excess over--threadsis gunicorn's keep-alive budget (gthread.py:73), so collapsing it onto the thread count would disable HTTP keep-alive.Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
No — the change is confined to Gunicorn CLI arguments, all three variables are optional, and no application code is touched. Log volume drops because
--log-level debugwas previously unconditional including in production mode; setDEFAULT_LOG_LEVEL=DEBUGto restore it.Database Migrations
Env Config
GUNICORN_WORKERS(optional, default2).GUNICORN_THREADS(optional, default512) — must exceed concurrent browser tabs per worker.GUNICORN_WORKER_CONNECTIONS(optional, default1000).DEFAULT_LOG_LEVEL(defaultINFO) now also drives Gunicorn's--log-level.Relevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing
bash -npasses; argument array expanded and inspected with no env set and with all four variables overridden.loglevelusesvalidate_stringandglogging.py:196lowercases before lookup.--threads 2, open two logged-in tabs and loopGET /api/v1/health; one worker stops responding whiless -tanp | grep :8000shows ESTABLISHED sockets with non-zeroRecv-Qand an empty LISTEN backlog.Screenshots
N/A — no user-facing surface.
Checklist
I have read and understood the Contribution Guidelines.