[ethanzhoucool]

Summary

POST /v1/explain/stream is an async def handler, but it ran two blocking things directly on the event loop:

1. explain(...), the CPU-bound static pass, which can take seconds on a large paste.
2. for chunk in provider.stream(...). Both providers stream over blocking HTTP: the Anthropic SDK's sync messages.stream, and httpx.stream for Gemini with a 120 s timeout.

While an enrichment streamed, the event loop could not run anything else. Every other request waited until the model finished, including /healthz, page loads and plain /v1/explain/all-levels calls from other users. The Procfile runs a single uvicorn worker, so that is the whole service. Static-only streams were affected too, for the length of the analysis.

Changes

• api/main.py: the analysis runs via starlette.concurrency.run_in_threadpool, and the provider's sync generator is consumed via iterate_in_threadpool. These are the helpers Starlette provides for calling blocking code from async handlers. The SSE frames, their order and the error handling are unchanged.

Testing

• pytest on Python 3.13, 3.12 and 3.11: 148 passed, 3 skipped. ruff is clean, and mypy reports the same 26 errors as before.
• New test_streaming_enrichment_does_not_block_other_requests. It uses a fake provider (no network, no API key) whose stream() blocks for 0.3 s before each of 3 chunks, the way a real SDK read does. The test sends GET /healthz 0.1 s into the stream, through httpx.ASGITransport on the same event loop.
  • On main: /healthz waited 0.91s behind the stream, so the test fails.
  • On this branch: /healthz returns right away while the stream is still running. I ran it 5 times with no flakes.
• The existing streaming tests still pass: the static frame comes first, and a missing provider is reported without failing the request.

Notes

I couldn't exercise a real provider here (no keys, and none should be used). The change only alters where the generator's next() runs, not what it yields.

---

Generated by Claude Code