Stream LLM responses instead of blocking the servlet thread during generation

[devops-thiago]

Background

QueryService.generateWithTimeout (src/main/java/br/com/arquivolivre/myjavagenie/service/QueryService.java, ~line 260) runs the LLM call in CompletableFuture.supplyAsync (the common ForkJoinPool) and then blocks the servlet thread on future.get(timeoutSeconds, ...). The HTTP request stays open for up to query.timeout-seconds (300 s in the Docker profile) with no token streaming. The UI only receives the final answer via the REST response, plus coarse status updates over WebSocket.

Impact

• Tomcat worker threads are pinned for the entire generation duration, limiting concurrency under load.
• Perceived latency is high: users wait for the full answer with no partial tokens, even though the UI already streams status updates.
• future.cancel(true) on timeout only interrupts the pool thread; the underlying provider HTTP request can keep running.
• Using the common ForkJoinPool for blocking I/O can starve other async tasks in the JVM.

How to reproduce

1. Set query.timeout-seconds: 300 and model.max-tokens: 4096.
2. Ask a question that produces a long answer.
3. Observe /api/chat/query stays pending for the full generation and no token-level output reaches the UI until completion.

Where the fix should land

Two complementary options (pick per scope):

• Short term: keep the timeout logic but return Callable/DeferredResult from the controller via Spring MVC async so the servlet thread is released during generation.
• Long term: use LangChain4j StreamingChatModel (OpenAiStreamingChatModel) and push token deltas over the existing WebSocket (/ws/chat) or SSE, so the UI renders tokens as they arrive.

Files touched

• src/main/java/br/com/arquivolivre/myjavagenie/service/QueryService.java
• src/main/java/br/com/arquivolivre/myjavagenie/controller/ChatController.java and/or QueryController.java
• src/main/java/br/com/arquivolivre/myjavagenie/websocket/ChatWebSocketHandler.java
• chat-ui/src/context/ChatContext.tsx, chat-ui/src/services/websocket.ts (streaming UI)
• LanguageModelProvider / OpenAIModelProvider if a streaming method is added

Acceptance criteria

• Servlet threads are released during LLM generation (or responses are streamed).
• UI renders partial tokens (or, at minimum, no longer shows a spinner for the full duration).
• Timeout behavior preserved: still 504/ModelTimeoutException after query.timeout-seconds.
• Existing QueryServiceTest updated and green; mvn spotless:check passes.