Continuous batching: a GPU lane can reach a state where the servable stays up and every request hangs indefinitely until restart

[lusoris]

Relationship to #4428

@mzegla asked for this on 2026-08-24 in #4428:

@lusoris as for the orphaned generations accumulation, could you file another issue to track it separately?

Filing it now. It is late; that is on me.

This is not the #4428 failure. #4428 is a process-fatal assertion in the
GenAI block manager (OPENVINO_ASSERT(m_block_table.count(seq_id) > 0, "Invalid sequence group.") in free_group_partially()), reached under static-KV-cache
exhaustion and preemption; the process exits and every batched generation dies
with it, and the candidate fix is openvinotoolkit/openvino.genai#4332.

What is described here has no process exit. The serving process stayed up
and had not restarted itself — a manual restart is what cured it — and the proxy
in front of it went on reporting the model ready. What stopped was generation.
In #4428 the process dies and clients get connection failures; here the process
lives and clients get nothing at all.

I want to be careful about one thing: I cannot tell you that nothing was
logged, because we retained no logs from the window (see below). What I can say
is that the process did not exit and the batch was not lost to a crash.

If the two turn out to share a root cause, that would be a finding — but the
observables do not overlap, and #4332 cannot be validated against this one, so
keeping them separate seemed right. Same fleet, same cards, same image.

Confidence, stated up front

I would rather under-claim this than have it chase a phantom.

• This is one retained incident (2026-08-22), plus one adjacent incident
  on the other card the previous day, reconstructed from our operations ledger.
• We have no OVMS-side logs from either window. They were not captured and
  are gone. Everything below that is not marked as measured is a report of what
  an operator recorded at the time.
• We have not reproduced this on demand, and we do not have a reproducer to
  hand you. I am not claiming a defect in OVMS — I am reporting a state we
  reached and cannot explain, and asking whether it is expected.
• A plausible explanation is entirely client-side (our reverse proxy, described
  below). I could not rule that out from what we retained.

Environment

Identical stack to #4428 except for the OVMS build, which is newer.

• Image: docker.io/openvino/model_server:weekly@sha256:7eb60804f86d7f47fd278a0cc1958fde5f8493534062ae69bef27dc5ba439cf2

  • Verified today: that digest still resolves on Docker Hub, its image config
    created is 2026-08-17T00:11:05+02:00, base ubuntu:24.04, labels
    supported-devices=CPU=1 GPU=1 NPU=1.
  • The image carries no version label, so I cannot confirm its OVMS version
    string first-hand. Our own build note records it as a 2026.4 development
    build (we pin weekly by digest because no 2026.4 release tag existed).
  • This base is wrapped in a local layer that adds tini and a reverse proxy
    binary. Nothing in /ovms is modified.

• GPU: Intel Arc B580 12 GB, xe kernel driver, PCIe passthrough into a VM.
  Host kernel version was not retained. In-image compute runtime, per our
  recorded inspection of this digest (not re-verified today):
  intel-opencl-icd / libze-intel-gpu1 26.18.38308.1-0, intel-igc 2.34.4,
  libigdgmm12 22.10.0.

• Model: empero-ai/Qwythos-9B-v2 (Apache-2.0), INT4 OpenVINO IR, exported
  by us at upstream revision 2178f73a9b5ea28ccd8f6096ef6bac5cd59c9d8b.
  Its model card at that revision describes it as a fine-tune of a Qwen3.5-9B
  hybrid-attention base — i.e. the same broad class as the hybrid side of
  the hybrid/SDPA split in SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428, rather than an SDPA model. Our own record
  files that as an upstream claim, not a verified measurement: we have not
  confirmed the attention type from the IR ourselves, so do not weigh it
  heavily. IR is 23 files,
  6,129,309,145 B total; openvino_language_model.bin is 4,614,719,734 B,
  sha256 ffd40ec4…. Measured solo device footprint 10,533,100,000 B.

• Serving arguments (verbatim from our deployment record for that build):

  --task=text_generation
  --cache_dir /models/.ovcache/qwythos-9b-v2
  --target_device=GPU
  --plugin_config='{"ENABLE_CPU_PINNING":false}'
  --tool_parser qwen3coder
  --reasoning_parser qwen3
  --kv_cache_precision u8
  --cache_size 4
  --enable_prefix_caching true
  --cache_interval_multiplier 64
  --max_num_seqs 16
  

• Traffic: OpenAI-compatible POST /v3/chat/completions with
  "stream": true, long agentic generations (multi-minute is normal), reaching
  OVMS through a reverse proxy. Effective client budgets on this lane were
  12288 input / 4096 output tokens (from our deployment record). I did not
  retain the served context window for this model and am not quoting the 40960
  figure from SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428, which belongs to the 27B.

The proxy in front matters, so here it is explicitly

OVMS is fronted by llama-swap v247
with a local patch, and that patch is directly relevant:

• the upstream request is issued with context.WithoutCancel(...), so a client
  that vanishes does not cancel the request context; and
• if the client stops reading mid-stream, the response body is handed to a
  background drain (io.Copy(io.Discard, ...)) instead of being closed,
  bounded at 5 minutes or 64 MiB, after which the body is closed for real.

That patch exists because of #4428 (it was our first, later-retracted, theory
about that crash). Its effect here is that from OVMS's point of view the HTTP
client is still connected and still reading for up to five minutes after the
real client is gone. Also configured: a 1800 s response-header timeout on the
proxy's upstream transport, and a 30 s grace before the proxy SIGKILLs a child
it is unloading.

What happened (2026-08-22)

Verbatim from our operations ledger for that day
(an internal file, not public; quoted rather than paraphrased because it is the
contemporaneous record and I do not want to smooth it in retelling):

B580 wedge (second leg of the day's fleet incident): the 10:58
paperclip pod roll orphaned in-flight q9 streams; OVMS accumulated
orphaned generations until qwythos-9b-v2 sat state=ready with every
request hanging forever; the idle reaper's 30-min SIGKILL cycles added
MORE orphans each round (self-reinforcing); q9-class agents stall-looped
(reap -> re-admit -> silent -> reap). The autorecover generation probe
walked past it: a timeout was unconditionally "inconclusive". Manual pod
bounce cured the lane instantly.

Unpacked, with our terminology replaced by yours:

1. 10:58 — a deployment roll restarted the agent runtime that owns the HTTP
   clients. Many long streaming generations lost their consumer at the same
   instant. "Orphaned generation" is our term for one of these: a generation
   whose originating client process no longer exists. It is not an OVMS term
   and does not assert anything about OVMS-side state.
2. Over the following ~90 minutes the lane degraded until every new request
   to that model hung. Not slow — hung: no response at all until the client
   itself gave up.
3. The servable did not report a problem. Our proxy's manager kept the model at
   state=ready and its own health endpoint green. (That state=ready is the
   proxy's state, not OVMS's. We did not retain a probe of OVMS's own
   /v2/health/ready for this window, so I cannot tell you what OVMS was
   reporting.)
4. ~12:25 — a manual restart of the serving pod cleared it immediately. To
   be accurate about this: a restart is the only thing we found that worked, but
   we did not systematically try lighter remedies first, so I cannot tell you
   that nothing lighter would have worked.

Our own monitoring is part of why this ran for ~90 minutes rather than minutes:
our health probe treated a generation timeout as inconclusive rather than as a
failure, so it walked straight past a model that was completing nothing at all.
That is our bug and we have since fixed it on our side — I mention it only
because it is the same blind spot ask (3) below is about, and because it is the
reason nobody was looking at OVMS while it was in this state.

The 10:58 → ~12:25 span is ~87 minutes, which matches the "~90 min" recorded
independently in the code comment we added after the incident. (The ledger does
not stamp a timezone on those two clock times; I read them as UTC because that
span is self-consistent and because the ledger entry itself is stamped
2026-08-22T12:35:00Z, ten minutes after the recorded bounce.)

Measured, and the one hard number we have: a one-token generation
request against the wedged, ready model timed out at 45 s. The same probe
against the same model after the restart completed in 5.8 s average. So this
is not "slow under load" — a minimal generation that normally takes seconds
returned nothing at all.

The self-reinforcing part

This is the bit I most want to flag, because it turns a transient into a
sustained outage.

Our orchestrator reaps agent runs that have produced no output for a while and
kills the client process. Each reap therefore abandons another in-flight
generation, and the reaped work is then re-admitted, which issues another
request into a lane that is already not generating — which produces no output —
which gets reaped. The ledger records the loop as reap -> re-admit -> silent -> reap, on a cycle it describes as 30 minutes, each round adding more abandoned
generations.

So whatever the initial cause, an automatic recovery mechanism upstream of the
server made it monotonically worse. Any deployment with a client-side timeout
and retry — which is most of them — has this shape.

Caveat on "30-min": that interval is the ledger's characterisation. Going
back through the retained configuration I could not identify a reaper with a
30-minute period. The closest configured 1800-second values are the proxy's
upstream response-header timeout and our orchestrator's run deadlines. I am
reporting the recorded phrase and flagging that I could not confirm the exact
mechanism or interval.

Adjacent occurrence, other card, previous day

Same class, different device and model, recorded in our ledger for
2026-08-21T19:14Z:

The roll orphaned 4 in-flight q27 generations: OVMS kept grinding dead
streams, jamming both dual-sequence slots -> ALL q27 calls hung (even
direct); B60 llama-swap pod bounce cleared it (same class as the morning
B580 wedge).

• Device: Intel Arc Pro B60 24 GB, xe, same image, same proxy.
• Model: a 27B INT4 engine (our q27 lane), configured for two concurrent
  sequences. I did not re-verify which 27B revision was resident that day, so I
  am not claiming its attention type here.
• Same trigger (a deployment roll abandoning in-flight streams), same
  observable (all requests hang), same and only remedy (restart).
• "even direct" means the hang reproduced when bypassing our load balancer and
  calling the OVMS endpoint straight, so it is not a routing-layer artifact.

Two occurrences on two different cards with two different models in two days,
both after a mass client restart, is why I think the shape is worth reporting
even though the evidence is thin.

What we retained, and what we did not

Being explicit, because most of the interesting evidence is in the second list.

Retained (first-hand, re-read for this report):

• The exact image digest, and its registry-side metadata re-verified today.
• The exact serving arguments, model artifact digests, and measured device
  footprint, from a versioned deployment record.
• The contemporaneous ledger entries quoted above, and the code comments added
  in response to the incident.
• The 45 s / 5.8 s probe measurement.
• The proxy patch source, so the client-side behaviour is exactly known.

NOT retained — please read this as a limit on everything above:

• Any OVMS log line from either window. No --log_level DEBUG, no INFO, no
  stdout. The cache-usage lines that made SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428 diagnosable have no counterpart
  here.
• Any OVMS metrics scrape (ovms_current_requests, ovms_current_graphs) from
  the wedged period, so we cannot say how many requests OVMS believed were in
  flight.
• Any OVMS-side health probe result during the wedge.
• Kernel logs for the window, so we cannot say whether an xe engine reset was
  involved. Given Model Server hangs when inference on Intel GPU #2336, that alternative deserves ruling out and we cannot rule
  it out.
• Any thread dump, gdb attach, or core from the hung process.
• Per-request bodies or IDs, so no generation can be traced end to end.

We also never opened an internal bug for this class, which is part of why the
evidence was not preserved. Both of those are process failures on our side, not
inference from absence.

Downstream corroboration — and its limits

There is public downstream history, but I want to be precise about what it does
and does not support, because I got the diagnosis wrong there once already.

• mostlygeek/llama-swap#980
  — opened by me 2026-08-04, closed as completed 2026-08-18, closed by me.
  It establishes, and this part still stands, that Go's
  httputil.ReverseProxy propagates a client disconnect into the upstream
  request, so a vanished client aborts the OVMS generation unless the proxy
  prevents it.
  It does not corroborate a crash cause. I closed it retracting my own
  diagnosis: I had blamed client aborts for the SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428 crash, patched the proxy so
  aborts could not propagate, and the crash continued unchanged. Anyone reading
  CVS-67137 Custom ReLU cpu extension #980 as evidence that disconnects crash OVMS is reading my retracted claim.
• mostlygeek/llama-swap#981
  — the surviveClientAbort implementation. Opened and closed unmerged the
  same day (2026-08-04, ~51 minutes). Maintainer @mostlygeek:

  This is a bug in OpenVino and their repo is the best place to receive a PR.
  and, on CVS-67137 Custom ReLU cpu extension #980:
  This sounds like a bug in OpenVino. OpenVino should be handling client
  disconnects gracefully and not discard a whole batch when it happens.
  He was right to decline it. We carry the change as a local patch anyway, which
  is why it is part of the environment described above.

Net: the downstream issues establish the plumbing (a disconnect reaches OVMS
as a cancellation unless something stops it), and they establish that we
deliberately stopped it. They do not independently corroborate the hang.

What we checked in the current source

Read first-hand at openvinotoolkit/model_server main
3607c3572c58de28ba161d7325b6723d0ff85756, to understand the mechanism rather
than to claim a defect:

• src/llm/language_model/continuous_batching/servable.cpp:120 registers the
  cancellation:

  cbExecutionContext->payload.client->registerDisconnectionCallback(
      [genHandle = cbExecutionContext->generationHandle]() { genHandle->stop(); });

• src/drogon_http_async_writer_impl.cpp shows that signal is driven by the
  TCP connection close callback (connPtr->setCloseCallback(...)), with
  IsDisconnected() returning isDisconnected || !requestPtr->connected() —
  i.e. a failed write or a closed socket.
• grep -rni timeout src/llm/ at that commit returns no matches, so as far
  as I can see there is no per-generation deadline in that layer.

If I read that correctly, a generation is bounded by its own stopping criteria
and by the client's socket, and by nothing else. Our proxy holds that socket
open and keeps reading for up to five minutes after the human-visible client is
gone — so for that window OVMS is behaving exactly as designed, and the
generations are not "orphaned" from its point of view at all. That is a
plausible complete explanation of the first five minutes and I want it on the
record. What it does not explain is the state that persisted for roughly
eighty more minutes, after every drain had hit its five-minute bound and every
socket had been closed for real.

(Unrelated but adjacent, noted so nobody wastes time on it: docs/mediapipe.md
documents that a mid-stream client disconnect can strand a Python-node generator
on a pooled graph. That is Python generative nodes with graph pool enabled,
which is not our configuration.)

What we are asking for

Not a fix — we cannot justify asking for one on this evidence. Three questions,
in the order that would help us most:

1. Is an unbounded generation lifetime the intended design? As far as I can
   tell from the source above, a continuous-batching generation ends when its
   own stopping criteria are met or when the client's socket closes, and there
   is no third bound. Is that right? And if it is, is there any supported way to
   cap generation wall time server-side? We would use one immediately — a
   proxy that holds a socket open on behalf of a client that no longer exists
   is, on that reading, indistinguishable from a legitimate slow reader.
2. Can a stop()-ed sequence keep occupying a slot? When
   GenerationHandle::stop() fires from the disconnect callback while the
   executor is mid-step — particularly with the cache at 100% and preemption
   active, the state SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428 lives in — is the sequence guaranteed to leave the
   scheduler? If a stop can be lost or deferred indefinitely, that would match
   what we saw, and it is adjacent to what Fix stale sequence handling during partial preemption openvino.genai#4332
   touches.
3. Can a stuck lane be detected from outside? Today a wedged lane is
   externally indistinguishable from a healthy busy one: health is green and
   nothing logs. ovms_current_requests and ovms_current_graphs count
   requests and connections, not scheduler occupancy. A metric exposing the
   continuous-batching scheduler's live sequence count and free-block count
   would let operators detect this in seconds instead of ~90 minutes. We would
   scrape it immediately.

If you want to try to reproduce it

Untested — this is a suggestion, not a recipe we have run:

1. Continuous batching on GPU, small static cache (--cache_size 4,
   --max_num_seqs 16), a hybrid-attention model, so the cache saturates with
   very few sequences.
2. Start several long streaming generations, enough to reach 100% cache.
3. Abandon their clients without closing the sockets — e.g. keep reading into
   /dev/null from a separate process — then close them some minutes later.
4. Repeat every few minutes while continuing to send new requests.
5. Watch for the point at which a one-token request to the still-ready model
   stops returning.

If that lands nowhere, I would treat this report as unconfirmed rather than
spend your time on it.

What we can offer

The honest answer is: less than I would like, and I do not want to write another
cheque here that I cannot cash. Our two Intel lanes carry production traffic, and
I already owe #4428 a validation run of the wovchena-genai image that I have
not been able to schedule. So, concretely:

• If this recurs, we will capture OVMS stdout at DEBUG, a metrics scrape, and a
  thread dump before restarting, and post them here. That instrumentation is
  cheap and we should have had it already.
• If you point at a specific metric or log line you want, we will wire it into
  the standing capture rather than waiting for the next incident.
• If this is expected behaviour — the socket is the contract, hold it open and
  you own the consequences — say so and I will close this and fix it on our
  side.