server: tell a streaming client when its slot is parked and restored

[danielhanchen]

Stacked on #184. Only the last commit is new; the rest is the swap branch.

Summary

A streaming client is told when its slot is parked and when it is restored, as SSE comments (: preempted, : resumed), with a : preempt-keepalive every 2 s while parked. Comments are legal SSE and invisible to every existing client, so the body is unchanged by preemption. Non-streaming requests see nothing.

Without this a parked slot is a silent connection: the client cannot tell "waiting for cells" from "slow" or "dead", and any client-side read timeout or proxy idle timeout fires during a wait that is long by design. Unsloth Studio uses the comments to show a paused line on the chat that was parked and to stand its own preemption down when the server can do it.

Policy

Unchanged. The notice is emitted at the two park sites (the KV-full victim and the LLAMA_SERVER_PREEMPT_EVERY test knob) and at the restore site, through a server_task_result_preempt_notice pushed to the task's result queue for streaming tasks only. Notices that precede the first result (a slot parked while still processing its prompt) are sent in front of it. While parked the ping period is 2 s regardless of --sse-ping.

Results

Raw stream under LLAMA_SERVER_PREEMPT_EVERY=64: park and resume pairs every 64 tokens, body identical to the unparked run. Through Unsloth Studio on Qwen3.5-4B with two MTP drafts at -c 8192 and four slots: four concurrent chats 4 of 4 with the pause shown in the GUI on every chat that was parked, zero errors.

Exactness

The generated tokens with forced parks equal the unparked run token for token (harness test), and through Studio both 1200 token prompts are byte-identical across 18 parks each (seed 0, temperature 0).

Cost when it does not fire

None. No notice is created unless a slot is parked, and the keepalive is the existing --sse-ping path with one extra branch.

Tests

tools/server/tests/unit/test_preempt_notify.py: comments arrive in order and the body is unchanged on /completion, the same on /v1/chat/completions, a non-streaming request is untouched, and two overflowing streams both finish with the parked one saying so. 10 passed together with the existing preemption tests.

Limitations

No keepalive before the first result, so a park during prompt processing is covered by the client's first-token deadline rather than the 2 s ping.

Follow-up commit

The keepalive promised above had never fired. The server's result queue notifies one condition variable for every result of every task, and the reader's timed wait restarted on each wakeup, so on any busy server the timeout never elapsed; the same wait serves the ordinary --sse-ping. The wait now targets a fixed deadline. Paired on this branch against an isolated build of the previous head, a 13.9 s park carried zero keepalives before and four after, spaced at the 2 s period plus the reader's 1 s polling granularity, none outside the park.

[danielhanchen]

One more commit, 5a791e03c: the keepalive this PR promises never actually arrived, and the reason is in the result queue rather than in the notice path.

server_response::send() calls notify_all() on a single condition variable for every result of every task, and server_response::recv_with_timeout() waited with wait_for(), which restarts the wait on each wakeup. A reader waiting on a task that is producing nothing is therefore woken by every token every other task produces and its timeout never elapses, so on a server with any traffic at all the timeout is not a timeout. A parked slot is the worst possible case by construction, because a slot is only ever parked while the others are busy, which is exactly when the wakeups never stop. The same applies to the ordinary --sse-ping, which could only ever fire on an otherwise idle server, and to the should_stop polling in server_response_reader::next(), whose own comment says it happens every polling_interval_seconds and did not. The fix is to compute the deadline once and wait_until() it, so a spurious wakeup re-checks the queue and goes back to waiting for the same instant, which is what every caller already reads the argument as meaning.

Measured with two streaming completions on the 4B at -c 2560 with two slots, sized so they do not fit together and one is parked until the other finishes, every SSE line timestamped as it arrives. Before, on an isolated copy of 65f8f5464 with only this file reverted, the parked stream shows a park at 17.40 s and a resume at 31.27 s and nothing in between, a 13.87 s silence on a stream whose entire purpose is to say something every 2 s. After, on the same workload, the park lands at 17.39 s and the resume at 31.25 s with keepalives at 20.39, 23.39, 26.39 and 28.39 s, four across a 13.86 s park, spaced at the 2 s period plus the reader's 1 s polling granularity, and none outside the park. Both runs produced 1201 tokens on each stream.

Cost is none measurable: nothing extra happens per result, and on the branch this was found on the same four-chat control gives 139.6 tok/s before the change and 139.2 after. There is no harness test for it, because at the context sizes the test model runs at a park lasts about as long as the keepalive period itself, and a test straddling the period it is testing would be worse than none, so this is pinned by the probe above. tools/server/tests/unit/test_preempt_notify.py is 4 passed and test_preempt.py 6 passed on the new head.