feat(server): --parallel-ctx-threshold admission gate + UM prefetch net (hydra#747 baseline) - #110
Merged
ddvnguyen merged 5 commits intoSep 9, 2026
Conversation
Advise read-mostly + prefetch to the allocating device for unified-memory buffers so lazily-migrated pages settle on the owning GPU. Co-Authored-By: opencode <noreply@opencode.ai>
Defer queued requests when resident-ctx-sum + candidate prompt would reach the threshold; deferred tasks retry on slot release. Auto-slot path only (explicit id_slot bypasses). Vanilla slot accounting: max(prompt-cache tokens, full task prompt length) per processing slot. Env: LLAMA_ARG_PARALLEL_CTX_THRESHOLD. Co-Authored-By: opencode <noreply@opencode.ai>
ddvnguyen
added a commit
to ddvnguyen/hydra_vortex
that referenced
this pull request
Sep 8, 2026
… (hydra#747) Per #747 direction: the baseline delivery targets ddvnguyen/llama.cpp's 'baseline' branch (clean v0.4.0 5266f24da) — admission gate ported with vanilla slot accounting (no hydra fields) + UM prefetch net. RPC reconnect-loop fix ships separately via ddvnguyen/llama.cpp#109 (hydra-fork), not bundled here. Fork PR: ddvnguyen/llama.cpp#110 Co-Authored-By: opencode <noreply@opencode.ai>
Mirror the defer SRV_INF with an admit line (resident + candidate < threshold) so boundary tests can show the exact accounting either way. Co-Authored-By: opencode <noreply@opencode.ai>
… exceeding threshold alone The gate deferred any request with resident + candidate >= threshold, including resident == 0. A lone request bigger than the threshold on an otherwise idle pool then defers forever: nothing is resident, so no slot release ever retries it. The threshold guards combined oversubscription between concurrent requests, not a single request's own size (that is bounded by the per-slot cap). Defer now requires resident > 0; a lone request always admits. Admit log distinguishes the lone case so the printed comparison stays accurate. Co-Authored-By: opencode <noreply@opencode.ai>
… one task A release event re-posted a single deferred task (FIFO head, or one explicitly requesting the slot). If that task re-deferred - e.g. the hydra#747 threshold gate deferring a large candidate - the release was burned and every other waiter stayed queued even with a slot idle, while brand-new arrivals kept getting served from the main queue (priority inversion, silent client hangs). Re-post the entire deferred FIFO in order (explicit-slot matches first) to the front of the main queue so every waiter is re-evaluated against the freed capacity before newer arrivals; tasks that still cannot proceed re-defer to the back, preserving FIFO order. Deterministic repro + production case: hydra#747 task-12239 hang (docs/investigations/740-results-report.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hydra issue: ddvnguyen/hydra_vortex#747 · baseline delivery (clean v0.4.0
5266f24da+ 2 commits, no hydra-fork patches)Commits
--parallel-ctx-threshold N(envLLAMA_ARG_PARALLEL_CTX_THRESHOLD, server-only): a queued request is admitted into an auto-selected free slot only ifresident-ctx-sum + candidate-prompt < N; otherwise deferred (queue_tasks.defer) and retried on slot release viapop_deferred_task. Explicitid_slot >= 0bypasses. Vanilla accounting (no hydra fields): per processing slot,max(prompt-cache tokens, full task prompt length)— the max covers early prefill (cache small, task long) and deep generation.cudaMemAdvise(SetPreferredLocation)+cudaMemPrefetchAsyncfor managed allocations.The RPC reconnect-loop fix is NOT here — that bug lives in hydra-fork's ggml-org#470 reconnection machinery which doesn't exist on
baseline; it shipped separately via #109 (hydra-fork).Rig verification (2×RTX: 5060 Ti CUDA0 + 3060 RPC CUDA1, Qwen3.8-27B-UD-Q5_K_M, tensor-split 27,38, UM on)
-DGGML_CUDA_FA_ALL_QUANTS=ONis mandatory for K q8_0 / V q4_1|q5_1 flash-attn on v0.4.0 — without it the RPC peer GGML_ABORTs atfattn.cu:707on first FA op (documented in arm093's yml; cost me one boot).defer task 563 (resident 88857 + candidate 68727 >= threshold 100000); first task completed and released → deferred task auto-admitted and prefilled to completion (405 t/s). 0 spurious defers across all boots/loops.Do not merge before the hydra_vortex parent PR lands its submodule bump. No deploy workflow triggered.