feat(serving): throughput sentinel — placement-contract readback + decode-rate floor (#441) - #2311
Merged
Merged
Conversation
…code-rate floor (#441) Joel 2026-08-15: "we need very good tok/sec or it's a failure. We need warnings when shit hits the fan. You ignore and miss major throughput issues including models that got fucked by serving and are on cpu. You never catch it and we are waiting an eternity." Two seams, both riding EXISTING machinery — no new monitor task, no new tick, no second connection to the engine: 1. PLACEMENT CONTRACT READBACK (inference/placement_watch.rs, new file). The governor plans a placement; llama.cpp decides the real allocation at load and can land somewhere else (Metal init failure, VRAM exhaustion at map time). The stderr pump already reads every engine line for the log cap + wedge watch — an OffloadWatch on the same pump now records the engine's own `offloaded X/Y layers to GPU` banner into a shared cell, and serve() compares it against the PLANNED placement after readiness: a GPU-intent lane at 0/N layers probes serving.placement.cpu_fallback as a PLACEMENT VIOLATION. A lease nobody reads back is a suggestion; this is the readback half of the contract. Partial offload stays silent (MoE cold-expert -ot splits offload a subset by design); no banner observed is reported as its own fact, never conflated with healthy. The watcher only REPORTS — lifecycle stays with the serving daemon, same doctrine as the wedge flag. 2. DECODE-RATE FLOOR (ai/openai_adapter.rs). Every streamed call already parses the lane's own timings; measured decode t/s (predicted-only, undiluted by prefill so long prompts can't false-positive) is now compared against the catalog row's tokens_per_second. Below a quarter of expectation on a real sample (≥16 decoded tokens) probes serving.throughput.degraded naming the measured vs expected rates and the suspect causes. Coarse by design: an order-of-magnitude-collapse alarm, not a perf tracker — a 2×-optimistic catalog row stays silent. Rows without a registry expectation (cloud adapters) have no lease to breach and stay silent. Regression test pins the banner parse (both prefix spellings, lookalike rejection, no-banner ≠ reported-zero) — a silent parse miss here means a CPU-fallback lane reads healthy forever, the exact class this kills. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…30x the lease's heartbeats (Joel: heartbeat needs fix) A 30-minute claim lease was renewed on EVERY 60s presence tick — ~30x the events one lease needs, crossing the wire to every subscriber on every node (the churn that burned this week's session tokens through an unfiltered feed). Presence keeps its 60s pulse (roster liveness IS a fast question); renewal now runs at TTL/3, derived from the one lease constant. A citizen still gets three renewal opportunities per lease, and any roster-read or per-card failure falls back to retrying on the next 60s tick until clean — degraded mode is exactly the old cadence, never a wider gap. Context: the render half (#275, airc ac4936c) and the store half (#323, Durable-only DurableSink) are ALREADY FIXED in airc HEAD — but the box runs airc daemons started Aug 5, two days BEFORE that fix merged (#272 deploy gap). This commit is the source-reduction half that no repo had; the daemon refresh rides the next reboot window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…lls the guard-tripping consts The de-hardcode guard (context_budget) correctly flagged MIN_SAMPLE_TOKENS/ FLOOR_FRACTION as new bare-literal consts. Rather than dressing them up, this converges the two parallel implementations: warn_if_decode_collapsed is now the first production consumer of inference::throughput_expectation::classify_throughput (which was built on canary but had zero callers). Thresholds are call-site policy arguments — sample gate 16 tokens, collapse floor 0.25, no above-par ceiling — and the classification itself lives in ONE place per the compression law. Guard test + placement_watch + throughput_expectation tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Joel's directive: "we need very good tok/sec or it's a failure. We need warnings when shit hits the fan. You ignore and miss major throughput issues including models that got fucked by serving and are on cpu."
Two detectors, both riding existing machinery (no new monitor task, no new tick):
Placement contract readback (
inference/placement_watch.rs, new): the stderr pump that already feeds the log cap + wedge watch now also records the engine's ownoffloaded X/Y layers to GPUbanner. After readiness,serve()compares it against the planned placement — a GPU-intent lane at 0/N layers probesserving.placement.cpu_fallbackas a PLACEMENT VIOLATION (the model is on CPU; every consumer is waiting an eternity). Partial offload stays silent (MoE-otsplits are by design); "no banner" is reported as its own fact, never conflated with healthy. Watcher reports, daemon owns lifecycle — the wedge doctrine.Decode-rate floor (
ai/openai_adapter.rs): every streamed call already parses lane timings; measured decode t/s (predicted-only, prefill can't false-positive it) is compared against the catalog row'stokens_per_second. Below 25% of expectation on a ≥16-token sample →serving.throughput.degradednaming measured vs expected + suspect causes. Coarse by design: an order-of-magnitude-collapse alarm, not a perf tracker.Why this shape
This is the readback half of the governor's lease: plan → actuate → verify the engine's own account of what it allocated. A lease nobody reads back is a suggestion — the exact bypass class Joel called out.
Verification
cargo check -p continuum-core --features metal,accelerateclean🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo