feat: midstream retries - #1453
Draft
hachall wants to merge 49 commits into
Draft
Conversation
…ontinuation key purpose Two mechanical prerequisites for the resume middleware: Strict onwards completions schema gains the generation-control fields a token-id resume replay needs: ignore_eos, min_tokens, stop_token_ids, skip_special_tokens, include_stop_str_in_output. The strict router previously dropped these on re-serialization, silently breaking resume semantics on the prod (strict) path. Round-trip + absent-by-default tests included; token-id prompt forms already supported. ApiKeyPurpose gains Continuation: a hidden per-user key purpose (like batch/playground) the resume middleware will issue continuation legs on. The purpose is what model_traffic_rules keys on to steer resume traffic to a model's continuation deployment; it also authenticates the internal request through onwards and carries per-key rate limits as a resume throttle. Wired through is_inference_purpose + both onwards key-sync SQL mirrors, hidden-key creation, tariff purpose mapping, and the manual-key- creation guard (reserved for internal use, like batch/playground). No migration needed: purpose columns are VARCHAR throughout. .sqlx cache regenerated (workspace, PG18) for the touched sync queries.
Deploying control-layer with
|
| Latest commit: |
c261932
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://30ba499e.control-layer.pages.dev |
| Branch Preview URL: | https://preview-midstream-retries.control-layer.pages.dev |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for “mid-stream retries / continuation” by preserving additional completions request fields in strict mode and introducing a dedicated internal continuation API key purpose so continuation legs can be routed and authorized distinctly in onwards/dwctl.
Changes:
- Extend strict Completions request schema to retain continuation-related generation controls across parse → re-serialize, with tests verifying round-trip/omission behavior.
- Add
ApiKeyPurpose::Continuationand thread it through dwctl’s inference-purpose allowlists and onwards config key-sync SQL filters. - Update internal purpose-string mappings (tariffs + API key creation/validation) and refresh SQLx offline metadata for the modified queries.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| onwards/src/strict/schemas/completions.rs | Adds continuation-related request fields to strict schema and tests that they survive strict forwarding. |
| dwctl/src/sync/onwards_config/mod.rs | Extends onwards key-sync SQL purpose allowlist to include continuation. |
| dwctl/src/db/models/api_keys.rs | Introduces Continuation purpose and updates inference-purpose allowlist helper + tests. |
| dwctl/src/db/handlers/tariffs.rs | Maps Continuation purpose to "continuation" for tariff lookup/creation. |
| dwctl/src/db/handlers/api_keys.rs | Writes Continuation purpose to DB as "continuation" in multiple match arms. |
| dwctl/src/api/models/api_keys.rs | Documents continuation as an internal-only reserved purpose in API create schema docs. |
| dwctl/src/api/handlers/api_keys.rs | Rejects manual creation of continuation keys (reserved for internal use). |
| .sqlx/query-d5be88341f10e2b20ca435b2ec6ffa0831e96183fec04f4ba1ed600fbade86f0.json | Updates SQLx offline metadata for composite model key-sync query (adds continuation). |
| .sqlx/query-8d6eb646125cfa3196928a7272a5c008c2d9d393447ecea8976b6fe4942da4b6.json | Updates SQLx offline metadata for non-composite deployment key-sync query (adds continuation). |
| .sqlx/query-32a4cfc90e4b10722ac6df7a746962a4d1e16abb84b9dccb311bd2f47cdcbb7e.json | Removes unused SQLx offline metadata entry. |
Files not reviewed (1)
- .sqlx/query-32a4cfc90e4b10722ac6df7a746962a4d1e16abb84b9dccb311bd2f47cdcbb7e.json: Generated file
Comment on lines
122
to
129
| // Convert purpose enum to string for database | ||
| let purpose_str = match request.purpose { | ||
| ApiKeyPurpose::Platform => "platform", | ||
| ApiKeyPurpose::Realtime => "realtime", | ||
| ApiKeyPurpose::Batch => "batch", | ||
| ApiKeyPurpose::Playground => "playground", | ||
| ApiKeyPurpose::Continuation => "continuation", | ||
| }; |
Comment on lines
37
to
42
| /// Single source of truth for those Rust-side checks. The onwards key-sync | ||
| /// queries in `sync::onwards_config` mirror this list as a SQL | ||
| /// `ak.purpose IN (...)` filter (SQL cannot call this) - keep them in sync. | ||
| pub fn is_inference_purpose(purpose: &str) -> bool { | ||
| matches!(purpose, "realtime" | "batch" | "playground") | ||
| matches!(purpose, "realtime" | "batch" | "playground" | "continuation") | ||
| } |
…lowlists Copilot review follow-ups on the continuation plumbing: "continuation" was written to api_keys.purpose but never parsed back — both string→enum sites (ApiKeyDBResponse, get_user_and_purpose_by_secret) and the billing batcher's parse_api_key_purpose fell through to Realtime, so a continuation key would misreport purpose downstream (traffic rules, analytics, tariff selection). The /ai/v1/models key allowlist also still excluded continuation keys.
…oning Groundwork for the mid-stream resume middleware: ContinuationConfig (DWCTL_CONTINUATION__*): global kill switch (default off — stack byte-identical to today), per-origin eligibility gates (realtime leads), chain-resume attempt cap, per-attempt resume deadline, per-stream accumulator cap, dynamo scheduling priority for resume legs (positive = ahead of new realtime, intentionally: a resume finishes an already-accepted stream on a strict seam budget), and a per-model in-flight resume cap (deaths cluster in incidents; bound the stampede at the continuation provider). The continuation key is ONE GLOBAL hidden key (admin-owned), not per-user like batch/playground: resume legs must survive the requesting user's keys being pulled mid-stream (credit exhaustion — the user still bills normally via the merged frame on the original request), key cardinality must not scale with users (onwards sync cost), and resumes are provider faults so throttling is per-model in the middleware, not per-user on a key. Provisioned idempotently at startup so it has synced into the onwards key cache before the first resume. Migration 132 widens the api_keys purpose CHECK constraint (043), which predates the purpose and rejected the insert.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 17 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- .sqlx/query-32a4cfc90e4b10722ac6df7a746962a4d1e16abb84b9dccb311bd2f47cdcbb7e.json: Generated file
Suppressed comments (1)
dwctl/src/continuation/mod.rs:37
- The doc comment says the upsert is keyed on “owner + purpose”, but
ApiKeys::get_or_create_hidden_keyactually usesON CONFLICT (user_id, created_by, purpose) ...for hidden keys. This function is still idempotent because it passes the same ID foruser_idandcreated_by, but the comment is currently inaccurate and could mislead future callers.
/// Idempotent (`ON CONFLICT` upsert keyed on owner + purpose): the startup call
/// guarantees existence/sync, and the resume middleware calls it again to
/// obtain the secret without caring which call created the row.
…omy, accumulator Three dependency-free modules the resume middleware is built from: - rewrap: completions-chunk to chat-chunk reframing on the original stream envelope, and the merged terminal usage arithmetic (the client is billed for their prompt once and for every token they received, with a render-derived fallback when a provider undercounts the leg prompt). - detect: the mid-stream death taxonomy as one pure classify(). 499 is two populations (worker cancellation resumes, client disconnect never does) and a 4xx envelope inside a 200 stream is not resumable. - accumulate: the StreamAccumulator seam plus the v1 plain-content impl, which disarms rather than guesses on reasoning/tool deltas, n>1 and the buffer cap.
render.rs issues the one-hop resume prefix call (messages + generation stub + continuation_text, token ids back) and treats a missing continuation_tokens as a hard error: a silent zero would bill the customer's own partial generation as input on the merged usage frame. metrics.rs follows the prompt_cache conventions; the model label is only ever attached past the route gate, so it stays bounded by the admin-configured set.
The tee: leg-1 frames are forwarded byte-for-byte while the generation accumulates; on a resumable death the chain renders the prefix, dispatches a /v1/completions leg on the global continuation key, and reframes its chunks onto the client's original envelope, ending with one merged usage frame and [DONE]. The layer sits between the cache layer and error enrichment. The load-bearing consequence is the resume target: it is the router clone taken at exactly that point, so a resume leg re-enters below outlet and the cache and produces no second analytics row, billing record or cache classify. Guards: per-model in-flight cap, per-stream buffer cap, structured-output and origin gates, an attempt budget, and a deadline shared by render and TTFB. Every exit path — including a client disconnect, which drops the whole chain with the response body — records exactly one outcome and releases the model's slot.
…e upstream Sixteen end-to-end tests over a fake inner service that plays every death mode from the fault-injection matrix: cut between frames, transport reset, stall, error envelope in a 200, dynamo 499 cancellation, 4xx-in-SSE, no_done/no_usage. The load-bearing assertions: the client receives one seamless generation with exactly ONE merged usage frame (prompt counted once, completion across all legs); the resume leg carries token ids, the global key, the priority hint, include_usage and a decremented max_tokens; and the leg re-enters at the path the inner router actually expects, asserted through a real nest rather than assumed. Also covers what must NOT happen: no resume for structured output, unrouted models, disabled origins, oversized bodies, the kill switch, unreconstructable deltas, a saturated model, or a disconnected client.
A counting layer standing in for outlet and the cache asserts they see the customer's request exactly once however many legs served it — the invariant the resume_target capture point exists to provide. Also treats a terminal usage frame as a completion signal alongside finish_reason: a generation that has already reported its accounting is over, and resuming past it would emit the second usage frame that outlet and the cache layer must never see.
Ports `reconstruct_dsv4.py` from the fidelity harness, which the flash report validated byte-exactly against ground truth read from `/v1/completions` with token-id prompts — the one capture point that never runs the chat parser (24/24 resume cut points, 23/23 end-to-end on real fragmented deltas). Three of its rules came from measured failures rather than from the format, and each is pinned by a test: - a prefix ending exactly at `</｜DSML｜parameter>` or `</｜DSML｜invoke>` makes the model emit EOS immediately, so the structurally-implied newline is restored before the prefix is handed to a resume; - `</｜DSML｜tool_calls>` is never closed speculatively: a completed invoke does not mean the block ended, and closing it drops the sibling calls of a parallel block and breaks monotonicity; - the `\n\n` before the tool block is parser-dependent, so it is injected only when the accumulated content does not already carry it. The port deviates from Python in one place it had to: `serde_json::Value` sorts object keys, which reorders tool parameters into bytes the model never emitted, so non-scalar arguments round-trip through an order-preserving value instead. Fixtures are the harness's own captures, trimmed to the fields the tests need, covering all three observed provider shapes (fragmenting args, whole-object args, direct dynamo with ground truth). The goldens replay all 247 captured cut points against the Python reconstructor's output.
The tee hard-coded `PlainContent`, so no model could ever benefit from a family reconstructor. Reconstruction is now chosen by a capability lookup on the requested model — `continuation.model_reconstructors`, where the value `dsv4` selects the DeepSeek-V4 reconstructor. A model absent from the map, or naming a family we do not know, keeps today's behaviour exactly: a typo degrades resumability rather than corrupting a prefix. The cap and disarm semantics are unchanged and apply to both. The map is config because that is what lets us canary one model without a schema change; it belongs on the per-route DB row next to the `continuation` traffic rule once more than one family is live. `envelope`/`saw_finish_reason`/`disarmed` move onto the trait so the tee can hold the accumulator behind it.
A composite can now carry a member that exists only to serve /v1/completions (a validated token-id continuation target) alongside the chat failovers it already has. Each member gains a `serves` tag (both/chat/completions) and the pool filters on the request's class — derived from the path alone — before the load-balance strategy and the failover loop, so limits, guards and retries are untouched. The filter engages only when a pool actually has a completions member, so every existing pool selects byte-identically; `serves` defaults to `both` and untagged configs parse exactly as before.
`priority` orders the dynamo scheduler's queue, so a caller able to set it can jump ahead of every realtime request on the platform. The chat schema already forwarded it through its `#[serde(flatten)]` extras bag; the completions schema now models it (plus `stream_options`) and gains the same extras bag for parity. Both are stripped — typed field and extras — unless the request authenticates with a `continuation`-purpose key, whose legs finish a stream we already accepted. The purpose comes from the key labels dwctl's sync already stamps, the same map the routing rules match on.
A composite member can now be scoped: `both` (on-prem/dynamo), `chat` (today's third-party failovers) or `completions` (the validated continuation target for token-id resume legs). The sync emits it to onwards as the provider's `serves` tag, so a resume leg reaches the target that was actually validated for it instead of a chat failover that would answer plausibly and wrongly. The completions row also carries the per-route continuation config (strip_leading_bos, render_kwargs, continuation_validated_at) so "is this model resumable" and "how do we render for it" live in one place. Behaviour is unchanged everywhere today: onwards only engages the filter for a pool that has a completions member, and nothing has one until it is attached.
…mbers
A composite that needs a validated target for `/v1/completions` token-id
resume legs now expresses that as STRUCTURE rather than as a tag on each
member: it holds named pools — `{default, completions?}` — and a small
resolver picks one from the request class (the path) BEFORE any provider
is selected. Everything below that point (the load-balance strategy,
`select_iter`, the failover loop, rate and concurrency limits) runs on
the chosen pool and never learns that other pools exist.
This replaces the per-member `serves` tag from a1b58f1, whose filtering
lived inside the pool and had to be threaded through selection. The
filter mechanism is deleted outright: `load_balancer.rs` returns to its
pre-branch shape, ~400 lines lighter.
Why structure wins here:
- "never serves chat" becomes a fact about membership, not a predicate
evaluated during selection — the validated target simply is not a
member of the default pool, so no chat failover path can reach it;
- the same hosted model can be in BOTH pools with independent ordering
(dynamo at position 0 of each), which a single tag cannot express;
- the resolver is one seam for future request classes: a new class is
an arm plus a pool name, and selection is untouched.
Backwards compatible by construction. Every existing config parses into
a composite whose only pool is `default` — which is the same object the
single pool was — and re-serializes without a `pools` wrapper, so a
config written by this build still reads on one without pools. A model
with no completions pool serves completions from `default`, exactly as
before.
Access control stays a property of the ALIAS: a non-default pool
inherits the default's keys and routing rules unless it states its own,
so adding a pool for one request class can never become a way around the
alias's keys or deny rules.
dwctl side: migration 134 is amended in place (unpushed) from the v1
`role` column to `pool VARCHAR NOT NULL DEFAULT 'default'`, with
uniqueness per (composite, deployed_model, pool) so a model can join two
pools, and sort_order dense per pool rather than per composite. No
backfill is needed — every existing membership defaults to `default`.
The components API accepts and returns `pool` (PATCH/DELETE name the
membership with `?pool=`), and bulk component replacement now touches
the default pool only, so editing a model's member list cannot silently
delete its completions pool. The continuation route cache keys off "the
composite has a completions pool with >=1 enabled member".
`TargetPools::pool_count` is deliberately not `len`/`is_empty`: a
composite used to BE a pool, and those names would have gone on
compiling while answering a different question.
…ompletions strict Two defects in 6ad4122's priority scrub. **Batch scheduling was collateral damage.** The scrub let only the `continuation` key through, which would have stripped `priority` from fusillade's requests too — and fusillade derives a NEGATIVE priority from each request's deadline so batch work sorts behind realtime traffic and, within itself, by urgency. Stripping it flattens batch scheduling to a single tier and lets a batch due next week compete with one about to miss its window: a live behaviour change on the busiest path through the platform, invisible from the response. The policy is now tri-level — `batch` passthrough, `continuation` passthrough, everything else stripped — with a regression test asserting a batch key's negative priority survives re-serialization on both schemas. Batch priorities are negative by construction, so passthrough cannot jump the realtime queue. **Completions stopped being strict.** The commit gave the completions schema a `#[serde(flatten)]` extras bag "for parity" with chat, which silently turned a strict schema into a passthrough one: every unmodelled field a caller sent now reached the upstream. That is a widening of the proxy's contract, not a priority fix. The bag is removed; completions drops unknown fields again, as it did before this branch. The typed `priority` and `stream_options` stay — `priority` is modelled precisely because it must sometimes survive re-serialization, which an unmodelled field could not. Chat keeps its pre-existing extras bag, so it keeps needing the extras scrub: an unmodelled field there is forwarded verbatim, which is exactly how a caller would smuggle a `priority` past a schema that never modelled one.
Spec rule 3 covers the empty pool as well as the missing one, and the first implementation only covered missing: a `completions` pool whose members had all been removed or disabled resolved to itself and answered 503, for traffic the default pool had served perfectly well the day before. Operators empty a pool by disabling its last member; that must not take the alias's `/v1/completions` down with it. An empty pool is now indistinguishable from an absent one at the resolver, including for the `onwards.pool` span field — an empty pool is not a picked pool. This does not reopen the routing hazard it looks like it might. What keeps a resume leg off an unvalidated member is not this fallback but dwctl's route cache, which calls a model resumable only while its completions pool has at least one enabled member. An empty pool means no resume legs exist to be misrouted, while ordinary completions traffic keeps flowing.
Semantic resolution, not just textual: main's COR-522/COR-519 refactor made strict onwards validate-then-forward-original-bytes, with dwctl owning the single parse-and-shape. The tri-level priority policy moves accordingly: onwards no longer scrubs anything (its schemas are validation surfaces); dwctl's inference middleware strips external priority unconditionally — fusillade daemon requests bypass it (keeping batch's deadline priority) and continuation resume legs enter the stack below it (keeping theirs). Strict tests rewritten to pin the new verbatim-forwarding contract. TargetPools gains first_target/evaluate_routing_rules delegates (default pool) for main's composite-level adapter checks.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 63 out of 70 changed files in this pull request and generated no new comments.
Files not reviewed (7)
- .sqlx/query-0b4d867dd83c99c4d73663a3a6365ae8573b2313eee7008c05ecadd751800a4f.json: Generated file
- .sqlx/query-4360e11a7a72ee02c7855ccb93d9e0bc03545232e52b6af316b4bf0fd50c21f1.json: Generated file
- .sqlx/query-7fb9723160601670cee9c6c4dbd69e717e5eb265665a5f64ab0c521ffbc2f3e7.json: Generated file
- .sqlx/query-af33f6cb710d24b945d3cebaee7b5e89dcd796839a6e6feea9c4864d4399ff10.json: Generated file
- .sqlx/query-c17bf7fa9145f94fc81b90a696a47114d8ad3bba818f7c2faedcf970d40c55e7.json: Generated file
- .sqlx/query-c4f2969f3df936f57f0ddae1de7565d62d2568e3cee46ece3f3b252e791ec724.json: Generated file
- .sqlx/query-ff7ac8269ce3a7191aa51302b1b216e074df674a0edd531953136f4cdf806044.json: Generated file
Suppressed comments (3)
onwards/src/handlers.rs:552
- On routing-rule redirects,
poolis resolved from the redirect alias, butresolved_pool_nameis not updated. This makesonwards.pooltracing inaccurate for redirected traffic (it can show the source alias’s pool name). Updateresolved_pool_namewhen switching to the redirect target’s pools.
onwards/src/handlers.rs:480 resolved_pool_nameis computed before routing-rule redirects, but later used for span fieldonwards.pool. If a request is redirected to a different alias, the request may be served by a different pool name, and the span will report the original alias’s pool instead of the actual one.
This issue also appears on line 548 of the same file.
onwards/src/strict/mod.rs:1367
- The comment block describes a tri-level policy where onwards strips
priorityfor some key purposes, but the tests below (and the stated COR-522 contract) assert onwards forwards the original bytes and does not enforce privileged-field stripping. This mismatch is likely to confuse future changes; please update the comment to match the actual enforcement point (dwctl inference middleware).
The dynamo `priority` field is only meaningful to a non-default pool's primary member; third-party fallback members reject unknown fields outright (Fireworks: "Extra inputs are not permitted, field: 'priority'"), which turned every continuation resume-leg fallback into a 400 in the cl-1453 canary. Strip it per attempt at the one place member identity is known — the provider loop — for non-primary members of non-default pools only. Default-pool traffic is untouched on every attempt: batch/flex deadline priorities keep reaching dynamo exactly as today (regression-tested).
…m header Batch loopback bodies are not streaming when they pass the continuation layer: fusillade marks stream-intent with the x-fusillade-stream header and the outbound middleware forces `stream: true` into the body BELOW us. The streaming eligibility gate only read the body, so batch-origin requests — the largest mid-stream death population (gemma-4-31B prod baseline: ~2.8k mid-stream batch deaths/week vs 0-2 realtime) — could never arm. Honor the header as stream-intent alongside the body flag.
…etions pool
The route cache built RouteInfo from a single representative member row, so
render_kwargs was first-member-wins. In production a member with NULL
render_kwargs at sort 0 (the blackhole/dynamo primary) overrode the validated
target's {"thinking_mode": "chat"} at sort 1: the resume prefix was rendered in
thinking mode and a real model emitted a literal </think> into a client's
content stream.
render_kwargs describes how THE MODEL is served, not which member answers, so
any member stating it states it for the pool and a silent member must not shout
down one that is not. Its eventual home is a pool-level column; until then this
is an aggregate over the pool's members. strip_leading_bos keeps the
representative row's value because BOS-prepending genuinely is per-member, and
stays approximate until the strip moves into onwards' per-member forwarding.
A named (completions) pool inherited the default pool's routing rules whole. The two actions are not the same kind of decision: Deny is access control on the alias and must follow every request class, or adding a completions pool would quietly reopen a batch-denied model. Redirect names another model ALIAS, and following one from inside a named pool serves the class from a different model — for a resume leg, token-ids rendered against model X decoded by model Y, which generates garbage from the first token. Inheritance now filters to Deny; a named pool with rules of its own keeps them verbatim, redirects included. dwctl's sync copies the model's rules onto every pool it emits rather than relying on onwards' inheritance, so it applies the same filter to non-default pools.
- Stall timer arms only after leg 1's first event: pre-first-token silence is admission/prefill time, which severing turned into a fabricated empty 200 on every slow-TTFT stream. Resume legs stay timed from first read. - Global key is SYSTEM-owned (nil UUID): the onwards keyset queries exempt the system user from the group-access and balance gates, without which every resume leg 403'd on group-restricted or priced composites. - Mid-stream 4xx envelopes after partial output now resume (error_envelope_4xx): a genuine rejection lands as the first frame, where no prefix exists and the error surfaces unchanged; bad input self-corrects via attempts exhaustion. - Refused deaths drain leg 1's trailing frames so the close stays byte-identical to passthrough. - n>1 is double-gated: eligibility rejects it, and the accumulator disarms on choice.index != 0 (providers stream n>1 as alternating single-choice chunks that would interleave into one corrupt prefix). - DSV4 disarms on interleaved tool-call fragments instead of encoding a truncated slot as complete. - A failed ContinuationState build now emits background_error and the new dwctl_continuation_layer_wired gauge instead of silently serving without resume.
It was the one POST inference path outside should_intercept, and exactly where priority is now a typed strict-schema field forwarded verbatim — an external key could queue-jump the dynamo scheduler. Fusillade keeps its header early-return above the strip and resume legs enter below the middleware, so both legitimate senders are unaffected. Also documents the sso-stack ingress header-strip this bypass depends on.
Pool resolution now happens inside the targets-map guard: the previous shape cloned the whole TargetPools (every pool's owned KeySet) and then the resolved pool again — 2-3 full key-set deep copies per request, fleet-wide, where one pool clone suffices.
…-key keyset access A named pool's ordering is a validated failover list, not a load-balancing surface — under the composite's own weighted_random default, resume legs would split randomly between the free dynamo hop and the paid provider. Tests pin the strategy override and pin the system-owned continuation key into the keysets of a gated, priced composite and a group-restricted regular model.
The components list now returns memberships from every pool while the PATCH/DELETE endpoints default to the default-pool membership — so rendering completions rows here let a toggle disable chat serving, made completions-only rows 404, and let drag-reorder scramble chat failover. The tab filters to pool=default (ModelComponent gains the pool field); a pool-aware surface for completions pools comes later.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 67 out of 74 changed files in this pull request and generated 1 comment.
Files not reviewed (7)
- .sqlx/query-06816f2ab961c5dc85297a021c7321311bd03715cfd4489e1e3997c7ac167b48.json: Generated file
- .sqlx/query-0b4d867dd83c99c4d73663a3a6365ae8573b2313eee7008c05ecadd751800a4f.json: Generated file
- .sqlx/query-4360e11a7a72ee02c7855ccb93d9e0bc03545232e52b6af316b4bf0fd50c21f1.json: Generated file
- .sqlx/query-af33f6cb710d24b945d3cebaee7b5e89dcd796839a6e6feea9c4864d4399ff10.json: Generated file
- .sqlx/query-c17bf7fa9145f94fc81b90a696a47114d8ad3bba818f7c2faedcf970d40c55e7.json: Generated file
- .sqlx/query-c4f2969f3df936f57f0ddae1de7565d62d2568e3cee46ece3f3b252e791ec724.json: Generated file
- .sqlx/query-ff7ac8269ce3a7191aa51302b1b216e074df674a0edd531953136f4cdf806044.json: Generated file
Suppressed comments (4)
onwards/src/handlers.rs:551
- When routing rules redirect to another alias, this replaces
poolbut leavesresolved_pool_namedescribing the original alias. A continuation redirect from a default pool to a target'scompletionspool therefore keeps this asNone, so the latermember_idx > 0branch does not strippriorityfrom third-party fallbacks and those providers can reject the resume request. Update both the resolved pool and its name when following the redirect.
dwctl/src/continuation/detect.rs:85 - A stall is always treated as mid-generation even if a
finish_reasonor terminal usage frame was already received. Providers emit the finish chunk before usage/[DONE]; if that trailer hangs, this classification starts another generation leg and can append text after an already-complete answer. Carry the terminal-state signal into this event and returnLostTrailer, matching the EOF behavior.
dwctl/src/continuation/resume.rs:194 - This timeout only covers rendering plus receipt of response headers; it does not cover the first SSE event. The tee loop then starts a fresh full
resume_deadline_secstimeout for that first event, so a resume seam can approach twice the configured per-attempt budget. Preserve this absolute deadline (or remaining duration) inLegfor the first stream read, then use the full duration only for subsequent inter-frame stalls.
dashboard/src/components/features/models/manage/ProvidersTab.tsx:1125 - The unfiltered cache can contain the same
model.idin bothdefaultandcompletions, but updates are matched only by that ID. This changes the completions row optimistically, and the laterpreviousComponents.find(...)may select that row and incorrectly skip the required default-pool PATCH when its sort order happens to match. Restrict both lookups to(pool ?? "default") === "default".
// Create optimistically updated components with new sort_order. Built from
// the UNFILTERED list: the query cache holds every pool's memberships, and
// the updates only ever address default-pool rows.
const optimisticComponents = (allComponents ?? []).map((component) => {
| // extension fields and a legitimate `previous_response_id` are left intact. | ||
| scrub_request_id_fields(&mut request_value); | ||
|
|
||
| strip_scheduling_priority(&mut request_value); |
…e first frame Some engines emit a content-less role preamble the moment a request is admitted; the first-frame rule armed on it and a pre-first-token silence was still severed into an empty 200 at the deadline (observed live in preview-cl-1453 with the sim). Keying on accumulated bytes means the timer only runs when there is something to resume from — which is also exactly the pre-continuation passthrough for disarmed streams.
…sition The named-pool strip keyed on member index 0 being dynamo. In preview the dynamo member was disabled and Fireworks became index 0 — every resume leg was rejected with 400 "Extra inputs are not permitted", exactly the incident shape the strip exists to survive. inference_endpoints.accepts_scheduling_priority (migration 135, default false) says whether an endpoint's serving stack understands the field; it flows endpoint → sync → ProviderSpec → Target, and onwards strips priority from any named-pool attempt whose member lacks it, whatever its position. Default-pool traffic is untouched (batch/flex deadline priorities keep reaching dynamo). Unflagged dynamo = legs queue at priority 0, never a rejected leg. Exposed on the endpoint create/update/read API.
…ed DB The previous regeneration ran on a local DB where main's 129 had been applied after 133, so `users` SELECT * column order differed from CI's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.