feat!: probe-based claim gate — replace model_filters with leaf-probe health - #1497
Draft
rushilbhat wants to merge 2 commits into
Draft
feat!: probe-based claim gate — replace model_filters with leaf-probe health#1497rushilbhat wants to merge 2 commits into
rushilbhat wants to merge 2 commits into
Conversation
…ate map Replace the claim gate's state SOURCE: the batchless, batch, and background claim queries no longer consult the model_filters event log. Each claim now receives a per-model gate map (ModelGateState: open / throttled), marshalled as two aligned TEXT[] arrays into a model_gates(model, state) CTE. The throttle MACHINERY is unchanged: the deadline-ramp escape hatch, the one-per-(user, window-class, model) leaky-bucket trickle, and cooldowns behave exactly as before, with "throttled" taking the role of not-live. - fusillade-core: new serializable enum ModelGateState with as_str() -> "open"/"throttled". Open claims at full capacity and is the default for models absent from the map -- absence must never strangle. - Storage trait: claim_batchless_requests / claim_requests / claim_batch_requests and the background-claim family now take gate_states: &HashMap<String, ModelGateState>. - DaemonConfig: new model_gate_states: Arc<DashMap<String, ModelGateState>> mirroring model_concurrency_limits (same serde treatment, default empty = everything open); the claim loops snapshot it each cycle next to available_capacity. dwctl populates the map in the next commit -- until then it is empty and the gate is a no-op. - Rename batch_claim_require_live -> batch_claim_require_open (serde alias keeps the old config key working) across dwctl config, fusillade DaemonConfig, and PostgresStorageConfig. - Background-claim semantic change: previously strictly live-only (INNER JOIN requiring the latest model_filters event = 'live'); now a model absent from the gate map is treated as open, per the absence-is-open invariant, so background claiming no longer requires an explicit signal. - The model_filters table and its append/list/purge/ current_filter_states APIs remain in place (deprecated as a claim source); scouter stops writing the log in a separate change.
…obe health
Populate the per-model claim-gate map the previous commit injected into
the fusillade daemon.
- Probe auto-creation: reconcile_auto_probes inserts an 'auto-<alias>'
probe (60s interval, active, POST) for every non-deleted,
non-composite deployed model with a hosting endpoint, via a single
INSERT ... SELECT ... ON CONFLICT DO NOTHING -- idempotent and
multi-pod safe. It never touches existing (manual) probes and never
deletes probes (model deletion cascades instead). Runs on onwards-sync
startup and on every full_reload, behind the new `probes.auto_create`
config flag (default: true).
- Health rule per leaf entry: healthy unless the most recent
probe_results row within the last 3 * interval_seconds has
success = false. No probe, an inactive probe, or no result inside the
window all count as healthy -- absence of monitoring must never
strangle a queue, and a result older than the window is stale, not
damning.
- Gate per catalog alias: a non-composite entry follows its own leaf
health (healthy -> Open, else Throttled); a composite resolves its
PRIMARY enabled component (lowest sort_order, tie weight DESC /
created_at ASC) recursively to a leaf, depth-bounded at 8, and
follows that leaf. Any resolution dead-end is Open.
- Sparse-map invariant: the updater only inserts Throttled entries and
evicts everything else, so the shared DashMap only ever names
throttled models and absent = Open stays the daemon-side default.
State transitions (throttled / reopened) log at info; steady states
do not.
- Refresh cadence: a claim-gate refresher task recomputes the map from
the DB every 15s, spawned under the same batch_daemon.enabled
(Always | Leader) condition as the fusillade daemon itself; failures
emit dwctl_background_errors_total{component="claim_gate"}.
Deploying control-layer with
|
| Latest commit: |
0847bb9
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://bd89ca37.control-layer.pages.dev |
| Branch Preview URL: | https://feat-probe-based-claim-gate.control-layer.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
COR-611 part A, in two commits:
feat(fusillade)!— the claim gate's state source becomes an injected per-model map (ModelGateState::{Open, Throttled}, sharedArc<DashMap>onDaemonConfig, mirroringmodel_concurrency_limits). All claim paths (batchless Source A/B, batch, background) consume amodel_gatesCTE from aligned arrays instead of joiningmodel_filters. The throttle machinery — leaky-bucket trickle, deadline ramp, cooldowns — is untouched; absent from the map = Open, so nothing can be strangled by missing state.batch_claim_require_live→batch_claim_require_open(serde alias kept). Themodel_filterstable and APIs remain (deprecated; scouter#158 stops writing them).feat(dwctl)— populates the map from real health:auto-<alias>probe via one idempotent insert, run at sync startup + every catalog full_reload. Manual probes are never clobbered; nothing is ever deleted. Flag:probes.auto_create(default true). The existing (currently dormant) probe scheduler executes them through the local/aiproxy — the exact path dispatched work takes.Why
The 2026-08-14 incident (COR-610): fusillade obeyed a scouter-written liveness note that could never be updated after a model rename, silently throttling a customer's queue to ~1 req/min for four hours with zero errors. Full design rationale in COR-611 — the gate now asks a question the control layer can answer itself, per-model, through the real serving path, with no cross-system name coupling and no stored state that a rename can strand. Bonus: a dead external provider now holds its queue instead of blind-firing into the outage.
Sequencing
Land and deploy before scouter#158 (which stops writing
model_filters) — this PR is what replaces the protection that removal takes away. Independent of #1496 (demand fold).Tests
Full
just test rustgreen (dwctl lib 2,047; fusillade-arsenal 214; all suites exit 0). COR-432 gate tests adapted to the injected map (incl. two repurposed: gate ignores model_filters events; absent-from-map claims full capacity). 14 new dwctl tests: auto-creation idempotency/skips/no-clobber, health-window semantics (recent failure throttles → reopens; stale failure is healthy; recent failure beats older success), composite resolution (primary, disabled-primary skip, recursion bound, dead-end open), sparse-map invariant. Known local-onlysqlx prepare --checkmismatch on pre-existingquery-aabf6640…(PG17 drift, present on main).