feat(fusillade): discover per-model concurrency from 529 backpressure

[JoshC8C7]

feat(fusillade): discover per-model concurrency from 529 backpressure

Workstream 1 of "1M Output Tokens per Second". Little's law puts the target at
roughly 60,000 requests in flight per model. Today's ceiling is a static
batch_capacity, which the project brief describes as "too high at 1 replica and
far too low at 100".

Builds on #1460, keeping its escalation accounting fix and replacing the
controller. Both commits are here so that authorship stays intact; #1460 can
close as superseded.

What changed

With adaptive_concurrency on, the configured per-model limit becomes where a
model starts rather than what it runs at, and the controller owns the number
from there.

This is the main departure from #1460, which clamped the controller to the
configured value so it could only ratchet down. That fixes the "too high" half
and leaves the "too low" half untouched - and the second half is the one that
blocks saturating the fleet, because nothing downstream tells fusillade the fleet
grew.

A memory gate (new) bounds the process instead. Before computing per-model
capacity the daemon reads its own cgroup working set against its own limit and
claims nothing while usage is at or above memory_gate_high_fraction, resuming
under memory_gate_low_fraction. Claiming is the only way in-flight grows, so
suppressing it means the level can only fall.

It measures rather than predicts, which is the point. Per-request memory varies
by more than an order of magnitude between workloads - a short answer against a
long reasoning chain from a prompt of the same size - so a count of in-flight
requests cannot express the risk, and nor can anything that reserves bytes
up-front from the request body.

An earlier revision of this PR used a count, max_total_in_flight. That has been
removed: it was applied inside available_capacity, so it only ever affected the
claim decision, which is the same point and the same mechanics as the gate,
differing only in the unit. Keeping both would mean tuning a redundant number at
the wrong quantity.

Enabling adaptive_concurrency without a gate now fails closed. Since the
configured limit becomes a starting point, that combination leaves nothing
bounding in-flight work. The daemon logs a critical background error and runs at
configured per-model limits instead.

Turning adaptive_concurrency off returns every model to its configured value
exactly as before, so the flag is safe to flip in either direction.

Control law

Multiplicative both ways, with adjustments gated on a stamp rather than a clock.

• Down: a 529 multiplies the limit by adaptive_cut_factor (0.8).
• Up: adaptive_growth_factor (1.5), at most once per claim cycle, for a
  model that used every slot it was offered on the last claim. A model that used
  fewer had run out of work, so a bigger limit would sit unused until a burst
  dispatched the lot at once.

Nothing is sized to the fleet. #1460's += 1 is the opposite: recovering a
40,000 gap in steps of 1 - or even 16 - takes most of an hour, since raises are
capped at one per claim cycle however fast the requests go out.

Fast recovery is load-bearing, not a nicety. Dynamo rejects by priority, so batch
work is pushed to almost no concurrency whenever realtime is busy and has to
climb straight back when that lifts. The controller cannot tell "the model is
full" from "I am being outranked".

Why requests carry a stamp

In-flight work is never cancelled, so after a cut the requests sent under the old
limit keep failing for up to a request lifetime. Reacting to those cuts
repeatedly for one overload event, and a scale-down evicting thousands at once
drives the limit to its floor of 1.

Each request carries a counter bumped on every adjustment; a 529 with a stale
stamp is discarded. One overload event costs one cut, while sustained overload
keeps producing fresh reports and keeps cutting.

This replaces #1460's adaptive_concurrency_recovery_interval_ms. One wall-clock
interval governed both directions there, and no setting works: the tail of stale
failures lasts a request lifetime, so any interval short enough to react promptly
is short enough to cut dozens of times on the same event.

A raise has to be earned

As many requests must be sent under the current limit as the raise will add.

Without this the limit runs away. A raise fires every claim cycle while a cut can
only fire once per generation, so the limit nets growth x cut per cycle - above
1 for any sensible pair. Under simulation it reached u64 overflow within seconds
while every single request was being rejected.

Covered by a_model_being_rejected_does_not_climb.

Growth moved off the completion path

Raising the limit whenever a request succeeds would push a model with five
requests of work up to a limit of thousands, since all five keep succeeding. It
happens in the claim loop instead, where we can see whether the model actually
wanted the slots.

Background no longer feeds the controller

Background work runs on top of the foreground limit rather than inside it, and is
admitted only while foreground is quiet. Its rejections mean background
overflowed, not that the foreground ceiling is too high - cutting on them shrinks
SLA-bearing traffic because spare-capacity traffic bounced. #1460 shared the
feedback across all claim loops.

Retry metric labels

fusillade_requests_retried_total gains reason and status_code. A retried
failure never reaches fusillade_requests_completed_total, which records
terminal outcomes only, so a sustained stream of rejections is currently
invisible: production shows ~780k retries a week on one model with no way to tell
upstream 529s from onwards' own 429s. That distinction decides whether the
controller has a signal at all.

Rollout

adaptive_concurrency is a bool, off by default. Set memory_gate_high_fraction
first - the daemon refuses to enable the controller without it. Enable per
deployment, on one model, watching fusillade_adaptive_concurrency_limit against
dwctl_model_batch_capacity.

The gate is independently useful and can go on first, without the controller: it
bounds in-flight work against configured per-model limits just as well.

No shadow mode. It would be actively misleading: with claims still sized by the
configured limit the daemon never dispatches beyond it, so the controller never
tests its own higher limit, never earns a 529, and grows monotonically forever.

Not in this PR

• Claim rate. claim_batch_size (100) per claim_interval_ms (1000) is 100
  claims/s against a target of ~1,000/s. Until that moves, the controller
  measures the claim loop rather than the fleet - it reads "under-filled, no
  rejections" and correctly declines to move.
• Shedding in-flight work. The gate stops admission; it cannot cancel what is
  already running. If a workload's memory grows over each request's lifetime
  faster than requests complete, the gate will hold the door shut while the pod
  still exceeds its limit. A lower high mark buys headroom for that; nothing here
  reclaims memory from a request already in flight.
• 429 versus 529. onwards returns 429 from its own concurrency limiter and
  never generates 529. If that becomes the binding wall, the controller watches
  for a signal that never arrives. The retry labels above are how to find out.

Validating it

A standalone rig (kept outside the repo) stands in for an inference engine at
capacity: it admits N concurrent, rejects the rest with 529, holds admitted
requests for a spread of durations, and takes capacity changes at runtime so a
scale-up or collapse can be triggered by hand.

Production cannot be used for this - there is no way to force a scale-down, and
production currently shows no 529s at all.

Worth confirming with it before prod:

• off is genuinely off (in-flight pins at the configured value)
• the limit converges near a capacity it was never told
• a collapse settles near the new capacity rather than at 1
• a scale-up recovers in a handful of claim cycles
• the limit does not drift upward while being rejected
• fusillade_claim_duration_seconds and fusillade_state_writes_in_flight at a
  raised claim rate, since that is the untested database question

Tests

• Controller unit tests: starts at the configured value and can exceed it,
  recovery from a cut takes single-digit steps at a 60,000 limit, a rejected
  model never climbs, one cut per burst, sustained overload still walks down,
  stale stamps ignored, small limits still make downward progress without
  reaching zero, model independence, proportional scaling of the total in-flight
  cap.
• Integration: a 529 from an escalation target cuts the model it was claimed
  against; a clean capacity-bound model grows past its configured limit; a
  background 529 does not reduce foreground capacity.
• just lint rust, just test rust, cargo fmt --all --check.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_0188uHGf9akbJUadVMiaUZe7

Measuring the per-pod ceiling

fusillade_in_flight_at_gate records in-flight at the moment the gate engages.
That is the per-pod ceiling under real payload mix, measured rather than assumed,
and it is what a replica count should be divided out of. Alongside it:

metric	use
fusillade_in_flight_at_gate	the ceiling
fusillade_memory_gate_engagements_total	how often it bites
fusillade_memory_working_set_ratio	how close it runs
fusillade_memory_gate_engaged	whether it is holding now

If it never engages, the ceiling is unknown and something else is the limit.

🤖 Generated with Claude Code

https://claude.ai/code/session_015rS1wXEJLqSMyk1YuXWDpD

[Copilot]

Pull request overview

Adds an adaptive per-model concurrency controller to fusillade that learns sustainable concurrency from downstream HTTP 529 backpressure, allowing limits to grow beyond configured starting points while bounding total in-flight via a new process-wide cap.

Changes:

• Introduces an AIMD-style adaptive concurrency controller with generation “stamps” to avoid repeated cuts from stale 529s, plus proportional scaling when max_total_in_flight binds.
• Wires the controller into daemon claim/dispatch flow (including escalation capacity accounting) and extends retry metrics with reason and status_code labels.
• Adds integration coverage and documents new configuration knobs across fusillade and dwctl.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
fusillade/src/daemon/adaptive_concurrency.rs	New adaptive concurrency controller + total in-flight cap scaling logic and unit tests.
fusillade/src/daemon/mod.rs	Integrates controller into daemon capacity calculation, escalation accounting, and retry metrics.
fusillade/src/daemon/config.rs	Adds adaptive concurrency and total in-flight cap knobs with defaults + serde roundtrip tests.
fusillade/tests/integration.rs	Adds integration tests for escalation 529 cuts, growth past configured limit, and background isolation.
fusillade/README.md	Documents adaptive concurrency behavior and configuration options.
dwctl/src/config.rs	Exposes/mappings for new fusillade adaptive concurrency knobs in dwctl config.
config.yaml	Documents new background_services.batch_daemon adaptive concurrency settings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	57225f0
Status:	✅  Deploy successful!
Preview URL:	https://4a6938af.control-layer.pages.dev
Branch Preview URL:	https://preview-adaptive-concurrency.control-layer.pages.dev

View logs