feat: adapt model concurrency to downstream 529 responses - #1460
Open
pjb157 wants to merge 1 commit into
Open
Conversation
Deploying control-layer with
|
| Latest commit: |
384e383
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f72be175.control-layer.pages.dev |
| Branch Preview URL: | https://peter-adaptive-529-throttle.control-layer.pages.dev |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an adaptive per-daemon, per-model AIMD concurrency controller to fusillade so claim capacity reacts to downstream overload (HTTP 529) by halving effective limits and then recovering additively, with shared feedback across claim loops (foreground/background and batch) and operator-facing configuration/docs.
Changes:
- Introduces an
AdaptiveConcurrencyControllerand wires it into claim-capacity calculations plus per-request success/overload feedback. - Extends daemon configuration surfaces (fusillade + dwctl + config.yaml) with
adaptive_concurrency_recovery_interval_ms, including mapping and tests. - Adds integration/unit tests and documentation describing the adaptive behavior and its metrics.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| fusillade/tests/integration.rs | Adds integration tests covering 529-triggered halving, cross-loop sharing, and escalation accounting. |
| fusillade/src/daemon/mod.rs | Integrates adaptive concurrency into capacity computation and request completion feedback; adds metrics/logging and tests. |
| fusillade/src/daemon/config.rs | Adds configurable recovery interval with default + serde behavior and a round-trip test. |
| fusillade/src/daemon/adaptive_concurrency.rs | New AIMD controller implementation with unit tests. |
| fusillade/README.md | Documents adaptive concurrency behavior and the new configuration knob. |
| dwctl/src/config.rs | Exposes and maps the new fusillade knob through dwctl config, with defaulting + tests. |
| config.yaml | Documents the new config key in the sample configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1249
to
+1260
| if is_downstream_overload(&failed.state.reason) | ||
| && let Some(configured_limit) = model_concurrency_limits | ||
| .get(&capacity_model_clone) | ||
| .map(|limit| *limit) | ||
| && let Some(adjustment) = adaptive_concurrency.record_overload( | ||
| &capacity_model_clone, | ||
| configured_limit, | ||
| Instant::now(), | ||
| ) | ||
| { | ||
| emit_concurrency_decrease(&capacity_model_clone, adjustment); | ||
| } |
Comment on lines
+1208
to
+1218
| if let Some(configured_limit) = model_concurrency_limits | ||
| .get(&capacity_model_clone) | ||
| .map(|limit| *limit) | ||
| && let Some(adjustment) = adaptive_concurrency.record_success( | ||
| &capacity_model_clone, | ||
| configured_limit, | ||
| Instant::now(), | ||
| ) | ||
| { | ||
| emit_concurrency_increase(&capacity_model_clone, adjustment); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tests
SQLX_OFFLINE=true just lint rustjust test rust