feat: CBA idle re-modeling and separate scale-up / scale-down task-count boundaries by Fly-Style · Pull Request #19378 · apache/druid

Fly-Style · 2026-04-26T12:23:30Z

This PR updates the seekable-stream cost-based autoscaler to make task-count decisions more stable and easier to reason about.

The main behavioral change is replacing the previous linear idle cost with a U-shaped idle cost centered around an ideal idle ratio. This penalizes both under-provisioning, where tasks have too little idle headroom, and over-provisioning, where tasks spend too much time idle. The predictedIdleRatio clamp at 0 made the U-shape's under-provisioning penalty saturate, so smaller taskCount always won the cost race — the optimizer scaled down a
busy, lag-free cluster. The goal is to keep ingestion tasks near a practical operating point instead of treating all additional idle time as uniformly bad.

This also separates task-count boundary controls for scale-up and scale-down. Scale-up remains unbounded by default so the autoscaler can react aggressively to lag, while scale-down is bounded by default to avoid large drops in task count. Candidate task counts are still generated from valid partitions-per-task ratios, but the optimizer can now limit which candidates are evaluated depending on the configured scale direction boundary.

What Changed

Added U-shaped idle cost in WeightedCostFunction, with an ideal idle ratio and asymmetric penalties for under- and over-provisioning.
Updated default cost weights to favor a more balanced lag/idle tradeoff.
Split the single useTaskCountBoundaries setting into separate for scale up/scale down
Added CostResult.INFINITE_COST so skipped candidates can still be represented safely in cost tables.
Updated supervisor docs for the cost-based autoscaler behavior and config options.
Expanded tests around U-shaped idle cost, config serialization/defaults, valid task-count generation, and bounded vs unbounded task-count jumps.
Removed highLagThreshold and useTaskCountBoundaries parameter, as obsolete, but kept in ctor for b/w compatibility.
lag amplification multiplier was increased due to changes in idle calculation formula from a point of view where scale-up/scale-down decisions are 'normal' in terms of normal distribution near 0.5/0.5 weights. 0.4 is a good amplification multiplier that was picked after a series of tests and calculations. 0.4/0.6 as default weights was picked from a conservative point of view. A Python script with computations is available by request.

Details

Updated scaleup scenery, visualized.

Details

Task boundaries are disabled, lag = 50k, current taskCount is 1. Plot contains p* as partitions count, and idle as current poll-idle-avg-ratio metric.

Scaledown scenery, visualized.

Details

Task boundaries for scale-down are disabled, taskCount = partitionCount = 128, lag = 0.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

…of scale-up and scale-down boundaries application

FrankChen021 · 2026-04-27T13:35:31Z

-    this.useTaskCountBoundaries = Configs.valueOrDefault(useTaskCountBoundaries, false);
-    this.highLagThreshold = Configs.valueOrDefault(highLagThreshold, -1);
-    this.minScaleUpDelay = Configs.valueOrDefault(minScaleUpDelay, Duration.millis(this.minTriggerScaleActionFrequencyMillis));
+    this.useTaskCountBoundariesOnScaleUp = Configs.valueOrDefault(useTaskCountBoundariesOnScaleUp, false);


[P2] Legacy boundary setting is accepted but ignored

The constructor still accepts the legacy useTaskCountBoundaries property, but the new scale-up/down fields are initialized only from useTaskCountBoundariesOnScaleUp and useTaskCountBoundariesOnScaleDown. Existing supervisor specs with useTaskCountBoundaries: true will silently lose the scale-up boundary after upgrade, allowing unbounded jumps to any candidate task count. Map the legacy value to the new fields when the new fields are absent, or reject/document a breaking config change.

This autoscaler was in experimental mode, but I will log on warn level and document the breaking change.

Done in : 5b61b6d

gianm

Something seems off with the "Scaledown scenery, visualized" plot. It says the conditions are:

Task boundaries for scale-down are enabled, taskCount = partitionCount = 128, lag = 0.

It shows that when current idle ratio is < 0.15 or so, the optimal task count becomes ~40. Scaling down 3x when current idle ratio is low will likely lead to the new set of tasks being overloaded.

gianm · 2026-04-27T16:36:28Z

+   * Maximum number of candidate task counts to evaluate above or below the current task count
+   * when scale-up or scale-down boundaries are enabled.
+   * <p>
+   * The misspelling is preserved to avoid unnecessary churn in this package-private constant.


I don't understand this comment. The constant is new in this patch. Please fix the spelling.

gianm · 2026-04-27T16:38:14Z

+
+At every evaluation interval, Druid computes the score for each candidate task count and picks the one with the lowest total cost.

 Note: Kinesis is not supported yet, support is in progress.


Is it really in progress?

I need to verify if anybody have a Kinesis workload with CBA working. If you want, we can remove that part.

gianm · 2026-04-27T16:39:38Z

   * during extensive testing as the most balanced multiplier for high-lag recovery.
   */
-  static final double LAG_AMPLIFICATION_MULTIPLIER = 0.05;
+  static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;


Why this change?

I will note it in the patch notes. Generally, the intention was to find a point where scale-up/scale-down decisions are 'normal' in terms of normal distribution near 0.5/0.5 weights. 0.4 is a good amplification multiplier. 0.4/0.6 as default weights was picked from conservativity point of view.

gianm · 2026-04-27T16:39:59Z

        .minTriggerScaleActionFrequencyMillis(1000)
-        .lagWeight(0.2)
-        .idleWeight(0.8)
+        .lagWeight(0.8)


What will be the effect of the change to lag and idle weights?

It was passing without any problems in normal circumstances. The main idea of the change is to reduce the potential of not scaling over the timeout due to CI CPU pressure.

Fly-Style · 2026-04-27T17:23:29Z

It shows that when current idle ratio is < 0.15 or so, the optimal task count becomes ~40. Scaling down 3x when current idle ratio is low will likely lead to the new set of tasks being overloaded.

Oh, this is a lag in my head, apologies. 🤦🏻

Fly-Style added 2 commits April 26, 2026 12:34

Introduce U-shaped idle calculation, prepare a ground for separation …

e38fbb6

…of scale-up and scale-down boundaries application

Separate scale barriers, update documentation and defaults

0266cb3

github-actions Bot added Area - Documentation Area - Ingestion labels Apr 26, 2026

Correct docs

df864db

Fly-Style changed the title ~~Cba autoscaler tweaking~~ feat: CBA idle re-modeling and separate scale-up / scale-down task-count boundaries Apr 26, 2026

Fly-Style requested a review from jtuglu1 April 27, 2026 06:39

Fly-Style added 2 commits April 27, 2026 12:15

Keep old config props for bw compatibility

e367f87

Fix issue with bounds check, which was rolled back accidentally

9efa91d

FrankChen021 reviewed Apr 27, 2026

View reviewed changes

Fly-Style self-assigned this Apr 27, 2026

Log the legacy useTaskCountBoundaries and highLagThreshold usage

5b61b6d

Fly-Style force-pushed the cba-autoscaler-tweaking branch from ec723ee to 5b61b6d Compare April 27, 2026 15:31

gianm reviewed Apr 27, 2026

View reviewed changes

Correct idle function for scale-down

3f7dcf1

Fly-Style requested review from FrankChen021 and gianm April 27, 2026 21:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: CBA idle re-modeling and separate scale-up / scale-down task-count boundaries#19378

feat: CBA idle re-modeling and separate scale-up / scale-down task-count boundaries#19378
Fly-Style wants to merge 7 commits intoapache:masterfrom
Fly-Style:cba-autoscaler-tweaking

Fly-Style commented Apr 26, 2026 •

edited

Loading

Uh oh!

FrankChen021 Apr 27, 2026

Uh oh!

Fly-Style Apr 27, 2026 •

edited

Loading

Uh oh!

Fly-Style Apr 27, 2026

Uh oh!

gianm left a comment

Uh oh!

gianm Apr 27, 2026

Uh oh!

gianm Apr 27, 2026

Uh oh!

Fly-Style Apr 27, 2026

Uh oh!

gianm Apr 27, 2026

Uh oh!

Fly-Style Apr 27, 2026 •

edited

Loading

Uh oh!

gianm Apr 27, 2026

Uh oh!

Fly-Style Apr 27, 2026

Uh oh!

Fly-Style commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		At every evaluation interval, Druid computes the score for each candidate task count and picks the one with the lowest total cost.

		Note: Kinesis is not supported yet, support is in progress.

Conversation

Fly-Style commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Details

Updated scaleup scenery, visualized.

Scaledown scenery, visualized.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fly-Style Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fly-Style Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fly-Style commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fly-Style commented Apr 26, 2026 •

edited

Loading

Fly-Style Apr 27, 2026 •

edited

Loading

Fly-Style Apr 27, 2026 •

edited

Loading