Implement grouped quota enforcement and failover metadata in RLS by AyushSawant18588 · Pull Request #1243 · envoyproxy/ratelimit

AyushSawant18588 · 2026-09-20T11:51:58Z

Problem
When a QuotaPolicy defines multiple quota buckets that apply to the same request, for example a per-tenant bucketRule plus a model defaultBucket, or several models on one route, the rate limit service treated all quota-mode descriptors as one flat set. Its rule was:

return OVER_LIMIT only when every quota descriptor is over its limit.

This produced two problems:

Per-tenant/clientSelectors limits were not enforced. For a matching tenant, both the tenant bucket and the model's always-firing defaultBucket are sent to the RLS. If the tenant bucket was exhausted (e.g. 100/100) but the default bucket still had room (e.g. 60/150), the "all descriptors over" rule returned OK. The tenant limit was counted in Redis but never actually rejected.
No independent per-model enforcement. With multiple models' quota descriptors present, the request was only rejected once every model was simultaneously exhausted, and there was no correct signal indicating which model still had quota.

The root cause was that the aggregation logic did not understand that some descriptors belong to the same enforcement scope (a model) while others are independent scopes that should support failover.

Overview of changes

Quota-mode descriptors are now grouped by their enforcement scope (the model, identified by the backend_name + model_name_override descriptor entries).
OR within a group: a group is over the limit if any of its descriptors is over (tenant bucket or default ceiling) this enforces per-tenant and default limits.
AND across groups: the overall request is OVER_LIMIT only when every quota group is over, this preserves cross-model failover semantics.
Non-quota rate-limit descriptors still reject immediately; unlimited descriptors are unchanged.
Derives a group key from the descriptor's backend_name + model_name_override entries, with a safe fallback to the full entry list when those keys aren't present (e.g. service-level catch-all quotas), so unrelated descriptors are never merged into one group.
When RESPONSE_DYNAMIC_METADATA is enabled, a passed descriptor whose group is exhausted is now excluded from the advertised metadata, so an over-limit model is never offered as an available routing target. Descriptors in non-exhausted groups are still advertised.

Testing

go build ./src/... passes.
go test ./src/service/... ./test/service/... passes, including all pre-existing quota and metadata tests.
Manually tested the changes end to end by deploying the updated rate limit service with QuotaPolicy and verifying quota enforcement behavior for tenant buckets, default buckets, and multiple model quota groups.

Signed-off-by: Ayush Sawant <ayush.sawant@nutanix.com>

missBerg · 2026-09-21T15:56:02Z

@yanavlasov can we get someone from Envoy proxy to review this?

yanavlasov · 2026-09-22T01:35:20Z

@AyushSawant18588 there are format errors.

/wait

soumya-acharya

Thanks for the changes! A few requirements and suggestions from Envoy's perspective.

soumya-acharya · 2026-09-23T05:19:00Z

+				availableDescriptors = append(availableDescriptors, idx)
+			}
+		}
+		response.DynamicMetadata = ratelimitToMetadata(request, availableDescriptors, limitsToCheck)


Filtering exhausted groups out of availableDescriptors is the right hook. ratelimitToMetadata (pre-existing) still dumps all req.Descriptors into descriptors and merges limit.Metadata into one map. Envoy cannot join that to hosts: agent-router does not set descriptor metadata, and two live backends collapse on the same key.
We should emit a list from availableDescriptors, deduped by quotaGroupKey:

"passed_backends": [ {"backend_name": "ns/be", "model_name_override": "model-b"} ]

Leave domain / descriptors / metadata if other callers need them. Envoy will read only passed_backends.

Will be making change on agent-router to set descriptor metadata and yes in metadata two live backends collapse on the same key but instead of having keys like backend_name and model_name_override which will collapse, it can directly be the values as the keys itself like:

"metadata": [ {"model-a": "ns/be"}, {"model-b": "ns/be"}, ]

So that none of them will be collapsed. With this we don't need to add new passed_backends field. Let me know if this structure is fine for envoy.

We should not invert keys into metadata and please don’t block this on agent-router descriptor.metadata.
ratelimitToMetadata still emits metadata as one merged Struct from limit.Metadata. A list of {model: backend} is a different type and a different source. If that field stays a map, two backends with the same model still collapse on one key. That is the case we have to route.

Descriptor entries already have backend_name and model_name_override. availableDescriptors is the right filter. From those indices, the best way is to emit a list as suggested above, we don’t need to change the existing metadata map at all. Envoy will read only passed_backends.

Oh yes model name can be same for different backends in a route so model-name or backed-name can't be a reliable key. I agree that it will be better to return a list of model and backend pairs of which quota is not exhausted and as this information is already available with RLS the agent-router does not need to send limit.Metadata. I have added passed backends field in dynamic metadata as you proposed. Can you review the new changes?
Thanks!

soumya-acharya · 2026-09-23T05:24:43Z

-				totalQuotaDescriptors += 1
+			// OR the statuses within a quota group.
+			if over {
+				group.over = true


I think group.over doesn't not check limitsToCheck[i].ShadowMode. A shadow over-limit bucket would mark the whole group over, it'd be better to ignore ShadowMode buckets. Can you verify this?

over is derived purely from descriptorStatus.Code == OVER_LIMIT. But a per-rule shadow-mode bucket never reaches this loop with an OVER_LIMIT code. ShadowMode is resolved much earlier, inside the cache/limiter layer. DoLimit (redis fixed_cache_impl.go) generates every descriptor status through this GetResponseDescriptorStatus, which downgrades a shadow over-limit bucket's Code to OK before it's returned. By the time shouldRateLimitWorker sees it, descriptorStatus.Code == OK, so over is false and it does not set group.over = true.

soumya-acharya · 2026-09-23T06:17:00Z

+	t.assert.True(ok)
+	// Only model-b (the group that still has quota) is advertised; model-a's
+	// passed default bucket is excluded because model-a's group is exhausted.
+	t.assert.Equal("model_b", nameVal.GetStringValue())


This makes merged metadata.name the failover signal. That will not exist in our stack. Please assert passed_backends has {backend_name=ns/be, model_name_override=model-b} and not model-a. Add two available groups so the list cannot collapse.

Whether the metadata lists collapses or not and what data ratelimit puts in metadata is depends on what we set as limit.metadata in agent-router, If we sent it as mentioned here: #1243 (comment) then it should be fine right?

Based on the above explanation we would need the test changes

I have updated the test to check for passed backends too

soumya-acharya · 2026-09-23T06:18:20Z

+// TestServiceQuotaModeMultiModelFailover verifies that with two model groups on a
+// route, one model being fully exhausted does not reject the request while
+// another model still has quota (AND across groups → failover preserved).
+func TestServiceQuotaModeMultiModelFailover(test *testing.T) {


Can we add two backend_names, same model_name_override, one group over -> overall_code=OK and only the live backend in passed_backends?

not needed if #1243 (comment) is fine

I have added this test case

Signed-off-by: Ayush Sawant <ayush.sawant@nutanix.com>

soumya-acharya

Thanks for the changes Ayush.

Implement grouped quota enforcement and failover metadata in RLS

a8251da

Signed-off-by: Ayush Sawant <ayush.sawant@nutanix.com>

AyushSawant18588 force-pushed the quota-mode-logic-update branch from 2bb7f86 to a8251da Compare September 20, 2026 11:52

yanavlasov previously approved these changes Sep 22, 2026

View reviewed changes

repokitteh-read-only Bot added the waiting label Sep 22, 2026

soumya-acharya suggested changes Sep 23, 2026

View reviewed changes

fixed formating issue

2714186

Signed-off-by: Ayush Sawant <ayush.sawant@nutanix.com>

AyushSawant18588 dismissed yanavlasov’s stale review via 2714186 September 23, 2026 07:05

repokitteh-read-only Bot removed the waiting label Sep 23, 2026

Added passed backend metadata for quota routing

504bd6d

Signed-off-by: Ayush Sawant <ayush.sawant@nutanix.com>

yanavlasov approved these changes Sep 24, 2026

View reviewed changes

soumya-acharya approved these changes Sep 25, 2026

View reviewed changes

yanavlasov merged commit 63c7e9c into envoyproxy:main Sep 28, 2026
6 checks passed

Conversation

AyushSawant18588 commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

missBerg commented Sep 21, 2026

Uh oh!

yanavlasov commented Sep 22, 2026

Uh oh!

soumya-acharya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AyushSawant18588 Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AyushSawant18588 Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soumya-acharya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AyushSawant18588 commented Sep 20, 2026 •

edited

Loading

AyushSawant18588 Sep 23, 2026 •

edited

Loading

AyushSawant18588 Sep 24, 2026 •

edited

Loading