atenet-router: raise actor-cluster circuit breakers - #806
atenet-router: raise actor-cluster circuit breakers#806Chuang Wang (chuangw6) wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
d98eeca to
5cfc492
Compare
Chuang Wang (chuangw6)
changed the title
17658f0 to
3c308a7
Compare
The actor cluster configured no breakers, leaving Envoy's 1,024 default, which caps concurrent requests through the router far below what it can carry. Connections, pending and requests rise to 20,000, kept under the ~28k source-port budget so overload trips a counted breaker instead of the kernel's opaque EADDRNOTAVAIL; max_retries keeps its default since nothing here retries. The ext_proc breaker now applies its ceiling to max_pending_requests as well as max_requests. The manifest also enables dispatcher stats and mutex tracing for observability and pins --concurrency at 8.
3c308a7 to
d9139a5
Compare
Bowei Du (bowei)
left a comment
Bowei Du (bowei)
left a comment
There was a problem hiding this comment.
I see you added new limits and constants -- but how do we expect it to behave for people who install it on a smaller machine than you are giving space for?
| cpu: 250m | ||
| memory: 256Mi | ||
| limits: | ||
| cpu: "8" |
There was a problem hiding this comment.
You should put a comment why the limit needs to be set:
Otherwise Envoy configures too many event loops on a large machine?
Does this affect scheduling on a small machine?
| - name: "drain-signal" | ||
| mountPath: "/var/run/atenet" | ||
| - name: envoy | ||
| # Not v1.39+: it measured 20-38% lower sustained throughput on this |
There was a problem hiding this comment.
remove this comment
| # Envoy sizes worker threads from the node's CPU count unless told | ||
| # otherwise; on a large node that is dozens of event loops. | ||
| - "--concurrency" | ||
| - "8" |
There was a problem hiding this comment.
But what if you install on a smaller machine? Are we ok with setting this to 8 overall or should this be tuned?
| - "8" | ||
| # Observability: mutex-contention counters on the admin /contention | ||
| # endpoint. Costs one atomic per contended acquisition. | ||
| - "--enable-mutex-tracing" |
There was a problem hiding this comment.
What does this give us in production
2 participants
Part of #665.
The actor cluster configured no circuit breakers, leaving Envoy's 1,024 default — which caps concurrent requests through the router far below what it can carry and reads, from outside, as the router running out of capacity.
max_connections,max_pending_requestsandmax_requestsrise to 20,000, kept under the ~28k ephemeral-port budget so overload trips a counted breaker instead of the kernel's opaqueEADDRNOTAVAIL.max_retrieskeeps Envoy's default — nothing on this cluster retries.--extproc-max-requestsceiling now coversmax_pending_requestsas well asmax_requests; the derived default is unchanged.--concurrencypinned at 8, and resources with small requests plus 8-core limits so the pod schedules anywhere while CPU stays capped.Tested with
go test ./cmd/atenet/internal/router/.🤖 Generated with Claude Code