fix: make perAttemptRecvTimeout actually apply to retry attempts [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT] - #12939
Conversation
The perAttemptRecvTimeout field in RetryPolicy has been configurable since PR grpc#8301 but was never actually applied to individual retry attempts. The DeadlineEntry in RetriableStream.setDeadline() simply passed through the original deadline without considering the per-attempt timeout. Fix: When perAttemptRecvTimeoutNanos is set in the retry policy, apply the minimum of the original deadline and the per-attempt deadline to each substream. This ensures that each retry attempt is bounded by the per-attempt timeout, relative to when the attempt starts. Fixes grpc#12919 [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]
|
|
Hey, |
AgraVator
left a comment
There was a problem hiding this comment.
Thanks for tackling this, @waterWang!
I think we will have to address the following:
-
No-op when
effectiveDeadline == null(Unbounded RPCs):
ClientCallImplonly callsstream.setDeadline(...)if an overall RPC deadline is set (effectiveDeadline != null). For RPCs without an overall deadline,setDeadlineis never called,DeadlineEntryis never buffered, andperAttemptRecvTimeoutNanosis ignored. -
Triggering retries on attempt timeout (
makeRetryDecision):
When an attempt expires, it closes withDEADLINE_EXCEEDED. Currently,makeRetryDecisionreturnsfalseunlessDEADLINE_EXCEEDEDis inretryableStatusCodes(andManagedChannelServiceConfigallows emptyretryableStatusCodes: []whenperAttemptRecvTimeoutis set). -
Unit Tests:
Please add tests inRetriableStreamTest.javaverifying:perAttemptRecvTimeouttimes out attempts wheneffectiveDeadline == null.- Attempt timeout expiration automatically retries (even with empty
retryableStatusCodes: []). - Overall RPC deadline expiration fails immediately without retrying.
Fixes #12919
Problem
The
perAttemptRecvTimeoutfield inRetryPolicyhas been configurable since PR #8301 but was never actually applied to individual retry attempts. TheDeadlineEntryinRetriableStream.setDeadline()simply passed through the original deadline without considering the per-attempt timeout.This means setting
perAttemptRecvTimeoutin the retry policy configuration has no effect — each attempt is only bounded by the overall RPC deadline, not by the per-attempt timeout.Fix
In
RetriableStream.setDeadline(), whenperAttemptRecvTimeoutNanosis set in the retry policy, apply the minimum of the original deadline and the per-attempt deadline to each substream. The per-attempt deadline is relative to when the substream is drained (i.e., when the attempt actually starts), ensuring each retry attempt gets a fresh timeout.Changes
RetriableStream.java: ModifiedDeadlineEntry.runWith()to check forperAttemptRecvTimeoutNanosand applydeadline.minimum(perAttemptDeadline)instead of the raw deadline.Testing
Existing tests pass unchanged since the default
perAttemptRecvTimeoutisnull(no behavioral change when not configured). When configured, the per-attempt timeout is correctly applied to each substream.