Repository navigation
fix: end each attempt with a typed outcome and decide retries before the hooks - #60
Merged
Merged
Conversation
…eout Two scenarios pin what a request does when its connection is closed during it, on a read and on a write: each further attempt fails at once, with its hooks, until the retries run out. TestEngineRequestTimeoutError makes every attempt of a request time out on a real UDP socket and checks what a caller that classifies errors sees: the "request timeout" text, unwrapping to the read deadline, a timeout net.Error. It fails until the error wraps the last attempt's reason: today it wraps nothing, so Netdata classifies every gosnmp request timeout as "unknown".
…the hooks An attempt's outcome was an error and a retry flag: a timeout was recognized by the word "timeout" in the error's text, a TCP reconnect returned a nil error (which the timeout check dereferenced under a context deadline), OnRetry ran before the decision whether another attempt follows (so also after the last one), and the error of a request whose attempts timed out wrapped nothing: Netdata classifies every gosnmp request timeout as "unknown". Each attempt now ends with a kind: reply, timeout, transport error, invalid reply, reconnected, closed, context done or failed. A failed write or read is classified once, with errors.Is: os.ErrDeadlineExceeded is a timeout, net.ErrClosed ends the request at once. After an attempt that another may cure, one decision runs before any hook: a done Context ends the request with its error, a timeout at the Context's deadline with context.DeadlineExceeded (a deadline equal to the attempt's now counts as the Context's), and when no attempt is left the request fails with the last attempt's error; after a timeout the text stays "request timeout (after N retries)" and after a reconnect "max retries (N) exceeded", each unwrapping to the attempt's reason (the deadline error, a timeout net.Error; io.EOF). OnRetry, the retry log line and ExponentialTimeout run only when another attempt follows. So: OnRetry no longer runs after the last attempt; a write error that mentions a timeout is returned as it is; a cancellation during the last attempt is context.Canceled, not a request timeout; a connection closed during a request ends it at once.
In every scenario that reached them, the Context's timer fired at the same instant as the attempt's deadline, so the rule that a timeout at the Context's deadline gives context.DeadlineExceeded, and the rule that an equal deadline is the Context's, were pinned only when the timer happened to run late: mutants removing either survived some runs. Two scenarios use a Context that reports its deadline but not its expiry, the moment a real clock has between the two. Two more pin a Context whose deadline passed during a hook before a refused read (no OnRetry), and a Context canceled before its deadline (context.Canceled, not the deadline). The closed-connection scenarios now model a closed socket, whose deadline cannot be set either. TestSendOneRequest_TCP_EOF_Reconnect checks its error with errors.Is instead of its text, and TestEngineTCPReconnect loses its unused knownBug field. Docs: the order of the Context rules, a failed reconnect under a done Context, and a closed connection excluded from the errors that start another attempt.
ilyam8
added this pull request to stack #61
October 13, 2026 22:36
stelfrag
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second step of the bug-fix phase: each attempt of a request ends with a typed outcome, and the decision whether
another attempt follows is made from that type before any hook runs.
An attempt's outcome used to be an error and a retry flag, which caused four problems:
OnRetryran before the decision, so it also ran after the last attempt.as "unknown".
Now each attempt ends with one kind: reply, timeout, transport error, invalid reply, reconnected, closed, context done
or failed. A failed write or read is classified once, with
errors.Is:os.ErrDeadlineExceededis a timeout, andnet.ErrClosedends the request at once. After an attempt that another may cure, one decision runs in this order:context.DeadlineExceeded.OnRetry, the retry log line andExponentialTimeoutrun only when another attempt follows.Behavior changes:
OnRetryno longer runs after the last attempt.deadline error, a timeout
net.Error. After a TCP reconnect, "max retries (N) exceeded" unwraps toio.EOF.context.Canceled, not a request timeout. Thecancellation is still noticed only when the attempt's deadline passes; a later step interrupts the read.
net.ErrClosed) ends it at once instead of failing every remaining attempt.context.DeadlineExceededwhichever timer fires first.
Tests
TestEngineRequestTimeoutError, on a real UDP socket. It checks what an error classifier sees and fails beforethe fix.
OnRetrylines are gone;os.ErrDeadlineExceededand a timeoutnet.Error;between the two. In the synctest bubble the Context's timer fired with the attempt's deadline, so removing the
deadline rule or the tie rule went unnoticed in some runs;
context.DeadlineExceeded, noOnRetry;context.Canceled.Validation
and
netsnmpplayback pass.SendOneRequest,Get,GetV3,Walk; 12 interleaved rounds): no time, byte or allocation change.retryable changes nothing, because the decision returns the Context's error first.
above) and wording fixes, in the last commit.