Skip to content

fix: end each attempt with a typed outcome and decide retries before the hooks - #60

Merged
ilyam8 merged 3 commits into
salt-per-messagefrom
attempt-outcomes
Oct 14, 2026
Merged

ilyam8 merged 3 commits into
salt-per-messagefrom
attempt-outcomes

Conversation

@ilyam8

@ilyam8 ilyam8 commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Second step of the bug-fix phase: each attempt of a request ends with a typed outcome, and the decision whether
another attempt follows is made from that type before any hook runs.

An attempt's outcome used to be an error and a retry flag, which caused four problems:

  • A timeout was recognized by the word "timeout" in the error's text.
  • A TCP reconnect returned a nil error, which the timeout check dereferenced under a context deadline.
  • OnRetry ran before the decision, so it also ran after the last attempt.
  • The error of a request whose attempts timed out wrapped nothing, so Netdata classifies every gosnmp request timeout
    as "unknown".

Now each attempt ends with one kind: reply, timeout, transport error, invalid reply, reconnected, closed, context done
or failed. A failed write or read is classified once, with errors.Is: os.ErrDeadlineExceeded is a timeout, and
net.ErrClosed ends the request at once. After an attempt that another may cure, one decision runs in this order:

  1. A done Context ends the request with its error.
  2. A timeout at the Context's deadline ends it with context.DeadlineExceeded.
  3. Otherwise another attempt follows, while any are left.
  4. When none is left, the request fails with the last attempt's error.

OnRetry, the retry log line and ExponentialTimeout run only when another attempt follows.

Behavior changes:

  • OnRetry no longer runs after the last attempt.
  • A request whose attempts time out still fails with "request timeout (after N retries)". The error now unwraps to the
    deadline error, a timeout net.Error. After a TCP reconnect, "max retries (N) exceeded" unwraps to io.EOF.
  • A write error that mentions a timeout is returned as it is.
  • A cancellation during the last attempt ends the request with context.Canceled, not a request timeout. The
    cancellation is still noticed only when the attempt's deadline passes; a later step interrupts the read.
  • A connection closed during a request (net.ErrClosed) ends it at once instead of failing every remaining attempt.
  • A Context deadline equal to the attempt's counts as the Context's, so the result is context.DeadlineExceeded
    whichever timer fires first.

Tests

  • First commit, pinning the current behavior:
    • two scenarios where the connection is closed during a request, on a read and on a write;
    • TestEngineRequestTimeoutError, on a real UDP socket. It checks what an error classifier sees and fails before
      the fix.
  • Golden flips:
    • the trailing OnRetry lines are gone;
    • the request-timeout results are typed and match os.ErrDeadlineExceeded and a timeout net.Error;
    • the context, closed and timeout-in-text scenarios above;
    • the TCP reconnect cases, including the former nil-error panic, which now runs out of retries.
  • Last commit, from the reviews, pinning the Context rules deterministically:
    • two scenarios with a Context that reports its deadline but not yet its expiry, the moment a real clock has
      between the two. In the synctest bubble the Context's timer fired with the attempt's deadline, so removing the
      deadline rule or the tie rule went unnoticed in some runs;
    • a deadline that passes during a hook before a refused read: context.DeadlineExceeded, no OnRetry;
    • a cancellation before the deadline: context.Canceled.

Validation

  • Local: golangci-lint, race tests (30 runs of the request scenarios), FIPS 140-only, six cross-builds, govulncheck
    and netsnmp playback pass.
  • Docker: GOARCH=386 and the end-to-end suite against snmpd 5.9.3 pass.
  • Benchmarks (SendOneRequest, Get, GetV3, Walk; 12 interleaved rounds): no time, byte or allocation change.
  • 20 mutants of the changed code: all killed in every run but one equivalent. Marking a Context-done attempt
    retryable changes nothing, because the decision returns the Context's error first.
  • Two independent reviews (the retry decision; test integrity and docs): one test-integrity gap (the Context rules
    above) and wording fixes, in the last commit.

…eout

Two scenarios pin what a request does when its connection is closed
during it, on a read and on a write: each further attempt fails at once,
with its hooks, until the retries run out.

TestEngineRequestTimeoutError makes every attempt of a request time out on
a real UDP socket and checks what a caller that classifies errors sees:
the "request timeout" text, unwrapping to the read deadline, a timeout
net.Error. It fails until the error wraps the last attempt's reason: today
it wraps nothing, so Netdata classifies every gosnmp request timeout as
"unknown".
…the hooks

An attempt's outcome was an error and a retry flag: a timeout was
recognized by the word "timeout" in the error's text, a TCP reconnect
returned a nil error (which the timeout check dereferenced under a context
deadline), OnRetry ran before the decision whether another attempt follows
(so also after the last one), and the error of a request whose attempts
timed out wrapped nothing: Netdata classifies every gosnmp request timeout
as "unknown".

Each attempt now ends with a kind: reply, timeout, transport error, invalid
reply, reconnected, closed, context done or failed. A failed write or read
is classified once, with errors.Is: os.ErrDeadlineExceeded is a timeout,
net.ErrClosed ends the request at once. After an attempt that another may
cure, one decision runs before any hook: a done Context ends the request
with its error, a timeout at the Context's deadline with
context.DeadlineExceeded (a deadline equal to the attempt's now counts as
the Context's), and when no attempt is left the request fails with the
last attempt's error; after a timeout the text stays "request timeout
(after N retries)" and after a reconnect "max retries (N) exceeded", each
unwrapping to the attempt's reason (the deadline error, a timeout
net.Error; io.EOF). OnRetry, the retry log line and ExponentialTimeout
run only when another attempt follows.

So: OnRetry no longer runs after the last attempt; a write error that
mentions a timeout is returned as it is; a cancellation during the last
attempt is context.Canceled, not a request timeout; a connection closed
during a request ends it at once.
In every scenario that reached them, the Context's timer fired at the same
instant as the attempt's deadline, so the rule that a timeout at the
Context's deadline gives context.DeadlineExceeded, and the rule that an
equal deadline is the Context's, were pinned only when the timer happened
to run late: mutants removing either survived some runs. Two scenarios use
a Context that reports its deadline but not its expiry, the moment a real
clock has between the two. Two more pin a Context whose deadline passed
during a hook before a refused read (no OnRetry), and a Context canceled
before its deadline (context.Canceled, not the deadline).

The closed-connection scenarios now model a closed socket, whose deadline
cannot be set either. TestSendOneRequest_TCP_EOF_Reconnect checks its error
with errors.Is instead of its text, and TestEngineTCPReconnect loses its
unused knownBug field.

Docs: the order of the Context rules, a failed reconnect under a done
Context, and a closed connection excluded from the errors that start
another attempt.
@ilyam8
ilyam8 added this pull request to stack #61 October 13, 2026 22:36
@ilyam8 ilyam8 changed the title attempt outcomes fix: end each attempt with a typed outcome and decide retries before the hooks Oct 13, 2026
@ilyam8
ilyam8 merged commit 45e514c into master Oct 14, 2026
22 checks passed
@ilyam8
ilyam8 deleted the attempt-outcomes branch October 14, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants