fix: give the client one connection it owns and let Close end a request

[ilyam8]

Summary

Fourth step of the bug-fix phase: the client owns one connection, and Close can end a running request.

The client used to replace its connection in place and close nothing it replaced:

• A second Connect leaked the first connection; a failed dial dropped the old one without closing it.
• A failed unconnected UDP listen left a nil *net.UDPConn in Conn, which is not nil.
• The TCP reconnect leaked the connection the agent closed, left Conn nil when its dial failed, and dialed with
  the client's Target, Port, LocalAddr and Control at that moment, so a hook redirected the running request.

Close was not safe while a request ran:

• Requests read Conn without the lock Close takes: a Close from another goroutine was a data race, and a request
  after it dereferenced a nil Conn (the recover turned the panic into an error with stacks).
• The Handler's Close closed Conn itself: it panicked without a connection and closed it again on a second call.
  Netdata's SNMP topology calls it from a goroutine to cancel walks, and worked only because of that.

A cancelled Context was noticed only when the attempt's deadline passed.

Now:

• The client records the connection it opened: the net.Conn, the settings that open it again, and the agent's
  address of an unconnected socket.
• Connect and a TCP reopen open a connection in locals and replace the client's only when it opened, closing the old
  one. A failed Connect or reopen leaves the client as it was.
• A TCP connection whose write or read fails, other than by the attempt's deadline or a close (net.ErrClosed), is
  lost: the agent closed it (EOF) or reset it. The next attempt, of the same request or a later one, opens it again
  before writing, with the settings it was opened with (a Conn the caller set: with the request's configuration),
  within the request's timeout and Context; a refused reopen leaves it lost for the attempt after, so the client
  recovers once the agent accepts again. This is the only place a TCP connection is reopened. A connection a hook
  replaced is not reopened.
• Opening a connection (Connect, a TCP reopen) takes one deadline from the Context and Timeout before its first step:
  the lookups of the local address and of an unconnected socket's agent run within it, as the dial does, with the
  standard library's queries and rules for choosing among the addresses, zones of scoped IPv6 addresses included. One
  intended difference, documented in lookupIPAddrs: on a network with "4" or "6", a zone a hosts file gives an
  address that is not scoped (IPv4, global IPv6) is dropped; keeping it would take a second lookup of every name. A canceled or timed-out opening fails with an error that
  matches context.Canceled or context.DeadlineExceeded.
• Connect, Close and a reopen change the connection under the client's lock; each attempt takes it under the lock.
• Names are resolved only by Connect and by a TCP reopen's dial, never during a request. Connect records the agent's
  address it resolved for an unconnected UDP socket (a Connect in connected mode clears it; Close keeps it); a Conn
  the caller sets with UseUnconnectedUDPSocket, such as a wrapper of the client's socket, writes to it. A caller's
  socket without such a Connect has no address, as before.
• Each attempt registers its connection with the client after setting its deadline. Close and a Context that can be
  cancelled interrupt the registered attempt through one rule: a deadline in the past on the attempt's connection,
  set outside the client's lock, keyed by the attempt's number so a late interrupt cannot reach a later attempt, and
  waited for when the attempt ends. A Context that cannot be cancelled costs nothing.
• Close is safe from any goroutine at any time. It interrupts the running attempt, whatever connection it uses (a
  hook may have replaced Conn), and closes the connection Conn holds; a connection a hook replaced is the
  caller's again. Once Close has run, an attempt that ends without a reply ends the request with an error wrapping
  net.ErrClosed, whatever else ended it (a refused reopen, a timeout, a cancelled Context), for any net.Conn;
  later requests fail the same way until Connect opens a connection or Conn is set. Conn is nil after it, and
  calls after the first return nil. Close does not cut the dial of a TCP reopen short; its timeout and the Context
  bound it.
• The check that the connection is still the client's never panics: a Conn of a type == cannot compare counts as
  unchanged.
• The Handler's Close is the client's.

Behavior changes:

Case	Before	After
Close during a request, from another goroutine	data race; nil dereference	the request fails at once with net.ErrClosed
Close after a hook replaced Conn during the attempt	the attempt waits for its timeout	the request fails at once with net.ErrClosed
Request after Close	"Conn is missing"	error wrapping net.ErrClosed
Handler Close without a connection / twice	panic / closes again	nil / nil
Second Connect	first connection leaked	first connection closed
Failed Connect	Conn nil, old connection leaked	client unchanged
Agent closes the TCP connection	reconnect at once, in the read; old connection leaked; dials the client's live settings	the next attempt reopens it, after OnRetry; old connection closed; dials the connection's settings
Failed TCP reopen	Conn nil: every later request fails	the connection stays lost; the next attempt tries again
Agent resets the TCP connection (no EOF)	retried on the dead connection: "broken pipe" forever	the next attempt reopens it
Slow name service during Connect or a TCP reopen (named LocalAddr, unconnected agent)	the lookup ignores the Context and Timeout	bounded by them, with their error
Unconnected UDP socket the caller set	writes to the address the last unconnected Connect resolved, also after a later connected one	writes to the address the last Connect resolved; none after a Connect in connected mode
Context cancelled during an attempt	noticed at the attempt's deadline	ends the attempt at once

Tests

• First commit: the engine harness's in-memory transport becomes safe for concurrent use (a deadline set or a Close
  from another goroutine wakes a waiting read; after Close, I/O fails with net.ErrClosed), with the existing
  transcripts unchanged, and pins of the old behavior: Close before, during and between attempts, the Handler's
  Close, a second Connect, the TCP reconnect's old connection, a refused reconnect and a hook that redirected it.
• Golden flips: the Close scenarios end with net.ErrClosed; cancellations end at the cancellation.
• New: Close from another goroutine during a request and during a walk (in the harness and on real sockets, through
  the Handler, two goroutines at once, as Netdata does); Close during Connect; a Context cancelled on a real socket and
  while the attempt sets its deadline; the interrupt held until after the reply arrived; a Context cancelled after its
  request; Close during the TCP reconnect's dial (refused and opened, on the request's goroutine and on another); a
  Conn a hook set during the attempt; a caller's or a hook's Conn of a type == cannot compare; the reconnect's
  settings, Context and timeout for a Conn the caller set; a Close after the reply was read.
• From the final review: Close after a hook removed or replaced Conn (with and without a Context); a Close while the
  Context's interrupt is still setting its deadline; a Close after a request whose hook replaced Conn; each attempt
  unregisters its Context interrupt; recovery after the agent closes the TCP connection and two reopens are refused,
  after a reset, and none after a timeout or a close; a reopen's dial bounded by the request's Context; where an
  unconnected UDP client writes for its own socket, a wrapper of it (also after ConnectIPv6) and a socket the caller
  opened, after Close and after a Connect in connected mode; no name lookup during a request; a Context interrupt that
  runs late leaves the next attempt alone; Connect's and a TCP reopen's lookups within the Context and Timeout (both
  connection origins; cancel, Timeout, Close), with the address choice checked against net.ResolveUDPAddr and
  net.ResolveTCPAddr on 102 inputs over a stub DNS; zones of names a hosts file maps to scoped IPv6 addresses.

Validation

• Local: golangci-lint, race tests, FIPS 140-only, six cross-builds, govulncheck and netsnmp playback pass.
  GOARCH=386 and the end-to-end suite run in CI.
• Benchmarks (10 rounds): the library's allocations are unchanged and time is within noise; a cancellable Context
  costs about 5 allocations and 180 ns per attempt.
• 37 mutants of the changed code: all killed but two that are equivalent (a stale record nothing reaches, an
  allocation-only change). Removing any of the six lock sites fails under -race. The goroutine tests pass 50 runs
  each with one and two CPUs under -race.
• Two independent reviews (concurrency and ownership; tests, harness and docs). They found a Close during a refused
  reconnect ending without net.ErrClosed, a caller Conn of an uncomparable type panicking under the lock (which
  then stayed held), a flaky test on few CPUs, a stale Handler doc, and test gaps (two lock sites, the order of the
  deadline and the interrupt, the reconnect's timeout). All fixed with regression tests in the third commit; the
  follow-up review's precision and wording notes are in the fourth.
• A final external review found three more, each reproduced first and fixed as a class in its own commit: Close did
  not reach the connection an attempt used after a hook replaced Conn (fifth commit: one interrupt rule for Close
  and the Context); a caller's unconnected UDP socket lacked the agent's address (sixth); a refused TCP reconnect never
  recovered, because only an EOF read triggered it and later writes failed with a broken pipe (seventh: the lost
  rule, which also recovers a connection the agent reset); the eighth and ninth fix comments. A re-review of those
  found the sixth commit's per-attempt name lookup ran outside the request's Context, Close and timeout and lost
  ConnectIPv6's address family: the eleventh commit replaces it (names are resolved only by Connect); the tenth pins
  the late-interrupt guard the re-review showed testable. A full review then found Connect and a TCP reopen resolving
  a named local address outside the Context and Timeout: the twelfth commit gives the whole opening one deadline; the
  next review found its lookups dropped the zone of a name's scoped IPv6 address, which the thirteenth keeps. All 59
  mutants of these fixes are killed. A cancellable Context costs two allocations less per attempt than before
  these fixes; other benchmarks are unchanged.