tcp: connections are never abandoned when the sender retries a blocked send

[tinic]

Summary

NX_TCP_MAXIMUM_RETRIES cannot be reached on a socket whose sender retries a blocked
send. The connection is never abandoned: it retransmits until the interval grows without
bound, and the application waiting on it blocks with no error for as long as it is left
running.

Two changes in two files, 31 insertions and 5 deletions. No API or configuration change.

Symptom

With a router silently dropping oversized datagrams, and NX_TCP_MAXIMUM_RETRIES 6 /
NX_TCP_RETRY_SHIFT 1 — which should abandon the connection at 127 s — retransmissions of
the same sequence number were observed at:

+1s  +2s  +4s  +8s  +16s  +32s  +64s  +128s   ... and no reset

The doubling shows NX_TCP_RETRY_SHIFT taking effect. The +128 shows the limit did not.

Root cause

_nx_tcp_fast_periodic_processing() (nx_tcp_fast_periodic_processing.c:129) tests the
retry limit against one of two counters, selected by
nx_tcp_socket_zero_window_probe_has_data:

else if (((socket_ptr -> nx_tcp_socket_timeout_retries >= socket_ptr -> nx_tcp_socket_timeout_max_retries) &&
          (socket_ptr -> nx_tcp_socket_zero_window_probe_has_data == NX_FALSE)) ||
         ((socket_ptr -> nx_tcp_socket_zero_window_probe_failure >= socket_ptr -> nx_tcp_socket_timeout_max_retries) &&
          (socket_ptr -> nx_tcp_socket_zero_window_probe_has_data == NX_TRUE)))

Two defects mean neither branch can fire.

1. nx_tcp_socket_send_internal.c arms the probe for any send it cannot queue.

There are three reasons data does not go out — the receiver's advertised window, the
congestion window, and the transmit queue depth — and only the first is a zero window.
Both sites test only zero_window_probe_has_data == NX_FALSE before declaring a probe, so
a send blocked by congestion or by queue depth marks the socket as being in the persist
state when it is not. That moves the limit onto zero_window_probe_failure, which the
ordinary data path never advances.

2. nx_tcp_socket_retransmit.c clears zero_window_probe_failure on every probe
rather than when a probe begins, pinning it at 1. So even a genuine persist against a peer
that has stopped answering never reaches the limit either.

Why neither defect alone is sufficient

This is the part that is not visible in either hunk on its own, and it is why the bug
survives casual reading.

A single blocked send does not hide the limit: the next retransmission clears the flag and
the ordinary counter resumes. What hides it is the caller. A blocking sender that
re-enters nx_tcp_socket_send() after NX_WINDOW_OVERFLOW or NX_TX_QUEUE_DEPTH — and a
non-blocking sender returning on its own select loop — re-arms the flag faster than the
retransmission interval doubles. From the second interval onwards, the flag is set every
time the timer looks, so the limit is tested against the wrong counter every time.

That is the ordinary shape of a blocking socket API, which is why this is reachable in
normal use rather than only under a contrived sequence.

The fix

nx_tcp_socket_send_internal.c, both sites: require
nx_tcp_socket_tx_window_advertised == 0 in addition to the existing condition, so the
flag means what its name says.

nx_tcp_socket_retransmit.c: clear zero_window_probe_failure only when a new probe
starts, matching the two sites above.

nx_tcp_fast_periodic_processing.c is deliberately untouched — with the flag correct, its
condition is correct.

Why this does not drop peers that are legitimately holding a zero window

The obvious alternative fix — ignoring the probe counter — would abandon a live peer
advertising a zero window, which is exactly what the persist state exists to avoid.

That path is preserved. A peer that answers its probes still clears
zero_window_probe_failure in nx_tcp_socket_state_ack_check.c:555, so a socket in a
genuine persist against a responsive peer is never reset however long the window stays
shut. Only a peer that has stopped answering is given up on, after the configured number
of retries.

Reproduction

A regression test drives six of these translation units against a host shim and steps the
timer directly, so 600 simulated seconds run in about 0.3 s with no target hardware:

case                              behaviour
--------------------------------  --------------------------------------
caller retrying its write         +1 +3 +7 +15 +31 +63   reset at 127 s
zero window, probes unanswered    +1 +3 +7 +15 +31 +63   reset at 127 s
zero window, probes answered      +1 ... +511            no reset

Before this change the first two cases print +1 +3 +7 +15 +31 +63 +127 +255 +511 NO RESET. The third case is the guard against over-fixing described above, and is unchanged
by the patch.

I am happy to contribute the test in whatever form suits the project's test layout; it is
currently written against a downstream harness rather than this repository's.