Established connection never detected as dead on silent socket loss — client hangs firing time bursts forever

[TermeHansen]

Summary

When the server's socket dies silently (TCP connection blackholed — no FIN, no RST, no WebSocket close frame), the client detects nothing. The connection stays admitted as current and is_connected() stays true indefinitely, so the client sits in a half-open state firing time-sync bursts that all time out and repeat forever. There is no liveness detection at any layer, and no path ever tears down the stale connection.

Expected behavior

A connection that stops producing any traffic (or stops acknowledging time messages) for a bounded period should be treated as dead: dropped and reconnected. The client should not block/hang indefinitely on a silent peer.

Observed behavior

In the sendspin-client this has been reported as an issue (TermeHansen/sendspin-client#14), with a log output similar to:

W sendspin.time_burst: Time message 8/8 timed out
W sendspin.time_burst: Time message 1/8 timed out   (+20s)
W sendspin.time_burst: Time message 2/8 timed out   ... repeats indefinitely
Burst completes (8/8), restarts after the inter-burst interval, and never gives up. No "connection lost" log is ever emitted. Once the kernel send buffer fills on the dead socket, the time-message send blocks the main loop for minutes at a time — the client hangs between bursts.

Root cause

Liveness is never enforced on an established connection:

1. No application-level heartbeat anywhere. No ping/keepalive is configured on the host or ESP transport (sendspin-cpp never calls IXWebSocket's ping/closeOnPongTimeout or any other keepalive). The server dying silently therefore produces no Close event, so SendspinClientConnection::connected_ stays true and the connection stays managed.
2. The existing periodic traffic — the time burst — is not used as a liveness signal. SendspinTimeBurst::loop() increments burst_index_ and restarts at 8/8 with no give-up (src/time_burst.cpp:57), regardless of whether the peer ever answers.
3. ConnectionManager::loop() only reaps nursery (pre-handshake) connections via NURSERY_ESTABLISH_TIMEOUT_US; there is no equivalent timeout for an admitted/established connection, so nothing ever drops it. And once the dead socket's TX window fills, time_burst_'s send_time_message() → ws_->send() blocks the main loop (client.loop()), stalling the whole client.

[kahrendt]

Thanks for reporting! Let me know if that fix doesn't handle it.

[TermeHansen]

@kahrendt Thanks for the quick fix! I tested bbc5958 , and it resolves the detection half but not recovery — the client still ends up offline permanently.

Setup: the client runs with a configured server URL, so it connects outbound via connect_to() (not the inbound/discovery direction where the server connects in).
What we see now with the new build:

W sendspin.time_burst: Time message 1/8 timed out
W sendspin.time_burst: Time message 2/8 timed out
...
W sendspin.conn_mgr: Current connection silent for >60000 ms, dropping as lost
<no further activity>

So the liveness watchdog fires correctly (that part works, and the endless timeout spam is gone — thanks for that). But after the drop, nothing ever re-establishes the connection:

• The manager sets auto_reconnect(false) on outbound transports (connect_to()), so the transport deliberately doesn't recover.
• The restart goodbye asks "a live server" to reconnect — that holds for inbound connections, but on a blackholed socket the goodbye is undeliverable, and there's no server watching this client anyway. The client initiated the connection, so recovery has to come from the client side.
• The app only calls connect_to() once at startup, so with the current slot empty and the nursery empty, the client just sits disconnected forever.

Before your fix I made a local patch that added the missing half: on reaping an outbound connection, arm a reconnect with exponential backoff (the manager already knows the target URL from connect_to()) and re-issue connect_to() until it establishes again. That combination — detection + reconnect-with-backoff for outbound — seemed to fully fix it.

Should I make a PR along those lines?

[kahrendt]

Yes, please open a PR with it. Thanks!

[TermeHansen]

@kahrendt see the PR #127