Skip to content

Fix/reconnect multinode timeout - #1898

Open
Ente3 wants to merge 4 commits into
CloudNetService:nightlyfrom
Ente3:fix/reconnect-multinode-timeout
Open

Ente3 wants to merge 4 commits into
CloudNetService:nightlyfrom
Ente3:fix/reconnect-multinode-timeout

Conversation

@Ente3

@Ente3 Ente3 commented Sep 30, 2026

Copy link
Copy Markdown

Motivation

When the connection between two nodes dropped, even for a moment, both nodes removed each
other from the cluster immediately and never tried to reconnect. The cluster stayed split
until a node was restarted, and each node continued as its own head node. This happens
more often with nodes in different datacenters and with more nodes in the cluster.

The reconnect grace period (cloudnet.max.node.disconnect.millis) only applied to nodes
that stopped sending updates while their socket stayed open. When the socket itself was
closed, the channel close handlers called NodeServer#close() directly. The reconnect
handshake had also never really been used, and it had several problems once it was.

Modification

  • A lost node connection now enters the reconnect grace period when
    cloudnet.max.node.disconnect.millis is greater than 0, instead of removing the node
    directly. The default of 0 keeps the previous behaviour.
  • Only one reconnect attempt per node is in flight at a time, because concurrent
    connections can break the reconnect handshake. Nodes stuck in the syncing state after
    an interrupted reconnect are removed once the grace period expires.
  • Nodes send their startup time when they authorize. The other node can then detect a
    restart during the grace period and let the node join as a newly started node, instead
    of reconnecting it into its old session (which failed with an NullPointerException in
    AuthorizationResponsePacketListener). Nodes on older versions don't send it, and
    older nodes ignore it, so mixed versions keep working.
  • Remote service providers look up the current channel of their node on each call,
    instead of keeping the channel that was used during registration. That channel is
    closed after a reconnect.
  • The service sync during a reconnect now applies the reconnecting node's services.
    Before, they were skipped because the node wasn't available() yet.
  • Queued packets are flushed when a reconnect ends in a plain authorization, and local
    services are re-published to the other node.
  • The head node is re-selected when a node becomes ready again, so all nodes agree on
    the same head node.
  • The grace period is read when NodeDisconnectHandler is created, so it can be set in
    tests.
  • Added unit tests for NodeDisconnectHandler, NodeDisconnectTrackerTask,
    AuthorizationPacketListener and AuthorizationResponsePacketListener.
  • Updated the launcher.cnl description of cloudnet.max.node.disconnect.millis.

Result

With a grace period configured, nodes reconnect after short network interruptions and
agree on the same head node afterwards. A node that restarts within the grace period
joins cleanly as a new node. If a node does not come back within the grace period, it is
removed like before. With the default setting, behaviour is unchanged.

Tested manually with two nodes (grace period 30s/60s):

  • Killing the node connection (ss -K): before, both nodes dropped each other and never
    reconnected. Now both are READY again within a few seconds, with the same head node.
  • Blocking the connection with a firewall (idle timeout): reconnects, and the head node
    now matches on both nodes (before, each node showed itself as head).
  • Crashing and restarting a node within the grace period: the restart is detected and
    the node joins as a new node without errors.
  • Outage longer than the grace period: nodes are removed after the grace period as before.
  • Grace period 0: same behaviour as before.
Other context

Known limitations:

  • If a node does not reconnect within the grace period, the cluster stays split until a
    node is restarted, like before.
  • During the grace period, services and players of the disconnected node stay registered.
  • The service-related changes in DefaultCloudServiceManager are not covered by unit
    tests, as the class needs most of the node to be constructed.

Tests:
https://hastebin.baxnetzwerk.de/ct-Z4gFrfyk
With cloudnet.max.node.disconnect.millis set to 60.000:
https://hastebin.baxnetzwerk.de/dzvj7KguKXk
With cloudnet.max.node.disconnect.millis set to 0 or default:
https://hastebin.baxnetzwerk.de/Bz+rvVdSEei
(the links are available for 1 year.)

Fixes #1897

Ente3 and others added 4 commits September 30, 2026 17:36
A dropped node connection removed both nodes from the cluster right
away, and nothing tried to reconnect. A lost connection now enters the
reconnect grace period if cloudnet.max.node.disconnect.millis is
greater than 0. The default (0) keeps the old behaviour.

- only one reconnect attempt per node at a time
- remove nodes stuck in syncing once the grace period expires
- send the startup time on authorization, so a restart during the
  grace period is treated as a new node
- remote service providers look up the current node channel on each
  call instead of keeping the one from registration
- apply the reconnecting node's services during service sync
- flush queued packets and re-publish local services after a plain
  authorization
- re-select the head node when a node becomes ready again
- read the grace period when NodeDisconnectHandler is created, not at
  class load, so tests can set it
- don't track reconnect attempts for the local node
- add Javadoc (@SInCE 4.0) to the new public methods in
  NodeDisconnectHandler

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster nodes never reconnect after a short connection loss

1 participant