Repository navigation
Conversation
A dropped node connection removed both nodes from the cluster right away, and nothing tried to reconnect. A lost connection now enters the reconnect grace period if cloudnet.max.node.disconnect.millis is greater than 0. The default (0) keeps the old behaviour. - only one reconnect attempt per node at a time - remove nodes stuck in syncing once the grace period expires - send the startup time on authorization, so a restart during the grace period is treated as a new node - remote service providers look up the current node channel on each call instead of keeping the one from registration - apply the reconnecting node's services during service sync - flush queued packets and re-publish local services after a plain authorization - re-select the head node when a node becomes ready again - read the grace period when NodeDisconnectHandler is created, not at class load, so tests can set it - don't track reconnect attempts for the local node - add Javadoc (@SInCE 4.0) to the new public methods in NodeDisconnectHandler
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When the connection between two nodes dropped, even for a moment, both nodes removed each
other from the cluster immediately and never tried to reconnect. The cluster stayed split
until a node was restarted, and each node continued as its own head node. This happens
more often with nodes in different datacenters and with more nodes in the cluster.
The reconnect grace period (
cloudnet.max.node.disconnect.millis) only applied to nodesthat stopped sending updates while their socket stayed open. When the socket itself was
closed, the channel close handlers called
NodeServer#close()directly. The reconnecthandshake had also never really been used, and it had several problems once it was.
Modification
cloudnet.max.node.disconnect.millisis greater than0, instead of removing the nodedirectly. The default of
0keeps the previous behaviour.connections can break the reconnect handshake. Nodes stuck in the syncing state after
an interrupted reconnect are removed once the grace period expires.
restart during the grace period and let the node join as a newly started node, instead
of reconnecting it into its old session (which failed with an NullPointerException in
AuthorizationResponsePacketListener). Nodes on older versions don't send it, andolder nodes ignore it, so mixed versions keep working.
instead of keeping the channel that was used during registration. That channel is
closed after a reconnect.
Before, they were skipped because the node wasn't
available()yet.services are re-published to the other node.
the same head node.
NodeDisconnectHandleris created, so it can be set intests.
NodeDisconnectHandler,NodeDisconnectTrackerTask,AuthorizationPacketListenerandAuthorizationResponsePacketListener.launcher.cnldescription ofcloudnet.max.node.disconnect.millis.Result
With a grace period configured, nodes reconnect after short network interruptions and
agree on the same head node afterwards. A node that restarts within the grace period
joins cleanly as a new node. If a node does not come back within the grace period, it is
removed like before. With the default setting, behaviour is unchanged.
Tested manually with two nodes (grace period 30s/60s):
ss -K): before, both nodes dropped each other and neverreconnected. Now both are
READYagain within a few seconds, with the same head node.now matches on both nodes (before, each node showed itself as head).
the node joins as a new node without errors.
0: same behaviour as before.Other context
Known limitations:
node is restarted, like before.
DefaultCloudServiceManagerare not covered by unittests, as the class needs most of the node to be constructed.
Tests:
https://hastebin.baxnetzwerk.de/ct-Z4gFrfyk
With
cloudnet.max.node.disconnect.millisset to 60.000:https://hastebin.baxnetzwerk.de/dzvj7KguKXk
With
cloudnet.max.node.disconnect.millisset to 0 or default:https://hastebin.baxnetzwerk.de/Bz+rvVdSEei
(the links are available for 1 year.)
Fixes #1897