[Core] Avoid reconnection if the node is Gone #2619
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 4, 2025
Closed
Test Results 7 files ±0 7 suites ±0 3m 34s ⏱️ -54s Results for commit b446ccc. ± Comparison against base commit d641697. This pull request removes 2 tests.♻️ This comment has been updated with latest results. |
AhmedSoliman
force-pushed
the
pr2619
branch
2 times, most recently
from
255c962 to
ebc159e
Compare
tillrohrmann
reviewed
Feb 4, 2025
There was a problem hiding this comment.
Thanks for preventing connection attempts to gone nodes @AhmedSoliman. Modulo one comment about a condition the changes look good to me.
AhmedSoliman
force-pushed
the
pr2619
branch
2 times, most recently
from
770b6cb to
8fe1dcd
Compare
tillrohrmann
approved these changes
Feb 5, 2025
| .lock() | ||
| .observed_generations | ||
| .get(&node_id.as_plain()) | ||
| .map(|status| node_id.generation() <= status.generation && status.gone) |
There was a problem hiding this comment.
Could we make this filter even a bit more selective by node_id.generation() < status.generation || node_id.generation() == status.generation && status.gone?
AhmedSoliman
force-pushed
the
pr2619
branch
2 times, most recently
from
7f86d7e to
e005bdf
Compare
Fixes: - On graceful shutdown, we had a long-standing bug where draining of some connections can get stuck due to the connection-aware rpc router holding owned senders in their futures. This was addressed by not blocking on the receive stream on drain, we'll only process the messages we have received after we sent the shutdown signal. Note that connections that have been terminated by the peer will also skip the drain since we don't want to process further messages from them. - On graceful shutdown, we had a bug that peers would have ignored the Control Frame holding the shutting down signal since those messages have no header. This is now fixed, this will impact a future PR to mark this node generation as `Gone` to avoid reconnects - On system shutdown, we first stop cluster controller to: - Make sure it doesn't react to our own partial/complete loss of connectivity during shutdown - To avoid any competition with other controllers that might become leader during shutdown of this node - We now drain connections first and stop socket handlers gracefully before we continue the shutdown to give the best chance for the shutdown control frame to be sent to peers. This should make other controllers and parts of the system realise that this node is `gone` as early as possible to improve failover time (MTTR) - Minor logging changes ``` // intentionally empty ```
- RepairTail bugs fixed + restatectl's digest command now tolerates a failed node - RepairTail improved logging to explain what's happened - Inner retries removed, most retries are now on the out layers (tbd if more inner retries need to be removed). Note that this causes some of the outer operations to fail more often than before. This will be evaluated as we test and fixed on the higher level as needed - Minor logging fixes ``` // intentionally empty ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack created with Sapling. Best reviewed with ReviewStack.