[Core] Improve connection termination on shutdown #2614
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 3, 2025
Closed
AhmedSoliman
changed the title
AhmedSoliman
force-pushed
the
pr2614
branch
2 times, most recently
from
4db47c7 to
df1cfe2
Compare
tillrohrmann
approved these changes
Feb 4, 2025
| @@ -632,41 +642,43 @@ where | |||
| drop(connection); | |||
|
|
|||
| let drain_start = std::time::Instant::now(); | |||
| trace!("Draining connection"); | |||
| debug!("Draining connection"); | |||
There was a problem hiding this comment.
Should this be part of the if branch?
| PeerMetadataVersion::from(header), | ||
| if needs_drain { | ||
| // Draining of incoming queue | ||
| while let Some(Some(Ok(msg))) = incoming.next().now_or_never() { |
There was a problem hiding this comment.
The ConnectionAwareRpcRouter on the sending end of this stream is keeping it open because it can hold the sender for this stream in a future it created and which is being awaited on the other end of the connection. Did I understand it correctly? Must have been quite hard to track this down. Respect.
AhmedSoliman
force-pushed
the
pr2614
branch
2 times, most recently
from
ed7040c to
69a7ccd
Compare
Fixes: - On graceful shutdown, we had a long-standing bug where draining of some connections can get stuck due to the connection-aware rpc router holding owned senders in their futures. This was addressed by not blocking on the receive stream on drain, we'll only process the messages we have received after we sent the shutdown signal. Note that connections that have been terminated by the peer will also skip the drain since we don't want to process further messages from them. - On graceful shutdown, we had a bug that peers would have ignored the Control Frame holding the shutting down signal since those messages have no header. This is now fixed, this will impact a future PR to mark this node generation as `Gone` to avoid reconnects - On system shutdown, we first stop cluster controller to: - Make sure it doesn't react to our own partial/complete loss of connectivity during shutdown - To avoid any competition with other controllers that might become leader during shutdown of this node - We now drain connections first and stop socket handlers gracefully before we continue the shutdown to give the best chance for the shutdown control frame to be sent to peers. This should make other controllers and parts of the system realise that this node is `gone` as early as possible to improve failover time (MTTR) - Minor logging changes ``` // intentionally empty ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes:
Goneto avoid reconnectsgoneas early as possible to improve failover time (MTTR)Stack created with Sapling. Best reviewed with ReviewStack.