[Bifrost] Improve logging of seal failures #2621
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 4, 2025
Closed
AhmedSoliman
force-pushed
the
pr2621
branch
2 times, most recently
from
6cb2379 to
fd56ace
Compare
Test Results 7 files ±0 7 suites ±0 2m 51s ⏱️ - 1m 37s Results for commit a5c9ef7. ± Comparison against base commit d641697. This pull request removes 2 tests.♻️ This comment has been updated with latest results. |
tillrohrmann
approved these changes
Feb 5, 2025
AhmedSoliman
force-pushed
the
pr2621
branch
2 times, most recently
from
75f84c1 to
0e591e7
Compare
Fixes: - On graceful shutdown, we had a long-standing bug where draining of some connections can get stuck due to the connection-aware rpc router holding owned senders in their futures. This was addressed by not blocking on the receive stream on drain, we'll only process the messages we have received after we sent the shutdown signal. Note that connections that have been terminated by the peer will also skip the drain since we don't want to process further messages from them. - On graceful shutdown, we had a bug that peers would have ignored the Control Frame holding the shutting down signal since those messages have no header. This is now fixed, this will impact a future PR to mark this node generation as `Gone` to avoid reconnects - On system shutdown, we first stop cluster controller to: - Make sure it doesn't react to our own partial/complete loss of connectivity during shutdown - To avoid any competition with other controllers that might become leader during shutdown of this node - We now drain connections first and stop socket handlers gracefully before we continue the shutdown to give the best chance for the shutdown control frame to be sent to peers. This should make other controllers and parts of the system realise that this node is `gone` as early as possible to improve failover time (MTTR) - Minor logging changes ``` // intentionally empty ```
- RepairTail bugs fixed + restatectl's digest command now tolerates a failed node - RepairTail improved logging to explain what's happened - Inner retries removed, most retries are now on the out layers (tbd if more inner retries need to be removed). Note that this causes some of the outer operations to fail more often than before. This will be evaluated as we test and fixed on the higher level as needed - Minor logging fixes ``` // intentionally empty ```
Additionally, loglet now has `id()` and `provider()` and `LogletWrapper` will print nice Debug value when used in tracing. ``` // intentionally empty ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additionally, loglet now has
id()andprovider()andLogletWrapperwill print nice Debug value when used in tracing.Stack created with Sapling. Best reviewed with ReviewStack.