sentry: add exception handling for main tasks #2032
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces some additional exception handling in the following modules of
sentry
:Sentry
main tasks (i.e.StatusManager
,Discovery
...): these are long-running tasks which usually should never terminate (unless the user requests a graceful exit), so here we just catch any system exception to log a critical error and relaunch it. As a result, we will see what is the root cause of the error forcing the program to exit, currently it's just a somewhat obscureco_await: operation cancelled
errorrlpx::Peer
: during some tests I've found that sometimes astd::nested_exception
gets raised, which we can handle properly. Moreover, I've seenPeer::post_message
rarely failing to spawnPeer::send_message
, thus causing the program to terminate, so I've added a warning to at least detect this situation (it seems to me like an unavoidable race between peer close and message spawing in different sub-tasks) and avoid program terminationrlpx:: Server
: again, sometimesasync_accept
raises aboost::system::system_error
with error codeboost::system::errc::invalid_argument
, which we need to understand but seems to self-heal in some way