Validator observability infra #3116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 6966645 The changes in this PR will be included in the next version bump. This PR includes changesets to release 6 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
yorhodes
approved these changes
Jan 4, 2024
|
Should this be merged? It's already live right? |
jmrossy
reviewed
Jan 11, 2024
… trevor/validator-observability-infra
…cus-network/abacus-monorepo into trevor/validator-observability-infra
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #3116 +/- ##
=======================================
Coverage 67.18% 67.18%
=======================================
Files 101 101
Lines 1021 1021
Branches 106 106
=======================================
Hits 686 686
Misses 291 291
Partials 44 44
|
ltyu
pushed a commit
to ltyu/hyperlane-monorepo
that referenced
this pull request
Mar 13, 2024
### Description - Configures the relayers to start tracking the latest checkpoints of the manta TIA WR, arbitrum TIA WR, and helloworld sets (and, implicitly, the default ISM sets too) - deployed all agent roles & networks with the new image - created alerts and added to the nexus dashboard One thing that's a bit of a bummer is the metrics are only updated whenever we try to construct metadata for a message. Naturally the validators will poll & sign checkpoints slightly out of sync, so pretty frequently the relayer attempts to deliver a message where say 4/6 signatures are needed, and 2 of the validators happen to just have not polled & signed the checkpoint yet. The message is then successfully delivered, and the 2 remaining validators probably sign the checkpoint in a matter of seconds, but the metrics aren't updated to reflect this unless there's another message whose metadata is being constructed. This just means that the graph is a bit ugly when things are working. But when things aren't working, we'll still be able to clearly see which validators are behind. We may want to consider changing the metrics if this proves confusing. Some ideas: 1. Move to a separate task that occasionally polls the latest checkpoints of the configured app contexts 2. Also track the threshold. This way we can construct alerts & dashboards based off the threshold and not be overly concerned if there are some stragglers ### Drive-by changes added `bytesToBytes32` so that we can construct matching lists, which the agents expect to be 0x-prefixed, using a router address config that may include protocol-specific address formats ### Related issues Fixes hyperlane-xyz#3109 ### Backward compatibility yes ### Testing deployed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
One thing that's a bit of a bummer is the metrics are only updated whenever we try to construct metadata for a message. Naturally the validators will poll & sign checkpoints slightly out of sync, so pretty frequently the relayer attempts to deliver a message where say 4/6 signatures are needed, and 2 of the validators happen to just have not polled & signed the checkpoint yet. The message is then successfully delivered, and the 2 remaining validators probably sign the checkpoint in a matter of seconds, but the metrics aren't updated to reflect this unless there's another message whose metadata is being constructed. This just means that the graph is a bit ugly when things are working. But when things aren't working, we'll still be able to clearly see which validators are behind.
We may want to consider changing the metrics if this proves confusing. Some ideas:
Drive-by changes
added
bytesToBytes32so that we can construct matching lists, which the agents expect to be 0x-prefixed, using a router address config that may include protocol-specific address formatsRelated issues
Fixes #3109
Backward compatibility
yes
Testing
deployed