Observe 3rd Party validators are signing at the tip in the relayer #1762
Comments
|
@tkporter curious to get your thoughts |
|
@tkporter friendly ping :) |
|
Another friendly ping @tkporter :) |
|
i don't have any satisfactory ideas that I can come up with off the top of my head but will think about this more I'm a little hesitant to put this in the scraper, it doesn't feel too much more natural to put this there than in the relayer If we're happy with only monitoring the default ISM's validator set then it sort of works. But I don't think there's anything satisfactory that works for all validators. I'd be up for having a metrics loop that tries to fetch the latest signed checkpoint for the set relating to the default ISM every 30s or minute or so Feels like this could live in the relayer or scraper. and would probably make more sense for PI chains to live in the relayer, given not everyone will be operating a scraper and these metrics are probably useful |
|
i guess you could actually enumerate the validators on ValidatorAnnounce, but agree that defaultISM makes more sense, and i could see it be useful in the relayer |
nambrot
changed the title
|
@tkporter friendly ping |
avious00
changed the title
avious00
changed the title
avious00
changed the title
|
Thinking the move may be to monitor validators that are enrolled on:
Some interesting things:
Imo the most important thing to track to begin with are validator latest checkpoint indices.
Proposed metric: Name:
I’m open to just having the So to be clear we'd just update the metrics when fetching metadata for validators that are part of the default ISM or for a specially configured app. This also means that if there are no messages, then the metrics won't be updated because this is tapping into the metadata building logic |
### Description
Goal of this was to have insight into validators of important sets being
"up"
Introduces a new metric used by relayers:
`hyperlane_observed_validator_latest_index`, e.g.:
```
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 670
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 665
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 652
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664
hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 658
hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641
```
Tapping into metadata building for multisig ISMs, the relayer will
update the metric with the latest indices for the validators in a set.
In order to prevent the cardinality being ridiculously high, only
certain validator sets are tracked. This is done by introducing an
`app_context` label (I'm very open to other names here, for some reason
whenever idk how to name some kind of identifier I end up calling it a
context 😆)
The app context can either be:
- if a new setting, --metricAppContexts, is specified, a message will be
classified based off the first matching list it matches. E.g.
`--metricAppContexts '[{"name": "testapp", "matchingList":
[{"recipient_address": "0xd84379ceae14aa33c123af12424a37803f885889",
"destination_domain": 13371 }] }]'`. This is nice for e.g. warp route
deployments, where the ISM is maybe not a default ISM, and can be
changed
- if a message doesn't get classified this way, it can also be
classified with the "default_ism" app context, which is just for any
message that happens to use the default ISM as its "root" ISM
This way we have insight in to the default ISM and any
application-specific ISMs.
Some things to note:
- it's possible for a message to actually have more than one validator
set, e.g. if it's using an aggregation ISM. In this case, we'll have
metrics on the union of all validator sets for that app context
- Some effort is required to make sure that metrics don't stick around
for a validator that has actually been removed from the set. To handle
this, we cache the validator set for an app context and clear out the
entire set each time we set the metrics
### Drive-by changes
- Zod's nonempty function for strings is deprecated, moves to `.min(1)`
instead
### Related issues
- Fixes #1762
### Backward compatibility
yes
### Testing
Ran locally - I think i'll probably add something in e2e tests, but
opening now
Compared to v1, the relayer has no practical metric anymore to assess whether validators are signing "at the tip". We should consider adding this functionality to the scraper
Indexing in relayers v2 for validator signature. This will also show up for monitoring and alerting in Grafana.
Tasks
The text was updated successfully, but these errors were encountered: