feat(relayer): add core metric for the highest seen index in tree

[aroralanuk]

Description

• Instead of using the latest leaf inserted into the tree which is slower, we retrieve the highest index of leaf currently stored in the hyperlane_db while we're syncing.
• The syncer task is in a JoinHandle with an index_updater task, which interrupts the syncer every 5 seconds to update if the syncer saw a new higher leaf index. This is because the syncer is a continuously running process and this is the easiest way to check progress in between.
• Renamed latest_leaf_index to latest_tree_insertion for clarity.

Drive-by changes

None

Related issues

• fixes https://github.com/hyperlane-xyz/issues/issues/1396

Backward compatibility

Yes

Testing

invariant in e2e and manually testing

[changeset-bot]

⚠️ No Changeset found

Latest commit: b6a5a92

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.53%. Comparing base (2d4963c) to head (b6a5a92).
Report is 24 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5151   +/-   ##
=======================================
  Coverage   77.53%   77.53%           
=======================================
  Files         103      103           
  Lines        2110     2110           
  Branches      190      190           
=======================================
  Hits         1636     1636           
  Misses        453      453           
  Partials       21       21           

Components	Coverage Δ
core	87.80% <ø> (ø)
hooks	79.39% <ø> (ø)
isms	83.68% <ø> (ø)
token	91.27% <ø> (ø)
middlewares	79.80% <ø> (ø)

[tkporter]

imo creating a new task just to update this metric & using the DB as an intermediary for this info is unnatural

I was thinking instead of having this be super merkle-tree specific, this would be more of a generic sequence number that's tracked when sequence-aware indexing

And in poking around where I'd imagine this would live, it turns out that Mantas actually added exactly this metric just a couple weeks ago 😵‍💫 https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/main/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs#L456-L459

https://abacusworks.grafana.net/goto/t9kdDYDNg?orgId=1

So given we already have this metric, let's just use the one that was already added hyperlane_cursor_max_sequence for our alert. Can you have this PR just be to rename latest_leaf_index_gauge -> latest_tree_insertion_gauge?

Sorry for sending you down this path - I wasn't aware of this change

[tkporter]

Looking into it though, the metrics are currently updated only when the cursor's update fn is called

hyperlane-monorepo/rust/main/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs

Line 504 in fa3ef92

self.update_metrics().await;

We only hit this code path if we actually index something, so it means upon startup we don't set the metric until we actually index something. e.g. we haven't had any usage on zksync for a long time so there's no hyperlane_cursor_max_sequence metric for zksync

Can you change this so that we call update_metrics() from get_next_range like around here?

hyperlane-monorepo/rust/main/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs

Line 129 in fa3ef92

let (Some(onchain_sequence_count), tip) = self

And can you also make it so that the update_metrics fn doesn't do let max_sequence = self.target_sequence().await as i64; itself, and instead we pass it in from get_next_range?

[aroralanuk]

for some reason, the sealevel changes are showing up here so I'm closing this issue and making the changes here: #5178