feat(relayer): Relayer indexer/processor separation and refactor, messaging queue

[cyberhorsey]

Closes #14528

this PR updates the relayer codebase to be inline with the taiko-client architecture, which should make it easier to onboard new people, as well as cross-share work more - once they get familiar with one codebase, they are familiar with all of our Golang applications.

It splits the relayer into two separate apps, the indexer and the processor, that now communicate asynchronosly through a Queue interface, the first implementation of which is a RabbitMQ queue (attached picture, processor is on the left reading from the queue, indexer is on the right listenign to the blockchain and adding to the queue).

This has three main advantages:

1. The indexer will be caught up to the tip of the chain much quicker, allowing the bridge-ui to be up-to-date.
2. The persistence of the queue will allow us to more reliably know which messages have been processed, how many behind we are, and which ones failed to be processed.
3. We can spin up as many additional processors as we want, without affecting the indexing, if we see many messages coming in that one processor can not handle.

The main difference is all of the CLI args of the relayer can still be passed in through Environment Variables as well, for easier deployment/configuration.

Latest code is already deployed (matching k8s-config PR is open) to internal-devnet for L1-L2, L2-L1, L2-L3, L3-L2, as well as a RabbitMQ statefulset.

TODO:

• tests for optional flag vars, current only tests requirred (can come in another PR, perhaps a task roger can do to get familiar with this Go codebase as well)
  

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
bridge-ui-v2-internal	❌ Failed (Inspect)			Sep 5, 2023 4:02pm

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
bridge-ui-v2	⬜️ Ignored (Inspect)	Visit Preview		Sep 5, 2023 4:02pm

[dantaik]

@cyberhorsey does this PR address this issue ?

[cyberhorsey]

@cyberhorsey does this PR address this issue ?

Yes, but it's only started today and will be maybe two more days.