Add telemetry to hermes - bootstrap

[ancazamfir]

Crate

relayer

Summary

From DEX requirement list: "metrics to monitor the health of the relayer, exposed through business-standard frameworks like Prometheus."

Problem Definition

Proposal

Investigate existing rust crates (e.g. https://crates.io/crates/tracing-opentelemetry, opentelemetry-prometheus). Add telemetry metrics for:

• packet relaying (success or errors)
• retries
• RPC errors

WIP more detail to be added

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[andynog]

Some metrics that needs to be tracked (as per meeting with dex team)

METRIC	DESCRIPTION	UNIT
tx_count	Total number of txs processed via Relay tx	counter
tx_successful	Total number of successful txs processed via Relay tx	counter
tx_failed	Total number of failed txs processed via Relay tx	counter
UP	Program Exit Flag	gauge
ibc_transfer_send	Total number of IBC transfers sent from a chain (source or sink)	counter
ibc_transfer_receive	Total number of IBC transfers received to a chain (source or sink)	counter
tx_msg_ibc_recv_packet	Total number of IBC packets received	counter
tx_msg_ibc_acknowledge_packet	Total number of IBC packets acknowledged	counter
ibc_timeout_packet	Total number of IBC timeout packets	counter
ibc_client_misbehaviour	Total number of client misbehaviours	counter
relay_chains_num	Number of chains the relay is connecting to	counter
relay_clients_num	Number of clients the relay is connecting to	counter
relay_connects_num	Number of connects the relay is connecting to	counter
relay_channels_num	Number of channels the relay is connecting to	counter
relay_rpcconnect_num	Broken RPC Connection	counter
relay_rpcdelay_time	Delay rate for RPC connections specified in config	ms
relay_txcompleted_time	The rate at which tx is completed from chain A to B	ms
relay_trustingperiods_time	How many trusting_periods are left(Metrics on whether an update is required)	ms

[andynog]

Started to implement this. I believe adding support the telemetry using the Supervisor might be the best option afaiu. But I will check this with people who are more familiar with the Supervisor if this is the right direction.