Hermes telemetry requirements

[thanethomson]

Crate

ibc and ibc-telemetry

Summary

Following from #868 and recent discussions, we need to expand on the metrics we expose via our telemetry mechanism.

Problem Definition

We have been talking about parsing log files to obtain information regarding metrics that could easily be tracked by our telemetry system (i.e. Prometheus).

Proposal

Let's:

1. Use this issue to discuss and gather all of the metrics that would be useful for us to expose.
2. List them in the Acceptance Critera section of this issue.
3. Prioritize the metrics.
4. Implement as many of them as we can according to priority. Ideally only one metric per PR to ensure quick review.

Acceptance Criteria

When the following metrics are exposed:

Done	Metric name	Description	Meter type
	error_websocket_reconnect	Number of times Hermes had to reconnect to the WebSocket endpoint	Counter
	time_to_relay	Interval between when Hermes receives an event until the messages in the event are sent to the destination chain, per message	ValueRecorder
	tx_failed	Total number of failed txs processed	Counter
	tx_retry_failed	Total number of failed retries when submitting txs	Counter
	trusting_periods_left	How many trusting_periods are left (metrics on whether an update is required)	UpDownCounter
	relay_chains_num	Number of chains the relay is connecting to	UpDownCounter
	rpc_query_count	Number of RPC queries issued to chain	Counter
	rpc_query_latency	Latency of RPC queries issued to chain	ValueRecorder
	grpc_query_count	Number of gRPC queries issued to chain	Counter
	grpc_query_latency	Latency of gRPC queries issued to chain	ValueRecorder
✅	tx_msg_ibc_recv_packet	Total number of IBC packets received	Counter
✅	tx_msg_ibc_acknowledge_packet	Total number of IBC packets acknowledged	Counter
✅	ibc_timeout_packet	Total number of IBC timeout packets	Counter
✅	ibc_client_misbehaviour	Total number of client misbehaviours	Counter
	tx_successful	Total number of successful txs processed	Counter
	tx_count	Total number of txs processed	Counter
	error_rpc	Number of RPC errors encountered	Counter
	ibc_transfer_send	Total number of IBC transfers sent from a chain (source or sink)	Counter
	ibc_transfer_receive	Total number of IBC transfers received to a chain (source or sink)	Counter
	relay_clients_num	Number of clients the relay is connecting to	UpDownCounter
	relay_connects_num	Number of connections the relay is connecting to	Counter
	relay_channels_num	Number of channels the relay is connecting to	Counter
	UP	Program exit flag	ValueRecorder

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Closing in favor of #2112 which will take a more fine-grained, incremental approach to adding metrics.