feat: add simulate errors metrics

[freak12techno]

Closes: #3845

Description

Adds a simulate_metric_total indicating when a transaction simulation fails, with the following labels: recoverable (can the execution continue if this happened?), account and error description.

I have a few of these in my logs:

Feb 10 17:24:06 monitoring-logging hermes[952969]: 2024-02-10T16:24:06.446342Z  WARN ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}: task encountered ignorable error: link error: link failed with underlying error: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "spendable balance 2950583uosmo is smaller than 3000001uosmo: insufficient funds: insufficient funds [osmosis-labs/osmosis/v22/x/txfees/keeper/feedecorator.go:294] With gas wanted: '300000000' and gas used: '54519' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "13726883"} }

and with these, this is something how this metric looks like locally:

# HELP simulate_errors_total Number of errors observed by Hermes when simulating a Tx
# TYPE simulate_errors_total counter
simulate_errors_total{account="osmo17ndx5qfku28ymxgmq6zq4a6d02dvpfjjul0hyh",error_description="Unknown error",recoverable="false",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 4

---

PR author checklist:

• Added changelog entry, using unclog.
• Added tests: integration (for Hermes) or unit/mock tests (for modules).
• Linked to GitHub issue.
• Updated code comments and documentation (e.g., docs/).
• Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

• Reviewed Files changed in the GitHub PR explorer.
• Manually tested (in case integration/unit/mock tests are absent).

[freak12techno]

this one is to prevent error_descriptions like gRPC call send_tx_simulate failed with status: status: Unknown, message: "spendable balance 2950583uosmo is smaller than 3000001uosmo: insufficient funds: insufficient funds [osmosis-labs/osmosis/v22/x/txfees/keeper/feedecorator.go:294] With gas wanted: '300000000' and gas used: '529599' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "13728350"} }, and to kinda group them by error message, not sure if that can be done in a better way, please let me know if so

[freak12techno]

also I am not sure if there's a way to get the code of the underlying gRPC error as they are all Unknown, if there is, I guess it would be better to use this one (and its text description), but not sure if it is possible

[freak12techno]

@romac can you take a look? not sure who to ask for a review here as this is my first PR towards this repo

[romac]

Wow, that was quick, thank you! Will take a look next week, but looks good already at first glance :)

[ljoss17]

I think as long as gRPC is used fo the simulation, Hermes will get tonic’s Status error which won’t contain the SDK/ibc-go error code, so it loses some error information. I don’t see how to improve the labels for the simulation errors.

This is already a great indicator to spot if something is not working properly, thank you!!

I’m not sure if it would be possible to use the RPC to simulate the Tx, thus getting a Tendermint's Response which would allow using the same type as for the broadcast error metric, any thought on this @romac ?

[romac]

I’m not sure if it would be possible to use the RPC to simulate the Tx, thus getting a Tendermint's Response which would allow using the same type as for the broadcast error metric, any thought on this @romac ?

As far as I know, tx simulation is a feature of the SDK, not Comet/Tendermint, and is only available as a gRPC call. We could issue the very same gRPC call over RPC via ABCI but that wouldn't change the response format we get.

[freak12techno]

@ljoss17 anything to fix here or it's good as it is?

[ljoss17]

Great work, thank you!!
Some small nit in the changelog, and could you move the changelog to features/ibc-telemetry

[freak12techno]

@ljoss17 done

[ljoss17]

Great work!!

[romac]

@freak12techno Thanks so much for the PR! 🙏