Hermes submits a high number of redundant packets on Stride

[ancazamfir]

Summary of Bug

[Huge thanks to @romac @ljoss17 and @colin-axner for the help with the investigation]
When relaying on stride the relayers relay an unusual number of redundant ack and recv packets. After our analysis on the transactions, hermes logs and full node logs on thing we found is that stride chain does not emit events for the redundant relays.
When configured with tx_confirmation = true, hermes looks for the events associated with the packet relayed, write_acknowledgement for recv and acknowledge_packet for ack. If it does not find the processing events, it retries a number of times before re-evaluating the original, send_packet for recv and write_acknowledgment for ack.

We confirmed with @colin-axner that in newer ibc-go versions the events for redundant relays are not emitted for redundant relays. He provided these details:

• versions where we start to see this behaviour:
  • v5.2+
  • v6.1+
  • v7+
• the diffs for the change can be seen here cosmos/ibc-go@c912fd9#diff-b225d8b78960f1ed19ccabcbdcda5ff708075790adac8463fbb0a83a8a63b78aL399

Update
@romac has written tests that show that on ibc-go v4.x:

• events ARE emitted for redundant acks
• events ARE NOT emitted for redundant recv packets, so we need to investigate why we do not have a higher number of redundant recv on other chains using v4.

Version

all

Steps to Reproduce

Acceptance Criteria

The number of uneffected packets sent by Hermes on Stride should be comparable with other chains
We should also:

• if possible add integration test for this case once we understand the root cause and include in "Steps to reproduce"
• check uneffected packets for chains other than stride on the ibc-go versions mentioned above

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

An update on this, it seems that at least for Hermes the lack of events does not explain the high number of uneffected packets as we re-evaluate immediately without retrying.
Next steps are to gain access to stride and osmosis full nodes, get funds and run (potentially instrumented) hermes relayer ourselves.
Also in parallel waiting on full node and hermes logs from hermes operators to see if we get some hints on what happens.

[ancazamfir]

Probably caused by Stride-Labs/stride#807

[ancazamfir]

Probably caused by Stride-Labs/stride#807

And fixed in Stride-Labs/stride#882

[romac]

Re-opening until we can confirm once Stride v13 rolls out

[seanchen1991]

@romac Good to close this now? 🙂