Remove telemetry from gasKV and cacheKV store operation's get methods

[ValarDragon]

Summary

The current telemetry work in GasKV and CacheKV store get operations causes significant overheads, antithetical to their purpose.

These telemetry overheads are (seemingly) for the use case of find all I/O time. However the approach being used here does not make sense to me (its effectively measuring 'time spent in getting from cache', not 'time from underlying store'), and is actually causing the overhead that its measuring!

If these are viewed as necessary, then they should only happen if the node has telemetry enabled.

Problem Definition

In our osmosis epoch time benchmarks, where the CacheKV store's cache size is huge, at least a third of the CacheKV store's Get function is spent in telemetry related operations. However per my (back of the envelope) estimate, our hashmap get operation time should probably be at least double the time of normal CacheKV operations used in the SDK.

However this telemetry overhead is then paid again in the gas kv store that is calling the cache kv store. (As they both do the same telemetry operation on each get)

This means that most of the time in fetching data from a catch (even in the case of a miss) is spent in telemetry.

Proposal

I don't view time spent in caches as the helpful telemetry operation. I think telemetry for getting cache timings is also bad, as it ruins lots of hardware cache efficiency work going on here. If you wanted telemetry for your caches, you should be doing this at a higher level (e.g. overall time with and without the cache). imo you should not be benchmarking low level things like this, as it artificially screws up things like stack/registers, cache lines, which are important for data caches.

• Remove the GasKVStore get telemetry operation
• Remove the CacheKVStore get telemetry operation

The alternative take would be to 'gate' these defers to only be valid when telemetry is enabled, and make that propogate throughout the state machine. I think these telemetry ops should be removed personally.

pprof source timings for Osmosis epoch benchmarks' very heavy usage of cacheKVStore. (My understanding is that the runtime.NewObject comes from the defer? But I don't actually know that at all)

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned