Improve eth_getBlockByNumber query performance.

[i-norden]

This API endpoint is a wrapper around the EthModule.EthGetBlockByNumber method. This method calls parseBlkParam which is where I believe the problem occurs. When we pass in a blocknumber, that methods falls through to this switch statement, which calls ChainGetTipSetByHeight which calls GetTipsetByHeight. Importantly, it appears that no matter how far back we are looking for a tipset it passes the head (heaviest) tipset to these methods as the starting point for a backwards walk. Inside the GetTipsetByHeight method we call "ChainIndex".GetTipSetByHeight, but this method doesn't appear to actually use an epoch => tipset index, rather it is a hot cache wrapped around a very slow iterative search backwards through every tipset starting from the head until we find the tipset with the desired epoch: https://github.com/filecoin-project/lotus/blob/master/chain/store/index.go#L103.

[i-norden]

At a high level I think we can boil down the issues underpinning this and #19 to three missing indexes:

height => tipsetkey (this could exist in mesgindex.go, could index the 'epoch' field and use this table in that manner)
message_cid => tipsetkey (this exists in msgindex.go (sqlite messages table) index, but perhaps we could provide better guarantees that the index is populated? or maybe it is solely user error that is causing this index to be incorrectly populated)
message_cid => receipt_cid (for #19)

[i-norden]

epoch => tipsetkey and message_cid => tipsetkey indexes already exist in the msgindex.go sqlite index. The former does not appear to be leveraged in the search path described in this issue, and while the the latter is used by the search path described in #19 it seems index misses are frequent enough to cause the backwards walk approach to be taken often. So for the former case we could begin to leverage this index, and in both cases we need to have better guarantees that the index will populated and hit.

Naively, I envision we could also add to this table a receipt_cid field for the message_cid => receipt_cid index but I very likely am overlooking a reason why this would either be difficult to populate or maintain (due to replaceable messages).

[fridrik01]

The skiplist implementation introduced in this PR improved the ChainIndex.GetTipsetByHeight method to be "1000x faster". However, I think its slow the first time since it needs to hot-load the cache (which is not ideal).

But, with this skiplist all in cache, does that address this eth_getBlockByNumber query performance, or is it still a problem?

[fridrik01]

Relevant to filecoin-project/lotus#10807

[i-norden]

Thanks @fridrik01, warming up the cache did work (once we were running the right version of lotus 🙃). We are going to test the PR that warms-up on node standup too, haven't done that yet.

I still think using the msgindex.db table to go from epoch=>tipsetkey could bypass needing the cache entirely and that that would be preferable in cases where we already have msgindex.db enabled.

Also running into some general syncing issues, described in greater detail here which are limiting us from testing historical msg and receipt stuff.