Blockstore on all dagstore cids

[aarshkshah1992]

For IPFS <-> Filecoin interop.

A blockstore that can serve retrieval for any cid across all shards that the dagstore has.

TODO

• Add fuzzed test.
• GetSize should use the underlying blockstore and not load the entire block in memory.
• When a blockstore is removed from the cache it is closed. Use reference counting to ensure that the blockstore is not closed while there is still an outstanding operation against the blockstore

[raulk]

• How and when are shards released?
• Suggest naming as IndexBackedBlockstore.
• If possible, put in a different package (ex. indexbs?) as this is a utility and strictly not needed for the operation of the DagStore.
• How is the LRU cache configured?

[raulk]

Is there a reason that this needs to be a member of the DagStore?

[aarshkshah1992]

@raulk

• Have renamed it as IndexBackedBlockstore in a new indexbs package.
• Yeah, I think it dosen't need to be a member of dagstore. Have moved it outside.
• Shards are released when they are evicted from the lru blockstore cache. See

  dagstore/indexbs/indexbacked_bs.go

  Line 44 in 8d9286a

  bslru, err := lru.NewWithEvict(maxCacheSize, func(_ interface{}, val interface{}) {

  .

[aarshkshah1992]

@raulk @dirkmc

There was a race here where two Get calls for the same shard key create two shard accesors and only one gets added to the lru cache and eventually relased whereas we lose the reference to the other one without ever releasing it (because multiple add calls to the lru cache for the same key only updates the priority of the item).

I've fixed this with striped locking based on shard key to synchornize access to the lru cache.

[aarshkshah1992]

@raulk Please can you take a look ?

[MichaelMure]

May I ask what's the status of this work?

[Wondertan]

Hello. We would like to use DAGStore together with a top-level index. We also need a Blockstore over the top-level index to serve sampling requests to our light clients, so we would like to have this merged not to reinvent the wheel:

• What else needs to be done here to say it's done?
• Can we somehow facilitate the process?

Kindly ping @raulk, @dirkmc, and @aarshkshah1992. I know you are busy with FVM and Boost things, but this should not take too much time from you. Thanks!

[willscott]

@hannahhoward probably has the most recent context here, and I think we are hoping to finish the work needed to get this finished.

[dirkmc]

@Wondertan @MichaelMure we are hoping to get it merged this week

[aarshkshah1992]

lgtm