Executing a selector against a DagService

[ribasushi]

I found myself needing to copy this code nearly-verbatim to another project. Perhaps worth cleaning up the interface and packaging it somewhere in go-ipld-prime?

https://github.com/filecoin-project/lotus/blob/3598eff5d4361dbf03d65fa2f1431b23188b5d76/markets/utils/selectors.go#L23-L29

@willscott @warpfork @rvagg

[rvagg]

makes sense to have a go-ipld-prime equivalent of dag.Walk which gets used pretty widely, throw in an optional selector and I can see this getting a lot of use

[warpfork]

I would really like to figure out two things:

1. what our roadmap is for a longterm, coherent plan for what the storage interfaces look like.
2. where we want to put glue code and bridge functions until we get to that long term coherent goal.

On the long term front:

• I do not believe that all of datastore, blockstore, blockservice, and dagservice (and is there a dagstore, just to fill out the matrix? I don't even know) are coherent and important or useful (and even if they were, good heavens are they a confusing maze and questionably named).
  • I recently tried to rediscover all these in a comment on another code review in go-path and I literally brain-OOM'd trying to figure it out. This is not a good thing.
  • Maybe some of these are perfectly fine. Let's say, for the sake of argument, that at least one of them is perfect and could not possibly be improved upon; it's a shining gem of good API, high performance, everything that matters agreed to use it already, etc. Even if that were true -- and I don't think it is -- I still wouldn't know which one of those things it is. It's a paradox of choice, and I consider that a first class problem.

On the short term front:

• We keep seeing these things pop up, so clearly we need an answer.
  • There's your link here.
  • A bunch of glue like this has appeared in go-fetcher, independently.
  • It's probably popping up in other places I don't know about.
  • And people are still asking what to do on matrix/discord, too.
• So, for a while, I thought letting people solve this piecemeal was going to be fine. As questions keep coming, and a community preferred solution keeps failing to evolve naturally, I'm starting to update my stance.
• But I still want to maintain a firewall between any of these interfaces, and the go-ipld core, due to the above disbelief that these interfaces we need to bridge today are good in the long term.
• And yet I also don't really want to deal with more repos sprouting up. (e.g., I don't think go-fetcher helped anything, it just fragmented things even more and gave me more new github notifications.)

I suspect a good answer here might involve making some new, separate go modules within this repo. I understand that should let us control the transitive dependency sprawl, and generally enforce success at the design firewall. I don't have a ton of hands-on-experience with how that works out in practice, yet, though, so we'll want to step somewhat carefully on that idea.

[rvagg]

Sadly this interface bloat made its way to JavaScript as well and we have variations of the above. For the newer multiformats/ipld work we just punted entirely to the user - work with [cid,bytes] pairs everywhere. For loading: I give you a CID, you give me it's bytes. For storing: I give you both a CID and its bytes.
It's not ideal, but it partly arises from wanting to defer attacking the beast of these interfaces and the various ways they are used. From what I understand, their evolution is based somewhat on the abstractions needing to be increasingly punctured over time for optimisations and other non-standard access patterns from IPFS. I've never really wanted to look into the details of that!
This might be something to stay in sync on if we arrive at a more ideal interface, or suite of them. This is clearly some nasty technical debt that is going to continue to hurt us if we don't clarify it.

[rvagg]

we have the storage/* packages in here now which I think get us at least part of the way to this.. I'm going to close this to remove a bit of clutter but @ribasushi if you feel strongly that closing this is wrong and that there's more to discuss feel free to reopen