Propose traversal-based car creation

[willscott]

• Complete the file-based alternative writes with no-precalculated size when a file is provided, and then goes pack to fix up the size of the data in the header at the end.
• include carv1 stream support
• Handle options being set on the traversal (repeated blocks)
• Handle options being set for the car (type of index, etc.)
• Roundtrip test

[rvagg]

Lotus evolved away from needing this, so I don't think there's a consumer for prepare+dump anymore, it's just write now, thankfully. There used to be a need to get the size up-front to set up the commp calculation with padding, but that's not necessary anymore. So you could decomplicate this by removing this functionality if you like.

[willscott]

If we want to write out to a stream - e.g. network, this 2x scan would still be more efficient than having to write it out to a file, touch up the header, and then as a second step send the whole thing, i think

[rvagg]

Oh, so this would be for a CARv2 format and you need the offset and all that? Because for a selector-based CARv1 you have the root already, or else you can't run the traversal.

[willscott]

yeah, this is to get the data size field correct in the carv2 header when streaming.

I can make a carv1-only stream version that can do it in one pass as well. I suppose that's probably useful as well.

[willscott]

I think i have all the code here doing what I want. starting in on tests.

This would be a great time for reviews if you want to request changes to naming

[warpfork]

Okay, I'm not gonna try to steer here, this is more a question for me to understand how to serve this area better in the future: Did you go through a decision process on whether to do this wrap StorageReadOpener approach vs do some wrappers around the new storage.* APIs?

Is this wrapping strategy just familiar already?

Is this wrapping strategy preferable because it seems like the most broadly applicable place to intercept things?

Is this wrapping strategy preferable because it's clear how it composes with other StorageReadOpener magic, such as how graphsync uses it to hook block load events?

Is it because figuring out how to wrap the storage.* APIs is daunting because it would've required thinking about all the places feature detection might try kick in?

Other?

(These are not blocking questions; they just could be useful info for understanding the value of any future iterations on storage and link loading APIs and other possible event callbacks around them.)

[willscott]

I think it's that the object that is expected to be passed around is a LinkSystem - that linksystem will already have the storage.* object applied to it and turned into a StorageReadOpener, and in this library I won't know what the underlying readable storage was to be able to wrap it. If i assume the right thing to take in is a link system, which seems right because i need to care about things like 'should i hash reads', 'what reifiers are present', etc, then this is the place i can intercept block reads

[masih]

Thinking out loud, in a separate PR we probably should change NewHeader to take options and encapsulate the gymnastics needed to set correct header values.