Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement chain export functionality and car writer #861

Merged
merged 6 commits into from
Nov 24, 2020

Conversation

austinabell
Copy link
Contributor

@austinabell austinabell commented Nov 24, 2020

Summary of changes
Changes introduced in this pull request:

  • I made the writer async, because it makes things a lot cleaner, faster and not block the async executor with file io.
    • Essentially, one task is receiving blocks of (Cid, Vec<u8>) and writing it, while the other is traversing the chain and passing any unvisited blocks through the channel. Better imo to do in parallel than doing it all in sequence
  • There isn't currently any exposed usages of this (although the RPC endpoint can be easily added now) but the main reason I needed it was to be able to export chains and I updated forest-bench austinabell/forest-bench@ad49fb4

Car read is also async now, to avoid blocking executor on io reads (this is definitely less important, because all of our usages are before we start the node, but this is good to future proof for when we can import a car while the node is running)

Reference issue to close (if applicable)

Closes

Other information and links

@austinabell austinabell merged commit d690d48 into main Nov 24, 2020
@austinabell austinabell deleted the austin/chainexport branch November 24, 2020 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants