Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare subspace-farmer-components to retrieve pieces in batches #3132

Merged
merged 4 commits into from
Oct 16, 2024

Conversation

nazar-pc
Copy link
Member

This introduces PieceGetter::get_pieces() (with dummy implementation that just calls PieceGetter::get_piece() internally) and prepares subspace-farmer-components to use it.

This prepares lower-level part of the farmer implementation, next steps include properly implementing PieceGetter::get_pieces() in FarmerPieceGetter and ClusterPieceGetter to fully utilize new capabilities and then switching piece cache sync to PieceGetter::get_pieces().

Eventually we'll make similar changes in subspace-service and upcoming data retrieval application.

I'm also considering optimizing PieceGetter::get_piece() to take advantage of already connected peers if there remains a use case for it after all this, though I'll need to check how much sense there is since we can just call with a single piece index, though I realize we'll have a bunch of extra bookkeeping there that might be too heavy for the use case.

Note that in recover_missing_piece implementation is not of the highest possible efficiency (it waits for one batch to complete before starting to download more pieces), but that is a conscious decision to simplify implementation given that requiring reconstruction is an exceptional situation to begin with and should be a very cold code path overall, it really shouldn't happen on any real network in practice.

Code contributor checklist:

@nazar-pc nazar-pc force-pushed the subspace-farmer-components-pieces-in-batches branch from 8386bb5 to 66fc2f8 Compare October 16, 2024 07:39
@nazar-pc nazar-pc added this pull request to the merge queue Oct 16, 2024
github-merge-queue bot pushed a commit that referenced this pull request Oct 16, 2024
…eces-in-batches

Prepare `subspace-farmer-components` to retrieve pieces in batches
@nazar-pc nazar-pc removed this pull request from the merge queue due to a manual request Oct 16, 2024
@nazar-pc nazar-pc merged commit 40c8c0a into main Oct 16, 2024
8 checks passed
@nazar-pc nazar-pc deleted the subspace-farmer-components-pieces-in-batches branch October 16, 2024 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants