Optimized farmer piece getter #2565
Conversation
|
@EmilFattakhov @jim-counter this brings behavior change that will be mentioned in release notes, but want to make sure you're aware of this as well |
| @@ -32,6 +34,7 @@ struct Inner<PV, NC> { | |||
| node_client: NC, | |||
| plotted_pieces: Arc<Mutex<Option<PlottedPieces>>>, | |||
| dsn_cache_retry_policy: DsnCacheRetryPolicy, | |||
| in_progress_pieces: Mutex<HashMap<PieceIndex, Arc<AsyncMutex<Option<Piece>>>>>, | |||
There was a problem hiding this comment.
Why do you use both Mutex and AsyncMutex here?
There was a problem hiding this comment.
Mutex is generally faster and preferred, but I do need to hold guard of AsyncMutex across .await, hence async mutex is needed there.
| @@ -120,6 +158,46 @@ where | |||
|
|
|||
| /// Slow way to get piece using archival storage | |||
| pub async fn get_piece_slow(&self, piece_index: PieceIndex) -> Option<Piece> { | |||
| let in_progress_piece_mutex = Arc::new(AsyncMutex::new(None)); | |||
There was a problem hiding this comment.
This code is almost identical to the previous function and it is complex enough to remove the duplication.
There was a problem hiding this comment.
I anticipated this comment. Yes, there are 3 places with almost identical, but slightly different code. However, de-duplicating it (I guess without macros that reduce readability) is challenging due to MutexGuards that would require self-referrential types. I decided to not go that route, but feel free to play with it over the weekend and send a PR if you find a nicer way to handle this.
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
This builds on top of a few previous PRs and makes farmer's piece getter more efficient.
It is now possible to do node sync, piece cache sync and plotting all at the same time on Space Acres and due to sorting of pieces to be downloaded in piece cache sync, node and piece cache sync will have a significant overlap of pieces that they downloading, making the whole process faster than before.
Code contributor checklist: