Async Store #5117
Async Store #5117
Conversation
|
The individual commits should be useful to review independently. |
src/rust/engine/fs/src/store.rs
Outdated
| let mut hasher = Sha256::default(); | ||
| hasher.input(&bytes); | ||
| let fingerprint = Fingerprint::from_bytes_unsafe(hasher.fixed_result().as_slice()); | ||
| fn store_bytes(&self, bytes: Vec<u8>, db: Database) -> CpuFuture<Fingerprint, String> { |
There was a problem hiding this comment.
Why a CpuFuture not a BoxFuture?
There was a problem hiding this comment.
If this were a crate-public API, it would be good to expose BoxFuture to avoid leaking implementation details... but since it isn't, avoiding the boxing is just nice.
| }); | ||
| match mkdir { | ||
| Ok(()) => {} | ||
| Err(e) => return future::result(Err(e)).to_boxed(), |
| .into_iter() | ||
| .map(|file_node| { | ||
| let path = destination.join(file_node.get_name()); | ||
| materialize_file(store.clone(), path, file_node) |
There was a problem hiding this comment.
Seems odd that this takes a FileNode rather than a Fingerprint.
(And similarly odd that this take_files()s and into_iter()s rather than just iterating over a read-only view, which I think it only does because you're passing a FileNode here rather than a Fingerprint)
There was a problem hiding this comment.
Good call... this looked necessary before I extracted the method, but was not. This improved the symmetry quite a bit.
|
Great, thanks! :) |
Problem
Storeexecutes IO, but was doing so synchronously on the main thread, which might block other operations.Solution
Give
Storea reference to aResettablePool, and execute all database operations on the pool.It was also necessary to apply the "
Arc<Inner*>pattern" (maybe it has a name?) to allow theStoreto be cloned into the pool... this pattern seems much cleaner than the "impl Trait for Arc<Struct>pattern" that we've used with theVFSimplementations... should consider switching those over at some point.Finally, I added
Store::load_bytes_withto support copying directly from LMDB buffers into a destination to avoid copying, and execute the copy on the IO pool.Result
save_directoryandmaterialize_directoryinfs_util, andDigestFileinnodeswill execute concurrently. Helps to unblock #5105 and #5106.