You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The preparations are done in #46 but it turns out that we can't directly use rayon's par_iter because we make use of error handling in the loop and would have to pass the errors to the main thread somehow.
Current situation
The extraction is done sequential with each chunk being read from the file. One file is shared with seek+read.
Impact
Slow extraction
Ideal future situation
Maybe: Mmap the file once and have the verification/extraction functions read the "array" so that the for-loop could do bzip decompression in parallel.
But since we need to use pwrite anyway for parallel writing, we can also use it for parallel reading (or multiple FDs).
Here are Rust traits:
https://doc.rust-lang.org/std/os/unix/fs/trait.FileExt.html#tymethod.read_at
https://doc.rust-lang.org/std/os/unix/fs/trait.FileExt.html#method.read_exact_at
https://doc.rust-lang.org/std/os/unix/fs/trait.FileExt.html#method.write_all_at
Implementation options
Use rayon to run the extraction for-loop in parallel
The text was updated successfully, but these errors were encountered: