Allow AlignedBuffer to return "owned slices" (like Bytes), which refer to the same buffer in RAM

[JackKelly]

Use-cases

• Merge multiple ranges in get_ranges, and return the individual "owned slices".

Related

• Maybe use Bytes within AlignedBuffer #112
• Check if ndarray, numpy, & compression crates can consume &[u8] (and hence consume from AlignedBuffer) #113

Implementation

I think I can't just use Arc<AlignedBuffer> because each clone needs to have its own valid_slice.

[JackKelly]

Some more thoughts on this:

Use-cases

• Re-use buffers
  • Mark segment as "free to be reused" when all references to that segment go away (i.e. when each segment is Drop'd)
• Allocate once, and manage re-use ourselves (although maybe this is a premature optimisation?!)

[JackKelly]

For now, I'm going to ignore re-using buffers.

But I do need to support the two use-cases at the top of this thread:

1. The user requests multiple contiguous byte ranges.

Let's say the user requests byte_ranges: 0..1000, 1000..2000, 2000..3000, 3000..4000.

LSIO then:

• merges these ranges
• allocates a single 4000 byte AlignedBufferMut
• submits a single read operation to the operation system.
• when the single read op completes, we freeze the buffer, which turns the entire 4,000 byte AlignedBufferMut into a 4,000 byte AlignedBuffer (immutable).
• Get four owned slices into the AlignedBuffer. Return these to the user.
• Drop the 4,000 byte slice (but don't drop the underlying buffer!)
• The buffer is deallocated when all the owned slices are dropped by the user.

2. The user requests a single 8 GB file.

Linux can't read more than 2 GB at once.

LSIO:

• allocates a single 8 GB AlignedBufferMut
• splits this into a new 2 GB AlignedBufferMut and the old AlignedBufferMut is reduced to 6 GB. Both of these buffers must have their starts and ends aligned. Then repeat the process to get 4 x 2 GB AlignedBufferMuts.
• issues four read operations to the OS.
• when the first, second, and third read ops complete, drop that AlignedBufferMut (but that won't drop the underlying storage, it just removes its reference)
• when the last read op completes, freeze it to get an immutable view of the 8 GB slice requested by the user.

[JackKelly]

Implementation

InnerAlignedBuffer needs to keep track of which byte ranges are "owned" by AlignedBufferMuts. Only one owner can see those bytes.

Maybe something like this:

struct InnerAlignedBuffer {
  buf: *mut u8,  // Should be NotNull?
  layout: alloc::Layout,
  exclusive_ranges: Vec<Range<usize>>,
}

impl InnerAlignedBuffer {
  // `AlignedBufferMut.split` must call this function:
  fn get_exclusive_slice(&mut self, slice: Range<usize>) -> Result<()> {
    for exclusive_range in self.exclusive_ranges.iter() {
        if overlaps(slice, exclusive_range) {
          return Err
        }
    }
    Ok(())
  }
  
  // `AlignedBufferMut.drop` must call this function:
  fn remove_exclusive_slice(&mut self, slice: Range<usize>) -> Result<()) {
  }
}

[JackKelly]

OK, I think the main functionality is implemented in 030920d

Still TODO:

• Write tests!
• Simplify! See the TODO items in the code

[JackKelly]

todo: doctest for 2nd use-case above

[JackKelly]

Great! I think this is done! I just want to re-read the code with fresh eyes, and then I can close this issue!