pageserver: make BufferedWriter do double-buffering

[yliang412]

Closes #9387.

Problem

BufferedWriter cannot proceed while the owned buffer is flushing to disk. We want to implement double buffering so that the flush can happen in the background. See #9387.

Summary of changes

• Maintain two owned buffers in BufferedWriter.
• The writer is in charge of copying the data into owned, aligned buffer, once full, submit it to the flush task.
• The flush background task is in charge of flushing the owned buffer to disk, and returned the buffer to the writer for reuse.
• The writer and the flush background task communicate through a bi-directional channel.

For in-memory layer, we also need to be able to read from the buffered writer in get_values_reconstruct_data. To handle this case, we did the following

• Use replace VirtualFile::write_all with VirtualFile::write_all_at, and use Arc to share it between writer and background task.
• leverage IoBufferMut::freeze to get a cheaply clonable IoBuffer, one clone will be submitted to the channel, the other clone will be saved within the writer to serve reads. When we want to reuse the buffer, we can invoke IoBuffer::into_mut, which gives us back the mutable aligned buffer.
• InMemoryLayer reads is now aware of the maybe_flushed part of the buffer.

Caveat

• We removed the owned version of write, because this interface does not work well with buffer alignment. The result is that without direct IO enabled, download_object does one more memcpy than before this PR due to the switch to use _borrowed version of the write.
• "Bypass aligned part of write" could be implemented later to avoid large amount of memcpy.

Testing

• use an oneshot channel based control mechanism to make flush behavior deterministic in test.
• test reading from EphemeralFile when the last submitted buffer is not flushed, in-progress, and done flushing to disk.

Performance

We see performance improvement for small values, and regression on big values, likely due to being CPU bound + disk write latency.

Results

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

7018 tests run: 6710 passed, 0 failed, 308 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_pull_timeline[True]: release-x86-64

Code coverage* (full report)

• functions: 30.8% (8306 of 26946 functions)
• lines: 47.8% (65399 of 136789 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
1da4028 at 2024-12-03T15:31:43.605Z :recycle:}

[problame]

write_buffered vs write_buffered_borrowed: my gut feeling is that in practice on-demand downloads did benefit from the old behavior where we were able to bypass the buffer (lower CPU usage).

We have that pagebench sub-benchmark for on-demand downloads, you could compare CPU usage before and after this change.

But, might be faster to "just" address this TODO.

Maybe you can be generic over constraints on the buffer type by making the buffer type an associated type of the writer?

[problame]

Ok, again on write_buffered / write_bufferd_borrowed.
Let's call it "buffer bypass for aligned parts of the write".

I remembered that with O_DIRECT, we save one memcpy so we can spend one and come out net 0 wrt CPU efficiency.

It would be nice to have a CPU efficiency WIN, though I'm ok with net 0.

The only remaining CPU efficiency difference that I can think of right now is that write_buffered issues one giant write for the entire middle of the buffer, whereas write_buffered issues TAIL_SZ'd writes.

---

Left a couple of comments that need addressing. Let's discuss major unclarities on Slack.

[problame]

Let's see how this works in staging. Preprod deployment later this week.