refactor(VirtualFile::crashsafe_overwrite): avoid Handle::block_on in callers

[problame]

Some callers of VirtualFile::crashsafe_overwrite call it on the executor thread, thereby potentially stalling it.

Others are more diligent and wrap it in spawn_blocking(..., Handle::block_on, ... ) to avoid stalling the executor thread.

However, because crashsafe_overwrite uses VirtualFile::open_with_options internally, we spawn a new thread-local tokio-epoll-uring::System in the blocking pool thread that's used for the spawn_blocking call.

This PR refactors the situation such that we do the spawn_blocking inside VirtualFile::crashsafe_overwrite. This unifies the situation for the better:

1. Callers who didn't wrap in spawn_blocking(..., Handle::block_on, ...) before no longer stall the executor.
2. Callers who did it before now can avoid the block_on, resolving the problem with the short-lived tokio-epoll-uring::Systems in the blocking pool threads.

A future PR will build on top of this and divert to tokio-epoll-uring if it's configures as the IO engine.

Changes

• Convert implementation to std::fs and move it into crashsafe.rs

  • Yes, I know, Safekeepers (cc @arssher ) added durable_rename and fsync_async_opt recently. However, crashsafe_overwrite is different in the sense that it's higher level, i.e., it's more like std::fs::write and the Safekeeper team's code is more building block style.
  • The consequence is that we don't use the VirtualFile file descriptor cache anymore.
  • I don't think it's a big deal because we have plenty of slack wrt production file descriptor limit rlimit (see this dashboard)

• Use tokio::task::spawn_blocking in VirtualFile::crashsafe_overwrite to call the new crashsafe::overwrite API.

• Inspect all callers to remove any double-spawn_blocking

• spawn_blocking requires the captures data to be 'static + Send. So, refactor the callers. We'll need this for future tokio-epoll-uring support anyway, because tokio-epoll-uring requires owned buffers.

Related Issues

• overall epic to enable write path to tokio-epoll-uring: Epic: adopt tokio-epoll-uring on the write path #6663
• this is also kind of relevant to the tokio-epoll-uring System creation failures that we encountered in staging, investigation being tracked in tokio-epoll-uring: "Cannot allocate memory" in staging on 2024-02-07 #6667
  • why is it relevant? Because this PR removes two uses of spawn_blocking+Handle::block_on

[github-actions]

2436 tests run: 2319 passed, 0 failed, 117 skipped (full report)

---

Flaky tests (4)

Postgres 15

• test_create_snapshot: debug, release
• test_empty_tenant_size: release
• test_delete_timeline_exercise_crash_safety_failpoints[Check.RETRY_WITH_RESTART-timeline-delete-before-rm]: release

Code coverage (full report)

• functions: 55.9% (12848 of 22966 functions)
• lines: 82.5% (69298 of 83998 lines)

---

_{The comment gets automatically updated with the latest test results
d0f2ab3 at 2024-02-14T14:25:41.325Z :recycle:}

[koivunej]

Need to remove the REVIEW: tags. I see no issues apart from the cutpasted comment.

[problame]

Need to remove the REVIEW: tags. I

Ok, so, you don't see a problem with us now doing spawn_blocking in those places?

[jcsp]

LGTM. In hindsight, we should have done this way back when the prototype of crashsafe_overwrite was changed to async.

[koivunej]

Ok, so, you don't see a problem with us now doing spawn_blocking in those places?

No issues.