pageserver: flush deletion queue on detach

[jcsp]

Problem

If a caller detaches a tenant and then attaches it again, pending deletions from the old attachment might not have happened yet. This is not a correctness problem, but it causes:

• Risk of leaking some objects in S3
• Some warnings from the deletion queue when pending LSN updates and pending deletions don't pass validation.

Summary of changes

• Deletion queue now uses UnboundedChannel so that the push interfaces don't have to be async.
  • This was pulled out of pageserver: enable compaction to proceed while live-migrating #5397, where it is also useful to be able to drive the queue from non-async contexts.
  • Why is it okay for this to be unbounded? The only way the unbounded-ness of the channel can become a problem is if writing out deletion lists can't keep up, but if the system were that overloaded then the code generating deletions (GC, compaction) would also be impacted.
• DeletionQueueClient gets a new flush_advisory function, which is like flush_execute, but doesn't wait for completion: this is appropriate for use in contexts where we would like to encourage the deletion queue to flush, but don't need to block on it.
  • This function is also expected to be useful in next steps for seamless migration, where the option to flush to S3 while transitioning into AttachedStale will also include flushing deletion queue, but we wouldn't want to block on that flush.
• The tenant_detach code in mgr.rs invokes flush_advisory after stopping the Tenant object.

[github-actions]

2280 tests run: 2164 passed, 0 failed, 116 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_wal_lagging: release

Code coverage (full report)

• functions: 52.3% (8144 of 15558 functions)
• lines: 81.0% (47637 of 58775 lines)

---

_{The comment gets automatically updated with the latest test results
16977d2 at 2023-10-10T09:48:35.577Z :recycle:}

[arpad-m]

Needs a rebase.

Right now we would have requests get stuck and eventually time out if the queue gets stuck, which creates noisy errors.

I also wonder if we should somehow measure the queue length in a metric or warn if the queue length is larger than some limit. There is the risk of having a memory "leak" like the one fixed in #5472 ...

[jcsp]

I also wonder if we should somehow measure the queue length in a metric or warn if the queue length is larger than some limit. There is the risk of having a memory "leak" like the one fixed in #5472 ...

We do already have metrics that enable calculating queue length -- this was a good reminder for me to add charts for those to the pageserver dashboard. I'll follow up to add an alert for queue length.

[jcsp]

Push:

• Merge main into this branch
• Since the LocationConf stuff merged, we should also do this flush-on-detach behavior when we shut down a tenant in upsert_location -- added that.