pageserver: enable compaction to proceed while live-migrating

[jcsp]

Problem

Long ago, in #5299 the tenant states for migration are added, but respected only in a coarse-grained way: when hinted not to do deletions, tenants will just avoid doing all GC or compaction.

Skipping compaction is not necessary for AttachedMulti, as we will soon become the primary attached location, and it is not a waste of resources to proceed with compaction. Instead, per the RFC
https://github.com/neondatabase/neon/pull/5029/files), deletions should be queued up in this state, and executed later when we switch to AttachedSingle.

Avoiding compaction in AttachedMulti can have an operational impact if a tenant is under significant write load, as a long-running migration can result in a large accumulation of delta layers with commensurate impact on read latency.

Closes: #5396

Summary of changes

• Add a 'config' part to RemoteTimelineClient so that it can be aware of the mode of the tenant it belongs to, and wire this through for construction + updates
• Add a special buffer for delayed deletions, and when in AttachedMulti route deletions here instead of into the main remote client queue. This is drained when transitioning to AttachedSingle. If the tenant is detached or our process dies before then, then these objects are leaked.
• As a quality of life improvement, also use the remote timeline client's knowledge of the tenant state to avoid submitting remote consistent LSN updates for validation when in AttachedStale (as we know these will fail)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5535 tests run: 5309 passed, 0 failed, 226 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_pull_timeline[True]: release-arm64

Code coverage* (full report)

• functions: 31.4% (7940 of 25306 functions)
• lines: 49.3% (63030 of 127869 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
9f1baf6 at 2024-11-20T17:41:18.371Z :recycle:}

[skyzh]

Generally LGTM, but I can see there are potentially some races (in reality, very rare to happen):

(1) Thread 1 calling update_config (remote_timeline_client.rs L495), it move all blocked deletions to the queue, and is about to update self.config to the new value.

(2) Thread 2 obtained the queue lock and looked at the deletion operation in the queue right before thread 1 update self.config (remote_timeline_client.rs L1963). At this point, self.config still has block_deletions=true, and everything gets moved back to the blocked deletion buffer.

But (2) might take a relatively long time and would not finish within the short amount of the time when (1) is right about to update the config.

Anyways, I don't see such races would lead to very bad effects except files don't get deleted immediately, so it's fine to leave it there for now.

[jcsp]

Generally LGTM, but I can see there are potentially some races (in reality, very rare to happen):

This is a good point - I have tweaked the code to update remotetimeline conf before rather than after draining the queue