feat(pageserver): support partial gc-compaction for delta layers

[skyzh]

Problem

The final patch for partial compaction, part of #9114, close #8921 (note that we didn't implement parallel compaction or compaction scheduler for partial compaction -- currently this needs to be scheduled by using a Python script to split the keyspace, and in the future, automatically split based on the key partitioning when the pageserver wants to trigger a gc-compaction)

Summary of changes

• Update the layer selection algorithm to use the same selection as full compaction (everything intersect/below gc horizon)
• Update the layer selection algorithm to also generate a list of delta layers that need to be rewritten
• Add the logic to rewrite delta layers and add them back to the layer map
• Update test case to do partial compaction on deltas

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5372 tests run: 5152 passed, 0 failed, 220 skipped (full report)

---

Flaky tests (2)

Postgres 17

• test_readonly_node_gc: debug-x86-64
• test_storage_controller_metadata_health[2]: debug-x86-64

Code coverage* (full report)

• functions: 31.8% (7882 of 24824 functions)
• lines: 49.4% (62416 of 126226 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
6641f10 at 2024-11-11T20:39:49.439Z :recycle:}

[problame]

The final patch for partial compaction,

Hm, is this truly the final piece of work in partial bottommost compaction? We don't support partial bottommost compaction yet.

---

We should not have a not-partial-bottommost-compact going forward, to reduce the amount of potential code paths that need to be kept in mind / reviewed.

Hence, can we please switch the API to take the compaction_key_range a Range<Key> instead of Option<Range<Key>>.

Full bottommost compaction unit tests can pass Key::MIN..Key::MAX. Or should it be just .. ? Or Key::Min..=Key::MAX.

Anyway, hope you get the point.

[skyzh]

The full/partial code path is now consolidated, and ready for another round of review :) Next week I'll likely add compaction key range to the manual compaction API so that I can start some testing in staging.