Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support offloaded timelines during shard split #9489

Merged
merged 5 commits into from
Oct 25, 2024
Merged

Conversation

arpad-m
Copy link
Member

@arpad-m arpad-m commented Oct 23, 2024

Before, we didn't copy over the index-part.json of offloaded timelines to the new shard's location, resulting in the new shard not knowing the timeline even exists.

In #9444, we copy over the manifest, but we also need to do this for index-part.json.

As the operations to do are mostly the same between offloaded and non-offloaded timelines, we can iterate over all of them in the same loop, after the introduction of a TimelineOrOffloadedArcRef type to generalize over the two cases. This is analogous to the deletion code added in #8907.

The added test also ensures that the sharded archival config endpoint works, something that has not yet been ensured by tests.

Part of #8088

@arpad-m arpad-m requested a review from a team as a code owner October 23, 2024 16:15
@arpad-m arpad-m requested a review from jcsp October 23, 2024 16:15
@arpad-m arpad-m changed the title Support offloaded timelines during shard splits of offloaded timelines Support offloaded timelines during shard splits Oct 23, 2024
@arpad-m arpad-m changed the title Support offloaded timelines during shard splits Support offloaded timelines during shard split Oct 23, 2024
Copy link

github-actions bot commented Oct 23, 2024

5273 tests run: 5059 passed, 0 failed, 214 skipped (full report)


Flaky tests (4)

Postgres 17

Code coverage* (full report)

  • functions: 31.4% (7664 of 24427 functions)
  • lines: 48.8% (60268 of 123425 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
7b558cc at 2024-10-24T15:06:38.557Z :recycle:

pageserver/src/tenant.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@jcsp jcsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct, but the way OffloadedTimeline::remote_client is used is a bit tenuous & I think other stuff that uses it might not work (comment inline)

@arpad-m arpad-m requested a review from jcsp October 24, 2024 13:46
@arpad-m arpad-m merged commit 4d9036b into main Oct 25, 2024
81 checks passed
@arpad-m arpad-m deleted the arpad/offloaded_sharding branch October 25, 2024 10:32
jcsp pushed a commit that referenced this pull request Oct 29, 2024
As pointed out in
#9489 (comment) ,
we currently didn't support deletion for offloaded timelines after the
timeline has been loaded from the manifest instead of having been
offloaded.

This was because the upload queue hasn't been initialized yet. This PR
thus initializes the timeline and shuts it down immediately.

Part of #8088
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants