Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Snapshot V2] Support pinned timestamp in delete flow #15256

Merged
merged 67 commits into from
Sep 4, 2024
Merged

[Snapshot V2] Support pinned timestamp in delete flow #15256

merged 67 commits into from
Sep 4, 2024

Conversation

anshu1106
Copy link
Contributor

@anshu1106 anshu1106 commented Aug 15, 2024 •
edited
Loading

Description

This PR introduce changes to support pinned timestamp in delete flow

  • With pinned timestamp we don't create explicit lock files and we can avoid the logic to query and delete lock files
  • Unpin timestamp for deleted snapshots
  • support deletion of remote store segment files for deleted indices

Related Issues

Resolves #15255
#15466

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Anshu Agarwal added 10 commits August 6, 2024 10:43
Initial Commit to support centralize snapshot creation and implicit l…
9142a25 
…ocking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Fix deserilization error
d9bbc65 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Fix gradle spotless check
f72702f 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Fix listener
8e85231 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Merge branch 'main' into snapshot-pinned-timestamp
8a507ad
Fix test
2d404e8 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Fix snapshot generation
90c860c 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Modify cluster setting name
193da65 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Add more tests
fe2aaaf 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Snapshot V2 delete changes
afcc37a 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Snapshots labels Aug 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for afcc37a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for c91f1b5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Anshu Agarwal added 4 commits August 21, 2024 11:13
Modify log messages
626c2fa 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Add spotless check failure fix
be65f6d 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Fix completion listener for snapshot v2
62452ee 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Elevate cluster state update priority for repository metadata update …
00031ec 
…task

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
@anshu1106 anshu1106 marked this pull request as ready for review August 22, 2024 09:11
@mch2 mch2 mentioned this pull request Sep 4, 2024
Open
mch2 added a commit to mch2/OpenSearch that referenced this pull request Sep 4, 2024
@mch2
Revert "[Snapshot V2] Support pinned timestamp in delete flow (opense…
a9a2424 
…arch-project#15256)"

This reverts commit 5bf34d2.
mch2 added a commit to mch2/OpenSearch that referenced this pull request Sep 4, 2024
@mch2
Revert "[Snapshot V2] Support pinned timestamp in delete flow (opense…
589a530 
…arch-project#15256)"

This reverts commit 5bf34d2.

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
anshu1106 added a commit to anshu1106/OpenSearch that referenced this pull request Sep 5, 2024
@anshu1106
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
3bf2865 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 5bf34d2)
@anshu1106 anshu1106 mentioned this pull request Sep 5, 2024
Merged
3 tasks
gbbafna pushed a commit that referenced this pull request Sep 5, 2024
@anshu1106
[Snapshot V2] Support pinned timestamp in delete flow (#15256) (#15722)
c91f2f1 
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 5bf34d2)
anshu1106 added a commit to anshu1106/OpenSearch that referenced this pull request Sep 5, 2024
@anshu1106
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
c6317ec 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 5bf34d2)
@anshu1106 anshu1106 mentioned this pull request Sep 5, 2024
Merged
3 tasks
anshu1106 added a commit to anshu1106/OpenSearch that referenced this pull request Sep 5, 2024
@anshu1106
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
81a4408 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 5bf34d2)
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
sachinpkale pushed a commit that referenced this pull request Sep 5, 2024
@anshu1106
[Snapshot V2] Support pinned timestamp in delete flow (#15256) (#15730)
7f08d68 
(cherry picked from commit 5bf34d2)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
This was referenced Sep 6, 2024
Open
Open
Open
Open
Open
Open
Open
akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Sep 10, 2024
@anshu1106 @akolarkunnu
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
1a585ab 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
This was referenced Sep 11, 2024
Open
Open
Open
Open
Open
Open
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 6, 2024
Open
@prudhvigodithi prudhvigodithi mentioned this pull request Oct 16, 2024
Open
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 16, 2024
@anshu1106 @dk2k
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
d7f8cd4 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 17, 2024
@anshu1106 @dk2k
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
308d9a7 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 21, 2024
@anshu1106 @dk2k
[Snapshot V2] Support pinned timestamp in delete flow (opensearch-pro…
831779e 
…ject#15256)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sachinpkale sachinpkale sachinpkale left review comments

@gbbafna gbbafna gbbafna approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@jed326 jed326 Awaiting requested review from jed326

@harishbhakuni harishbhakuni Awaiting requested review from harishbhakuni

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch backport 2.17 backport-failed enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Snapshots
Projects
Storage Project Board
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Snapshot V2] Integrate delete snapshot flow with pinned timestamp
3 participants
@anshu1106 @sachinpkale @gbbafna