-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Segment Replication] Add PIT/Scroll compatibility with Segment Replication #6644
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
26477a8
to
ccd5694
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #6644 +/- ##
=======================================
Coverage ? 70.73%
Complexity ? 59206
=======================================
Files ? 4810
Lines ? 283497
Branches ? 40873
=======================================
Hits ? 200545
Misses ? 66459
Partials ? 16493
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Gradle Check (Jenkins) Run Completed with:
|
This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com>
Signed-off-by: Marc Handalian <handalm@amazon.com>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
…of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Closing this PR in favour of #6765 (author is on leave) CC @anasalkouz |
…cation #6644 (#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>
…cation opensearch-project#6644 (opensearch-project#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>
…cation (#7010) * [Segment Replication] Add PIT/Scroll compatibility with Segment Replication #6644 (#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Add param definition causing precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove unnecessary override annotation Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>
Description
This change makes updates to make PIT/Scroll queries compatible with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed.
Issues Resolved
#6519
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.