[Remote Translog] Introduce remote translog transfer support #4480

Bukhtawar · 2022-09-10T20:06:25Z

Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com

Description

The changes introduces support for blob store based remote translog store. The changes contain the remote transfer service interactions and snapshots data needed to be uploaded per commit.

Remote Translog Files and Location

.tlg file : base64_uuid/index_uuid/shard_id/primary_term/translog
.ckp file: base64_uuid/index_uuid/shard_id/primary_term/translog
metadata file : base64_uuid/index_uuid/shard_id/primary_term/metadata

Contents of Remote Translog Files

.tlg_N file: Same as .tlg file created on local disk per generation
.ckp_N file: Same as .ckp file created on local disk per generation
metadata file: metadata__primary_term__generation__timestamp
1. primary term
2. generation
3. timestamp
4. mapping between generation → primary_term(representative of the path in remote store, since we can derive the complete path from this)

Tests : Work In progress

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-09-10T20:07:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2946/
CommitID: 4ed18dd

github-actions · 2022-09-11T14:47:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2948/
CommitID: d988288

github-actions · 2022-09-11T18:40:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2949/
CommitID: a77c864

github-actions · 2022-09-11T20:08:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2950/
CommitID: df71151

github-actions · 2022-09-12T07:16:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2961/
CommitID: 593c8d1

github-actions · 2022-09-12T20:32:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2979/
CommitID: 593c8d1

github-actions · 2022-09-19T20:17:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3218/
CommitID: bd5f1de

github-actions · 2022-09-20T06:46:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3233/
CommitID: d696e19

github-actions · 2022-09-20T08:41:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3238/
CommitID: 0fcdc6e

github-actions · 2022-09-20T08:46:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3239/
CommitID: 8df5484

github-actions · 2022-09-20T14:08:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3246/
CommitID: 104108f

github-actions · 2022-09-20T14:49:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3249/
CommitID: f593c45

github-actions · 2022-09-20T15:02:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3251/
CommitID: 5291135

server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

sachinpkale · 2022-10-18T06:51:03Z

Pushed a commit which fixes #4820 as well.

github-actions · 2022-10-18T07:09:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4835/
CommitID: 8f1df36

github-actions · 2022-10-18T07:33:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4837/
CommitID: b32908a

github-actions · 2022-10-18T08:49:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4838/
CommitID: a1bad44

github-actions · 2022-11-15T18:50:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6916/
CommitID: 0bc8a30

gbbafna · 2022-11-16T07:09:11Z

server/src/main/java/org/opensearch/index/translog/BufferedChecksumStreamInput.java

+        try {
+            return readByte() & 0xFF;
+        } catch (EOFException e) {
+            return -1;
+        }


why do we need this change ?

This is due to an interface contract that OpenSearch was breaking. This is unrelated to the change

It looks like we have a base contract test for a stream input. Can you create an issue to create a contract test for this class since you've got the full context for this specific miss?

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

andrross · 2022-11-16T19:27:06Z

server/src/main/java/org/opensearch/index/translog/BufferedChecksumStreamInput.java

+        try {
+            return readByte() & 0xFF;
+        } catch (EOFException e) {
+            return -1;
+        }


It looks like we have a base contract test for a stream input. Can you create an issue to create a contract test for this class since you've got the full context for this specific miss?

server/src/main/java/org/opensearch/index/translog/transfer/FileSnapshot.java

andrross · 2022-11-16T19:36:24Z

server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java

+        Iterable<String> remoteTransferPath,
+        ActionListener<TransferFileSnapshot> listener
+    ) {
+        assert remoteTransferPath instanceof BlobPath;


Since you currently only have one instance of a TransferService, I would recommend not introducing an interface and do things like this. It's easy enough to refactor out an interface if/when you need it, but until you really need it you can only make educated guesses about what the generic interface should look like. Not a huge deal, I'll defer to your preference.

server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java

server/src/main/java/org/opensearch/index/translog/transfer/FileTransferException.java

CHANGELOG.md

andrross · 2022-11-16T19:45:55Z

...src/main/java/org/opensearch/index/translog/transfer/TranslogCheckpointTransferSnapshot.java

+            }
+            translogTransferSnapshot.setMinTranslogGeneration(highestGenMinTranslogGeneration);
+
+            assert this.primaryTerm == highestGenPrimaryTerm : "inconsistent primary term";


Again with me complaining about asserts :) but I would enforce these constraints unconditionally. Performance doesn't seem like a concern since the work is done unconditionally. Same applies for line 50.

I won't block on this issue if you strongly prefer asserts, but regardless we should have unit tests for these constraints.

I'll stick with this for now and make changes as needed once we added tests for this.

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-11-19T09:01:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7109/
CommitID: 8a5da20
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-11-21T13:37:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7138/
CommitID: cc4d85d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-11-21T17:06:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.index.shard.SegmentReplicationIndexShardTests.testNRTReplicaPromotedAsPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/7144/
CommitID: 2cbb582
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

andrross · 2022-11-22T00:25:00Z

CHANGELOG.md

@@ -50,6 +50,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
 - Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
 - Migrate client transports to Apache HttpClient / Core 5.x ([#4459](https://github.com/opensearch-project/OpenSearch/pull/4459))
+- Support remote translog transfer for request level durability([#4480](https://github.com/opensearch-project/OpenSearch/pull/4480))


This should go in the Unreleased 2.x section provided this will be backported.

opensearch-trigger-bot · 2023-01-04T05:17:02Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-4480-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a8b5d91bc8b6b2a79fa042d22775959043dac145
# Push it to GitHub
git push --set-upstream origin backport/backport-4480-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4480-to-2.x.

…rch-project#4480) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

…5693) * [Backport 2.x] Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

…rch-project#4480) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

…5773) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

…5693) * [Backport 2.x] Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar force-pushed the txlog-transfer branch from d988288 to a77c864 Compare September 11, 2022 18:39

Bukhtawar force-pushed the txlog-transfer branch from a77c864 to df71151 Compare September 11, 2022 20:07

Bukhtawar force-pushed the txlog-transfer branch from bd5f1de to d696e19 Compare September 20, 2022 06:32

Bukhtawar force-pushed the txlog-transfer branch from d696e19 to 0fcdc6e Compare September 20, 2022 08:24

Bukhtawar marked this pull request as ready for review September 20, 2022 08:35

Bukhtawar requested review from a team and reta as code owners September 20, 2022 08:35

Bukhtawar changed the title ~~WIP: Introduce remote translog transfer support~~ [Remote Translog] Introduce remote translog transfer support Sep 20, 2022

Bukhtawar force-pushed the txlog-transfer branch from 8df5484 to 104108f Compare September 20, 2022 13:45

Bukhtawar force-pushed the txlog-transfer branch from 104108f to f593c45 Compare September 20, 2022 14:23

sachinpkale reviewed Sep 22, 2022

View reviewed changes

sachinpkale force-pushed the txlog-transfer branch from 8f1df36 to b32908a Compare October 18, 2022 07:13

sachinpkale force-pushed the txlog-transfer branch from b32908a to a1bad44 Compare October 18, 2022 08:29

Bukhtawar requested a review from sachinpkale November 15, 2022 18:28

gbbafna reviewed Nov 16, 2022

View reviewed changes

Bukhtawar requested a review from gbbafna November 16, 2022 08:56

gbbafna reviewed Nov 16, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java Show resolved Hide resolved

Bukhtawar requested a review from gbbafna November 16, 2022 17:58

andrross reviewed Nov 16, 2022

View reviewed changes

andrross reviewed Nov 17, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java Outdated Show resolved Hide resolved

PR review comments

8a5da20

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Update changelog

cc4d85d

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Merge branch 'main' into txlog-transfer

2cbb582

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar mentioned this pull request Nov 21, 2022

[Remote Translog] Add Tests for base contracts for BufferedChecksumStreamInput #5322

Open

Bukhtawar requested review from andrross and gbbafna and removed request for gbbafna November 21, 2022 17:41

andrross approved these changes Nov 22, 2022

View reviewed changes

Bukhtawar merged commit a8b5d91 into opensearch-project:main Nov 22, 2022

gbbafna added the backport 2.x Backport to 2.x branch label Jan 4, 2023

gbbafna pushed a commit to gbbafna/OpenSearch that referenced this pull request Jan 4, 2023

[Remote Translog] Introduce remote translog transfer support (opensea…

9a8bd07

…rch-project#4480) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar pushed a commit that referenced this pull request Jan 9, 2023

[Remote Translog] Introduce remote translog transfer support (#4480) (#…

f6a86db

…5693) * [Backport 2.x] Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

gbbafna pushed a commit that referenced this pull request Jan 9, 2023

[Remote Translog] Introduce remote translog transfer support (#4480)

ffecf5f

* Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

gbbafna pushed a commit to gbbafna/OpenSearch that referenced this pull request Jan 10, 2023

[Remote Translog] Introduce remote translog transfer support (opensea…

d1cc1d5

…rch-project#4480) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

sachinpkale mentioned this pull request Jan 10, 2023

[Backport 2.5] [Remote Translog] Introduce remote translog transfer support (#4480) #5773

Merged

6 tasks

gbbafna pushed a commit that referenced this pull request Jan 10, 2023

[Remote Translog] Introduce remote translog transfer support (#4480) (#…

f9ea95e

…5773) * Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

kotwanikunal pushed a commit that referenced this pull request Jan 25, 2023

[Remote Translog] Introduce remote translog transfer support (#4480) (#…

d54b370

…5693) * [Backport 2.x] Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Translog] Introduce remote translog transfer support #4480

[Remote Translog] Introduce remote translog transfer support #4480

Bukhtawar commented Sep 10, 2022 •

edited

Loading

github-actions bot commented Sep 10, 2022

github-actions bot commented Sep 11, 2022

github-actions bot commented Sep 11, 2022

github-actions bot commented Sep 11, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 19, 2022

github-actions bot commented Sep 20, 2022

github-actions bot commented Sep 20, 2022

github-actions bot commented Sep 20, 2022

github-actions bot commented Sep 20, 2022

github-actions bot commented Sep 20, 2022

github-actions bot commented Sep 20, 2022

sachinpkale commented Oct 18, 2022

github-actions bot commented Oct 18, 2022

github-actions bot commented Oct 18, 2022

github-actions bot commented Oct 18, 2022

github-actions bot commented Nov 15, 2022

gbbafna Nov 16, 2022

Bukhtawar Nov 16, 2022

andrross Nov 16, 2022

andrross Nov 16, 2022

andrross Nov 16, 2022

andrross Nov 16, 2022

Bukhtawar Nov 21, 2022

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

andrross Nov 22, 2022

Bukhtawar Nov 22, 2022

opensearch-trigger-bot bot commented Jan 4, 2023

[Remote Translog] Introduce remote translog transfer support #4480

[Remote Translog] Introduce remote translog transfer support #4480

Conversation

Bukhtawar commented Sep 10, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Sep 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 11, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 11, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 11, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Oct 18, 2022

github-actions bot commented Oct 18, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 18, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 18, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 15, 2022

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 19, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 21, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 21, 2022

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 4, 2023

Bukhtawar commented Sep 10, 2022 •

edited

Loading