Make number of segment metadata files in remote segment store configurable #11329

sachinpkale · 2023-11-24T16:57:19Z

Description

Current Behavior

Remote segment store uploads a metadata file with each successful upload of segments post refresh.
To read any segment data from remote segment store, we need to read metadata file first.
All the flows that involves copying segments between 2 nodes, like replication, peer recovery are integrated with remote store.
Following are the steps to cleanup stale segments (that are not live on local) from remote segment store:
- On flush, we fetch all but latest 10 metadata files, let's call them stale metadata files.
- Create a list of segments that are referenced by the stale metadata files and not currently live, let's call them stale segment files.
- Delete stale segment files followed by deletion of stale metadata files.

Issue

Replication and peer recovery flows are async. When replica (or new primary in case of primary relocation) starts downloading data from remote segment store, it reads the latest metadata file and starts downloading segment files referred by the metadata file.
To avoid race condition where slow replica is taking a lot of time to download and the segment files became stale at primary (due to segment merges) and deleted from remote segment store, we kept a hard-coded value of 10 to keep last 10 metadata files.
If the amount of data to be downloaded is high and if the flush (and merge) happens at primary during that time, it is possible that replica will not find a segment file in the remote store and recovery fails with following exception.

IndexShardRecoveryException[Exception while copying segment files from remote segment store]; 
nested: IOException[Error occurred when downloading segments from remote store]; 
nested: NoSuchFileException[Blob object [<Segment Path>/_6yt.cfs__KegSxIsBOrEU3DYZHraf] not found: 
The specified key does not exist]

Solution

In this PR, we are mitigating the above mentioned issue by making last N metadata files to keep configurable.
This is not a long term solution to the problem and we need to come up with a lock based solution (used by snapshot interop) but until then this change provides a way to avoid NoSuchFileException.
We also made a change to peer recovery flow where, if downloading from remote segment store fails, recovery flow will continue with node-node segment copy. This is something that exists even today with remote store integration of peer recovery flow.

Related Issues

[BUG] NoSuchFileException during peer recovery of a larger shard using remote store #11328

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-11-24T17:20:02Z

Compatibility status:

Checks if related components are compatible with change 1a6cfad

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2023-11-24T17:26:56Z

❌ Gradle check result for d5ac8ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

ashking94

Looks good, but interested in plan for this setting as it will be user facing. I can review once UTs and ITs are added.

Signed-off-by: Sachin Kale <kalsac@amazon.com>

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

github-actions · 2023-11-25T14:47:25Z

❌ Gradle check result for 71e133c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-11-26T03:45:41Z

❌ Gradle check result for cb81e86: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-11-26T04:50:10Z

❌ Gradle check result for 98884c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-11-26T05:52:34Z

❕ Gradle check result for 250fdcc: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-11-26T05:53:08Z

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (5bb6cae) 71.21% compared to head (250fdcc) 71.29%.
Report is 3 commits behind head on main.

❗ Current head 250fdcc differs from pull request most recent head 1a6cfad. Consider uploading reports for the commit 1a6cfad to get more accurate results

Files	Patch %	Lines
...ch/indices/recovery/PeerRecoveryTargetService.java	0.00%	4 Missing ⚠️
...earch/index/store/RemoteSegmentStoreDirectory.java	0.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11329      +/-   ##
============================================
+ Coverage     71.21%   71.29%   +0.08%     
- Complexity    58926    58984      +58     
============================================
  Files          4890     4890              
  Lines        277434   277451      +17     
  Branches      40313    40313              
============================================
+ Hits         197567   197812     +245     
+ Misses        63464    63237     -227     
+ Partials      16403    16402       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

…ention.max_count setting Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-11-27T10:09:36Z

❕ Gradle check result for 1a6cfad: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-11-27T16:18:48Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11329-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ec5a0f99f89b82bbbd14cc0ae06ea8ea6ba08667
# Push it to GitHub
git push --set-upstream origin backport/backport-11329-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11329-to-2.x.

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

…rable (#11329) (#11357) Signed-off-by: Sachin Kale <kalsac@amazon.com>

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Make number of segment metadata files to keep configurable

d5ac8ac

Signed-off-by: Sachin Kale <kalsac@amazon.com>

ashking94 reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java Outdated Show resolved Hide resolved

ashking94 reviewed Nov 25, 2023

View reviewed changes

Sachin Kale added 2 commits November 25, 2023 18:37

Add UTs and ITs, fix existing UTs

fb10d90

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add changlog entry

71e133c

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale commented Nov 25, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java Show resolved Hide resolved

sachinpkale marked this pull request as ready for review November 25, 2023 13:28

sachinpkale requested review from abbashus, adnapibar, anasalkouz, andrross, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, peternied, reta, Rishikesh1159, ryanbogan, saratvemulapalli and setiah as code owners November 25, 2023 13:28

Bukhtawar reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java Outdated Show resolved Hide resolved

gbbafna reviewed Nov 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java Outdated Show resolved Hide resolved

Address PR comments

cb81e86

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix compile errors in test

98884c1

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix failing test

250fdcc

Signed-off-by: Sachin Kale <kalsac@amazon.com>

gbbafna reviewed Nov 27, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java Outdated Show resolved Hide resolved

Remove max limit from cluster.remote_store.index.segment_metadata.ret…

1a6cfad

…ention.max_count setting Signed-off-by: Sachin Kale <kalsac@amazon.com>

Bukhtawar approved these changes Nov 27, 2023

View reviewed changes

gbbafna approved these changes Nov 27, 2023

View reviewed changes

gbbafna merged commit ec5a0f9 into opensearch-project:main Nov 27, 2023
33 of 36 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Nov 27, 2023

opensearch-trigger-bot bot added the backport-failed label Nov 27, 2023

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Nov 28, 2023

Make number of segment metadata files in remote segment store configu…

5e3dd85

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale mentioned this pull request Nov 28, 2023

Make number of segment metadata files in remote segment store configurable (#11329) #11357

Merged

gbbafna pushed a commit that referenced this pull request Nov 28, 2023

Make number of segment metadata files in remote segment store configu…

cf7ba36

…rable (#11329) (#11357) Signed-off-by: Sachin Kale <kalsac@amazon.com>

fahadshamiinsta pushed a commit to fahadshamiinsta/OpenSearch270 that referenced this pull request Dec 4, 2023

Make number of segment metadata files in remote segment store configu…

628b29a

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request Dec 11, 2023

Make number of segment metadata files in remote segment store configu…

2009e07

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024

Make number of segment metadata files in remote segment store configu…

b973096

…rable (opensearch-project#11329) Signed-off-by: Sachin Kale <kalsac@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make number of segment metadata files in remote segment store configurable #11329

Make number of segment metadata files in remote segment store configurable #11329

sachinpkale commented Nov 24, 2023 •

edited

Loading

github-actions bot commented Nov 24, 2023 •

edited

Loading

github-actions bot commented Nov 24, 2023

ashking94 left a comment

github-actions bot commented Nov 25, 2023

github-actions bot commented Nov 26, 2023

github-actions bot commented Nov 26, 2023

github-actions bot commented Nov 26, 2023

codecov bot commented Nov 26, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023

opensearch-trigger-bot bot commented Nov 27, 2023

Make number of segment metadata files in remote segment store configurable #11329

Make number of segment metadata files in remote segment store configurable #11329

Conversation

sachinpkale commented Nov 24, 2023 • edited Loading

Description

Current Behavior

Issue

Solution

Related Issues

Check List

github-actions bot commented Nov 24, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Nov 24, 2023

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 25, 2023

github-actions bot commented Nov 26, 2023

github-actions bot commented Nov 26, 2023

github-actions bot commented Nov 26, 2023

codecov bot commented Nov 26, 2023 • edited Loading

Codecov Report

github-actions bot commented Nov 27, 2023

opensearch-trigger-bot bot commented Nov 27, 2023

sachinpkale commented Nov 24, 2023 •

edited

Loading

github-actions bot commented Nov 24, 2023 •

edited

Loading

codecov bot commented Nov 26, 2023 •

edited

Loading