Adding index create block when all nodes have breached high disk watermark #5852

RS146BIJAY · 2023-01-13T09:04:39Z

…rmark

Signed-off-by: Rishav Sagar rissag@amazon.com

Description

OpenSearch stops allocating any shards to nodes that have breached High Disk Watermark. If a scenario arises that all the nodes in the cluster breached high disk watermark, no new shards will be created on this cluster. Now if we try to create a new index on this cluster, it will be a red index (since no primary or replica shards can be created for this index).

In order to prevent red cluster, we propose that whenever all the nodes in the cluster has breached high disk watermark we apply a INDEX CREATE BLOCK on the cluster to prevent creation of any red indices. We will modify the DiskThresholdMonitor (which monitors for disk watermarks thresholds on a domain and take appropriate action) to include an extra check on whether low disk watermark is breached on all the nodes in cluster. If it has, it will apply an index create block on the entire cluster.

Issues Resolved

#4456

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-13T09:13:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9369/
CommitID: 41ea477
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T09:36:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9370/
CommitID: 901cfc1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T09:42:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9371/
CommitID: fd39829
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T09:50:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9372/
CommitID: 76b9782
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T11:32:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9376/
CommitID: dee1ea0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T12:00:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9379/
CommitID: 79bade6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T15:00:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testCancelPrimaryAllocation
      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancelPrimaryAllocation

URL: https://build.ci.opensearch.org/job/gradle-check/9382/
CommitID: 4ee5693
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-01-13T15:06:35Z

Codecov Report

Merging #5852 (0c5ed1f) into main (3e2d8cc) will increase coverage by 0.27%.
The diff coverage is 17.39%.

@@             Coverage Diff              @@
##               main    #5852      +/-   ##
============================================
+ Coverage     70.57%   70.84%   +0.27%     
- Complexity    58486    58730     +244     
============================================
  Files          4770     4770              
  Lines        280721   280743      +22     
  Branches      40536    40539       +3     
============================================
+ Hits         198123   198901     +778     
+ Misses        66228    65502     -726     
+ Partials      16370    16340      -30

Impacted Files	Coverage Δ
...uster/routing/allocation/DiskThresholdMonitor.java	`81.73% <17.39%> (-6.09%)`	⬇️
...g/opensearch/index/analysis/CharFilterFactory.java	`0.00% <0.00%> (-100.00%)`	⬇️
...rch/test/junit/listeners/ReproduceInfoPrinter.java	`9.52% <0.00%> (-68.26%)`	⬇️
.../java/org/opensearch/search/dfs/AggregatedDfs.java	`51.61% <0.00%> (-45.17%)`	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	`22.22% <0.00%> (-44.45%)`	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	`60.00% <0.00%> (-40.00%)`	⬇️
...ava/org/opensearch/search/dfs/DfsSearchResult.java	`47.87% <0.00%> (-39.37%)`	⬇️
...luster/routing/allocation/RoutingExplanations.java	`62.06% <0.00%> (-37.94%)`	⬇️
...ion/admin/cluster/node/info/PluginsAndModules.java	`53.12% <0.00%> (-34.38%)`	⬇️
...cluster/routing/allocation/RerouteExplanation.java	`70.00% <0.00%> (-30.00%)`	⬇️
... and 522 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-01-13T15:48:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9383/
CommitID: a8787d6

github-actions · 2023-01-13T16:08:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9384/
CommitID: 888c720
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-13T16:39:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancelPrimaryAllocation
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9385/
CommitID: b5b12f3
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-13T19:33:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9392/
CommitID: dceeb82
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T20:10:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testCancellation
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes

URL: https://build.ci.opensearch.org/job/gradle-check/9394/
CommitID: e6ac695
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-18T21:07:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9553/
CommitID: 08ae9be

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java

…rmark Signed-off-by: Rishav Sagar <rissag@amazon.com>

github-actions · 2023-01-19T10:37:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9586/
CommitID: b8d232d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T11:03:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9587/
CommitID: f401496

Bukhtawar · 2023-01-19T16:42:05Z

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java

-        ActionListener<Void> wrappedListener = ActionListener.wrap(r -> {
-            setLastRunTimeMillis();
-            listener.onResponse(r);
-        }, e -> {
+        ActionListener<Void> wrappedListener = ActionListener.wrap(r -> { listener.onResponse(r); }, e -> {


for my understanding y did we remove last run time?

As commented by Gaurav here, setLastRunTimeMillis() is only meant for reroute operations as it needs to honor diskThresholdSettings.getRerouteInterval().millis(), so removing it from updateIndicesReadOnly as well.

If we remove setLastRunTimeMillis it will try to reroute once even though we don't need to reroute.

Addressed the comment

Signed-off-by: Rishav Sagar <rissag@amazon.com>

github-actions · 2023-01-19T19:03:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9603/
CommitID: 0c5ed1f

…rmark (#5852) * Adding index create block when all nodes have breached high disk watermark Signed-off-by: Rishav Sagar <rissag@amazon.com> (cherry picked from commit 6309611) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rmark (#5852) (#6165) * Adding index create block when all nodes have breached high disk watermark (cherry picked from commit 6309611) Signed-off-by: Rishav Sagar <rissag@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

dennisoelkers · 2023-05-08T14:25:01Z

@RS146BIJAY: Will the index creation cluster block be lifted automatically when one or more nodes fall below the watermark again?

RS146BIJAY · 2023-05-08T16:12:04Z

Yes. See this: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java#L386

RS146BIJAY force-pushed the index-block-guardrails branch 3 times, most recently from fd39829 to 76b9782 Compare January 13, 2023 09:32

RS146BIJAY force-pushed the index-block-guardrails branch from 76b9782 to dee1ea0 Compare January 13, 2023 11:05

RS146BIJAY force-pushed the index-block-guardrails branch from dee1ea0 to 79bade6 Compare January 13, 2023 11:43

RS146BIJAY force-pushed the index-block-guardrails branch from 79bade6 to 4ee5693 Compare January 13, 2023 14:25

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from a8787d6 to 888c720 Compare January 13, 2023 15:30

RS146BIJAY force-pushed the index-block-guardrails branch from 888c720 to b5b12f3 Compare January 13, 2023 16:04

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from dceeb82 to b8dfc29 Compare January 13, 2023 19:15

RS146BIJAY force-pushed the index-block-guardrails branch from b8dfc29 to e6ac695 Compare January 13, 2023 19:39

RS146BIJAY marked this pull request as ready for review January 13, 2023 20:18

RS146BIJAY requested review from reta, anasalkouz, andrross and Bukhtawar as code owners January 13, 2023 20:18

RS146BIJAY force-pushed the index-block-guardrails branch from 6ba0f38 to 08ae9be Compare January 18, 2023 20:38

gbbafna reviewed Jan 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/allocation/DiskThresholdMonitor.java Show resolved Hide resolved

Adding index create block when all nodes have breached high disk wate…

fd5c656

…rmark Signed-off-by: Rishav Sagar <rissag@amazon.com>

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from b8d232d to f401496 Compare January 19, 2023 10:33

gbbafna approved these changes Jan 19, 2023

View reviewed changes

Arpit-Bandejiya approved these changes Jan 19, 2023

View reviewed changes

Bukhtawar reviewed Jan 19, 2023

View reviewed changes

Fixing flaky test cases and refactoring code

0c5ed1f

Signed-off-by: Rishav Sagar <rissag@amazon.com>

RS146BIJAY force-pushed the index-block-guardrails branch from f401496 to 0c5ed1f Compare January 19, 2023 18:30

Bukhtawar approved these changes Jan 19, 2023

View reviewed changes

Bukhtawar merged commit 6309611 into opensearch-project:main Jan 19, 2023

andrross added the backport 2.x Backport to 2.x branch label Jan 19, 2023

opensearch-trigger-bot bot mentioned this pull request Jan 19, 2023

[Backport 2.x] Adding index create block when all nodes have breached high disk watermark #5943

Closed

andrross mentioned this pull request Jan 20, 2023

DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock is flaky #5956

Closed

andrross added the v2.6.0 'Issues and PRs related to version v2.6.0' label Jan 26, 2023

andrross mentioned this pull request Jan 26, 2023

Add support for index create block #4607

Merged

6 tasks

andrross added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Feb 2, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 2, 2023

[Backport 2.x] Adding index create block when all nodes have breached high disk watermark #6165

Merged

RS146BIJAY mentioned this pull request Aug 20, 2023

Fixing unblock condition for index create block #9437

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding index create block when all nodes have breached high disk watermark #5852

Adding index create block when all nodes have breached high disk watermark #5852

RS146BIJAY commented Jan 13, 2023 •

edited

Loading

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

codecov-commenter commented Jan 13, 2023 •

edited

Loading

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

Bukhtawar Jan 19, 2023

RS146BIJAY Jan 19, 2023 •

edited

Loading

Bukhtawar Jan 19, 2023

RS146BIJAY Jan 19, 2023

github-actions bot commented Jan 19, 2023

dennisoelkers commented May 8, 2023

RS146BIJAY commented May 8, 2023

Adding index create block when all nodes have breached high disk watermark #5852

Adding index create block when all nodes have breached high disk watermark #5852

Conversation

RS146BIJAY commented Jan 13, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jan 13, 2023 • edited Loading

Codecov Report

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar Jan 19, 2023

Choose a reason for hiding this comment

RS146BIJAY Jan 19, 2023 • edited Loading

Choose a reason for hiding this comment

Bukhtawar Jan 19, 2023

Choose a reason for hiding this comment

RS146BIJAY Jan 19, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

dennisoelkers commented May 8, 2023

RS146BIJAY commented May 8, 2023

RS146BIJAY commented Jan 13, 2023 •

edited

Loading

codecov-commenter commented Jan 13, 2023 •

edited

Loading

RS146BIJAY Jan 19, 2023 •

edited

Loading