Introduce cluster default and minimum index refresh interval settings #9267

ashking94 · 2023-08-11T15:18:22Z

Description

This PR aims to introduce notion of cluster level default and minimum index refresh interval. More details are present in #9266.

Related Issues

Resolves #9266

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-trigger-bot · 2023-08-11T15:41:36Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 22m 50s

github-actions · 2023-08-11T16:06:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22482/
CommitID: d9d9b2f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-12T08:41:57Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 29m 5s

github-actions · 2023-08-12T08:53:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22570/
CommitID: fe9b338
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-12T08:55:52Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 8s

github-actions · 2023-08-12T09:19:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.classMethod
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

URL: https://build.ci.opensearch.org/job/gradle-check/22572/
CommitID: 418f2b3
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-08-12T09:20:16Z

Codecov Report

Merging #9267 (5b590b2) into main (6a5b464) will decrease coverage by 0.10%.
The diff coverage is 77.58%.

@@             Coverage Diff              @@
##               main    #9267      +/-   ##
============================================
- Coverage     71.12%   71.02%   -0.10%     
+ Complexity    57417    57407      -10     
============================================
  Files          4776     4776              
  Lines        270742   270790      +48     
  Branches      39578    39579       +1     
============================================
- Hits         192558   192328     -230     
- Misses        62044    62277     +233     
- Partials      16140    16185      +45

Files Changed	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	`93.18% <ø> (ø)`
...rc/main/java/org/opensearch/index/IndexModule.java	`82.35% <ø> (ø)`
...c/main/java/org/opensearch/index/IndexService.java	`74.56% <72.22%> (-0.22%)`	⬇️
...in/java/org/opensearch/indices/IndicesService.java	`71.28% <74.07%> (+0.11%)`	⬆️
...h/cluster/metadata/MetadataCreateIndexService.java	`79.24% <90.90%> (-0.32%)`	⬇️
...luster/metadata/MetadataUpdateSettingsService.java	`70.05% <100.00%> (-0.92%)`	⬇️
.../main/java/org/opensearch/index/IndexSettings.java	`86.29% <100.00%> (+0.03%)`	⬆️

... and 448 files with indirect coverage changes

opensearch-trigger-bot · 2023-08-12T09:45:38Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 30m 34s

github-actions · 2023-08-12T09:56:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.pit.DeletePitMultiNodeIT.testDeleteWhileSearch

URL: https://build.ci.opensearch.org/job/gradle-check/22574/
CommitID: 5a92df6
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-08-12T22:18:42Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 13s

github-actions · 2023-08-12T22:33:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22585/
CommitID: eee52f4

server/src/main/java/org/opensearch/indices/IndicesService.java

opensearch-trigger-bot · 2023-08-17T11:04:23Z

Compatibility status:

Checks if related components are compatible with change e1c40b4

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-17T11:05:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22968/
CommitID: c8d48ca
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

linuxpi

LGTM. please add the test for explicit index refresh interval less than cluster minimum

ashking94 · 2023-08-17T11:24:17Z

LGTM. please add the test for explicit index refresh interval less than cluster minimum

There are multiple tests around this already.
Create index test - https://github.com/opensearch-project/OpenSearch/pull/9267/files#diff-03ee2116467c4223c6ff16481ce75d1efbbc1732cc0479cf6e6fcd0a987d2641R145
Update index setting test - https://github.com/opensearch-project/OpenSearch/pull/9267/files#diff-03ee2116467c4223c6ff16481ce75d1efbbc1732cc0479cf6e6fcd0a987d2641R132

github-actions · 2023-08-17T11:28:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22969/
CommitID: f900c82

server/src/main/java/org/opensearch/indices/IndicesService.java

server/src/main/java/org/opensearch/index/IndexSettings.java

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-08-17T17:59:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22986/
CommitID: a381f00

github-actions · 2023-08-17T18:08:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22987/
CommitID: 5b590b2

…#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit 8e95a82) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…#9267) (#9428) (cherry picked from commit 8e95a82) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Kiran Reddy <kkreddy@amazon.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

ashking94 added the skip-changelog label Aug 11, 2023

ashking94 force-pushed the cluster-refresh-setting branch 2 times, most recently from fe9b338 to 418f2b3 Compare August 12, 2023 08:26

ashking94 force-pushed the cluster-refresh-setting branch from 418f2b3 to 5a92df6 Compare August 12, 2023 09:14

ashking94 force-pushed the cluster-refresh-setting branch from 5a92df6 to 0b97873 Compare August 12, 2023 10:21

ashking94 marked this pull request as ready for review August 12, 2023 10:23

ashking94 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159 and ryanbogan as code owners August 12, 2023 10:23

ashking94 force-pushed the cluster-refresh-setting branch from f5d7452 to eee52f4 Compare August 12, 2023 21:49

linuxpi requested changes Aug 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/IndicesService.java Show resolved Hide resolved

ashking94 force-pushed the cluster-refresh-setting branch from eee52f4 to c8d48ca Compare August 17, 2023 10:33

linuxpi approved these changes Aug 17, 2023

View reviewed changes

sachinpkale reviewed Aug 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/IndicesService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/indices/IndicesService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/IndexSettings.java Show resolved Hide resolved

ashking94 force-pushed the cluster-refresh-setting branch from f900c82 to a381f00 Compare August 17, 2023 17:18

Introduce cluster default and minimum index refresh interval settings

5b590b2

Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 force-pushed the cluster-refresh-setting branch from a381f00 to 5b590b2 Compare August 17, 2023 17:25

sachinpkale approved these changes Aug 18, 2023

View reviewed changes

sachinpkale merged commit 8e95a82 into opensearch-project:main Aug 18, 2023

sachinpkale added the backport 2.x Backport to 2.x branch label Aug 18, 2023

opensearch-trigger-bot bot mentioned this pull request Aug 18, 2023

[Backport 2.x] Introduce cluster default and minimum index refresh interval settings #9428

Merged

austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Aug 25, 2023

Introduce cluster default and minimum index refresh interval settings (…

bb324af

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 mentioned this pull request Aug 28, 2023

[DOC] Add documentation for cluster level default and minimum refresh interval settings opensearch-project/documentation-website#4905

Closed

4 tasks

Gaganjuneja pushed a commit to Gaganjuneja/OpenSearch that referenced this pull request Aug 28, 2023

Introduce cluster default and minimum index refresh interval settings (…

8b1d195

…opensearch-project#9267) Signed-off-by: Ashish Singh <ssashish@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce cluster default and minimum index refresh interval settings #9267

Introduce cluster default and minimum index refresh interval settings #9267

ashking94 commented Aug 11, 2023 •

edited

Loading

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

codecov bot commented Aug 12, 2023 •

edited

Loading

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

opensearch-trigger-bot bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

linuxpi left a comment

ashking94 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

Introduce cluster default and minimum index refresh interval settings #9267

Introduce cluster default and minimum index refresh interval settings #9267

Conversation

ashking94 commented Aug 11, 2023 • edited Loading

Description

Related Issues

Check List

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 12, 2023 • edited Loading

Codecov Report

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 12, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 17, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

linuxpi left a comment

Choose a reason for hiding this comment

ashking94 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 commented Aug 11, 2023 •

edited

Loading

codecov bot commented Aug 12, 2023 •

edited

Loading