-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds cluster setting to configure index state management jitter #153
Conversation
Signed-off-by: Clay Downs <downsrob@amazon.com>
d3a2a7c
to
ce9bb6f
Compare
Codecov Report
@@ Coverage Diff @@
## main #153 +/- ##
============================================
- Coverage 75.88% 75.82% -0.07%
+ Complexity 1922 1916 -6
============================================
Files 260 260
Lines 10992 11014 +22
Branches 1720 1721 +1
============================================
+ Hits 8341 8351 +10
- Misses 1699 1705 +6
- Partials 952 958 +6
Continue to review full report at Codecov.
|
.../kotlin/org/opensearch/indexmanagement/indexstatemanagement/settings/ManagedIndexSettings.kt
Outdated
Show resolved
Hide resolved
Signed-off-by: Clay Downs <downsrob@amazon.com>
src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
Show resolved
Hide resolved
src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/model/ManagedIndexConfig.kt
Show resolved
Hide resolved
@@ -179,5 +180,14 @@ class ManagedIndexSettings { | |||
Setting.Property.NodeScope, | |||
Setting.Property.Dynamic | |||
) | |||
|
|||
val JITTER: Setting<Double> = Setting.doubleSetting( | |||
"plugins.index_state_management.jitter", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor: Do you think it makes sense to set this setting at index_management level so it can be applied to other features of index management in future if needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that if a jitter setting is added to the other features of index management then it would make sense to have the setting feature-scoped by adding it as a setting at the transforms or rollups level, as I don't think a user's setting of jitter for index management would directly relate to how that user might want to configure a potential rollup jitter.
* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <aditjind@amazon.com> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Trying something else Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Addresses comments Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <dblock@amazon.com> * Security improvements (#126) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <dblock@dblock.org> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <downsrob@amazon.com> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes detekt issue Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Aditya Jindal <13850971+aditjind@users.noreply.github.com> Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
…search-project#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <downsrob@amazon.com>
Signed-off-by: Clay Downs downsrob@amazon.com
Description of changes: Adds cluster setting to allow users to enable and configure the jitter setting for index state management jobs. The purpose of the jitter is to add a randomized delay to the start time of ISM jobs to make sure that all indices do not become active at the same time. The jitter is stored as a double and represents the maximum percentage of the job interval to add to the job start time. The default jitter is 0.6, and the cluster setting resides under plugins.index_state_management.jitter.
The job scheduler has another configurable cluster setting, jitter_limit, which represents the maximum jitter which may be applied to the job. The jitter cluster setting may be higher than this limit, but jitter times will be capped to the jitter_limit percentage of the job interval. The job scheduler jitter_limit default is 0.6.
At this time, modifying the jitter cluster setting will only apply the changed jitter to newly created jobs.
CheckList:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.