Add changes to block calls in cat shards, indices and segments based on dynamic limit settings

[sumitasr]

Add changes to block calls in cat shards, indices and segments based on dynamic limit settings

Description

• Introduced a new class RequestLimitSettings to maintain dynamic settings for 3 actions - cat indices, shards and segments and method to evaluate if circuitLimitBreached for an action.

  • cat.indices.limit (The limit will be applied on number of indices)
  • cat.shards.limit (The limit will be applied on number of shards)
  • cat.segments.limit (The limit will be applied on number of indices)

• Introduced isRequestLimitCheckSupported method defaulting to false in AbstractCatAction so that we can override this if an action extending AbstractCatAction can define if it supports limit or circuit breaker checks introduced in RequestLimitSettings class. This value can be passed to respected transport action as param e.g. TransportCatShardsAction

• If circuit limit breached, CircuitBreakingException will be raised which is mapped to 429 error code.

Related Issues

Resolves #15954

Functional Testing

Action 1 : /_cat/shards/

Step 1 - Run OS cluster locally

Step 2 - PUT 2 index

Step 3 - Perform /_cat/shards/

curl -XGET "http://localhost:9200/_cat/shards/"
3m3y-index21 0 p STARTED 1 4.7kb 127.0.0.1 runTask-0
3m3y-index21 0 r UNASSIGNED
index2 0 p STARTED 1 4.7kb 127.0.0.1 runTask-0
index2 0 r UNASSIGNED

Step 4 - Update setting cat.shards.limit to 2

time curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient" : {
"cat.shards.limit" : 2
}
}'
{"acknowledged":true,"persistent":{},"transient":{"cat":{"shards":{"limit":"2"}}}}curl -X PUT "localhost:9200/_cluster/settings" -H 0.00s user 0.01s system 12% cpu 0.100 total

Step 5 - Perform /_cat/shards/

curl -XGET "http://localhost:9200/_cat/shards/"

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"Too many shards requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"Too many shards requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},"status":429}%

Step 6 - Update setting cat.shards.limit to 5

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient" : {
"cat.shards.limit" : 5
}
}'
{"acknowledged":true,"persistent":{},"transient":{"cat":{"shards":{"limit":"5"}}}}%

Step 7 - Perform /_cat/shards/

curl -XGET "http://localhost:9200/_cat/shards/"
3m3y-index21 0 p STARTED 1 4.7kb 127.0.0.1 runTask-0
3m3y-index21 0 r UNASSIGNED
index2 0 p STARTED 1 4.7kb 127.0.0.1 runTask-0
index2 0 r UNASSIGNED

Action 2 : /_cat/indices/

Step 1 - Using cluster started in previous section

Step 2 - Perform /_cat/indices/

curl -XGET "http://localhost:9200/_cat/indices/"
yellow open 3m3y-index21 5P24Sl2hTee4mhqdgu8cHQ 1 1 1 0 4.8kb 4.8kb
yellow open index2 clEIcfQpQUOLXZdWt_AqgA 1 1 1 0 4.8kb 4.8kb

Step 3 - Update settings

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient" : {
"cat.indices.limit" : 1
}
}'
{"acknowledged":true,"persistent":{},"transient":{"cat":{"indices":{"limit":"1"}}}}%

Step 4 - Perform /_cat/indices/

curl -XGET "http://localhost:9200/_cat/indices/"
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"Too many indices requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"Too many indices requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},"status":429}%

Step 5 - Update settings
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient" : {
"cat.indices.limit" : -1
}
}'

Step 6 - Perform /_cat/indices/

curl -XGET "http://localhost:9200/_cat/indices/"
yellow open 3m3y-index21 5P24Sl2hTee4mhqdgu8cHQ 1 1 1 0 4.8kb 4.8kb
yellow open index2 clEIcfQpQUOLXZdWt_AqgA 1 1 1 0 4.8kb 4.8kb

Action 3 : /_cat/segments/

Step 1 - Using cluster started in previous section

Step 2 - Perform /_cat/segments/

curl -XGET "http://localhost:9200/_cat/segments/"
3m3y-index21 0 p 127.0.0.1 _0 0 1 0 4.5kb 0 true true 9.12.0 true
index2 0 p 127.0.0.1 _0 0 1 0 4.5kb 0 true true 9.12.0 true

Step3 - Update settings

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient" : {
"cat.segments.limit" : 1
}
}'
{"acknowledged":true,"persistent":{},"transient":{"cat":{"segments":{"limit":"1"}}}}%

Step 4 - Perform /_cat/segments/
curl -XGET "http://localhost:9200/_cat/segments/"
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"Segments from too many indices requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"Segments from too many indices requested.","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},"status":429}%

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 8796ab6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 59b896a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ec96cdd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 973f3be: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1944512: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 515a699: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 27440d2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[sumitasr]

❌ Gradle check result for 27440d2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Reason - failed for task ':server:spotlessJavaCheck'.

Pushed fix post running ./gradlew :server:spotlessApply

[github-actions]

❌ Gradle check result for 454e2dd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8a2140f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 399fb59: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 7a7ae4b: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 67.44186% with 28 lines in your changes missing coverage. Please review.

Project coverage is 71.90%. Comparing base (3937ccb) to head (7a7ae4b).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
.../opensearch/rest/action/cat/RestIndicesAction.java	16.66%	15 Missing ⚠️
...opensearch/rest/action/cat/RestSegmentsAction.java	42.85%	4 Missing ⚠️
...admin/cluster/shards/TransportCatShardsAction.java	25.00%	3 Missing ⚠️
...java/org/opensearch/rest/RequestLimitSettings.java	93.47%	2 Missing and 1 partial ⚠️
.../action/admin/cluster/shards/CatShardsRequest.java	50.00%	2 Missing ⚠️
.../opensearch/rest/action/cat/AbstractCatAction.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15986      +/-   ##
============================================
- Coverage     71.90%   71.90%   -0.01%     
+ Complexity    64392    64368      -24     
============================================
  Files          5278     5281       +3     
  Lines        300877   300947      +70     
  Branches      43478    43492      +14     
============================================
+ Hits         216351   216390      +39     
- Misses        66747    66786      +39     
+ Partials      17779    17771       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for 66c962b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?