Make balanced shards allocator timebound #15239

imRishN · 2024-08-14T05:56:13Z

Description

This PR aims to time bound the reroute duration to finish within a specific timeout so that it allows for URGENT priority tasks that would otherwise be waiting in queue.

For instance time taken by rebalance -

% cat elasticsearch.log | grep "to compute"
[2024-08-21T03:44:42,385][WARN ][o.o.c.s.MasterService    ] [a64d304b34ad798faae32932ab14d605] took [7.8m], which is over [10s], to compute cluster state update for [cluster_reroute(reroute after starting shards)]
[2024-08-21T03:52:45,982][WARN ][o.o.c.s.MasterService    ] [a64d304b34ad798faae32932ab14d605] took [7.7m], which is over [10s], to compute cluster state update for [cluster_reroute(reroute after starting shards)]

Hot threads in master -

curl localhost:9200/_nodes/<node id>/hot_threads?interval=5s
   
   99.9% (4.9s out of 5s) cpu usage by thread 'opensearch[<>][clusterManagerService#updateTask][T#1]'
     9/10 snapshots sharing following 20 elements
       app//org.opensearch.cluster.routing.allocation.allocator.LocalShardsBalancer.balanceByWeights(LocalShardsBalancer.java:440)
       app//org.opensearch.cluster.routing.allocation.allocator.LocalShardsBalancer.balance(LocalShardsBalancer.java:204)
       app//org.opensearch.cluster.routing.allocation.allocator.BalancedShardsAllocator.allocate(BalancedShardsAllocator.java:324)
       app//org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:576)
       app//org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:538)
       app//org.opensearch.node.Node$$Lambda$2639/0x0000001800a9ec70.apply(Unknown Source)
       app//org.opensearch.cluster.routing.BatchedRerouteService$1.execute(BatchedRerouteService.java:136)
       app//org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67)
       app//org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:882)
       app//org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:434)
       app//org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:301)
       app//org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:212)
       app//org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:209)
       app//org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:247)
       app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:863)
       app//org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
       app//org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246)
       java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@17.0.9/java.lang.Thread.run(Thread.java:840)
     unique snapshot
       app//org.opensearch.cluster.routing.allocation.allocator.LocalShardsBalancer.balanceByWeights(LocalShardsBalancer.java:470)
       app//org.opensearch.cluster.routing.allocation.allocator.LocalShardsBalancer.balance(LocalShardsBalancer.java:204)
       app//org.opensearch.cluster.routing.allocation.allocator.BalancedShardsAllocator.allocate(BalancedShardsAllocator.java:324)
       app//org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:576)
       app//org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:538)
       app//org.opensearch.node.Node$$Lambda$2639/0x0000001800a9ec70.apply(Unknown Source)
       app//org.opensearch.cluster.routing.BatchedRerouteService$1.execute(BatchedRerouteService.java:136)
       app//org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67)
       app//org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:882)
       app//org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:434)
       app//org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:301)
       app//org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:212)
       app//org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:209)
       app//org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:247)
       app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:863)
       app//org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
       app//org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246)
       java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@17.0.9/java.lang.Thread.run(Thread.java:840)

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable. - [DOC] Add documentation related to allocator time out documentation-website#8086

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-14T06:07:37Z

❌ Gradle check result for 0e9151c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-15T17:19:35Z

❌ Gradle check result for 7fe10d7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-16T17:28:49Z

❌ Gradle check result for 8f558d7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-27T08:26:40Z

✅ Gradle check result for 87a5cfc: SUCCESS

codecov · 2024-08-27T08:28:33Z

Codecov Report

Attention: Patch coverage is 79.48718% with 8 lines in your changes missing coverage. Please review.

Project coverage is 71.93%. Comparing base (46a269e) to head (75145b7).
Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
.../allocation/allocator/BalancedShardsAllocator.java	76.47%	3 Missing and 1 partial ⚠️
...ting/allocation/allocator/LocalShardsBalancer.java	78.94%	0 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15239      +/-   ##
============================================
+ Coverage     71.88%   71.93%   +0.05%     
- Complexity    63242    63263      +21     
============================================
  Files          5224     5224              
  Lines        296137   296174      +37     
  Branches      42777    42785       +8     
============================================
+ Hits         212881   213067     +186     
+ Misses        65784    65613     -171     
- Partials      17472    17494      +22

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-27T09:36:31Z

❌ Gradle check result for fc9a8ff: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-27T11:18:02Z

❌ Gradle check result for 9a101b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

Bukhtawar

Thanks for the changes, lets add an integ test for the same with deterministic mechanisms to trigger timeouts

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-28T15:22:22Z

❌ Gradle check result for 72a10b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-29T10:32:14Z

✅ Gradle check result for 3ba1615: SUCCESS

github-actions · 2024-08-29T10:32:20Z

❕ Gradle check result for c18d44c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Bukhtawar

Do we need an explicit reroute?

imRishN · 2024-08-29T11:38:54Z

Do we need an explicit reroute?

Usually, if in the current round of reroute any amount of work was attempted (like allocating an unassigned shard, or moving a shard or rebalancing a shard), a reroute will eventually be triggered. If no work was done (allocating no unassigned shards, moving no shards, rebalancing no shards) a subsequent reroute would also be most probably wasteful. But there could be edge cases where a subsequent reroute might help. Opened issue to track this - #14945

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2024-08-29T12:51:20Z

✅ Gradle check result for 75145b7: SUCCESS

* Make balanced shards allocator time bound to prioritise critical operations waiting in the pending task queue Signed-off-by: Rishab Nahata <rnnahata@amazon.com> (cherry picked from commit e982a16) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Make balanced shards allocator time bound to prioritise critical operations waiting in the pending task queue Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Make balanced shards allocator timebound

0e9151c

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Randomise nodes shards iterator for move

7fe10d7

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Terminate rebalancing if time elapsed

8f558d7

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Add test for allocate unassigned time out

d591b26

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN marked this pull request as ready for review August 20, 2024 08:21

Add timer test

fc9a8ff

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Trigger Build

9a101b9

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar reviewed Aug 27, 2024

View reviewed changes

...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Aug 27, 2024

View reviewed changes

...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Aug 27, 2024

View reviewed changes

Remove null

72a10b4

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 2 commits August 29, 2024 15:04

Fix test

c18d44c

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix PR number

3ba1615

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar approved these changes Aug 29, 2024

View reviewed changes

Refactor

75145b7

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar merged commit e982a16 into opensearch-project:main Aug 29, 2024
34 checks passed

opensearch-trigger-bot bot mentioned this pull request Aug 29, 2024

[Backport 2.x] Make balanced shards allocator timebound #15503

Merged

opensearch-ci-bot mentioned this pull request Sep 6, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MasterServiceTests #15809

Open

BrewTestBot mentioned this pull request Sep 18, 2024

opensearch 2.17.0 Homebrew/homebrew-core#191052

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make balanced shards allocator timebound #15239

Make balanced shards allocator timebound #15239

imRishN commented Aug 14, 2024 •

edited

Loading

github-actions bot commented Aug 14, 2024

github-actions bot commented Aug 15, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 27, 2024

codecov bot commented Aug 27, 2024 •

edited

Loading

github-actions bot commented Aug 27, 2024

github-actions bot commented Aug 27, 2024

Bukhtawar left a comment

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

Bukhtawar left a comment

imRishN commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Aug 29, 2024

Make balanced shards allocator timebound #15239

Make balanced shards allocator timebound #15239

Conversation

imRishN commented Aug 14, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 14, 2024

github-actions bot commented Aug 15, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 27, 2024

codecov bot commented Aug 27, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 27, 2024

github-actions bot commented Aug 27, 2024

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

Bukhtawar left a comment

Choose a reason for hiding this comment

imRishN commented Aug 29, 2024 • edited Loading

github-actions bot commented Aug 29, 2024

imRishN commented Aug 14, 2024 •

edited

Loading

codecov bot commented Aug 27, 2024 •

edited

Loading

imRishN commented Aug 29, 2024 •

edited

Loading