Shuffle tasks on hosts with overutilized memory resources #1937
Conversation
baconmania
changed the title
baconmania
force-pushed
the
shuffle-tasks-for-memory
branch
from
9c0b981 to
6e25b84
Compare
There was a problem hiding this comment.
Overall strategy of 'fix the worst one first' seems fine to me for memory vs cpu. Left one comment about breaking out of the loop early if we think the current removed set of tasks should satisfy things. Other than that, my only other thought is if 'most overused' is still the best metric for sorting tasks to shuffle. We've seen some cases where a task using 10% of its cpu still gets shuffled, which seems odd. Though I'm not sure fo a good way around it either
| private boolean shuffleTasksForOverloadedSlaves = false; // recommended 'true' when oversubscribing cpu for larger clusters | ||
| private boolean shuffleTasksForOverloadedSlaves = false; // recommended 'true' when oversubscribing resources for larger clusters | ||
|
|
||
| private double shuffleTasksWhenSlaveMemoryUtilizationPercentageExceeds = 0.82; |
There was a problem hiding this comment.
was there any math behind this value?
There was a problem hiding this comment.
Sort of an arbitrary rule of thumb. Assuming we'd like to set a default "target" memory utilization of 85%, setting this config at 82% should give us enough time to shuffle tasks before actually hitting our target.
|
|
||
| for (TaskIdWithUsage taskIdWithUsage : possibleTasksToShuffle) { | ||
| if (requestsWithShuffledTasks.contains(taskIdWithUsage.getTaskId().getRequestId())) { | ||
| LOG.debug("Request {} already has a shuffling task, skipping", taskIdWithUsage.getTaskId().getRequestId()); | ||
| continue; | ||
| } | ||
| if (cpuOverage <= 0 || shuffledTasksOnSlave > configuration.getMaxTasksToShufflePerHost() || currentShuffleCleanupsTotal >= configuration.getMaxTasksToShuffleTotal()) { | ||
| LOG.debug("Not shuffling any more tasks (overage: {}, shuffledOnHost: {}, totalShuffleCleanups: {})", cpuOverage, shuffledTasksOnSlave, currentShuffleCleanupsTotal); | ||
| if ((mostOverusedResource.overusage <= 0) || shuffledTasksOnSlave > configuration.getMaxTasksToShufflePerHost() || currentShuffleCleanupsTotal >= configuration.getMaxTasksToShuffleTotal()) { |
There was a problem hiding this comment.
cpuOverage <= 0 worked as a condition for 'should this put us back in green?' because we updated it at the end of the loop. I don't currently see mostOverusedResource.overusage getting updated anywhere
|
🚢 |
|
🚢 |
No description provided.