[GLUTEN-8018][VL] Support adjusting stage resource profile dynamically #8209

zjuwangg · 2024-12-11T08:48:16Z

What changes were proposed in this pull request?

It's a follow PR of #8195 and demonstrate how to adjust resource profile through Rule.

How was this patch tested?

Unit test will be updated soon.

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

github-actions · 2024-12-11T08:48:35Z

#8018

github-actions · 2024-12-11T08:48:48Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2024-12-12T06:55:01Z

Thank you for splitting the PR into 2. Let's discuss in #8195 before moving forward to this one.

github-actions · 2024-12-17T13:29:18Z

Run Gluten Clickhouse CI on x86

zhztheplayer

Can we add some tests in this PR? Since the feature seems functional to users after the change. Thanks!

Also, could change [CORE] to [VL] in PR title if this is not ready for CH use yet.

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala

github-actions · 2024-12-18T15:53:42Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-18T17:54:34Z

Run Gluten Clickhouse CI on x86

FelixYBW · 2024-12-19T00:09:30Z

Thank you, @zjuwangg

Since it's a big feature. Would you like to create a doc for the featuer in docs? Or add in a follow up PR?

zjuwangg · 2024-12-19T02:53:37Z

Thank you, @zjuwangg

Since it's a big feature. Would you like to create a doc for the featuer in docs? Or add in a follow up PR?

Yeah, of course. I'll add detailed doc in the follow up PR.

github-actions · 2024-12-19T02:57:00Z

Run Gluten Clickhouse CI on x86

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala

github-actions · 2024-12-25T16:50:55Z

Run Gluten Clickhouse CI on x86

shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala

github-actions · 2024-12-26T05:21:32Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-02T10:10:05Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-13T02:33:55Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-01-15T04:56:12Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-01-15T06:12:04Z

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala

+    val c2RorR2CCnt = planNodes.count(
+      p => p.isInstanceOf[ColumnarToRowTransition] || p.isInstanceOf[RowToColumnarTransition])
+    val totalCount = planNodes.size
+
+    if (1.0 * c2RorR2CCnt / totalCount >= glutenConf.autoAdjustStageC2RorR2CRatioThreshold) {
+      val newMemoryAmount = memoryRequest.get.amount * glutenConf.autoAdjustStageRPHeapRatio;
+      val newExecutorMemory =
+        new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong)
+      executorResource.put(ResourceProfile.MEMORY, newExecutorMemory)
+      val newRP = new ResourceProfile(executorResource.toMap, taskResource.toMap)
+      return applyNewResourceProfileIfPossible(plan, newRP, rpManager)
+    }
+    plan


So we are still collecting c2RorR2CCnt? While @PHILO-HE suggested to count on fallen back nodes. Do you have any comment here? Thanks.

So we are still collecting c2RorR2CCnt? While @PHILO-HE suggested to count on fallen back nodes. Do you have any comment here? Thanks.

Good catch. I mistake the fallen node with r2cOrC2RCnt. Will address soon!

github-actions · 2025-01-15T10:09:58Z

Run Gluten Clickhouse CI on x86

zhztheplayer

Basically the code structure looks good to me. I didn't test or verify it, do you have any test result about the feature? Thanks.

...s-velox/src/test/scala/org/apache/gluten/execution/AutoAdjustStageResourceProfileSuite.scala

zjuwangg · 2025-01-17T10:52:59Z

Basically the code structure looks good to me. I didn't test or verify it, do you have any test result about the feature? Thanks.

Yes. Worked in our inner version in an similar way.

zhztheplayer · 2025-01-21T05:08:27Z

I didn't verify the change so let's hold on for some time to see if anyone likes to review before proceeding to merge. Thank you for the contribution in advance.

zhztheplayer · 2025-01-23T11:39:41Z

@zjuwangg I'm merging and since there's no one else helping verify this yet, would you like to mark the following as experimental with a new PR or so? Thanks.

Scala class AutoAdjustStageResourceProfileSuite
The config options added by this PR

zhztheplayer · 2025-01-23T11:40:37Z

@zjuwangg I'm merging and since there's no one else helping verify this yet, would you like to mark the following as experimental with a new PR or so? Thanks.

Scala class AutoAdjustStageResourceProfileSuite

The config options added by this PR

Oops, the PR has confilicts. So we can do this together with a rebase I guess.

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala

...s-velox/src/test/scala/org/apache/gluten/execution/AutoAdjustStageResourceProfileSuite.scala

…esource profile

github-actions · 2025-01-24T03:46:02Z

Run Gluten Clickhouse CI on x86

zhztheplayer

LGTM from my side

cc @jackylee-ch if you have further comments

jackylee-ch

LGTM

github-actions bot added CORE works for Gluten Core VELOX CLICKHOUSE labels Dec 11, 2024

zjuwangg mentioned this pull request Dec 11, 2024

[GLUTEN-8018][CORE] Introduce ApplyResourceProfileExec to apply resource profile for query stage #8195

Merged

zhztheplayer changed the title ~~[GLUTEN-8018]Introduce GlutenAutoAdjustStageResourceProfileRule to auto adjust stage resource profile~~ [GLUTEN-8018] Introduce GlutenAutoAdjustStageResourceProfileRule to auto adjust stage resource profile Dec 12, 2024

PHILO-HE changed the title ~~[GLUTEN-8018] Introduce GlutenAutoAdjustStageResourceProfileRule to auto adjust stage resource profile~~ [GLUTEN-8018][CORE] Support adjust stage resource profile dynamically Dec 17, 2024

PHILO-HE changed the title ~~[GLUTEN-8018][CORE] Support adjust stage resource profile dynamically~~ [GLUTEN-8018][CORE] Support adjusting stage resource profile dynamically Dec 17, 2024

zjuwangg force-pushed the m_stageRP_2 branch from 8073c47 to 7786636 Compare December 17, 2024 13:28

github-actions bot removed the CLICKHOUSE label Dec 17, 2024

zhztheplayer reviewed Dec 18, 2024

View reviewed changes

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala Outdated Show resolved Hide resolved

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala Outdated Show resolved Hide resolved

zjuwangg force-pushed the m_stageRP_2 branch from 7786636 to 38dde01 Compare December 18, 2024 15:53

zjuwangg changed the title ~~[GLUTEN-8018][CORE] Support adjusting stage resource profile dynamically~~ [GLUTEN-8018][VL] Support adjusting stage resource profile dynamically Dec 18, 2024

PHILO-HE reviewed Dec 20, 2024

View reviewed changes

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala Outdated Show resolved Hide resolved

zjuwangg force-pushed the m_stageRP_2 branch from e34ed0f to 2ea2934 Compare December 25, 2024 16:50

zjuwangg requested review from PHILO-HE and zhztheplayer December 25, 2024 16:53

jackylee-ch reviewed Dec 26, 2024

View reviewed changes

shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala Outdated Show resolved Hide resolved

...ait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala Outdated Show resolved Hide resolved

zjuwangg requested a review from jackylee-ch December 27, 2024 05:36

zjuwangg requested a review from zhztheplayer January 15, 2025 03:26

zhztheplayer reviewed Jan 15, 2025

View reviewed changes

zjuwangg requested a review from zhztheplayer January 17, 2025 06:57

zhztheplayer reviewed Jan 17, 2025

View reviewed changes

...s-velox/src/test/scala/org/apache/gluten/execution/AutoAdjustStageResourceProfileSuite.scala Outdated Show resolved Hide resolved

zhztheplayer approved these changes Jan 21, 2025

View reviewed changes

zhztheplayer added the ready to merge label Jan 21, 2025

jackylee-ch reviewed Jan 23, 2025

View reviewed changes

zjuwangg added 10 commits January 24, 2025 11:37

Introduce GlutenAutoAdjustStageResourceProfile to auto adjust stage r…

58cb17f

…esource profile

address comments and add test suit

3db937f

fix style problem

bf693d0

address comments and fix ut failure

1e2b690

address comments

cb7dd00

rebase and address comments

abfe159

fix compile problem

c7a4b61

fix compile problem

d557db1

address comments

0a1925c

address comments

d584637

zjuwangg force-pushed the m_stageRP_2 branch from 3c8a767 to d584637 Compare January 24, 2025 03:45

zhztheplayer approved these changes Jan 24, 2025

View reviewed changes

jackylee-ch approved these changes Jan 24, 2025

View reviewed changes

jackylee-ch merged commit 291ff35 into apache:main Jan 24, 2025
48 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-8018][VL] Support adjusting stage resource profile dynamically #8209

[GLUTEN-8018][VL] Support adjusting stage resource profile dynamically #8209

zjuwangg commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

zhztheplayer commented Dec 12, 2024

github-actions bot commented Dec 17, 2024

zhztheplayer left a comment

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

FelixYBW commented Dec 19, 2024

zjuwangg commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 25, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Jan 2, 2025

github-actions bot commented Jan 13, 2025

zhztheplayer commented Jan 15, 2025

zhztheplayer Jan 15, 2025

zjuwangg Jan 15, 2025

github-actions bot commented Jan 15, 2025

zhztheplayer left a comment

zjuwangg commented Jan 17, 2025

zhztheplayer commented Jan 21, 2025

zhztheplayer commented Jan 23, 2025

zhztheplayer commented Jan 23, 2025

github-actions bot commented Jan 24, 2025

zhztheplayer left a comment

jackylee-ch left a comment

[GLUTEN-8018][VL] Support adjusting stage resource profile dynamically #8209

[GLUTEN-8018][VL] Support adjusting stage resource profile dynamically #8209

Conversation

zjuwangg commented Dec 11, 2024

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

zhztheplayer commented Dec 12, 2024

github-actions bot commented Dec 17, 2024

zhztheplayer left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

FelixYBW commented Dec 19, 2024

zjuwangg commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 25, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Jan 2, 2025

github-actions bot commented Jan 13, 2025

zhztheplayer commented Jan 15, 2025

zhztheplayer Jan 15, 2025

Choose a reason for hiding this comment

zjuwangg Jan 15, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 15, 2025

zhztheplayer left a comment

Choose a reason for hiding this comment

zjuwangg commented Jan 17, 2025

zhztheplayer commented Jan 21, 2025

zhztheplayer commented Jan 23, 2025

zhztheplayer commented Jan 23, 2025

github-actions bot commented Jan 24, 2025

zhztheplayer left a comment

Choose a reason for hiding this comment

jackylee-ch left a comment

Choose a reason for hiding this comment