[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #4363

shenh062326 · 2015-02-04T09:45:21Z

If a blockManager has not send heartBeat more than 120s, BlockManagerMasterActor will remove it. But coarseGrainedSchedulerBackend can only remove executor after an DisassociatedEvent. We should expireDeadHosts at HeartbeatReceiver.

shenh062326 · 2015-02-04T09:46:52Z

add [SPARK-5529]

SparkQA · 2015-02-04T09:47:40Z

Test build #26744 has started for PR 4363 at commit c922cb0.

This patch merges cleanly.

SparkQA · 2015-02-04T09:48:39Z

Test build #26744 has finished for PR 4363 at commit c922cb0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T09:48:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26744/
Test FAILed.

sryza · 2015-02-04T18:47:24Z

Hi @shenh062326 , can you remove the dead host expiry in BlockManagerMasterActor so we don't have it on two places?

sryza · 2015-02-04T18:47:45Z

core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala

+        val msg = "Removing Executor " + executorId + " with no recent heart beats: "
+            +(now - lastSeenMs) + "ms exceeds " + slaveTimeout + "ms"
+        logWarning(msg)
+        if (scheduler.isInstanceOf[org.apache.spark.scheduler.TaskSchedulerImpl]) {


I think it would be better to add a method to TaskScheduler for this.

SparkQA · 2015-02-05T01:42:57Z

Test build #26800 has started for PR 4363 at commit b5c0441.

This patch merges cleanly.

SparkQA · 2015-02-05T01:44:02Z

Test build #26800 has finished for PR 4363 at commit b5c0441.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-05T01:44:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26800/
Test FAILed.

SparkQA · 2015-02-05T01:47:56Z

Test build #26802 has started for PR 4363 at commit fb5df97.

This patch merges cleanly.

SparkQA · 2015-02-05T01:48:57Z

Test build #26802 has finished for PR 4363 at commit fb5df97.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-05T01:48:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26802/
Test FAILed.

SparkQA · 2015-02-05T01:52:51Z

Test build #26804 has started for PR 4363 at commit e197e20.

This patch merges cleanly.

SparkQA · 2015-02-05T01:55:14Z

Test build #26804 has finished for PR 4363 at commit e197e20.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-05T01:55:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26804/
Test FAILed.

SparkQA · 2015-02-05T01:57:49Z

Test build #26808 has started for PR 4363 at commit c1dfda1.

This patch merges cleanly.

SparkQA · 2015-02-05T02:01:14Z

Test build #26808 has finished for PR 4363 at commit c1dfda1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-05T02:01:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26808/
Test FAILed.

SparkQA · 2015-02-05T02:02:33Z

Test build #26810 has started for PR 4363 at commit bccd515.

This patch merges cleanly.

SparkQA · 2015-02-05T02:59:42Z

Test build #26810 has finished for PR 4363 at commit bccd515.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-02-20T01:36:07Z

core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala

 import org.apache.spark.util.ActorLogReceive

 /**
 * A heartbeat from executors to the driver. This is a shared message used by several internal
- * components to convey liveness or execution information for in-progress tasks.
+ * components to convey liveness or execution information for in-progress tasks. It will also 
+ * expire the hosts that have not heartbeated for more than spark.driver.executorTimeoutMs.


you'll need to update this comment if you rename the configs

andrewor14 · 2015-02-20T01:36:54Z

@shenh062326 Thanks for addressing the feedback. I think the current solution works, and the comments I left inline are mostly minor. I will merge this once you address all of them.

shenh062326 · 2015-02-26T09:15:54Z

Sorry for late, I will change it.

SparkQA · 2015-02-26T09:22:37Z

Test build #27993 has started for PR 4363 at commit 2dc456e.

This patch merges cleanly.

SparkQA · 2015-02-26T09:24:36Z

Test build #27993 has finished for PR 4363 at commit 2dc456e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-26T09:24:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27993/
Test FAILed.

SparkQA · 2015-02-26T10:02:36Z

Test build #27997 has started for PR 4363 at commit 1a042ff.

This patch merges cleanly.

SparkQA · 2015-02-26T11:20:22Z

Test build #27997 has finished for PR 4363 at commit 1a042ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-26T11:20:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27997/
Test PASSed.

andrewor14 · 2015-02-26T21:02:50Z

core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala

+  // executor ID -> timestamp of when the last heartbeat from this executor was received
+  private val executorLastSeen = new mutable.HashMap[String, Long]
+
+  private val executorTimeout = sc.conf.getLong("spark.network.timeoutMs", 


please use the existing config spark.network.timeout without the Ms. The reason why I suggested using this config is to avoid introducing new ones.

SparkQA · 2015-02-27T01:13:02Z

Test build #28030 has started for PR 4363 at commit 2c9a46a.

This patch merges cleanly.

SparkQA · 2015-02-27T02:32:51Z

Test build #28030 has finished for PR 4363 at commit 2c9a46a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-27T02:32:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28030/
Test PASSed.

andrewor14 · 2015-02-27T02:43:00Z

Great, LGTM thanks for your work @shenh062326 I'm merging this into master.

If a blockManager has not send heartBeat more than 120s, BlockManagerMasterActor will remove it. But coarseGrainedSchedulerBackend can only remove executor after an DisassociatedEvent. We should expireDeadHosts at HeartbeatReceiver. Author: Hong Shen <hongshen@tencent.com> Closes apache#4363 from shenh062326/my_change3 and squashes the following commits: 2c9a46a [Hong Shen] Change some code style. 1a042ff [Hong Shen] Change some code style. 2dc456e [Hong Shen] Change some code style. d221493 [Hong Shen] Fix test failed 7448ac6 [Hong Shen] A minor change in sparkContext and heartbeatReceiver b904aed [Hong Shen] Fix failed test 52725af [Hong Shen] Remove assert in SparkContext.killExecutors 5bedcb8 [Hong Shen] Remove assert in SparkContext.killExecutors a858fb5 [Hong Shen] A minor change in HeartbeatReceiver 3e221d9 [Hong Shen] A minor change in HeartbeatReceiver 6bab7aa [Hong Shen] Change a code style. 07952f3 [Hong Shen] Change configs name and code style. ce9257e [Hong Shen] Fix test failed bccd515 [Hong Shen] Fix test failed 8e77408 [Hong Shen] Fix test failed c1dfda1 [Hong Shen] Fix test failed e197e20 [Hong Shen] Fix test failed fb5df97 [Hong Shen] Remove ExpireDeadHosts in BlockManagerMessages b5c0441 [Hong Shen] Remove expireDeadHosts in BlockManagerMasterActor c922cb0 [Hong Shen] Add expireDeadHosts in HeartbeatReceiver

If a blockManager has not send heartBeat more than 120s, BlockManagerMasterActor will remove it. But coarseGrainedSchedulerBackend can only remove executor after an DisassociatedEvent. We should expireDeadHosts at HeartbeatReceiver. Author: Hong Shen <hongshentencent.com> Closes #4363 from shenh062326/my_change3 and squashes the following commits: 2c9a46a [Hong Shen] Change some code style. 1a042ff [Hong Shen] Change some code style. 2dc456e [Hong Shen] Change some code style. d221493 [Hong Shen] Fix test failed 7448ac6 [Hong Shen] A minor change in sparkContext and heartbeatReceiver b904aed [Hong Shen] Fix failed test 52725af [Hong Shen] Remove assert in SparkContext.killExecutors 5bedcb8 [Hong Shen] Remove assert in SparkContext.killExecutors a858fb5 [Hong Shen] A minor change in HeartbeatReceiver 3e221d9 [Hong Shen] A minor change in HeartbeatReceiver 6bab7aa [Hong Shen] Change a code style. 07952f3 [Hong Shen] Change configs name and code style. ce9257e [Hong Shen] Fix test failed bccd515 [Hong Shen] Fix test failed 8e77408 [Hong Shen] Fix test failed c1dfda1 [Hong Shen] Fix test failed e197e20 [Hong Shen] Fix test failed fb5df97 [Hong Shen] Remove ExpireDeadHosts in BlockManagerMessages b5c0441 [Hong Shen] Remove expireDeadHosts in BlockManagerMasterActor c922cb0 [Hong Shen] Add expireDeadHosts in HeartbeatReceiver Author: Hong Shen <hongshen@tencent.com> Closes #5793 from alexrovner/SPARK-5529-backport-1.3-v2 and squashes the following commits: f238f94 [Hong Shen] [SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver

Add expireDeadHosts in HeartbeatReceiver

c922cb0

shenh062326 changed the title ~~Add expireDeadHosts in HeartbeatReceiver~~ [SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver Feb 4, 2015

sryza reviewed Feb 4, 2015
View reviewed changes

shenh062326 added 2 commits February 5, 2015 09:38

Remove expireDeadHosts in BlockManagerMasterActor

b5c0441

Remove ExpireDeadHosts in BlockManagerMessages

fb5df97

Fix test failed

e197e20

Fix test failed

c1dfda1

Fix test failed

8e77408

Fix test failed

bccd515

andrewor14 reviewed Feb 20, 2015
View reviewed changes

Change some code style.

2dc456e

Change some code style.

1a042ff

andrewor14 reviewed Feb 26, 2015
View reviewed changes

Change some code style.

2c9a46a

asfgit closed this in 18f2098 Feb 27, 2015

This was referenced Apr 28, 2015

SPARK-5529 Backported changes to 1.3 #5745

Closed

SPARK-5529 Backported changes to 1.3 #5747

Closed

alexrovner mentioned this pull request Apr 29, 2015

[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #5793

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #4363

[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #4363

shenh062326 commented Feb 4, 2015

shenh062326 commented Feb 4, 2015

SparkQA commented Feb 4, 2015

SparkQA commented Feb 4, 2015

AmplabJenkins commented Feb 4, 2015

sryza commented Feb 4, 2015

sryza Feb 4, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

andrewor14 Feb 20, 2015

andrewor14 commented Feb 20, 2015

shenh062326 commented Feb 26, 2015

SparkQA commented Feb 26, 2015

SparkQA commented Feb 26, 2015

AmplabJenkins commented Feb 26, 2015

SparkQA commented Feb 26, 2015

SparkQA commented Feb 26, 2015

AmplabJenkins commented Feb 26, 2015

andrewor14 Feb 26, 2015

SparkQA commented Feb 27, 2015

SparkQA commented Feb 27, 2015

AmplabJenkins commented Feb 27, 2015

andrewor14 commented Feb 27, 2015

[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #4363

[SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver #4363

Conversation

shenh062326 commented Feb 4, 2015

shenh062326 commented Feb 4, 2015

SparkQA commented Feb 4, 2015

SparkQA commented Feb 4, 2015

AmplabJenkins commented Feb 4, 2015

sryza commented Feb 4, 2015

sryza Feb 4, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

andrewor14 Feb 20, 2015

Choose a reason for hiding this comment

andrewor14 commented Feb 20, 2015

shenh062326 commented Feb 26, 2015

SparkQA commented Feb 26, 2015

SparkQA commented Feb 26, 2015

AmplabJenkins commented Feb 26, 2015

SparkQA commented Feb 26, 2015

SparkQA commented Feb 26, 2015

AmplabJenkins commented Feb 26, 2015

andrewor14 Feb 26, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 27, 2015

SparkQA commented Feb 27, 2015

AmplabJenkins commented Feb 27, 2015

andrewor14 commented Feb 27, 2015