[WIP]SPARK-1712: TaskDescription instance is too big causes Spark to hang #677
Conversation
|
Can one of the admins verify this patch? |
| @@ -414,6 +415,14 @@ private[spark] class TaskSetManager( | |||
| // we assume the task can be serialized without exceptions. | |||
| val serializedTask = Task.serializeWithDependencies( | |||
| task, sched.sc.addedFiles, sched.sc.addedJars, ser) | |||
| if (serializedTask.limit >= akkaFrameSize - 1024) { | |||
There was a problem hiding this comment.
The reference Executor.scala#L235.may not suitable for here
There was a problem hiding this comment.
@kayousterhout - you added this. Any reason you left the 1024 of wiggle room?
There was a problem hiding this comment.
real example:
serialized task with dependencies |
serialized LaunchTask |
|---|---|
| 4356 | 4797 |
| 2690585 | 2704141 |
| 5238584 | 5264580 |
The difference has been increasing
There was a problem hiding this comment.
ah I see - it's to leave room for the other contents of the message. In the second case here, do you know what is taking all the extra room? 2704141 - 2690585 = ~13KB which is very large for a few string/int fields!
There was a problem hiding this comment.
Yes, the problem is the value of uncertainty
| @@ -414,6 +415,14 @@ private[spark] class TaskSetManager( | |||
| // we assume the task can be serialized without exceptions. | |||
| val serializedTask = Task.serializeWithDependencies( | |||
| task, sched.sc.addedFiles, sched.sc.addedJars, ser) | |||
| if (serializedTask.limit >= akkaFrameSize - 1024) { | |||
| val msg = "Serialized task %s:%d were %d bytes which " + | |||
There was a problem hiding this comment.
It would be nicer to use string interpolation here to make it easier to parse.
There was a problem hiding this comment.
The reason for this is to keep the same style of the code.
|
Thanks for tracking down this fix. It looks good - added some small comments. |
witgo
changed the title
witgo
changed the title
witgo
changed the title
|
@pwendell |
No description provided.