Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]SPARK-1712: TaskDescription instance is too big causes Spark to hang #677

Closed
wants to merge 4 commits into from

Conversation

witgo
Copy link
Contributor

@witgo witgo commented May 7, 2014

No description provided.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@@ -414,6 +415,14 @@ private[spark] class TaskSetManager(
// we assume the task can be serialized without exceptions.
val serializedTask = Task.serializeWithDependencies(
task, sched.sc.addedFiles, sched.sc.addedJars, ser)
if (serializedTask.limit >= akkaFrameSize - 1024) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the 1024 for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reference Executor.scala#L235.may not suitable for here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kayousterhout - you added this. Any reason you left the 1024 of wiggle room?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

real example:

serialized task with dependencies serialized LaunchTask
4356 4797
2690585 2704141
5238584 5264580

The difference has been increasing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see - it's to leave room for the other contents of the message. In the second case here, do you know what is taking all the extra room? 2704141 - 2690585 = ~13KB which is very large for a few string/int fields!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the problem is the value of uncertainty

@@ -414,6 +415,14 @@ private[spark] class TaskSetManager(
// we assume the task can be serialized without exceptions.
val serializedTask = Task.serializeWithDependencies(
task, sched.sc.addedFiles, sched.sc.addedJars, ser)
if (serializedTask.limit >= akkaFrameSize - 1024) {
val msg = "Serialized task %s:%d were %d bytes which " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nicer to use string interpolation here to make it easier to parse.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for this is to keep the same style of the code.

@pwendell
Copy link
Contributor

pwendell commented May 7, 2014

Thanks for tracking down this fix. It looks good - added some small comments.

@witgo witgo changed the title SPARK-1712: TaskDescription instance is too big causes Spark to hang SPARK-1712: ParallelCollectionPartition instance is too big causes Spark to hang May 7, 2014
@witgo witgo changed the title SPARK-1712: ParallelCollectionPartition instance is too big causes Spark to hang SPARK-1712: TaskDescription instance is too big causes Spark to hang May 7, 2014
@witgo witgo changed the title SPARK-1712: TaskDescription instance is too big causes Spark to hang [WIP]SPARK-1712: TaskDescription instance is too big causes Spark to hang May 7, 2014
@witgo
Copy link
Contributor Author

witgo commented May 7, 2014

@pwendell
How about this solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants