Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes #1966

Closed
wants to merge 7 commits into from
Closed

[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes #1966

wants to merge 7 commits into from

Conversation

YanTangZhai
Copy link
Contributor

JobProgressPage could not show Fair Scheduler Pools section sometimes.
SparkContext starts webui and then postEnvironmentUpdate. Sometimes JobProgressPage is accessed between webui starting and postEnvironmentUpdate, then the lazy val isFairScheduler will be false. The Fair Scheduler Pools section will not display any more.

@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have started for PR 1966 at commit aac7f7b.

  • This patch merges cleanly.

@andrewor14
Copy link
Contributor

retest this please

andrewor14
andrewor14 reviewed Sep 11, 2014
View reviewed changes
core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -334,6 +333,9 @@ class SparkContext(config: SparkConf) extends Logging {
postEnvironmentUpdate()
postApplicationStart()

// Starting Spark UI service
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering here is very sensitive and it might be good to explain why this must be bound here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll update this comment when I merge it. Thanks.

@andrewor14
Copy link
Contributor

Thanks for submitting this patch. I have noticed this myself, except my encounters were usually NPEs instead (when we call sc.getAllPools from the JobProgressPage before sc.taskScheduler was set). Looks like this will fix that case as well.

@SparkQA
Copy link

SparkQA commented Sep 11, 2014

QA tests have started for PR 1966 at commit aac7f7b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA tests have finished for PR 1966 at commit aac7f7b.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA tests have started for PR 1966. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20212/consoleFull

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA results for PR 1966:
- This patch FAILED unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20212/consoleFull

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA tests have started for PR 1966. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20213/consoleFull

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA tests have started for PR 1966 at commit d4323f8.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA results for PR 1966:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20213/consoleFull

@SparkQA
Copy link

SparkQA commented Sep 12, 2014

QA tests have finished for PR 1966 at commit d4323f8.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in dedace8 Oct 17, 2014
asfgit pushed a commit that referenced this pull request Oct 17, 2014
@YanTangZhai @andrewor14
[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools sect…
35875e9 
…ion sometimes

JobProgressPage could not show Fair Scheduler Pools section sometimes.
SparkContext starts webui and then postEnvironmentUpdate. Sometimes JobProgressPage is accessed between webui starting and postEnvironmentUpdate, then the lazy val isFairScheduler will be false. The Fair Scheduler Pools section will not display any more.

Author: yantangzhai <tyz0303@163.com>
Author: YanTangZhai <hakeemzhai@tencent.com>

Closes #1966 from YanTangZhai/SPARK-3067 and squashes the following commits:

d4323f8 [yantangzhai] update [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes
8a00106 [YanTangZhai] Merge pull request #6 from apache/master
b6391cc [yantangzhai] revert [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes
d2226cd [yantangzhai] [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes
cbcba66 [YanTangZhai] Merge pull request #3 from apache/master
aac7f7b [yantangzhai] [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes
cdef539 [YanTangZhai] Merge pull request #1 from apache/master

(cherry picked from commit dedace8)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
@witgo witgo mentioned this pull request Oct 20, 2014
Closed
asfgit pushed a commit that referenced this pull request Oct 20, 2014
@witgo @andrewor14
[SPARK-4010][Web UI]Spark UI returns 500 in yarn-client mode
457ef59 
The problem caused by #1966
CC YanTangZhai andrewor14

Author: GuoQiang Li <witgo@qq.com>

Closes #2858 from witgo/SPARK-4010 and squashes the following commits:

9866fbf [GuoQiang Li] Spark UI returns 500 in yarn-client mode

(cherry picked from commit 51afde9)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
asfgit pushed a commit that referenced this pull request Oct 20, 2014
@witgo @andrewor14
[SPARK-4010][Web UI]Spark UI returns 500 in yarn-client mode
51afde9 
The problem caused by #1966
CC YanTangZhai andrewor14

Author: GuoQiang Li <witgo@qq.com>

Closes #2858 from witgo/SPARK-4010 and squashes the following commits:

9866fbf [GuoQiang Li] Spark UI returns 500 in yarn-client mode
@AngersZhuuuu AngersZhuuuu mentioned this pull request Jul 21, 2021
Closed
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Aug 7, 2024
[MINOR] build: Run network-common tests using sbt-nonroot (apache#1966)
b85fadc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@YanTangZhai @SparkQA @andrewor14