Skip to content

Commit

Permalink
Add context termination for History server on Yarn
Browse files Browse the repository at this point in the history
The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458
  • Loading branch information
moutai committed Feb 22, 2015
1 parent 275b1be commit 3ad3db4
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,8 @@ follows:
Note that in all of these UIs, the tables are sortable by clicking their headers,
making it easy to identify slow tasks, data skew, etc.

In Yarn, note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI.

# Metrics

Spark has a configurable metrics system based on the
Expand Down

0 comments on commit 3ad3db4

Please sign in to comment.