From 3ad3db4fcb47d33892334cab953daaeecb556502 Mon Sep 17 00:00:00 2001 From: moussa taifi Date: Sun, 22 Feb 2015 14:16:14 -0500 Subject: [PATCH] Add context termination for History server on Yarn The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway. Related to SPARK-2972 and SPARK-3458 --- docs/monitoring.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/monitoring.md b/docs/monitoring.md index 7a5cadc171d6d..eeea1ba70c9d4 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -150,6 +150,8 @@ follows: Note that in all of these UIs, the tables are sortable by clicking their headers, making it easy to identify slow tasks, data skew, etc. +In Yarn, note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI. + # Metrics Spark has a configurable metrics system based on the