[SPARK-44916][DOCS][TESTS] Document Spark Driver Live Log UI

core/src/test/scala/org/apache/spark/util/logging/DriverLoggerSuite.scala

@@ -97,7 +97,9 @@ class DriverLoggerSuite extends SparkFunSuite with LocalSparkContext {
   test("SPARK-44214: DriverLogger.apply returns None when only spark.driver.log.localDir exists") {
     val sparkConf = new SparkConf()
     assert(DriverLogger(sparkConf).isEmpty)
-    assert(DriverLogger(sparkConf.set(DRIVER_LOG_LOCAL_DIR, "file://tmp/")).isEmpty)
+    withTempDir { dir =>
+      assert(DriverLogger(sparkConf.set(DRIVER_LOG_LOCAL_DIR, dir.getCanonicalPath)).isEmpty)
+    }
   }
 
   private def getSparkContext(): SparkContext = {

docs/configuration.md

@@ -430,6 +430,14 @@ of the most common options to set are:
   </td>
   <td>1.3.0</td>
 </tr>
+<tr>
+  <td><code>spark.driver.log.localDir</code></td>
+  <td>(none)</td>
+  <td>
+    Specifies a local directory to write driver logs and enable Driver Log UI Tab.
+  </td>
+  <td>4.0.0</td>
+</tr>
 <tr>
   <td><code>spark.driver.log.dfsDir</code></td>
   <td>(none)</td>
@@ -460,7 +468,7 @@ of the most common options to set are:
   <td><code>spark.driver.log.layout</code></td>
   <td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex</td>
   <td>
-    The layout for the driver logs that are synced to <code>spark.driver.log.dfsDir</code>. If this is not configured,
+    The layout for the driver logs that are synced to <code>spark.driver.log.localDir</code> and <code>spark.driver.log.dfsDir</code>. If this is not configured,
     it uses the layout for the first appender defined in log4j2.properties. If that is also not configured, driver logs
     use the default layout.
   </td>

docs/running-on-kubernetes.md

@@ -441,6 +441,19 @@ $ kubectl port-forward <driver-pod-name> 4040:4040
 
 Then, the Spark driver UI can be accessed on `http://localhost:4040`.
 
+Since Apache Spark 4.0.0, Driver UI provides a way to see driver logs via a new configuration.
+
+```
+spark.driver.log.localDir=/tmp
+```
+
+Then, the Spark driver UI can be accessed on `http://localhost:4040/logs/`.
+Optionally, the layout of log is configured by the following.
+
+```
+spark.driver.log.layout="%m%n%ex"
+```
+
 ### Debugging
 
 There may be several kinds of failures. If the Kubernetes API server rejects the request made from spark-submit, or the