[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application #23158

shahidki31 · 2018-11-27T18:09:21Z

What changes were proposed in this pull request?

When the 'spark.history.fs.inProgressOptimization.enabled' is true, inProgress application's last updated time is not getting updated in the History UI. Also, during the cleaning time, InProgress application is getting removed from the listing, even if the last updated time is within the cleaning threshold time.

In this PR, if the fastInprogressOptimization enabled, we update the lastUpdateTime of the application as last scan time. This will update the lastUpdateTime in the historyUI and also while cleaning, it won't remove if the updateTime is within the cleaning interval

How was this patch tested?

Added UT, attached screen shot.
Before patch:

After Patch:

shahidki31 · 2018-11-27T18:11:47Z

cc @vanzin @srowen Kindly review

shahidki31 · 2018-11-27T18:36:12Z

Expired was determined by the lastUpdateTime which we need to update when ever an eventLog update happens

spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

Lines 788 to 793 in 2d89d10

    
           val expired = listing.view(classOf[ApplicationInfoWrapper]) 
        
             .index("oldestAttempt") 
        
             .reverse() 
        
             .first(maxTime) 
        
             .asScala 
        
             .toList

spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

Lines 1129 to 1130 in 2d89d10

    
           @JsonIgnore @KVIndexParam("oldestAttempt") 
        
           def oldestAttempt(): Long = attempts.map(_.info.lastUpdated.getTime()).min

SparkQA · 2018-11-27T22:13:19Z

Test build #99338 has finished for PR 23158 at commit 985470c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-11-28T22:07:55Z

core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala

+        "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1"))
+    )
+    clock.setTime(firstFileModifiedTime)
+    provider.checkForLogs()


You need to set the log file's modified time before calling this, otherwise the cleaner won't be checking what you expect.

added Thanks.
But for inProgress application, do we really need to set log file's last modified time, as the cleaner check only the application's lastUpdated time, which we update whenever size of the logFile changes.

Perhaps. Better to be consistent with other tests. Also because you're using a manual clock, and otherwise your mod times will be way higher than the clock's time.

I see, Thanks.

vanzin · 2018-11-28T22:08:06Z

core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala

+    val maxAge = TimeUnit.DAYS.toMillis(7)
+    val clock = new ManualClock(0)
+    val provider = new FsHistoryProvider(
+      createTestConf().set("spark.history.fs.cleaner.maxAge", s"${maxAge}ms"), clock)


Use config constant.

Done. Thanks

SparkQA · 2018-11-29T02:08:31Z

Test build #99409 has finished for PR 23158 at commit a07aaa4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-29T03:59:32Z

Test build #99419 has finished for PR 23158 at commit 5b2d6c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-29T05:30:21Z

Test build #99411 has finished for PR 23158 at commit 95587b8.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2018-11-29T06:38:59Z

retest this please

shahidki31 · 2018-11-29T06:39:35Z

Jenkins, retest this please

SparkQA · 2018-11-29T08:05:02Z

Test build #99433 has finished for PR 23158 at commit 5b2d6c9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2018-11-29T08:17:44Z

It is random failure. Jenkins, retest this please

SparkQA · 2018-11-29T12:17:02Z

Test build #99438 has finished for PR 23158 at commit 5b2d6c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-11-29T17:47:59Z

Merging to master / 2.4.

…ated for the Inprogress application ## What changes were proposed in this pull request? When the 'spark.history.fs.inProgressOptimization.enabled' is true, inProgress application's last updated time is not getting updated in the History UI. Also, during the cleaning time, InProgress application is getting removed from the listing, even if the last updated time is within the cleaning threshold time. In this PR, if the fastInprogressOptimization enabled, we update the `lastUpdateTime` of the application as last scan time. This will update the `lastUpdateTime` in the historyUI and also while cleaning, it won't remove if the updateTime is within the cleaning interval ## How was this patch tested? Added UT, attached screen shot. Before patch: ![screenshot from 2018-11-27 23-22-38](https://user-images.githubusercontent.com/23054875/49101600-9b5a3380-f29c-11e8-8efc-3fb594e4279a.png) ![screenshot from 2018-11-27 23-20-11](https://user-images.githubusercontent.com/23054875/49101601-9c8b6080-f29c-11e8-928e-643a8c8f4477.png) After Patch: ![screenshot from 2018-11-27 23-37-10](https://user-images.githubusercontent.com/23054875/49101911-669aac00-f29d-11e8-8181-663e4a08ab0e.png) ![screenshot from 2018-11-27 23-39-04](https://user-images.githubusercontent.com/23054875/49102010-a5306680-f29d-11e8-947a-e8a2a09a785a.png) Closes #23158 from shahidki31/HistoryLastUpdateTime. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 24e78b7) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

shahidki31 · 2018-11-29T18:15:51Z

Thanks a lot @vanzin

…ated for the Inprogress application ## What changes were proposed in this pull request? When the 'spark.history.fs.inProgressOptimization.enabled' is true, inProgress application's last updated time is not getting updated in the History UI. Also, during the cleaning time, InProgress application is getting removed from the listing, even if the last updated time is within the cleaning threshold time. In this PR, if the fastInprogressOptimization enabled, we update the `lastUpdateTime` of the application as last scan time. This will update the `lastUpdateTime` in the historyUI and also while cleaning, it won't remove if the updateTime is within the cleaning interval ## How was this patch tested? Added UT, attached screen shot. Before patch: ![screenshot from 2018-11-27 23-22-38](https://user-images.githubusercontent.com/23054875/49101600-9b5a3380-f29c-11e8-8efc-3fb594e4279a.png) ![screenshot from 2018-11-27 23-20-11](https://user-images.githubusercontent.com/23054875/49101601-9c8b6080-f29c-11e8-928e-643a8c8f4477.png) After Patch: ![screenshot from 2018-11-27 23-37-10](https://user-images.githubusercontent.com/23054875/49101911-669aac00-f29d-11e8-8181-663e4a08ab0e.png) ![screenshot from 2018-11-27 23-39-04](https://user-images.githubusercontent.com/23054875/49102010-a5306680-f29d-11e8-947a-e8a2a09a785a.png) Closes apache#23158 from shahidki31/HistoryLastUpdateTime. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

…ated for the Inprogress application ## What changes were proposed in this pull request? When the 'spark.history.fs.inProgressOptimization.enabled' is true, inProgress application's last updated time is not getting updated in the History UI. Also, during the cleaning time, InProgress application is getting removed from the listing, even if the last updated time is within the cleaning threshold time. In this PR, if the fastInprogressOptimization enabled, we update the `lastUpdateTime` of the application as last scan time. This will update the `lastUpdateTime` in the historyUI and also while cleaning, it won't remove if the updateTime is within the cleaning interval ## How was this patch tested? Added UT, attached screen shot. Before patch: ![screenshot from 2018-11-27 23-22-38](https://user-images.githubusercontent.com/23054875/49101600-9b5a3380-f29c-11e8-8efc-3fb594e4279a.png) ![screenshot from 2018-11-27 23-20-11](https://user-images.githubusercontent.com/23054875/49101601-9c8b6080-f29c-11e8-928e-643a8c8f4477.png) After Patch: ![screenshot from 2018-11-27 23-37-10](https://user-images.githubusercontent.com/23054875/49101911-669aac00-f29d-11e8-8181-663e4a08ab0e.png) ![screenshot from 2018-11-27 23-39-04](https://user-images.githubusercontent.com/23054875/49102010-a5306680-f29d-11e8-947a-e8a2a09a785a.png) Closes apache#23158 from shahidki31/HistoryLastUpdateTime. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 24e78b7) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

update lastUpdateTime for inprogress application

985470c

vanzin reviewed Nov 28, 2018

View reviewed changes

address comment

95587b8

shahidki31 force-pushed the HistoryLastUpdateTime branch from a07aaa4 to 95587b8 Compare November 28, 2018 22:49

nit

5b2d6c9

shahidki31 changed the title ~~[SPARK-26186][SPARK-26184] Last updated time is not getting updated for the Inprogress application~~ [SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application Nov 29, 2018

asfgit closed this in 24e78b7 Nov 29, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application #23158

[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application #23158

shahidki31 commented Nov 27, 2018 •

edited

Loading

shahidki31 commented Nov 27, 2018

shahidki31 commented Nov 27, 2018

SparkQA commented Nov 27, 2018

vanzin Nov 28, 2018

shahidki31 Nov 28, 2018

vanzin Nov 28, 2018

shahidki31 Nov 28, 2018

vanzin Nov 28, 2018

shahidki31 Nov 28, 2018

SparkQA commented Nov 29, 2018

SparkQA commented Nov 29, 2018

SparkQA commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

SparkQA commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

SparkQA commented Nov 29, 2018

vanzin commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application #23158

[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application #23158

Conversation

shahidki31 commented Nov 27, 2018 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

shahidki31 commented Nov 27, 2018

shahidki31 commented Nov 27, 2018

SparkQA commented Nov 27, 2018

vanzin Nov 28, 2018

Choose a reason for hiding this comment

shahidki31 Nov 28, 2018

Choose a reason for hiding this comment

vanzin Nov 28, 2018

Choose a reason for hiding this comment

shahidki31 Nov 28, 2018

Choose a reason for hiding this comment

vanzin Nov 28, 2018

Choose a reason for hiding this comment

shahidki31 Nov 28, 2018

Choose a reason for hiding this comment

SparkQA commented Nov 29, 2018

SparkQA commented Nov 29, 2018

SparkQA commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

SparkQA commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

SparkQA commented Nov 29, 2018

vanzin commented Nov 29, 2018

shahidki31 commented Nov 29, 2018

shahidki31 commented Nov 27, 2018 •

edited

Loading