[SPARK-33841][CORE][3.1] Fix issue with jobs disappearing intermittently from the SHS under high load

[vladhlinsky]

What changes were proposed in this pull request?

Mark SHS event log entries that were processing at the beginning of the checkForLogs run as not stale and check for this mark before deleting an event log. This fixes the issue when a particular job was displayed in the SHS and disappeared after some time, but then, in several minutes showed up again.

Why are the changes needed?

The issue is caused by SPARK-29043, which is designated to improve the concurrent performance of the History Server. The change breaks the "app deletion" logic because of missing proper synchronization for processing event log entries. Since SHS now filters out all processing event log entries, such entries do not have a chance to be updated with the new lastProcessed time and thus any entity that completes processing right after filtering and before the check for stale entities will be identified as stale and will be deleted from the UI until the next checkForLogs run. This is because updated lastProcessed time is used as criteria, and event log entries that missed to be updated with a new time, will match that criteria.

The issue can be reproduced by generating a big number of event logs and uploading them to the SHS event log directory on S3. Essentially, around 439(49.6 MB) copies of an event log file were created using shs-monitor script. Strange behavior of SHS counting the total number of applications was noticed - at first, the number was increasing as expected, but with the next page refresh, the total number of applications decreased. No errors were logged by SHS.

252 entities are displayed at 21:20:23:

178 entities are displayed at 21:22:15:

Does this PR introduce any user-facing change?

Yes, SHS users won't face the behavior when the number of displayed applications decreases periodically.

How was this patch tested?

Tested using shs-monitor script:

• Build SHS with the proposed change
• Download Hadoop AWS and AWS Java SDK
• Prepare S3 bucket and user for programmatic access, grant required roles to the user. Get access key and secret key
• Configure SHS to read event logs from S3
• Start monitor script to query SHS API
• Run producers
• Wait for SHS to load all the applications
• Verify that the number of loaded applications increases continuously over time

For more details, please refer to the shs-monitor repository.

[vladhlinsky]

cc @HeartSaVioR

[HeartSaVioR]

retest this, please

[HeartSaVioR]

ok to test

[HeartSaVioR]

+1 and I'd consider @tgravescs approved this PR as only target branch is different.

[SparkQA]

Test build #133037 has finished for PR 30847 at commit 1cb2e1f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37636/

[dongjoon-hyun]

Retest this please

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37636/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37638/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37638/

[dongjoon-hyun]

+1, LGTM. Thanks, @vladhlinsky .
Merged to branch-3.1 for Apache Spark 3.1.0.

[SparkQA]

Test build #133039 has finished for PR 30847 at commit 1cb2e1f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.