[SPARK-27071][CORE] Expose additional metrics in status.api.v1.StageData

[tomvanbussel]

What changes were proposed in this pull request?

This PR exposes additional metrics in status.api.v1.StageData. These metrics were already computed for LiveStage, but they were never exposed to the user. This includes extra metrics about the JVM GC, executor (de)serialization times, shuffle reads and writes, and more.

How was this patch tested?

Existing tests.

cc @hvanhovell

[hvanhovell]

@vanzin can you take a look?

[hvanhovell]

ok to test

[attilapiros]

Unfortunately this change also increases the event log size meanwhile there are problems caused by the too huge event logs (think about a long running streaming app).

As this metrics are available via the monitoring subsystem and can be sent to an external system I have some doubts whether this is really needed.

[vanzin]

@attilapiros I don't see any change that would affect the event log at all here?

(Haven't looked at everything in detail yet.)

[attilapiros]

@vanzin Oh I see SparkListenerStageCompleted already contains the TaskMetrics.

[hvanhovell]

Any more feedback here? Otherwise I am going to pull the trigger on this one.

[vanzin]

I most probably won't get to reviewing this. But tests haven't run.

This is also technically an API breakage, because you're removing fields from the public REST types.

[tomvanbussel]

If we cannot break the API, then I see two options:

1. We leave the old fields, and we also add the TaskMetrics. This would mean that some fields are duplicated.
2. We inline all the fields from TaskMetrics into StageData. This is not as nice (in my opinion), but there will be no duplicates.

Which option is better?

[hvanhovell]

@tomvanbussel lets go with option 2 for now.

[hvanhovell]

ok to test

[SparkQA]

Test build #104163 has finished for PR 24011 at commit 4974fa0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

LGTM. I will defer to @vanzin for final sign-off.

[hvanhovell]

retest this please

[SparkQA]

Test build #105429 has finished for PR 24011 at commit 4974fa0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

retest this please

[hvanhovell]

retest this please

[SparkQA]

Test build #105829 has finished for PR 24011 at commit 4974fa0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Merging to master. Thanks