Cherry pick commits for releasing v2.0.0

[ChenYi015]

Purpose of this PR

Cherry pick the following commits:

• Update release workflow and docs (Update release workflow and docs #2121)
• Fix: spark application does not respect time to live seconds (Fix: spark application does not respect time to live seconds #2165)
• Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (Account for spark.executor.pyspark.memory in Yunikorn gang scheduling #2178)
• Add specific error in log line when failed to create web UI service (Add specific error in log line when failed to create web UI service #2170)
• implement workflow to scan latest released docker image (implement workflow to scan latest released docker image #2177)
• Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 #2173)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 #2174)
• fix the make kind-delete-custer to avoid accidental kubeconfig deletion (fix the make kind-delete-custer to avoid accidental kubeconfig deletion #2172)
• Feature: Add pprof endpoint (Feature: Add pprof endpoint #2164)
• Upgrade to Spark 3.5.2 (Upgrade to Spark 3.5.2(#2012) #2157)
• replace datamechanics -> spot.io as adopter (acquisition) (replace datamechanics -> spot.io as adopter (acquisition) #2168)
• fix: The logger had an odd number of arguments, making it panic (fix: The logger had an odd number of arguments, making it panic #2166)
• fix: webhook not working when settings spark job namespaces to empty (fix: webhook not working when settings spark job namespaces to empty #2163)
• Update e2e tests (Update e2e tests #2161)
• Bump gocloud.dev from 0.37.0 to 0.39.0 (Bump gocloud.dev from 0.37.0 to 0.39.0 #2160)
• feat: support driver and executor pod use different priority (feat: support driver and executor pod use different priority #2146)
• Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 #2159)
• Upgrade to Spark 3.5.2 (Upgrade to Spark 3.5.2 #2154)
• Upgrade to Go 1.23.1 (Upgrade to Go 1.23.1 #2155)
• Fix: e2e test failes due to webhook not ready (Fix: e2e test failes due to webhook not ready #2149)
• Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces #2129)
• Set schedulerName to Yunikorn (Set schedulerName to Yunikorn #2153)
• Run e2e tests on Kind (Run e2e tests on Kind #2148)
• Support extended kube-scheduler as batch scheduler (Support extended kube-scheduler as batch scheduler #2136)
• Adding support for setting spark job namespaces to all namespaces (Adding support for setting spark job namespaces to all namespaces #2123)
• fix: unable to set controller/webhook replicas to zero (fix: unable to set controller/webhook replicas to zero #2147)
• Add default batch scheduler argument (Add default batch scheduler argument #2143)
• Reintroduce option webhook.enable (Reintroduce option webhook.enable #2142)
• Fix: Spark role binding did not render properly when setting spark service account name (Fix: Spark role binding did not render properly when setting spark service account name #2135)
• fix: Add default values for namespaces to match usage descriptions (fix: Add default values for namespaces to match usage descriptions  #2128)
• Update Makefile for building sparkctl (Update Makefile for building sparkctl #2119)
• Support gang scheduling with Yunikorn (Support gang scheduling with Yunikorn #2107)

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[jacobsalway]

I'm writing up a docs page for the YuniKorn integration in kubeflow/website#3872. Might be good to include in the 2.0.0 release notes? I'm aiming to finish it tomorrow

[ChenYi015]

I'm writing up a docs page for the YuniKorn integration in kubeflow/website#3872. Might be good to include in the 2.0.0 release notes? I'm aiming to finish it tomorrow

Sounds good, I think we can add the doc updates to the release notes as well.

[ChenYi015]

/assign @jacobsalway @vara-bonthu @yuchaoran2011

[vara-bonthu]

Nice work on the leading this work @ChenYi015 👏🏼

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jacobsalway]

/lgtm