Update e2e tests #2161

ChenYi015 · 2024-09-10T07:06:55Z

Purpose of this PR

Proposed changes:

Remove duplicate operator image build tasks in integration test
Add time sleep buffer to ensure that webhooks are ready before running the e2e tests
Check whether volumes are mounted in e2e tests
Check whether resource requests and limits are set in e2e tests
Remove the coreLimit field from all example spark applications except for spark-pi-custom-resource.yaml. For that, added coreRequest and coreLimit fields to it.

Change Category

Indicate the type of change by marking the applicable boxes:

Bugfix (non-breaking change which fixes an issue)
Feature (non-breaking change which adds functionality)
Breaking change (fix or feature that could affect existing functionality)
Documentation update

Checklist

Before submitting your PR, please review the following:

I have conducted a self-review of my own code.
I have updated documentation accordingly.
I have added tests that prove my changes are effective or that my feature works.
Existing unit tests pass locally with my changes.

Additional Notes

The e2e tests of the recent PRs still failed due to the webhook not ready, even after checking the CA bundle and webhook service enpoints.

… e2e tests Signed-off-by: Yi Chen <github@chenyicn.net>

Signed-off-by: Yi Chen <github@chenyicn.net>

ChenYi015 · 2024-09-10T07:20:52Z

/assign @yuchaoran2011 @vara-bonthu @jacobsalway

Signed-off-by: Yi Chen <github@chenyicn.net>

jacobsalway · 2024-09-12T06:56:17Z

test/e2e/suit_test.go

@@ -149,6 +149,8 @@ var _ = BeforeSuite(func() {
 	validatingWebhookKey := types.NamespacedName{Name: ValidatingWebhookName}
 	Expect(waitForMutatingWebhookReady(context.Background(), mutatingWebhookKey)).NotTo(HaveOccurred())
 	Expect(waitForValidatingWebhookReady(context.Background(), validatingWebhookKey)).NotTo(HaveOccurred())
+	// TODO: Remove this when there is a better way to ensure the webhooks are ready before running the e2e tests.
+	time.Sleep(10 * time.Second)


I think this is fine as a temporary workaround. Two ways we can solve this in the future:

Remove the dependency on the webhook altogether by constructing pod template files on the fly from the existing SparkApplication spec. Maybe via a ConfigMap into the driver pod?

If the webhook is enabled, the controller should only start reconciling if the webhook is ready to serve mutation requests.

jacobsalway · 2024-09-12T07:01:47Z

test/e2e/sparkapplication_test.go

+				}
+				Expect(container.Resources.Requests.Memory).NotTo(BeNil())
+				Expect(container.Resources.Limits.Memory).NotTo(BeNil())
+			}


I'm curious: was there any specific reason to add this assertion? The request and limit fields on the pods are set via spark-submit args and aren't patched by the webhook issue that I believe you raised this PR as part of

Yes, the cpu/memory requests/limits are not patched by the webhook. I added these assertions just to ensure that spark-pi-custom-resouce.yaml works as expected in setting resource requests/limits correctly.

What do you think about removing the coreLimit field from the existing example applications and creating a new application that specifies both coreRequest and coreLimit, creating a new end-to-end test for that, then moving this assertion logic to that test? I feel like that captures the intention better but open to your opinion too obviously.

@jacobsalway Sounds good. I have removed the coreLimit field from all the example applications except for spark-pi-custom-resource.yaml. For that, I added the coreRequest and coreLimit fields to it.

Signed-off-by: Yi Chen <github@chenyicn.net>

jacobsalway · 2024-09-12T12:04:03Z

/lgtm

ChenYi015 · 2024-09-13T03:23:31Z

/approve

google-oss-prow · 2024-09-13T03:23:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <github@chenyicn.net>

* Support gang scheduling with Yunikorn (#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add test cases Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add code comments Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add license comment Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Fix initial number implementation Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <github@chenyicn.net> * Update Makefile for building sparkctl (#2119) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: Add default values for namespaces to match usage descriptions (#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <pengfei4.li@ly.com> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <pengfei4.li@ly.com> --------- Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: Spark role binding did not render properly when setting spark service account name (#2135) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <github@chenyicn.net> * Reintroduce option webhook.enable (#2142) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <github@chenyicn.net> * Add default batch scheduler argument (#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add helm unit test Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: unable to set controller/webhook replicas to zero (#2147) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <github@chenyicn.net> * Adding support for setting spark job namespaces to all namespaces (#2123) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <github@chenyicn.net> * Support extended kube-scheduler as batch scheduler (#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <github@chenyicn.net> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <github@chenyicn.net> * Run e2e tests on Kind (#2148) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net> * Set schedulerName to Yunikorn (#2153) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <github@chenyicn.net> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (#2129) watching all namespaces Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: e2e test failes due to webhook not ready (#2149) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Go 1.23.1 (#2155) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2 (#2154) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <github@chenyicn.net> * feat: support driver and executor pod use different priority (#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove backup crd files Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> --------- Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests (#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: webhook not working when settings spark job namespaces to empty (#2163) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: The logger had an odd number of arguments, making it panic (#2166) Signed-off-by: tcassaert <tcassaert@inuits.eu> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2(#2012) (#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> --------- Signed-off-by: HyukSangCho <a01045542949@gmail.com> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <github@chenyicn.net> * Feature: Add pprof endpoint (#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> * add pprof support to helm chart Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> --------- Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <github@chenyicn.net> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172) Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <github@chenyicn.net> * Add specific error in log line when failed to create web UI service (#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <tcassaert@inuits.eu> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: tcassaert <tcassaert@protonmail.com> --------- Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Yi Chen <github@chenyicn.net> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <github@chenyicn.net> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: spark application does not respect time to live seconds (#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <github@chenyicn.net> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <github@chenyicn.net> * Update release workflow and docs (#2121) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> Signed-off-by: Yi Chen <github@chenyicn.net> Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: HyukSangCho <a01045542949@gmail.com> Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Jacob Salway <jacob.salway@gmail.com> Co-authored-by: Neo <56439757+snappyyouth@users.noreply.github.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <ruoshuidba@gmail.com> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> Co-authored-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: ha2hi <56156892+ha2hi@users.noreply.github.com> Co-authored-by: Sébastien Maintrot <3097030+ImpSy@users.noreply.github.com>

* Support gang scheduling with Yunikorn (kubeflow#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add test cases Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add code comments Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add license comment Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Fix initial number implementation Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <github@chenyicn.net> * Update Makefile for building sparkctl (kubeflow#2119) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: Add default values for namespaces to match usage descriptions (kubeflow#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <pengfei4.li@ly.com> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <pengfei4.li@ly.com> --------- Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: Spark role binding did not render properly when setting spark service account name (kubeflow#2135) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <github@chenyicn.net> * Reintroduce option webhook.enable (kubeflow#2142) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <github@chenyicn.net> * Add default batch scheduler argument (kubeflow#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add helm unit test Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: unable to set controller/webhook replicas to zero (kubeflow#2147) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <github@chenyicn.net> * Adding support for setting spark job namespaces to all namespaces (kubeflow#2123) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <github@chenyicn.net> * Support extended kube-scheduler as batch scheduler (kubeflow#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <github@chenyicn.net> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <github@chenyicn.net> * Run e2e tests on Kind (kubeflow#2148) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net> * Set schedulerName to Yunikorn (kubeflow#2153) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <github@chenyicn.net> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (kubeflow#2129) watching all namespaces Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: e2e test failes due to webhook not ready (kubeflow#2149) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Go 1.23.1 (kubeflow#2155) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2 (kubeflow#2154) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (kubeflow#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <github@chenyicn.net> * feat: support driver and executor pod use different priority (kubeflow#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove backup crd files Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> --------- Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump gocloud.dev from 0.37.0 to 0.39.0 (kubeflow#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests (kubeflow#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: webhook not working when settings spark job namespaces to empty (kubeflow#2163) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: The logger had an odd number of arguments, making it panic (kubeflow#2166) Signed-off-by: tcassaert <tcassaert@inuits.eu> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2(kubeflow#2012) (kubeflow#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> --------- Signed-off-by: HyukSangCho <a01045542949@gmail.com> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <github@chenyicn.net> * Feature: Add pprof endpoint (kubeflow#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> * add pprof support to helm chart Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> --------- Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <github@chenyicn.net> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (kubeflow#2172) Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (kubeflow#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (kubeflow#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <github@chenyicn.net> * Add specific error in log line when failed to create web UI service (kubeflow#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <tcassaert@inuits.eu> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: tcassaert <tcassaert@protonmail.com> --------- Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Yi Chen <github@chenyicn.net> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <github@chenyicn.net> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (kubeflow#2178) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: spark application does not respect time to live seconds (kubeflow#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <github@chenyicn.net> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <github@chenyicn.net> * Update release workflow and docs (kubeflow#2121) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> Signed-off-by: Yi Chen <github@chenyicn.net> Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: HyukSangCho <a01045542949@gmail.com> Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Jacob Salway <jacob.salway@gmail.com> Co-authored-by: Neo <56439757+snappyyouth@users.noreply.github.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <ruoshuidba@gmail.com> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> Co-authored-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: ha2hi <56156892+ha2hi@users.noreply.github.com> Co-authored-by: Sébastien Maintrot <3097030+ImpSy@users.noreply.github.com>

ChenYi015 added 2 commits September 10, 2024 15:03

Add sleep buffer to ensture the webhooks are ready before running the…

9433e28

… e2e tests Signed-off-by: Yi Chen <github@chenyicn.net>

Remove duplicate operator image build tasks

976471c

Signed-off-by: Yi Chen <github@chenyicn.net>

google-oss-prow bot requested review from andreyvelich, jacobsalway and yuchaoran2011 September 10, 2024 07:07

google-oss-prow bot added the size/S label Sep 10, 2024

google-oss-prow bot assigned jacobsalway, vara-bonthu and yuchaoran2011 Sep 10, 2024

google-oss-prow bot added size/M and removed size/S labels Sep 10, 2024

ChenYi015 changed the title ~~Add time sleep buffer to ensure that webhooks are ready before running the e2e tests~~ Update e2e tests Sep 10, 2024

ChenYi015 force-pushed the workflow/integration-test branch from 14b97d4 to ed1d525 Compare September 10, 2024 09:57

Update e2e tests

bbf1e88

Signed-off-by: Yi Chen <github@chenyicn.net>

ChenYi015 force-pushed the workflow/integration-test branch from ed1d525 to bbf1e88 Compare September 10, 2024 10:47

jacobsalway reviewed Sep 12, 2024

View reviewed changes

Update examples

5769940

Signed-off-by: Yi Chen <github@chenyicn.net>

google-oss-prow bot added size/L and removed size/M labels Sep 12, 2024

google-oss-prow bot added the lgtm label Sep 12, 2024

google-oss-prow bot added the approved label Sep 13, 2024

google-oss-prow bot merged commit e6a7805 into kubeflow:master Sep 13, 2024
7 checks passed

ChenYi015 deleted the workflow/integration-test branch September 13, 2024 04:34

ChenYi015 mentioned this pull request Sep 19, 2024

Cherry pick commits for releasing v2.0.0 #2156

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update e2e tests #2161

Update e2e tests #2161

ChenYi015 commented Sep 10, 2024 •

edited

Loading

ChenYi015 commented Sep 10, 2024

jacobsalway Sep 12, 2024 •

edited

Loading

jacobsalway Sep 12, 2024

ChenYi015 Sep 12, 2024

jacobsalway Sep 12, 2024

ChenYi015 Sep 12, 2024

jacobsalway commented Sep 12, 2024

ChenYi015 commented Sep 13, 2024

google-oss-prow bot commented Sep 13, 2024

Update e2e tests #2161

Update e2e tests #2161

Conversation

ChenYi015 commented Sep 10, 2024 • edited Loading

Purpose of this PR

Change Category

Checklist

Additional Notes

ChenYi015 commented Sep 10, 2024

jacobsalway Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

jacobsalway Sep 12, 2024

Choose a reason for hiding this comment

ChenYi015 Sep 12, 2024

Choose a reason for hiding this comment

jacobsalway Sep 12, 2024

Choose a reason for hiding this comment

ChenYi015 Sep 12, 2024

Choose a reason for hiding this comment

jacobsalway commented Sep 12, 2024

ChenYi015 commented Sep 13, 2024

google-oss-prow bot commented Sep 13, 2024

ChenYi015 commented Sep 10, 2024 •

edited

Loading

jacobsalway Sep 12, 2024 •

edited

Loading