Run e2e tests on Kind #2148

jacobsalway · 2024-08-29T12:49:06Z

Purpose

The existing end-to-end tests haven't been run for what seems like quite a long time. It's important that new or existing end-to-end tests are run in CI as a required check for PRs to be merged to ensure that bugs aren't introduced.

Changes:

Run all jobs under integration.yaml on ubuntu-latest
Create a new e2e-test job to replace the existing integration-test job that spins up a Kubernetes cluster using Kind, builds and loads the image to the cluster, then runs the existing four e2e tests

Closes #1416

Prerequisite work before adding an end to end test for the Yunikorn batch scheduler integration for #2098

Change Category

Indicate the type of change by marking the applicable boxes:

Bugfix (non-breaking change which fixes an issue)
Feature (non-breaking change which adds functionality)
Breaking change (fix or feature that could affect existing functionality)
Documentation update

Checklist

Before submitting your PR, please review the following:

I have conducted a self-review of my own code.
I have updated documentation accordingly.
I have added tests that prove my changes are effective or that my feature works.
Existing unit tests pass locally with my changes.

jacobsalway · 2024-08-30T05:31:22Z

.github/workflows/integration.yaml

@@ -91,7 +91,7 @@ jobs:
          done

  build-helm-chart:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest


I don't see any specific dependency on Ubuntu version in these commands and across the projects I've seen it seems common to run actions on ubuntu-latest.

jacobsalway · 2024-08-30T05:49:15Z

/assign @ChenYi015 @vara-bonthu

ChenYi015

Thanks for fixing the e2e test CI. I have left some comments.

ChenYi015 · 2024-08-30T06:13:08Z

test/e2e/suit_test.go

+
+	values := make(map[string]interface{})
+	if imageTag := os.Getenv("IMAGE_TAG"); imageTag != "" {
+		values["image.tag"] = imageTag
+	}
+	release, err := installAction.Run(chart, values)


We could use values defined in charts/spark-operator-chart/ci/ci-values.yaml as it is aimed to be used in CI workflow.

Using the chartutils package to load these values now

ChenYi015 · 2024-08-30T06:16:29Z

Makefile

-ifndef ignore-not-found
-  ignore-not-found = false
-endif


Why was it deleted as it is used in some targets e.g. uninstall-crd, deploy.

My bad, good catch. Have kept as part of keeping the kind targets

ChenYi015 · 2024-08-30T06:19:17Z

Makefile

-
-.PHONY: kind-create-cluster
-kind-create-cluster: kind ## Create a kind cluster for integration tests.
-	if ! $(KIND) get clusters 2>/dev/null | grep -q "^$(KIND_CLUSTER_NAME)$$"; then \
-		kind create cluster --name $(KIND_CLUSTER_NAME) --config $(KIND_CONFIG_FILE) --kubeconfig $(KIND_KUBE_CONFIG); \
-	fi
-
-.PHONY: kind-load-image
-kind-load-image: kind-create-cluster docker-build ## Load the image into the kind cluster.
-	$(KIND) load docker-image --name $(KIND_CLUSTER_NAME) $(IMAGE)
-
-.PHONY: kind-delete-custer
-kind-delete-custer: kind ## Delete the created kind cluster.
-	$(KIND) delete cluster --name $(KIND_CLUSTER_NAME) && \
-	rm -f $(KIND_KUBE_CONFIG)
-


The kind related targets are used in local tests if the developer does not have a k8s cluster already, perhaps we can keep these targets.

Fair point, I've personally used k3s/k3d and minikube before as well but it looks like kind is fairly common among open source projects

ChenYi015 · 2024-08-30T06:22:31Z

.github/workflows/integration.yaml


-      - name: Build local spark-operator docker image for minikube testing
+      - name: Build and load image to Kind cluster
        run: |
          docker build -t docker.io/kubeflow/spark-operator:local .


We can utilize the docker-build target in the Makefile:

make docker-build IMAGE_TAG=local

Changed this step to run make kind-load-image which also runs the docker-build target

ChenYi015 · 2024-08-30T06:35:59Z

Remove local Kind commands from the Makefile. The existing kind-create-cluster target was broken anyway and developers may wish to use other distributions e.g. Minikube or k3d

@jacobsalway I forgot to push the charts/spark-operator-chart/kind-config.yaml file, which breaks the kind-create-cluster target in makefile. You can make a kind config file as follows and it can be used in e2e CI and local tests:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.29.2
- role: worker
  image: kindest/node:v1.29.2

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

jacobsalway · 2024-08-31T02:46:29Z

Makefile

@@ -246,7 +246,7 @@ endif
 .PHONY: kind-create-cluster
 kind-create-cluster: kind ## Create a kind cluster for integration tests.
 	if ! $(KIND) get clusters 2>/dev/null | grep -q "^$(KIND_CLUSTER_NAME)$$"; then \
-		kind create cluster --name $(KIND_CLUSTER_NAME) --config $(KIND_CONFIG_FILE) --kubeconfig $(KIND_KUBE_CONFIG); \
+		kind create cluster --name $(KIND_CLUSTER_NAME) --config $(KIND_CONFIG_FILE) --kubeconfig $(KIND_KUBE_CONFIG) --wait=1m; \


Waiting for the control plane to become ready before returning success from this target seems sensible to me. I've always used this flag when running kind create cluster myself.

google-oss-prow · 2024-09-03T07:06:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ChenYi015 · 2024-09-03T07:25:28Z

/lgtm

Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece)

Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net>

* Support gang scheduling with Yunikorn (#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add test cases Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add code comments Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add license comment Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Fix initial number implementation Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <github@chenyicn.net> * Update Makefile for building sparkctl (#2119) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: Add default values for namespaces to match usage descriptions (#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <pengfei4.li@ly.com> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <pengfei4.li@ly.com> --------- Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: Spark role binding did not render properly when setting spark service account name (#2135) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <github@chenyicn.net> * Reintroduce option webhook.enable (#2142) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <github@chenyicn.net> * Add default batch scheduler argument (#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add helm unit test Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: unable to set controller/webhook replicas to zero (#2147) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <github@chenyicn.net> * Adding support for setting spark job namespaces to all namespaces (#2123) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <github@chenyicn.net> * Support extended kube-scheduler as batch scheduler (#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <github@chenyicn.net> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <github@chenyicn.net> * Run e2e tests on Kind (#2148) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net> * Set schedulerName to Yunikorn (#2153) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <github@chenyicn.net> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (#2129) watching all namespaces Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: e2e test failes due to webhook not ready (#2149) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Go 1.23.1 (#2155) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2 (#2154) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <github@chenyicn.net> * feat: support driver and executor pod use different priority (#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove backup crd files Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> --------- Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests (#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: webhook not working when settings spark job namespaces to empty (#2163) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: The logger had an odd number of arguments, making it panic (#2166) Signed-off-by: tcassaert <tcassaert@inuits.eu> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2(#2012) (#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> --------- Signed-off-by: HyukSangCho <a01045542949@gmail.com> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <github@chenyicn.net> * Feature: Add pprof endpoint (#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> * add pprof support to helm chart Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> --------- Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <github@chenyicn.net> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172) Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <github@chenyicn.net> * Add specific error in log line when failed to create web UI service (#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <tcassaert@inuits.eu> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: tcassaert <tcassaert@protonmail.com> --------- Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Yi Chen <github@chenyicn.net> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <github@chenyicn.net> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: spark application does not respect time to live seconds (#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <github@chenyicn.net> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <github@chenyicn.net> * Update release workflow and docs (#2121) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> Signed-off-by: Yi Chen <github@chenyicn.net> Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: HyukSangCho <a01045542949@gmail.com> Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Jacob Salway <jacob.salway@gmail.com> Co-authored-by: Neo <56439757+snappyyouth@users.noreply.github.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <ruoshuidba@gmail.com> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> Co-authored-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: ha2hi <56156892+ha2hi@users.noreply.github.com> Co-authored-by: Sébastien Maintrot <3097030+ImpSy@users.noreply.github.com>

* Support gang scheduling with Yunikorn (kubeflow#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add test cases Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add code comments Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add license comment Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Fix initial number implementation Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <github@chenyicn.net> * Update Makefile for building sparkctl (kubeflow#2119) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: Add default values for namespaces to match usage descriptions (kubeflow#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <pengfei4.li@ly.com> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <pengfei4.li@ly.com> --------- Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: Spark role binding did not render properly when setting spark service account name (kubeflow#2135) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <github@chenyicn.net> * Reintroduce option webhook.enable (kubeflow#2142) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <github@chenyicn.net> * Add default batch scheduler argument (kubeflow#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Add helm unit test Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: unable to set controller/webhook replicas to zero (kubeflow#2147) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <github@chenyicn.net> * Adding support for setting spark job namespaces to all namespaces (kubeflow#2123) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <github@chenyicn.net> * Support extended kube-scheduler as batch scheduler (kubeflow#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <github@chenyicn.net> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <github@chenyicn.net> * Run e2e tests on Kind (kubeflow#2148) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net> * Set schedulerName to Yunikorn (kubeflow#2153) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <github@chenyicn.net> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (kubeflow#2129) watching all namespaces Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: e2e test failes due to webhook not ready (kubeflow#2149) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Go 1.23.1 (kubeflow#2155) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2 (kubeflow#2154) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (kubeflow#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <github@chenyicn.net> * feat: support driver and executor pod use different priority (kubeflow#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove backup crd files Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> --------- Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump gocloud.dev from 0.37.0 to 0.39.0 (kubeflow#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests (kubeflow#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <github@chenyicn.net> * Update e2e tests Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: webhook not working when settings spark job namespaces to empty (kubeflow#2163) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <github@chenyicn.net> * fix: The logger had an odd number of arguments, making it panic (kubeflow#2166) Signed-off-by: tcassaert <tcassaert@inuits.eu> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <github@chenyicn.net> * Upgrade to Spark 3.5.2(kubeflow#2012) (kubeflow#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <a01045542949@gmail.com> --------- Signed-off-by: HyukSangCho <a01045542949@gmail.com> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <github@chenyicn.net> * Feature: Add pprof endpoint (kubeflow#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> * add pprof support to helm chart Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> --------- Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <github@chenyicn.net> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (kubeflow#2172) Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (kubeflow#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <github@chenyicn.net> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (kubeflow#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <github@chenyicn.net> * Add specific error in log line when failed to create web UI service (kubeflow#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <tcassaert@inuits.eu> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: tcassaert <tcassaert@protonmail.com> --------- Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Yi Chen <github@chenyicn.net> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <github@chenyicn.net> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (kubeflow#2178) Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <github@chenyicn.net> * Fix: spark application does not respect time to live seconds (kubeflow#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <github@chenyicn.net> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <github@chenyicn.net> * Update release workflow and docs (kubeflow#2121) Signed-off-by: Yi Chen <github@chenyicn.net> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> Signed-off-by: Yi Chen <github@chenyicn.net> Signed-off-by: pengfei4.li <pengfei4.li@ly.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Signed-off-by: tcassaert <tcassaert@inuits.eu> Signed-off-by: HyukSangCho <a01045542949@gmail.com> Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com> Signed-off-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: Jacob Salway <jacob.salway@gmail.com> Co-authored-by: Neo <56439757+snappyyouth@users.noreply.github.com> Co-authored-by: pengfei4.li <pengfei4.li@ly.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <ruoshuidba@gmail.com> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai> Co-authored-by: tcassaert <tcassaert@protonmail.com> Co-authored-by: ha2hi <56156892+ha2hi@users.noreply.github.com> Co-authored-by: Sébastien Maintrot <3097030+ImpSy@users.noreply.github.com>

google-oss-prow bot added the do-not-merge/work-in-progress label Aug 29, 2024

google-oss-prow bot requested review from mwielgus and yuchaoran2011 August 29, 2024 12:49

google-oss-prow bot added the size/M label Aug 29, 2024

jacobsalway force-pushed the fix/e2e-test-ci branch 4 times, most recently from 251c7e3 to 439db1d Compare August 30, 2024 05:30

jacobsalway commented Aug 30, 2024

View reviewed changes

jacobsalway force-pushed the fix/e2e-test-ci branch from 439db1d to 0c2fb25 Compare August 30, 2024 05:43

jacobsalway marked this pull request as ready for review August 30, 2024 05:48

google-oss-prow bot removed the do-not-merge/work-in-progress label Aug 30, 2024

google-oss-prow bot requested review from ChenYi015 and vara-bonthu August 30, 2024 05:48

google-oss-prow bot assigned ChenYi015 and vara-bonthu Aug 30, 2024

ChenYi015 reviewed Aug 30, 2024

View reviewed changes

Run e2e-test in CI using Kind

5940969

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

jacobsalway force-pushed the fix/e2e-test-ci branch from 0c2fb25 to 5940969 Compare August 31, 2024 02:35

jacobsalway commented Aug 31, 2024

View reviewed changes

jacobsalway mentioned this pull request Sep 1, 2024

[FEATURE] Support Yunikorn as a batch scheduler #2098

Open

5 tasks

ChenYi015 approved these changes Sep 3, 2024

View reviewed changes

google-oss-prow bot added the approved label Sep 3, 2024

google-oss-prow bot added the lgtm label Sep 3, 2024

google-oss-prow bot merged commit c810ece into kubeflow:master Sep 3, 2024
7 checks passed

jacobsalway deleted the fix/e2e-test-ci branch September 3, 2024 07:32

jacobsalway mentioned this pull request Sep 3, 2024

Update Golang to 1.23 and Spark to 3.5.2 #2150

Closed

8 tasks

ChenYi015 pushed a commit to ChenYi015/spark-operator that referenced this pull request Sep 9, 2024

Run e2e tests on Kind (kubeflow#2148)

30b5f6c

Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece)

ChenYi015 mentioned this pull request Sep 9, 2024

Cherry pick commits for releasing v2.0.0 #2156

Merged

4 tasks

ChenYi015 pushed a commit to ChenYi015/spark-operator that referenced this pull request Sep 9, 2024

Run e2e tests on Kind (kubeflow#2148)

7c4902b

Signed-off-by: Jacob Salway <jacob.salway@gmail.com> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <github@chenyicn.net>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run e2e tests on Kind #2148

Run e2e tests on Kind #2148

jacobsalway commented Aug 29, 2024 •

edited

Loading

jacobsalway Aug 30, 2024 •

edited

Loading

jacobsalway commented Aug 30, 2024

ChenYi015 left a comment

ChenYi015 Aug 30, 2024

jacobsalway Aug 31, 2024

ChenYi015 Aug 30, 2024

jacobsalway Aug 31, 2024 •

edited

Loading

ChenYi015 Aug 30, 2024

jacobsalway Aug 31, 2024

ChenYi015 Aug 30, 2024

jacobsalway Aug 31, 2024

ChenYi015 commented Aug 30, 2024

jacobsalway Aug 31, 2024 •

edited

Loading

google-oss-prow bot commented Sep 3, 2024

ChenYi015 commented Sep 3, 2024

Run e2e tests on Kind #2148

Run e2e tests on Kind #2148

Conversation

jacobsalway commented Aug 29, 2024 • edited Loading

Purpose

Change Category

Checklist

jacobsalway Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

jacobsalway commented Aug 30, 2024

ChenYi015 left a comment

Choose a reason for hiding this comment

ChenYi015 Aug 30, 2024

Choose a reason for hiding this comment

jacobsalway Aug 31, 2024

Choose a reason for hiding this comment

ChenYi015 Aug 30, 2024

Choose a reason for hiding this comment

jacobsalway Aug 31, 2024 • edited Loading

Choose a reason for hiding this comment

ChenYi015 Aug 30, 2024

Choose a reason for hiding this comment

jacobsalway Aug 31, 2024

Choose a reason for hiding this comment

ChenYi015 Aug 30, 2024

Choose a reason for hiding this comment

jacobsalway Aug 31, 2024

Choose a reason for hiding this comment

ChenYi015 commented Aug 30, 2024

jacobsalway Aug 31, 2024 • edited Loading

Choose a reason for hiding this comment

google-oss-prow bot commented Sep 3, 2024

ChenYi015 commented Sep 3, 2024

jacobsalway commented Aug 29, 2024 •

edited

Loading

jacobsalway Aug 30, 2024 •

edited

Loading

jacobsalway Aug 31, 2024 •

edited

Loading

jacobsalway Aug 31, 2024 •

edited

Loading