Support extended kube-scheduler as batch scheduler

[ChenYi015]

Purpose of this PR

Close #1943

Proposed changes:

• Support extended kube-scheduler as batch scheduler by using scheduler-plugins.

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[ChenYi015]

Testing

1. Deploy a kube-scheduler with coscheduling enabled. Ref: kubernetes-sigs/scheduler-plugins.

The following example can be used to deploy a kube-scheduler named kube-scheduler:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-scheduler

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-scheduler
rules:
- apiGroups: ["", "events.k8s.io"]
  resources: ["events"]
  verbs: ["create", "patch", "update"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
  resourceNames: ["kube-scheduler"]
  resources: ["leases"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["create"]
- apiGroups: [""]
  resourceNames: ["kube-scheduler"]
  resources: ["endpoints"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["delete", "get", "list", "watch"]
- apiGroups: [""]
  resources: ["bindings", "pods/binding"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/status"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["replicationcontrollers", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps", "extensions"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims", "persistentvolumes"]
  verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["subjectaccessreviews"]
  verbs: ["create"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csinodes", "storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["scheduling.sigs.k8s.io"]
  resources: ["podgroups", "elasticquotas"]
  verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["scheduling.x-k8s.io"]
  resources: ["podgroups"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csistoragecapacities", "csidrivers"]
  verbs: ["get", "list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-scheduler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-scheduler
subjects:
- kind: ServiceAccount
  name: kube-scheduler
  namespace: default

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-scheduler-config
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: true
    profiles:
    - schedulerName: kube-scheduler
      plugins:
        multiPoint:
          enabled:
          - name: Coscheduling
        queueSort:
          enabled:
          - name: Coscheduling
          disabled:
          - name: "*"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      component: kube-scheduler
  template:
    metadata:
      labels:
        component: kube-scheduler
    spec:
      containers:
      - name: kube-scheduler
        image: registry.k8s.io/scheduler-plugins/kube-scheduler:v0.28.9
        command:
        - kube-scheduler
        - --config=/etc/kubernetes/config/scheduler-config.yaml
        - --leader-elect-resource-name=kube-scheduler
        - --leader-elect-resource-namespace=default
        - --v=5
        volumeMounts:
        - name: scheduler-config
          mountPath: /etc/kubernetes/config
      volumes:
      - name: scheduler-config
        configMap:
          name: kube-scheduler-config
      serviceAccount: kube-scheduler

2. Build the image and pushed to your image registry:

IMAGE_REGISTRY=xxx
IMAGE_REPOSITORY=xxx
IMAGE_TAG=kube-scheduler-test

make docker-buildx IMAGE_REGISTRY=${IMAGE_REGISTRY} IMAGE_REPOSITORY=${IMAGE_REPOSITORY} IMAGE_TAG=${IMAGE_TAG}

3. Install the Helm chart with custom kube-scheduler enabled:

helm install spark-operator charts/spark-operator-chart \
    --create-namespace \
    --namespace spark-operator \
    --set image.registry=${IMAGE_REGISTRY} \
    --set image.repository=${IMAGE_REPOSITORY} \
    --set image.tag=${IMAGE_TAG} \
    --set controller.batchScheduler.enable=true \
    --set 'controller.batchScheduler.kubeSchedulerNames={kube-scheduler}'

4. Submit example SparkApplication:

kubectl apply -f examples/spark-pi-kube-scheduler.yaml

A pod group will be created:

$ kubectl describe podgroups.scheduling.x-k8s.io spark-pi-kube-scheduler-pg
Name:         spark-pi-kube-scheduler-pg
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  scheduling.x-k8s.io/v1alpha1
Kind:         PodGroup
Metadata:
  Creation Timestamp:  2024-08-22T02:49:17Z
  Generation:          1
  Owner References:
    API Version:           sparkoperator.k8s.io/v1beta2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  SparkApplication
    Name:                  spark-pi-kube-scheduler
    UID:                   063db00f-e1fd-4469-8e78-cffe181240d1
  Resource Version:        36451569
  UID:                     4bab4bbd-d410-44a4-b7d1-86757ab16b15
Spec:
  Min Member:  1
  Min Resources:
    Cpu:     101
    Memory:  10752m
Events:      <none>

If there is no enough resources for all spark pods, then driver pod will be pending:

$ kubectl describe pod spark-pi-kube-scheduler-driver
...
Events:
  Type     Reason               Age                  From            Message
  ----     ------               ----                 ----            -------
  Warning  FailedScheduling     19m                  kube-scheduler  0/3 nodes are available: resource gap: map[cpu:{{13375 -3} {<nil>}  DecimalSI}]. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling., PodGroup default/spark-pi-kube-scheduler-pg gets rejected due to Pod spark-pi-kube-scheduler-driver is unschedulable even after PostFilter.

[ChenYi015]

/assign @vara-bonthu @yuchaoran2011

[vara-bonthu]

Testing

1. Deploy a kube-scheduler with coscheduling enabled. Ref: kubernetes-sigs/scheduler-plugins.

The following example can be used to deploy a kube-scheduler named kube-scheduler:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-scheduler

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-scheduler
rules:
- apiGroups: ["", "events.k8s.io"]
  resources: ["events"]
  verbs: ["create", "patch", "update"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
  resourceNames: ["kube-scheduler"]
  resources: ["leases"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["create"]
- apiGroups: [""]
  resourceNames: ["kube-scheduler"]
  resources: ["endpoints"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["delete", "get", "list", "watch"]
- apiGroups: [""]
  resources: ["bindings", "pods/binding"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/status"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["replicationcontrollers", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps", "extensions"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims", "persistentvolumes"]
  verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["subjectaccessreviews"]
  verbs: ["create"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csinodes", "storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["scheduling.sigs.k8s.io"]
  resources: ["podgroups", "elasticquotas"]
  verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["scheduling.x-k8s.io"]
  resources: ["podgroups"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csistoragecapacities", "csidrivers"]
  verbs: ["get", "list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-scheduler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-scheduler
subjects:
- kind: ServiceAccount
  name: kube-scheduler
  namespace: default

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-scheduler-config
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: true
    profiles:
    - schedulerName: kube-scheduler
      plugins:
        multiPoint:
          enabled:
          - name: Coscheduling
        queueSort:
          enabled:
          - name: Coscheduling
          disabled:
          - name: "*"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      component: kube-scheduler
  template:
    metadata:
      labels:
        component: kube-scheduler
    spec:
      containers:
      - name: kube-scheduler
        image: registry.k8s.io/scheduler-plugins/kube-scheduler:v0.28.9
        command:
        - kube-scheduler
        - --config=/etc/kubernetes/config/scheduler-config.yaml
        - --leader-elect-resource-name=kube-scheduler
        - --leader-elect-resource-namespace=default
        - --v=5
        volumeMounts:
        - name: scheduler-config
          mountPath: /etc/kubernetes/config
      volumes:
      - name: scheduler-config
        configMap:
          name: kube-scheduler-config
      serviceAccount: kube-scheduler

2. Build the image and pushed to your image registry:

IMAGE_REGISTRY=xxx
IMAGE_REPOSITORY=xxx
IMAGE_TAG=kube-scheduler-test

make docker-buildx IMAGE_REGISTRY=${IMAGE_REGISTRY} IMAGE_REPOSITORY=${IMAGE_REPOSITORY} IMAGE_TAG=${IMAGE_TAG}

3. Install the Helm chart with custom kube-scheduler enabled:

helm install spark-operator charts/spark-operator-chart \
    --create-namespace \
    --namespace spark-operator \
    --set image.registry=${IMAGE_REGISTRY} \
    --set image.repository=${IMAGE_REPOSITORY} \
    --set image.tag=${IMAGE_TAG} \
    --set controller.batchScheduler.enable=true \
    --set 'controller.batchScheduler.kubeSchedulerNames={kube-scheduler}'

4. Submit example SparkApplication:

kubectl apply -f examples/spark-pi-kube-scheduler.yaml

A pod group will be created:

$ kubectl describe podgroups.scheduling.x-k8s.io spark-pi-kube-scheduler-pg
Name:         spark-pi-kube-scheduler-pg
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  scheduling.x-k8s.io/v1alpha1
Kind:         PodGroup
Metadata:
  Creation Timestamp:  2024-08-22T02:49:17Z
  Generation:          1
  Owner References:
    API Version:           sparkoperator.k8s.io/v1beta2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  SparkApplication
    Name:                  spark-pi-kube-scheduler
    UID:                   063db00f-e1fd-4469-8e78-cffe181240d1
  Resource Version:        36451569
  UID:                     4bab4bbd-d410-44a4-b7d1-86757ab16b15
Spec:
  Min Member:  1
  Min Resources:
    Cpu:     101
    Memory:  10752m
Events:      <none>

If there is no enough resources for all spark pods, then driver pod will be pending:

$ kubectl describe pod spark-pi-kube-scheduler-driver
...
Events:
  Type     Reason               Age                  From            Message
  ----     ------               ----                 ----            -------
  Warning  FailedScheduling     19m                  kube-scheduler  0/3 nodes are available: resource gap: map[cpu:{{13375 -3} {<nil>}  DecimalSI}]. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling., PodGroup default/spark-pi-kube-scheduler-pg gets rejected due to Pod spark-pi-kube-scheduler-driver is unschedulable even after PostFilter.

Please add this to examples, as it's helpful for users to build a custom scheduler on k8s.

[vara-bonthu]

Thank you for this contribution. It would be beneficial to add this to the examples section, as it provides users with valuable guidance on building a custom scheduler for Kubernetes.

I've observed that with Spark workloads, the default Kubernetes scheduler uses a LeastAllocated approach for pod bin-packing. However, this can lead to significant resource wastage across nodes. A more efficient method would be to use the MostAllocated approach, which can be configured using custom batch schedulers like the one demonstrated here.

This example showcases how to leverage custom schedulers to optimize resource allocation, a feature currently available with other batch schedulers such as YuniKorn. Including this example would provide users with a practical demonstration of advanced scheduling techniques for Spark on Kubernetes.

Could you please consider adding this example to the repository? It would serve as an excellent resource for users looking to maximize cluster efficiency with Spark workloads.

An update to your example

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-scheduler-config
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: true
    profiles:
    - schedulerName: kube-scheduler
      plugins:
        multiPoint:
          enabled:
          - name: Coscheduling
        queueSort:
          enabled:
          - name: Coscheduling
          disabled:
          - name: "*"
        score:
          enabled:
          - name: NodeResourcesFit
            weight: 1
      pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated
            resources:
            - name: cpu
              weight: 1
            - name: memory
              weight: 1

[jacobsalway]

If we add docs or examples on running a customer scheduler, I wonder if it'd be worth adding a warning on the complexities of doing so? At least on managed Kubernetes services like EKS, the scheduler is abstracted away from you and running a multi-scheduler cluster comes with its own headaches and gotchas.

[vara-bonthu]

If we add docs or examples on running a customer scheduler, I wonder if it'd be worth adding a warning on the complexities of doing so? At least on managed Kubernetes services like EKS, the scheduler is abstracted away from you and running a multi-scheduler cluster comes with its own headaches and gotchas.

That's a valid point. I agree that we may need to address potential issues in the example. Let's not include that as an example.
While EKS abstracts schedulers, the default scheduler can lead to resource inefficiencies due to poor bin packing, an issue I've seen with several customers. Although implementing custom batch schedulers like YuniKorn could be a solution, not all customers are eager to adopt an additional addon; many would rather see enhancements to the existing scheduler. Hopefully, this will soon be a configurable option for EKS clusters.

[ChenYi015]

The example kube-scheduler deployment is intended only for testing purpose and is not production-ready. And it is not recommended to deploy multiple kube-schedulers in managed Kubernetes services, where the scheduler is typically managed by the provider. I think K8s service providers should offer an option to configure the default scheduler to enable coscheduling.

[yuchaoran2011]

The above statement on why this setup is not production ready should be included in the documentation (a simple paragraph will do). Otherwise the PR LGTM

[ChenYi015]

The above statement on why this setup is not production ready should be included in the documentation (a simple paragraph will do).
@yuchaoran2011 Agree, I will update the related docs once release 2.0.0 is published.

[yuchaoran2011]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu, yuchaoran2011

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu,yuchaoran2011]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment