Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two phase admission documentation #1181

Merged
merged 3 commits into from
Oct 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions site/content/en/docs/concepts/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,18 +40,25 @@ Kueue. Sometimes referred to as _job_.
independently from [pod priority](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/).
This priority value from a `WorkloadPriorityClass` is only used for managing the queueing and preemption of [Workloads](#workload).

### [Admission Check](/docs/concepts/admission_check)

A mechanism allowing internal or external components to influence the timing of workloads admission.

![Components](/images/queueing-components.svg)

## Glossary

### Admission

The process of admitting a Workload to start (Pods to be created). A Workload
is admitted by a ClusterQueue according to the available resources and gets
resource flavors assigned for each requested resource.
### Quota Reservation

Sometimes referred to as _workload scheduling_ or _job scheduling_
(not to be confused with [pod scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)).
Is the process during which the kueue scheduler locks the resources needed by a workload within the targeted [ClusterQueues ResourceGroups](/docs/concepts/cluster_queue/#resource-groups)

### Admission

The process of admitting a Workload to start (Pods to be created). A Workload
is admitted when it has a Quota Reservation and all its [AdmissionCheckStates](/docs/concepts/admission_check)
are `Ready`.

### [Cohort](/docs/concepts/cluster_queue#cohort)

Expand Down
85 changes: 85 additions & 0 deletions site/content/en/docs/concepts/admission_check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
title: "Admission Check"
date: 2023-10-05
weight: 6
description: >
A mechanism allowing internal or external components to influence the timing of
workloads admission.
---

AdmissionChecks are a mechanism that allows Kueue to consider additional criteria before admitting a Workload.
When a ClusterQueue has AdmissionChecks configured, each of the checks has to provide a
positive signal to the Workload before it can be [Admitted](docs/concepts/#admission).

## Components

You can configure the mechanism using the following components:

### Admission Check

Admission check is a non-namespaced API object used to define details about an `AdmissionCheck` like:

- **controllerName** - It's an identifier for the controller that processes this AdmissionCheck, not necessarily a Kubernetes Pod or Deployment name. Cannot be empty.
- **retryDelayMinutes** - Specifies how long to keep the workload suspended after a failed check (after it transitioned to False). After that the check state goes to "Unknown". The default is 15 min.
- **parameters** - Identifies an additional resource providing additional parameters for the check.

An AdmissionCheck object looks like the following:
```yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
name: prov-test
spec:
controllerName: kueue.x-k8s.io/provisioning-request
retryDelayMinutes: 15
parameters:
apiGroup: kueue.x-k8s.io
kind: ProvisioningRequestConfig
name: prov-test-config
```
### ClusterQueue admissionChecks
Once defined, an AdmissionCheck can be referenced in the ClusterQueues' spec. All Workloads associated with the queue need to be evaluated by the AdmissionCheck's controller before being admitted.
Similarly to `ResourceFlavors`, if an `AdmissionCheck` is not found or its controller has not marked it as `Active`, the ClusterQueue will be marked as Inactive.

### AdmissionCheckState

AdmissionCheckState is the way the state of an AdmissionCkeck for a specific Workload is tracked.

AdmissionCheckStates are listed in the Workload's `.status.admissionCheckStates` field.

The status of a Workload that has pending AdmissionChecks looks like the following:
```yaml
status:
admission:
<...>
admissionChecks:
- lastTransitionTime: "2023-10-20T06:40:14Z"
message: ""
name: sample-prov
podSetUpdates:
- annotations:
cluster-autoscaler.kubernetes.io/consume-provisioning-request: job-prov-job-9815b-sample-prov
name: main
state: Ready
<...>
```

A list of states being maintained in the Status of all the monitored Workloads.

Kueue ensures that the list of the Workloads AdmissionCheckStates is in sync with the list of its associated ClusterQueue, Kueue adds new checks with the `Pending` state.

- Once a workload has QuotaReservation and all its AdmissionChecks are in "Ready" state it will become Admitted.
- If at least one of the Workloads AdmissionCheck is in the `Retry` state.
- If `Admitted` the workload is evicted.
- If the workload has `QuotaReservation` it will be release released.
- If at least one of the Workloads AdmissionCheck is in the `Rejected`:
- If `Admitted` the workload is evicted.
- If the workload has `QuotaReservation` it will be release released.
- The workload is marked as 'Finished' with a relevant failure message.

### Admission Check Controller

Is a component that monitors Workloads maintaining the content of its specific `admissionCheckStates` and the `Active` condition of the AdmissionChecks it's controlling.
The logic for how an AdmissionCheck changes states is not part of Kueue.
21 changes: 21 additions & 0 deletions site/static/examples/sample-admission-check.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
name: prov-test
spec:
controllerName: ProvisioningRequestController
parameters:
apiGroup: kueue.x-k8s.io
kind: ProvisioningRequestConfig
name: prov-test-config

---

apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
name: prov-test-config
spec:
provisioningClassName: test-class
parameters:
parma1: value1