Skip to content

Commit

Permalink
Two phase admission documentation (kubernetes-sigs#1181)
Browse files Browse the repository at this point in the history
* Two phase admission documentation

* Review Remarks

* Review Remarks
  • Loading branch information
trasc authored and PBundyra committed Oct 26, 2023
1 parent d1c00a2 commit a2d0ca2
Show file tree
Hide file tree
Showing 3 changed files with 118 additions and 5 deletions.
17 changes: 12 additions & 5 deletions site/content/en/docs/concepts/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,18 +40,25 @@ Kueue. Sometimes referred to as _job_.
independently from [pod priority](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/).
This priority value from a `WorkloadPriorityClass` is only used for managing the queueing and preemption of [Workloads](#workload).

### [Admission Check](/docs/concepts/admission_check)

A mechanism allowing internal or external components to influence the timing of workloads admission.

![Components](/images/queueing-components.svg)

## Glossary

### Admission

The process of admitting a Workload to start (Pods to be created). A Workload
is admitted by a ClusterQueue according to the available resources and gets
resource flavors assigned for each requested resource.
### Quota Reservation

Sometimes referred to as _workload scheduling_ or _job scheduling_
(not to be confused with [pod scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)).
Is the process during which the kueue scheduler locks the resources needed by a workload within the targeted [ClusterQueues ResourceGroups](/docs/concepts/cluster_queue/#resource-groups)

### Admission

The process of admitting a Workload to start (Pods to be created). A Workload
is admitted when it has a Quota Reservation and all its [AdmissionCheckStates](/docs/concepts/admission_check)
are `Ready`.

### [Cohort](/docs/concepts/cluster_queue#cohort)

Expand Down
85 changes: 85 additions & 0 deletions site/content/en/docs/concepts/admission_check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
title: "Admission Check"
date: 2023-10-05
weight: 6
description: >
A mechanism allowing internal or external components to influence the timing of
workloads admission.
---

AdmissionChecks are a mechanism that allows Kueue to consider additional criteria before admitting a Workload.
When a ClusterQueue has AdmissionChecks configured, each of the checks has to provide a
positive signal to the Workload before it can be [Admitted](docs/concepts/#admission).

## Components

You can configure the mechanism using the following components:

### Admission Check

Admission check is a non-namespaced API object used to define details about an `AdmissionCheck` like:

- **controllerName** - It's an identifier for the controller that processes this AdmissionCheck, not necessarily a Kubernetes Pod or Deployment name. Cannot be empty.
- **retryDelayMinutes** - Specifies how long to keep the workload suspended after a failed check (after it transitioned to False). After that the check state goes to "Unknown". The default is 15 min.
- **parameters** - Identifies an additional resource providing additional parameters for the check.

An AdmissionCheck object looks like the following:
```yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
name: prov-test
spec:
controllerName: kueue.x-k8s.io/provisioning-request
retryDelayMinutes: 15
parameters:
apiGroup: kueue.x-k8s.io
kind: ProvisioningRequestConfig
name: prov-test-config
```
### ClusterQueue admissionChecks
Once defined, an AdmissionCheck can be referenced in the ClusterQueues' spec. All Workloads associated with the queue need to be evaluated by the AdmissionCheck's controller before being admitted.
Similarly to `ResourceFlavors`, if an `AdmissionCheck` is not found or its controller has not marked it as `Active`, the ClusterQueue will be marked as Inactive.

### AdmissionCheckState

AdmissionCheckState is the way the state of an AdmissionCkeck for a specific Workload is tracked.

AdmissionCheckStates are listed in the Workload's `.status.admissionCheckStates` field.

The status of a Workload that has pending AdmissionChecks looks like the following:
```yaml
status:
admission:
<...>
admissionChecks:
- lastTransitionTime: "2023-10-20T06:40:14Z"
message: ""
name: sample-prov
podSetUpdates:
- annotations:
cluster-autoscaler.kubernetes.io/consume-provisioning-request: job-prov-job-9815b-sample-prov
name: main
state: Ready
<...>
```

A list of states being maintained in the Status of all the monitored Workloads.

Kueue ensures that the list of the Workloads AdmissionCheckStates is in sync with the list of its associated ClusterQueue, Kueue adds new checks with the `Pending` state.

- Once a workload has QuotaReservation and all its AdmissionChecks are in "Ready" state it will become Admitted.
- If at least one of the Workloads AdmissionCheck is in the `Retry` state.
- If `Admitted` the workload is evicted.
- If the workload has `QuotaReservation` it will be release released.
- If at least one of the Workloads AdmissionCheck is in the `Rejected`:
- If `Admitted` the workload is evicted.
- If the workload has `QuotaReservation` it will be release released.
- The workload is marked as 'Finished' with a relevant failure message.

### Admission Check Controller

Is a component that monitors Workloads maintaining the content of its specific `admissionCheckStates` and the `Active` condition of the AdmissionChecks it's controlling.
The logic for how an AdmissionCheck changes states is not part of Kueue.
21 changes: 21 additions & 0 deletions site/static/examples/sample-admission-check.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
name: prov-test
spec:
controllerName: ProvisioningRequestController
parameters:
apiGroup: kueue.x-k8s.io
kind: ProvisioningRequestConfig
name: prov-test-config

---

apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
name: prov-test-config
spec:
provisioningClassName: test-class
parameters:
parma1: value1

0 comments on commit a2d0ca2

Please sign in to comment.