Default LocalQueue

[KPostOffice]

What would you like to be added:

I would like to be able to indicate a default LocalQueue per namespace to be used for a workload

Why is this needed:

If an administrator creates a single LocalQueue per namespace that all workloads are expected to use, defaulting the queue let's quota consumers ignore the need to include Queue specific request labels in their workloads and enforces the need to be associated with a Queue in order to be scheduled.

Completion requirements:

• API changes to LocalQueue spec which allow admin to specify it as default
• Mutating webhook which will get the default LocalQueue and apply the appropriate label
• Validating webhook which ensures that only one LocalQueue is marked as default

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[tenzen-y]

@KPostOffice Do you assume that automatically https://kueue.sigs.k8s.io/docs/tasks/manage/setup_job_admission_policy/ generation?

[KPostOffice]

@KPostOffice Do you assume that automatically https://kueue.sigs.k8s.io/docs/tasks/manage/setup_job_admission_policy/ generation?

Are you referring to the example AdmissionPolicy outlined in the documentation?

[tenzen-y]

https://kueue.sigs.k8s.io/docs/tasks/manage/setup_job_admission_policy/

Yes, I indicated the ValidatingAdmissionPolicy.

[yaroslava-serdiuk]

/assign

[KPostOffice]

Yeah, the assumption here would be that the policy would be in place

[yaroslava-serdiuk]

If we are going to modify API to have "default" field, it might be tricky to validate that only one LQ is a default. Are we going to abandon setting LQ as default if another default LQ is present in the namespace (in this case we do not validate the object itself)? If we can have multiple default LQs, which one is an actual default? It could be a first one or the last one.

To avoid those questions and the API change, I suggest to default LocalQueue that has default name, WDYT?
However this might be a breaking change for someone.

[yaroslava-serdiuk]

@KPostOffice could you please tell if #2936 (comment) makes sense

[mwielgus]

Let's have the "default" default LocalQueue and put the whole feature behind a feature gate, so we don't break anyone by accident.

[KPostOffice]

Using the default LQ name behind a feature gate would be a fine solution from my perspective

[mimowo]

It feels this feature requires a KEP, even if there is no "API" changed, we still introduce a feature-gate. Also, I feel the mechanism will evolve in the future, so having some reference we can update will be useful. It does not need to be long, just one pager to collect the use-cases and the proposed solution.

[tenzen-y]

It feels this feature requires a KEP, even if there is no "API" changed, we still introduce a feature-gate. Also, I feel the mechanism will evolve in the future, so having some reference we can update will be useful. It does not need to be long, just one pager to collect the use-cases and the proposed solution.

+1

[mimowo]

/reopen
For the documentation update

[k8s-ci-robot]

@mimowo: Reopened this issue.

In response to this:

/reopen
For the documentation update

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yaroslava-serdiuk]

@mimowo could you specify what should be added to documentation?

[mimowo]

Basically, we want to have all features documented. Two places come to my mind (could be separate PRs):

• LocalQueue page: https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/docs/concepts/local_queue.md?plain=1
• a new section in the administrative tasks: https://github.com/kubernetes-sigs/kueue/tree/main/site/content/en/docs/tasks/manage/enforce_job_management, could be entitled "Setup Local Queue defaulting"

cc @PBundyra wdyt?

[PBundyra]

Basically, we want to have all features documented. Two places come to my mind (could be separate PRs):

• LocalQueue page: https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/docs/concepts/local_queue.md?plain=1
• a new section in the administrative tasks: https://github.com/kubernetes-sigs/kueue/tree/main/site/content/en/docs/tasks/manage/enforce_job_management, could be entitled "Setup Local Queue defaulting"

cc @PBundyra wdyt?

Agreed, I would also consider adding some note to the Run A Kubernetes Job page. However, I believe the note shouldn't be limited only to default LQ but it should also cover jobs without the LQ

[yaroslava-serdiuk]

Agreed, I would also consider adding some note to the Run A Kubernetes Job page. However, I believe the note shouldn't be limited only to default LQ but it should also cover jobs without the LQ

LocalQueueDefaulting is not limited to Kubernetes Job only, seems sigs/kueue/tree/main/site/content/en/docs/tasks/manage/enforce_job_management is the best place.