Asynchronous preemption in the scheduler

[sanposhiho]

Enhancement Description

• One-line enhancement description (can be used as a release note): The scheduler runs the preemption (PostFilter) asynchronously to enhance the scheduling throughput on the failure scenarios.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4832-async-preemption/README.md

• Discussion Link: Run scheduler PostFilters asynchronously to improve throughput kubernetes#126858

• Primary contact (assignee): @sanposhiho

• Responsible SIGs: SIG-Scheduling

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): v1.32
  • Beta release target (x.y):
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): KEP-4832: Asynchronous preemption in the scheduler #4833
  • Code (k/k) update PR(s): feature(KEP-4832): asynchronous preemption kubernetes#128170
  • Docs (k/website) update PR(s): KEP-4832: add SchedulerAsyncPreemption to the doc website#48407

• Beta

  • KEP (k/enhancements) update PR(s): kep-4832: promote to beta #5127
  • Code (k/k) update PR(s): feat: graduate the async preemption feature to beta kubernetes#130550
  • Docs (k/website) update(s): KEP-4832: graduate the async preemption feature to beta website#49996

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig scheduling
/assign

[alculquicondor]

/label lead-opted-in

[impact-maker]

Hello @sanposhiho @alculquicondor 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage alpha for v1.32 (correct me, if otherwise).

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday, October 3rd, 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[tjons]

Hello 👋, v1.32 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.32, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

[tjons]

/milestone clear

[sanposhiho]

@alculquicondor
Looks like we just missed the deadline.
I believe we can make an exception request because we only need an approval for PRR, and actually @/wojtek-t already mentioned it's generally OK for alpha. What do you think?

[sanposhiho]

I filed an exception request for it. cc @wojtek-t.

[sreeram-venkitesh]

/stage alpha

[tjons]

With all the requirements met, this enhancement's exception is granted and this enhancement is now tracked for enhancements freeze!

[chanieljdan]

Hi @sanposhiho 👋, 1.32 Release Docs Lead here.

Does this enhancement work planned for 1.32 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[chanieljdan]

Hi @sanposhiho 👋, 1.32 Release Docs Lead here.

Does this enhancement work planned for 1.32 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

Hi @sanposhiho 👋,

Just a reminder to open a placeholder PR against dev-1.32 branch in the k/website repo for this (steps available here). The deadline for this is a week away at Thursday October 24, 2024 18:00 PDT.

Thanks,

Daniel

[sanposhiho]

Created: kubernetes/website#48407

[wrkode]

👋 Hi there, William here from v1.32 Comms
We'd love for you to consider writing a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt-in, let us know and open a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[sanposhiho]

I believe we don't need a blog post for this. (at least, within this release)

[wrkode]

Thank's for your prompt feedback @sanposhiho

[tjons]

Hey again @sanposhiho 👋 v1.32 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Friday 8th November 2024 / 19:00 PDT Thursday 7th November 2024 .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• [Umbrella] KEP-4832(alpha): Asynchronous preemption alpha requirement kubernetes#128014
• KEP-4832(alpha): The core implementation change in the preemption plugin, along with the feature gate kubernetes#128020

Additionally, please let me know if there are any other PRs in k/k not listed in the description or not linked with this GitHub issue that we should track for this KEP, so that we can maintain accurate status.

The status of this enhancement is marked as at risk for code freeze.

If you anticipate missing code freeze, you can file an exception request in advance. Thank you!

[dom4ha]

Not sure for how long we usually wait to spot potential issues, so not certain what to suggest here.

As a side note, the preemption process itself has a few important issues. One of them is the performance, which async preemption helps to mitigate, so it's an argument for moving forward.

However, the current process cannot preempt pods running on a different node than the candidate wants to schedule. I'm looking into this issue and I hope we will be able to do some brainstorming soon how it could be addressed as well. Once we do that, maybe we;d have more clear picture what actions we should take and when.

[sanposhiho]

We have a plugin for that as a subproject.
https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg%2Fcrossnodepreemption%2FREADME.md

We can consider implementing the same if there's many-enough people wanting it. But, I'm not sure if we're getting such feedback from various folks.

[dom4ha]

Thanks Kensei for the pointer. IIUC the plugin tries the brute force approach, which would impact performance a lot, so I'd think about something more performance oriented.

Regarding the interest, I suspect that development around DRA and other more complex dependencies, will force us to look closer into this problem.

[dom4ha]

@sanposhiho Is there also an issue describing the problem and considered solutions?

[sanposhiho]

cc @Huang-Wei, might know more context for the plugin.

I'd think about something more performance oriented

Nice, I didn't grasp its algorithm actually.
It would be great to make an improvement in the algorithm, regardless of whether we decide to or not to move it to the upstream.

I suspect that development around DRA and other more complex dependencies, will force us to look closer into this problem.
...
Is there also an issue describing the problem and considered solutions?

DRA has its own PostFilter and hence I'm not sure if they really would.
But, feel free to create an issue for the discussion. (AFAIK, we don't have an issue) Either way, here isn't an appropriate place to continue this discussion.
We can discuss the solution, whether we need to implement it within the upstream scheduler, etc.

[alculquicondor]

Can we move discussions about cross-node preemption to a separate issue? It's not related to this enhancement.

In the meantime, I'm not hearing anything against moving to beta.

/label lead-opted-in

[ArkaSaha30]

Hello @sanposhiho 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage beta for v1.33 (correct me, if otherwise)
/stage beta

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.33.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• KEP needs to be updated with the target release
• Description needs to reflect the correct Beta release target (x.y): v1.33
• KEP needs to be updated with prr-approvers as per PRR yaml

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hi @sanposhiho 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #4832 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[sanposhiho]

@dipesh-rawat
Oops, looks like I missed some fields to update. Will be updated at
#5161

[dipesh-rawat]

Hello @sanposhiho 👋, 1.33 Enhancements team here,

Now that PRs #5161 #5127 has been merged, all the KEP requirements are in place and merged into k/enhancements.

Before the enhancement freeze, it would be appreciated if following nit could be addressed:

• Update issue description to add following:
  • Beta release target (x.y): 1.33
  • Add section for Beta stage
    • Beta
      • KEP (k/enhancements) update PR(s):
        Update kep.yaml for KEP-4832 #5161
        kep-4832: promote to beta #5127
      • Code (k/k) update PR(s):
      • Docs (k/website) update PR(s):

Aside from the minor nit mentioned above, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

(cc: @ArkaSaha30)

/label tracked/yes

[dipesh-rawat]

/stage beta

[ArvindParekh]

Hello, @sanposhiho 👋, v1.33 Docs Shadow here.

Does this enhancement work planned for v1.33 require any new docs or modifications to existing docs?

If so, please follow the steps here to open a PR against the dev-1.33 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 27th February 2025 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[aakankshabhende]

Hi @sanposhiho 👋 -- this is Aakanksha (@aakankshabhende ) from the 1.33 Communications Team!

For the 1.33 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we'd love for you to consider writing one for your enhancement!

As you may be aware, feature blogs are a great way to communicate to users about features which fall into (but not limited to) the following categories:

• This introduces some breaking change(s)
• This has significant impacts and/or implications to users
• ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Wednesday, 5th March, 2025? For more information about writing a blog, please find the blog contribution guidelines 📚

Tip

Some timeline to keep in mind:

• 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
• Monday, 7th April, 2025: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[sanposhiho]

I think this feature is so small and internal that we don't have to write a blog post.