Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update affinity assistant creation implementation #6596

Merged
merged 1 commit into from
May 24, 2023
Merged

update affinity assistant creation implementation #6596

merged 1 commit into from
May 24, 2023

Conversation

pritidesai
Copy link
Member

@pritidesai pritidesai commented Apr 28, 2023 •
edited
Loading

Changes

Before this commit, the affinity assistant was created in the beginning of the pipelineRun. And the same affinity assistant was used throughout the entire lifecycle of a pipelineRun. Now, there could be a case when the node on which affinity assistant pod is created is cordoned for maintenance purpose. In this case, the rest of the pipelineRun is stuck and cannot run to the competition since the affinity assistant (StatefulSet) tries to schedule rest of the pods (taskRuns) on the same node but that node is cordoned or not scheduling anything new.

This commit always makes an attempt to create Affinity Assistant (StatefulSet) in case it does not exist. If it exist, the controller checks if the node on which Affinity Assistant pod is created is healthy to schedule subsequent pods. If not, the controller deletes Affinity Assistant pod so that StatefulSet can upscale the replicas (set to 1) on other node in the cluster.

Closes #6586.

Testing is pending, will update the PR after testing locally.

These changes are now tested locally on a cluster with two nodes and works as expected. The affinity assistant pod along with remaining taskRun pods are created on a healthy node when the existing node is cordoned.

/kind bug

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
  • Has Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Resilient Affinity Assistant - make sure the Affinity Assistant pod is always on a healthy node during the entire life cycle of the pipelineRun

@tekton-robot
Copy link
Collaborator

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@tekton-robot tekton-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Apr 28, 2023
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Apr 28, 2023 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: pritidesai / name: Priti Desai (f19cef0)

@tekton-robot tekton-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 28, 2023
@tekton-robot tekton-robot requested review from abayer and imjasonh April 28, 2023 05:48
@pritidesai
Copy link
Member Author

@lbernick @skaegi

This is an alternate solution to fix the pipelineRun with affinity assistant where the node could go down while the pipelineRun is still running.

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 75.7% -17.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 75.7% -17.3

@pritidesai pritidesai force-pushed the affinity_assistant branch from f5e2595 to c741725 Compare May 6, 2023 05:48
@tekton-robot tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 6, 2023
@pritidesai pritidesai force-pushed the affinity_assistant branch from c741725 to c548577 Compare May 6, 2023 05:48
@pritidesai pritidesai marked this pull request as ready for review May 6, 2023 05:48
@tekton-robot tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2023
@tekton-robot tekton-robot requested review from dibyom and lbernick May 6, 2023 05:48
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@pritidesai pritidesai force-pushed the affinity_assistant branch from c548577 to e28348b Compare May 6, 2023 23:35
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@pritidesai pritidesai force-pushed the affinity_assistant branch from e28348b to 4dfbf1f Compare May 7, 2023 00:13
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 92.6% -0.3

@pritidesai
Copy link
Member Author

thanks a bunch @lbernick 🙏

@QuanZhang-William please let me know if you have any feedback on these changes, thanks!

@pritidesai pritidesai force-pushed the affinity_assistant branch from 7453f53 to bb122cb Compare May 19, 2023 05:51
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@QuanZhang-William
Copy link
Member

thanks a bunch @lbernick 🙏

@QuanZhang-William please let me know if you have any feedback on these changes, thanks!

Thanks @pritidesai ! LGMT!

@pritidesai pritidesai force-pushed the affinity_assistant branch from bb122cb to 0b245f7 Compare May 19, 2023 16:35
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@imjasonh imjasonh removed their request for review May 19, 2023 16:52
@jerop jerop self-assigned this May 22, 2023
afrittoli
afrittoli reviewed May 22, 2023
View reviewed changes
docs/developers/affinity-assistant.md
affinity-assistant-c7b485007a-0 1/1 Running 0 4s 10.244.2.144 kind-multinode-worker2 <none> <none>
```

And, the `pipelineRun` finishes to completion:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's something missing here.
For the pipeline run to succeed, the operator performing the cluster upgrade needs to wait until there is no Tekton running pod on the node before draining it.

Since the affinity assistant moved to a new node, new tasks will start on different nodes, but running tasks must complete before the update may continue, or the pipeline run will fail.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

draining is disruptive whereas cordoning is not.

We are emphasizing in this doc that the affinity assistant mechanism is capable of handling the node being cordoned (scheduling is disabled).

Now, cordon that node to mark it unschedulable for any new pods:

kubectl cordon kind-multinode-worker1
node/kind-multinode-worker1 cordoned

The node is cordoned:

kubectl get node
NAME                           STATUS                     ROLES           AGE   VERSION
kind-multinode-control-plane   Ready                      control-plane   13d   v1.26.3
kind-multinode-worker1         Ready,SchedulingDisabled   <none>          13d   v1.26.3
kind-multinode-worker2         Ready                      <none>          13d   v1.26.3

@pritidesai
update affinity assistant creation implementation
0b7ead8 
Before this commit, the affinity assistant was created in the beginning of
the pipleineRun. And the same affinity assistant was relied on for the
entire lifecycle of a PR. Now, there could be a case when the node
on which affinity assistant pod is created goes down. In this case,
the rest of the pipelineRun is stuck and cannot run to the completition
since the affinity assistant (StatefulSet) tries to schedule rest
of the pods (taskRuns) on the same node but that node is cordoned or not
scheduling anything new.

This commit always makes an attempt to create Affinity Assistant (StatefulSet)
in case it does not exist. If it exist, the controller checks if the node on
which Affinity Assistant pod is created is healthy to schedule subsequent
pods. If not, the controller deletes Affinity Assistant pod so that
StatefulSet can upscale the replicas (set to 1) on other node in the cluster.

Signed-off-by: Priti Desai <pdesai@us.ibm.com>
@pritidesai pritidesai force-pushed the affinity_assistant branch from 0b245f7 to 0b7ead8 Compare May 22, 2023 18:52
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/affinity_assistant.go 93.0% 94.5% 1.5

@pritidesai
Copy link
Member Author

Hey @afrittoli @jerop any outstanding review on this? I would like to get it in 0.48. This is open since past three weeks and have gone through multiple rounds of reviews. Please let me know if there is anything I need to address. Thanks!

@skaegi
Copy link
Contributor

skaegi commented May 24, 2023 •
edited
Loading

Just to add further voice to this as we would really like to see this in 0.48. This PR is important to us as we currently have a problem when performing maintenance on nodes in our cluster. This PR let's us run more of a lights out operation vs. requiring manual intervention.

afrittoli
afrittoli reviewed May 24, 2023
View reviewed changes
Copy link
Member

@afrittoli afrittoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates!
/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label May 24, 2023
@tekton-robot tekton-robot merged commit 6590b21 into tektoncd:main May 24, 2023
@pritidesai pritidesai deleted the affinity_assistant branch May 24, 2023 18:01
@pritidesai
Copy link
Member Author

We would like to upgrade to the latest LTS release which is 0.47. These changes along with the PR #6860 are required for us to manage the clusters. I would like to request cherry picking these changes into 0.47. Thanks!

/cherry-pick release-v0.47.x

@tekton-robot tekton-robot mentioned this pull request Jun 22, 2023
Merged
@tekton-robot
Copy link
Collaborator

@pritidesai: new pull request created: #6863

In response to this:

We would like to upgrade to the latest LTS release which is 0.47. These changes along with the PR #6860 are required for us to manage the clusters. I would like to request cherry picking these changes into 0.47. Thanks!

/cherry-pick release-v0.47.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@QuanZhang-William QuanZhang-William QuanZhang-William left review comments

@afrittoli afrittoli afrittoli left review comments

@lbernick lbernick lbernick approved these changes

@abayer abayer Awaiting requested review from abayer

@dibyom dibyom Awaiting requested review from dibyom

Assignees

@afrittoli afrittoli

@jerop jerop

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
Pipelines v0.49
Development

Successfully merging this pull request may close these issues.

Affinity assistant deadlock during node maintenance
7 participants
@pritidesai @tekton-robot @lbernick @afrittoli @QuanZhang-William @skaegi @jerop