Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add defence logic for PV without claime ref #796

Merged

Conversation

zezaeoh
Copy link

@zezaeoh zezaeoh commented Oct 14, 2022

What type of PR is this?

/kind bug

What this PR does / why we need it:
App crashes when deleting PVs without ClaimRef.

I1014 03:34:47.331898       1 controller.go:1471] delete "pvc-a6f2adab-036b-4833-b5c7-eaf84626c351": started
E1014 03:34:47.332068       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 257 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1805020?, 0x29703c0})
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0005af920?})
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x1805020, 0x29703c0})
	/go/pkg/csiprow.XXXXDeFLHD/go-1.18/src/runtime/panic.go:838 +0x207
github.com/kubernetes-csi/external-provisioner/pkg/controller.(*csiProvisioner).getSecretsFromSC(0xc0003b8640, {0x1cdf738, 0xc0005b4300}, 0xc00071e500, 0x0, 0xc0006f08c0)
	/workspace/pkg/controller/controller.go:1272 +0x2a2
...

When we use Custom Resource RunnerSets of action-runner-controller,

That controller sets specific PV's ClaimRef to nil to release from PVC resource
https://github.com/actions-runner-controller/actions-runner-controller/blob/master/controllers/sync_volumes.go#L161-L176

In that case, this can lead to many PVs without a ClaimRef.

Which issue(s) this PR fixes:

Fixes #None

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix nil pointer crash for PV without ClaimRef

@k8s-ci-robot k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Oct 14, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Oct 14, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: zezaeoh / name: Zezae Oh (06ccf62)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Oct 14, 2022
@k8s-ci-robot
Copy link
Contributor

Welcome @zezaeoh!

It looks like this is your first PR to kubernetes-csi/external-provisioner 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/external-provisioner has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 14, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @zezaeoh. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 14, 2022
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 14, 2022
@RaunakShah
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 14, 2022
@RaunakShah
Copy link
Contributor

RaunakShah commented Oct 14, 2022

Can you add some logs to the PR description to show that this error is triggered?

@zezaeoh zezaeoh requested review from RaunakShah and removed request for ggriffiths and Jiawei0227 October 14, 2022 07:57
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 14, 2022
@zezaeoh zezaeoh force-pushed the fix/defence-non-claimed-pv branch from ff760c9 to 84454df Compare October 14, 2022 07:57
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 14, 2022
@zezaeoh
Copy link
Author

zezaeoh commented Oct 27, 2022

@RaunakShah

Could you review again? :)

@RaunakShah
Copy link
Contributor

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 27, 2022
Copy link
Contributor

@Madhu-1 Madhu-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commits need to be squashed into one.

@@ -1283,6 +1283,9 @@ func (p *csiProvisioner) getSecretsFromSC(ctx context.Context, volume *v1.Persis
}

// Resolve provision secret credentials.
if volume.Spec.ClaimRef == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we dont have the secrets, dont we need to allow deletion? lets the CSI driver try to handle it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, action-runner-controller deletes PV's ClaimRef intentionally to make PV orphan and reuse.
So, i think we should leave this PV as orphan

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is for the ReclaimPolicy=Retained PV right, not for the ReclaimPolicy=Delete?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh.. you are right! @Madhu-1
I'll follow your suggestion :) thx!!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM, you need review or approval from @msau42 @jsafrane

@zezaeoh zezaeoh force-pushed the fix/defence-non-claimed-pv branch from 84454df to 3441650 Compare October 27, 2022 23:50
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 27, 2022
@zezaeoh zezaeoh requested review from Madhu-1 and removed request for RaunakShah October 27, 2022 23:58
@zezaeoh zezaeoh force-pushed the fix/defence-non-claimed-pv branch from 3441650 to bb3185c Compare October 29, 2022 03:52
@zezaeoh zezaeoh requested review from RaunakShah and Madhu-1 and removed request for Madhu-1 and RaunakShah October 29, 2022 04:15
@zezaeoh
Copy link
Author

zezaeoh commented Nov 2, 2022

@Madhu-1 @RaunakShah

Could you review again?

@RaunakShah
Copy link
Contributor

@zezaeoh changes look good, can you add also add some testing results to the description? For example, the logs showing that csi-provisioner doesn't crash but also that the CSI driver eventually does not delete the volume?

@jsafrane
Copy link
Contributor

jsafrane commented Nov 4, 2022

@zezaeoh do you know how it's possible that a volume for PV without ClaimRef is deleted? That should be nearly impossible.

@jsafrane
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 29, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, RaunakShah, zezaeoh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 29, 2022
@k8s-ci-robot k8s-ci-robot merged commit 3bfb915 into kubernetes-csi:master Nov 29, 2022
@zezaeoh zezaeoh deleted the fix/defence-non-claimed-pv branch December 3, 2022 07:34
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants