Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delay removal of orphanPVC to avoid the removal of PVC in use #67

Merged
merged 4 commits into from
Jun 22, 2023

Conversation

gurjotkaur20
Copy link
Contributor

Fixes #32

Description

Possibly a race condition is causing removal of PVC in use.

The solution in the PR: Added two labels toBeDeleted and deletionTimestamp to mark the pvc for deletion. Do not delete the pvc right away, wait for 60s to delete it and avoid the race condition.


This PR has:

  • been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
  • been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

Key changed/added files in this PR
  • controllers/druid/handler.go
  • controllers/druid/util.go

@AdheipSingh
Copy link
Contributor

@gurjotkaur20 this is great !

The only thing which i might want to do is add a finalizer instead of labels ( cc @avtarOPS ) Regardless, ill take look at it.
Thanks

return err

if _, ok := pvc.GetLabels()["toBeDeleted"]; ok {
deletionTS := pvc.GetLabels()["deletionTimestamp"]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can change the label name as the same label is used by the finalizer.

err := writers.Delete(ctx, sdk, drd, pvcList[i], emitEvents, &client.DeleteAllOfOptions{})
if err != nil {
return err

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This became way too big, we should refactor this, separating it into functions

if err != nil {
return err

if _, ok := pvc.GetLabels()["toBeDeleted"]; ok {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the below should be consts

@gurjotkaur20
Copy link
Contributor Author

I have refactored the code and updated the label name.

@itamar-marom
Copy link
Collaborator

I have refactored the code and updated the label name.

Fine by me :)
@AdheipSingh

Comment on lines 512 to 514
} else {
// do nothing
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this block ?

Comment on lines 526 to 528
} else {
// do nothing
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, why do we need this block ? can return nil ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, why do we need this block ? can return nil ?

Done changes for both else blocks.

@gurjotkaur20 gurjotkaur20 force-pushed the master branch 2 times, most recently from d6a50d2 to 7212552 Compare June 21, 2023 10:34
@AdheipSingh AdheipSingh merged commit 45b59e1 into datainfrahq:master Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

deleteOrphanPvc deleted PVC in use
5 participants