[local-volume] remove pv if disk/dir is unmounted/removed

[weiwei04]

like #250 mentioned, local-volume current usage is like:

• you set up a hostdir for each storage-class (eg. /mnt/disks/)
• within that folder, you mount disks (vol1, vol2, whatever...)
• if an app requests a pvc using that class, one of the /mnt/disks/{volume} is bounded exclusively

Here I have a small questions:
Shall we delete a pv if the backing disk/dir for that pv is unmounted/deleted?

1. record the pv's backend storage media into pv spec(maybe use annotation), type maybe: MountedDisk, LocalDir, BlockDevice...
2. create a reconciler object, to ReconcileLocalVolumes(), for unbouned pv:
   a. If the MountedDisk is unmounted, we should delete the pv.
   b. If the LocalDir is deleted, we should delete the pv.

[ianchakeres]

Extending this concept a little... I think it would be interesting if the underlying capacity of a backing disk changed, that the related unbound PV be updated. This update could be achieved by an unbound PV delete followed by an unbound PV create.

[weiwei04]

@msau42 any comments on this issue? If it is a sound needs(to delete pv if the backing disk/dir is unmounted/deleted), I just have the time and needs to open a pr to implement this feature :)

[msau42]

I can see the use case, but I'm a little wary of automatically deleting the PV because of race conditions between deleting the PV and a PVC binding to it. We would need to add a safeguard that you can't delete a PV if it's still bound to a PVC. This is also something that no other volume plugin does.

If I understand correctly, you need this as a workaround until we have dynamic provisioning. Can you just add the PV deletion in your external entity that is creating/deleting directories? Note, you would still have to deal with the race condition.

[msau42]

It's possible a disk could disappear temporarily and come back. We don't want to just blindly delete the PV and potentially cause the application to lose their data unnecessarily.

Instead of directly deleting the PV, the approach I prefer to take is to have health monitoring on the PV and be able to mark a PV as unhealthy if there's a problem with the underlying disk, and let the user define policies on how they want to handle it. This way we can also retain some history on the PV.

[msau42]

/assign

[msau42]

/area local-volume

[msau42]

Now that we have PVC and PV finalizers in 1.10, I think this logic can be implemented. Anyone interested in working on this?

[msau42]

Need to walk through the scenario of what will happen if we hit the race condition of deleting a PV after it gets bound

[msau42]

Some of this might be covered by @NickrenREN health monitoring proposal

[msau42]

Tracking this as part of health monitoring: kubernetes-sigs/sig-storage-local-static-provisioner#10
/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

Tracking this as part of health monitoring: kubernetes-sigs/sig-storage-local-static-provisioner#10
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.