Provide a way to delete PV/PVCs

[smanpathak]

Upon mysql cluster deletion, the associated PV/PVCs are left intact. This is problematic when testing mysql-operator by keeping mysql cluster name same.
If mysql cluster is created with same name again, old PVs are re-attached. This might be fine when restoring accidentally deleted cluster, but problematic with testing. When existing volumes are attached, the percona image startup scripts find old data directories and do not update passwords specified for the users with new mysql cluster.
I think one way of doing this is by specifying attributes which permit pv deletion along with mysql cluster.

[calind]

I see two distinct issues here:

1. We should reset passwords when those set in secret change (Reset passwords when the ones set in secret change #75)
2. Keeping around/deleting PVCs when cluster is deleted or scaled. The reason I would include the scale scenario is that if you scale the cluster up and have stale data on PVCs, you can end up with broken replication.

For 2. I suggest the following:

1. add a cleanupGracePeriod under volumeSpec
2. controller deletes the no longer needed PVCs cleanupGracePeriod seconds after the cluster has been scaled
3. If the cluster has the mysql.presslabs.org/pvc-cleanup finalizer, the controller deletes the PVCs cleanupGracePeriod seconds after the cluster has been deleted

[smanpathak]

@calind by the finalizer you mean annotation right? Also, in case of scale event shouldn't the replication still work? e.g. if a master is at 1000 and replica A is at 900 and replica B is added, the replication for A continues from 901 and from 1 for replica B.
It makes sense to delete the claim when scaling down.

[AMecea]

Me and @calind discussed and concluded that: we have to simplify this issue by deleting PVC when scale-down only and move in a new issue the second part where a cluster is deleted.

Steps to implement PVC deletion when scale down:

• first we need to add a finalizer to PVCs, at creation time;
• create a PVC reconciliation loop. This can be implemented in two ways:
  • iterate over all PVCs created by controller and delete those that have the time elapsed from deletionTimestamp grated then cleanupGracePeriod (using a PVC informer);
  • or iterate only over a list of PVCs that is updated by cluster controller (using a map);
• when a scale-down event is "detected" (this can be done by comparing the statefulset replicas with the cluster replicas) then evicted PVCs are deleted (but because the finalizer is set, PVCs are not deleted but deletionTimestamp field is set on PVC, done automatically by k8s), those PVC are deleted eventually by reconciliation loop;
• when scale-up if there exists a PVC the deletionTimestamp is removed.

For the second part of this issue: can be done by adding a finalizer on the cluster resource and when the cluster is deleted to be scaled at 0 and when cleanupGracePeriod expire remove the finilizer and delete the cluster. (I will describe this in another issue)

[calind]

Closing this in favor of #136.