Add support scripts to drain a node and backup old resources

README.md

@@ -92,6 +92,7 @@ Please review the [security checklist](./security-checklist.md) before deploying
 - [Scheduling Guide](./docs/scheduling.md)
 - [Drive Replacement Guide](./docs/drive-replacement.md)
 - [Volume Expansion](./docs/volume-expansion.md)
+- [Drain a node](./docs/drain-node.md)
 - [Driver Specification](./docs/specification.md)
 - [Monitoring & Metrics](./docs/metrics.md)
 - [Developer Guide](./docs/development-and-testing.md)

docs/drain-node.md

@@ -0,0 +1,27 @@
+Drain a node
+-------------
+
+Draining will forcefully remove the DirectPV resources from a node. This [bash script](./tools/drain.sh) can be used for draining and has to be executed cautiously as it is an irreversible operation and may incur data loss.
+
+You can consider draining a node in the following circumstances
+
+#### When a node is detached from kubernetes
+
+If a node which was used by DirectPV is detached from kubernetes, the DirectPV resources from that node will remain intact until the resources are drained.
+
+#### When DirectPV is unselected to run on a specific node
+
+If a node which was used by DirectPV is decided to be a "non-storage" node.
+
+For example, If DirectPV is decided to not run on a specific node by changing the node-selectors like the following example
+
+```sh
+$ kubectl directpv uninstall
+$ kubectl directpv install --node-selector node-label-key=node-label-value
+```
+
+the resources from the detached node can then be cleaned up by
+
+```sh
+./drain.sh <node-name>
+```

docs/tools/drain.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+#
+# This file is part of MinIO DirectPV
+# Copyright (c) 2023 MinIO, Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# This script drains the DirectPV resources from a selected node
+# 
+# **CAUTION**
+#
+# This operation is irreversible and may incur data loss if not used cautiously.
+#
+
+set -e -C -o pipefail
+
+function drain() {
+    rm -f /tmp/directpvdrain/objects.json /tmp/directpvdrain/objects-to-patch.json
+
+    # Get the objects
+    kubectl get "${1}" --selector="directpv.min.io/node=${2}" -o json > /tmp/directpvdrain/objects.json
+
+    # Use jq if available else fallback to python3
+    if command -v jq > /dev/null
+    then
+        jq '.items[].metadata.finalizers = []' < /tmp/directpvdrain/objects.json > /tmp/directpvdrain/objects-to-patch.json
+    elif command -v python3 &> /dev/null
+    then
+       python3 <<EOF
+import json
+with open("/tmp/directpvdrain/objects.json") as f:
+     d = json.load(f)
+for item in d['items']:
+     item['metadata']['finalizers'] = []
+with open("/tmp/directpvdrain/objects-to-patch.json", "w") as f:
+     json.dump(d, f)
+EOF
+    else
+       echo "either jq or python3 is required"
+       exit 255
+    fi
+
+    # Apply the modifications
+    kubectl apply -f /tmp/directpvdrain/objects-to-patch.json || true
+
+    # delete the objects
+    kubectl delete "${1}" --selector="directpv.min.io/node=${2}" --ignore-not-found=true
+}
+
+function init() {
+
+    mkdir -p /tmp/directpvdrain
+
+    if [[ $# -ne 1 ]]; then
+        echo "usage: drain.sh <NODE>"
+        echo
+        echo "This script forcefully removes all the DirectPV resources from the node"
+        echo "This operation is irreversible and may incur data loss if not used cautiously."
+        exit 255
+    fi
+
+    if ! which kubectl >/dev/null 2>&1; then
+        echo "kubectl not found; please install"
+        exit 255
+    fi
+
+    # The `directpv-min-io` CSI Driver shouldn't be registered on the node which is requested to drain
+    rm -rf /tmp/directpvdrain/csinode.json
+    if kubectl get csinode "${1}" -o json > /tmp/directpvdrain/csinode.json; then
+        if command -v jq > /dev/null
+        then
+            if [ -n "$( jq '.spec.drivers | .[]? | select(.name == "directpv-min-io") | .name' < /tmp/directpvdrain/csinode.json )" ]; then
+                echo "the node is still under use by DirectPV CSI Driver; please remove DirectPV installation from the node to drain"
+                exit 255
+            fi
+        elif command -v python3 &> /dev/null
+        then
+             python3 <<EOF
+import json
+with open("/tmp/directpvdrain/csinode.json") as f:
+     d = json.load(f)
+if d['spec']['drivers'] is not None:
+   for driver in d['spec']['drivers']:
+       if driver['name'] == "directpv-min-io":
+          print("the node is still under use by DirectPV CSI Driver; please remove DirectPV installation from the node to drain")
+          exit(1)
+EOF
+        else
+            echo "either jq or python3 is required"
+            exit 255
+        fi
+    fi
+}
+
+function main() {
+    node="$1"
+
+    drain "directpvvolumes" "${node}"
+    drain "directpvdrives" "${node}"
+    drain "directpvinitrequests" "${node}"
+    kubectl delete directpvnode "${node}" --ignore-not-found=true
+}
+
+init "$@"
+main "$@"

docs/tools/remove-directcsi.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# This file is part of MinIO DirectPV
+# Copyright (c) 2023 MinIO, Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# This script removes direct-csi drives and volumes after taking backup YAMLs
+# to directcsidrives.yaml and directcsivolumes.yaml
+#
+
+set -e -C -o pipefail
+
+function init() {
+    if [[ $# -ne 0 ]]; then
+        echo "usage: remove-directcsi.sh"
+        echo
+        echo "This script removes direct-csi drives and volumes after taking backup YAMLs"
+        echo "to directcsidrives.yaml and directcsivolumes.yaml"
+        exit 255
+    fi
+
+    if ! which kubectl >/dev/null 2>&1; then
+        echo "kubectl not found; please install"
+        exit 255
+    fi
+}
+
+function main() {
+    kubectl get directcsivolumes -o yaml > directcsivolumes.yaml
+    kubectl get directcsidrives -o yaml > directcsidrives.yaml
+
+    # Unset the finalizers
+    # Use jq if available else fallback to python3
+    if command -v jq1 > /dev/null
+    then
+        kubectl get directcsivolumes -o json | jq '.items[].metadata.finalizers = []' | kubectl apply -f -
+        kubectl get directcsidrives -o json | jq '.items[].metadata.finalizers = []' | kubectl apply -f -
+    elif command -v python3 &> /dev/null
+    then
+       python3 <<EOF
+import yaml
+with open("directcsivolumes.yaml") as f:
+     v = yaml.safe_load(f)
+for item in v['items']:
+     item['metadata']['finalizers'] = []
+with open("/tmp/directcsivolumes-to-remove.yaml", "w") as f:
+     yaml.dump(v, f)
+with open("directcsidrives.yaml") as f:
+     d = yaml.safe_load(f)
+for item in d['items']:
+     item['metadata']['finalizers'] = []
+with open("/tmp/directcsidrives-to-remove.yaml", "w") as f:
+     yaml.dump(d, f)
+EOF
+       kubectl apply -f /tmp/directcsivolumes-to-remove.yaml || true
+       kubectl apply -f /tmp/directcsidrives-to-remove.yaml || true
+    else
+       echo "either jq or python3 is required"
+       exit 255
+    fi
+
+    # delete the resources
+    kubectl delete directcsivolumes --all
+    kubectl delete directcsidrives --all
+}
+
+init "$@"
+main "$@"

docs/tools/replace.sh

@@ -16,6 +16,10 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+#
+# This script replaces source drive to destination drive in the specified node
+#
+
 set -e
 
 # usage: get_drive_id <node> <drive-name>
@@ -47,6 +51,8 @@ function get_pod_namespace() {
 function init() {
     if [[ $# -eq 4 ]]; then
         echo "usage: replace.sh <NODE> <SRC-DRIVE> <DEST-DRIVE>"
+        echo
+        echo "This script replaces source drive to destination drive in the specified node"
         exit 255
     fi
 

docs/upgrade.md

@@ -34,6 +34,8 @@ quay.io/minio/csi-resizer:v1.7.0
 
 - If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18`
 
-If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes.
+- If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes.
 
 You need to change `direct.csi.min.io/access-tier: <your_access_tier_value>` to `directpv.min.io/access-tier: <your_access_tier_value>` in the respective storage class parameters section.
+
+- The older CRDs (directcsidrives and directcsivolumes) are deprecated and not used in versions > v4.0.0, it can be removed after upgrading. Please use the [bash script](./tools/remove-directcsi.sh) to remove the older objects after upgrading to latest.