Update documentation for Volume Group Snapshot

book/src/external-snapshotter.md

@@ -34,13 +34,17 @@ To use the snapshot beta and GA feature, a snapshot controller is also required.
 
 ### Description
 
-Starting with the Beta version, the snapshot controller will be watching the Kubernetes API server for `VolumeSnapshot` and `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar only watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar is also responsible for calling the CSI RPCs CreateSnapshot, DeleteSnapshot, and ListSnapshots.
+The CSI `external-snapshotter` sidecar watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar is also responsible for calling the CSI RPCs `CreateSnapshot`, `DeleteSnapshot`, and `ListSnapshots`.
 
-#### VolumeSnapshotClass Parameters
+Volume Group Snapshot support can be enabled with the `--enable-volume-group-snapshots` option. When enabled, the CSI `external-snapshotter` sidecar watches the API server for `VolumeGroupSnapshotContent` CRD object, and will be responsible for calling the CSI RPCs `CreateVolumeGroupSnapshot`, `DeleteVolumeGroupSnapshot` and `GetVolumeGroupSnapshot`.
+
+#### VolumeSnapshotClass and VolumeGroupSnapshotClass Parameters
 
 When provisioning a new volume snapshot, the CSI `external-snapshotter` sets the `map<string, string> parameters` field in the CSI `CreateSnapshotRequest` call to the key/values specified in the `VolumeSnapshotClass` it is handling.
 
-The CSI `external-snapshotter` also reserves the parameter keys prefixed with `csi.storage.k8s.io/`. Any `VolumeSnapshotClass` keys prefixed with `csi.storage.k8s.io/` are not passed to the CSI driver as an opaque `parameter`.
+When volume group snapshot support is enabled, the `map<string, string> parameters` field is set in the CSI `CreateVolumeGroupSnapshotRequest` call to the key/values specified in the `VolumeGroupSnapshotClass` it is handling.
+
+The CSI `external-snapshotter` also reserves the parameter keys prefixed with `csi.storage.k8s.io/`. Any `VolumeSnapshotClass` or `VolumeGroupSnapshotClass` keys prefixed with `csi.storage.k8s.io/` are not passed to the CSI driver as an opaque `parameter`.
 
 The following reserved `VolumeSnapshotClass` parameter keys trigger behavior in the CSI `external-snapshotter`:
 
@@ -51,9 +55,9 @@ The following reserved `VolumeSnapshotClass` parameter keys trigger behavior in
 
 For more information on how secrets are handled see [Secrets & Credentials](secrets-and-credentials.md).
 
-#### VolumeSnapshot and VolumeSnapshotContent Parameters
+#### VolumeSnapshot, VolumeSnapshotContent, VolumeGroupSnapshot and VolumeGroupSnapshotContent Parameters
 
-The CSI `external-snapshotter` (v4.0.0+) introduces the `--extra-create-metadata` flag, which automatically sets the following `map<string, string> parameters` in the CSI `CreateSnapshotRequest`:
+The CSI `external-snapshotter` (v4.0.0+) introduces the `--extra-create-metadata` flag, which automatically sets the following `map<string, string> parameters` in the CSI `CreateSnapshotRequest` and `CreateVolumeGroupSnapshotRequest`:
 
 * `csi.storage.k8s.io/volumesnapshot/name`
 * `csi.storage.k8s.io/volumesnapshot/namespace`
@@ -69,6 +73,8 @@ For detailed information about volume snapshot and restore functionality, see [V
 
 CSI drivers that support provisioning volume snapshots and the ability to provision new volumes using those snapshots should use this sidecar container, and advertise the CSI `CREATE_DELETE_SNAPSHOT` controller capability.
 
+CSI drivers that support provisioning volume group snapshots should use this side container too, and advertise the CSI `CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT` controller capability.
+
 For detailed information (binary parameters, RBAC rules, etc.), see [https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md](https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md).
 
 ### Deployment

book/src/group-snapshot.md

@@ -14,12 +14,16 @@ are taken at the same point-in-time. A group snapshot can be used either to rehy
 new volumes (pre-populated with the snapshot data) or to restore existing volumes to
 a previous state (represented by the snapshots).
 
-Kubernetes CSI currently enables CSI Drivers to expose the following functionality via the Kubernetes API:
+Kubernetes CSI currently enables CSI Drivers to expose the following
+functionality via the Kubernetes API:
 
-1. Creation and deletion of volume group snapshots via [Kubernetes native API](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/apis/volumegroupsnapshot/v1alpha1). 
-2. Creation of new volumes pre-populated with the data from a snapshot that is part of the volume group snapshot via Kubernetes [dynamic volume provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/).
+1. Creation and deletion of volume group snapshots via [Kubernetes native
+   API](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/apis/volumegroupsnapshot/v1alpha1).
+2. Creation of new volumes pre-populated with the data from a snapshot that is
+   part of the volume group snapshot via Kubernetes [dynamic volume
+   provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/).
 
-## Implementing Volume Group Snapshot Functionality in Your CSI Driver
+## Implementing Volume Group Snapshot Functionality in your CSI Driver
 
 To implement the volume group snapshot feature, a CSI driver MUST:
 
@@ -31,13 +35,21 @@ For details,  see the [CSI spec](https://github.com/container-storage-interface/
 
 ## Sidecar Deployment
 
-The Kubernetes CSI development team maintains the [external-snapshotter](external-snapshotter.md) Kubernetes CSI [Sidecar Containers](sidecar-containers.md). This sidecar container implements the logic for watching the Kubernetes API objects and issuing the appropriate CSI volume group snapshot calls against a CSI endpoint. For more details, see [external-snapshotter documentation](external-snapshotter.md).
+The Kubernetes CSI development team maintains the
+[external-snapshotter](external-snapshotter.md) Kubernetes CSI [Sidecar
+Containers](sidecar-containers.md). This sidecar container implements the logic
+for watching the Kubernetes API objects and issuing the appropriate CSI volume
+group snapshot calls against a CSI endpoint. For more details, see
+[external-snapshotter documentation](external-snapshotter.md).
 
 ## Volume Group Snapshot APIs
 
-With the introduction of Volume Group Snapshot, the user can create and delete a group snapshot using Kubernetes APIs. 
+With the introduction of Volume Group Snapshot, the user can create and delete a
+group snapshot using Kubernetes APIs.
 
-The schema definition for the custom resources (CRs) can be found [here](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd). The CRDs should be installed by the Kubernetes distributions.
+The schema definition for the custom resources (CRs) can be found
+[here](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd).
+The CRDs should be installed by the Kubernetes distributions.
 
 There are 3 APIs:
 
@@ -63,21 +75,34 @@ created. including the driver information, the deletion policy, etc.
 
 ## Controller
 
-* The controller logic for volume group snapshot is added to the snapshot controller and the CSI external-snapshotter sidecar.
+* The controller logic for volume group snapshot is added to the snapshot
+  controller and the CSI external-snapshotter sidecar.
 
-The snapshot controller is deployed by the Kubernetes distributions and is responsible for watching the VolumeGroupSnapshot CRD objects and manges the creation and deletion lifecycle of volume group snapshots.
+The snapshot controller is deployed by the Kubernetes distributions and is
+responsible for watching the VolumeGroupSnapshot CRD objects and manges the
+creation and deletion lifecycle of volume group snapshots.
 
-The CSI external-snapshotter sidecar watches Kubernetes VolumeGroupSnapshotContent CRD objects and triggers CreateVolumeGroupSnapshot/DeleteVolumeGroupSnapshot against a CSI endpoint.
+The CSI external-snapshotter sidecar watches Kubernetes
+VolumeGroupSnapshotContent CRD objects and triggers
+CreateVolumeGroupSnapshot/DeleteVolumeGroupSnapshot against a CSI endpoint.
 
 ## Snapshot Validation Webhook
 
-The validating webhook server is updated to validate volume group snapshot objects. This SHOULD be installed by the Kubernetes distros along with the snapshot-controller, not end users. It SHOULD be installed in all Kubernetes clusters that has the volume group snapshot feature enabled. See [Snapshot Validation Webhook](snapshot-validation-webhook.md) for more details on how to use the webhook.
+The validating webhook server is updated to validate volume group snapshot
+objects. This SHOULD be installed by the Kubernetes distros along with the
+snapshot-controller, not end users. It SHOULD be installed in all Kubernetes
+clusters that has the volume group snapshot feature enabled. See [Snapshot
+Validation Webhook](snapshot-validation-webhook.md) for more details on how to
+use the webhook.
 
 ## Kubernetes Cluster Setup
 
-See the Deployment section of [Snapshot Controller](snapshot-controller.md) on how to set up the snapshot controller and CRDs.
+See the Deployment section of [Snapshot Controller](snapshot-controller.md) on
+how to set up the snapshot controller and CRDs.
 
-See the Deployment section of [Snapshot Validation Webhook](snapshot-validation-webhook.md) for more details on how to use the webhook.
+See the Deployment section of [Snapshot Validation
+Webhook](snapshot-validation-webhook.md) for more details on how to use the
+webhook.
 
 ## Test Volume Group Snapshot Feature
 
@@ -88,30 +113,51 @@ Create a _StorageClass_:
 kubectl create -f storageclass.yaml
 ```
 
-Create _PVC_:
+Create _PVCs_:
 ```
+# This will create a PVC named hpvc
 kubectl create -f pvc.yaml
-kubectl create -f pvc2.yaml
+
+# This will create a PVC named hpvc-2
+sed "s/hpvc/hpvc-2/" pvc.yaml | kubectl create -f -
 ```
 
 Add a label to _PVC_:
 ```
-kubectl label pvcs <your-pvc-name> group:my-apps
+kubectl label pvc hpvc hpvc-2 app.kubernetes.io/name=postgresql
 ```
 
 Create a _VolumeGroupSnapshotClass_:
 ```
-kubectl create -f group-snapshot-class.yaml
+kubectl create -f groupsnapshotclass-v1alpha1.yaml
 ```
 
 Create a _VolumeGroupSnapshot_:
 ```
-kubectl create -f group-snapshot.yaml
+kubectl create -f groupsnapshot-v1alpha1.yaml
+```
+
+Once the _VolumeGroupSnapshot_ is ready, the `pvcVolumeSnapshotRefList` status field will contain the names of the generated _VolumeSnapshot_ objects:
+```
+kubectl get volumegroupsnapshot new-groupsnapshot-demo  -o yaml | sed -n '/pvcVolumeSnapshotRefList/,$p'
+
+  pvcVolumeSnapshotRefList:
+  - persistentVolumeClaimRef:
+      name: hpvc
+    volumeSnapshotRef:
+      name: snapshot-4bcc4a322a473abf32babe3df5779d14349542b1f0eb6f9dab0466a85c59cd42-2024-06-19-12.35.17
+  - persistentVolumeClaimRef:
+      name: hpvc-2
+    volumeSnapshotRef:
+      name: snapshot-62bd0be591e1e10c22d51748cd4a53c0ae8bf52fabb482bee7bc51f8ff9d9589-2024-06-19-12.35.17
+  readyToUse: true
 ```
 
 Create a _PVC_ from a _VolumeSnapshot_ that is part of the group snapshot:
 ```
-kubectl create -f restore-from-snapshot-in-group.yaml
+# In the command below, the volume snapshot name should be chosen from
+# the ones listed in the output of the previous command
+sed 's/new-snapshot-demo-v1/snapshot-4bcc4a322a473abf32babe3df5779d14349542b1f0eb6f9dab0466a85c59cd42-2024-06-19-12.35.17/' restore.yaml | kubectl create -f -
 ```
 
 ## Examples

book/src/sidecar-containers.md

@@ -1,8 +1,11 @@
 # Kubernetes CSI Sidecar Containers
 
-Kubernetes CSI Sidecar Containers are a set of standard containers that aim to simplify the development and deployment of CSI Drivers on Kubernetes.
+Kubernetes CSI Sidecar Containers are a set of standard containers that aim to
+simplify the development and deployment of CSI Drivers on Kubernetes.
 
-These containers contain common logic to watch the Kubernetes API, trigger appropriate operations against the “CSI volume driver” container, and update the Kubernetes API as appropriate.
+These containers contain common logic to watch the Kubernetes API, trigger
+appropriate operations against the “CSI volume driver” container, and update the
+Kubernetes API as appropriate.
 
 The containers are intended to be bundled with third-party CSI driver containers and deployed together as pods.
 

book/src/snapshot-controller.md

@@ -34,10 +34,14 @@ For more information on the CSI external-snapshotter sidecar, see [this external
 
 The snapshot controller will be watching the Kubernetes API server for `VolumeSnapshot` and `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar only watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects. The snapshot controller will be creating the `VolumeSnapshotContent` CRD object which triggers the CSI `external-snapshotter` sidecar to create a snapshot on the storage system.
 
+The snapshot controller will be watching for `VolumeGroupSnapshot` and `VolumeGroupSnapshotContent` CRD objects when Volume Group Snapshot support is enabled via the `--enable-volume-group-snapshots` option.
+
 For detailed snapshot beta design changes, see the design doc [here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/README.md).
 
 For detailed information about volume snapshot and restore functionality, see [Volume Snapshot & Restore](snapshot-restore-feature.md).
 
+For detailed information about volume group snapshot and restore functionality, see [Volume Snapshot & Restore](group-snapshot.md).
+
 For detailed information (binary parameters, RBAC rules, etc.), see [https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md](https://github.com/kubernetes-csi/external-snapshotter/blob/release-6.2/README.md).
 
 ## Deployment

book/src/snapshot-restore-feature.md

@@ -83,12 +83,12 @@ kubectl create -f pvc.yaml
 
 Create a _VolumeSnapshotClass_:
 ```
-kubectl create -f snapshotclass.yaml
+kubectl create -f snapshotclass-v1.yaml
 ```
 
 Create a _VolumeSnapshot_:
 ```
-kubectl create -f snapshot.yaml
+kubectl create -f snapshot-v1.yaml
 ```
 
 Create a _PVC_ from a _VolumeSnapshot_: