Unable to get "CSI Snapshot Data Movement" to MinIO S3 running

[Zalfsten]

Hello everybody,

I'm trying to backup CSI volumes to S3 (MinIO), using the "CSI Snapshot Data Movement" approach -- but yet without success. I'm encountering three different error scenarios -- partly inconsistently, partly because of different configurations. Now I only describe the issue I'm currently facing.

When creating a backup from a demo nginx deployment in the default namespace, consisting of a pod and a pvc, I receive the warning VolumeSnapshotter plugin doesn't support data movement, no backups
of PVCs are created. Find the details below.

How can this be fixed?

Thanks and Regards,
Carsten

Steps to reproduce

$ velero backup create test2 --include-namespaces=default --snapshot-volumes --snapshot-move-data
Backup request "test2" submitted successfully.
Run `velero backup describe test2` or `velero backup logs test2` for more details.
$ velero backup describe test2
Name:         test2
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.6
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  Completed


Warnings:
  Velero:     <none>
  Cluster:   resource: /persistentvolumes name: /pvc-76ffe0d8-c7d0-4e6e-aea3-8adeece42916 message: /VolumeSnapshotter plugin doesn't support data movement.
  Namespaces: <none>

Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-08-13 12:19:27 +0000 UTC
Completed:  2024-08-13 12:19:30 +0000 UTC

Expiration:  2024-09-12 12:19:27 +0000 UTC

Total items to be backed up:  12
Items backed up:              12

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

The VolumeSnapshotClass

kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: truenas-snapshot
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: csi.hpe.com
parameters:
  description: "Snapshot created on TrueNAS"
  csi.storage.k8s.io/snapshotter-secret-name: truenas-secret
  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system
deletionPolicy: Delete

My velero config

I'm using the helm chart version 7.1.2 to setup velero. This is my value file:

velero:
  image:
    repository: velero/velero
    tag: v1.13.2
    pullPolicy: IfNotPresent

  initContainers:
    - name: velero-plugin-for-csi
      image: velero/velero-plugin-for-csi:v0.7.1
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /target
          name: plugins
    - name: velero-plugin-for-aws
      image: velero/velero-plugin-for-aws:v1.10.0
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /target
          name: plugins

  cleanUpCRDs: false
  deployNodeAgent: true

  configuration:
    features:
      - EnableCSI
    defaultSnapshotMoveData: true
    uploaderType: kopia
    backupStorageLocation:
      - provider: aws
        bucket: velero
        config:
          region: minio
          s3ForcePathStyle: "true"
          s3Url: <my URL>
  credentials:
    useSecret: true
    existingSecret: velero-secret

My setup

• kubectl verion:

  Client Version: v1.29.1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.6

  on-premise, setup using kubespray 2.24.0

• cat /etc/os-release:

  NAME="AlmaLinux"
  VERSION="9.4 (Seafoam Ocelot)"
  ID="almalinux"
  ID_LIKE="rhel centos fedora"
  VERSION_ID="9.4"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="AlmaLinux 9.4 (Seafoam Ocelot)"
  ANSI_COLOR="0;34"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
  HOME_URL="https://almalinux.org/"
  DOCUMENTATION_URL="https://wiki.almalinux.org/"
  BUG_REPORT_URL="https://bugs.almalinux.org/"
  
  ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
  ALMALINUX_MANTISBT_PROJECT_VERSION="9.4"
  REDHAT_SUPPORT_PRODUCT="AlmaLinux"
  REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
  SUPPORT_END=2032-06-01

• velero version:

  Client:
          Version: v1.13.2
          Git commit: 4d961fb6fec384ed7f3c1b7c65c818106107f5a6
  Server:
          Version: v1.13.2

  (I've used v1.14.0 before, but also without success)

• velero client config get features:

  features: <NOT SET>

the debug file

bundle-2024-08-13-12-46-36.tar.gz

[Zalfsten]

Ok, that one was easy:
replacing

  configuration:
    features:
      - EnableCSI

by

  configuration:
    features: EnableCSI

fixed this warning. Now I am in one of the other mentioned error scenarios. I will open a new issue for this.