Failed to find parent dir for mountpoint

[joselbr2099]

Describe the bug
After deploy fio service (from tutorial) I get this issue:

    Warning  FailedMount             5s (x5 over 13s)  kubelet                  MountVolume.SetUp failed for volume "pvc-8e567cc1-256d-4475-a102-d382cb9038b0" : rpc error: code = Internal desc = Failed to find parent dir for mountpoint /var/snap/microk8s/common/var/lib/kubelet/pods/159a9f0b-f9a7-4e2c-a69b-9e02f15f8163/volumes/kubernetes.io~csi/pvc-8e567cc1-256d-4475-a102-d382cb9038b0/mount, volume 8e567cc1-256d-4475-a102-d382cb9038b0

checking in /var/snap/microk8s/common/var/lib/kubelet/pods/:

[root@node]# ls /var/snap/microk8s/common/var/lib/kubelet/pods/159a9f0b-f9a7-4e2c-a69b-9e02f15f8163
ls: cannot access '/var/snap/microk8s/common/var/lib/kubelet/pods/159a9f0b-f9a7-4e2c-a69b-9e02f15f8163': The file or directory does not exist
[root@node]#  ls /var/snap/microk8s/common/var/lib/kubelet/pods/
3a20a9aa-ea5a-4989-a537-2e0e3da5eae8  4671d44f-112b-448d-8e06-eda70cc056b2  5656dd1a-f558-43f5-864a-dfa7b89d78b9  56ff7f4f-fcc9-45f6-8625-68c7cdedfb66  5a75dcce-532c-44f9-a8eb-ff970edc46af  7b03624c-a249-4576-a67d-0cfc8e1abdb5  7b7de57d-0202-4410-9362-699a847eb7fc  b0265a99-375c-4e74-b707-c67cfca6cd36  dd4add58-b3e0-42ea-834f-c787fa2acd43  e9e55eca-2f5c-403a-b55f-8ebad0c37b22
[root@node]# 

To Reproduce
launch storeClass.yaml

 cat storeClass.yaml 
apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata:
  name: mayastor-test-1 
parameters: 
  ioTimeout: "30" 
  protocol: nvmf
  repl: "1"
provisioner: io.openebs.csi-mayastor

launch pvc.yaml

kind: PersistentVolumeClaim
metadata:
  name: ms-volume-claim
spec: 
  accessModes:
   - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: mayastor-test-1

launch fio.yaml

$ cat fio.yaml 
kind: Pod
apiVersion: v1
metadata:
  name: fio
spec:
  # Run on a storage node so that the nexus is local to the app
  nodeSelector:
    openebs.io/engine: mayastor
  volumes:
    - name: ms-volume
      persistentVolumeClaim:
       claimName: ms-volume-claim
  containers:
    - name: fio
      image: mayadata/fio
      args:
        - sleep
        - "1000000"
      volumeMounts:
        - mountPath: "/volume"
          name: ms-volume
      #volumeDevices:
      #  - devicePath: /dev/xvda
      #    name: ms-volume

finally

$ kubectl describe po fio
Name:             fio
Namespace:        test
Priority:         0
Service Account:  default
Node:             --obscure--
Start Time:       Wed, 18 Oct 2023 12:39:22 -0400
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Containers:
  fio:
    Container ID:  
    Image:         mayadata/fio
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      sleep
      1000000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zrxzq (ro)
      /volume from ms-volume (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ms-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ms-volume-claim
    ReadOnly:   false
  kube-api-access-zrxzq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              openebs.io/engine=mayastor
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Normal   Scheduled               14s               default-scheduler        Successfully assigned qapruebas/fio to usecc-srv3.apps.gob.bo
  Normal   SuccessfulAttachVolume  14s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-8e567cc1-256d-4475-a102-d382cb9038b0"
  Warning  FailedMount             5s (x5 over 13s)  kubelet                  MountVolume.SetUp failed for volume "pvc-8e567cc1-256d-4475-a102-d382cb9038b0" : rpc error: code = Internal desc = Failed to find parent dir for mountpoint /var/snap/microk8s/common/var/lib/kubelet/pods/159a9f0b-f9a7-4e2c-a69b-9e02f15f8163/volumes/kubernetes.io~csi/pvc-8e567cc1-256d-4475-a102-d382cb9038b0/mount, volume 8e567cc1-256d-4475-a102-d382cb9038b0

diskpool.yaml

$ cat test1.yaml 
apiVersion: "openebs.io/v1beta1" 
kind: DiskPool
metadata: 
  name: poolnode1 
  namespace: mayastor
spec: 
  node: node2
  disks: ["/dev/loop7"]

Expected behavior
finish tutorial and getting a cluser working with mayastor

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: Red Hat Enterprise Linux release 8.8 (Ootpa)
• Kernel version: Linux srv1o 4.18.0-477.27.1.el8_8.x86_64 Reserve space for metadata #1 SMP Thu Aug 31 10:29:22 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
• MayaStor revision: from HELM

[tiagolobocastro]

@Abhinandan-Purkait @datacore-tilangovan any idea on what this could be?

[Abhinandan-Purkait]

@joselbr2099 Can we see the csi-node pod logs, the pod on the node where application was scheduled.

Also, did you specify the kubeletDir to point to what microk8s uses while installing mayastor ?
ref:- https://github.com/openebs/mayastor-extensions/blob/1eb37015463121ace8d57a60b66a2c2efea8ca77/chart/values.yaml#L324

[joselbr2099]

I solved this issue reinstalling microk8s cluster