[Test-Only] build: test minikube v1.22.0 kernel v4.19.182 for rbd-nbd

[pkalever]

Describe what this PR does

Assuming kubernetes/minikube#12268 introduced a regression in NBD kernel driver.

updates: #2610
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

---

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

• /retest ci/centos/<job-name>: retest the <job-name> after unrelated
  failure (please report the failure too!)
• /retest all: run this in case the CentOS CI failed to start/report any test
  progress or results

[Rakshith-R]

@pkalever This would require changes in centos ci branch too, particularly revert of this pr #2490
and simultaneous backport changes to all supported release branches.

[nixpanic]

Assuming kubernetes/minikube#12268 introduced
bugs in NBD kernel driver.

What makes you assume this? Is it about the timing since when the failures started, or something else?

[humblec]

Assuming kubernetes/minikube#12268 introduced bugs in NBD kernel driver.

I am sorry, for some reason I dont get whats the exact issue we are facing with NBD driver. Could you please explain some more on the PR ?

[pkalever]

@nixpanic @humblec very recently we started seeing issues with rbd-nbd tests. We are hitting frequent failures with rbd-nbd encryption test, not just that, we have also seen mkfs failures on rbd-nbd "create a PVC and bind it to an app using rbd-nbd mounter" which was not seen before.

I'm investigating in various directions with not many clues. One perspective was from "what changed recently?" the answer is minikube version.

Please note this is just a hypothesis and nothing concrete ATM.

[pkalever]

@nixpanic @humblec @Madhu-1 @Rakshith-R

I have hit the same locally with minikube v1.22.0, kernel version 4.19.182

STEP: create a PVC and bind it to an app using rbd-nbd mounter with encryption
Nov 10 16:01:49.851: INFO: waiting for kubectl (delete -f args []) to finish
Nov 10 16:01:49.851: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.39.103:8443 --kubeconfig=/home/pkalever/.kube/config --namespace=cephcsi-e2e-9e58a444 delete -f -'
Nov 10 16:01:49.976: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n"
Nov 10 16:01:49.976: INFO: stdout: "storageclass.storage.k8s.io \"csi-rbd-sc\" deleted\n"
Nov 10 16:01:49.984: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-7467d8bf8-wdwdp ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Nov 10 16:01:49.984: INFO: >>> kubeConfig: /home/pkalever/.kube/config
Nov 10 16:01:51.377: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-694    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state
Nov 10 16:01:51.378: INFO: waiting for PVC rbd-pvc (0 seconds elapsed)
Nov 10 16:01:53.390: INFO: waiting for PVC rbd-pvc (2 seconds elapsed)
Nov 10 16:01:53.403: INFO: Waiting for PV pvc-c83c16d7-bf8f-46c2-82f7-f7509074c463 to bind to PVC rbd-pvc
Nov 10 16:01:53.403: INFO: Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
Nov 10 16:01:53.407: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (3.388272ms)
Nov 10 16:01:53.407: INFO: Waiting up to 10m0s for PersistentVolume pvc-c83c16d7-bf8f-46c2-82f7-f7509074c463 to have phase Bound
Nov 10 16:01:53.411: INFO: PersistentVolume pvc-c83c16d7-bf8f-46c2-82f7-f7509074c463 found and phase=Bound (4.44586ms)
Nov 10 16:01:53.427: INFO: Waiting up to csi-rbd-demo-pod to be in Running state
[...]

[pkalever@dhcp53-159 ceph-csi]$ kubectl get pods -n rbd-694
NAME               READY   STATUS              RESTARTS   AGE                                        
csi-rbd-demo-pod   0/1     ContainerCreating   0          48s                                        
[pkalever@dhcp53-159 ceph-csi]$ kubectl describe pod/csi-rbd-demo-pod -n rbd-694                     
Name:         csi-rbd-demo-pod                                                                       
Namespace:    rbd-694                                                                                
Priority:     0                                                                                      
Node:         minikube/192.168.39.103
Start Time:   Wed, 10 Nov 2021 16:01:53 +0530              
Labels:       <none>                                       
Annotations:  kubernetes.io/psp: e2e-test-privileged-psp   
Status:       Pending                                      
IP:                                                        
IPs:          <none>                                       
Containers:                                                
  web-server:             
    Container ID:         
    Image:          docker.io/library/nginx:latest         
    Image ID:                                              
    Port:           <none>                                                                           
    Host Port:      <none>
    State:          Waiting                                
      Reason:       ContainerCreating
    Ready:          False                                  
    Restart Count:  0                                                                                
    Environment:    <none>                                 
    Mounts:               
      /var/lib/www/html from mypvc (rw)                    
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n5t4p (ro)                  
Conditions:                                                
  Type              Status                                 
  Initialized       True                                   
  Ready             False                                  
  ContainersReady   False                                  
  PodScheduled      True                                   
Volumes:                                                   
  mypvc:                                                   
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) 
    ClaimName:  rbd-pvc                                    
    ReadOnly:   false                                      
  kube-api-access-n5t4p:                                   
    Type:                    Projected (a volume that contains injected data from multiple sources)  
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s               
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s             
Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Normal   Scheduled               60s               default-scheduler        Successfully assigned rbd-694/csi-rbd-demo-pod to minikube
  Normal   SuccessfulAttachVolume  60s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-c83c16d7-bf8f-46c2-82f7-f7509074c463"
  Warning  FailedMount             0s (x7 over 53s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-c83c16d7-bf8f-46c2-82f7-f7509074c463" : rpc error: code = Internal desc = failed to encrypt rbd image replicapool
/csi-vol-6abe4d22-4211-11ec-babb-be1b076a52a3: failed to encrypt volume replicapool/csi-vol-6abe4d22-4211-11ec-babb-be1b076a52a3: an error (exit status 1) occurred while running cryptsetup args: [-q luksFormat --type luks2 --hash sha256 /
dev/nbd0 -d /dev/stdin]
[pkalever@dhcp53-159 ceph-csi]$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ uname -a 
Linux minikube 4.19.182 #1 SMP Fri Jul 2 00:45:17 UTC 2021 x86_64 GNU/Linux
$ exit
logout
ssh: Process exited with status 127

[pkalever@dhcp53-159 ceph-csi]$ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e
[pkalever@dhcp53-159 ceph-csi]$ 

Closing this PR now.