Unable to mount volumes for pod mysql

[QxiaoQ]

/kind bug

What steps did you take and what happened:
When I start the mysql POD , met this error : Unable to mount volumes for pod “mysql-7559c6b79b-rffqd_kubeflow(95a1aee5-cfef-11e9-870d-0020091e6d26)“: timeout expired waiting for volumes to attach or mount for pod “kubeflow”/“mysql-7559c6b79b-rffqd”. list of unmounted volumes=[mysql-persistent-storage default-token-q4lzm]. list of unattached volumes=[mysql-persistent-storage default-token-q4lzm]

Anything else you would like to add:
The current PVC mysql-pv-claim is using a google PV instead of local one

Environment:

• kfctl version: Build from source code
• Kubernetes platform: ICP
• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1+icp", GitCommit:"90febef3a98de5efb2e1248525e24a57e01ad386", GitTreeState:"clean", BuildDate:"2018-09-04T08:03:44Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/ppc64le"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1+icp", GitCommit:"90febef3a98de5efb2e1248525e24a57e01ad386", GitTreeState:"clean", BuildDate:"2018-09-04T08:03:44Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/ppc64le"}

• OS :linux/ppc64le

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.96. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[QxiaoQ]

@xauthulei @jinchihe

[krishnadurai]

@QxiaoQ could you please tell us a bit more about your installation - which config have you used for installing with kfctl?

[QxiaoQ]

@QxiaoQ could you please tell us a bit more about your installation - which config have you used for installing with kfctl?
Sure.
I have build my kfctl from source code based on ppc64le.
And I install & deploy kubeflow following this doc: https://deploy-preview-1134--competent-brattain-de2d6d.netlify.com/docs/started/cloud/getting-started-icp/

The following command is optional, to make kfctl binary easier to use.

export PATH=$PATH:

export KFAPP=

Default uses IAP.

kfctl init ${KFAPP}
cd ${KFAPP}
kfctl generate all -V
kfctl apply all -V

[jinchihe]

/assign @jinchihe

@QxiaoQ Let me take a look onve get chance, thanks.

[jinchihe]

@QxiaoQ What type PV used? if NFS, could you please try to restart the NFS service?

[xauthulei]

@jinchihe ,we prefer to use the private local NFS, but it still used its predefine the PV:

Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
   PDName:     dls-kf-storage-artifact-store
   FSType:     ext4
   Partition:  0
   ReadOnly:   false

Maybe we miss something here?

[jinchihe]

Seems we should not create the PV, since if we install kubeflow on premise cluster, will failed to mount PV/PVC. Better solution is to check the platform, only create the PVs for GCP, othwise do not generate the PV file or just do not create PV for all platforms for simplying?

../kfctl generate all
...
[root@jinchi1 app3]# cat kustomize/mysql/overlays/mysqlPd/kustomization.yaml |grep -i persistent-volume.yaml
- persistent-volume.yaml
[root@jinchi1 app3]# cat persistent-volume.yaml
cat: persistent-volume.yaml: No such file or directory
[root@jinchi1 app3]# cat kustomize/mysql/overlays/mysqlPd/persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: $(mysqlPvName)
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteOnce
  gcePersistentDisk:
    pdName: $(mysqlPd)
    fsType: ext4
[root@jinchi1 app3]#

[jinchihe]

/assign @QxiaoQ
/assign @xauthulei

[xauthulei]

@jinchihe , I will check this with @QxiaoQ , Thanks.

[yanniszark]

Hi @QxiaoQ!
I faced a similar problem with the kfctl_existing_arrikto config, trying to remove all cloud specific parts.

The kfctl_existing_arrikto installation uses PVCs and only depends on having a default StorageClass.
This deployment works well for on-prem installations and doesn't depend on any cloud-specific feature, so it should work in your case as well.

[xauthulei]

@yanniszark , I fond we hard code here:
https://github.com/kubeflow/manifests/blob/master/pipeline/mysql/overlays/mysqlPd/persistent-volume.yaml#L10
Because , when execute cmd kfctl init, it fetch the latest manifests release package and cach it local. then we will base on the.cach under ${KFAPP} to generate Kustomize yaml's . Could we disable the PVC to reference specify PV.
For example, metadata-db, it will find a available local PV.

# kubectl get pv  | grep metadata-mysql

kubeflow-pv2                      20Gi       RWO            Retain           Bound       kubeflow/metadata-mysql                                                          56d

and its PVC is defined: https://github.com/kubeflow/manifests/blob/master/metadata/base/metadata-db-pvc.yaml

What is your suggestion for better deal with this ?

@jinchihe , @jlewi what is your comments , Thanks

[jinchihe]

@kkasravi I noticed the below two file is added by you in the PR kubeflow/manifests#42. That will cause some problems for Kubeflow install for on-prem cluster since the PV type is hardcode for gcePersistentDisk.

pipeline/mysql/overlays/mysqlPd/persistent-volume.yaml
pipeline/minio/overlays/minioPd/persistent-volume.yaml

We suggest to remove the two files, if the volumne dynamic is not enabled, user can create PV manually. Agree? or better solution. Thanks.

[krishnadurai]

@jinchihe I think one alternative is to create PVs manually using default storage class. For that you need to set a default storage class in the Kubernetes cluster even before the PVCs are created through deployment. Right now the 'storageclass' parameter in base directories for PVCs are left empty signifying that they will be populated by Kubernetes with the default storage class. So as soon as you create a PV in the default storage class they should be able to be assigned to the waiting PVCs.

[krishnadurai]

Right now, I believe, we are following Kubernetes best practice by leaving the storage classes empty:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#writing-portable-configuration

You could read this issue for further clarification:
#3774

[xauthulei]

@krishnadurai , The above description info are so clear. From my points, it is not easy to debug this default GCEPersistentDiskPV issue. If we delete the PV yaml file and leave the PVC there, it will be easier to local user to debug. also, the use will be quickly noted to create a storage class or local matched PV. what is your comments ?

[krishnadurai]

@xauthulei just correct me if I'm wrong here.
It looks like you might be using one of the GCP configs for deployment during kfctl init. For example, these configs contain an overlay specification to use the PV overlay you are referring to above:

kubeflow/bootstrap/config/kfctl_gcp_iap.yaml

Line 240 in 64f4f79

- mysqlPd

If you use any of the methods mentioned here for deployment, it shouldn't be creating those PVs:
https://www.kubeflow.org/docs/started/k8s/

[xauthulei]

@krishnadurai, you are right. Maybe I am wrong to deploy the kubeflow in my local cluster, please correct me :
1> build the binary kfctl from its master
2>kfctl init all -V (instead of kfctl init --config ****)
3>kfctl generate all -V
4>kfctl apply all -V

In my local ${KFAPP}, it will generate the app.yaml with the mysql sections :

  - kustomizeConfig:
      overlays:
      - mysqlPd
      parameters:
      - name: mysqlPd
        value: dls-kf-storage-metadata-store
      - name: mysqlPvName
        value: mysql-pv
      - name: mysqlPvcName
        value: mysql-pv-claim
      repoRef:
        name: manifests
        path: pipeline/mysql
    name: mysql

Now even if I the above mysql sections from the the configure, It will still create the PV.

Based on you latest comments, if we create a local.yaml delete the mysql section, it will don't create the PV, Is that right ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.