Can not overwrite MySQL configuration from Kubernetes YAML

[crauterb]

What happened:

I tried to deploy the api-server with a configuration that should allow it to communicate with a previously set up mysql-database. However, despite setting the according environment variables, the service keeps on crashing!

What did you expect to happen:

I expected the application to correctly use the set up environment variables.

What steps did you take:

I have set up a mysql deployment:

NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
mysql-argo   ClusterIP   10.247.50.60   <none>        3306/TCP   1h

with the according secret

NAME                  TYPE     DATA   AGE
mysql-argo-password   Opaque   1      1h

My deployment for the api-server looks like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  generation: 1
  labels:
    app: ml-pipeline
  name: ml-pipeline
  namespace: kubeflow
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: ml-pipeline
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: ml-pipeline
    spec:
      containers:
      - env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: MYSQL_SERVICE_HOST
          value: 172.16.0.7
        - name: MYSQL_SERVICE_PORT
          value: "3306"
        - name: DBCONFIG_USER
          value: herbert
        - name: DBCONFIG_PASSWORD
          valueFrom:
            secretKeyRef:
              key: mysql-argo-password
              name: mysql-argo-password
        image: gcr.io/ml-pipeline/api-server:0.1.24
        imagePullPolicy: IfNotPresent
        name: ml-pipeline-api-server
        ports:
        - containerPort: 8888
          protocol: TCP
        - containerPort: 8887
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      serviceAccount: ml-pipeline
      serviceAccountName: ml-pipeline

as I have read #1455 such that I can overwrite the settings of the config.json using the environment variables in the .yaml file. I also used the newest version in which this merge is applied.

Now, I am trying to deploy the according pipeline components, yet I am stuck when trying to deploy the api-server: After creating the deployment, the created pod does try to connect to the database for the specified amount of "InitConnectionTimeout": "6m" and then crashes with:

I0716 14:42:37.542471       8 client_manager.go:123] Initializing client manager
F0716 14:48:50.981437       8 error.go:296] Error 1045: Access denied for user 'herbert'@'172.16.0.203' (using password: NO)
goroutine 1 [running]:
github.com/golang/glog.stacks(0xc000399100, 0xc000498000, 0x7d, 0x9b)
	external/com_github_golang_glog/glog.go:769 +0xd4
github.com/golang/glog.(*loggingT).output(0x295a520, 0xc000000003, 0xc000222fd0, 0x283c0ab, 0x8, 0x128, 0x0)
	external/com_github_golang_glog/glog.go:720 +0x329
github.com/golang/glog.(*loggingT).printf(0x295a520, 0x3, 0x1a79b1d, 0x2, 0xc0000179b0, 0x1, 0x1)
	external/com_github_golang_glog/glog.go:655 +0x14b
github.com/golang/glog.Fatalf(0x1a79b1d, 0x2, 0xc0000179b0, 0x1, 0x1)
	external/com_github_golang_glog/glog.go:1148 +0x67
github.com/kubeflow/pipelines/backend/src/common/util.TerminateIfError(0x1c2b280, 0xc0004ce3c0)
	backend/src/common/util/error.go:296 +0x79
main.initMysql(0xc0003c9d3a, 0x5, 0x53d1ac1000, 0x0, 0x0)
	backend/src/apiserver/client_manager.go:260 +0x37d
main.initDBClient(0x53d1ac1000, 0x15)
	backend/src/apiserver/client_manager.go:190 +0x5c0
main.(*ClientManager).init(0xc000017cd8)
	backend/src/apiserver/client_manager.go:125 +0x80
main.newClientManager(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	backend/src/apiserver/client_manager.go:300 +0x7b
main.main()
	backend/src/apiserver/main.go:56 +0x5e

Now, with that error message I have several problems:

1. Apparently, the overwriting of the User did work: It no longer uses the root-User
2. The setting of the MySQL-Host did not work: I did specify an IP, yet the client apparently tries to connect to a different IP address ( which in this case is the IP address of the pod itself )
3. Apparently, the setting of the Password did not work out, since it still tells me, it is using NO PASSWORD

... which leads me to the question: What did I do wrong here ;-)

Anything else you would like to add:

I am truly hoping that I did read all the docs correctly and my limited knowledge of Go does not come around to bite me in the blubber ;)

[crauterb]

Thank you so much for the quick fix @IronPan.

Although I have no idea why the error message produced did list a different host than the one I specified - but the error is resolved using your fix!