How to configure riff to run kservices with images from a registry signed by a custom CA

[jldec]

According to /knative/serving#2136, it should be possible to configure Knative serving to trust custom CA certificates to work with an on-prem registry like Harbor.

It would be helpful if riff made it simpler for users to perform this additional configuration. This could either take to form of documentation, or extensions to the riff cli.

The PKS-Harbor installation guidelines document how to automate the installation of a certificate from Harbor into the PKS kubernetes environment.

Initial testing appears to confirm that the PKS Harbor cert is sufficient for pulling images to install and run "normal" kubernetes deployments, but does not meet the requirements to run Knative services.

knative/serving#1996 has more details regarding why Knative services are different in how they pull images.

[trisberg]

The cert added according to PKS-Harbor installation guidelines document works for allowing k8s to pull images. It does not seem to be included in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt so it is not available to knative. Not sure how it could be made available.

[trisberg]

These are the steps that will show the cert error "x509: certificate signed by unknown authority"

$ riff service create square --image harbor.haas-75.pez.pivotal.io/dev/square:test

riff service create completed successfully

The --image flag references a repository that is hosted in a registry with a self-signed certificate. This certificate has been added to the PKS configuration so Kubernetes nodes can pull images but the Knative serving controller can't access the registry/

$ riff service list
NAME   STATUS                                                                    
square RevisionMissing: Configuration "square" does not have any ready Revision. 

riff service list completed successfully

$ kubectl describe kservice square
Name:         square
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  serving.knative.dev/v1alpha1
Kind:         Service
Metadata:
  Creation Timestamp:  2019-02-15T14:48:16Z
  Generation:          1
  Resource Version:    2669466
  Self Link:           /apis/serving.knative.dev/v1alpha1/namespaces/default/services/square
  UID:                 ba318227-3130-11e9-aa36-005056893a3c
Spec:
  Generation:  1
  Run Latest:
    Configuration:
      Revision Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec:
          Container:
            Image:  harbor.haas-75.pez.pivotal.io/dev/square:test
            Name:   
            Resources:
Status:
  Address:
    Hostname:  square.default.svc.cluster.local
  Conditions:
    Last Transition Time:        2019-02-15T14:48:16Z
    Message:                     Revision "square-00001" failed with message: "Get https://harbor.haas-75.pez.pivotal.io/v2/: x509: certificate signed by unknown authority".
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady
    Last Transition Time:        2019-02-15T14:48:17Z
    Message:                     Configuration "square" does not have any ready Revision.
    Reason:                      RevisionMissing
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2019-02-15T14:48:17Z
    Message:                     Configuration "square" does not have any ready Revision.
    Reason:                      RevisionMissing
    Status:                      False
    Type:                        RoutesReady
  Domain:                        square.default.example.com
  Domain Internal:               square.default.svc.cluster.local
  Latest Created Revision Name:  square-00001
  Observed Generation:           1
Events:                          <none>

[trisberg]

Summary of where we are at the moment.

The CA certificate needs to be available for:

1. Kubernetes, to pull images from the registry. This seems to be managed by the admin installing Kubernetes - https://kubernetes.io/docs/setup/certificates/, for minikube you can copy the ca.crt file to your local filesystem under a path like: ~/.minikube/files/etc/docker/certs.d/registry.springdeveloper.com/ca.crt - anything under ~/.minikube/files will be copied to the node.

2. Knative serving controller, to get the digest for the image. It is possible to modify the serving configuration to mount a volume with the cert and then set an environment variable so the controller can pick up the cert - SSL_CERT_FILE='/var/run/secrets/kubernetes.io/registry/ca.crt'

3. The buildpack, to be able to access the registry for the analyze step. It is unclear how we could do that at the moment - https://github.com/buildpack/roadmap/issues/57

[jonjohnsonjr]

@trisberg

Re: 1, for knative digest resolution, we have an open issue to follow the same pattern as docker/minkube, linking here as a breadcrumb: google/go-containerregistry#211

Re: 2, we should already be picking up /var/run/secrets/kubernetes.io/registry/ca.crt without having to modify the environment. Does that not work? Any other certs would have to be added that way, though.

Re: 3, if buildpacks are using google/go-containerregistry, we can probably address that in the same way. I think we'd want to follow the pattern that docker/containerd do for providing custom certs.

[trisberg]

@jonjohnsonjr

Thanks for the feedback. Re: 2, it does seem to work but I'm not sure how to add my self-signed cert to the ca.crt file that is provided as /var/run/secrets/kubernetes.io/registry/ca.crt, that's why I added it separately and used SSL_CERT_FILE env var.

For buildpacks we'll just have to wait for this issue to be addressed in the buildpack project before moving further along here.

[jonjohnsonjr]

Oh 🤦‍♂️ I didn't read correctly registry vs serviceaccount, sorry for the confusion.

[almartino]

Hi, this is how i managed to inject a custom ca in deployment/controller of knative-serving namespace:

create a generic secret under knative-serving ns:

kubectl --namespace knative-serving create secret generic customca --from-file=customca.crt=CA_FILE_PATH

configure knative-serving:

kubectl edit deployment controller --namespace knative-serving

add customca secret and mount point:

env section:

- env:
  - name: CONFIG_LOGGING_NAME
    value: config-logging
  - name: SSL_CERT_DIR
    value: /etc/customca

volumeMounts section:

volumeMounts:
  - mountPath: /etc/customca
    name: customca

volumes section:

volumes:
  - name: customca
    secret:
      defaultMode: 420
      secretName: customca

with those configurations knative-serving/controller can successfully pull an image from a private docker repository.

p.s. I noticed that installing knative following the guide a controller service account is created and the controller pod mounts /var/run/secrets/kubernetes.io/serviceaccount/ca.crt from the controller's serviceaccount secret. This is why I configured SSL_CERT_DIR env