Using local docker registry inside minikube cluster with 'none' driver

[mikwieczorek]

Hi,

I am learning kubernetess using minikube and I encountered a problem with using local docker registry inside minikube.
I have a docker registry running on my laptop with 5000:5000 port-forwarding.
Minikube is run with none driver and with --insecure-registry flag.

However, when I enter into a pod and run:
curl localhost:5000/v2/_catalog <- does not work
curl 127.0.0.1:5000/v2/_catalog <- does not work
curl <laptop-ip>:5000/v2/_catalog <- works

registry add-on seems not to work with none driver.
Deployment of docker registry inside cluster and port-forwarding of 5000:5000 does not work as port 5000 is already taken by my host-machine docker registry.

Is there any way around to be able to use localhost:5000 from the inside of a cluster or is the only way in these settings?

I tried different approaches, but I am new to k8s and I could not find any working solution in docs.

[afbjorklund]

If you want to use your external registry inside the cluster (without using the real IP), you need to modify the proxy from the "registry" add-on. By default it will proxy to registry.kube-system.svc.cluster.local but you want it to use the host registry.

We don't recommend running the "none" driver on your laptop, since there is no isolation between "your files" and the cluster. For instance if the disk goes full then the kubelet will start deleting images to free up some space on the control plane node.

[afbjorklund]

It's actually something that could be a nice feature, and would deserve some better documentation.

• Add possibility to use host docker registry #12087

[afbjorklund]

Alternatively one could set up a real (https) registry inside the cluster, and avoid the localhost (http) hack.

This would involve creating and distributing the ssl certs for the registry ClusterIP, to the other nodes/pods.

[mikwieczorek]

@afbjorklund thank you for your prompt and detailed reply. I really appreciate your help.

I took you advice and deployed minikube with docker driver this time.

As for the localhost and docker registry I tried to create the proxy to expose localhost:5000 inside pods and to point to my host-machine IP and docker registry. Unfortunately, I could not successfully use the localhost:5000 inside pods.

My settings are as follows: minikube with docker driver and following flags pertaining registry:

--insecure-registry localhost:5000 \
--insecure-registry <MY-IP>:5000 

I tried to reply the registry and proxy using the manifest below. I also tried deploying the proxy alone, but it did not help. The only this I changed in the original deployment (was substitution in the proxy env.value).

kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
    spec:
      containers:
      - name: registry
        image: registry:2.5.1
        imagePullPolicy: Always
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        hostPath:
          path: /data/registry/

---

apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    protocol: TCP

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-registry-proxy
  namespace: kube-system
  labels:
    k8s-app: kube-registry-proxy
    kubernetes.io/cluster-service: "true"
    version: v0.4
spec:
  selector:
    matchLabels:
      k8s-app: kube-registry-proxy
      version: v0.4
  template:
    metadata:
      labels:
        k8s-app: kube-registry-proxy
        version: v0.4
    spec:
      containers:
      - name: kube-registry-proxy
        image: gcr.io/google_containers/kube-registry-proxy:0.4
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        env:
        - name: REGISTRY_HOST
          value: <MY-IP>      ### Used to be kube-registry.kube-system.svc.cluster.local
        - name: REGISTRY_PORT
          value: "5000"
        ports:
        - name: registry
          containerPort: 80
          hostPort: 5000

I really appreciate one more helpful hand in this problem. Of course, the working solution will be posted here to allow other users to solve this problem quickly.

[afbjorklund]

I thought you already had a registry, and just wanted to run the "proxy" (nginx) part of the hack

There was some more useful documentation, before the proxy got removed from kubernetes...

• Multiarch support for registry addon #10780 (comment)

But running with the real host IP is much cleaner, even if it requires either "insecure" or certs:

https://docs.docker.com/registry/deploying/

The "internal" host IP should be available

[mikwieczorek]

I have a registry, but on the host machine, outside the k8s. The only thing I would like to have to resolve localhost:5000 inside the cluster to my host-machine docker registry.
So just the proxy should be enough in this case?

[afbjorklund]

Here was the direct link: cluster/addons/registry/README.md#expose-the-registry-on-each-node

And here is the proxy: cluster/addons/registry/images/Dockerfile (it is a fairly basic nginx image)

[klaases]

Hi @mikwieczorek, were you able to resolve this based on @afbjorklund's comment above?

[mikwieczorek]

@klaases No, I did not use proxy (nginx). I sticked to using my host-machine IP instead of localhost. Works, but is not that elegant and versatile maybe

[klaases]

Hi @mikwieczorek - this issue seems to be related to local networking settings.

Please feel free to re-open the issue by commenting with /reopen. This issue will be closed as additional information was unavailable and some time has passed.

Additional information that may be helpful:

• Whether the issue occurs with the latest minikube release

• The exact minikube start command line used

• Attach the full output of minikube logs, run minikube logs --file=logs.txt to create a log file

Thank you for sharing your experience!