DNS name resolving not working in kubernetes #339

henningWoehr · 2023-10-10T14:52:18Z

Hi, I just started using your project on kubernetes and got a problem with the newest patches

Bug description

Since the version 2.5.3, the NGINX_PROXY_PASS_URL can't be resolved anymore.
My guess is, that it is related with this #333 issue and maybe the fix of that caused this issue, but I'm not really into nginx to understand the config there.

How to Reproduce

With these manifests and the ui version set to 2.5.2, everything works fine, but any higher version doesn't work.

My container-registry manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: container-registry
  namespace: default
spec:
  selector:
    matchLabels:
      app: container-registry
  template:
    metadata:
      labels:
        app: container-registry
    spec:
      volumes:
        - name: repos
          persistentVolumeClaim:
            claimName: container-registry
      containers:
        - name: container-registry
          image: registry:2
          ports:
          - containerPort: 5000
          env:
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
          resources:
            limits:
              memory: "64Mi"
              cpu: "200m"
            requests:
              memory: "15Mi"
              cpu: "10m"
          volumeMounts:
          - name: repos
            mountPath: "/var/lib/registry"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: container-registry
  namespace: default
spec:
  resources:
    requests:
      storage: 10G
  accessModes:
    - ReadWriteOnce
---
apiVersion: v1
kind: Service
metadata:
  name: container-registry
  namespace: default
spec:
  selector:
    app: container-registry
  type: NodePort
  ports:
  - port: 80
    targetPort: 5000
    nodePort: 30500

My container-registry-ui manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: container-registry-ui
  namespace: default
spec:
  selector:
    matchLabels:
      app: container-registry-ui
  template:
    metadata:
      labels:
        app: container-registry-ui
    spec:
      containers:
        - name: container-registry-ui
          image: joxit/docker-registry-ui:2.5.4
          env:
            - name: REGISTRY_TITLE
              value: Local Registry
            - name: DELETE_IMAGES
              value: "true"
            - name: NGINX_PROXY_PASS_URL
              value: http://container-registry:80
            - name: SINGLE_REGISTRY
              value: "true"
          resources: 
            limits:
              memory: "32Mi"
              cpu: "100m"
            requests:
              memory: "5Mi"
              cpu: "1m"
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: container-registry-ui
  namespace: default
spec:
  selector:
    app: container-registry-ui
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30501

If applicable, add screenshots to help explain your problem.

System information

Not neccessary in my opinion

Additional context

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/90-docker-registry-ui.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/10/10 14:34:19 [notice] 1#1: using the "epoll" event method
2023/10/10 14:34:19 [notice] 1#1: nginx/1.25.2
2023/10/10 14:34:19 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2023/10/10 14:34:19 [notice] 1#1: OS: Linux 5.15.89-talos
2023/10/10 14:34:19 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/10/10 14:34:19 [notice] 1#1: start worker processes
2023/10/10 14:34:19 [notice] 1#1: start worker process 354
2023/10/10 14:34:19 [notice] 1#1: start worker process 355
2023/10/10 14:34:19 [notice] 1#1: start worker process 356
2023/10/10 14:34:19 [notice] 1#1: start worker process 357
127.0.0.1 - - [10/Oct/2023:14:34:54 +0000] "GET / HTTP/1.1" 200 2083 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"
2023/10/10 14:34:54 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:34:54 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:34:59 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:35:04 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:35:09 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:35:14 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:35:19 [error] 355#355: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
127.0.0.1 - - [10/Oct/2023:14:35:22 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"
2023/10/10 14:35:22 [error] 354#354: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/10 14:35:22 [error] 354#354: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
127.0.0.1 - - [10/Oct/2023:14:35:23 +0000] "GET /v2/_catalog?n=1000 HTTP/1.1" 499 0 "http://localhost:10002/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"

The text was updated successfully, but these errors were encountered:

Slyke · 2023-10-15T10:08:50Z

Same issue for me:

2023/10/15 10:03:07 [error] 397#397: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/15 10:03:12 [error] 397#397: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/15 10:03:17 [error] 397#397: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/15 10:03:22 [error] 397#397: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/15 10:03:27 [error] 397#397: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
2023/10/15 10:03:32 [error] 397#397: *22 docker-registry.pipelines.svc.cluster.local could not be resolved (110: Operation timed out), client: 10.7.1.229, server: localhost, request: "GET /v2/_catalog?n=1000 HTTP/1.1", host: "10.3.1.211", referrer: "http://10.3.1.211/"
10.7.1.229 - - [15/Oct/2023:10:03:32 +0000] "GET /v2/_catalog?n=1000 HTTP/1.1" 502 497 "http://10.3.1.211/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"

In the browser, this is returned as a 502 error.

From inside the pod:

/usr/share/nginx/html # ping google.com
PING google.com (142.251.215.238): 56 data bytes
64 bytes from 142.251.215.238: seq=0 ttl=117 time=5.823 ms
64 bytes from 142.251.215.238: seq=1 ttl=117 time=5.837 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 5.823/5.830/5.837 ms
/usr/share/nginx/html # ping docker-registry.pipelines.svc.cluster.local.
PING docker-registry.pipelines.svc.cluster.local. (10.100.13.197): 56 data bytes
^C
--- docker-registry.pipelines.svc.cluster.local. ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
/usr/share/nginx/html # nslookup google.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   google.com
Address: 142.250.217.78

Non-authoritative answer:
Name:   google.com
Address: 2607:f8b0:400a:804::200e

/usr/share/nginx/html # nslookup google.com
Server:         10.96.0.10
Address:        10.96.0.10:53

Non-authoritative answer:
Name:   google.com
Address: 142.250.69.206

Non-authoritative answer:
Name:   google.com
Address: 2607:f8b0:400a:804::200e

I'm using v2.5.4. This error does not happen when changing the version to v2.5.2

Joxit · 2023-10-16T23:59:36Z

Hi both of you, thank you for using my project and posting issues.

The bug is confirmed, I'm looking for the best way to support both of the use cases (standard network and kubernetes)

Joxit · 2023-10-17T00:45:51Z

Thank you for your patience.
This should be fixed in 2.5.5, you will also be able to set a custom dns resolver if you need to with the option NGINX_RESOLVER 😄

Joxit self-assigned this Oct 16, 2023

Joxit added the bug label Oct 16, 2023

Joxit added this to docker-registry-ui Oct 16, 2023

github-project-automation bot moved this to To do in docker-registry-ui Oct 16, 2023

Joxit closed this as completed in 7991442 Oct 17, 2023

github-project-automation bot moved this from To do to Done in docker-registry-ui Oct 17, 2023

Joxit mentioned this issue Oct 17, 2023

NGINX_PROXY_PASS_URL use long cache DNS resolution #333

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS name resolving not working in kubernetes #339

DNS name resolving not working in kubernetes #339

henningWoehr commented Oct 10, 2023

Slyke commented Oct 15, 2023 •

edited

Loading

Joxit commented Oct 16, 2023

Joxit commented Oct 17, 2023

DNS name resolving not working in kubernetes #339

DNS name resolving not working in kubernetes #339

Comments

henningWoehr commented Oct 10, 2023

Bug description

How to Reproduce

System information

Additional context

Slyke commented Oct 15, 2023 • edited Loading

Joxit commented Oct 16, 2023

Joxit commented Oct 17, 2023

Slyke commented Oct 15, 2023 •

edited

Loading