Update ingress-nginx addon #7997

aledbf · 2020-05-04T18:58:20Z

Update version to 0.32
Enable validation webhook (enabled by default now)
Fix image naming (multi-arch now)
Remove configmap setting (map-hash-bucket-size default is 256 now)
References in ingress-dns were wrong.
Add missing rules for new IngressClass resource in k8s v1.18

k8s-ci-robot · 2020-05-04T18:58:35Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aledbf
To complete the pull request process, please assign afbjorklund
You can assign the PR to them by writing /assign @afbjorklund in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tstromberg · 2020-05-04T19:05:37Z

/ok-to-test

medyagh · 2020-05-04T19:11:58Z

@aledbf do you mind sharing the output of using the addon with these changes?

minikube-pr-bot · 2020-05-04T19:16:45Z

kvm2 Driver
docker Driver

aledbf · 2020-05-04T19:17:04Z

make;out/minikube delete;out/minikube start --vm-driver kvm2;out/minikube addons enable ingress 
make: 'out/minikube' is up to date.
[sudo] password for aledbf: 
🔥  Deleting "minikube" in kvm2 ...
💀  Removed all traces of the "minikube" cluster.
😄  minikube v1.10.0-beta.2 on Debian bullseye/sid
✨  Using the kvm2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.18.1 on Docker 19.03.8 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"
🌟  The 'ingress' addon is enabled

k get pods -A
NAMESPACE     NAME                                        READY   STATUS      RESTARTS   AGE
kube-system   coredns-66bff467f8-9gw96                    1/1     Running     0          54s
kube-system   coredns-66bff467f8-p4vtm                    1/1     Running     0          54s
kube-system   etcd-minikube                               1/1     Running     0          55s
kube-system   ingress-nginx-admission-create-2lvcn        0/1     Completed   0          53s
kube-system   ingress-nginx-admission-patch-nswsl         0/1     Completed   0          53s
kube-system   ingress-nginx-controller-7bb4c67d67-zz425   1/1     Running     0          54s
kube-system   kube-apiserver-minikube                     1/1     Running     0          55s
kube-system   kube-controller-manager-minikube            1/1     Running     0          55s
kube-system   kube-proxy-shrgt                            1/1     Running     0          54s
kube-system   kube-scheduler-minikube                     1/1     Running     0          55s
kube-system   storage-provisioner                         1/1     Running     0          60s
 aledbf@me  ~/.../k8s.io/minikube   ingress ●  curl https://gist.githubusercontent.com/aledbf/74296caad6e95c2f5b92517a4f47c256/raw/0c781ae9abb2e373c21af8332ce7b2f5a11db0a6/xip.io | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   860  100   860    0     0   2014      0 --:--:-- --:--:-- --:--:--  2014
deployment.apps/http-svc created
service/http-svc created
Generating a RSA private key
.................................................................++++
.........................................................................................................................................................++++
writing new private key to 'xip.key'
-----
secret/xip-tls created
ingress.networking.k8s.io/ingress created
*   Trying 192.168.39.15:80...
* TCP_NODELAY set
* Connected to 192.168.39.15.xip.io (192.168.39.15) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.39.15.xip.io
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Server: nginx/1.17.10
< Date: Mon, 04 May 2020 19:15:25 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
< Location: https://192.168.39.15.xip.io/
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>
* Connection #0 to host 192.168.39.15.xip.io left intact

curl https://192.168.39.15.xip.io -k

Hostname: http-svc-64f85bcc78-w4gc2

Pod Information:
	node name:	minikube
	pod name:	http-svc-64f85bcc78-w4gc2
	pod namespace:	default
	pod IP:	172.17.0.3

Server values:
	server_version=nginx: 1.12.2 - lua: 10010

Request Information:
	client_address=172.17.0.2
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://192.168.39.15.xip.io:8080/

Request Headers:
	accept=*/*
	host=192.168.39.15.xip.io
	user-agent=curl/7.68.0
	x-forwarded-for=192.168.39.1
	x-forwarded-host=192.168.39.15.xip.io
	x-forwarded-port=443
	x-forwarded-proto=https
	x-real-ip=192.168.39.1
	x-request-id=c99e19679d0e1c8a52e143cf8a2eb108
	x-scheme=https

Request Body:
	-no body in request-

k logs -f -n kube-system ingress-nginx-controller-7bb4c67d67-zz425
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.32.0
  Build:         git-446845114
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.17.10

-------------------------------------------------------------------------------

W0504 19:14:28.841026       7 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0504 19:14:28.841129       7 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0504 19:14:28.841447       7 main.go:220] Creating API client for https://10.96.0.1:443
I0504 19:14:28.848711       7 main.go:264] Running in Kubernetes cluster version v1.18 (v1.18.1) - git (clean) commit 7879fc12a63337efff607952a323df90cdc7a335 - platform linux/amd64
I0504 19:14:28.957712       7 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0504 19:14:28.958724       7 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
W0504 19:14:28.961366       7 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I0504 19:14:28.964296       7 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0504 19:14:29.008909       7 nginx.go:263] Starting NGINX Ingress controller
I0504 19:14:29.024439       7 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"tcp-services", UID:"4271c349-ae12-4873-a2fa-53c1238fde79", APIVersion:"v1", ResourceVersion:"380", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/tcp-services
I0504 19:14:29.025814       7 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"nginx-load-balancer-conf", UID:"5d68e01a-f466-4a68-bd09-0ae5d302c738", APIVersion:"v1", ResourceVersion:"379", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/nginx-load-balancer-conf
I0504 19:14:29.027556       7 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"udp-services", UID:"aa36256e-acae-4b75-a233-28c171ca8b41", APIVersion:"v1", ResourceVersion:"381", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/udp-services
I0504 19:14:30.210551       7 nginx.go:307] Starting NGINX process
I0504 19:14:30.210600       7 leaderelection.go:242] attempting to acquire leader lease  kube-system/ingress-controller-leader-nginx...
I0504 19:14:30.211145       7 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0504 19:14:30.211428       7 controller.go:139] Configuration changes detected, backend reload required.
I0504 19:14:30.238514       7 leaderelection.go:252] successfully acquired lease kube-system/ingress-controller-leader-nginx
I0504 19:14:30.238515       7 status.go:86] new leader elected: ingress-nginx-controller-7bb4c67d67-zz425
I0504 19:14:30.282362       7 controller.go:155] Backend successfully reloaded.
I0504 19:14:30.282391       7 controller.go:164] Initial sync, sleeping for 1 second.
W0504 19:15:15.145311       7 controller.go:909] Service "default/http-svc" does not have any active Endpoint.
W0504 19:15:15.145453       7 controller.go:1119] Error getting SSL certificate "default/xip-tls": local SSL certificate default/xip-tls was not found. Using default certificate
I0504 19:15:15.178286       7 main.go:115] successfully validated configuration, accepting ingress ingress in namespace default
I0504 19:15:15.182278       7 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress", UID:"d27df886-447a-4ce7-97a8-71f735489981", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"693", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/ingress
I0504 19:15:15.182346       7 backend_ssl.go:66] Adding Secret "default/xip-tls" to the local store
W0504 19:15:17.544746       7 controller.go:909] Service "default/http-svc" does not have any active Endpoint.
I0504 19:15:17.544830       7 controller.go:139] Configuration changes detected, backend reload required.
I0504 19:15:17.618423       7 controller.go:155] Backend successfully reloaded.
192.168.39.1 - - [04/May/2020:19:15:25 +0000] "GET / HTTP/1.1" 308 172 "-" "curl/7.68.0" 84 0.000 [default-http-svc-80] [] - - - - 418f1e68a3a55f013e0074f665e3d6b3
I0504 19:15:30.265822       7 status.go:275] updating Ingress default/ingress status from [] to [{192.168.39.15 }]
I0504 19:15:30.277533       7 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress", UID:"d27df886-447a-4ce7-97a8-71f735489981", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"737", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/ingress
192.168.39.1 - - [04/May/2020:19:16:04 +0000] "GET / HTTP/2.0" 200 710 "-" "curl/7.68.0" 35 0.002 [default-http-svc-80] [] 172.17.0.3:8080 722 0.002 200 c99e19679d0e1c8a52e143cf8a2eb108

codecov-io · 2020-05-04T20:44:51Z

Codecov Report

Merging #7997 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #7997   +/-   ##
=======================================
  Coverage   35.78%   35.78%           
=======================================
  Files         143      143           
  Lines        9174     9174           
=======================================
  Hits         3283     3283           
  Misses       5490     5490           
  Partials      401      401

tstromberg · 2020-05-04T21:47:56Z

This appears to break the TestAddons/parallel/Ingress test:

https://storage.googleapis.com/minikube-builds/logs/7997/bc85e70/KVM_Linux.html#fail_TestAddons%2fparallel%2fIngress

addons_test.go:133: failed to get response from ngninx ingress on 127.0.0.1:80: out/minikube-linux-amd64 -p addons-20200504135432-16769 ssh "curl http://127.0.0.1:80 -H 'Host: nginx.example.com'" stdout = "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.17.10</center>\r\n</body>\r\n</html>\r\n", want "Welcome to nginx!"

To debug this, you may find this useful to run:

make integration -e TEST_ARGS="-test.run TestAddons/parallel/Ingress --profile=minikube --cleanup=false"

Maybe the URL mapping changed?

aledbf · 2020-05-04T22:13:52Z

make integration -e TEST_ARGS="-test.run TestAddons/parallel/Ingress --profile=minikube --cleanup=false"

Thank you for the tip.

The error is

Error from server (InternalError): Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.101.110.130:443: connect: connection refused

This happens because the admission webhook is not available. I added a retry for kubectl command

TravisBuddy · 2020-05-04T22:17:12Z

Travis tests have failed

Hey @aledbf,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

TravisBuddy Request Identifier: fea47810-8e54-11ea-80eb-6b769f7cd14d

medyagh · 2020-05-04T22:52:27Z

@aledbf thanks for the putting the output, btw I like your helper script that u use to create eample ingress

aledbf@me  ~/.../k8s.io/minikube   ingress ●  curl https://gist.githubusercontent.com/aledbf/74296caad6e95c2f5b92517a4f47c256/raw/0c781ae9abb2e373c21af8332ce7b2f5a11db0a6/xip.io | bash

I think in a separate PR, I would love to see that as part of a Tutorial on our website.
our current ingress tutorial seems to be broken.I believe it is missing generating secrets as u did in your script.

aledbf · 2020-05-05T00:49:30Z

/retest

minikube-pr-bot · 2020-05-05T19:47:13Z

kvm2 Driver
Times for minikube: [66.099230051 65.097445058 65.566100407]
Average time for minikube: 65.58759183866667

Times for Minikube (PR 7997): [62.180647117999996 62.374720954000004 63.571860662]
Average time for Minikube (PR 7997): 62.70907624466667

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7997) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on   |  0.062740 |           0.059311 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.022105 |           0.022754 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.004275 |           0.002289 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.750895 |          39.650198 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.1 | 22.835735 |          21.490187 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.373809 |           1.285600 |
| components...                  |           |                    |
| * Enabled addons:              |  0.441477 |           0.124175 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.092457 |           0.068490 |
| configured to use "minikube"   |           |                    |
|                                |  0.004099 |           0.006073 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.247194621000002 28.965363975 27.037102151000003]
Average time for minikube: 27.749886915666668

Times for Minikube (PR 7997): [27.045140738 28.42986858500001 28.268166965000002]
Average time for Minikube (PR 7997): 27.914392096

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7997) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on           |  0.075196 |           0.077134 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002584 |           0.003079 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.059289 |           0.065365 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.791817 |           7.806055 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.1         |  0.122576 |           0.120056 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 18.201596 |          18.022808 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.423146 |           1.751639 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.002601 |           0.002698 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.065085 |           0.060584 |
| configured to use "minikube"           |           |                    |
|                                        |  0.005996 |           0.004974 |
+----------------------------------------+-----------+--------------------+

aledbf · 2020-05-06T16:31:13Z

@tstromberg @medyagh the errors seems not related to the PR. Is there anything else I need to do/check? Thanks

tstromberg · 2020-05-07T17:08:45Z

Thank you!

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 4, 2020

k8s-ci-robot requested review from RA489 and sharifelgamal May 4, 2020 18:58

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 4, 2020

aledbf force-pushed the ingress branch from f982780 to bc85e70 Compare May 4, 2020 20:37

aledbf force-pushed the ingress branch from bc85e70 to ac0569e Compare May 4, 2020 22:11

tstromberg mentioned this pull request May 4, 2020

validateIngressAddon: cleanup output, particularly stderr warnings #8005

Merged

aledbf added 2 commits May 5, 2020 15:36

Update ingress-nginx addon

c3a529f

Fix invalid ingress-dns references and ingress

8209421

aledbf force-pushed the ingress branch from ac0569e to 8209421 Compare May 5, 2020 19:36

tstromberg merged commit 9d32294 into kubernetes:master May 7, 2020

tstromberg mentioned this pull request May 8, 2020

KVM old-k8s-version: waiting for default service account: waited 6m for SA: timed out waiting for the condition #8048

Closed

fmotrifork mentioned this pull request May 12, 2020

minikube 1.10.0 fishworks/fish-food#725

Merged

sepetrov mentioned this pull request Jun 11, 2020

Fix deployment name in ingress Nginx tutorial #8451

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ingress-nginx addon #7997

Update ingress-nginx addon #7997

aledbf commented May 4, 2020 •

edited

Loading

k8s-ci-robot commented May 4, 2020

tstromberg commented May 4, 2020

medyagh commented May 4, 2020

minikube-pr-bot commented May 4, 2020

aledbf commented May 4, 2020 •

edited

Loading

codecov-io commented May 4, 2020 •

edited

Loading

tstromberg commented May 4, 2020 •

edited

Loading

aledbf commented May 4, 2020

TravisBuddy commented May 4, 2020

medyagh commented May 4, 2020 •

edited

Loading

aledbf commented May 5, 2020

minikube-pr-bot commented May 5, 2020

aledbf commented May 6, 2020

tstromberg commented May 7, 2020

Update ingress-nginx addon #7997

Update ingress-nginx addon #7997

Conversation

aledbf commented May 4, 2020 • edited Loading

k8s-ci-robot commented May 4, 2020

tstromberg commented May 4, 2020

medyagh commented May 4, 2020

minikube-pr-bot commented May 4, 2020

aledbf commented May 4, 2020 • edited Loading

codecov-io commented May 4, 2020 • edited Loading

Codecov Report

tstromberg commented May 4, 2020 • edited Loading

aledbf commented May 4, 2020

TravisBuddy commented May 4, 2020

Travis tests have failed

TravisBuddy Request Identifier: fea47810-8e54-11ea-80eb-6b769f7cd14d

medyagh commented May 4, 2020 • edited Loading

aledbf commented May 5, 2020

minikube-pr-bot commented May 5, 2020

aledbf commented May 6, 2020

tstromberg commented May 7, 2020

aledbf commented May 4, 2020 •

edited

Loading

aledbf commented May 4, 2020 •

edited

Loading

codecov-io commented May 4, 2020 •

edited

Loading

tstromberg commented May 4, 2020 •

edited

Loading

medyagh commented May 4, 2020 •

edited

Loading