Secrets not created

[jwaldrip]

Secrets are not getting created. My logs keep looping over the following messages:

21:16:05.000
time="2017-01-30T04:16:05Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:08.000
time="2017-01-30T04:16:08Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:11.000
time="2017-01-30T04:16:11Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:12.000
time="2017-01-30T04:16:12Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:15.000
time="2017-01-30T04:16:15Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps

Any idea why this would be happening?

[rimusz]

yup, seeing the same issue too

[FourSigma]

Seeing the same issue as well :-(

[orian]

For a brief moment I've thought I had the same issue. Please notice that:

1. you need to properly configure your domain in ingress.yaml
2. you need to point dns to the assigned IP, otherwise it let's encrypt cannot properly verify the domain

[simonswine]

I think that might be a problem, with the log level not high enough. Can you try running the kube-lego pod with debug flags:

   env:
        - name: LEGO_LOG_LEVEL
          value: debug

[simonswine]

@FourSigma @rimusz @jwaldrip

[huysamen]

Not sure if mine is the same issue, but it seems that the secret is getting created incorrectly, since after creating the secret, I get the following error:

Error while process certificate requests: Secret \"app-dev-tls\" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value]" context=kubelego

[simonswine]

@huysamen please enable debug logging and provide a bit more info (K8S objects, ...)

[pierreozoux]

So this is related to #77 and #62
I hit the same issue.

I activated debug, but not much more help:

time="2017-02-24T16:20:59Z" level=debug msg="worker: begin processing true" context=kubelego 
time="2017-02-24T16:20:59Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=nginx 
time="2017-02-24T16:20:59Z" level=debug msg=reset context=provider provider=gce 
time="2017-02-24T16:20:59Z" level=debug msg=finialize context=provider provider=gce 
time="2017-02-24T16:20:59Z" level=debug msg=reset context=provider provider=nginx 
time="2017-02-24T16:20:59Z" level=debug msg=finialize context=provider provider=nginx 
time="2017-02-24T16:21:00Z" level=info msg="process certificates requests for ingresses" context=kubelego 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=tls namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=app namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="requesting certificate for ***domainname***" context="ingress_tls" name=app namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=kube-lego-account namespace=nginx 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=tls namespace=production 
time="2017-02-24T16:21:01Z" level=error msg="Error while process certificate requests: Secret \"tls\" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value]" context=kubelego 
time="2017-02-24T16:21:01Z" level=debug msg="worker: done processing true" context=kubelego 

Here is my ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app
  annotations:
    kubernetes.io/tls-acme: 'true'
    kubernetes.io/ingress.class: nginx
spec:
  tls:
    - secretName: tls
      hosts:
        - ***domainname***
  rules:
    - host: ***domainname***
      http:
        paths:
          - path: /
            backend:
              serviceName: app
              servicePort: 3000

I did one suggestion, I added one container to the lego pod, exec'ed inside it and run the following:

ping 8.8.8.8
wget https://acme-v01.api.letsencrypt.org

And it worked as expected.
I couldn't try inside the lego container itself. (It reminds me unikernel :) )

For information, I'm running on GKE, and I installed everything with helm:

helm install --namespace nginx --name nginx stable/nginx-ingress
helm install --namespace nginx --name lego -f k8s/values-lego.yml stable/kube-lego

(The values are just the lego API endpoint and my email)

Is there anything I can do to help debug this? Thanks a lot for your work!

[pierreozoux]

Edit:

I found my issue:

here was the value of :

LEGO_URL: Lhttps://acme-v01.api.letsencrypt.org/directory

You got it? Yes, me too... Lost 2 hours...

It would be a nice to have to have it slightly more verbose :)

Everything is working on my side! Have a wonderful week-end!

[galexrt]

Old comment text. Click to expand

Same issue here. But in my case some secrets get created, but others don't. ``` time="2017-03-27T21:37:40Z" level=info msg="kube-lego 0.1.3-d425b293 starting" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="connected to kubernetes api v1.5.3" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="start watching ingress objects" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="server listening on http://:8080/" context=acme time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test2/testapp-1" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test2/testapp-2" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test1/testapp-3" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/monitoring/examplenet-monitoring" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/monitoring/examplenet-testapp-4" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:37:40Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:37:40Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:37:40Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:37:40Z" level=debug msg=finialize context=provider provider=nginx time="2017-03-27T21:37:40Z" level=info msg="process certificates requests for ingresses" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="cert expires in 80.9 days, no renewal needed" context="ingress_tls" expire_time=2017-06-16 19:24:00 +0000 UTC name=examplenet-monitoring namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="no cert request needed" context="ingress_tls" name=examplenet-monitoring namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-4 namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="requesting certificate for testapp-1.example.net" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:37:41Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:37:56Z" level=debug msg="error while authorizing: reachabily test failed: wrong status code '504'" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:06Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:07Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:07Z" level=info msg="authorization successful" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:08Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=info msg="requesting certificate for testapp-1.example-test2.net" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=debug msg="testing reachablity of http://testapp-1.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=info msg="authorization successful" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-2 namespace=example-test2 time="2017-03-27T21:38:10Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:10Z" level=info msg="requesting certificate for testapp-2.example-test2.net" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:11Z" level=debug msg="testing reachablity of http://testapp-2.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=info msg="authorization successful" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:12Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:12Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:13Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:15Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=testapp-3.example.net token="[REMOVED]" time="2017-03-27T21:38:17Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:17Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:17Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:17Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:17Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:18Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:19Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=testapp-3.example.net token="[REMOVED]" time="2017-03-27T21:38:21Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:21Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:22Z" level=info msg="successfully got certificate: domains=[testapp-3.example.net] url=https://acme-v01.api.letsencrypt.org/acme/cert/[REMOVED]" context=acme time="2017-03-27T21:38:22Z" level=debug msg="certificate pem data:\n-----BEGIN CERTIFICATE-----\n[REMOVED]\n-----END CERTIFICATE-----\n" context=acme time="2017-03-27T21:38:22Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=error msg="Error while process certificate requests: error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net" context=kubelego time="2017-03-27T21:38:22Z" level=debug msg="worker: done processing true" context=kubelego time="2017-03-27T21:38:22Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:38:22Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:38:22Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:38:22Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:38:22Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:38:22Z" level=debug msg=finialize context=provider provider=nginx time="2017-03-27T21:38:22Z" level=info msg="process certificates requests for ingresses" context=kubelego time="2017-03-27T21:38:22Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-06-25 20:38:00 +0000 UTC name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=info msg="no cert request needed" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=info msg="cert expires in 80.9 days, no renewal needed" context="ingress_tls" expire_time=2017-06-16 19:24:00 +0000 UTC name=examplenet-monitoring namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="no cert request needed" context="ingress_tls" name=examplenet-monitoring namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-4 namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="requesting certificate for testapp-1.example.net" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:38:23Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=info msg="authorization successful" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-1 namespace=example-test2 time="2017-03-27T21:38:24Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:24Z" level=info msg="requesting certificate for testapp-1.example-test2.net" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:25Z" level=debug msg="testing reachablity of http://testapp-1.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:26Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:26Z" level=info msg="authorization successful" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:27Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=info msg="requesting certificate for testapp-2.example-test2.net" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=debug msg="testing reachablity of http://testapp-2.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:28Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:28Z" level=info msg="authorization successful" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:29Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:30Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:30Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:31Z" level=error msg="Error while process certificate requests: error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net" context=kubelego time="2017-03-27T21:38:31Z" level=debug msg="worker: done processing true" context=kubelego time="2017-03-27T21:38:31Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:38:31Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:38:31Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:38:31Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:38:31Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:38:31Z" level=debug msg=finialize context=provider provider=nginx ``` (The original domains have been replaced with mostly `example*.net`. The domains are valid and when using the `certbot` manually I got working certificates out)

---

It seems that because one certificate failed, kube-lego went into a loop and hit the rate limit in my case.

[andrejpk]

I had the same problem.. forgot to update my email address in the template.. a better error would have saved me some time. :)