Docker renewing certificate(s) on startup even if certificate exists

[stevmon]

We are trying to switch a client to use jrcs for certificates so we can use v2 for letsencrypt certificates.

I have a test environment set up using docker-compose since there are big changes to the docker-compose.yml file.

I was using the staging URL in the YML file but wanted to check to make sure that we could retrieve a "real" certificate for the site and not have to accept a self-signed certificate. (I think that was what the browser needed...)

So I commented out the line "# ACME_CA_URI: https://acme-staging-v02.api.letsencrypt.org/directory" in the YML file and everything seemed well. The certificate was accepted in the browser no problem.

But we noticed that each time I performed a "docker-compose down" and then a "docker-compose up" it would reload/pull new certificates for the site.

I don't think this should be normal operation as the certificates should already be persistent according to all I have read.

A couple of excerpts that I think might be relevant from the logs:

letsencrypt-proxy    | Info: running acme-companion version v2.1.0-5-g1b01042
nginx-proxy          | WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one
nginx-proxy          | is being generated in the background.  Once the new dhparam.pem is in place, nginx will be reloaded.
...
nginx-proxy          | dockergen.1 | 2021/05/04 17:29:33 Generated '/etc/nginx/conf.d/default.conf' from 4 containers
nginx-proxy          | dockergen.1 | 2021/05/04 17:29:33 Running 'nginx -s reload'
...
letsencrypt-proxy    | Warning: '/etc/acme.sh' does not appear to be a mounted volume.
letsencrypt-proxy    | Info: Custom Diffie-Hellman group found, generation skipped.

My YML file is as follows:

version: "2"

services:

  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - letsencrypt-certs:/etc/nginx/certs
      - letsencrypt-vhost-d:/etc/nginx/vhost.d
      - letsencrypt-html:/usr/share/nginx/html
      
  letsencrypt-proxy:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt-certs:/etc/nginx/certs
      - letsencrypt-vhost-d:/etc/nginx/vhost.d
      - letsencrypt-html:/usr/share/nginx/html
    environment:
#      ACME_CA_URI: https://acme-staging-v02.api.letsencrypt.org/directory
      DEFAULT_EMAIL: mail@sio.midco.net
      NGINX_PROXY_CONTAINER: nginx-proxy
 
 
  mariadb:
    image: mariadb:latest
    expose:
      - 3306
    volumes:
      - ./logs/mariadb:/var/log/mysql
      - ./mariadb-data:/var/lib/mysql
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: pass
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: pass
    container_name: mariadb
 
  wordpress:
    depends_on:
      - mariadb
    image: wordpress:latest
    expose:
      - 80
    restart: unless-stopped
    volumes:
      - ./webroot:/var/www/html
      - ./logs/wordpress:/var/log/wordpress
    environment:
      VIRTUAL_HOST: HOST.com
      LETSENCRYPT_HOST: HOST.com
      WORDPRESS_DB_HOST: mariadb:3306
      WORDPRESS_DB_USER: user
      WORDPRESS_DB_NAME: dbname
      WORDPRESS_DB_PASSWORD: dbpass
      WORDPRESS_TABLE_PREFIX: prefix
    container_name: wordpress
 

networks:
  default:
    external:
      name: nginx-proxy
 
volumes:
  letsencrypt-certs:
  letsencrypt-vhost-d:
  letsencrypt-html:

Is it normal behavior to re-fetch certificates on startup? If it is my concern is that on the production server if we are doing other updates or changes that might require multiple restarts I might run out of quota for certificates and mess up the website for customers as far as certificates go.

According to the 120,000 YML files I have reviewed the certificate(s) should be persistent but that does not seem to be the case.

Any idea what I am missing?

Thanks!

[buchdag]

Hi, the front of the README has a warning about this:

Required read if you use the latest version : the v2.0.0 release of this project mark the switch of the ACME client used by the Docker image from simp.le to acme.sh. This switch result in some backward incompatible changes, so please read this issue and the updated docs for more details before updating your image. The single most important change is that the container now requires a volume mounted to /etc/acme.sh in order to persist ACME account keys and SSL certificates. The last tagged version that uses simp_le is v1.13.1.

Also please note that this project's name and DockerHub registry have changed. We are still pushing to jrcs/letsencrypt-nginx-proxy-companion but it's considered legacy, the new registry is nginxproxy/acme-companion.

Lastly, I'd strongly advise against using latest version of any image in production. I'd rather use versioned tags whenever available, and update the versions used in a controlled manner (and read the project's release notes ^^) to avoid surprises like this one.

[stevmon]

Thank you for your response.

I had read that section but did not really comprehend how to implement it. I followed the links and did not gather any information that was specific enough for me and actually confused me on what I needed to do.

In fact I spent part of the day searching for items pertaining to acme.sh and implementing it in a YML file - which really took me further off course.

But! I finally did realize that what I needed to do was to create a volume in the volumes section of both my letsencrypt-proxy and nginx-proxy containers in the YML containing:

  - letsencrypt-acme:/etc/acme.sh

This was not obvious to me from that snippet. Also there is a link further down in that page that was supposed to be related to persistence. There was no reference information there about needing that volume. So that maybe should be adjusted?

Thanks again for your response.

[Pryla]

Thank you both so much!

But! I finally did realize that what I needed to do was to create a volume in the volumes section of both my letsencrypt-proxy and nginx-proxy containers in the YML containing:

  - letsencrypt-acme:/etc/acme.sh

Also please note that this project's name and DockerHub registry have changed. We are still pushing to jrcs/letsencrypt-nginx-proxy-companion but it's considered legacy, the new registry is nginxproxy/acme-companion.

Those two bits of information helped me out. I spent so much time and a sleepless night with this. The linked issue might not be too helpful for newbies.

And btw: docker hub for jrcs/letsencrypt-nginx-proxy-companion points out that the latest version for acme 1.0 is v1.11 and not v1.13.1 or what am I missing here?

Thank you!

[buchdag]

@stevmon @Pryla any PR with clarification to the README.md are welcome

And btw: docker hub for jrcs/letsencrypt-nginx-proxy-companion points out that the latest version for acme 1.0 is v1.11 and not v1.13.1 or what am I missing here?

Unfortunately you're not missing anything, I don't have direct modification right over the Readme of https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion, I want to fix this but I haven't had the time yet.

[buchdag]

But! I finally did realize that what I needed to do was to create a volume in the volumes section of both my letsencrypt-proxy and nginx-proxy containers in the YML containing

You don't need to add this volume to the nginx-proxy container, only the acme-companion container reads from / writes to /etc/acme.sh

[stevmon]

Actually zanzara should be able to add the line with ACME_CA_URI below and use a test certificate:

letsencrypt-proxy:
image: nginxproxy/acme-companion
container_name: letsencrypt-proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- letsencrypt-certs:/etc/nginx/certs
- letsencrypt-vhost-d:/etc/nginx/vhost.d
- letsencrypt-html:/usr/share/nginx/html
- letsencrypt-acme:/etc/acme.sh
environment:
ACME_CA_URI: https://acme-staging-v02.api.letsencrypt.org/directory
DEFAULT_EMAIL: somemail@yahoo.com
NGINX_PROXY_CONTAINER: nginx-proxy
restart: always

[zanzara]

Thx, but this is what I just did in the meantime (look at my post above).
The issue with the staging image is, that browsers are complaining too intensive about the invalid cert.
Users out there won't click through the browser's exceptions buttons to see however the page - which is rather reasonable.

Even worse as app artifact on the phone. Cause I cannot configure capacitor that way, that it accepts such an invalid cert.

So in the end, I have to wait now until rating limit is refreshed again for my subdomains.

[buchdag]

@zanzara you can also get valid certificates from another ACME compatible CA like Zero SSL.

Let me know if Zero SSL usage isn't clear enough in the docs.

[zanzara]

Hooray, you made my day (although it's midnight here now) !
It works like charm with Zero SSL (as you wrote it in your docs). Great. Also app is working again now.

I'm not stucking so deep in this certification stuff as you obviously. I did only know about LetsEncrpyt.
So, good luck and good night

[SeungWookHan]

@buchdag Hello! Now I'm using Zero SSL because of rate limiting issue
my docker-compose file is like this

version: "3.9"
services:
  nginx-proxy:
    container_name: nginx-proxy
    image: nginxproxy/nginx-proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - certs:/etc/nginx/certs:ro      
      - ../nginx/log:/var/log/nginx 
      - html:/usr/share/nginx/html
      - vhost:/etc/nginx/vhost.d
      - dhparam:/etc/nginx/dhparam
      - conf:/etc/nginx/conf.d
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/tmp/docker.sock:ro
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx-proxy"
    restart: always

  letsencrypt-nginx-proxy:
    container_name: letsencrypt-nginx-proxy
    image: nginxproxy/acme-companion
    depends_on:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - html:/usr/share/nginx/html
      - vhost:/etc/nginx/vhost.d
      - dhparam:/etc/nginx/dhparam:ro
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: always
    environment:
      - NGINX_PROXY_CONTAINER=nginx-proxy
      - ACME_CA_URI=https://acme.zerossl.com/v2/DV90
      - ZEROSSL_API_KEY="my api key.."
  
  user-frontend:
    environment:
      - VIRTUAL_HOST=${HOST}
      - LETSENCRYPT_HOST=${HOST}
    expose:
      - 3000

  admin-frontend:
    environment:
      - VIRTUAL_HOST=admin.${HOST}
      - LETSENCRYPT_HOST=admin.${HOST}
    expose:
      - 3000

  express:
    environment:
    - VIRTUAL_HOST=api.${HOST}
    - LETSENCRYPT_HOST=api.${HOST}
    expose:
      - ${PORT}
  
  adminer:
    environment:
    - VIRTUAL_HOST=db.admin.${HOST}
    - LETSENCRYPT_HOST=db.admin.${HOST}
    expose:
      - 8080

volumes:
  certs:
  conf:
  vhost:
  html:
  dhparam:
  acme:

some of container is work but some of still doesn't have ssl verification...
Is there any wrong in my compose file or should I add something on it?

[stevmon]

Thank you for your responses. I was really suffering not knowing what I was doing wrong.