Create certificates for multiple hosts without adding a CNAME record for each?

[mdbraber]

First off, thanks for building acme-dns - it's just what I was looking for! I'm trying to do the following: I want to generate certificates for the servers foo.example.org, bar.example.org etc. This works fine. However I have to manually add the following in my providers DNS config:

_acme-challenge CNAME 3882beda-a862-4de8-b3ab-4f528fe88845.auth.example.org
_acme-challenge.foo CNAME _acme-challenge
_acme-challenge.bar CNAME _acme-challenge

As the hosts (foo, bar etc.) are being created rather dynamic I would prefer to prevent creating them by hand if possible (also I would prefer to keep the names of the hosts out of my providers DNS. Wildcards obviously won't work and I don't want to generate a wildcard certificate on each hosts as that seems more of a workaround than a real solution.

The only solution so far that I could think of would be to use my providers API to somehow dynamically create the _acme-challenge.foo record. I would have need to have the client (e.g. lego) to do this before requesting the _acme-challenge record. But unfortunately this implies a 'race' condition because to do that I would put my API keys on each host (which is exactly what I'm trying to prevent with acme-dns).

Would there be any (other) ideas to do this smarter / more dynamic?

[joohoi]

Thanks for your kind words!
As you correctly pointed out, acme-dns was created to prevent saving the all-powerful DNS zone credentials on each box that needs to automate DNS challenges. The automation will unfortunately require keeping the CNAME records in the main zone intact to be able to do its magic.

The nature of the records is static, so they need to be created only once, so kind of fire & forget approach to the issue as they will be reused with the created pair of credentials on every automated renewal.

Something to point up though; to steer people towards secure configuration practices, acme-dns only allows two simultaneous TXT record per generated subdomain (for the purpose of getting wildcard and the main domain to the same certificate: *.example.org and example.org, both of which need to use the _acme-challenge.example.org domain for TXT records).

Back to the original question: could you help me understand the situation better? I would like to know why you'd prefer removing the CNAME records from the main zone outside of the challenge validation. I know there are multiple possible reasons for that though, just don't know what you are facing currently!

[mdbraber]

First off - this is all more or less artificial as I'm not running this in production but rather in my homelab. But still I'm trying to see if I can make this work. It's mostly that I would prefer not to have to create separate CNAME records when hosts are brought up / taken down rather quickly. Yes, normally this would be fire and forget, but with containers as I'm using this is rather frequent.

Does that clarify things?

[mdbraber]

Here's my idea for a solution so far:

• Use acmedns with Caddy (which offers Automatic & On-Demand TLS)
• Let Caddy ask whether a domain is allowed to be created: https://caddyserver.com/docs/automatic-https#on-demand

Then I can specifiy a server listening to the ask request (e.g. on the same host as acme-dns) that can dynamically create the host if it's allowed to be created. As I have all my hostnames mapped internally to the domain I'd like to use (using an internal DNS resolver), I could resolve the requesting IP and use that name to determine which DNS record to add.

Any other ideas are welcome!

[mdbraber]

Another idea: if I could supply the username when registering (instead of being dynamic) I could easily create a DNS record like:

_acme-challenge.foo CNAME foo.auth.example.org

That would probably be the most clean / dynamic (and also easiest to keep track of created domains / certificates. Would there be something that prevents usernames from being dynamic (security concerns or otherwise?)

[Ajedi32]

I'm confused. Are you saying you want to automate the registration of new hostnames, but without allowing a service to get certificates for a hostname that a different service already registered, and with some other arbitrary restrictions on what hostnames are allowed to be created?

[mdbraber]

@Ajedi32 I'm sorry, it might be a bit convoluted, but I'll try to clarify:

• I have a lot of dynamic hosts that I want to create SSL certs for
• I don't want my DNS API credentials on all of those hosts
• Therefore I'm using ACME DNS.

The only 'problem' is that I still have to manually add a _acme-challenge.foo record to my DNS. To prevent that I'm trying to think of a solution to create those records dynamically (because I can't use wildcards for something like _acme-challenge.*).

My idea is now is to use Caddy with acmedns so it could automatically request a certificate. Before requesting a certificate (the tls ask variable) it would be able to query a URL to see if the host is allowed to receive a certificate. If I would have a server listening on that URL that creates the _acme-challenge.foo record when it's queried (based on the internally mapped hostname linked to the requesting IP). That way the certificates that can be created are limited to which hosts I have intnerally mapped and automatic

[Ajedi32]

If I would have a server listening on that URL that creates the _acme-challenge.foo record when it's queried (based on the internally mapped hostname linked to the requesting IP). That way the certificates that can be created are limited to which hosts I have intnerally mapped and automatic

Ah. Yeah, that seems like it would work. That way you can program the server to enforce any restrictions you want for hostname creation.

[mdbraber]

I'm trying to understand what goes on when a certificate is being issues by LE. I've tried researching, but I can't yet fully wrap my head around what's going on. So here's my understanding so far:

Suppose we have an acme-dns configuration for domain test.example.org which needs to redirect with a CNAME to UUID.auth.example.org:

1. Client requests certificate validation using acme-dns for domain test.example.org with credentials that link to domain UUID.auth.example.org
2. Client receives challenge from LE and calls acme-dns /update endpoint with the provided challenge for UUID.auth.example.org
3. Let's Encrypt server checks _acme-challenge.test.example.org to see if it contains a CNAME
4. If the CNAME redirects to UUID.auth.example.org that host is queried and acme-dns provides the correct TXT response

Would it work if between step 2 and 3 I let acme-dns dynamically create the CNAME? That way I would only have my DNS provider credentials stored at one single location.

[mdbraber]

I've created this simple Go server to create / destroy the required CNAME records (which can be called by just a simple POST request with the corresponding JSON: https://github.com/mdbraber/update-cname-server

[Ajedi32]

3. Let's Encrypt server checks _acme-challenge.test.example.org to see if it contains a CNAME

Well, sort of. It check's if _acme-challenge.test.example.org contains the correct TXT record, but finds a CNAME instead, so it follows the CNAME redirect to UUID.auth.example.org and checks if that contains the correct TXT record.

Would it work if between step 2 and 3 I let acme-dns dynamically create the CNAME? That way I would only have my DNS provider credentials stored at one single location.

Yes, but at that point I don't see why you wouldn't just have it create the TXT record directly. No need for acme-dns when you have a way to fulfill the challenge yourself.

[Ajedi32]

It seems like you might be better off writing an acme-dns alternative instead of using ACME-DNS. You basically just want some kind of DNS API proxy that fulfills requests to set TXT records on your upstream DNS (just like ACME clients do) but with restrictions so clients can't just update whatever records they want.

That can be done without needing a whole separate DNS server, so I think it's better implemented as a separate project rather than an add-on to ACME-DNS.

I've actually kinda been wanting something like that for a while now; a centralized server that can fulfill ACME challenges for other devices on the network, provided they meet certain authorization conditions. ACME-DNS sorta fulfills that use-case, but only for one "authorization condition": manual approval (by setting a CNAME record) for a specific acme-dns account to authorize certs for a specific domain.

[mdbraber]

@Ajedi32 you're right. My setup works, but it's indeed not needed to use acme-dns for this. Therefore I've quickly written acme-proxy . This might be an alternative for those people who do have access to their DNS providers' API, but would like to centralize requests.

It integrates with lego and can basically proxy all the providers. To use it now, use my current fork of lego that integrates acme-proxy as a provider. I've created a PR with lego to gather suggestions for a better implementation: go-acme/lego#708. It works great at this moment:

Running the proxy:

mdbraber-mbp:acme-proxy mdbraber$ ACMEPROXY_PROVIDER="transip" TRANSIP_ACCOUNT_NAME="mdbraber" TRANSIP_PRIVATE_KEY_PATH="/Users/mdbraber/transip.key" go run acme-proxy.go

Requesting a certificate:

mdbraber-mbp:lego mdbraber$ ACMEPROXY_URL="http://127.0.0.1:9095/" ./lego -m m@mdbraber.com -a -x http-01 -x tls-alpn-01 --dns acme-proxy --dns-resolvers ns0.transip.nl -s https://acme-staging-v02.api.letsencrypt.org/directory -d mdbraber.net -d *.mdbraber.net run
2018/11/08 10:11:38 [INFO] acme: Registering account for m@mdbraber.com
2018/11/08 10:11:38 !!!! HEADS UP !!!!
2018/11/08 10:11:38
		Your account credentials have been saved in your Let's Encrypt
		configuration directory at "/Users/mdbraber/go/src/github.com/mdbraber/lego/.lego/accounts/acme-staging-v02.api.letsencrypt.org/m@mdbraber.com".
		You should make a secure backup	of this folder now. This
		configuration directory will also contain certificates and
		private keys obtained from Let's Encrypt so making regular
		backups of this folder is ideal.
2018/11/08 10:11:38 [INFO] [mdbraber.net, *.mdbraber.net] acme: Obtaining bundled SAN certificate
2018/11/08 10:11:39 [INFO] [*.mdbraber.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/lBFxlzA3lbOJ8a7cAmIv-vP-Qe1OU4ZSR_q4tmD5Af4
2018/11/08 10:11:39 [INFO] [mdbraber.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/RY19RDOYqi2UbTBqmU5FmU1ZVx5FT7kP1xsO5dkodIc
2018/11/08 10:11:39 [INFO] [mdbraber.net] acme: Could not find solver for: tls-alpn-01
2018/11/08 10:11:39 [INFO] [mdbraber.net] acme: Could not find solver for: http-01
2018/11/08 10:11:39 [INFO] [mdbraber.net] acme: Preparing to solve DNS-01
2018/11/08 10:11:41 [INFO] [mdbraber.net] acme: Preparing to solve DNS-01
2018/11/08 10:11:43 [INFO] [mdbraber.net] acme: Trying to solve DNS-01
2018/11/08 10:11:43 [INFO] [mdbraber.net] Checking DNS record propagation using [ns0.transip.nl:53]
2018/11/08 10:11:43 [INFO] Wait [timeout: 10m0s, interval: 10s]
2018/11/08 10:14:34 [INFO] [mdbraber.net] The server validated our request
2018/11/08 10:14:34 [INFO] [mdbraber.net] acme: Trying to solve DNS-01
2018/11/08 10:14:34 [INFO] [mdbraber.net] Checking DNS record propagation using [ns0.transip.nl:53]
2018/11/08 10:14:34 [INFO] Wait [timeout: 10m0s, interval: 10s]
2018/11/08 10:14:40 [INFO] [mdbraber.net] The server validated our request
2018/11/08 10:14:42 [INFO] [mdbraber.net, *.mdbraber.net] acme: Validations succeeded; requesting certificates
2018/11/08 10:14:44 [INFO] [mdbraber.net] Server responded with a certificate.

[wdoekes]

a centralized server that can fulfill ACME challenges for other devices on the network, provided they meet certain authorization conditions

You might wat to take a look at our changes mentioned in #158, where you can use "readable" CNAMEs and a single set of credentials to manage multiple TXT records.

[mdbraber]

Thanks @wdoekes looks interesting. For anyone interested in a different solution to the same problem, check out https://github.com/mdbraber/acmeproxy for an updated version of acmeproxy that's basically relaying requests from other hosts inside a network (having credential stored in a single location) and can use username/password or IP based authentication.