DNSMadeEasy Certificate renewal / Creation failing

[mrTimGilroy]

I am having some issues when issuing or renewing LE certificates using DNSMadeEasy as the DNS provider. This has previously worked well and been fairly reliable, but right now it's failing almost every time.

There are two failure scenarios:

1. Issuing a certificate when there's no existing acme challenge TXT record present

Orchestrator function 'IssueCertificate' failed: The activity function 'CheckIsReady' failed: "ACME validation status is invalid. Required retry at first.
LastError = {"type":"urn:ietf:params:acme:error:unauthorized","detail":"No TXT record found at _acme-challenge.txxxxn.cxxxxxxxxxx3.com","status":403}". See the function execution logs for additional details.

On the face of it this suggests that the DNS provider is not creating the record, but when I look at the DNSMadeEasy activity log I can see the create request was received and acted on, and when I check the domain TXT records the acme challenge record is present and correct (_acme-challenge.txxxxxn.cxxxxxxxxxx3.com). I can delete this record, wait a minute or two, and then try again - but the same thing happens. The record is created but I still get an error saying no TXT record was found.

2. Issuing a certificate when there IS an existing acme challenge TXT record present
   (In normal operation the acme challenge record should not be permanent and should be deleted once the challenge record has been validated but at the moment the process is leaving TXT records in place because it's failing).

Orchestrator function 'IssueCertificate' failed: The activity function 'CheckIsReady' failed: "ACME validation status is invalid. Required retry at first.
LastError = {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect TXT record "5S9X3uX-GV2AeOE5Uyk4rMT6TBP007YbDThakpRLgh0" found at _acme-challenge.txxxxn.cxxxxxxxxxx3.com","status":403}". See the function execution logs for additional details.

When I check the DNSMadeEasy activity log I can see an initial request to delete the pre-existing TXT record, followed by a request to create a fresh one. If I check the DNS records the new acme challenge record is present and correct so those DNS operations did complete correctly. The error message however is referring to the pre-existing TXT record.

In both scenarios it looks like the "CheckIsReady" code is checking for the acme challenge too quickly after issuing the commands to the DNS provider. Even though we can see the required deletions and creations happening pretty much instantly it seems that the code is not waiting long enough for those operations to complete.

Has anyone seen this behaviour before?

[JohnLBevan]

I'm not familiar with DNSMadeEasy, but is it possible that the zone to which AcmeBot has access to update records is different to the zone used by public DNS (i.e. if you were to query the name servers, Resolve-DnsName 'cxxxxxxxxxx3.com' -Type NS -Server 1.1.1.1 does that return the same NS values as you see listed in this zone? Asking as it could be AcmeBot is correctly updating its records, but DNS is looking elsewhere so not seeing those records.
Unlikely as it sounds like it had worked until recently, but thought it worth suggesting/checking. Hope that's some help / at least eliminates a potential option.

[mrTimGilroy]

John,
I have checked the name server entries as recommended by yourself and they do match. An external query does return the same values as shown in the DNSMadeEasy Control panel.
Regards
Tim

[mrTimGilroy]

I have removed the application and redeployed using the same KeyVault. This also had the advantage of bring the Runtime versions (4.636.0.23246 -- 4.836.0.23246) and Function App up to latest variants. Unfortunately, this has not resolved the issue.
I do not see similar errors returned with CloudFlare, which I have tested with and may consider it to be an issue with DNS provider blocking some LE requests or something of that nature. I have noted discussion of additional queries from other countries.
Whilst performing the investigation, I have struggled to find documentation on redeploying the application, (How to upgrade\ update) and also removal and redeploy.
If anyone has further information or links on the above I would appreciate it.

One of the issues appears to be we have installed the application, it has worked for a period of time, but issues have now appeared and that is when requirements such as "how do I ensure update ?"has been asked.

My process to Redeploy. But upgrade requires a different process I assume.
Note of all Custom Environmental Variables (Namely)
API Key details for the DNS Provider (Example shown DNSMadeEasy):
Acmebot:DnsMadeEasy:ApiKey & Value
Acmebot:DnsMadeEasy:SecretKey & Value

Acmebot:VaultBaseUrl (Critical)

Then remove all resources generated by original deployment of KeyVault-acmebot with exception of KeyVault if you are preserving the current certificates.
Remove RBAC entries within KeyVault for remove Function App
Remove the Identity for enterprise application that was created in Azure AD

Redeploy the application using deployment templates provided.
Register ID with Azure AD
Add RBAC role (Certificate Officer) to Function App
Reinsert API Details for DNS Provider in Environmental Variables
Reinsert Acmebot:VaultBaseUrl in Environmental Variables
Save and Apply settings.

Application will restart and on browsing the UI, all certificate details should be displayed as they were prior to redeployment.

[mrTimGilroy]

The answer to the issue we had been having was mainly down to DNS.
The organisation used wildcard on the DNS zone supplying the certificates.
Any exceptions thrown during the creation of the certificate which left a txt record for a requested subdomain caused the whole DNS resolution process to fail for that subdomain. This then caused issues with the acme protocol on retry of certificate creation process.
The simple workaround was to add A records for all the subdomains to the zone. This resolved the issue with DNSMadeEasy which appears to have a different standard than others (Cloudflare for example)