auto managed domain mistakenly using self-signed

[arpitjindal97]

Expected Behaviour:

I want to use self-signed certificate for arpit-test.msmartpay.in domain only. I want caddy to automatically manage other domains.
When i visit arpit.msmartpay.in, I should be presented with let's encrypt certificate
when I visit arpit-test.msmartpay.in, I should be presented with self-signed certificate

Actual Behaviour:

• Caddy gets certificates from let's encrypt for both domains (why both)
• Caddy is using self-signed for both

Caddyfile:

{
    servers {
        metrics
    }
    auto_https ignore_loaded_certs
}

arpit.msmartpay.in {
    respond "Hello World"
}

arpit-test.msmartpay.in {
    tls /ssl/msmartpay.in/self/domain.crt /ssl/msmartpay.in/self/privkey.pem
    respond "Hello World"
}

domain.crt:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4a:b0:51:03:a6:50:ec:05:d7:78:7d:17:52:c2:ca:cd:55:25:ac:ab
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SL, ST=Western, L=Colombo, OU=ABC, CN=arpit.msmartpay.in
        Validity
            Not Before: Nov 15 21:44:09 2024 GMT
            Not After : Nov 13 21:44:09 2034 GMT
        Subject: C=SL, ST=Western, L=Colombo, OU=ABC, CN=arpit.msmartpay.in
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bc:46:76:77:dd:16:77:75:8e:32:87:36:75:ac:
                    .................
                    b4:17
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:arpit.msmartpay.in, DNS:*.msmartpay.in
            X509v3 Subject Key Identifier:
                E9:51:26:A4:56:31:40:CA:D5:DA:C2:35:92:32:9D:2B:2C:9E:7B:6A
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        3b:ae:10:b6:d9:cd:18:54:cd:0b:97:2b:b2:3d:52:9e:91:9f:
        ..................................
 

[mholt]

ignore_loaded_certs will tell Caddy to ignore loaded certificates when choosing what to manage certificates for, so it doesn't matter that you have loaded a certificate for your site, it will still obtain one for them to manage.

Seems to be a duplicate of #5933.

[arpitjindal97]

the problem is when I visit arpit.msmartpay.in, I'm presented with self-signed certificate

[arpitjindal97]

@mholt Please re-open the issue, Please understand the fully before closing it

[mholt]

@arpitjindal97 Not sure what I am missing here, please enlighten me. This seems to be a duplicate, as I said. The other issue will track this.

[mohammed90]

Caddy gets certificates from let's encrypt for both domains (why both)

Because you used ignore_loaded_certs.

Caddy is using self-signed for both

You haven't presented evidence of this.

the problem is when I visit arpit.msmartpay.in, I'm presented with self-signed certificate

We haven't seen evidence of this.

[arpitjindal97]

Ignore auto_https ignore_loaded_certs parameter for a while, I will talk about it later. First take a look at this

@mholt @mohammed90
I'm using above mentioned Caddyfile

Here is the evidence of this:

$ curl https://arpit.msmartpay.in -v -k                                                                                                         
* Host arpit.msmartpay.in:443 was resolved.
* IPv6: (none)
* IPv4: 77.248.167.27
*   Trying 77.248.167.27:443...
* Connected to arpit.msmartpay.in (77.248.167.27) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=SL; ST=Western; L=Colombo; OU=ABC; CN=arpit.msmartpay.in
*  start date: Nov 15 21:44:09 2024 GMT
*  expire date: Nov 13 21:44:09 2034 GMT
*  issuer: C=SL; ST=Western; L=Colombo; OU=ABC; CN=arpit.msmartpay.in
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://arpit.msmartpay.in/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: arpit.msmartpay.in]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: arpit.msmartpay.in
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< content-type: text/plain; charset=utf-8
< server: Caddy
< content-length: 11
< date: Thu, 21 Nov 2024 16:31:52 GMT
<
* Connection #0 to host arpit.msmartpay.in left intact
Hello World

[mohammed90]

The certificate CN is literally arpit.msmartpay.in. Configuring it under arpit-test doesn't make it valid.

[mholt]

What's the difference between this issue and #5933?

[arpitjindal97]

I have configured the self-signed certificate under arpit-test so I expect caddy to use that only for arpit-test.

for arpit.msmartpay.in, I expect it to use let's encrypt certificate.

It shouldn't matter what the CN of certificates are.

[arpitjindal97]

@mholt My apologies, you are correct it is related to that issue. I didn't have a closer look earlier.

When can we expect a fix?

[mholt]

force_automate should be released in 2.9.

[arpitjindal97]

Even after using 2.9, it doesn't seem to work. wildcard cert is still being picked

{
    servers {
        metrics
    }
}

arpit.msmartpay.in {
    tls force_automate
    respond "Hello World"
}

arpit-test.msmartpay.in {
    tls /ssl/msmartpay.in/self/domain.crt /ssl/msmartpay.in/self/privkey.pem
    respond "Hello World"
}

[polarathene]

Caddy 2.9 works fine.

You can try reproduce with the following example below to troubleshoot what is going wrong for your actual setup :)

compose.yaml:

networks:
  default:
    name: example-net

volumes:
  custom-certs:
    name: example-tls

services:
  reverse-proxy:
    image: caddy:2.9
    depends_on:
      - get-certs
    volumes:
      - custom-certs:/srv/tls
    # Config is embedded for copy/paste of single `compose.yaml` to run example:
    configs:
      - source: caddy-config
        target: /etc/caddy/Caddyfile
    # For this example, DNS lookups from containers on this network
    # will resolve these FQDN to this Caddy container:
    networks:
      default:
        aliases:
          - caddy.example.test
          - wild.example.test
          - example.test
          - sub.example.test
          - also-wild.example.test

  # Since Caddy depends on this service before it starts,
  # compose will first run this service to provision the external certificates:
  get-certs:
    image: smallstep/step-ca
    volumes:
      - custom-certs:/tmp/certs/
    # This service runs as non-root (1000:1000) by default,
    # change to desired UID/GID ownership of certs generated:
    user: root
    # Support for running the custom script below:
    working_dir: /tmp/certs
    entrypoint: /tmp/generate-certs.sh
    configs:
      - source: generate-certs
        target: /tmp/generate-certs.sh
        # Make script executable:
        mode: 500

configs:
  caddy-config:
    content: |
      # Global Settings
      {
        # Optional: For testing purposes (otherwise defaults to a public CA)
        # Have Caddy provision certs locally (self-signed):
        local_certs
      }

      caddy.example.test {
        tls force_automate
        respond "I am using a certificate provisioned by Caddy"
      }

      wild.example.test {
        respond <<HEREDOC
          I am using the external wildcard cert loaded
          from the tls directive of another site block
          HEREDOC
      }

      example.test, sub.example.test, also-wild.example.test {
        tls /srv/tls/example.test/cert.pem /srv/tls/example.test/key.pem
        respond "I am using an externally loaded certificate"
      }

  # NOTE: For `smallstep/step-ca` container to run it's `step` CLI to create a cert:
  generate-certs:
    content: |
      #!/usr/bin/env bash

      mkdir -p ca example.test

      step certificate create 'Smallstep Root CA' ca/cert.pem ca/key.pem \
        --profile root-ca --no-password --insecure --force

      step certificate create 'Smallstep Leaf' example.test/cert.pem example.test/key.pem \
        --san 'sub.example.test' --san '*.example.test' \
        --ca ca/cert.pem --ca-key ca/key.pem \
        --profile leaf --no-password --insecure --force

Verify:

$ docker compose up -d --force-recreate
$ docker run --rm -it --volume example-tls:/srv/tls --network example-net alpine
$ apk add curl step-cli jq

#
## All sites responding:
#

# NOTE: Insecure flag is used due to Caddy's `local_certs` self-signed CA cert missing:
$ curl --insecure https://caddy.example.test
I am using a certificate provisioned by Caddy

$ curl --insecure https://wild.example.test
  I am using the external wildcard cert loaded
  from the tls directive of another site block

$ curl --cacert /srv/tls/ca/cert.pem https://sub.example.test
I am using an externally loaded certificate

#
## Certificates used are the ones expected:
#

$ step certificate inspect --insecure --format json https://caddy.example.test \
  | jq '{ issuer_dn, subject_dn, names }'

{
  "issuer_dn": "CN=Caddy Local Authority - ECC Intermediate",
  "subject_dn": null,
  "names": [
    "caddy.example.test"
  ]
}

$ step certificate inspect --roots /srv/tls/ca --format json https://wild.example.test \
  | jq '{ issuer_dn, subject_dn, names }'

{
  "issuer_dn": "CN=Smallstep Root CA",
  "subject_dn": "CN=Smallstep Leaf",
  "names": [
    "sub.example.test",
    "*.example.test"
  ]
}

$ step certificate inspect --roots /srv/tls/ca --format json https://sub.example.test \
  | jq '{ issuer_dn, subject_dn, names }'

{
  "issuer_dn": "CN=Smallstep Root CA",
  "subject_dn": "CN=Smallstep Leaf",
  "names": [
    "sub.example.test",
    "*.example.test"
  ]
}

# Due to the site-block forcing external via `tls` directive, this will return an invalid cert:
$ step certificate inspect --insecure --format json https://example.test \
  | jq '{ issuer_dn, subject_dn, names }'

{
  "issuer_dn": "CN=Smallstep Root CA",
  "subject_dn": "CN=Smallstep Leaf",
  "names": [
    "sub.example.test",
    "*.example.test"
  ]
}

NOTE: If the site-block with example.test site-address had the tls directive shifted to the wild.example.test site-block instead, then it would have been provisioned with the cert by Caddy's CA instead, while the other two would match the wildcard certificate and load that.

I do not know for sub.example.test if it's just preferring it because the certificate is already an explicit SAN match and the cert is valid (not expired), or if it's because it's also a match to the wildcard SAN. I'm not sure why you'd provision a cert with two SANs like that, but it's what you've shown in your report 🤷‍♂

[arpitjindal97]

The wild card certificate I use has arpit.msmartpay.in as CN. Could that be an issue?

[polarathene]

@arpitjindal97 yes, sorry I missed that 😅

Prior to using SANs for provisioning, the FQDN could be in the leaf certificate CN instead which would also resolve in the names array from the step certificate inspect command above.

That is, yes Caddy would select that externally loaded certificate instead of provisioning a separate one, even if there was no wildcard with that certificate, tls force_automate doesn't appear to force preferring automated provisioning in that scenario (probably a bug).

---

I also verified that if the FQDN is an explicit SAN as well, this would use the external certificate (or any other one provisioned by Caddy), even with tls force_automate. auto_https prefer_wildcard is not required in that scenario as while the certificate may offer a wildcard certificate, the explicit FQDN match will still have it selected. That was unexpected given how I've seen tls force_automate described by @francislavoie thus may not be intentional 🤷‍♂

So for your arpit.msmartpay.in site block to not use the wildcard certificate, you need to ensure the wildcard certificate is not provisioned with that FQDN in the CN or SANs. You can confirm this with my reference example (which uses step certificate create followed by the CN value, and SANs added via --san args).

---

Reference

This is a variant of the earlier reference above, specifically tailored to replicate your FQDN + wildcard scenario.

I self-contained the certs (formally get-certs) service if that's easier to grok and experiment with this way. It switches the image to plain Alpine and installs the necessary packages for cert provisioning + inspection, with the inspection commands put into a shell script that you can call with an FQDN to inspect 👍

networks:
  default:
    name: example-net

volumes:
  custom-certs:
    name: example-tls

services:
  reverse-proxy:
    image: caddy:2.9
    depends_on:
      - certs
    volumes:
      - custom-certs:/srv/tls
    configs:
      - source: caddy-config
        target: /etc/caddy/Caddyfile
    networks:
      default:
        aliases:
          - arpit.msmartpay.in
          - arpit-test.msmartpay.in

  # Provision the externally loaded wildcard cert + provide cert inspection:
  certs:
    image: localhost/certs
    volumes:
      - custom-certs:/srv/tls
    pull_policy: build
    build:
      dockerfile_inline: |
        FROM alpine:3.21
        RUN <<HEREDOC
          apk add curl jq step-cli
          mkdir /srv/tls
        HEREDOC

        # This is a small shell script that you can with: docker compose run --rm certs cert-info arpit.msmartpay.in
        # NOTE: The `$$` is required to escape `$` from the Docker Compose variable interpolation feature.
        COPY --chmod=755 <<"HEREDOC" /usr/local/bin/cert-info
        #! /usr/bin/env sh
        FQDN="$${1}"
        curl -w '\n' --insecure "https://$${FQDN}"
        step certificate inspect --format json --insecure "https://$${FQDN}" | jq '{ issuer_dn, subject_dn, names }'
        HEREDOC

    # Provision a locally signed certificate with a private CA root:
    # This command is provided via the `exec` format instead of `shell`,
    # Thus `ash -c '<string>'` is the equivalent form:
    command:
      - ash
      - -c
      - |
        cd /srv/tls
        mkdir -p ca msmartpay.in

        step certificate create 'Smallstep Root CA' ca/cert.pem ca/key.pem \
          --profile root-ca --no-password --insecure --force

        step certificate create 'Smallstep Leaf' msmartpay.in/cert.pem msmartpay.in/key.pem \
          --san 'arpit.msmartpay.in' --san '*.msmartpay.in' \
          --ca ca/cert.pem --ca-key ca/key.pem \
          --profile leaf --no-password --insecure --force

configs:
  caddy-config:
    content: |
      {
        local_certs
        auto_https prefer_wildcard
      }

      # If the wildcard certificate has this site-address as an explicit CN or SAN,
      # Caddy will use that certificate instead of provisioning a separate certificate:
      arpit.msmartpay.in {
        tls force_automate
        respond "I should be using a certificate provisioned by Caddy"
      }

      arpit-test.msmartpay.in {
        tls /srv/tls/msmartpay.in/cert.pem /srv/tls/msmartpay.in/key.pem
        respond "I am using an externally loaded certificate"
      }

The CN Smallstep Leaf value can be replaced with arpit.msmartpay.in, and you can remove the related --san to run the example again and get the same results.

$ docker compose up -d --force-recreate
$ docker compose run --rm certs cert-info arpit.msmartpay.in

I am using a certificate provisioned by Caddy
{
  "issuer_dn": "CN=Smallstep Root CA",
  "subject_dn": "CN=Smallstep Leaf",
  "names": [
    "arpit.msmartpay.in",
    "*.msmartpay.in"
  ]
}

# If you swap the SAN for the leaf cert CN instead:
$ docker compose up -d --force-recreate
$ docker compose run --rm certs cert-info arpit.msmartpay.in

I am using a certificate provisioned by Caddy
{
  "issuer_dn": "CN=Smallstep Root CA",
  "subject_dn": "CN=arpit.msmartpay.in",
  "names": [
    "arpit.msmartpay.in",
    "*.msmartpay.in"
  ]
}

[arpitjindal97]

So for your arpit.msmartpay.in site block to not use the wildcard certificate, you need to ensure the wildcard certificate is not provisioned with that FQDN in the CN or SANs

I can't really change wildcard certificate in my production because the impact is bigger. When can we expect a fix for this?

[polarathene]

When can we expect a fix for this?

I'm not a developer for Caddy, I just helped you to properly identify and reproduce the problem.

If you do not get a response from a maintainer by like Monday, maybe it's because the issue was close as "Not Planned" and you'd need to open a new issue. If so, just link to my comment above which details what appears to be a bug.

---

Your external certificate has a 10 year expiry and given the CN is clearly self-signed? So I'm not sure why the impact of correcting your certificate is a blocker sorry? It'd be faster to resolve that then wait on a fix to be formally released.

• Provision LetsEncrypt for arpit.msmartpay.in with tls force_automate in it's site block.
• Provision your own wildcard certificate for any other *.msmartpay.in subdomain like you wanted for arpit-test. Caddy just needs one site block to have that wildcard loaded and auto_https prefer_wildcard will default to using that for anything else that qualifies.
• Provision a separate certificate with your private CA for just arpit.msmartpay.in if you need one that is not provisioned via LetsEncrypt under some other scenario and load that via tls explicitly.

Provisioning a certificate locally without LetsEncrypt is quite simple, as shown above.

[mholt]

The wild card certificate I use has arpit.msmartpay.in as CN. Could that be an issue?

Yes. Caddy sees it as a certificate that already serves that domain.