Agent/Server cert mismatch, but why?

[rsteckler]

My agents won't connect to the server and the server logs show a cert has mismatch.

The weird thing is the server says it's loading the cert with hash cf96a9f15a and the clients are connecting with the same hash, but getting rejected. The server is saying it wants one of two other hash options. What are those from?

MeshCentral HTTP server running on port 443.
Loaded web certificate from "https://mesh.xxx.casa", host: "mesh.xxx.casa"
  SHA384 cert hash: cf96a9f15a019cfd541fc22927b707904589db470c9fd3c5f0ea398d5b50ccab163a3d88a7356b1668628bb4561c07ca
  SHA384 key hash: 23e49176da3e14fb8fc4719ad31fda78763bec15d48139f16e4816c29a405895a57ff8a12a80cd53055c4d371b22277a
Agent bad web cert hash (Agent:cf96a9f15a != Server:2b8b9ab5a8 or 95b4b79469), holding connection (192.168.2.1:38492).
Agent reported web cert hash:cf96a9f15a019cfd541fc22927b707904589db470c9fd3c5f0ea398d5b50ccab163a3d88a7356b1668628bb4561c07ca.

It seems odd the server would say, "Loading cert hash cf96a9f15a. Your hash is bad because it's cf96a9f15a"

[rsteckler]

It's worth mentioning that the sha384 hashes of the certificate files don't match what the logs are saying:

# sha384sum --tag webserver-cert-private.key
SHA384 (webserver-cert-private.key) = 22d981603d49e96c485a8915085f08ced76e1b36354e64a20de03d880b16aac6598bdca4d9f88a765d91d899af7a7197
# sha384sum --tag webserver-cert-public.crt
SHA384 (webserver-cert-public.crt) = 6ed5fe40b295063bd523d937b74beb72a50614f2d4b7b7179f57defc5218932dc52bcdd022b272094cdc526a5c6708ee
# sha384sum --tag agentserver-cert-private.key
SHA384 (agentserver-cert-private.key) = 17e8dd05f765775f747f66a8f51f661956ce6c66db3999ff7eb354882365a4a5ed1cc9a13d0b45fd35089d835d2b7113
# sha384sum --tag agentserver-cert-public.crt
SHA384 (agentserver-cert-public.crt) = cc56d916a5e96e014714be25421abb674640243fb9921b6503b5b99abf7c03c021bc4444a9789963d9c51f75e2daca66

None of those are the hashes mentioned by the server logs.

[rsteckler]

config.json. Note that I have certUrl setup.

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "plugins":{"enabled": false},
    "_mongoDb": null,
    "cert": "mesh.xxx.casa",
    "WANonly": false,
    "LANonly": false,
    "sessionKey": "redacted",
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "AgentPong": 50,
    "BrowserPong": 50,
    "TLSOffload": true,
    "trustedProxy": true,
    "SelfUpdate": false,
    "AllowFraming": false,
    "WebRTC": true
  },
  "domains": {
    "mesh.xxx.casa": {
      "_title": "Home",
      "_title2": "xxx",
      "minify": true,
      "NewAccounts": false,
      "localSessionRecording": false,
      "_userNameIsEmail": true,
      "certUrl": "https://mesh.xxx.casa"
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
    "_email": "xxx@gmail.com",
    "_names": "mesh.xxx.casa",
    "production": false
  }
}

I've also verified the cert is the same from where MC pulls it and from what the clients see:
From the MC server (inside the docker container):

* Server certificate:
*  subject: CN=mesh.xxx.casa
*  start date: Jan  4 05:06:25 2025 GMT
*  expire date: Apr  4 05:06:24 2025 GMT
*  subjectAltName: host "mesh.xxx.casa" matched cert's "mesh.xxx.casa"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.

From a client (agent) machine:

* Server certificate:
*  subject: CN=mesh.xxx.casa
*  start date: Jan  4 05:06:25 2025 GMT
*  expire date: Apr  4 05:06:24 2025 GMT
*  subjectAltName: host "mesh.xxx.casa" matched cert's "mesh.xxx.casa"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.

[rsteckler]

Ok - I resolved this, but I still don't understand why it wasn't working. To fix it, I visited my domain from chrome and exported the public certificate to a file. I copied that into the mesh central container's webserver-cert-public.crt file. That caused everything to match up.

My open question is why when MC starts and pulls the cert itself, it stores something else in the web.crt file.

[si458]

sorry for delay, didnt see this, webserver-cert-public.crt is a self-signed certificate that gets generated on start up
then if you use certurl it will pull that certificate down and use that
but im guessing since the timestamps on your certs are only this morning start date: Jan 4 05:06:25 2025 GMT
meshcentral might not of refreshed its certificates correctly,
normally a stop of meshcentral and wait like 30 seconds then start meshcentral again should let it fetch the certificate again
but whats also weird is im sure if it knows ur using certurl and the hash doesnt match it will fetch the certificate again and then use that hash instead, so it might be matching the wrong hash values when it refreshes the certurl ?

[rsteckler]

Please don't apologize for a 1 hour delay of support on an open source project on a Saturday :)

The reason the certs are from this morning is that I deleted them to make sure MC did a fresh pull.
What's odd is that when I delete the web cert and restart the container, it pulls down new certificates just as the log indicates, from the correct certUrl. But what it stores in the web.crt file is different than the certificate I get when I curl from that very same container instance to the same certUrl.

I suppose that makes no sense, so I must be wrong somewhere, but here's what I see:
Exporting the certificate from chrome on my desktop:

-----BEGIN CERTIFICATE-----
MIIE9TCCA92gAwIBAgISA420KkIS2KYc6FM1GuBa+DtSMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjUwMTA0MDUwNjI1WhcNMjUwNDA0MDUwNjI0WjAdMRswGQYDVQQD
ExJtZXNoLmVsZnVlcnRlLmNhc2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCnM2h05gYjImXFEMR6aLEojyU7dAVnTBkpYCj7ztZiex+0J2Amt89oTvMz
gOOSQk0KPrzx0B+NQY1wh81dyNm7Fiu7VXwwc4cY9GeV4nC2ARpFJupC4pFFJ2Vo
sHva4x2dt/DJvePSPeAN9wAt7HQCxK5lTFP0raSWRimvYN4ABFg+VURYxWJdVv1a
w9Y5QfqJeF8zkG3QJlcMVTJbjoX0aXmet1xc7e/jRJcsyO5M1Hw8zQUJJ0oe9UY6
8ew9wASPhNr7MVXgYuhy4QRSZQcfrEhOAkmxM6+n2OZMbCw0TXgMPU70RaEwRTiK
5bDQO8h4WMAe74DulE/4IBUdG7O7AgMBAAGjggIXMIICEzAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFMSikraG7LkMnl61sKHqrAoBJl2kMB8GA1UdIwQYMBaAFMXPRqTq
9MPAemyVxC2wXpIvJuO5MFcGCCsGAQUFBwEBBEswSTAiBggrBgEFBQcwAYYWaHR0
cDovL3IxMS5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYXaHR0cDovL3IxMS5pLmxl
bmNyLm9yZy8wHQYDVR0RBBYwFIISbWVzaC5lbGZ1ZXJ0ZS5jYXNhMBMGA1UdIAQM
MAowCAYGZ4EMAQIBMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAcyAiDwgWivnz
xKaLCrJqmkoA7vV3hYoITQUA1KVCRFkAAAGUL+qpywAABAMARzBFAiBADEEXlUNJ
TSvru44yBIcoKmnzXUwd2DhdK/FquOtUgQIhAPpsX+S8T3yX86w8TJ/+v4eDrcFs
Y+ZeBT6vuBtWU9y4AHcATnWjJ1yaEMM4W2zU3z9S6x3w4I4bjWnAsfpksWKaOd8A
AAGUL+qpzgAABAMASDBGAiEAuiylmYhCfCGaRwLM2jzsdZHnYs9fFonOOl1TC62c
0dsCIQDK/DGUUuYmGS/06znTJuJ4Gdp+442GvURQYzcTEw3XBjANBgkqhkiG9w0B
AQsFAAOCAQEAFkpCgavYAJlbBz+iius0GKtUnBOn7ZGds1zPC3gBgyAR3HacRO5g
AJvKi+4iOtetom1hFbU55gN2yuu6sQBKxD996pdFxA0nz8C5z3wtkhpdOcjHmgEm
m1x94yHkplxD9EOTjzRupNrBS4KasxXIPOAMmLYfCF87A6OPBrkkghRcn8drCQ5p
QcQ7zzj28V66fBvMNrvMhzvjsvDtgkdeyK3YlLd0QAe/TtQfz9pVof6abHt5WJMQ
wdjLryHUCQsJOSKo+y+UI694BHKr5Hv7RVvJ9z76qJfd8wqRPHYJOqGkxlqchvGp
gqjSTJ2B5ux6EVMUNr9HqC8lOYNzMQf79Q==
-----END CERTIFICATE-----

Pulling the certificate with openssl from inside the MC server docker container (exactly where MC should pull it from):

Certificate chain
 0 s:CN=mesh.xxx.casa
   i:C=US, O=Let's Encrypt, CN=R11
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan  4 05:06:25 2025 GMT; NotAfter: Apr  4 05:06:24 2025 GMT
-----BEGIN CERTIFICATE-----
MIIE9TCCA92gAwIBAgISA420KkIS2KYc6FM1GuBa+DtSMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjUwMTA0MDUwNjI1WhcNMjUwNDA0MDUwNjI0WjAdMRswGQYDVQQD
ExJtZXNoLmVsZnVlcnRlLmNhc2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCnM2h05gYjImXFEMR6aLEojyU7dAVnTBkpYCj7ztZiex+0J2Amt89oTvMz
gOOSQk0KPrzx0B+NQY1wh81dyNm7Fiu7VXwwc4cY9GeV4nC2ARpFJupC4pFFJ2Vo
sHva4x2dt/DJvePSPeAN9wAt7HQCxK5lTFP0raSWRimvYN4ABFg+VURYxWJdVv1a
w9Y5QfqJeF8zkG3QJlcMVTJbjoX0aXmet1xc7e/jRJcsyO5M1Hw8zQUJJ0oe9UY6
8ew9wASPhNr7MVXgYuhy4QRSZQcfrEhOAkmxM6+n2OZMbCw0TXgMPU70RaEwRTiK
5bDQO8h4WMAe74DulE/4IBUdG7O7AgMBAAGjggIXMIICEzAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFMSikraG7LkMnl61sKHqrAoBJl2kMB8GA1UdIwQYMBaAFMXPRqTq
9MPAemyVxC2wXpIvJuO5MFcGCCsGAQUFBwEBBEswSTAiBggrBgEFBQcwAYYWaHR0
cDovL3IxMS5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYXaHR0cDovL3IxMS5pLmxl
bmNyLm9yZy8wHQYDVR0RBBYwFIISbWVzaC5lbGZ1ZXJ0ZS5jYXNhMBMGA1UdIAQM
MAowCAYGZ4EMAQIBMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAcyAiDwgWivnz
xKaLCrJqmkoA7vV3hYoITQUA1KVCRFkAAAGUL+qpywAABAMARzBFAiBADEEXlUNJ
TSvru44yBIcoKmnzXUwd2DhdK/FquOtUgQIhAPpsX+S8T3yX86w8TJ/+v4eDrcFs
Y+ZeBT6vuBtWU9y4AHcATnWjJ1yaEMM4W2zU3z9S6x3w4I4bjWnAsfpksWKaOd8A
AAGUL+qpzgAABAMASDBGAiEAuiylmYhCfCGaRwLM2jzsdZHnYs9fFonOOl1TC62c
0dsCIQDK/DGUUuYmGS/06znTJuJ4Gdp+442GvURQYzcTEw3XBjANBgkqhkiG9w0B
AQsFAAOCAQEAFkpCgavYAJlbBz+iius0GKtUnBOn7ZGds1zPC3gBgyAR3HacRO5g
AJvKi+4iOtetom1hFbU55gN2yuu6sQBKxD996pdFxA0nz8C5z3wtkhpdOcjHmgEm
m1x94yHkplxD9EOTjzRupNrBS4KasxXIPOAMmLYfCF87A6OPBrkkghRcn8drCQ5p
QcQ7zzj28V66fBvMNrvMhzvjsvDtgkdeyK3YlLd0QAe/TtQfz9pVof6abHt5WJMQ
wdjLryHUCQsJOSKo+y+UI694BHKr5Hv7RVvJ9z76qJfd8wqRPHYJOqGkxlqchvGp
gqjSTJ2B5ux6EVMUNr9HqC8lOYNzMQf79Q==
-----END CERTIFICATE-----

Deleting the certificate and letting MC pull it the next time it starts up:

Log file:

Loaded web certificate from "https://mesh.xxx.casa", host: "mesh.xxx.casa"
  SHA384 cert hash: cf96a9f15a019cfd541fc22927b707904589db470c9fd3c5f0ea398d5b50ccab163a3d88a7356b1668628bb4561c07ca
  SHA384 key hash: 23e49176da3e14fb8fc4719ad31fda78763bec15d48139f16e4816c29a405895a57ff8a12a80cd53055c4d371b22277a

Resulting file:

 cat webserver-cert-public.crt
-----BEGIN CERTIFICATE-----
MIIEkDCCAvigAwIBAgIFFJEzOYcwDQYJKoZIhvcNAQEMBQAwRTEfMB0GA1UEAxMW
TWVzaENlbnRyYWxSb290LTBiMDhlMjEQMA4GA1UEChMHdW5rbm93bjEQMA4GA1UE
BhMHdW5rbm93bjAeFw0xODAxMDEwMDAwMDBaFw00OTEyMzEwMDAwMDBaMB0xGzAZ
BgNVBAMTEm1lc2guZWxmdWVydGUuY2FzYTCCAaIwDQYJKoZIhvcNAQEBBQADggGP
ADCCAYoCggGBAL9iWoeOTEe8dVTzf6VwPzJwPB74a+lxq0HKj6J66V45JKnzsNT2
SF6NX0tO4EoBcM2bhjtfuSChDa0vWUdd8s9xetnmhh9ValuP1HtsUb18VzlDW+Hr
pK1Fh1LF4Q9QDQAvn9ByG6bpGDocFnaEhkNTkf1UydCopNaNEaq5e5VGOE+jB24b
OgPfHAWnMo0IqPFb5JgGNtMIN/lTdAJIbWTgIGRY8hzQ8Z7s7UVd9zYYolKH1Sg6
iNOb+fG/HkUaaw4vo+Lic3XVVuzYiX+jJgCSKvvj32zYlA5xtnAUrC+icUSb0LVB
gwnr8p2vg/ojQJqmAIbTUoOkHVNNn7kCWl2og2wu1AomJddRe9VQ8DET0loxwqVg
bSY4hHrw/J2GdGHfo3ItbzItIdyTXtaiXx13iQMlNMtFe/nm95aTn9kcj6z0pL7t
Vbw5XPsqUl7MeIYw8VQCYrcm00DBEN5nF9ZdDYKwpcuyaYG8jnZd+9wK2MFVWuo+
Zf0qJRXFqvyyTQIDAQABo4GuMIGrMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMG
A1UdJQQMMAoGCCsGAQUFBwMBMB0GA1UdDgQWBBRBn05BT44yCgIq4caPpRVulln/
1jBdBgNVHREEVjBUghJtZXNoLmVsZnVlcnRlLmNhc2GGGmh0dHA6Ly9tZXNoLmVs
ZnVlcnRlLmNhc2Evgglsb2NhbGhvc3SGEWh0dHA6Ly9sb2NhbGhvc3QvhwR/AAAB
MA0GCSqGSIb3DQEBDAUAA4IBgQBF+DZrqym4nSMKE5Uzv3euL5vT+TATHveiMiif
EjFXZJQVbEc0skpKbOIFLc8lg1S9Y3A8isVxZN6ZNG3Rrk6wfVMtatRCDezXG12Q
FxLP83CaWA0MM9EqaFFn2c9GqIPD8+nrSmixYZt4uedgCn3pIwPdiFamt+gTbl9x
MNCPv8MhUU7nbdvILk8Og7qeHu3LUgxWFAC0lnxi7+FaHyEvb2EtbgJ4ctU34VTz
LDQtxGCafXV6VqvzRIIgM8oq1QnWa3fdqqzMUj766st20Tle7NioyZBZ0SZaCpyc
o+eOp1oD9uvjnAsi3oeT/EfhzrSBK+WkfyNqlczTMmyCEZngtBPpw7YtXP3zfBh3
rIAFy6Y490pnmbdP/a3I2z2byKi7/kv9/81nxETJSJyEij9ai1AtdrJOeJNamWm4
MmWg1vZ8gzeurQUY+LJYc1Aiuj/fn0TW30NKVa7YcqhLk2Li4BS5T2R9xRIyjwwe
8sNRHcNC6uMxjdTZKgOkij/gyko=
-----END CERTIFICATE-----

The first two match, but it sure looks like MC is self-generating it's own cert each time it starts up, rather than downloading one from certurl. I triple-checked my config and it sure looks like certURL is in the correct place. I don't believe it's case-sensitive. And the url is definitely correct. Plus, the log files say it's pulling the certificate correctly.

In any case, I'm good until my certificate renews on letsencrypt, at which point I'll need to manually replace the certificate file on MC. If this ends up being something you want to investigate, I'm happy to do whatever I can to help debug it.

[si458]

DONT DELETE webserver-cert-public.crt this is a self-signed cert and is used IF it cant get an SSL cert for anything!
so if the file isnt there, it will just regenerate it, thats the idea!
the cert thats pulled from certurl isnt stored anywhere, its stored in memory,
then as i explained, IF your agent provides a different hash for some reason, meshcentral SHOULD just pull the cert again from certurl and just verify if its changed or not, and if it has changed, then use that cert hash instead

[rsteckler]

I see. It sounds like my installation isn't storing the cert in memory correctly, or at least isn't using it when the agents connect. I think it goes back to the original log file I posted:

Loaded web certificate from "https://mesh.xxx.casa", host: "mesh.xxx.casa"
  SHA384 cert hash: cf96a9f15a019cfd541fc22927b707904589db470c9fd3c5f0ea398d5b50ccab163a3d88a7356b1668628bb4561c07ca
  SHA384 key hash: 23e49176da3e14fb8fc4719ad31fda78763bec15d48139f16e4816c29a405895a57ff8a12a80cd53055c4d371b22277a
Agent bad web cert hash (Agent:cf96a9f15a != Server:2b8b9ab5a8 or 95b4b79469), holding connection (192.168.2.1:38492).
Agent reported web cert hash:cf96a9f15a019cfd541fc22927b707904589db470c9fd3c5f0ea398d5b50ccab163a3d88a7356b1668628bb4561c07ca.

It looks like it's loading the correct certificate and identifying it with the hash cf96a9f15a
But when a client tries to connect with that exact same hash, it fails and says it wants 2b8b9ab5a8 (cert) or 95b4b79469 (full stack) instead.

I'll go ahead and close the discussion, as I'm able to move forward by just replacing the self-signed web cert with my own. Hopefully this discussion helps someone else in the future.
Thanks for the help!