SMTP using Let's Encrypt cannot send emails

[cweilguny]

Hi,
I have a new self hosted instance of bitwarden, and configured my mailserver as SMTP server. The mailserver uses a Let's Encrypt certificate for transport encryption. So I set port 587 and globalSettings__mail__smtp__ssl=true in the mailsettings and tried to send the verification mail. But sending fails, bwdata/logs/api/Api/20180929.txt shows this:

2018-09-29 18:40:45.414 +00:00 [Error] Mail send failed.
System.Net.Mail.SmtpException: Failure sending mail. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.TlsStreamAuthenticateCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
   at System.Net.Mail.SmtpTransport.EndGetConnection(IAsyncResult result)
   at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
   --- End of inner exception stack trace ---

On my mailserver the mail.log says:

Sep 29 20:40:45 mail postfix/submission/smtpd[7786]: connect from bitwarden.example.com[1.2.3.4]
Sep 29 20:40:45 mail postfix/submission/smtpd[7786]: lost connection after STARTTLS from bitwarden.example.com[1.2.3.4]
Sep 29 20:40:45 mail postfix/submission/smtpd[7786]: disconnect from bitwarden.example.com[1.2.3.4] ehlo=1 starttls=1 commands=2

As the container has no timezone set, it's "kind of" 2 hours off (see the times of the two logs). So I tried to set the timezone in the bitwarden-api container to Europe/Vienna, to exclude the time for certificate issues. That didn't change anything.

The mail settings are as correct as I can set them. Port 587, SSL, username and password do work too (tried in my webmail client). The settings work in different clients, there I also don't have to accept invalid certificates. I also checked if the certificate is valid with nmap -p 587 --script ssl-cert my-mail-server.example.com and it's valid for the domain I used. Googling lost connection after STARTTLS gives some hints, that the sending server didn't accept the certificate, well, just like the bitwarden log says.

So, here is the point, where I don't know how to research further. Any hints? Is it probably a known issue (didn't find anything here but I can't imagine that I'm the only bitwarden self-hoster that uses a mailserver with a let's encrypt certificate)?

[kspearrin]

Sounds like you need to provide your the LE trust chain cert to the Bitwarden containers, so that they properly trust it. Place these certificates in ./bwdata/cs-certificates and they will be loaded into the trusted store on each container.

[cweilguny]

Thanks! I had some time offline, so it took a couple of days I could try that. I copied the ISRG Root X1 and the two active intermediate authorities to bwdata/cs-certificates, stopped the containers, and restarted them. Nothing changed. Do you have another idea?

Side note: I'm gonna use Bitwarden only for me and another person, actually I don't know if I really need to send emails, cause I don't know Bitwarden yet. Are there any important emailing functionalities?

[qspweissen]

Same problem here with my mail server and a Let's Encrypt certificate.

The remote certificate is invalid according to the validation procedure.

[cweilguny]

@qspweissen: I switched to a temporary GMail account. I only needed the email to verify the two user accounts, don't know why I needed to verify them, if I remember it right, accounts only can join organizations if they are verified. After that, I didn't need emails anymore. So if you only have a few users, maybe just using GMail or another free service for this is a fast solution for you too.

But I would also prefer this to be fixed, or at least a more detailed message in the logs, why the certificate can't be validated. I have no other troubles with my mailserver, so I'm not sure if it's a faulty webserver or a faulty certifiacte validation.

[egyptianbman]

I'm having a similar problem with the e-mail provider, mailjet.com. Their ssl cert is by digicert. I added all the ca certs from digicert into the ca-certificates directory and restarted but it still wouldn't work, I got the same error.

[kspearrin]

Can you check the smtp address here to make sure there are no issues with the ssl cert? https://www.digicert.com/help/

[egyptianbman]

Yes, I tested in-v3.mailjet.com as well as in-v3.mailjet.com:587 and both came back as valid.

[qspweissen]

Same for me. Everthing is valid

[cweilguny]

Six green checkmarks for me too.

[kspearrin]

I am not sure why the connection is not being trusted then. Does anyone have a set of test SMTP credentials to a server with this problem that I can use to debug this with? You can PM me on the dev channel here https://gitter.im/bitwarden/Lobby

[egyptianbman]

@kspearrin you can create an account with mailjet.com. They offer 200 e-mails per day for free, which is why I use them for this. I don't anticipate needing to send anywhere near that many e-mails for personal use.

[kspearrin]

@egyptianbman I created a free mailjet account and ran some tests tonight using in-v3.mailjet.com.

• Using Bitwarden running through Docker, I am able to reproduce the same exception:

  System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

• Debugging Bitwarden through my local Windows machine with the same mailjet server, all email sends just fine without issue.

Very weird.

Like you, I thought that this would suggest that the Debian docker container didn't have the DigiCert Root CA for whatever reason. So I bashed into the bitwarden-api container (this is the one that is sending emails) like so:

docker exec -it bitwarden-api /bin/bash

I then did a curl GET to DigiCert's Root CA test page and it works just fine:

# curl https://global-root-ca.chain-demos.digicert.com/

I am a little confused as to what it could be at this point.

[kspearrin]

@egyptianbman I then tried it without SSL, using port 25 and I get another exception:

System.Net.Mail.SmtpException: Failure sending mail. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Connection refused 104.199.96.85:25

[egyptianbman]

I was actually able to get it to work with ssl disabled and used that to get access to the admin, then restarted the service to clear the credentials (since the link was sent insecurely) and re-enabled ssl.

Yeah, I'm not sure what the deal is. This is a C# application so could it be that mono needs something to be able to access the root certs?

[kspearrin]

I opened an issue here: dotnet/aspnetcore#6152

[egyptianbman]

Thank you!

[egyptianbman]

Any luck on this? I still get the following when Bitwarden attempts to send an email:

[egyptianbman]

I took another look at mailjet to see if anything had changed since quite some time has passed and I noticed a note that if port 587 doesn't work, to try 465. I did another trial with port 25, 587 and finally -- 465 -- which worked! This means my issue is resolved.

My working configuration:

globalSettings__mail__smtp__host=in-v3.mailjet.com
globalSettings__mail__smtp__port=465
globalSettings__mail__smtp__ssl=true

[Skyrant]

local smtp server with let's encrypt certificate and i am getting the same errors.

Which files do i need to copy to ca-certificates or cs-certificates and what are the filenames the app is looking for?

Is there any other workaround?

[bitwarden-bot]

Hi @cweilguny,
We're cleaning up our repositories in preparation for a major reorganization. Issues from last year will be marked as stale and closed after two weeks. If you still need help, comment to let us know and we'll look into it.
Thanks!