Hostname provided by SNI and in the Host header doesn't match in federation requests

[dani]

Running synapse since several months behind a proxypass, including the federation port. I'm using apache httpd as reverse proxy. I have to servers (same config). As soon as I update one of those to 0.33.3 (or now 0.33.4), federation don't work anymore. The homeserver 0.33.4 can't send anything to the one still at 0.33.2

I can see a lot of resp 400 from synapse to the rev proxy each time a synapse >= 0.33.3 talks to me (0.33.2). Upgrading this synapse instance to 0.33.4 doesn't change anything. So it's not an incompatibility between >= 0.33.3 and older, but something which broke federation behind a proxypass.

What info would you need to debug this ?

[ara4n]

@dani: we need to see the logs for the requests which are failing - probably a large block of logs at INFO log level on both the source & destination server would help.

[dani]

In fact, it seems that synapse >= 0.33.3 who sends put requests on the federation endpoints are not dispatched correctly to synapse backend, but are being answered by a 400 error message by the frontal httpd directly. It might have something to do with the recent SNI change brought in 0.33.3, which somehow doesn't play well with my proxypass setup (request treated by wrong vhost ?). I'll gather the info and post it here ASAP

[dani]

Ok, looks like I got it. Requests from other synapse servers (but only those >= 0.33.3) are not valid, because the SNI doesn't match the Host header. Eg, my matrix server is lapiole.org. My ID is @dani:lapiole.org but the DNS name of the box handling it is matrix.lapiole.org (published through the SRV DNS entry), and here are the result:

[Tue Sep 18 08:26:15.954501 2018] [ssl:error] [pid 31587] AH02032: Hostname matrix.lapiole.org provided via SNI and hostname lapiole.org provided via HTTP are different

And the corresponding access_log line

matrix.lapiole.org 52.211.y.x - - [18/Sep/2018:08:26:15 +0200] "PUT /_matrix/federation/v1/send/1537216206283/ HTTP/1.1" 400 - "-" "Synapse/0.33.4" scheme="-"

There was no SNI before that, so the request was being treated (through a dedicated vhost listening on a specific port), but now that there's SNI, and that it doesn't match Host, httpd automatically responds with a 400 bad request. It doesn't even reach synapse backend.

[dani]

Not sure how nginx (or other proxies) are handling this, but those requests are invalid. And it looks like there's no way to ignore this and serve them anyway with Apache httpd. So indeed, 0.33.3 broke reverse-proxyfied federation

[dani]

The problem has already been rised in issue #2525 which has been closed: "the conclusion was that the current behaviour is actually correct". I do not agree with the conclusion. Requests with a mismatch between SNI and Host headers are invalid (not only Apache httpd rejects them, cloudflare also does for example, see https://support.cloudflare.com/hc/en-us/articles/204872628-Error-1013-HTTP-hostname-and-TLS-SNI-hostname-mismatch)

[richvdh]

Indeed there should not be a mismatch between SNI and Host. (The topic of #2525 was what they should be set to.)

I'll have a look at this later.

[richvdh]

@dani: thanks for the excellent detective work on this. #3907 should fix the problem for future synapse releases.