federation issue: "Unauthorized" trying to join foreign room #1142

steckerhalter · 2016-09-25T14:04:08Z

I've setup a homeserver (using example.com because of privacy here) matrix.example.com and set an SRV record:

dig srv _matrix._tcp.example.com
;; ANSWER SECTION:
_matrix._tcp.example.com. 299 IN    SRV 10 0 443 matrix.example.com.

In addition I have proxied via Apache to port 8008 (with letsencrypt ssl certs). I can use the integrated webclient and also other clients (like Riot on android) with my homeserver.

But when I try to join a room on matrix.org like #bible:matrix.org I get Unauthorized (M_UNKNOWN) and the log says:

2016-09-25 15:53:51,497 - synapse.http.outbound - 122 - INFO - GET-7656- {GET-O-7} [matrix.org] Sending request: GET matrix://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23bible%3Amatrix.org
2016-09-25 15:53:51,504 - twisted - 154 - INFO - - DNSDatagramProtocol starting on 4690
2016-09-25 15:53:51,507 - twisted - 154 - INFO - - Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0xb405764c>
2016-09-25 15:53:51,540 - twisted - 154 - INFO - GET-7656- DNSDatagramProtocol starting on 14191
2016-09-25 15:53:51,544 - twisted - 154 - INFO - GET-7656- Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0xb554b7cc>
2016-09-25 15:53:51,551 - twisted - 154 - INFO - GET-7656- (UDP Port 4690 Closed)
2016-09-25 15:53:51,556 - twisted - 154 - INFO - GET-7656- Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0xb405764c>
2016-09-25 15:53:51,561 - synapse.http.endpoint - 184 - INFO - GET-7656- Connecting to 83.166.64.33:8448
2016-09-25 15:53:51,564 - twisted - 154 - INFO - GET-7656- Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0xb554912c>
2016-09-25 15:53:51,568 - twisted - 154 - INFO - GET-7656- (UDP Port 14191 Closed)
2016-09-25 15:53:51,571 - twisted - 154 - INFO - GET-7656- Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0xb554b7cc>
2016-09-25 15:53:52,191 - synapse.handlers.presence - 327 - INFO - GET-7656- Handling presence timeouts
2016-09-25 15:53:52,285 - synapse.storage.TIME - 208 - INFO - GET-7656- Total database time: 0.000% {who_forgot(0): 0.000%, user_get_threepids(0): 0.000%, update_pusher_last_stream_ordering_and_success(0): 0.000%} {}
2016-09-25 15:53:53,646 - synapse.http.outbound - 207 - INFO - GET-7656- {GET-O-7} [matrix.org] Result: 401 Unauthorized
2016-09-25 15:53:53,653 - root - 180 - WARNING - GET-7656- Error retrieving alias
2016-09-25 15:53:53,659 - synapse.http.server - 126 - ERROR - GET-7656- 401: Unauthorized
Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 116, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1183, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 255, in _async_render
    callback_return = yield callback(request, **kwargs)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1183, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/rest/client/v1/directory.py", line 48, in on_GET
    res = yield dir_handler.get_association(room_alias)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1183, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/directory.py", line 177, in get_association
    retry_on_dns_fail=False,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1183, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/transport/client.py", line 185, in make_query
    timeout=10000,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1183, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 398, in get_json
    timeout=timeout,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1185, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 217, in _create_request
    response.code, response.phrase, body
HttpResponseException: 401: Unauthorized

I haven't set any email address. Could that be a problem? How should I continue to analyze this?

The text was updated successfully, but these errors were encountered:

steckerhalter · 2016-09-26T12:13:56Z

I tried several things and couldn't make it work so now I reverted to expose port 8448 and using synapse directly and removed the SRV record. Now I can join rooms on other servers.

icaruseffect · 2016-09-28T13:02:12Z

I encountered the same problem.
In contrast to @steckerhalter i cannot choose my ports freely, because i'm using a shared host.
I tried exactly the same steps (including srv to port 443) and failed with the same result.

Is it possible to configure federation for ports different than 8448?

richvdh · 2016-09-28T16:14:45Z

Is it possible to configure federation for ports different than 8448?

Yes, this is exactly what the SRV record is for.

Remember it may take a few hours for a changed DNS record to propagate. If you can't get it to work, we're better able to provide support in https://matrix.to/#/#matrix:matrix.org than in github issues.

donpdonp · 2016-10-15T01:51:03Z

I have this exact problem! I've been in matrix-dev multiple times but the fix is still a mystery. Can this ticket be re-opened? The configuration of a reverse-proxy and no ports beyond https is the most desirable setup.

After seeing the 401 in the logs, I have verified that matrix.org is requesting the signing key from my homeserver and its being delivered (status 200). Then matrix.org rejects the federation request (401) and I dont know why.

richvdh · 2016-10-16T11:09:36Z

This ticket amounts to "federation isn't working for me", which isn't a useful thing to track as an issue, because there are a number of things which could cause it - mostly related to configuration. So no, sorry.

richvdh · 2016-10-16T11:11:17Z

We're currently working on a tool which will check that servers are set up correctly for federation, which might help.

leonerd · 2016-10-16T12:43:47Z

We're currently working on a tool which will check that servers are set up correctly for federation, which might help.

For now at least, you can try the commandline tool:

$ cpan App::MatrixTool
$ matrixtool server-key your.server.name.here

If that can see the server keys correctly then federation is probably working. If not, follow up on what failed.

Try also whether matrix.org can see it

$ matrixtool notary your.server.name.here matrix.org

simonszu · 2016-11-23T13:03:02Z

Sorry to post here again, but did you resolve this, @icaruseffect ?
I am struggling with this issue for days, and this blog post which is the only documentation for this is somewhat outdated. I am at a state where matrixtool produces this output:

$ matrixtool server-key matrix.simonszu.de
[INFO] Connected to 5.189.143.28:443
[FAIL] TLS fingerprint does not match any listed
[OK] Verified using origin=matrix.simonszu.de key_id=ed25519:a_LiWb
v2 keys from matrix.simonszu.de:

Key id ed25519:a_LiWb
base64::VF2Cxq3wVEe8NIplwnHK+yKIhdBkgBmzqUfT1k0aMgg

I am sure that the handshake fails due to the TLS fingerprint mismatch, but i ran out of ideas to fix this. The chat room is also no good place to seek help since it is crowded, and need-help-messages disappear quite fast :(

donpdonp · 2016-11-25T03:34:13Z

@leonerd thx for the assistance, the output from matrixtool is

[INFO] Connected to 64.62.134.149:443
[FAIL] TLS fingerprint does not match any listed
[OK] Verified using origin=donp.org key_id=ed25519:a_oekW
v2 keys from donp.org:

Key id ed25519:a_oekW
  base64::azmxIZEG6OR5QQvqMFtieK5yNPSVQ2EjEgwqUF9NNTs

Whats the connection between the TLS cert delivered during an HTTPS connection and the tls related homeserver.yaml settings:

tls_certificate_path: "/home/donp/matrix-org/synapse/donp.org.tls.crt"
tls_private_key_path: "/home/donp/matrix-org/synapse/donp.org.tls.key"
tls_dh_params_path: "/home/donp/matrix-org/synapse/donp.org.tls.dh"
no_tls: True

for instance I have a reverse proxy answering the HTTPS port with a letsencrypt key that is different than the generated-during-install TLS key that is specified in the yaml snipped above. With TLS being turned off in the config, its unclear if the TLS key is used/important.

icaruseffect · 2016-11-25T16:01:12Z

Hi @simonszu, i sadly cannot help you further, because i deactivated my server because of job related time constraints.
I guess the reason for the failure of federation in my setup were cause by using a reverse proxy, not using the initially generated keys. They must be hard coded somewhere after initialization.
But sadly i cannot further contribute to this problem because of i don't use matrix any longer in production.

simonszu · 2016-11-26T14:25:34Z

@icaruseffect That's no problem, i can fully understand this. ;)
I have made a question on serverfault in general, and it seems that @donpdonp is at the same work in progress as i am, so i will check this issue from time to time and see if anything new pops up.

ara4n · 2016-11-27T12:40:12Z

The problems here are just failure to configure federation correctly - it's not a bug per se. I think we could be supporting it more productively though - eg by publishing a better guide or troubleshooting mechanism for it. Until that point, please come to #matrix:matrix.org and don't leave until someone has helped you debug what's gone wrong.

imazes · 2016-12-21T10:49:42Z

ye ,i gt the same problem.srv looks like useless

donpdonp · 2017-01-14T19:47:24Z

just to follow up on this ticket, I've picked apart the problem and want to leave the solution here for the other ticket participants with the same problem.

the root of the problem is python twisted does not support SNI (#1491 is waiting on the fix), so when matrix.org tries to connect to my homeserver on 443, nginx returns the wrong(default) SSL cert which doesnt match what matrix.org was expecting.

another key piece of info is the SRV record is only to adjust the federation port (8448). The default client port is the usual https port. So its enough to remove the SRV record and simply open port 8448 directly to the python script and let the script use its generated self-signed certs.

Kakise · 2017-02-12T16:13:01Z

Hello,
I have the exact same problem.

Proxying matrix server to kakise.xyz:443 via nginx
Port 8008 and 8448 are accessible from the outside
Using the default cert

This error:

2017-02-12 17:08:37,178 - synapse.access.http.8008 - 91 - INFO - GET-80- 87.88.37.184 - 8008 - {None} Processed request: 170ms (11ms, 4ms) (0ms/0) 62B 401 "GET /_matrix/client/api/v1/directory/room/%23matrix%3Amatrix.org HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0"
2017-02-12 17:08:41,461 - synapse.handlers.presence - 328 - INFO - GET-80- Handling presence timeouts
2017-02-12 17:08:41,462 - synapse.handlers.typing - 79 - INFO - GET-80- Checking for typing timeouts
2017-02-12 17:08:41,463 - synapse.handlers.presence - 239 - INFO - GET-80- Performing _persist_unpersisted_changes. Persisting 1 unpersisted changes
2017-02-12 17:08:41,476 - synapse.handlers.presence - 251 - INFO - GET-80- Finished _persist_unpersisted_changes
2017-02-12 17:08:41,624 - synapse.storage.TIME - 207 - INFO - - Total database time: 0.112% {update_presence(1): 0.109%, update_cached_last_access_time(1): 0.003%, user_get_threepids(0): 0.000%} {}
2017-02-12 17:08:46,461 - synapse.handlers.presence - 328 - INFO - - Handling presence timeouts
2017-02-12 17:08:46,462 - synapse.handlers.typing - 79 - INFO - - Checking for typing timeouts
2017-02-12 17:08:48,823 - synapse.access.http.8008 - 59 - INFO - GET-81- 87.88.37.184 - 8008 - Received request: GET /_matrix/client/api/v1/directory/room/%23matrix%3Amatrix.org
2017-02-12 17:08:48,824 - synapse.http.outbound - 124 - INFO - GET-81- {GET-O-9} [matrix.org] Sending request: GET matrix://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23matrix%3Amatrix.org
2017-02-12 17:08:48,852 - synapse.http.outbound - 209 - INFO - GET-81- {GET-O-9} [matrix.org] Result: 401 Unauthorized
2017-02-12 17:08:48,854 - root - 180 - WARNING - GET-81- Error retrieving alias
2017-02-12 17:08:48,855 - synapse.http.server - 126 - ERROR - GET-81- 401: Unauthorized
Traceback (most recent call last):
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 116, in wrapped_request_handler
yield request_handler(self, request, request_metrics)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 255, in _async_render
callback_return = yield callback(request, **kwargs)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/rest/client/v1/directory.py", line 49, in on_GET
res = yield dir_handler.get_association(room_alias)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/handlers/directory.py", line 177, in get_association
retry_on_dns_fail=False,
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/federation/transport/client.py", line 185, in make_query
timeout=10000,
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 390, in get_json
timeout=timeout,
File "/root/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "/root/.synapse/local/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 219, in _create_request
response.code, response.phrase, body
HttpResponseException: 401: Unauthorized

No SRV records (I tried redirecting to 443 via srv but didn't changed anything)
I tried using letsencrypt cert (the same as kakise.xyz) but didn't seemed to resolve anything
Server using hsts, kakise.xyz:8448 is not accessible from browser (except when using letsencrypt)

Kakise · 2017-02-12T16:16:50Z

(.synapse)root@kakise:~# matrixtool server-key kakise.xyz
[INFO] Connected to 89.234.183.52:8448
[OK] TLS fingerprint matches (SHA-256)
[OK] Verified using origin=kakise.xyz key_id=ed25519:a_FwNn
v2 keys from kakise.xyz:

Key id ed25519:a_FwNn
  base64::OVGCeYtKzTx04ZGvPiN47dAFvqNpSGQ//Q7WvD2xKtM
[INFO] Matches cached key
(.synapse)root@kakise:~# matrixtool notary kakise.xyz matrix.org
[INFO] Connected to 83.166.64.33:8448
Keys from kakise.xyz via notary matrix.org

Key id ed25519:a_FwNn
  base64::OVGCeYtKzTx04ZGvPiN47dAFvqNpSGQ//Q7WvD2xKtM
[OK] Verified using server_name=kakise.xyz key_id=ed25519:a_FwNn
[INFO] Matches cached key

donpdonp · 2017-03-13T22:00:02Z

I'm not sure if the matrixtool is already doing this but a very helpful test is to
curl 'https://matrix.org/federationtester/api/report?server_name=myhomeserver.org'

and inspect the json

npodbielski · 2021-10-24T21:06:14Z

If anyone will happen to end up here from search engine here are official docs on how to fix it :)
https://github.com/matrix-org/synapse/blob/develop/docs/delegate.md

steckerhalter changed the title ~~federation issue: "not authorized" trying to join room~~ federation issue: "Unauthorized" trying to join foreign room Sep 25, 2016

richvdh closed this as completed Sep 28, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

federation issue: "Unauthorized" trying to join foreign room #1142

federation issue: "Unauthorized" trying to join foreign room #1142

steckerhalter commented Sep 25, 2016

steckerhalter commented Sep 26, 2016

icaruseffect commented Sep 28, 2016

richvdh commented Sep 28, 2016

donpdonp commented Oct 15, 2016

richvdh commented Oct 16, 2016

richvdh commented Oct 16, 2016 •

edited

Loading

leonerd commented Oct 16, 2016

simonszu commented Nov 23, 2016

donpdonp commented Nov 25, 2016 •

edited

Loading

icaruseffect commented Nov 25, 2016 •

edited

Loading

simonszu commented Nov 26, 2016

ara4n commented Nov 27, 2016 via email

imazes commented Dec 21, 2016

donpdonp commented Jan 14, 2017 •

edited

Loading

Kakise commented Feb 12, 2017

Kakise commented Feb 12, 2017

donpdonp commented Mar 13, 2017

npodbielski commented Oct 24, 2021

federation issue: "Unauthorized" trying to join foreign room #1142

federation issue: "Unauthorized" trying to join foreign room #1142

Comments

steckerhalter commented Sep 25, 2016

steckerhalter commented Sep 26, 2016

icaruseffect commented Sep 28, 2016

richvdh commented Sep 28, 2016

donpdonp commented Oct 15, 2016

richvdh commented Oct 16, 2016

richvdh commented Oct 16, 2016 • edited Loading

leonerd commented Oct 16, 2016

simonszu commented Nov 23, 2016

donpdonp commented Nov 25, 2016 • edited Loading

icaruseffect commented Nov 25, 2016 • edited Loading

simonszu commented Nov 26, 2016

ara4n commented Nov 27, 2016 via email

imazes commented Dec 21, 2016

donpdonp commented Jan 14, 2017 • edited Loading

Kakise commented Feb 12, 2017

Kakise commented Feb 12, 2017

donpdonp commented Mar 13, 2017

npodbielski commented Oct 24, 2021

richvdh commented Oct 16, 2016 •

edited

Loading

donpdonp commented Nov 25, 2016 •

edited

Loading

icaruseffect commented Nov 25, 2016 •

edited

Loading

donpdonp commented Jan 14, 2017 •

edited

Loading