Unable to login after enabling hidden read receipts (MSC2285)

[aaronraimist]

Description

I enabled MSC2285 in Synapse and Element yesterday and everything seemed to be working properly. Then today I randomly got logged out of Element (unsure if browser issue or Synapse/Element bug) and when I tried to sign back in /sync kept returning errors.

Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]: During handling of the above exception, another exception occurred:
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]: Traceback (most recent call last):
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     result = current_context.run(gen.send, result)
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/sync.py", line 353, in _wait_for_sync_for_user
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     result: SyncResult = await self.current_sync_for_user(
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/sync.py", line 397, in current_sync_for_user
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     sync_result = await self.generate_sync_result(
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/sync.py", line 1063, in generate_sync_result
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     res = await self._generate_sync_entry_for_rooms(
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/sync.py", line 1448, in _generate_sync_entry_for_rooms
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     now_token, ephemeral_by_room = await self.ephemeral_by_room(
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/sync.py", line 458, in ephemeral_by_room
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     receipts, receipt_key = await receipt_source.get_new_events(
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/receipts.py", line 224, in get_new_events
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     events = ReceiptEventSource.filter_out_hidden(events, user.to_string())
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:   File "/usr/local/lib/python3.8/site-packages/synapse/handlers/receipts.py", line 190, in filter_out_hidden
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]:     hidden = user_rr.get("hidden", None)
Aug 13 20:35:51 michaelpanderson matrix-synapse-worker-generic_worker-18111[2353406]: AttributeError: 'str' object has no attribute 'get'

After setting msc2285_enabled: false /sync works again and I can login.

Version information

• Homeserver: raim.ist
• Version: 1.40.0
• Install method: https://github.com/spantaleev/matrix-docker-ansible-deploy/
• Platform: Ubuntu 20.04 + Docker

I also recently started using workers but I don't think that is related.

cc @SimonBrandner

[SimonBrandner]

It is quite unclear to me how this could happen. The only way I see this happening is the data in the database being broken? Though, I don't see how that could happen either. I just don't see a simple explanation as to why user_rr would be a string. 🤔

I am also unable to reproduce

[aaronraimist]

It looks like I have 71 receipts in my database where the content is a string

synapse=# SELECT LEFT(data, 1) AS first_char, COUNT(*) FROM receipts_graph GROUP BY first_char;
 first_char | count
------------+--------
 {          | 100541
 "          |     71
(2 rows)

This looks related to #10376. Synapse doesn't validate it properly. I would guess you can do something like these answers https://stackoverflow.com/questions/28807935/attributeerror-str-object-has-no-attribute-get to prevent sync from erroring in this situation.

[SimonBrandner]

This looks related to #10376. Synapse doesn't validate it properly. I would guess you can do something like these answers stackoverflow.com/questions/28807935/attributeerror-str-object-has-no-attribute-get to prevent sync from erroring in this situation.

Thanks for the help! :)

[richvdh]

Since this will only affect people on servers with msc2285_enabled: true, we're not planning to release a Synapse 1.40.1 to fix this.