Federated device list not removing removed devices

[paddycarver]

Description

My friend has an account on matrix.org. I run my own homeserver using Synapse. He has clients with encryption keys that I've verified before, and one client with encryption keys that I've not verified before. He removed some of the clients that I've verified and the one that I haven't. His device list (Room Info -> People -> his name -> N verified sessions in Element desktop) doesn't show the devices he removed, but mine does. And I now get the scary red "unencrypted" badge for the room because there's an unencrypted client in the room, and we can't resolve it because he doesn't have the client anymore.

Steps to reproduce

• add devices
• sync them to a federated server
• ???? (unclear, some confounding condition here, I assume?)
• remove devices

Version information

• Homeserver:

If not matrix.org: tricuties.com

• Version: {"server_version":"1.50.1","python_version":"3.7.3"}

(I don't know what version it originally showed up on, but I can confirm it's happening in 1.48.0 and 1.50.1.)

• Install method:

Debian packages from the Matrix repository

• Platform:

Running as a VM in GCP on debian 10.

[paddycarver]

Debugging attempts:

I've verified that requesting the client-server /keys/query endpoint for my homeserver returns all the devices I'm seeing in the client. I tried to reproduce the server-server /keys/query endpoint on the matrix.org homeserver, but I've been unable so far to get it to accept my authentication header, leading me to suspect I'm not signing the JSON correctly or I'm building the request incorrectly or something.

[aaronraimist]

Related: #8631

[paddycarver]

I think this is probably a duplicate of #8631, my bad.

[DMRobertson]

Nonetheless, thanks for the report @paddycarver. I'm very grateful for your efforts to debug!

The server-server communication seems to be where this is going wrong, and I've added some optional debug logging to try and track this down. See #8631 (comment) if you'd like to give it a try.