den.half-shot.uk is confused about whether it is in !EgoVrPyOnPSVXERlxj:gruenhage.xyz

[Half-Shot]

• I (@ half-shot:den.half-shot.uk) joined !EgoVrPyOnPSVXERlxj:gruenhage.xyz a long time ago from a previous dendrite install.
• I recently reinstalled Dendrite, kept the same keys but vaped the database. I created the user with the same mxid.
• I kicked my user from the above room, and then invited them.
• Dendrite joined the room and sent a few messages, but then backfilled.
• Dendrite backfilled the kick event, and applied it over my join event so that I was kicked from the room.
  • This didn't apply quite cleanly. The client stopped being able to send events, and a cache refresh showed that the user was kicked.
• Other servers in the room think the user is still joined.
• After trying to kick the user again from the room from Synapse, and trying to invite again, I get "You do not have permission to invite people to this room.". The error from Synapse reads: {"errcode":"M_FORBIDDEN","error":"User is already joined to room"}

Logs:
invite_fail.txt

[neilalexander]

First step in diagnosing this is working out whether the roomserver and the sync API agree on what your membership is. Please run the database queries below and check the event that's returned—they should be the same:

Roomserver:

SELECT event_json FROM roomserver_rooms
INNER JOIN roomserver_state_snapshots
	ON roomserver_rooms.state_snapshot_nid = roomserver_state_snapshots.state_snapshot_nid
INNER JOIN roomserver_state_block
	ON roomserver_state_block.state_block_nid = ANY(roomserver_state_snapshots.state_block_nids)
INNER JOIN roomserver_event_state_keys
	ON roomserver_state_block.event_state_key_nid = roomserver_event_state_keys.event_state_key_nid
INNER JOIN roomserver_event_types
	ON roomserver_state_block.event_type_nid = roomserver_event_types.event_type_nid
INNER JOIN roomserver_event_json
	ON roomserver_state_block.event_nid = roomserver_event_json.event_nid
WHERE roomserver_rooms.room_id = '!EgoVrPyOnPSVXERlxj:gruenhage.xyz'
	AND event_state_key = '@half-shot:den.half-shot.uk';

Sync API:

SELECT headered_event_json FROM syncapi_current_room_state
WHERE state_key = '@half-shot:den.half-shot.uk'
	AND room_id = '!EgoVrPyOnPSVXERlxj:gruenhage.xyz';

[Half-Shot]

Room server:

{
    "auth_events": [
        "$gctSxKL5BC7cY9ZeLjCIg0GtSQxznduM3YwYxSHg7Yw",
        "$pMrgHzPcFE3JkFCeNdWlT530oPVSNOry_n_6Yyw_Aww",
        "$geJZI-v1PNGkyRA29XU_XEM6K8FgU1WWUVThdBJWyDE",
        "$cUQz_hJ8K-xpp9HqZsgDJ1KBhI8nKBBbIPKelNuHVKk"
    ],
    "content": {
        "avatar_url": "mxc:\/\/den.half-shot.uk\/8c234263641af0de0a1860ecfe91208ee5725944f0d6c55027e3031b88c3591b",
        "displayname": "half-shot",
        "membership": "join"
    },
    "depth": 43125,
    "hashes": {
        "sha256": "LHEwdM9Ecj1cycxk0FHF7i3MRQ44V8dRxI6yUERHMtM"
    },
    "origin": "den.half-shot.uk",
    "origin_server_ts": 1603228896489,
    "prev_events": [
        "$geJZI-v1PNGkyRA29XU_XEM6K8FgU1WWUVThdBJWyDE"
    ],
    "prev_state": [],
    "room_id": "!EgoVrPyOnPSVXERlxj:gruenhage.xyz",
    "sender": "@half-shot:den.half-shot.uk",
    "state_key": "@half-shot:den.half-shot.uk",
    "type": "m.room.member",
    "signatures": {
        "den.half-shot.uk": {
            "ed25519:fg4A": "xVHiLLMa3rGWPpS\/91y5VuMumqj5NlwKAAh53fJvtld+prcezqRewJUl6KT7wjKK4RswVsbwGyM8vuSdCPNcCg"
        }
    },
    "unsigned": {
        "replaces_state": "$geJZI-v1PNGkyRA29XU_XEM6K8FgU1WWUVThdBJWyDE",
        "prev_content": {
            "avatar_url": "mxc:\/\/den.half-shot.uk\/8c234263641af0de0a1860ecfe91208ee5725944f0d6c55027e3031b88c3591b",
            "displayname": "half-shot",
            "membership": "invite"
        },
        "prev_sender": "@Half-Shot:half-shot.uk"
    }
}

{
    "auth_events": [
        "$cUQz_hJ8K-xpp9HqZsgDJ1KBhI8nKBBbIPKelNuHVKk",
        "$WVPwhjLoXA5ls3hWBXNAjrd-VPUFd7kUgD0HjXnK7FY",
        "$pMrgHzPcFE3JkFCeNdWlT530oPVSNOry_n_6Yyw_Aww",
        "$nMOAar2VvdE4VBe-00fK_6Nq-Uc4qhJqKp8UO80onj0"
    ],
    "content": {
        "membership": "leave"
    },
    "depth": 43155,
    "hashes": {
        "sha256": "wZCbxNqolfxarMqOyZLda0KnT11pG1FHb2sFQkMIflw"
    },
    "origin": "half-shot.uk",
    "origin_server_ts": 1603229256684,
    "prev_events": [
        "$FQhOjiTrRKJ6YP5Wwy6ocGCG35MYNx1N7EE2QgxFClg"
    ],
    "prev_state": [],
    "room_id": "!EgoVrPyOnPSVXERlxj:gruenhage.xyz",
    "sender": "@Half-Shot:half-shot.uk",
    "signatures": {
        "half-shot.uk": {
            "ed25519:a_fBAF": "NYes55VCAwoRzdVnhr94eG71ISKC4y22AKbgnt7Wq8ux76lboxcs51vLthZHm73LqTWnRdKvyTq7YZex61y7BA"
        }
    },
    "state_key": "@half-shot:den.half-shot.uk",
    "type": "m.room.member",
    "unsigned": {
        "age_ts": 1603229256684,
        "replaces_state": "$WVPwhjLoXA5ls3hWBXNAjrd-VPUFd7kUgD0HjXnK7FY"
    }
}

Sync

{
    "auth_events": [
        "$cUQz_hJ8K-xpp9HqZsgDJ1KBhI8nKBBbIPKelNuHVKk",
        "$WVPwhjLoXA5ls3hWBXNAjrd-VPUFd7kUgD0HjXnK7FY",
        "$pMrgHzPcFE3JkFCeNdWlT530oPVSNOry_n_6Yyw_Aww",
        "$nMOAar2VvdE4VBe-00fK_6Nq-Uc4qhJqKp8UO80onj0"
    ],
    "content": {
        "membership": "leave"
    },
    "depth": 43155,
    "hashes": {
        "sha256": "wZCbxNqolfxarMqOyZLda0KnT11pG1FHb2sFQkMIflw"
    },
    "origin": "half-shot.uk",
    "origin_server_ts": 1603229256684,
    "prev_events": [
        "$FQhOjiTrRKJ6YP5Wwy6ocGCG35MYNx1N7EE2QgxFClg"
    ],
    "prev_state": [],
    "room_id": "!EgoVrPyOnPSVXERlxj:gruenhage.xyz",
    "sender": "@Half-Shot:half-shot.uk",
    "signatures": {
        "half-shot.uk": {
            "ed25519:a_fBAF": "NYes55VCAwoRzdVnhr94eG71ISKC4y22AKbgnt7Wq8ux76lboxcs51vLthZHm73LqTWnRdKvyTq7YZex61y7BA"
        }
    },
    "state_key": "@half-shot:den.half-shot.uk",
    "type": "m.room.member",
    "unsigned": {
        "age_ts": 1603229256684,
        "replaces_state": "$WVPwhjLoXA5ls3hWBXNAjrd-VPUFd7kUgD0HjXnK7FY"
    },
    "_room_version": "6"
}

[neilalexander]

Looks to be related to #1547 and #1548 - the roomserver has the correct state but the sync API has taken an old event as if it's new.

[Half-Shot]

Out of interest, we're unable to invite me to the room because Dendrite is reporting to others that I am joined to the room. Is that related?

[neilalexander]

Yes, the problem here is that the roomserver (which thinks you're in the room) doesn't agree with the sync API (which doesn't think you're in the room). That's why no one can invite you - your server believes you're already in the room and therefore responds to any federated invite requests as such.

The fault here is the sync API for consuming old events as if they were new ones, which is why the problem occurred after backfilling.

[Half-Shot]

Aha.

[Half-Shot]

I created a new account and joined the room fine. I then kicked (technically just sent a leave->leave) the user to try and unstick dendrite, but it still thinks I'm joined. Any ideas for remediation steps?

[neilalexander]

You can try modifying the roomserver_memberships table to make the roomserver accept an invite again.

You'll need the:

• Room NID from roomserver_rooms
• State key NID for your user from roomserver_event_state_keys

Then just find the relevant row in roomserver_memberships and set membership_nid to 0. That should allow you to accept invites again.

[Half-Shot]

That worked thanks!