Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Server notice rooms are created rapidly if triggered by maybe_send_server_notice_to_user #12686

Closed
Half-Shot opened this issue May 10, 2022 · 2 comments · Fixed by #12713
Closed
Assignees
Labels
S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@Half-Shot
Copy link
Collaborator

In the case where a server has reached it's MAU limit, the ResourceLimitsServerNotices class is meant to send a notice to users informing them of this. However, it ends up creating a ton of duplicate rooms and never actually invites the target for the notice to the room.

  • ServerNoticesManager.get_or_create_notice_room_for_user is used to create the room for the user but does not invite them, that happens in maybe_invite_user_to_room which is called together in send_notice
  • ResourceLimitsServerNotices is cheeky and calls ServerNoticesManager.get_or_create_notice_room_for_user in maybe_send_server_notice_to_user, presumably making the fatal mistake that get_or_create_notice_room_for_user does NOT invite users.
  • Hence, if your server is over the MAU limit, it's probably calling ResourceLimitsServerNotices.maybe_send_server_notice_to_user often, and therefore creating new rooms often.

This is causing some servers to bloat their database with empty rooms, and of course any interfaces polling the admin API for rooms will see a load of extra redundant rooms.

@Half-Shot Half-Shot added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. S-Minor Blocks non-critical functionality, workarounds exist. labels May 10, 2022
@babolivier babolivier self-assigned this May 11, 2022
@babolivier
Copy link
Contributor

Thanks for investigating @Half-Shot! This looks like it's indeed the culprit, I've opened #12704 to fix this (by making get_or_create_notice_room_for_user always check if the user needs inviting).

@babolivier
Copy link
Contributor

babolivier commented May 11, 2022

Actually it looks like this alone isn't enough. It turns out the MAU code calls get_or_create_notice_room_for_user even without the MAU limit being reached, without inviting the user on purpose. This is in an attempt to figure out whether, if a room exists, we need to update the notice associated with it. This means if the room doesn't exist one is created and nothing is done unless the MAU limit is hit.

Now this can be an issue but get_or_create_notice_room_for_user is cached, which means that it should be returning the same room ID on every call (and only run the actual code of the function the first time it's called with a given MXID). The only exception to this is if the homeserver is restarted (in which case the cache is cleared and new rooms are created when the function is first called with an MXID), but I've confirmed with @Half-Shot out of bounds that the server hasn't been restarted in the problematic time period.

I'll keep investigating.

Edit: it looks like this cache on this host is having its entries invalidated because of a low TTL (10min).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
2 participants