Endless chain of replies to messages should be disabled or limited. Causes server crash. #7192
Comments
|
The default could be that only three levels of quotes are shown, and after that there is an expand button for deeper quote levels, also three levels at a time? |
|
I can't believe how easy it is to take down a whole server doing this "exploit". I think this needs to be put up to one of the higher priorities. |
|
ouch! how many did it take? I agree there should be a limit not only for sanity, but obviously having so many nested attachments like that is causing a loop of some sort that is bringing the server to its knees |
|
@geekgonecrazy Honestly, i couldn't even go to check because the server was very unresponsive. When i tried to scroll up to see who was doing this the screen started going blank. I just saw enough to take that screenshot. We have 32 gigs of ram just dedicated to Rocket Chat also so its definitely causing some crazy loops as you said. |
|
Erm... just to verify this: The server is actually crashing, yeah? If so: with what error message? High CPU load, exhausted memory? Anything that strikes? It's not just the client-side JavaScript that makes "the server" (that is actually the client then) appear slow, right? Cheers |
|
yes both server and client end. On my end i am seeing the 3 dots load when refreshing the page. Those 3 dots are going on endlessly most of the time. Messages might appears a long time after a refresh. As for the server, i think my admin said it was starting to show 100% CPU usage. That may or may not be due to the vulnerability. Unfortunately, i didnt check the other instances of RC i had installed on the server to see if those were affected. I have since removed RC from the high traffic part of our site due to this issue (its basically unusable with this vulnerability) and some other issues with many users on at once (which could be related, who knows). |
|
While those first hand descriptions are quite valuable to see what's up, it's the actual error messages, process traces or the real effects on the machine parameters that make the debugging more feasible. So you might wanna get those from your admin. Cheers |
|
I understand but i couldnt even get into the admin panel to get those errors. Unless you are talking about the errors showing up in the actual server, then i would have to reproduce it again but not sure i want to put this back on our website right now. I think we definitely need to limit the replies to replies no matter what errors show up. |
This might or might not be true from a technical perspective, but I totally agree from a UI perspective. And yes, I'm talking about the actual logfiles from the server. Cheers |
|
Yeah sorry thats what i meant. From a UI perspective is reason enough. From a technical perspective, i'm pretty confident it has an affect also, as that was the only thing going on in our chat at the time and it slowed our server down big time. We did not have file uploads enabled at the time. Cams were disabled too. Only endless "replies" being spammed by one user. (which i thought the character limit would stop but apparently not) |
This is getting off topic and in to HA stuff. But an instance of Rocket.Chat its self is only going to use up to 1.7GB unless you specify an option to allow more: From your case it seems like in conjunction with the other issue you have opened here: #7183 that your server being at 100% might be the other issue. The nested replies is likely effecting the client and causing it to lock up. The fact that you are getting the 3 dots means the Rocket.Chat server its self is responded so has not crashed. Its likely that something is happening client side. If you can provide some logs from the browsers console log we could probably narrow this down. But to get your server back up you can hop into the mongo database. Do a find in the users collection for the user that was spamming. Get his/her id. Do a find on the rooms collection with the room that was spammed. Grab its id. Then do a delete for records in the messages collection with the users id and the room id. Obviously that doesn't take away from us needing to be able to limit replys. But either way this will be needed to get your server back up. |
|
I apologize but I am just really going from memory (which admittedly isnt great) about the 3 dots. That could have been 3 dots i saw with the other issue i'm having that you have already referenced. I 100% remember when the spammer was doing it, the screen was all blank white. After a long while, his long messages might show up, but then if you want to follow the messages up to see whos doing it, it would just turn white again and on my browser the Safari beach ball would start spinning. Pretty much locking up my browser. I think if i restart the server and booted that guy from doing it, the browser would have responded to the server. Agreed, the 100% CPU thing can be related to the other issue so dont take that as 100% sure its related to this tickets issue. The only thing i can 100% confirm is the client issues from the browser. I doubt i can get those logs from my browser now as that was a few days ago. |
|
Just had this happen on another website running RC. It's going to be the new way to "take down" an RC server as soon as word spread. Well hope this is going to be in the next update. |
|
Happening right now. Checked CPU usage and its at 105% lol. Memory usage is low. Client browser totally locks up. Almost impossible to log into the server Admin panel to restart. I am guessing anyone would be able to lock up the demo rocket chat doing this same method so this should be an urgent fix. Is there any temporary fix to disable reply to messages that we can add to keep our RC applications up? |
|
@WebSavvyGuy This is just a test: it will NOT fix the actual messages just how they look in YOUR browser (until you reload), but if this works we can provide a script to fix the messages in the database. |
|
@gdelavald Hi. Am i putting this under "SOURCES" "+Snippet" in Chrome? I apologize i've never used that tool before. Also, its very difficult to find out when the chains will happen again |
|
@WebSavvyGuy if you deleted the message chains already, there is no much point in trying this :P |
|
Oh yeah it was deleted and even reinstalled long long ago. It was just an exploit some kiddies were constantly using. |
|
Hi, Some news about this problem ? The script of @gdelavald is ok, but no option in admin page to disable the quote loop, or just disable quote ? |
|
It is located in the Administration panel under Settings > Messages |
|
... I'm sorry, maybe I'm stupid but I don't find this setting. Just in message ? Not under an expand menu ? |
|
Its above the "Hide User Join Messages" |
Rocket.Chat Version: 0.56
Running Instances:
DB Replicaset OpLog:
Node Version:
Endless chains of replies causes crashing. When a user replies to a reply to a reply such as the image shown below, the entire server begins to lag and slow very badly. There should be a limit to the number of replies a message can be replied to. This will be a way RC chat rooms can be abused.
The text was updated successfully, but these errors were encountered: