WebSocketManager disconnects 30-60 minutes at a time #8558

adam-a · 2022-08-27T01:38:28Z

Which package is this bug report for?

ws

Issue description

Running WebSocketManager for a bot on my own personal Discord server

Tested this locally and on a staging within an EC2 instance -- the same issue applies

Randomly throughout the day this will occur and could go on for upwards of 30-60 minutes.

How can I ensure no/minimal disconnection? (Code sample below -- very straightforward implementation)

Connecting to [wss://gateway.discord.gg?v=10&encoding=json](wss://gateway.discord.gg/?v=10&encoding=json)
info:    Debug
Waiting for event hello for 60000ms
info:    Heartbeat
info:    Debug
Starting to heartbeat every 41250ms
info:    Debug
Identifying
        shard id: 0
        shard count: 1
        intents: 34305
        compression: none
info:    Debug
Disconnected normally from code 1000
info:    Debug
Destroying shard
        Reason: Got disconnected by Discord
        Code: 1000
        Recover: Reconnect
info:    Debug
Connecting to [wss://gateway.discord.gg?v=10&encoding=json](wss://gateway.discord.gg/?v=10&encoding=json)
info:    Debug
Waiting for event hello for 60000ms
info:    Heartbeat
info:    Debug
Starting to heartbeat every 41250ms
info:    Debug
Identifying
        shard id: 0
        shard count: 1
        intents: 34305
        compression: none
info:    Debug
Disconnected normally from code 1000
info:    Debug
Destroying shard
        Reason: Got disconnected by Discord
        Code: 1000
        Recover: Reconnect
info:    Debug
Connecting to [wss://gateway.discord.gg?v=10&encoding=json](wss://gateway.discord.gg/?v=10&encoding=json)
info:    Debug
Waiting for event hello for 60000ms
info:    Heartbeat
info:    Debug
Starting to heartbeat every 41250ms
info:    Debug
Identifying
        shard id: 0
        shard count: 1
        intents: 34305
        compression: none
info:    Debug
Disconnected normally from code 1000
info:    Debug
Destroying shard
        Reason: Got disconnected by Discord
        Code: 1000
        Recover: Reconnect
info:    Debug
Connecting to [wss://gateway.discord.gg?v=10&encoding=json](wss://gateway.discord.gg/?v=10&encoding=json)
info:    Debug
Waiting for event hello for 60000ms
info:    Heartbeat
info:    Debug
Starting to heartbeat every 41250ms
info:    Debug
Identifying
        shard id: 0
        shard count: 1
        intents: 34305
        compression: none

Code sample

if (!authToken) throw Error('BOT AuthToken not set');

  const rest = new REST().setToken(authToken);

  const manager = new WebSocketManager({
    handshakeTimeout: 20000,
    token: authToken,
    intents,
    rest,
  });

  manager.setStrategy(new SimpleShardingStrategy(manager));

  manager.addListener(WebSocketShardEvents.Debug, async (payload: { message: string }) =>
    logger.info(`Debug\n${payload.message}`)
  );

  manager.addListener(WebSocketShardEvents.Ready, () => logger.info('Ready'));

  manager.addListener(WebSocketShardEvents.Hello, () => logger.info('Heartbeat'));

  manager.addListener(WebSocketShardEvents.Resumed, () => logger.info('Resumed'));

  manager.addListener(
    WebSocketShardEvents.Dispatch,
    (
      event: {
        data: GatewayDispatchPayload;
      } & {
        shardId: number;
      }
    ) => logEvent(event.data)
  );

  await manager.connect();

Package version

14.3.0

Node.js version

18.6.0

Operating system

No response

Priority this issue should have

High (immediate attention needed)

Which partials do you have configured?

Message, Reaction

Which gateway intents are you subscribing to?

Guilds, GuildMessages, GuildMessageReactions, MessageContent

I have tested this issue on a development release

No response

The text was updated successfully, but these errors were encountered:

didinele · 2022-08-27T08:20:34Z

What version of @discordjs/ws are you on?

adam-a · 2022-08-27T17:35:23Z

What version of @discordjs/ws are you on?

Hi! Thanks for responding.

I'm using 0.3.0.

It's driving me nuts. Even overrode updateSessionInfo and retrieveSessionInfo at some point to keep track of session info since it seems like it is being utilized in https://github.com/didinele/discord.js/blob/874ee5cee006edad6d7fe138d5dbe83f96d3f0e5/packages/ws/src/ws/WebSocketShard.ts#L105 (check line 128)

But that did not work as well.

didinele · 2022-08-27T17:58:08Z

Sorta familiar with this issue. We've been trying to fix it for a while and across multiple PRs. You're not experiencing Discord constantly telling you to reconnect (at least to my understanding), but rather a reconnect loop bug.

My latest attempt at patching this up was #8479, but seems like that wasn't enough.

cc @SpaceEEC

adam-a · 2022-08-27T18:10:26Z

Sorta familiar with this issue. We've been trying to fix it for a while and across multiple PRs. You're not experiencing Discord constantly telling you to reconnect (at least to my understanding), but rather a reconnect loop bug.

My latest attempt at patching this up was #8479, but seems like that wasn't enough.

cc @SpaceEEC

Do you think it's an issue in the underlying WebSocket package?

I implemented my bot with the standard WebSocket package before finding discord.js, and I had the same issue.

Also, side note, today I noticed this:

Waiting for event hello for 60000ms
info:    Heartbeat
info:    Debug
Starting to heartbeat every 41250ms
info:    Debug
Resuming session
info:    Debug
Resuming session
/Users/some/Documents/Github/some/shared_services/node_modules/ws/lib/websocket.js:441
      throw new Error('WebSocket is not open: readyState 0 (CONNECTING)');
            ^
Error: WebSocket is not open: readyState 0 (CONNECTING)
    at WebSocket.send (/Users/some/Documents/Github/some/shared_services/node_modules/ws/lib/websocket.js:441:13)
    at WebSocketShard.send (/Users/some/Documents/Github/some/shared_services/node_modules/@discordjs/ws/dist/ws/WebSocketShard.cjs:168:21)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async WebSocketShard.connect (/Users/some/Documents/Github/some/shared_services/node_modules/@discordjs/ws/dist/ws/WebSocketShard.cjs:97:7)
    at async WebSocketShard.onMessage (/Users/some/Documents/Github/some/shared_services/node_modules/@discordjs/ws/dist/ws/WebSocketShard.cjs:322:11)
[nodemon] app crashed - waiting for file changes before starting...

didinele · 2022-08-27T19:08:15Z

Do you think it's an issue in the underlying WebSocket package?

No.

adam-a · 2022-08-27T19:16:03Z

Do you think it's an issue in the underlying WebSocket package?

No.

I'd like to clone discord.js and assist in this effort. What's a good starting point for me particularly for this recurring loop bug?

didinele · 2022-08-27T19:48:54Z

packages/ws/src/ws/WebSocketShard.ts - mainly the onClose handler and the destroy method.

Dossar · 2022-10-07T02:52:52Z

Sorta familiar with this issue. We've been trying to fix it for a while and across multiple PRs. You're not experiencing Discord constantly telling you to reconnect (at least to my understanding), but rather a reconnect loop bug.

My latest attempt at patching this up was #8479, but seems like that wasn't enough.

cc @SpaceEEC

Hello, it looks like this thread might be the same root cause for bots disconnecting after an hour or earlier:
#8486 (comment)

To summarize the findings of what I saw in that thread,

Bot randomly exiting process or going offline/unresponsive #8486 (comment) mentioned discordjs 13.8.1 is not showing these issues.
The person in the OP was using discord.js@14.1.2
Bot randomly exiting process or going offline/unresponsive #8486 (comment) reported v13.10.3 has the same problem.

The common data point seems to be that somewhere between v13.8.1 and v13.10.3, something introduced this issue of bots disconnecting from the client. And then that ended up getting into v14.x

legendhimself · 2022-10-07T22:27:00Z

might be the same root cause for bots disconnecting after an hour or earlier:
#8486 (comment)

@Dossar
this is an issue from the new WS package and it has nothing to do with those issues since even in the latest versions DJS still does not make use of the WS package for creating the WebSockets or connecting to the discord gateway.

didinele · 2022-10-12T18:55:47Z

Hi! Are you still able to reproduce this issue on 0.4.1?

adam-a · 2022-11-02T19:06:50Z

Hi! Are you still able to reproduce this issue on 0.4.1?

Hi there! I just updated it and will be watching it. Will lyk! Thank you for the fix.

adam-a · 2022-11-03T18:54:14Z

@didinele Unfortunately, the issue persists with 0.4.1 :/

Nov 3, 2022 @ 11:54:13.921 | Debug Identifying 	shard id: 0 	shard count: 1 	intents: 34305 	compression: none
-- | --

  | Nov 3, 2022 @ 11:54:13.920 | Heartbeat

  | Nov 3, 2022 @ 11:54:13.920 | Debug Starting to heartbeat every 41250ms

  | Nov 3, 2022 @ 11:54:13.757 | Debug Waiting for event hello for 60000ms

  | Nov 3, 2022 @ 11:54:13.756 | Debug Connecting to wss://gateway.discord.gg?v=10&encoding=json

  | Nov 3, 2022 @ 11:54:13.755 | Debug Destroying shard 	Reason: Got disconnected by Discord 	Code: 1000 	Recover: Reconnect

  | Nov 3, 2022 @ 11:54:13.755 | Debug Disconnected normally from code 1000

  | Nov 3, 2022 @ 11:53:18.760 | Debug Starting to heartbeat every 41250ms

  | Nov 3, 2022 @ 11:53:18.760 | Debug Identifying 	shard id: 0 	shard count: 1 	intents: 34305 	compression: none

  | Nov 3, 2022 @ 11:53:18.760 | Heartbeat

  | Nov 3, 2022 @ 11:53:18.604 | Debug Waiting for event hello for 60000ms

  | Nov 3, 2022 @ 11:53:18.602 | Debug Disconnected normally from code 1000

  | Nov 3, 2022 @ 11:53:18.602 | Debug Destroying shard 	Reason: Got disconnected by Discord 	Code: 1000 	Recover: Reconnect

  | Nov 3, 2022 @ 11:53:18.602 | Debug Connecting to wss://gateway.discord.gg?v=10&encoding=json

  | Nov 3, 2022 @ 11:53:09.169 | HTTP Request

  | Nov 3, 2022 @ 11:52:23.398 | Debug Starting to heartbeat every 41250ms

  | Nov 3, 2022 @ 11:52:23.398 | Debug Identifying 	shard id: 0 	shard count: 1 	intents: 34305 	compression: none

  | Nov 3, 2022 @ 11:52:23.397 | Heartbeat

  | Nov 3, 2022 @ 11:52:23.247 | Debug Waiting for event hello for 60000ms

  | Nov 3, 2022 @ 11:52:23.246 | Debug Connecting to wss://gateway.discord.gg?v=10&encoding=json

  | Nov 3, 2022 @ 11:52:23.245 | Debug Disconnected normally from code 1000

  | Nov 3, 2022 @ 11:52:23.245 | Debug Destroying shard 	Reason: Got disconnected by Discord 	Code: 1000 	Recover: Reconnect

  | Nov 3, 2022 @ 11:52:01.024 | HTTP Request

didinele · 2022-11-30T13:55:51Z

After another month of on-and-off debugging and a lot of frustration with being generally unable to reproduce this issue in a consistent matter, I've finally had a breakthrough and tracked it down to be how we handle the send ratelimit - the issue boils down to some reversed logic that was missed during review, essentially meaning that after ~120 heartbeats we think we're about to get ratelimited and block the entire send queue for... a lot of time (could be something like ~20 minutes in some cases).

WS connections auto-magically close with a 1000 close code if enough time passes with no flowing payloads. The queue stays stuck after we reconnect, so our IDENTIFY never goes through, once again, causing the connection to eventually close due to inactivity (causing the "reconnect loop" that we see) - eventually the sleep call ends and we successfully reconnect and start the cycle over again.

I'll have a PR fixing this later today, appreciate the flag and the effort in providing me with logs!

JMTK · 2022-11-30T16:32:54Z

@didinele would this also alleviate issues related to #8486 / #4095 ?

didinele · 2022-11-30T17:00:54Z

@didinele would this also alleviate issues related to #8486 / #4095 ?

discord.js (mainlib) does not currently use @discordjs/ws, it has its own own WS implementation (the "old" one), so no, those changes do not help at all at this time, but they will potentially will in the future when the main library starts using the ws package.

adam-a added bug need repro labels Aug 27, 2022

Jiralite added the packages:ws label Aug 27, 2022

Jiralite removed the need repro label Aug 27, 2022

didinele mentioned this issue Oct 10, 2022

fix(WebSocketShard): dispatch race condition #8731

Merged

didinele mentioned this issue Nov 30, 2022

fix(WebSocketShard): send ratelimit handling #8887

Merged

Jiralite added the has PR label Nov 30, 2022

kodiakhq bot closed this as completed in #8887 Dec 1, 2022

kevinlul mentioned this issue Dec 20, 2022

[RESOLVED] ⚠️Production outage (partial): bot ceased receiving events for certain servers from Discord after about 2022-12-14 11:01 UTC DawnbrandBots/bastion-bot#246

Closed

github-actions bot locked as resolved and limited conversation to collaborators Dec 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebSocketManager disconnects 30-60 minutes at a time #8558

WebSocketManager disconnects 30-60 minutes at a time #8558

adam-a commented Aug 27, 2022 •

edited by vladfrangu

Loading

didinele commented Aug 27, 2022

adam-a commented Aug 27, 2022 •

edited

Loading

didinele commented Aug 27, 2022 •

edited

Loading

adam-a commented Aug 27, 2022

didinele commented Aug 27, 2022

adam-a commented Aug 27, 2022 •

edited

Loading

didinele commented Aug 27, 2022

Dossar commented Oct 7, 2022

legendhimself commented Oct 7, 2022 •

edited

Loading

didinele commented Oct 12, 2022

adam-a commented Nov 2, 2022

adam-a commented Nov 3, 2022 •

edited

Loading

didinele commented Nov 30, 2022 •

edited

Loading

JMTK commented Nov 30, 2022

didinele commented Nov 30, 2022

WebSocketManager disconnects 30-60 minutes at a time #8558

WebSocketManager disconnects 30-60 minutes at a time #8558

Comments

adam-a commented Aug 27, 2022 • edited by vladfrangu Loading

Which package is this bug report for?

Issue description

Code sample

Package version

Node.js version

Operating system

Priority this issue should have

Which partials do you have configured?

Which gateway intents are you subscribing to?

I have tested this issue on a development release

didinele commented Aug 27, 2022

adam-a commented Aug 27, 2022 • edited Loading

didinele commented Aug 27, 2022 • edited Loading

adam-a commented Aug 27, 2022

didinele commented Aug 27, 2022

adam-a commented Aug 27, 2022 • edited Loading

didinele commented Aug 27, 2022

Dossar commented Oct 7, 2022

legendhimself commented Oct 7, 2022 • edited Loading

didinele commented Oct 12, 2022

adam-a commented Nov 2, 2022

adam-a commented Nov 3, 2022 • edited Loading

didinele commented Nov 30, 2022 • edited Loading

JMTK commented Nov 30, 2022

didinele commented Nov 30, 2022

adam-a commented Aug 27, 2022 •

edited by vladfrangu

Loading

adam-a commented Aug 27, 2022 •

edited

Loading

didinele commented Aug 27, 2022 •

edited

Loading

adam-a commented Aug 27, 2022 •

edited

Loading

legendhimself commented Oct 7, 2022 •

edited

Loading

adam-a commented Nov 3, 2022 •

edited

Loading

didinele commented Nov 30, 2022 •

edited

Loading