Dealing with high player count and high latency

[bryanmylee]

I'm using this library with a game that expects 12 players across different platforms, including HTML5. For context, due to the HTML5 support, I'm using the WebSocketMultiplayerPeer instead of ENetMultiplayerPeer.

The rollback and time synchronization works really well when I'm debugging locally with low latency, but during a test deployment of my game with a server, I was getting lots of rollback issues like:

[ERR][netfox::RollbackSynchronizer] Received state for 14506, rejecting because older than 64 frames

I know I could increase the history limit to be more lenient to high latency clients, but it seems unavoidable that eventually someone will have a bad connection.

I'm wondering if there is any common advice for dealing with high latency?

[elementbound]

What is your tickrate and what is the measured latency? Because if you use the default 30, then 64 frames is more than two seconds of latency, which is quite a lot. I'd like to make sure that it's indeed network latency and not an issue with the time sync.

During testing I have aimed to keep games playable at ~250ms ping and kept the 64 frame history limit as a safe default. Unfortunately, at some point you have to draw the line because you run out of tradeoffs to make. There might be better approaches out there, but I'm not the one aware of them I'm afraid 🙂

But depending on the game, you can try to keep the tickrate as low as possible, and increase history as high as sensible, as you've mentioned. Higher history limit means more memory, but lower tickrate means you get more seconds of history for the same memory usage.

[bryanmylee]

I have my tickrate set to 30, and a history limit of 64.

With 5 clients connected, some of my clients get RTTs to the server of around 0.16s, with spikes to around 0.82s-1.06s.

I assume that means the latency therefore should at most be at most 503ms to the server, and at most 1.06s between clients? Well within the 2.13s history buffer I have.

For what it's worth, I did implement the workaround at #175 since I need nodes with RollbackSynchronizer to spawn at any time, so I'm not sure if that's causing issues.

EDIT: I'm getting RTTs up to 16 seconds when new clients join an existing game, causing a global lag spike.

[bryanmylee]

I'm not sure if this is relevant, but the issue pops up the most when I connect my Android client to the network.

EDIT: In hindsight, it might have less to do with the platform and more to do with the number of clients connected at the time.

[bryanmylee]

6.players.mp4

This is a demo with 6 players (2 HTML5, 4 native), with a server hosted on a server with at most 1s RTT.

2.players.mp4

The rollback issues seem to tone down greatly when the number of players are reduced.

[bryanmylee]

The issue seems to stem from new nodes being added. When a 4th client is added later in the match, existing clients get a spike that causes >2s RTTs.

adding.new.nodes.causes.lag.spike.mp4

[bryanmylee]

My guess is with larger multiplayer lobbies, too many rollbacks are occurring whenever any single player has a lag spike, causing the entire network to grind to a halt.

Maybe rollback netcode isn't suitable for large multiplayer lobbies. Is it possible to disable rollbacks, and set up netfox to:

1. optimistically update player A's local state based on input (at t=0)
2. send player A's input to the server (received at t=1) which then updates its server state and broadcasts player A's server state to others (received on peers at t=2)
3. occasionally sync player A's server state (at t=1) back to local (received on local at t=2) to check for drift

On step 3, instead of rolling back player A's local state to t=1 and resimulating to t=2, we just need to verify that its local state at t=1 is close-enough to the server state at t=1, with "close enough" defined via custom predicates for each property.

If player A's local state at t=1 fails verification, we snap player A's local state at t=2 (current time) to the server state received at t=1.

Peers will not need to adjust their local states for player A, since they've only received server state for player A.

This does mean that state is more inconsistent across the network, but it would be more scalable for large lobbies.

[elementbound]

Thank you for the detailed findings, really appreciated!

I would also suspect that there's a lot of resimulation going on, which becomes more and more of an issue with more players. What are your planned latency values to support?

Also, I've just merged #195, which adds custom perf monitors. I'd be particularly interested in rollback ticks simulated for your case - is it consistently high, or does it spike occasionally? I've played around with 1s RTT, and noticed that there's an occasional spike of resimulated rollback ticks, from the usual 25 to 100+, before settling down. Very much looks like an input packet being stuck, and since the inputs are sent as reliable, probably all other inputs are halted before that single missed one arrives. Re-requesting packets at 1s RTT sounds very expensive. #160 might help with that, not sure yet. Limiting the number of ticks resimulated is also an option I want to investigate, but I'd be cautious with that for now - it can mask bigger issues, and not sure if it won't cause runaway desync problems.

Anyway, I'm wondering whether the same is happening in your case too. Would you be willing to take a look with #195 merged? You can just copy NetworkPerformance to your project, and add it as an autoload or a regular node to your scene.

---

You can certainly do the optimistic updates, although netfox doesn't have built-in support for this. You could either build your own specialized synchronizer node for this ( maybe by carving it from RollbackSynchronizer's code ), or building your own from scratch. Either way, the property classes could be useful.

Also, feels a bit weird to say this, but depending on your game, you could also forego the server-authoritative setup, by just making each player own their avatar. At that point, you can either use Godot's MultiplayerSynchronizer, or the upcoming StateSynchronizer ( #196 ). It acts the same as MultiplayerSynchronizer, but is tied to netfox's tick loop, and works with TickInterpolator. I've shared the source here.

[bryanmylee]

I would also suspect that there's a lot of resimulation going on, which becomes more and more of an issue with more players. What are your planned latency values to support?

I'm planning to support mobile players so I expect RTTs of minimally 200ms, which does make rollback seem infeasible, especially with 12-player lobbies.

...is it consistently high, or does it spike occasionally?

It actually does spike occasionally, especially when I'm running other RPCs to spawn and sync player data. In the last demo I sent above, you'll notice the RTT spiking when a new player joins the match.

---

I'll take a look with #195 when I have the time, but I really really appreciate the work you're doing. Your expertise has been wonderful, and I wish I could contribute more.

I'll also experiment with the idea of a server-authoritative-"lite" / server-verified mode for my use-case. I still don't want to give up control to clients completely, and if data has to travel through the server before reaching peers, I might as well use that to verify user inputs.

[elementbound]

I'm planning to support mobile players so I expect RTTs of minimally 200ms

That is indeed the higher end of what I've originally considered, although I'm suspecting that rollback also shouldn't take that much time. I'll see if I can make the rollback process more lightweight, but don't hold your breath on that 🙂 Better support for larger player count is also a good direction for netfox.

It actually does spike occasionally, especially when I'm running other RPCs to spawn and sync player data

Thanks for the confirm!

---

I really really appreciate the work you're doing. Your expertise has been wonderful, and I wish I could contribute more.

Thank you, much appreciated! So is your willingness to provide further info and work with me to investigate! 🙂 If you'd be interested, I can spend some time to write a more detailed ticket ( compared to "build something like RollbackSynchronizer but different" ) so you can pick it up. If it works out, we can merge it to netfox as an additional sync strategy.

[bryanmylee]

Yep! I'll open an issue to describe what I'm trying to build in more detail!

Started a thread at #197.

[albertok]

Very much looks like an input packet being stuck, and since the inputs are sent as reliable, probably all other inputs are halted before that single missed one arrives

Would it be better to move to sending inputs unreliably but each packet includes the previous x inputs?

[elementbound]

I think the best would be to have a "reliable_unordered" mode in Godot, which just makes sure that things arrive eventually, with order not being that important. At least, if I were to implement something over unreliable, that would be it.

[TheYellowArchitect]

Maybe rollback netcode isn't suitable for large multiplayer lobbies.

The only real limitation is the hardware of the server (to simulate multiple ticks per frame) and the bandwidth of the network. By extension, in a normal setup, 200ms of 16 players should be possible imo, especially if you have a tiny state consisting of player data only

I am curious myself as to those spikes. That said, perhaps when a player joins but it tries to rollback, it tries to rollback on a frame before the player joined and causes problems?

Also has been a while since I saw how the input is sent to the server in forest brawl, is it a dictionary? If yes, that's the problem

[elementbound]

It's a dictionary, yes. Why is that the problem?

[TheYellowArchitect]

12 players are sending input each tick to 12 players, leading to a network saturation. Each player exponentially worsens the network, instead of utilizing the server.
Anyway, this is solved by the combination of #221 and #230

If bandwidth really is the problem, the PRs #251 #254 should solve it. But I haven't looked deep into the join/sync logic, its the final culprit if this problem still exists in latest version.

[annamaria-szentpeteri]

I'm a bit confused: how is your last comment related to the input being sent as a dictionary?

[TheYellowArchitect]

Also has been a while since I saw how the input is sent to the server in forest brawl, is it a dictionary? If yes, that's the problem

It's an expansion of this^

[annamaria-szentpeteri]

How is sending a dictionary leads to network saturation?