Large Memory Consumption Tracking Issue (OOM)

[AgeManning]

Description

We are aware of an issue on the mainnet network which is causing Lighthouse to consume more memory than it should. This is leading to Out of Memory (OOM) process terminations on some machines.

The root cause of the issue (we believe) are messages being queued on gossipsub to be sent out. This is a combination of messages being published, messages being forward and gossipsub control messages. The queues are filling up and the memory is not being dropped. This appears to only be occuring on mainnet, we assume in part to the size of the network and the number of messages being transmitted.

There are a number of solutions being put in place and being tested. This issue is mainly a tracking issue, so users can follow along with development updates as we correct this issue.

Primarly the end solution will consist of more efficient memory management (avoid duplicating any memory in messages when sending) this should reduce allocations, a priortisation of messages so that we can prioritise published, forward and control messages individually and finally a dropping mechanism that allows us to drop messages when the queues grow too large.

Memory Allocations:

• feat(quick-protobuf-codec): reduce allocations during encoding libp2p/rust-libp2p#4782
• Arc-ing messages in gossipsub: libp2p/rust-libp2p@d28ff63

Message Prioritisation:

• refactor(gossipsub): send more specific messages to ConnectionHandler libp2p/rust-libp2p#4811

[thomaseizinger]

Memory Allocations:

• RawMessage::raw_protobuf_len can be very expensive as well because it performs many temporary allocations.

[tobidae-cb]

Following - this may be related to an issue we're also seeing - #4953

[AgeManning]

To keep people updated on the progress.

We need to update lighthouse to the latest libp2p:
#4935

We have message priority sorted:
libp2p/rust-libp2p#4914

And we have a form of time-bound message dropping:
sigp/rust-libp2p#555

We are going to combine these into a rust-libp2p fork and start testing on live networks.

[AgeManning]

Further updates.

We have a rust-libp2p fork which we are now testing. The new features we have added are:

• Backpressure in messages and priorities of messages (Gossipsub backpressure rust-libp2p#549)
• Time-based message dropping from non-critical messages (Implement message time-bound dropping rust-libp2p#555)
• Gossipsub scoring and the ability of application-level (think Lighthhouse) scoring for slow peers, which will cycle them from the mesh optimizing the gossipsub channel away from slow peers (Slow peer scoring rust-libp2p#556)
• Extended metrics so we can visualize what is happening at the gossipsub level (Queue metrics rust-libp2p#558)
• Improved speed of control messages, as this appears to be an old optimization which appears to be worse on performance. (feat(gossipsub): remove control pool rust-libp2p#559)

We are adding these to Lighthouse and thoroughly testing ahead of a new lighthouse release.

[thomaseizinger]

Great! Looking forward to seeing these in rust-libp2p!

[paulhauner]

Are you happy to close this @AgeManning? If not, perhaps we can at least drop the v4.6.0 tag.

[AgeManning]

This has been resolved in 4.6.0 and the RC.