Peers Not Stable in 24.12.1

[StefanBratanov]

Users reported that Teku struggles to get enough peers since 24.12.1

1. A user setting the p2p-peer-upper-bound to 256, used to hit 256 but since 24.12.1 they're getting around 100 - 150:
   https://discord.com/channels/697535391594446898/697539289042649190/1323510723921580116
2. Another user experienced the same issue. They're setting the p2p-peer-upper-bound to 256, used to hit 256 but since 24.12.1 they're getting around 85 - 125:
   https://discord.com/channels/697535391594446898/697539289042649190/1323903016428113982

[mehdi-aouadi]

Some investigation results:

• The described issue didn't happen in any of our nodes (holesky/mainnet)
• Our mainnet nodes are configured with 120 upper bound peer limit and they hit that target:

• Our Holesky nodes are configured with 160 upper bound peer limit and hit that target too

• Some Holesky boot nodes are configured with 300 upper bound peer limit and they also manage to hit that target
  The Holesky boot node are running an older version: 24.10.3

• The only networking related change we introduced in the 24.12.1 was the Deprecate TTFB, RESP_TIMEOUT, introduce MAX_CONCURRENT_REQUESTS #8912 but it doesn't introduce any peer limiting or disconnecting logic

No conclusions can be made based on our nodes logs for the moment

[mehdi-aouadi]

Some new observations:

• When setting the p2p-peer-upper-bound to 250 we're unable to reach that target with the 24.12.1 (tested on nightly-01), stuck at around 180 peers
• The same issue was consistent after multiple node restart
• There are many TOO_MANY_PEERS disconnect reasons
• Rolling back to 24.12.0 solves the issue and we're able to reach even 300 peers when setting p2p-peer-upper-bound to 300
• This issue is observed only when the p2p-peer-upper-bound is set to a value > 180

[tbenr]

so seems like #8912 is the key:

upperbound set to 300 peers

with it, when we approach ~180 peers we see a lot of drops due to PING timeout

without it:

so applying MAX_CONCURRENT_REQUESTS to ping, with reaching ~180 cause us to start queuing more than 2 outbound pings for each peers and it is interpreted as remote peer not responding while is our throttling kicking in

  void sendPeriodicPing(final Eth2Peer peer) {
    if (peer.getUnansweredPingCount() >= eth2RpcOutstandingPingThreshold) {
      LOG.debug("Disconnecting the peer {} due to PING timeout.", peer.getId());
      peer.disconnectCleanly(DisconnectReason.UNRESPONSIVE).ifExceptionGetsHereRaiseABug();
    } else {