Increase push fanout from 6 to 9

[gregcusack]

Problem

A Push Fanout of 6 is too small, resulting in the following:

• slow message propagation
• sparse network

The key here is that by increasing push fanout, gossip is able to reach more nodes in fewer hops. The pruning logic then removes the redundant paths, leaving us with a more dense MST without significant overhead (in steady state).

All numbers and claims below are based off of a simulated gossip framework here: gossip-sim
Data from simulated gossip framework can be found here: Grafana Dashboard

Summary of Changes

• Increase CRDS_GOSSIP_PUSH_FANOUT from 6 to 9, while leaving CRDS_GOSSIP_PUSH_ACTIVE_SET_SIZE at 12

Supporting Data

Increasing Push Fanout from 6 to 9:
Pros

• Reduces the mean and median number of hops per gossip iteration by ~14.5% (~1 hop)
  • Result: More nodes receive gossip messages sooner
• Reduces Last Delivery Hop (essentially message latency) by 16.6% (3 hops)
• Reduces Relative Message Redundancy by ~2.8%
• Reduces the number of total messages sent in the network by ~1-2%
• Reduces mean # of egress messages for 99.7% of validators
• The increase in the number of prunes in steady state is nominal

Cons

• Slight decrease in network coverage by 0.1% -- translates to an average of ~5 additional nodes (out of 4210 - including unstaked nodes) left out of an origin's MST
• Some higher staked nodes see a slight increase in egress messages
• obviously, each origin node will be sending to 3 additional nodes per message

Fixes #

[codecov]

Codecov Report

Merging #32323 (857d4e2) into master (91186d3) will increase coverage by 0.0%.
The diff coverage is n/a.

❗ Current head 857d4e2 differs from pull request most recent head 2d7b7de. Consider uploading reports for the commit 2d7b7de to get more accurate results

@@           Coverage Diff           @@
##           master   #32323   +/-   ##
=======================================
  Coverage    82.0%    82.0%           
=======================================
  Files         773      773           
  Lines      209745   209670   -75     
=======================================
- Hits       172153   172096   -57     
+ Misses      37592    37574   -18