[network] Expand STUN-based IP self-discovery to all nodes

[saketh-are]

To create a direct connection between two nodes, one of the nodes must know the public IP address of the other node’s connection listener.

Currently, SnapshotHostsInfo contains only the PeerId of the node hosting a state snapshot, not its IP address. To support connectivity for decentralized state sync, all nodes will perform IP self-discovery by querying the STUN server(s) indicated in their config. The IP address returned by the STUN server will be verified by establishing a loopback connection.

This approach is successfully used today by validators to support Tier 1 connectivity; the only change is that we will expand it to all nodes.

[saketh-are]

On further consideration, the requirements for IP discovery in Tier3 are a bit different than that of Tier1. In Tier1 nodes either configure their IP explicitly or query some trusted 3rd-party for it. They then establish self-connections to validate the IP. Finally, they sign and broadcast the information with their validator key. The ramifications of publishing an incorrect IP are much higher and more long-lasting in Tier1.

The security requirements for decentralized state sync are much lower, and it is anyway possible for peers to interfere with state part requests (which are routed messages) by dropping them. Hence we will take a much simpler approach of having nodes look up their own IP in their peer store.

The peer store is an LRU cache of limited size. We will need to ship a small change to the peer store so that nodes preferentially retain their own IP address.

Another issue is that STUN is sensitive to the details of the network setup and currently is not working for ~20% of mainnet validators. This approach avoids placing a dependency on STUN for decentralized state sync.