discovery - [feature]: implement bandwidth based, daemon wide gossip rate limiting.

[Roasbeef]

Is your feature request related to a problem? Please describe.

Today we rate limit based on the the number of messages a peer sends, within the syncer:

lnd/discovery/syncer.go

Lines 392 to 396 in 3526f82

	// rateLimiter dictates the frequency with which we will reply to gossip
	// queries from a peer. This is used to delay responses to peers to
	// prevent DOS vulnerabilities if they are spamming with an unreasonable
	// number of queries.
	rateLimiter *rate.Limiter

.

With this, we're able to rate limit the messages sent during the interactive sync/reconciliation protocol, but not also what we'll send/recv from peers as a part of the normal gossip trickle.

We also rate limit based on the number of channel updates sent/recv'd:

lnd/discovery/gossiper.go

Lines 484 to 491 in 3526f82

	// chanUpdateRateLimiter contains rate limiters for each direction of
	// a channel update we've processed. We'll use these to determine
	// whether we should accept a new update for a specific channel and
	// direction.
	//
	// NOTE: This map must be synchronized with the main
	// AuthenticatedGossiper lock.
	chanUpdateRateLimiter map[uint64][2]*rate.Limiter

.

These values were set long ago, and not properly tuned before settling in a value. These values also don't give users precise control w.r.t exactly how much bandwidth we'll dedicate to gossip data. By using more precise, yet global rate limiting, we'd also be able to give node operators first-class telemetry w.r.t how much bandwidth is actually being used for gossip.

This is a brainstorming issue to explore switching all the gossip related rate limiting to a stricter/simpler bandwidth based approach.

Describe the solution you'd like

Simplify and unify rate limiting, using Go's token bucket rate limiter. In other areas we set the limit based on the number of messages. Instead, I propose we set the limit based on a bytes/second target.

Implementation wise, we may want to have this closer to the peer, occupying the role of the current msg queue into the gossiper. we'll want to take care that we avoid unnecessary blocking and also compromising the uptime of an existing channel based on bandwidth based rate limiting.

[morehouse]

Implementation wise, we may want to have this closer to the peer, occupying the role of the current msg queue into the gossiper.

Sounds good for incoming messages.

Might be a little tricky for outgoing messages. We don't want the outgoing rate limit at the peer level to bottleneck the syncer/gossiper, causing outgoing messages to get queued up and waste resources.