Determine default target for broker fanout delivery attempts per second

[grantr]

Problem
What is our default target for delivery attempts per second handled by the broker fanout service?

Since fanout only attempts one delivery per trigger, this may be defined as a factor of ingress rate (#1596), for example "average fanout rate is 2x average ingress rate".

Persona:
User

Exit Criteria
We have a goal in delivery attempts per second to tune the default fanout resource requirements.

Additional Context
Part of #1552.

[grantr]

@liu-cong suggests fanout ratio of 20x (20 triggers each matching every incoming event).

[Ectelion]

Note that this task has evolved from settling on targets for a single fanout pod to determining targets for the final fanout deployment with autoscaling. This is because by settling on the Ingress max rate (which was 1000 rps with 256kb payload), we have also implicitly settled on the maximal fanout inbound rate, but a single fanout pod can not handle that load.

Based on our experiments with autoscaling (shared in the internal doc), we've determined the following targets for the final deployment:

• Maximal inbound delivery rate: ~256 Mb/s or the delivery rate supported by Ingress
• Maximal outbound delivery rate: ~2.5 Gb/s or 10x of Ingress delivery rate

More specifically, the above may transfer to:

• 1000 requests per second
• 256 kb payload
• On average, 10 triggers activated per request

The above targets assume that there are enough allocatable resources in the cluster for Fanout deployment to scale out without delay. Another thing to keep in mind is that this throughput turns out to be higher than the default Pub/Sub throughput, so, additional quotas may need to be requested.

Speaking of a single fanout pod without autoscaling, our initial experiments indicated that, for the given RPS of 1000 and the average of 10 triggers activated on each request, the payload size needs to be 32kb or less in order for the throughput to become sustainable. For reference, here is a summary of single-pod results:

[Ectelion]

It seems we are good with these targets. The rest of discussions are around proposed changes to support them.