Adjust work queue rate limiter config

[tpantelis]

Scale testing revealed excessive artificial delays from the bucket rate limiter used for the informer work queues. This was due to many failure requeues/retries occurring for many items in a short amount of time such that both the exponential and bucket rate limiters kicked in resulting in delays > 2 hours.

We can make some adjustments to the rate limiter config to alleviate excessive delays.

• Increase the exponential rate limiter base delay from 5 to 50 ms. This reduces the number of re-queues before hitting the max delay from 13 to 10.

  At 5 ms:

  1: Delay: 5ms
  2: Delay: 10ms
  3: Delay: 20ms
  4: Delay: 40ms
  5: Delay: 80ms
  6: Delay: 160ms
  7: Delay: 320ms
  8: Delay: 640ms
  9: Delay: 1.28s
  10: Delay: 2.56s
  11: Delay: 5.12s
  12: Delay: 10.24s
  13: Delay: 20.48s
  14: Delay: 30s

  At 50 ms:

  1: Delay: 50ms
  2: Delay: 100ms
  3: Delay: 200ms
  4: Delay: 400ms
  5: Delay: 800ms
  6: Delay: 1.6s
  7: Delay: 3.2s
  8: Delay: 6.4s
  9: Delay: 12.8s
  10: Delay: 25.6s
  11: Delay: 30s

• Increase the bucket rate limiter burst size from 100 to 500. This is the number of items at which the limiter will start adding delays if the rate is exceeded.

• Cap the maximum delay at 5 min.

[submariner-bot]

🤖 Created branch: z_pr898/tpantelis/q_rate_limiter

[submariner-bot]

🤖 Closed branches: [z_pr898/tpantelis/q_rate_limiter]