Konnect push fails instead of throttling on HTTP 429

[mflendrich]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

If the Konnect config push results in a HTTP 429 (meaning that KIC hit Konnect's rate limit), this gets reported back to the user as an error

time="2023-05-02T11:03:53-07:00" level=error msg="Failed pushing configuration to Konnect" error="performing update for https://us.kic.api.konghq.com/kic/api/runtime_groups/REDACTED failed: failed getting current state for https://us.kic.api.konghq.com/kic/api/runtime_groups/REDACTED: loading configuration from kong: acls: HTTP status 429 (message: \"API rate limit exceeded\")"

Expected Behavior

HTTP 429 should cause KIC to throttle the config push rather than to fail.

@GGabriele says we can achieve this by leveraging an equivalent of Kong/deck#705.

Steps To Reproduce

No response

Kong Ingress Controller version

2.9.*

Kubernetes version

n/a

Anything else?

No response

[randmonkey]

Retryable is the only option to configure the kong client to retry on 429, but the parameter of retry is not configurable, and the maximum retry time could be near 10 minutes. IHAC that the retry requests may be sent after newer configuration is successfully uploaded, then the outdated configurations may be uploaded.

[pmalek]

@randmonkey Given that Update() call is synchronous after it gets triggered by the dataplane synchronizer timer and also given the fact that KongClient has a lock used in Update() how would you see the old config being applied after the new has been already been sent to datalplane(s)?

[czeslavo]

I think that given the fact that Update calls are synchronous, we should implement the backoff mechanism for Konnect on a higher level than HTTP's client (while still we can somehow try to propagate 429 headers to properly react when it happens).

I imagine we could wrap UpdateStrategy for Konnect to be aware of the backoff strategy and simply skip the update when conditions for the next retry aren't met.

[czeslavo]

Just to visualise my idea with code: 2e743f1