KIM should stop processing any action on Gardener shoot after configured timeout

[koala7659]

Description

All operations processed by the KIM service in the Runtime CR controller reconciliation loop should be interrupted after the configured timeout period has elapsed. Each runtime operation kind (provisioning / deprovisioning / upgrade) should have its own duration timeout defined.

If an operation times out:

• The Runtime CR should be marked as "Failed" state
• A specific condition should be added, indicating which operation has timed out

AC:

• KIM deployment should be configured with three time-out values for each runtime operation type (provisioning / deprovisioning / upgrade)
• Runtime operations should be interrupted after when timeout period has passed
• RuntimeCR Status should be set correctly when an operation has timed out

Relates to:
#193

Recovery
If the timeout during Update function occurs - user (KEB) can fix broken configuration that has timeout by manually deleting annotation in the Runtime instance:
kyma-project.io/runtime-operation-started:
In such a case Runtime reconciler will Patch Gardner shoot with fixed configuration and will start next upgrade cycle. If the operation succeedes on Gardener side the Runtime CR status will be switched in Ready state.

[koala7659]

Output examples:

Output

Provisioning timeout:

 status:
    conditions:
    - lastTransitionTime: "2024-08-08T11:17:23Z"
      message: Shoot creation timeout
      reason: ShootCreationTimeout
      status: "False"
      type: Provisioned
    state: Failed

Deprovisioning timeout:

 status:
    conditions:
    - lastTransitionTime: "2024-08-08T11:17:23Z"
      message: Runtime deprovisioning timeout
      reason: ShootDeletionTimeout
      status: "False"
      type: Deprovisioned
    state: Failed

Upgrade timeout:

 status:
    conditions:
    - lastTransitionTime: "2024-08-08T11:17:23Z"
      message: Shoot reconcile timeout
      reason: ShootProcessingTimeout
      status: "False"
      type: Provisioned
    state: Failed

[piotrmiskiewicz]

Question: The recovery scenario described above is about "update" operation. What about deprovisioning, should KEB remove the annotation or not?

[piotrmiskiewicz]

Question: update scenario. First update - KEB changes machineType, KIM do the proper changes in the shoot but a timeout occurs. KIM set the status failed. Then, let's say - one day after, KEB is sending another update (for example an autoscaler max change). Should KEB do something with the annotation mentioned above?

[tobiscr]

Decided to descope this feature as it could cause side-effects we don't want to cope with.