oci provider: fail fast, recover fast, when instance-pool/node-group is out of capacity

[jlamillan]

Which component this PR applies to?

cluster-autoscaler (OCI provider)

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds support to the OCI provider for Instance Pools to fail fast (and potentially recover fast) when an underlying instance-pool for a node-group has run out of quota / capacity.

It accomplishes this by taking a similar approach to this PR where it monitors the cloud-providers work request queue for errors relating to capacity/quota, and quickly fails any on-going scale-up operation if they are detected. Additionally, it creates "placeholder instances" for yet-to-be-fulfilled cloud instances, which allows the Cluster Autoscaler to short-circuit the --max-node-provision-time timeout and gives it (unfulfilled) instances to delete. This approach frees the Cluster Autoscaler to (1) detect the failure fast and (2) try to scale a different node-group to fulfill the missing instances (provided it meets the node requirements).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

• These changes have been running in production for a number of weeks.

Does this PR introduce a user-facing change?

support monitoring instance-pool work-requests for capacity/quota issues during scale-up

[jlamillan]

FYI @gjtempleton, @x13n thanks for mentioning that AWS and GKE take a similar approach the other week in the Autoscaling SIG.

[mwielgus]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlamillan, mwielgus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [mwielgus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment