Azure: Fast nodegroup backoff on failed provisioning

[domenicbozzuto]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds a case to the Azure VMSS instance status check to cover an instance that failed provisioning. If the instance fails to provision (generally due to Azure being unable to provide capacity for the instance type), the error percolate to the cluster state registry and put the nodegroup into backoff (faster than if it had to wait maxNodeProvisionTime).

This is inspired by the other cloudProviders that implement similar behavior, like #4489

Testing Methodology

An nodegroup using an instance type that was actively experiencing Azure capacity issues was selected. A scale up was triggered for that nodegroup which promptly fails. The nodegroup is put into backoff in ~20s, and another compatible nodegroup can be upscaled instead, without having to wait maxNodeProvisionTime (15m):

2023-02-28T14:14:49.615Z: virtualMachineScaleSetsClient.WaitForCreateOrUpdateResult - updateVMSSCapacity for scale set ""<NODEGROUP>"" failed: Code=""ZonalAllocationFailed"" Message=""Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance"" Target=""53""
2023-02-28T14:14:49.615Z: Failed to update the capacity for <NODEGROUP> with error Code=""ZonalAllocationFailed"" Message=""Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance"" Target=""53"", invalidate the cache so as to get the real size from API
2023-02-28T14:14:49.619Z: Provisioning has failed for VM: azure:///subscriptions/<SUBSCRIPTION>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Compute/virtualMachineScaleSets/<NODEGROUP>/virtualMachines/53
2023-02-28T14:14:49.621Z: Nodegroup is nil for azure:///subscriptions/<SUBSCRIPTION>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Compute/virtualMachineScaleSets/<NODEGROUP>/virtualMachines/53
2023-02-28T14:14:49.621Z: Disabling scale-up for node group <NODEGROUP> until 2023-02-28 14:19:49.558793898 +0000 UTC m=+59470.373573234; errorClass=OutOfResource; errorCode=provisioning-state-failed

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[gandhipr]

/lgtm
/assign @tallaxes

[domenicbozzuto]

@tallaxes a gentle bump if you have a chance to take a look at this

[bpineau]

@tallaxes , @feiskyer or @nilo19 any chance one of you could have a look please? 🙇

[tallaxes]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: domenicbozzuto, tallaxes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/cloudprovider/azure/OWNERS [tallaxes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tallaxes]

/area provider/azure