AWSMachinePool reconciliation stuck if ASG could not be created

[AndiDog]

/kind bug

What steps did you take and what happened:

We created a new cluster having a AWSMachinePool. Upon adding a second AWSMachinePool, for some reason, the IAM instance profile wasn't ready/found yet and the ASG creation failed. That's out of scope for the issue. Just mentioning the error for completeness and visibility:

failed to create AWSMachinePool: failed to create autoscaling group: ValidationError: You must use a valid fully-formed launch template. Value (nodes-nodepool2-theclustername) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name

This means that the AWSMachinePool reconciler had successfully created the launch template (and set the IAM instance profile name on the launch template), but then failed on ASG creation.

The problem is that reconciliation is now stuck due to a bug: on next Reconcile attempt, ReconcileLaunchTemplate detects a diff, calls canUpdateLaunchTemplate and that in turn calls CanStartASGInstanceRefresh. In there, an AWS request DescribeInstanceRefreshesInput{AutoScalingGroupName: aws.String(scope.Name())} is made and fails because the ASG doesn’t exist yet. The error is:

E1123 13:38:44.326581       1 logger.go:83] "failed to reconcile launch template" err=<
	ValidationError: AutoScalingGroup name not found - AutoScalingGroup theclustername-nodepool2 not found
		status code: 400, request id: 17090c90-0275-40dd-9a1d-2e2994105934

We must handle, log and ignore that error. If the ASG does not exist, CanStartASGInstanceRefresh should return false. Or ReconcileLaunchTemplate should not even consider instance refresh through some other means.

What did you expect to happen:

Reconciliation continues and eventually succeeds.

Environment:

• Cluster-api-provider-aws version: roughly v2.2.4

[richardcase]

/triage accepted