Automated cherry pick of #16778: dns: don't use IMDS region resolver when it previously failed

[rifelpet]

Cherry pick of #16778 on release-1.30.

#16778: dns: don't use IMDS region resolver when it previously failed

For details on the cherry pick process, see the cherry pick requests page.

[rifelpet]

/hold
waiting to confirm this fixes the cluster creation of these jobs:

https://testgrid.k8s.io/kops-misc#kops-aws-external-dns

https://testgrid.k8s.io/kops-misc#kops-aws-pod-identity-webhook

https://testgrid.k8s.io/kops-misc#kops-aws-apiserver-nodes

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

For kops-aws-pod-identity-webhook, I think this is the error:

Error: error running tasks: deadline exceeded executing task ManagedFile/discovery.json. Example error: error creating ManagedFile ".well-known/openid-configuration": error writing s3://k8s-kops-ci-prow-state-store/e2e-49d63c55eb-ac683.tests-kops-aws.k8s.io/.well-known/openid-configuration (with ACL="public-read"): operation error S3: PutObject, https response error StatusCode: 400, RequestID: M0XPHJR0EZRXMPXD, HostID: d+e5r1Lc7v47eYOlLjCTJe8L/r5lmPJ6gvuY7nNY2pEU1wlEN+3LJayd33Os1KQFoddackAfxeE=, api error AccessControlListNotSupported: The bucket does not allow ACLs

So ACLs on the S3 bucket - I think we try to detect that, so maybe that's going wrong.

---

For https://testgrid.k8s.io/kops-misc#kops-aws-apiserver-nodes, we're getting further, but the most recent test failed to connect to the cluster and then could not SSH to the VMs to dump more info. It almost looked like they didn't come up at all, which is ... odd. On the toolbox dump the instances do have state=running (but I can't easily tell for how long). We do create the cluster from a template here, so it's not unlikely that something else is going wrong here.

[justinsb]

(I have no objection to delaying the backport by the way, while we figure this out!)

[rifelpet]

@justinsb this is my main concern with merging this as-is. I think a follow up is needed:

#16778 (comment)

[rifelpet]

The remaining failures are unrelated, so this should be safe to merge

/unhold