Error: Post https://xxxx.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp xxx:443: i/o timeout

[max-rocket-internet]

On every example on first run:

Error: Post https://C39B1571479D46ED13A85BC0105510E2.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp 44.229.104.205:443: i/o timeout

  on ../../aws_auth.tf line 45, in resource "kubernetes_config_map" "aws_auth":
  45: resource "kubernetes_config_map" "aws_auth" {

This is because theres a small between when the cluster endpoint is returned after creation but is not available.

If you rerun terraform apply the configmap is created fine.

Quite annoying 😅

[dpiddockcmp]

Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?

[xposix]

I keep on having the same issue. IMO, the kubernetes provider should wait / retry few times before coming back with an error.

[max-rocket-internet]

Related: hashicorp/terraform-provider-kubernetes#96

[max-rocket-internet]

Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?

I think a socket error should be retried by the provider.

In the mean time, could we make a work around somehow? Like a null resource that sleeps for 10 seconds after cluster creation?

[ivanich]

I'm using kubectl hack for this but it requires kubectl and a lot of depends_on and in general it ugly

  provisioner "local-exec" {
    command = "until kubectl --kubeconfig ${path.root}/kubeconfig_${var.cluster_id} -n kube-system get pods >/dev/null 2>&1;do echo 'Waiting for EKS API';sleep 5;done"
  }
}

in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.

[max-rocket-internet]

in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.

You can just use release v7.0.1, no?

[ivanich]

You can just use release v7.0.1, no?

7.0.1 has faulty kubernetes provider commit, so staying with 7.0.0 for now, and if it's will be an issue for a while fork is the way to go at least we can have new features provided by upstream.

[max-rocket-internet]

7.0.1 has faulty kubernetes provider commit

No it doesn't 🙂 I made it specifically the commit before: #630

https://github.com/terraform-aws-modules/terraform-aws-eks/releases/tag/v7.0.1

[ivanich]

No it doesn't 🙂 I made it specifically the commit before: #630

Nice, thanks for the tip, haven't checked commits since kubernetes provider merge. 👍

[dpiddockcmp]

This is definitely a "feature" of the AWS EKS API. It returns creation complete before the cluster is actually ready to be used 😞

This requires all tools creating clusters to have a retry on their first access. e.g.:

• eksctl's WaitForControlPlane
• logic in the "quickstart" example run_command
• v7 of this module local_exec

We need to go back to having a retry loop hack. Which kind of defeats the point of moving to the kubernetes provider 😞

I was comparing with GCP's kubernetes offering. Their service only returns ready once the cluster is actually usable. You can easily chain GKE creation with the kubernetes terraform provider.
Test code here

Raised an issue in the containers roadmap here . See what they say

[barryib]

FYI, I opened a PR in the AWS provider to wait for kubernetes endpoint hashicorp/terraform-provider-aws#11426. I don't know if it the right quick win before aws/containers-roadmap#654 get solved. Feedbacks are welcome.

[dpiddockcmp]

Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.

Can we go back to kubectl and forget this whole exercise?

Edit: Raised an issue with the k8s provider. See what they say. hashicorp/terraform-provider-kubernetes#719

[max-rocket-internet]

Can we go back to kubectl and forget this whole exercise?

I'm leaning towards this also.

[max-rocket-internet]

What should we do? Revert the k8s provider change? Or implement something like in this PR? #639

[dpiddockcmp]

Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.

Solved this problem with a null_data_source to enforce the ordering. So the main issue remaining is the endpoint not always being available after first creation. #639 looks like it might be the solution but that PR needs a little cleaning up.

[max-rocket-internet]

Solved this problem with a null_data_source to enforce the ordering

Great!

[jamescross91]

Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.

Solved this problem with a null_data_source to enforce the ordering. So the main issue remaining is the endpoint not always being available after first creation. #639 looks like it might be the solution but that PR needs a little cleaning up.

Just run into this on master - can you give an example of how we can fix using a null_data_source?

[dpiddockcmp]

Just run into this on master - can you give an example of how we can fix using a null_data_source?

Basically:

• Have a data null_data_source nds with inputs:
  • cluster_name. Doesn't matter where this comes from
  • kubernetes_config_map.aws_auth.id
• Use data.null_data_source.nds.outputs["cluster_name"] as your cluster_name variable in the node groups

You'd have to fork the module to do it. It's not usable from outside. It only solves the race condition on cluster first creation with node groups.

[erabug]

Today I ran into both this issue and the Error: configmaps "my-config" already exists issue @dpiddockcmp described. I switched to the branch from #639 and it solved the problem 🎉

[max-rocket-internet]

Resolved in #639

[ngortheone]

@max-rocket-internet I am not familiar with the release cadence. When is this fix going to be available for regular users?

[barryib]

@ngortheone it's already available in v8.0.0

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.