How to pin a version of the EKS AMI

[mightyguava]

We used the terraform modules to provision some EKS worker nodes a bit over a month ago. Recently I ran terraform again with no changes, but terraform wanted changes to the launch configuration. It's due to the use of most_recent = true here: https://github.com/pingcap/tidb-operator/blob/master/deploy/modules/aws/tidb-cluster/data.tf#L7

This caused issues because the new version of the AMI lowered ulimits, causing TiKV to stop working. This has already been fixed master here. So it was kind of a pain to track down.

It'd be nice to be able to pin a version. I think by setting var.worker_ami_name_filter I should be able to pin a version? But I haven't been able to figure out what exactly I'm supposed to put there. Where do I get this version from? The var help says:

could be set to a release from amazon-eks-ami, e.g. "v20190220\

I think it's supposed to be the "Packer version" here https://docs.aws.amazon.com/eks/latest/userguide/eks-linux-ami-versions.html. It'd be nice if the docs were more clear.

[cofyc]

cc @aylei @DanielZhangQD do you have any idea?

[DanielZhangQD]

From the releases https://github.com/awslabs/amazon-eks-ami/releases, I think it should be the Packer version.
BTW, does the AMI update cause TiKV node re-creation? Does the data of TiKV persist?

[mightyguava]

Changes to the AMI version only applies to newly launched nodes. I wanted to scale up TiKV, so I added a new node, and the new TiKV instance couldn’t start

[DanielZhangQD]

@mightyguava We have set the ulimits in userdata.sh https://github.com/pingcap/tidb-operator/blob/v1.0.6/deploy/modules/aws/tidb-cluster/templates/userdata.sh.tpl, do you mean that the ulimits settings do not take effect with the new AMI?

[mightyguava]

I mean that every time I run terraform apply a new AMI gets picked up even if I didn't change anything. When I ran apply earlier to scale up my cluster, it picked up the bad AMI, but I hadn't pulled this repo yet so it didn't include the new userdata.

I added this to my module declaration to pin the AMI.

module "default-cluster" {
  source  = "./deploy/modules/aws/tidb-cluster"
  worker_ami_name_filter    = "v20190927"
}

The worker_ami_name_filter variable was what I was looking for. It just wasn't clear how to use it.

[DanielZhangQD]

@mightyguava OK, Thanks for the feedback.
I think it's OK to not pin an AMI version to leverage the potential fixes and improvements in the new AMI and we will make sure that we configure the necessary settings the TiDB requires in our scripts.
Is it OK with you?

[mightyguava]

Sure. All I was looking for from this ticket was instructions for how to pin the version. It’d be really useful if there was a link in the comments, that’s all.

[DanielZhangQD]

@mightyguava Thanks for the clarification, #1903 add the link.