Recalling AMI Release v20230211 due to it breaking Kernel upgrades

[saurav-agarwalla]

Description

If you don't upgrade the Kernel in EKS Optimized AMIs, you can stop reading.

If you DO upgrade the Kernel that the EKS Optimized AMIs ship with, you should know that a recent bug caused Kernel upgrades to behave unexpectedly in the AMI Release v20230211.

The bug was introduced as a result of this change where we added a version lock to the Kernel. This was done to make sure that customers don't accidentally end up updating the Kernel version that the AMI ships with because our tests only validate the AMI with the Kernel that it ships with and updating it to a different version could cause issues/instabilities.

What went wrong in that change was that instead of locking only the Kernel which is shipped with the AMI by default, it version locks both 4.14 and 5.4 Kernels (for the AMIs which have 5.4 as default) as seen from the output in one of the affected AMIs:

$ sudo yum versionlock
Loaded plugins: priorities, update-motd, versionlock
0:kernel-4.14.301-224.520.amzn2.
0:kernel-5.4.228-132.418.amzn2.*
0:runc-1.1.4-1.amzn2.*
0:containerd-1.6.6-1.amzn2.0.2.*
0:docker-20.10.17-1.amzn2.0.1.*
versionlock list done

As a result of this, if you upgrade the Kernel, it can end up bringing it down to 4.14. Effectively, something like the following:

$ sudo amazon-linux-extras install -y kernel-5.10
Installing kernel
Loaded plugins: priorities, update-motd, versionlock
Cleaning repos: amzn2-core amzn2extra-docker amzn2extra-kernel-5.10
...
...
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:4.14.301-224.520.amzn2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                                Version                                                                              Repository                                                               Size
============================================================================================================================================================================================================================================================================================================
Installing:
 kernel                                                              x86_64                                                              4.14.301-224.520.amzn2                                                               amzn2-core                                                               22 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 22 M
Installed size: 102 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-4.14.301-224.520.amzn2.x86_64.rpm                                                                                                                                                                                                                                             |  22 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-4.14.301-224.520.amzn2.x86_64                                                                                                                                                                                                                                                     1/1
  Verifying  : kernel-4.14.301-224.520.amzn2.x86_64                                                                                                                                                                                                                                                     1/1

Installed:
  kernel.x86_64 0:4.14.301-224.520.amzn2

Workaround and mitigation

We are working on releasing a new set of AMIs which would fix this problem and are also recalling AMI Release v20230211.

If you need to work around this in the meanwhile, you can use sudo yum versionlock delete kernel as shown in the steps below:

$ sudo yum versionlock delete kernel
Loaded plugins: priorities, update-motd, versionlock
Deleting versionlock for: 0:kernel-4.14.301-224.520.amzn2.*
Deleting versionlock for: 0:kernel-5.4.228-132.418.amzn2.*
versionlock deleted: 2

$ sudo yum versionlock
Loaded plugins: priorities, update-motd, versionlock
0:runc-1.1.4-1.amzn2.*
0:containerd-1.6.6-1.amzn2.0.2.*
0:docker-20.10.17-1.amzn2.0.1.*
versionlock list done

And then proceed with the Kernel upgrade.

[sbenderli]

Will adding a kernel versionlock in the future mean that all kernel upgrades, including kernel-livepatching, will only work after the kernel versionlock is explicitly removed?

[saurav-agarwalla]

Will adding a kernel versionlock in the future mean that all kernel upgrades, including kernel-livepatching, will only work after the kernel versionlock is explicitly removed?

That's correct. Our recommendation is still relying on our AMI releases to get the latest Kernel since that'll make sure that the Kernel version being used is the one that has passed our testing. But if there's a strong reason to upgrade the Kernel that the AMI ships with, then yes, you'll need to remove the version lock before the upgrade.

[saurav-agarwalla]

https://github.com/awslabs/amazon-eks-ami/releases/tag/v20230217 fixes this issue. Resolving this as a result.