Supported AMI types on Modernisation Platform EKS(/Cloud Platform EKS)

[jhpyke]

During the build out of the EKS Cluster in Modernisation Platform, the question of how tightly bound we want this compute we're setting up (which is explicitly intended to be a temporary solution) to be with the current Cloud Platform Compute. Specifically, the issue of support for BottleRocket OS. Currently all our compute internally is using AWS_L2 Linux AMIs, including the current Airflow Clusters, as well as the Cloud Platform AMIs.

Bottlerocket includes a number of features that may be desirable, but it is unclear what knock on implications there may be to supporting an AMI that's not aligned with the rest of the estate. We likely would want to either get buy-in across the estate for bottlerocket support, or to understand where bottlerocket and L2 diverge, so that any implications to supporting it in the short to mid term are fully understood and we're going to be able to seemlessly migrate existing customer tools/work onto the new cluster.

To that end, I'd suggest it's worth creating an ADR on our part on if we want to pursue bottlerocket, as this document will be valuable in gaining buy-in across the org if we want to pursue, and will be a useful reference if we go with L2.

There's a ticket that breaks down some of the pros and cons as a user journey. Key takeaway is that Bottlerocket likely offers improvements relevant to our largely containerised jobs, as it is specifically optimised for containerised workflows. It will, however, inherently be disjoint with the state of L2 - unix core function may change between the two where operations that work one way on L2 will have to do done another for bottlerocket.

For a specific example, looking at update strategy, there's no 'in-place' installation of packages at runtime, as this is all managed via images. This means we'd need a different update strategy for core packages if this is relevant for any of our Tools/Apps on BR vs L2.

[julialawrence]

I'm curious about the usecase for these for us right now. More details on that would be good.

[jhpyke]

I'd be keen to see performance of some of our high-throughput workloads on there (a lot of technical hurdles before that can happen, such as data account access, IAM, etc.) Theoretically the lightweight nature of BR may be of benefit to some of our High-Memory Node tasks, which are already butting against memory limitations, and it would allow us to simplify the security policy situation, as BR has a lot of built in security options that are sensible to airflow. Bottlerocket is also directly supported by AWS, which should facilitate proper support of the product via our enterprise agreement, so there's not as much of a concern as much about the tech becoming outmoded/incompatible over time.

[jhpyke]

From AWS's own docs on Bottlerocket:

Note

Can I move my containers running on Amazon Linux 2 to Bottlerocket?

• Yes, you can move your containers across Amazon Linux 2 and Bottlerocket without modifications.

Note

Q: When should I not use Bottlerocket?

• If your operational workflows to run containers involve installing software on the host OS with yum, directly ssh-ing into instances, customizing each instance individually, or running a third-party ISV software that is not containerized (e.g., agents for logging and monitoring), Amazon Linux 2 may be a better fit. Bottlerocket is optimized to run and manage large containerized deployments and does not easily allow many of these activities.

I'm less familar with the AP apps, but it seems likely that most(/all?) of our existing use cases would work interchangeably on L2/Bottlerocket

[Gary-H9]

Pro's and Con's

(from the ticket mentioned above)

Amazon Linux:

• Versatility: General-purpose OS, suitable for a wide range of applications and services.
• Customisation: Offers more flexibility for installing software and making configuration changes.
• Management: Supports traditional management methods like SSH, more familiar to system administrators.
• Compatibility: Well-supported and compatible with a wide range of software and services.
• Performance: May not be as optimised for container workloads as Bottlerocket.

Bottlerocket OS:

• Purpose-Built: Designed specifically for running containers, making it a good fit for EKS.
• Security: Offers enhanced security features, including read-only root filesystem and eBPF-based networking.
• Performance: Optimised for high performance and reduced resource usage, beneficial for containerised applications.
• Updates: Supports atomic updates, reducing the risk of update failures.
• Management: Requires container orchestrators like EKS for management, not suitable for direct SSH access or customisation.

[bagg3rs]

So zero modification for security and performance gains?
If we do find an issue we can migrate workloads with relative ease?
There seems to be not much risk especially atm to try bottle 🚀

[jacobwoffenden]

Bottlerocket has been implemented into the new cluster