☸️ EKS Cluster Consolidation

[Gary-H9]

Note

If we are looking at SageMaker etc in 6+ months then this should be done.

Important

If we are looking at them soon then this does not need to be done.

Brief summary of what this Epic is

We currently run a number of EKS clusters which are about to go into extended support - 31st January.

Additionally they use Flux for deployment and we would like to stop using this technology.

Issues in Epic

• 🔥 Decommission analytical-platform-development cluster #2945
• 🔥 Decommission analytical-platform-production Cluster #2946
• 🔥 Decommission analytical-platform-management-production Cluster #2947
• 🔥 Decommission analytical-platform-management-production Cluster #2948
• 🚚 Migrate Analytical Platform EFS #2949
• 🗺️ DNS #2950
• ☸️ EKS Cluster Consolidation #2943
• 💨 Apps and Tools EKS Cluster: Create Airflow Node Group #1950
• 🕵️ Investigate the Usage of an Alternative CNI for EKS cluster to be built in MP #1371

[julialawrence]

I think we should do this unless the horizon for new tooling is less than 90 days because

1. We don't know that SageMaker will be the tool of choice although we strongly suspect it.
2. Any migration to new tooling offerings will almost certainly be gradual rather than shut-off-turn-on which means maintaining both environments out of the same cluster will make things easier
3. The window between "we're starting adoption of new tooling" vs "we're making new tooling available to users" is very likely to take us beyond the 6 months window anyway.
4. We anticipate being able to move containerised workloads to CP at some point and kicking the tires on migration would probably be a good thing
5. With EFS being the unknown, I'm not sure the effort involved is actually that big so therefore narrower window to make it worth it.

[julialawrence]

We will deliver this work as part of the refined architecture for AP.