Karpenter Day 2 Operations - Lifecycle Management!

[prad9192]

Hey everyone,

We're looking to gather some insights and best practices around Day 2 operations with Karpenter. Specifically, we're interested in how other organizations are managing the lifecycle of Karpenter, including upgrades and ongoing maintenance.

Our Current Approach (and Challenges)

Currently, we use helm template to render all the Karpenter Kubernetes manifests (CRD, Deployment, etc.) within a Git repository. We then leverage a Cluster resource set to apply these manifests to our target clusters from an management cluster (using CAPI).

However, we've encountered some challenges with this approach, especially during upgrades:

• Cumbersome and Time-Consuming: The process of updating manifests and reapplying them across clusters has become increasingly cumbersome and time-consuming with recent Karpenter upgrades.
• Custom Controller Dependency: We have a custom component (Kubernetes controllers) that interacts with Karpenter. This component takes Machinepool specifications from developers and creates corresponding Nodepools on the target cluster. Each Karpenter upgrade requires us to:
  • Update the Go module in our custom controller to reference the new Karpenter version.
  • Modify the business logic of our controller to accommodate changes in the Nodepool/Ec2nodeclass specification (e.g., kubelet configuration moving from Nodepool to EC2NodeClass).

Questions for the Community

• How are you managing Karpenter upgrades and lifecycle in your organization?
• Are you using Helm, Kustomize, or other tools to manage Karpenter manifests?
• How are you handling dependencies between Karpenter and other components, especially in the context of upgrades?
• Do you have any strategies for minimizing disruption and ensuring smooth transitions during Karpenter upgrades?
• Any advice on automating the upgrade process and reducing manual intervention?

We're eager to hear your thoughts, experiences, and any recommendations you might have.

Thanks in advance for your inputs!