EKS : Cluster destroy failing for Fargate w/ CoreDNS addon

[Howlla]

Describe the bug

When using the CoreDNS addon for a Fargate Cluster, the teardown fails and the resources created are not deleted. The error is caused by CoreDNSComputeType patch which is automatically applied to Fargate clusters.

Expected Behavior

The cluster created by cdk deploy should delete and clean up all resources using cdk destroy

Current Behavior

Delete fails and some of the resources are kept intact

This shows up in the terminal running destroy

2:12:41 PM | DELETE_FAILED        | Custom::AWSCDK-EKS-KubernetesPatch    | HelloEKSCoreDnsComputeTypePatch45115688
Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (NotFound): deployments.apps "coredns" not found\n'

at invokeUserFunction (/var/task/framework.js:2:6)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async onEvent (/var/task/framework.js:1:369)
at async Runtime.handler (/var/task/cfn-response.js:1:1573) (RequestId: a925b3d0-0e54-4c3e-af1e-07a7dc63ad3c)


 ❌  CdkStack: destroy failed Error: The stack named CdkStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [HelloEKSCoreDnsComputeTypePatch45115688]. )
    at destroyStack (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:467:1792)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CdkToolkit.destroy (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:470:187853)
    at async exec4 (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:525:53102)

The stack named CdkStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [HelloEKSCoreDnsComputeTypePatch45115688]. )

Reproduction Steps

In the terminal run cdk init sample-app --language=typescript

Replace lib/cdk-stack.ts with the code below

import * as eks from 'aws-cdk-lib/aws-eks'
import { Construct } from 'constructs';

export class CdkStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    var cluster = new eks.FargateCluster(this, 'HelloEKS', {
      version: eks.KubernetesVersion.V1_27,
    });
    new eks.CfnAddon(this, 'MyCoreDnsAddon', {
      addonName: 'coredns',
      clusterName: cluster.clusterName,
    })
  }
}

run cdk deploy

wait for deployment to complete

run cdk destroy

Possible Solution

Reason for bug:

The k8s-patch for computeType annotations are applied with no dependency on the coreDNS addon themselves.
When we run the cdk destroy command, first the addons are removed. This means that the coredns deployment is deleted first and then the kubectl patch is attempted to be removed. This results in the console output Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (NotFound): deployments.apps "coredns" not found\n'

Possible Fix

Set removal policy for either the CoreDNSComputeType Patch or the coreDNS addon to RemovalPolicy.Retain

Additional Information/Context

No response

CDK CLI Version

2.100.0(build e1b5c77)

Framework Version

No response

Node.js Version

20.5.0

OS

Mac Sonoma

Language

TypeScript

Language Version

No response

Other information

Related Issue: #14968 - it states that CDK should be using core addons as opposed to relying on some default configuration from EKS.

[pahud]

I am not sure if we should add any default retain policy on the CoreDnsComputeTypePatch but adding the removal policy from the cfnAddOn as proposed in this PR makes sense to me.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.