Skip to content

Commit

Permalink
fix(eks): k8s resources accidentally deleted due to logical ID change (…
Browse files Browse the repository at this point in the history 
…#12053)

The `KubernetesManifest` construct used `kubectl apply` for both CREATE and UPDATE operations. This means that if, for example, two manifests had resources with the same k8s name (`metadata.name`), the second manifest created will not fail, but rather override the resource definition.

As a consequence, if the logical ID of a `KubernetesManifest` resource was changed (without a change in the physical name), CFN would perform a replacement process which involves a CREATE of the new resource and then a DELETE of the old one. Since the CREATE operation was implemented through `apply`, it succeeded (with no-op) but then the DELETE operation would delete the resource. The result is that the resource was deleted.

The solution is to use `kubectl create --save-config` instead of `kubectl apply` for CREATE operations. This yields the desired CREATE semantics (dah!).

Now, if a `KubernetesManifest` resource is defined with a K8S object name that already exists, the CREATE operation will fail as expected. The logical ID change scenario (resource replacement), would also issue a CREATE operation first which will fail.

To change logical IDs of `KubernetesManifest` resources, users will have to either delete the old resource or change its physical name.

Since this is quite hard to test (due to multi-phase deployments and failure modes), this was tested manually:

1. Defined a manifest with logical name X1 and physical name Y1 -> CREATE was issued
2. Changed logical name to X2 (physical remains Y1) -> update failed because CFN issues a CREATE operation first (#10397)
3. Changed also the physical name to Y2 -> deploy succeeded, new resource created, old resource pruned.

This fixes #10397


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
Elad Ben-Israel authored Dec 13, 2020
1 parent d276b02 commit 019852e
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 20 deletions.
5 changes: 4 additions & 1 deletion packages/@aws-cdk/aws-eks/lib/kubectl-handler/apply/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ def apply_handler(event, context):

logger.info("manifest written to: %s" % manifest_file)

if request_type == 'Create' or request_type == 'Update':
if request_type == 'Create':
# --save-config will allow us to use "apply" later
kubectl('create', manifest_file, '--save-config')
elif request_type == 'Update':
opts = []
if prune_label is not None:
opts = ['--prune', '-l', prune_label]
Expand Down
38 changes: 19 additions & 19 deletions packages/@aws-cdk/aws-eks/test/integ.eks-cluster.expected.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3836,7 +3836,7 @@
},
"/",
{
"Ref": "AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956S3BucketE0699DCA"
"Ref": "AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aS3BucketCF0594BD"
},
"/",
{
Expand All @@ -3846,7 +3846,7 @@
"Fn::Split": [
"||",
{
"Ref": "AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956S3VersionKey09D9D914"
"Ref": "AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aS3VersionKey1CB38323"
}
]
}
Expand All @@ -3859,7 +3859,7 @@
"Fn::Split": [
"||",
{
"Ref": "AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956S3VersionKey09D9D914"
"Ref": "AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aS3VersionKey1CB38323"
}
]
}
Expand All @@ -3881,11 +3881,11 @@
"Arn"
]
},
"referencetoawscdkeksclustertestAssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3Bucket43161A4BRef": {
"Ref": "AssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3Bucket7C9DE002"
"referencetoawscdkeksclustertestAssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3Bucket4AFE6229Ref": {
"Ref": "AssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3BucketFC3C258F"
},
"referencetoawscdkeksclustertestAssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3VersionKeyDF17FE61Ref": {
"Ref": "AssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3VersionKey45F6B914"
"referencetoawscdkeksclustertestAssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3VersionKey49D5E273Ref": {
"Ref": "AssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3VersionKey6820D47C"
},
"referencetoawscdkeksclustertestVpcPrivateSubnet1Subnet32A4EC2ARef": {
"Ref": "VpcPrivateSubnet1Subnet536B997A"
Expand Down Expand Up @@ -4541,17 +4541,17 @@
"Type": "String",
"Description": "Artifact hash for asset \"daeb79e3cee39c9b902dc0d5c780223e227ed573ea60976252947adab5fb2be1\""
},
"AssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3Bucket7C9DE002": {
"AssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3BucketFC3C258F": {
"Type": "String",
"Description": "S3 bucket for asset \"340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2\""
"Description": "S3 bucket for asset \"81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74\""
},
"AssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2S3VersionKey45F6B914": {
"AssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74S3VersionKey6820D47C": {
"Type": "String",
"Description": "S3 key for asset version \"340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2\""
"Description": "S3 key for asset version \"81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74\""
},
"AssetParameters340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2ArtifactHashEDD75501": {
"AssetParameters81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74ArtifactHash1DF738E9": {
"Type": "String",
"Description": "Artifact hash for asset \"340a58e595482ebd1921fce6a3eb5df2c7afea6183bb3da2a531bd7f48e776e2\""
"Description": "Artifact hash for asset \"81ef9ae09d999514914c8c39a2f87e135a40bd56cd33f2fad771824d5072fd74\""
},
"AssetParametersb075459e6bf309093fbd4b9a9e576a5f172b91c14d84eedb0f069566f6abb0deS3Bucket14156880": {
"Type": "String",
Expand Down Expand Up @@ -4601,17 +4601,17 @@
"Type": "String",
"Description": "Artifact hash for asset \"a69aadbed84d554dd9f2eb7987ffe5d8f76b53a86f1909059df07050e57bef0c\""
},
"AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956S3BucketE0699DCA": {
"AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aS3BucketCF0594BD": {
"Type": "String",
"Description": "S3 bucket for asset \"25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956\""
"Description": "S3 bucket for asset \"fc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384a\""
},
"AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956S3VersionKey09D9D914": {
"AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aS3VersionKey1CB38323": {
"Type": "String",
"Description": "S3 key for asset version \"25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956\""
"Description": "S3 key for asset version \"fc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384a\""
},
"AssetParameters25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956ArtifactHash2F859D25": {
"AssetParametersfc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384aArtifactHash25E56295": {
"Type": "String",
"Description": "Artifact hash for asset \"25aed688c0803654674984565f9e68dec4fbd4f8427b6e9db1be3c61b4fa2956\""
"Description": "Artifact hash for asset \"fc3e757b1108608694a6c0e92715f352dc4c7ca05a94991b95050f3cbe71384a\""
},
"SsmParameterValueawsserviceeksoptimizedami118amazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter": {
"Type": "AWS::SSM::Parameter::Value<String>",
Expand Down

0 comments on commit 019852e

Please sign in to comment.