SSM: Cannot update a value on deploy

[asdfractal]

Describe the bug

When I change the value of a resource that is creating an SSM parameter, the deploy fails with error
/infra/redacted/ecr/registry already exists in stack

In this case it is renaming the ecr, which we are storing as SSM parameters for cross stack references.

To be transparent - I'm very new to using cdk and not in a platform team, we just want some test pipelines set up. So I'm not sure if this is a bug but what I'm looking for is a way I could update an ssm parameter value when updating a stack deployment.

Expected Behavior

Running cdk diff gives this output

[-] AWS::SSM::Parameter infra-redacted-app-pr-registry infraredactedappprregistry6D04DDAA destroy
[-] AWS::SSM::Parameter infra-redacted-app-pr-repository infraredactedappprrepository7BAF1037 destroy
[+] AWS::SSM::Parameter infra-redacted-app-jgtest-registry infraredactedappjgtestregistryFBE1EDA9
[+] AWS::SSM::Parameter infra-redacted-app-jgtest-repository infraredactedappjgtestrepository21F07B04
[~] AWS::ECR::Repository redactedAppECR/ECR redactedAppECRC28DCB14 replace
 └─ [~] RepositoryName (requires replacement)
     ├─ [-] redacted-app-pr
     └─ [+] redacted-app-jgtest

Expected behavior is that this is executed as shown in the diff. Destroying the current parameter and then creating the new one.
Even further expected behavior would be that it is not required to destroy it but instead update the value of the parameter...but I can't find any information about doing this.

Current Behavior

Deploy failed due to resource already existing

redactedAPP-CICDResources:  start: Building 668fd9aed2ce5c9336e190f8d0b9535d12129024d109d7ba4f85e5f3b6578376:123456789999-ap-southeast-2
redactedAPP-CICDResources:  success: Built 668fd9aed2ce5c9336e190f8d0b9535d12129024d109d7ba4f85e5f3b6578376:123456789999-ap-southeast-2
redactedAPP-CICDResources:  start: Publishing 668fd9aed2ce5c9336e190f8d0b9535d12129024d109d7ba4f85e5f3b6578376:123456789999-ap-southeast-2
redactedAPP-CICDResources:  success: Published 668fd9aed2ce5c9336e190f8d0b9535d12129024d109d7ba4f85e5f3b6578376:123456789999-ap-southeast-2
redactedAppPackerCICDResources (redactedAPP-CICDResources): deploying... [1/1]
redactedAPP-CICDResources: creating CloudFormation changeset...
12:05:37 PM | CREATE_FAILED        | AWS::SSM::Parameter                   | infraredactedappjgtestregistryFBE1EDA9
/infra/redactedapp/ecr/registry already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-C
ICDResources/64287a60-5525-11ef-b8b1-02c30c85c353

12:05:37 PM | CREATE_FAILED        | AWS::SSM::Parameter                   | infraredactedappjgtestrepository21F07B04
/infra/redactedapp/ecr/repository already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP
-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353


 ❌  redactedAppPackerCICDResources (redactedAPP-CICDResources) failed: Error: The stack named redactedAPP-CICDResources failed to deploy: UPDATE_ROLLBACK_COMPLETE: /infra/redactedapp/ecr/registry already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353, /infra/redactedapp/ecr/repository already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353
    at FullCloudFormationDeployment.monitorDeployment (/home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:446:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:449:199716)
    at async /home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:449:181438

 ❌ Deployment failed: Error: The stack named redactedAPP-CICDResources failed to deploy: UPDATE_ROLLBACK_COMPLETE: /infra/redactedapp/ecr/registry already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353, /infra/redactedapp/ecr/repository already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353
    at FullCloudFormationDeployment.monitorDeployment (/home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:446:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:449:199716)
    at async /home/username/.nvm/versions/node/v20.13.1/lib/node_modules/aws-cdk/lib/index.js:449:181438

The stack named redactedAPP-CICDResources failed to deploy: UPDATE_ROLLBACK_COMPLETE: /infra/redactedapp/ecr/registry already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353, /infra/redactedapp/ecr/repository already exists in stack arn:aws:cloudformation:ap-southeast-2:123456789999:stack/redactedAPP-CICDResources/64287a60-5525-11ef-b8b1-02c30c85c353

Reproduction Steps

const SSMparam = new SSM.StringParameter(this, 'ssmParam', {
  parameterName: '/ssmParam/name',
  stringValue: 'value',
  description: 'description',
});

Deploy the parameter.
Change the value to something else, with the same name.

const SSMparam = new SSM.StringParameter(this, 'ssmParam', {
  parameterName: '/ssmParam/name',
  stringValue: 'othervalue',
  description: 'description',
});

Deploy again.

Possible Solution

Allow an update of parameter values easily, either through destroy/re-create or just an update to the value. I'm not sure if this is the intended way to update an SSM but there doesn't seem to be any documention around it through SSM.

If this constructor is not intended for update, then maybe there should be something like create_or_update?

Additional Information/Context

No response

CDK CLI Version

2.151.0 (build b8289e2)

Framework Version

No response

Node.js Version

v20.13.1

OS

Ubuntu

Language

TypeScript

Language Version

Version 5.5.4

Other information

No response

[ashishdhingra]

@asdfractal Good morning. The above error is thrown by the CloudFormation, not the CDK. However, somehow, I'm unable to reproduce the issue at my end. Used the below code (used CDK version 2.151.0):

import * as cdk from 'aws-cdk-lib';
import * as ssm from 'aws-cdk-lib/aws-ssm';

export class CdktestStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const ssmParam = new ssm.StringParameter(this, 'ssmParam', {
      parameterName: '/someparameter/name',
      stringValue: 'value',
      description: 'description',
    });
  }
}

• Deployed the stack.
• Updated stringValue to othervalue.
• Re-deployed the stack.

This created new version of SSM parameter:

Are you re-deploying the SSM parameter as part of same stack or a different stack? I used the same stack.

Thanks,
Ashish

[asdfractal]

Hi @ashishdhingra. Thanks for the reply and for testing. Unfortunately my reproduction steps were not great, sorry about that.
I am re-deploying as part of the same stack.

I'm having difficulty reproducing this, I've tried to simulate it with a simple stack but it works as expected. So it is most likely due to an issue with our infrastructure. I have only recently joined a new team and just working on it, so please forgive my ignorance on the subject.

This is the comparison between the real app stack diff, and my test stack

Resources
[-] AWS::SSM::Parameter infra-redacted-app-pr-registry infraredactedappprregistry6D04DDAA destroy
[-] AWS::SSM::Parameter infra-redacted-app-pr-repository infraredactedappprrepository7BAF1037 destroy
[+] AWS::SSM::Parameter infra-redacted-app-pr-test-registry infraredactedappprtestregistry1BB674CB
[+] AWS::SSM::Parameter infra-redacted-app-pr-test-repository infraredactedappprtestrepositoryA3DA673B
[~] AWS::ECR::Repository redactedAppECR/ECR redactedAppECRC28DCB14 replace
 └─ [~] RepositoryName (requires replacement)
     ├─ [-] redacted-app-pr
     └─ [+] redacted-app-pr-test

---------------
Resources
[~] AWS::ECR::Repository jgECR/ECR jgECR629E43A9 replace
 └─ [~] RepositoryName (requires replacement)
     ├─ [-] jg-test
     └─ [+] jg-test-rename
[~] AWS::SSM::Parameter infra-jg-rep infrajgrepDB22CE12 replace
 └─ [~] Name (requires replacement)
     ├─ [-] /infra/jg-test/rep
     └─ [+] /infra/jg-test-rename/rep
[~] AWS::SSM::Parameter infra-jg-reg infrajgregBCDE9FE3 replace
 └─ [~] Name (requires replacement)
     ├─ [-] /infra/jg-test/reg
     └─ [+] /infra/jg-test-rename/reg

As you can see on the first output (real stack), the parameters will be destroyed and created. This causes the error.
But on the test stack it will replace them - this is the behavior I want and works as expected.

To create the test stack this is the full code I have used

## app.cdk.ts
import * as cdk from "aws-cdk-lib";
import { CdktestStack } from "../stacks/CdktestStack"

const app = new cdk.App();

new CdktestStack(app, "CdktestStack", {})


## CdktestStack.ts
import { Construct } from "constructs";
import { RemovalPolicy } from "aws-cdk-lib";
import * as ecr from "aws-cdk-lib/aws-ecr";
import * as cdk from 'aws-cdk-lib';
import * as ssm from 'aws-cdk-lib/aws-ssm';

export class CdktestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const ecrName = 'jg-test'
    const repository = new ecr.Repository(this, "jgECR", {
      repositoryName: ecrName,
      removalPolicy: RemovalPolicy.DESTROY,
      lifecycleRules: [{ description: "Last 120 images", maxImageCount: 120 }],
    });

    new ssm.StringParameter(this, 'infra-jg-rep', {
      parameterName: `/infra/${ecrName}/rep`,
      stringValue: repository.repositoryUri,
      description: "ecr registry",
    });

    new ssm.StringParameter(this, 'infra-jg-reg', {
      parameterName: `/infra/${ecrName}/reg`,
      stringValue: repository.repositoryUri.split('/')[0],
      description: "ecr repo",
    });
  }
  }
}

Then I would rename ecrName to jg-test-rename

This is as close as I can to copy our infrastructure without creating too much to provide you with a simple example.
So I will continue to test to find the issue, now that I see it is possible to do the replacement as expected.

If you have any understanding of why it might show destroy vs replace please share.
Thanks.

[ashishdhingra]

@asdfractal Thanks for your response.

• Is it possible to share minimal repro code that creates ECR repository and SSM parameters from real stack? (we can remove unnecessary code)?

• What is the version of aws-cdk-lib customer is using in real stack?

• Is the CDK application for the real stack somehow deploying the same stack multiple times?

• In real stack output:

  Resources
  [-] AWS::SSM::Parameter infra-redacted-app-pr-registry infraredactedappprregistry6D04DDAA destroy
  [-] AWS::SSM::Parameter infra-redacted-app-pr-repository infraredactedappprrepository7BAF1037 destroy
  [+] AWS::SSM::Parameter infra-redacted-app-pr-test-registry infraredactedappprtestregistry1BB674CB
  [+] AWS::SSM::Parameter infra-redacted-app-pr-test-repository infraredactedappprtestrepositoryA3DA673B
  [~] AWS::ECR::Repository redactedAppECR/ECR redactedAppECRC28DCB14 replace
   └─ [~] RepositoryName (requires replacement)
       ├─ [-] redacted-app-pr
       └─ [+] redacted-app-pr-test
  

  Why is the ID of SSM parameter changing? Notice that:

  • infra-redacted-app-pr-registry is getting changed to infra-redacted-app-pr-test-registry.
  • infra-redacted-app-pr-repository is getting chnaged to infra-redacted-app-pr-test-repository.

  Since the logical ID changed, it might be happening that the old SSM parameter is marked for deletion, but CloudFormation is somehow creating new SSM parameter before the old SSM parameter is deleted. Please see if it makes sense.

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[asdfractal]

@ashishdhingra
Thanks, you have helped me realise we had a variable as the ID as well, which is also being changed.
Ideally would like to be able to keep them updated as well, but it may be as you said with the order of operation.

At least now I can confirm the value is updated as expected, so not changing the ID can be dealt with.

Thanks for the assistance!

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.