(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

AllanOricil · 2024-05-08T06:02:15Z

Describe the bug

I have a lambda full of env variables referencing a "parameter", like the one shown below

![img]

INFO: pay attention that I'm loading the right .env file as shown in the above image. I always set NODE_ENV before I ran any cdk command. And I print the .env path to make sure the right one was selected when building things locally.

![img]

The above screenshots show a template that was synthesized by CDK after I changed the .env file that provide values to my parameters. However, even after deploying this new stack, CDK did not change my env variable values in my lambda functions. In the image below you can see that "ALLOWED_ORIGINS" was not updated.

![img]

Another weird thing is the CDK doesnt even realize that this change will happen. Below you can see the output of cdk diff for the same stack.

Expected Behavior

CDK must update Lambda Env Variables values that reference parameters whenever they are changed.

Current Behavior

CDK does not update Lambda Env Variables values when they reference parameters that have changed

Reproduction Steps

create a construct with one lambda
create a cfn parameter that is loaded with values from a .env file. You can use dotenv. like I did.
![img]
![img]
add an env variable to your lambda, and fill its value with the value from the parameter.

![img]

synth and deploy it. Open the synthesized template and verify the parameter value is the same one from your .env.
change the .env value
synth. Again verify the synthesized template and verify that it has changed, and it again matches the one from your .env.
deploy it and verify that the value was not changed in AWS
conclude it is a bug

Possible Solution

Couldn't find any woraround

Additional Information/Context

No response

CDK CLI Version

2.138.0

Framework Version

No response

Node.js Version

node 18

OS

macos latest

Language

TypeScript

Language Version

typescript@v5.2.2

Other information

No response

The text was updated successfully, but these errors were encountered:

AllanOricil · 2024-05-08T07:39:10Z

I also deleted ALL env variables in one of my functions, using the console. Then I redeployed the stack, and verified my env variables were not added back. There were a bunch of other env variables there that are just gone.

![img]

AllanOricil · 2024-05-08T08:46:49Z

I had to invest around 1 hour to destroy and recreate my stack in order to have env variables updated

![img]

AllanOricil · 2024-05-08T08:49:22Z

Please, create a dedicated flag in cdk that "updates env variables only".

--update-lambda-env-variables-only

Look at this! 28min to deploy + the time to destroy the stack + the time I now have to invest to update the API url in some other stacks

![img]

ashishdhingra · 2024-05-08T18:36:52Z

@AllanOricil Somehow I'm unable to reproduce the issue with CDK version 2.140.0. Used the code below:
TypescriptStack.ts

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as dotenv from 'dotenv';

export class TypescriptStack extends cdk.Stack {
  private readonly allowedOrigins: cdk.CfnParameter;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    dotenv.config();
    console.log("ALLOWED_ORIGINS: " + this.getEnvVar("ALLOWED_ORIGINS"));

    this.allowedOrigins = new cdk.CfnParameter(this, "allowedOrigins", {
      type: "String",
      noEcho: true,
      default: this.getEnvVar("ALLOWED_ORIGINS")
    });

    const lambdaFunction = new lambda.Function(this, 'testLambda', {
      runtime: lambda.Runtime.NODEJS_20_X,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'hello.handler',
      environment: {
         ALLOWED_ORIGINS: this.getEnvVar("ALLOWED_ORIGINS")
      }
    });
  }

  getEnvVar(envVarName: string): string {
    return process.env.ALLOWED_ORIGINS ?? "";
  }
}

lambda\hello.js

exports.handler = async function(event) {
    console.log("request:", JSON.stringify(event, undefined, 2));
    return {
      statusCode: 200,
      headers: { "Content-Type": "text/plain" },
      body: `Hello, CDK! You've hit ${event.path}\n`
    };
  };

.env (on 1st deploy)

ALLOWED_ORIGINS=http://localhost:8080,http://testdomain.com

.env (on 2nd deploy)

ALLOWED_ORIGINS=http://localhost:8080,http://testdomain.com,http://makemydomain.com

package.json

{
  "name": "typescript",
  "version": "0.1.0",
  "bin": {
    "typescript": "bin/typescript.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.12",
    "@types/node": "20.12.7",
    "aws-cdk": "2.140.0",
    "jest": "^29.7.0",
    "ts-jest": "^29.1.2",
    "ts-node": "^10.9.2",
    "typescript": "~5.4.5"
  },
  "dependencies": {
    "aws-cdk-lib": "2.140.0",
    "constructs": "^10.0.0",
    "dotenv": "^16.4.5",
    "source-map-support": "^0.5.21"
  }
}

Executing cdk diff before 2nd deployment gave the below output:

Parameters
[~] Parameter allowedOrigins allowedOrigins: {"Type":"String","Default":"http://localhost:8080,http://testdomain.com","NoEcho":true} to {"Type":"String","Default":"http://localhost:8080,http://testdomain.com,http://makemydomain.com","NoEcho":true}

Resources
[~] AWS::Lambda::Function testLambda testLambdaC13632F6 
 └─ [~] Environment
     └─ [~] .Variables:
         └─ [~] .ALLOWED_ORIGINS:
             ├─ [-] http://localhost:8080,http://testdomain.com
             └─ [+] http://localhost:8080,http://testdomain.com,http://makemydomain.com

Executing cdk deploy updated the environment variable in Lambda function.

Thanks,
Ashish

AllanOricil · 2024-05-08T21:04:07Z

@ashishdhingra your lambda is not using the value from the parameter. It is using the env variable. Your rendered template must have a Ref pointing to the parameter.

ashishdhingra · 2024-05-08T22:17:08Z

@ashishdhingra your lambda is not using the value from the parameter. It is using the env variable. Your rendered template must have a Ref pointing to the parameter.

@AllanOricil Thanks for pointing it out. I modified the code to below and the issue is now reproducible:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as dotenv from 'dotenv';

export class TypescriptStack extends cdk.Stack {
  private readonly allowedOrigins: cdk.CfnParameter;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    dotenv.config();
    console.log("ALLOWED_ORIGINS: " + this.getEnvVar("ALLOWED_ORIGINS"));

    this.allowedOrigins = new cdk.CfnParameter(this, "allowedOrigins", {
      type: "String",
      noEcho: true,
      default: this.getEnvVar("ALLOWED_ORIGINS")
    });

    const lambdaFunction = new lambda.Function(this, 'testLambda', {
      runtime: lambda.Runtime.NODEJS_20_X,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'hello.handler',
      environment: {
         ALLOWED_ORIGINS: this.allowedOrigins.value.toString()
      }
    });
  }

  getEnvVar(envVarName: string): string {
    return process.env.ALLOWED_ORIGINS ?? "";
  }
}

It produces below CloudFormation template (as an example), with environment variable as Ref to allowedOrigins parameter:

Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com,http://makemydomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Metadata:
      aws:cdk:path: TypescriptStack/testLambda/ServiceRole/Resource
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket:
          Fn::Sub: cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289
    Metadata:
      aws:cdk:path: TypescriptStack/testLambda/Resource
      aws:asset:path: asset.b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706
      aws:asset:is-bundled: false
      aws:asset:property: Code
...
...

Running cdk diff (before 2nd deployment), produces the following output:

ALLOWED_ORIGINS: http://localhost:8080,http://testdomain.com,http://makemydomain.com
Stack TypescriptStack
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
Parameters
[~] Parameter allowedOrigins allowedOrigins: {"Type":"String","Default":"http://localhost:8080,http://testdomain.com","NoEcho":true} to {"Type":"String","Default":"http://localhost:8080,http://testdomain.com,http://makemydomain.com","NoEcho":true}


✨  Number of stacks with differences: 1

I'm unsure though if this is a CDK issue or CloudFormation issue since the synthesized template is handled by CloudFormation, and CloudFormation allows declaring parameters.

Need to discuss this with team.

Thanks,
Ashish

AllanOricil · 2024-05-08T22:46:45Z

@ashishdhingra keep us updated
Thank you!

AllanOricil · 2024-05-09T03:18:57Z

Sorry, I thought I had found another related issue. But I confused an execution record with a config record.

skinny85 · 2024-05-09T04:09:14Z

@AllanOricil why do you need to use a Parameter at all? In general, CloudFormation Parameters are discouraged when using CDK.

AllanOricil · 2024-05-09T04:14:42Z

@skinny85

For my use case, I can definitely remove it. Thanks for pointing that out 🤙

However, I did it like that because I wanted to verify if it was possible to override an env variable if necessary using cli parameters. This is useful in complex scenarios (I did not find one yet) which the value of the env variable in the lambda has to be evaluated at deploy time, with information outside of the cdk project.

With a parameter like that, I can deploy my stack as so:

DYNAMIC_VARIABLE=$(...)
cdk synth stack-name -e --parameters myDynamicVariable $DYNAMIC_VARIABLE

If AWS does not recommend it. What about removing this feature? For example, if one tries to do what I did, throw an exception and stop the synthesis. However I do think it could be useful. I just don't know a real use case to show as an example.

skinny85 · 2024-05-09T05:04:21Z

@AllanOricil No worries! A few thoughts:

There's no reason to use CloudFormation Parameters if all you need is to read an environment variable. You can always just read an environment variable directly in your CDK code:
```
const dynamicVariable = process.env.DYNAMIC_VARIABLE ?? 'DEFAULT_VALUE';
// use dynamicVariable in your code...
```
The CDK Context is a good mechanism if you need to reference something outside your code.

In general, the results of synthesizing your CDK app should be deterministic, and avoid things like external environment variables as much as possible.

Do you need to use environment variables perhaps because you are using the same CDK Stack instance for multiple deployed Stacks in CloudFormation? That's not the best practice when using CDK.

Let me know if this makes sense!

ashishdhingra · 2024-05-09T17:06:28Z

For testing purposes, without using CDK:

Deployed the below CloudFormation stack from AWS console (manually uploaded the lambda code ZIP file to S3 bucket):

Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: testbucket-issue1880
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289

This created Lambda along with environment variable ALLOWED_ORIGINS having value http://localhost:8080,http://testdomain.com.

Modified the value of CloudFormation parameter allowedOrigins to http://localhost:8080,http://testdomain.com,http://makemydomain.com. Deployed the below template (updated CloudFormation stack to use a new template):

Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com,http://makemydomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: testbucket-issue1880
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289

During AWS console workflow, in Specify stack details step showing there was a checkbox Use previous value in Parameters section. By default, this checkbox was checked. In the last step Review testlambdastack, it displayed message No updates are to be performed..
Went back to Specify stack details step and unchecked the checkbox Use previous value. Unchecking the checkbox blanked out the value (could be a UI issue since it should have read updated value from uploaded YAML file). Hence, manually specified value http://localhost:8080,http://testdomain.com,http://makemydomain.com for allowedOrigins parameter.
Going to last Review testlambdastack now displayed Lambda function to be modified in Change set preview section.
Submitting the changes updated the Lambda function with environment variable ALLOWED_ORIGINS changed to http://localhost:8080,http://testdomain.com,http://makemydomain.com .

So, by default, any changes to CloudFormation parameter are not used (based on observation in AWS console UI).

If we look at the CDK guide Deploying with parameters, is explicitly specifies that By default, the AWS CDK retains values of parameters from previous deployments and uses them in subsequent deployments if they are not specified explicitly. Use the --no-previous-parameters flag to require all parameters to be specified. (this may be due to limitation of CloudFormation).

Going with the above understanding, using cdk deploy --no-previous-parameters on 2nd deployment updated the value of environment variable in Lambda configuration.

@AllanOricil Please use cdk deploy --no-previous-parameters to use the updated value of CloudFormation parameter.

Thanks,
Ashish

AllanOricil · 2024-05-09T21:28:59Z

@ashishdhingra Good job!

I didnt know about this flag.

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

AllanOricil · 2024-05-09T21:29:03Z

@ashishdhingra Good job!

I didnt know about this flag.

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

ashishdhingra · 2024-05-09T21:51:29Z

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

@AllanOricil Based on my observation in last #30101 (comment), one possible reasoning is one-to-one parity with CloudFormation UI in AWS console. By default, in AWS console, checkbox Use previous value is checked. This is just an assumption. Also, if we change the default behavior, it might break some small set of customers where by default it is currently assumed that new value for CfnParameter would not be used by default.

AllanOricil · 2024-05-10T12:34:36Z

I will close this. Thank you all for the help.

github-actions · 2024-05-10T12:34:59Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws-cdk-automation · 2024-07-25T21:10:34Z

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

AllanOricil added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 8, 2024

github-actions bot added the @aws-cdk/core Related to core CDK functionality label May 8, 2024

AllanOricil changed the title ~~(core): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack~~ (Lambda): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack May 8, 2024

github-actions bot added the @aws-cdk/aws-lambda Related to AWS Lambda label May 8, 2024

AllanOricil mentioned this issue May 8, 2024

(cli): --hotswap does not detect function env variables changes #16444

Closed

piradeepk removed the @aws-cdk/core Related to core CDK functionality label May 8, 2024

ashishdhingra self-assigned this May 8, 2024

ashishdhingra added needs-reproduction This issue needs reproduction. and removed needs-triage This issue or PR still needs to be triaged. labels May 8, 2024

ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-reproduction This issue needs reproduction. labels May 8, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 9, 2024

AllanOricil changed the title ~~(Lambda): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack~~ (Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack May 9, 2024

ashishdhingra added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 9, 2024

ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels May 9, 2024

AllanOricil closed this as completed May 10, 2024

This was referenced May 13, 2024

Weekly issue metrics report #30173

Closed

Weekly issue metrics report #30175

Closed

github-actions bot mentioned this issue Jun 1, 2024

Monthly issue metrics report #30413

Closed

aws locked as resolved and limited conversation to collaborators Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

ashishdhingra commented May 8, 2024

AllanOricil commented May 8, 2024 •

edited

Loading

ashishdhingra commented May 8, 2024

AllanOricil commented May 8, 2024

AllanOricil commented May 9, 2024

skinny85 commented May 9, 2024

AllanOricil commented May 9, 2024 •

edited

Loading

skinny85 commented May 9, 2024

ashishdhingra commented May 9, 2024

AllanOricil commented May 9, 2024

AllanOricil commented May 9, 2024

ashishdhingra commented May 9, 2024

AllanOricil commented May 10, 2024

github-actions bot commented May 10, 2024

aws-cdk-automation commented Jul 25, 2024

(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

Comments

AllanOricil commented May 8, 2024 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

AllanOricil commented May 8, 2024 • edited Loading

AllanOricil commented May 8, 2024 • edited Loading

AllanOricil commented May 8, 2024 • edited Loading

ashishdhingra commented May 8, 2024

AllanOricil commented May 8, 2024 • edited Loading

ashishdhingra commented May 8, 2024

AllanOricil commented May 8, 2024

AllanOricil commented May 9, 2024

skinny85 commented May 9, 2024

AllanOricil commented May 9, 2024 • edited Loading

skinny85 commented May 9, 2024

ashishdhingra commented May 9, 2024

AllanOricil commented May 9, 2024

AllanOricil commented May 9, 2024

ashishdhingra commented May 9, 2024

AllanOricil commented May 10, 2024

github-actions bot commented May 10, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

aws-cdk-automation commented Jul 25, 2024

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 8, 2024 •

edited

Loading

AllanOricil commented May 9, 2024 •

edited

Loading