aws-lambda: Adding an encrypted SQS queue as a Lambda DLQ does not grant Lambda role necessary KMS permissions

[pbsinclair42]

Describe the bug

When creating a Lambda, you have the option to specify a DLQ. Doing so correctly automatically grants the Lambda's role sqs:SendMessage permissions on that SQS queue resource. However, if that SQS queue is encrypted with a customer-managed KMS key in the same stack, the Lambda's role is not automatically granted the required permissions on the KMS key resource. This leads to the Lambda being unable to write to the DLQ and messages being lost.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The Lambda role to be automatically granted the required permissions on the KMS key

Current Behavior

The Lambda role did not have any permissions on the KMS key

Reproduction Steps

import { Capture, Template } from "aws-cdk-lib/assertions";
import { Key } from "aws-cdk-lib/aws-kms";
import { Function, InlineCode, Runtime } from "aws-cdk-lib/aws-lambda";
import { Queue } from "aws-cdk-lib/aws-sqs";
import { App, Stack } from "aws-cdk-lib/core";
import { Construct } from "constructs";

class TestStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const lambdaDlqKey = new Key(this, "LambdaDlqKey", {
      enableKeyRotation: true,
    });

    const lambdaDlq = new Queue(this, "LambdaDlq", {
      queueName: "LambdaDlq",
      encryptionMasterKey: lambdaDlqKey,
    });

    const lambda = new Function(this, "Lambda", {
      code: new InlineCode("throw new Error()"),
      handler: "someHandler",
      runtime: Runtime.NODEJS_18_X,
      deadLetterQueue: lambdaDlq,
    });
  }
}

describe("Function with customer managed KMS encrypted DLQ", () => {
  it("does not grant KMS permissions", () => {
    const mockApp = new App();

    const testInfraStack = new TestStack(mockApp, "testStack");

    const template = Template.fromStack(testInfraStack);

    const statementCapture = new Capture();
    template.hasResourceProperties("AWS::IAM::Policy", {
      PolicyDocument: {
        Statement: statementCapture,
      },
    });
    // Succeeds (correctly grants required sqs:SendMessage permission)
    expect(statementCapture.asArray().filter((statement) => hasActionMatching(statement, /sqs:SendMessage/))).not.toHaveLength(0);
    // Fails: (does not grant required kms permission)
    expect(statementCapture.asArray().filter((statement) => hasActionMatching(statement, /kms:.*/))).not.toHaveLength(0);
  });
});

function hasActionMatching(statement: { Action: string | string[] }, action: RegExp) {
  if (typeof statement.Action === "string") {
    return action.test(statement.Action);
  }
  return statement.Action.filter((statementAction) => action.test(statementAction)).length > 0;
}

This can also be confirmed by deploying the TestStack, sending an async message to the Lambda, and validating that it is unable to write the failed message to its DLQ (and a dead letter queue failure metric is published).

Possible Solution

This can of course be worked around by the user manually granting KMS permissions (e.g. lambdaDlqKey.grantEncrypt(lambda.role!), or even lambdaDlq.grantSendMessages(lambda)), however the fact that this is not done automatically by CDK seems like a bug to me, or at very least a poor customer experience that can lead to customer data loss due to accidental misconfiguration.

A fix would be to check if the DLQ is encrypted when the Lambda is initialised in CDK, and grant the necessary KMS permissions if so.

Additional Information/Context

No response

CDK CLI Version

2.175.1

Framework Version

No response

Node.js Version

18.20.5

OS

Ubuntu 24.04.1

Language

TypeScript

Language Version

TypeScript (5.7.3)

Other information

No response

[ashishdhingra]

Analysis:
Tested using CDK version 2.177.0.

Using code below without the encryptionMasterKey set for DLQ:

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as sqs from 'aws-cdk-lib/aws-sqs';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const lambdaDlq = new sqs.Queue(this, "LambdaDlq", {
      queueName: "LambdaDlq",
    });

    const lambdaFunction = new lambda.Function(this, "Lambda", {
      code: new lambda.InlineCode("exports.someHandler = async (event, context) => { throw new Error();};"),
      handler: "index.someHandler",
      runtime: lambda.Runtime.NODEJS_18_X,
      deadLetterQueue: lambdaDlq,
    });
  }
}

and sending asynchronous event using AWS CLI using command aws lambda invoke --function-name CdktestStackNew-Lambda<<SOME_Random_ID>> --invocation-type Event response.json, sends the error to DLQ after 2 retries (refer Understanding retry behavior in Lambda). We could monitor retry by inspecting CloudWatch logs to error logs. Also, for testing, we would need to poll for messages in AWS Lambda SQS console for the specified DLQ.

Changing CDK code to below:

import * as cdk from 'aws-cdk-lib';
import * as kms from "aws-cdk-lib/aws-kms";
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as sqs from 'aws-cdk-lib/aws-sqs';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const lambdaDlqKey = new kms.Key(this, "LambdaDlqKey", {
      enableKeyRotation: true,
    });

    const lambdaDlq = new sqs.Queue(this, "LambdaDlq", {
      queueName: "LambdaDlq",
      encryptionMasterKey: lambdaDlqKey,
    });

    const lambdaFunction = new lambda.Function(this, "Lambda", {
      code: new lambda.InlineCode("exports.someHandler = async (event, context) => { throw new Error();};"),
      handler: "index.someHandler",
      runtime: lambda.Runtime.NODEJS_18_X,
      deadLetterQueue: lambdaDlq,
    });
  }
}

generates the below CloudFormation template:

Resources:
  LambdaDlqKey3D43DCB9:
    Type: AWS::KMS::Key
    Properties:
      EnableKeyRotation: true
      KeyPolicy:
        Statement:
          - Action: kms:*
            Effect: Allow
            Principal:
              AWS: arn:aws:iam::139480602983:root
            Resource: "*"
        Version: "2012-10-17"
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    Metadata:
      aws:cdk:path: CdktestStackNew/LambdaDlqKey/Resource
  LambdaDlq3949784F:
    Type: AWS::SQS::Queue
    Properties:
      KmsMasterKeyId:
        Fn::GetAtt:
          - LambdaDlqKey3D43DCB9
          - Arn
      QueueName: LambdaDlq
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: CdktestStackNew/LambdaDlq/Resource
  LambdaServiceRoleA8ED4D3B:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Metadata:
      aws:cdk:path: CdktestStackNew/Lambda/ServiceRole/Resource
  LambdaServiceRoleDefaultPolicyDAE46E21:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: sqs:SendMessage
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - LambdaDlq3949784F
                - Arn
        Version: "2012-10-17"
      PolicyName: LambdaServiceRoleDefaultPolicyDAE46E21
      Roles:
        - Ref: LambdaServiceRoleA8ED4D3B
    Metadata:
      aws:cdk:path: CdktestStackNew/Lambda/ServiceRole/DefaultPolicy/Resource
  LambdaD247545B:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: exports.someHandler = async (event, context) => { throw new Error();};
      DeadLetterConfig:
        TargetArn:
          Fn::GetAtt:
            - LambdaDlq3949784F
            - Arn
      Handler: index.someHandler
      Role:
        Fn::GetAtt:
          - LambdaServiceRoleA8ED4D3B
          - Arn
      Runtime: nodejs18.x
    DependsOn:
      - LambdaServiceRoleDefaultPolicyDAE46E21
      - LambdaServiceRoleA8ED4D3B
    Metadata:
      aws:cdk:path: CdktestStackNew/Lambda/Resource
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/zWMyw6CMBBFv4V9GYGY4N7EDRvFDzCljMlAH4GhEtL03w1FV/fkvioo6xqKTK6cq37MNXUQnotUo5Arv8JoGEKDm7i+bYNbFDwxhIdHj7uVIAotTddLCDdv1ULO7tGfoyBpILROp0XSu9Ok0udBMYoW2flZHZ0fR2FdjzDw6VNeoCrgnA1MlM/eLmQQ2kO/+DKsHsQAAAA=
    Metadata:
      aws:cdk:path: CdktestStackNew/CDKMetadata/Default
Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]

• Deploying it does grant sqs:SendMessage permission.
• However, invoking Lambda function asynchronously via AWS CLI doesn't send message to DLQ.
• Manually adding below permissions to the Lambda role and re-invoking Lambda function aynchronously, sent message to DLQ after 2 retries.

  {
  	"Version": "2012-10-17",
  	"Statement": [
  		{
    		"Sid": "VisualEditor0",
    		"Effect": "Allow",
    		"Action": [
    			"kms:Decrypt",
    			"kms:GenerateDataKey"
    		],
    		"Resource": "arn:aws:kms:us-east-2:<<ACCOUNT-ID>>:key/2975e042-c5c8-4c07-8b5d-18671ce12a7b"
    	}
    ]
  }

Refer Grant Amazon SNS KMS permissions to Amazon SNS to publish messages to the queue at Access management for encrypted Amazon SQS queues with least privilege policies for policy details.