aws-s3: CDK Version 2.147.0+ has error with creating Bucket with autoDeleteObjects=true setting

[ecs-jnguyen]

Describe the bug

Upgrading from cdk 2.146.0 -> 2.147.0 causes the following error when deploying a stack with an S3 bucket. In 2.147.0 if I set auto_delete_objects=True, the deployment fails with an error (error message is in a different section below). When I set auto_delete_objects=False, the deployment is successful.

The asset that fails to upload is CustomS3AutoDeleteObjectsCustomResourceProviderHandler.

Expected Behavior

I expect the asset CustomS3AutoDeleteObjectsCustomResourceProviderHandler to publish successfully

Current Behavior

During deployment it is failing to upload the asset

CdkBucketDebugStack:  start: Building faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2
CdkBucketDebugStack:  success: Built faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2
CdkBucketDebugStack:  start: Publishing faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2
CdkBucketDebugStack:  fail: Unexpected close tag
Line: 126
Column: 7
Char: >

 ❌ Deployment failed: Error: Failed to publish asset faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2
    at Deployments.publishSingleAsset (/Users/myusername/.n/lib/node_modules/aws-cdk/lib/index.js:454:11645)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.publishAsset (/Users/myusername/.n/lib/node_modules/aws-cdk/lib/index.js:454:197644)
    at async /Users/myusername/.n/lib/node_modules/aws-cdk/lib/index.js:454:181545

Failed to publish asset faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2

Reproduction Steps

Here is my python cdk app

# app.py
import aws_cdk as cdk

from aws_cdk import Stack, aws_s3, RemovalPolicy

from aws_cdk.aws_s3 import (
    BucketEncryption,
    BlockPublicAccess,
)
from constructs import Construct


CF_ACCOUNT_ID = "123456789"
CF_REGION = "us-west-2"


class CdkBucketDebugStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here

        aws_s3.Bucket(
            self,
            "TestingBucket",
            bucket_name=f"testingbucket-asdfasdf-{CF_ACCOUNT_ID}-{CF_REGION}",
            encryption=BucketEncryption.S3_MANAGED,
            versioned=True,
            enforce_ssl=True,
            minimum_tls_version=1.2,
            block_public_access=BlockPublicAccess.BLOCK_ALL,
            auto_delete_objects=True,
            removal_policy=RemovalPolicy.DESTROY,
            event_bridge_enabled=True,
        )


app = cdk.App()
CdkBucketDebugStack(
    app,
    "CdkBucketDebugStack",
    env=cdk.Environment(account=CF_ACCOUNT_ID, region=CF_REGION),
)

app.synth()

Possible Solution

N/A

Additional Information/Context

This is working fine in 2.146.0. It breaks from 2.147.0 to 2.149.0(latest version of as writing)

CDK CLI Version

2.147.0 (build 3338fc0)

Framework Version

No response

Node.js Version

v18.17.1

OS

mac

Language

Python

Language Version

3.11.8

Other information

No response

[pahud]

Hi

I was able to deploy this with 2.148.0, 2.147.0 and 2.146.0 with no error using the code below:

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'Bucket', {
      encryption: s3.BucketEncryption.S3_MANAGED,
      versioned: true,
      enforceSSL: true,
      minimumTLSVersion: 1.2,
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      autoDeleteObjects: true,
      removalPolicy: RemovalPolicy.DESTROY,
      eventBridgeEnabled: true,
    });
  }
} 

And this CDK in python with 2.147.0 with no error.

s3.Bucket(
	self,
	"TestingBucket",
	# bucket_name=f"testingbucket-asdfasdf-{CF_ACCOUNT_ID}-{CF_REGION}",
	encryption=s3.BucketEncryption.S3_MANAGED,
	versioned=True,
	enforce_ssl=True,
	minimum_tls_version=1.2,
	block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
	auto_delete_objects=True,
	removal_policy=RemovalPolicy.DESTROY,
	event_bridge_enabled=True,
)

my versions

% npx cdk --version
2.147.0 (build e5740c0)
(.venv) % grep aws-cdk-lib requirements.txt 
aws-cdk-lib==2.147.0

I can't reproduce this issue both in CDK TypeScript and Python.

Questions:

1. You mentioned

The asset that fails to upload is CustomS3AutoDeleteObjectsCustomResourceProviderHandler.

But I didn't see CustomS3AutoDeleteObjectsCustomResourceProviderHandler in your provided error message, what made you think it was failing on uploading that asset?

2. Can you try my code snippet above without specifying the bucket name and see if it works?

3. Can you check both CDK CLI version and aws-cdk-lib version using the command as I mentioned above. Are they both 2.147.0 ?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[ecs-jnguyen]

Hi @pahud, thanks for the quick reply.

I tried your code without specifying the s3 bucket name and it is still failing. However I noticed that deploying in a different AWS account works okay. I think this issue is somehow tied only to this particular AWS account.

To answer your questions:

1. The error message Failed to publish asset faa95a81ae7d7373f3e1f242268f904eb748d8d0fdd306e8a6fe515a1905a7d6:123456789-us-west-2 led me to open the asset in my cdk.out folder. Upon further investigation it looks like the index.js file in there is the lambda code for the autoDeleteObjects flag. When I had autoDeleteObjects=false, it did not try to upload that asset.
2. It still fails in the problem AWS account, however it is working in a different account
3. For step 2, I updated my cdk cli to 2.147.0 and my aws-cdk-lib in python to 2.147.0

I think for now we can probably close this issue since it is specific to my account. When I took the zip file asset from the good standing AWS account and uploaded it to the problematic AWS account via the browser, it looks like the deployment when okay.

I think we can close this issue for now.

EDIT: upon further investigation, I believe that there is a network configuration on my end that is blocking uploads to this s3 bucket.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[pahud]

@ecs-jnguyen

Thank you for the follow-up comment.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.