aws-s3-deployment: BucketDeployment fails with rate exceeded (Lambda ThrottlingException)

[borgoat]

Describe the bug

I just upgraded to 2.93 and started getting an error during a BucketDeployment preparation. It looks like it happens even before the Lambda that copies data from the asset bucket into my target bucket runs.
It seems to be an issue while configuring the Lambda Layer, and the error apparently comes from the Lambda Control Plane. AFAIK, it is not possible to raise those limits.

Expected Behavior

I need to deploy a bunch of Lambda Functions together. Probably, I'm updating a lot of them since I did a general upgrade of dependencies, and so they're all deployed at the same time. I'm not even sure if this is a CDK issue to be honest...

Current Behavior

Dev-Stateles |  51/143 | 5:02:36 PM | DELETE_COMPLETE      | AWS::Lambda::LayerVersion            | Dev/Stateles/InstrumentsStack/SqliteDb/AwsCliLayer (InstrumentsStackSqliteDbAwsCliLayer3E9EBD96)
Dev-Stateles |  52/143 | 5:02:36 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack           | Dev-Stateles
Failed resources:
Dev-Stateles | 4:59:38 PM | UPDATE_FAILED        | AWS::Lambda::LayerVersion            | Dev/Stateles/InstrumentsStack/SqliteDb/AwsCliLayer (InstrumentsStackSqliteDbAwsCliLayer3E9EBD96) Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException; Request ID: d2daf379-98f1-4b16-b1f7-00b864bcec81; Proxy: null)" (RequestToken: c8b03f3e-21e2-8ca8-bd9d-fa6d0d1c94f4, HandlerErrorCode: GeneralServiceException)

 ❌  Dev/Stateles (Dev-Stateles) failed: Error: The stack named Dev-Stateles failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException; Request ID: d2daf379-98f1-4b16-b1f7-00b864bcec81; Proxy: null)" (RequestToken: c8b03f3e-21e2-8ca8-bd9d-fa6d0d1c94f4, HandlerErrorCode: GeneralServiceException)
    at FullCloudFormationDeployment.monitorDeployment (/yeekatee/node_modules/aws-cdk/lib/index.js:443:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/yeekatee/node_modules/aws-cdk/lib/index.js:446:153718)
    at async /yeekatee/node_modules/aws-cdk/lib/index.js:446:137166

 ❌ Deployment failed: Error: The stack named Dev-Stateles failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException; Request ID: d2daf379-98f1-4b16-b1f7-00b864bcec81; Proxy: null)" (RequestToken: c8b03f3e-21e2-8ca8-bd9d-fa6d0d1c94f4, HandlerErrorCode: GeneralServiceException)
    at FullCloudFormationDeployment.monitorDeployment (/yeekatee/node_modules/aws-cdk/lib/index.js:443:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/yeekatee/node_modules/aws-cdk/lib/index.js:446:153718)
    at async /yeekatee/node_modules/aws-cdk/lib/index.js:446:137166
The stack named Dev-Stateles failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException; Request ID: d2daf379-98f1-4b16-b1f7-00b864bcec81; Proxy: null)" (RequestToken: c8b03f3e-21e2-8ca8-bd9d-fa6d0d1c94f4, HandlerErrorCode: GeneralServiceException)

Reproduction Steps

export type InstrumentsStackProps = {
  readonly bucket: s3.IBucket;
  readonly cloudFrontDistribution: cloudfront.IDistribution;
}

export class InstrumentsStack extends Construct {
  constructor(scope: Construct, id: string, props: InstrumentsStackProps) {
    super(scope, id);

    const { bucket, cloudFrontDistribution } = props;

    const filePath = path.join(__dirname, 'static-data', 'bundle');
    const asset = s3deploy.Source.asset(filePath);

    new s3deploy.BucketDeployment(this, 'SqliteDb', {
      destinationBucket: bucket,
      distribution: cloudFrontDistribution,
      sources: [asset],
      destinationKeyPrefix: 'public/instruments/',
      memoryLimit: 1024,
    });
  }
}

The folder static-data/bundle is around ~150MB of data.

In the end I fear the real problem is not this Construct per se, but the fact that I'm deploying some other 70 Lambda Functions in parallel?

Possible Solution

I'm wondering if pre-publishing this layer that contains just the CLI could solve this, instead of deploying it every time as a ZIP.

Additional Information/Context

No response

CDK CLI Version

2.93.0 (build 724bd01)

Framework Version

No response

Node.js Version

v18.16.0

OS

macOS

Language

Typescript

Language Version

TypeScript 5.1.6

Other information

No response

[peterwoodworth]

Thanks for reporting @borgoat, if you try and downgrade again does the error stop? Also, what is the diff on this resource that's failing?

[borgoat]

Hey @peterwoodworth!

Yes, the deployment pre-upgrade keeps working. To be honest, it has nothing to deploy when it comes to that specific Lambda Layer, since this upgrade got rolled back by CloudFormation. However, the rest of the stack is deploying correctly, including other changes to Lambda Functions.

Before the upgrade I was using v2.87. I also tested now upgrading to v2.94 instead, but the issue is the same.

The BucketDeployment done in isolation does not seem to cause any trouble, but when combined with a few other Lambdas (as mentioned I'm upgrading multiple dependencies, so this is 53 Functions (plus the 1 Layer) being freshly bundled and updated as part of the stack), it fails.

I commented out the BucketDeployment, deployed, and then uncommented and deployed again. This worked: no rate limits were hit.
It's not really a great solution though, as I can't replicate this in the CD pipeline, and would have to get my whole team to do that manually too...

This is the CDK diff before and after the upgrade: diff.txt

I find it interesting that it is always the AWS::Lambda::LayerVersion AwsCliLayer to fail, rather than any other Lambda Function being deployed. I wonder if a different quota is being applied there, but I couldn't find anything in the docs...

I'm not sure of all the implications and tradeoffs, but anyway I would suggest investigating whether this AWS CLI Layer¹ - that is deployed as part of this BucketDeployment construct, but surely also used by others - could be delivered independently of CDK, as a pre-built Layer. Similar to what Powertools does². It would just be a ARN for CDK, versioning could still be enforced, and it would be 1 resource less in the customer stack (which isn't bad being a scarce resource). I didn't try this case, but I suspect if I had 2 BucketDeployment I would then get 2 Layers with the exact same content? If that's the case, it would be really valuable to reuse an existing Layer.

What do you think?

Footnotes

1. https://github.com/cdklabs/awscdk-asset-awscli ↩

2. https://docs.powertools.aws.dev/lambda/python/latest/#lambda-layer ↩

[eladrkatz]

Had similar issue - thanks to this thread I figured that downgrading CDK version mitigates the issue. Thanks!

[jmustonen]

Any development on this front? We started facing this very same issue a couple of weeks ago, and downgrading cdk to v2.87 seemed to help. But unfortunately it was just temporary - today the issue resurfaced, and we started again getting the Rate exceeded ThrottlingException 😞

We currently have 5 stacks that we're deploying in a single CodeBuild pipeline - it might very well be that we're doing "too much" in one go, but still this used to work mostly without any hiccups.