ec2: InvalidPermission.NotFound: The specified rule does not exist in this security group when creating RestrictDefaultSecurityGroupCustomResource

[alehuo]

Describe the bug

After updating aws-cdk-lib from 2.77.0 to 2.87.0 the following error occurs with the @aws-cdk/aws-ec2:restrictDefaultSecurityGroup feature flag set to true. We had the feature flag set to true prior to the update, and there were no errors regarding the deployment.

Expected Behavior

cdk deploy should complete with no UPDATE_ROLLBACK_COMPLETE errors on the CloudFormation stack.

Current Behavior

The stack deployment fails, because the creation of RestrictDefaultSecurityGroupCustomResource fails. The following error occurs:

1:57:09 PM | CREATE_FAILED        | Custom::VpcRestrictDefaultSG                | Rest...omResource/Default
Received response status [FAILED] from custom resource. Message returned: InvalidPermission.NotFound: The specified rule does not exist in this security group.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: 0b911b0b-c81b-439a-9591-83cb123924af)
1:57:10 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack                  | network-stack
The following resource(s) failed to create: [vpcRestrictDefaultSecurityGroupCustomResource].
1:57:37 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | network-stack
1:57:41 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function                       | Custom::VpcRestric...ceProvider/Handler
1:57:09 PM | CREATE_FAILED        | Custom::VpcRestrictDefaultSG                | vpcRestric...omResource
Received response status [FAILED] from custom resource. Message returned: InvalidPermission.NotFound: The specified rule does not exist in this security group.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: 0b911b0b-c81b-439a-9591-83cb123924af)

Reproduction Steps

Use aws-cdk-lib version 2.77.0, set the feature flag ec2:restrictDefaultSecurityGroup to true. Create a new VPC and deploy it:

 const vpc = new Vpc(scope, id, {
    cidr: "10.4.0.0/21",
    maxAzs: 3,
    subnetConfiguration: [
      {
        cidrMask: 24,
        name: 'public-subnet',
        subnetType: SubnetType.PUBLIC
      },
      {
        cidrMask: 24,
        name: 'private-subnet',
        subnetType: SubnetType.PRIVATE_WITH_EGRESS
      },
      {
        cidrMask: 27,
        name: 'isolated-subnet',
        subnetType: SubnetType.PRIVATE_ISOLATED
      }
    ]
  })

After deployment, upgrade aws-cdk-lib to 2.87.0 and run cdk diff - you should see a RestrictDefaultSecurityGroupCustomResource resource being created. At least in our case aws-cdk-lib version 2.87.0 wanted to create the custom resource. Run cdk deploy and you should get an error about InvalidPermission.NotFound.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0 (build 9fca790)

Framework Version

No response

Node.js Version

v16.18.1

OS

MacOS Ventura 13.4.1, ARM64

Language

Typescript

Language Version

TypeScript (5.1.6)

Other information

No response

[khushail]

Hi @alehuo , I am able to reproduce this and got the mentioned resource diff -

and deployment is also succeeded -

Could you check your account settings to see if its not an issue with the enovironment

[alehuo]

I am now testing if it is environment related. I'm getting the same diff as you.

[alehuo]

Apparently, the default security group of the VPC has no inbound or outbound rules defined, and in our case AWS CDK assumes it has and the deployment fails. What kind of rules does the default security group of your created VPC have?

[alehuo]

Digging deeper into this issue:

If I create a completely new VPC with restrictDefaultSecurityGroup set to false, the deployment completes successfully. VPCs that are newly created this way have the following inbound & outbound rules:

Inbound: Allow all traffic
Outbound: Allow all IPv4 traffic

The deployment of a new VPC also succeeded with restrictDefaultSecurityGroup set to true. If restrictDefaultSecurityGroup is set to true, the custom lambda attempts to delete these rules.

I suspect that if the VPC's default security group did not have any inbound or outbound rules and later on we switch the restrictDefaultSecurityGroup flag to true, the deployment fails. I tried toggling between true and false on the parameter, but I did not manage to replicate the behaviour that exists in our environment.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.