(elasticsearch): Two Domains in same account/region cause LogGroup Policy Failure

[peterb154]

When 2 or more elasticsearch Domains are created in the same account/region both with logging configurations, Cloudformation will throw an error when the second domain is deleted:

Cloudformation resource DELETE failed error:

ESLogGroupPolicy	Custom::CloudwatchLogResourcePolicy	DELETE_FAILED	Failed to delete resource. Policy with name [ESLogPolicy] does not exist.

Cloudformation event:

2020-12-10 17:16:37 UTC-0600	ElasticsearchTestTsESDomain2ESLogGroupPolicy7BC68A84	DELETE_FAILED	Failed to delete resource. Policy with name [ESLogPolicy] does not exist.

Reproduction Steps

#!/usr/bin/env node
import { App, Stack } from '@aws-cdk/core';
import { Domain, ElasticsearchVersion } from '@aws-cdk/aws-elasticsearch';

const app = new App();
const stack = new Stack(app, 'es-bug-stack');
new Domain(stack, 'domain1', {
    version: ElasticsearchVersion.V7_7,
    logging: { appLogEnabled: true }
})
const stack2 = new Stack(app, 'es-bug-stack2');
new Domain(stack2, 'domain2', {
    version: ElasticsearchVersion.V7_7,
    logging: { appLogEnabled: true }
})
app.synth()

$ cdk deploy '*'
$ cdk destroy

Detailed example:

$ cdk deploy '*' --require-approval never
 es-bug-stack
 es-bug-stack: deploying...
 ✅  es-bug-stack
 es-bug-stack2
 es-bug-stack2: deploying...
 ✅  es-bug-stack2

$ cdk destroy es-bug-stack --force
 es-bug-stack: destroying... 
 9:52:38 AM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack          | es-bug-stack
 9:58:56 AM | DELETE_IN_PROGRESS   | AWS::IAM::Role                      | AWS679f53fac002430...bd2287/ServiceRole

 ✅  es-bug-stack: destroyed

$ cdk destroy es-bug-stack2 --force
 10:18:32 AM | DELETE_FAILED        | Custom::CloudwatchLogResourcePolicy | domain2ESLogGroupPolicyC8E72CCB
 Failed to delete resource. Policy with name [ESLogPolicy] does not exist.

        new CustomResource (/Users/req96588/Projects/moo-elastic-testing/node_modules/@aws-cdk/core/lib/custom-resource.ts:115:21)
        \_ new AwsCustomResource (/Users/req96588/Projects/moo-elastic-testing/node_modules/@aws-cdk/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts:376:27)
        \_ new LogGroupResourcePolicy (/Users/req96588/Projects/moo-elastic-testing/node_modules/@aws-cdk/aws-elasticsearch/lib/log-group-resource-policy.ts:28:5)
        \_ new Domain (/Users/req96588/Projects/moo-elastic-testing/node_modules/@aws-cdk/aws-elasticsearch/lib/domain.ts:1416:32)
        \_ Object.<anonymous> (/Users/req96588/Projects/moo-elastic-testing/bin/app.ts:12:1)
10:18:32 AM | DELETE_FAILED        | Custom::CloudwatchLogResourcePolicy | domain2/ESLogGroup...y/Resource/Default
Failed to delete resource. Policy with name [ESLogPolicy] does not exist.
10:18:37 AM | DELETE_FAILED        | AWS::CloudFormation::Stack          | es-bug-stack2
The following resource(s) failed to delete: [domain2ESLogGroupPolicyC8E72CCB].

 ❌  es-bug-stack2: destroy failed Error: The stack named es-bug-stack2 is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [domain2ESLogGroupPolicyC8E72CCB]. )
    at Object.waitForStackDelete (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:277:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Object.destroyStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:381:28)
    at CdkToolkit.destroy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:252:9)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:200:9)
The stack named es-bug-stack2 is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [domain2ESLogGroupPolicyC8E72CCB]. )

What did you expect to happen?

That two elastic search domains can be created in the same AWS account/region without issue

What actually happened?

1. When the second domain is created, it overwrites the first domains CloudWatch LogGroup policy (becuase the Domain uses a statically named ESLogPolicy)
2. When the second domain is deleted, cloudformation throw an error. (Because when the first domain is deleted, it should not delete the second domain's policy)

Environment

• CDK CLI Version : 1.70.0
• Framework Version: 1.70.0
• Node.js Version: v12.18.4
• **OS : macos
• Language (Version): all | typescript@3.9.7

Other

It appears that @aws-cdk/aws-elasticsearch Domain creates a custom resources to set cloudwatch log group resource policy here and the yes the policy name "ESLogPolicy"

aws-cdk/packages/@aws-cdk/aws-elasticsearch/lib/domain.ts

Line 1418 in ae2e9c1

policyName: 'ESLogPolicy',

      // Use a custom resource to set the log group resource policy since it is not supported by CDK and cfn.
      // https://github.com/aws/aws-cdk/issues/5343
      logGroupResourcePolicy = new LogGroupResourcePolicy(this, 'ESLogGroupPolicy', {
        policyName: 'ESLogPolicy',
        policyStatements: [logPolicyStatement],
      });

The custom resource deletes that log policy when the Domain is deleted

aws-cdk/packages/@aws-cdk/aws-elasticsearch/lib/log-group-resource-policy.ts

Line 41 in 33ce50f

action: 'deleteResourcePolicy',

/**
 * Creates LogGroup resource policies.
 */
export class LogGroupResourcePolicy extends cr.AwsCustomResource {
  constructor(scope: cdk.Construct, id: string, props: LogGroupResourcePolicyProps) {
    const policyDocument = new iam.PolicyDocument({
      statements: props.policyStatements,
    });

    super(scope, id, {
      resourceType: 'Custom::CloudwatchLogResourcePolicy',
      onUpdate: {
        service: 'CloudWatchLogs',
        action: 'putResourcePolicy',
        parameters: {
          policyName: props.policyName,
          policyDocument: JSON.stringify(policyDocument),
        },
        physicalResourceId: cr.PhysicalResourceId.of(id),
      },
      onDelete: {
        service: 'CloudWatchLogs',
        action: 'deleteResourcePolicy',
        parameters: {
          policyName: props.policyName,
        },
        ignoreErrorCodesMatching: '400',
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({ resources: ['*'] }),
    });
  }
}

---

This is 🐛 Bug Report

[peterb154]

@iliapolo I'll write a PR for this a little later if that helps.

[iliapolo]

@peterb154 Yes that would be great 👍

[peterb154]

@peterb154 Yes that would be great
@iliapolo - Here is a PR #12056 - assumeing using this.node.addr in the LogGroup Resource Policy name is stable/desirable. Let me know if not.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.