aws_logs: CfnAccountPolicy Returns bad request

[LucasSymons]

Describe the bug

When creating an AccountPolicy via cdk, I get back a request error.
"Invalid request provided: AWS::Logs::AccountPolicy" (RequestToken: 717f24d1-38cd-2b20-b7a1-1b04be946d9f, HandlerErrorCode: InvalidRequest)

Testing the stack with cdk synth no errors are returned.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Expected successful creation or an obvious error about what the problem is.

Current Behavior

Deployment failed: Error: The stack named cloudwatch-data-protection-NONPROD failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: AWS::Logs::AccountPolicy" (RequestToken: 717f24d1-38cd-2b20-b7a1-1b04be946d9f, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/Cellar/aws-cdk/2.154.1/libexec/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:523:13)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at Object.deployStack2 [as deployStack] (/usr/local/Cellar/aws-cdk/2.154.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:332:24)
    at /usr/local/Cellar/aws-cdk/2.154.1/libexec/lib/node_modules/aws-cdk/lib/util/work-graph.ts:105:11

The stack named cloudwatch-data-protection-NONPROD failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: AWS::Logs::AccountPolicy" (RequestToken: 717f24d1-38cd-2b20-b7a1-1b04be946d9f, HandlerErrorCode: InvalidRequest)

Reproduction Steps

import json
from pathlib import Path as Lib_Path

import aws_cdk as cdk
from aws_cdk import aws_logs as logs
from constructs import Construct
from private_internal_lib import EnvironmentName, Utils


class CloudwatchDataProtectionStack(cdk.Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        env_name = EnvironmentName[self.node.try_get_context("env").upper()]

        _config = Utils.get_config_from_context(
            self.node, "cloudwatch-data-protection", env_name
        )

        _audit_log_group = logs.LogGroup(
            self,
            "AuditGroup",
            log_group_name="audit-findings",
            retention=logs.RetentionDays[_config["log_retention"]],
        )

        # Load Policy Doc
        dirname = Lib_Path(__file__).parent
        with Lib_Path.open(Lib_Path(dirname / "policy.json")) as json_file:
            json_data = json.load(json_file)

        _cfn_account_policy = logs.CfnAccountPolicy(
            self,
            "MyCfnAccountPolicy",
            policy_document=json.dumps(json_data),
            policy_name="DATAPROTECTION",
            policy_type="DATA_PROTECTION_POLICY",
            # the properties below are optional
            scope="ALL",
        )

The Policy file in use is below

{
    "Name": "ACCOUNT_DATA_PROTECTION_POLICY",
    "Description": "",
    "Version": "2021-06-01",
    "Statement": [
        {
            "Sid": "audit-policy",
            "DataIdentifier": [
                "arn:aws:dataprotection::aws:data-identifier/CreditCardMagneticStripe",
                "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber",
                "arn:aws:dataprotection::aws:data-identifier/CreditCardSecurityCode",
                "arn:aws:dataprotection::aws:data-identifier/DriversLicense-AU",
                "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-CA",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-DE",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-ES",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-FR",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-GB",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-IT",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-US",
                "arn:aws:dataprotection::aws:data-identifier/Address",
                "Medicare"
            ],
            "Operation": {
                "Audit": {
                    "FindingsDestination": {
                        "CloudWatchLogs": {
                            "LogGroup": "/audit-findings"
                        }
                    }
                }
            }
        },
        {
            "Sid": "redact-policy",
            "DataIdentifier": [
                "arn:aws:dataprotection::aws:data-identifier/CreditCardMagneticStripe",
                "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber",
                "arn:aws:dataprotection::aws:data-identifier/CreditCardSecurityCode",
                "arn:aws:dataprotection::aws:data-identifier/DriversLicense-AU",
                "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-CA",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-DE",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-ES",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-FR",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-GB",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-IT",
                "arn:aws:dataprotection::aws:data-identifier/PassportNumber-US",
                "arn:aws:dataprotection::aws:data-identifier/Address",
                "Medicare"
            ],
            "Operation": {
                "Deidentify": {
                    "MaskConfig": {}
                }
            }
        }
    ],
    "Configuration": {
        "CustomDataIdentifier": [
            {
                "Name": "Medicare",
                "Regex": "^[2-6]{1}\\d{3}[ ]?\\d{5}[ ]?\\d{1}[- ]?\\d?"
            }
        ]
    }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.154.1 (build febce9d)

Framework Version

No response

Node.js Version

v20.10.0

OS

MacOS 14.6.1

Language

Python

Language Version

No response

Other information

No response

[khushail]

Hi @LucasSymons , thanks for reporting this.
I am able to repro the issue given this code -

                with open('/policy_doc.json', 'r') as file:
                    data = json.load(file)
          
                cfn_account_policy = logs.CfnAccountPolicy(self, "MyCfnAccountPolicy",
                policy_document=json.dumps(data),
                policy_name="policyName",
                policy_type="DATA_PROTECTION_POLICY",
                scope="ALL",
                )

Although the cdk synth succeeded but error observed during cdk deploy -

Since CfnAccountPolicy is L1 construct , this is a Cloudformation issue.

[khushail]

Internal ticket filed with Cloudformation support - P151743364

[garysassano]

I experienced the same exact issue yesterday when trying to create the following resource:

new CfnAccountPolicy(this, "LogsAccountPolicy", {
  policyName: "LambdaSubscriptionPolicy",
  policyDocument: JSON.stringify({
    destinationArn: logProcessorFunction.functionArn,
    filterPattern: "{ $.__otel_otlp_stdout = * }",
    distribution: "Random",
  }),
  policyType: "SUBSCRIPTION_FILTER_POLICY",
  scope: "ALL",
  selectionCriteria: `LogGroupName NOT IN ["/aws/${logProcessorFunction.functionName}"]`,
});

The equivalent SAM template was working just fine:

LogsAccountPolicy:
  Type: AWS::Logs::AccountPolicy
  Properties:
    PolicyName: "LambdaSubscriptionPolicy"
    PolicyDocument: 
      Fn::Sub: |
        {
          "DestinationArn": "${LogProcessorFunction.Arn}",
          "FilterPattern": "{ $.__otel_otlp_stdout = * }",
          "Distribution": "Random"
        }
    PolicyType: "SUBSCRIPTION_FILTER_POLICY"
    Scope: "ALL"
    SelectionCriteria: !Sub "LogGroupName NOT IN [\"/aws/${LogProcessorFunction}\"]"

[nathan-ahn]

The error message is extremely vague here. In my case, I found that it was actually a permissions issue, not an issue with the CfnAccountPolicy itself. Here's the working version for us:

this.addPermission("LogProcessorPermission", {
	principal: new ServicePrincipal("logs.amazonaws.com"),
	action: "lambda:InvokeFunction",
	sourceArn: `arn:aws:logs:${props.env.region}:${props.env.account}:log-group:*`,
	sourceAccount: props.env.account,
});

Be very careful with sourceArn! We previously had :log-group:/* which caused the permissions issue. If you're having issues, I'd highly recommend debugging using the CLI. Errors there are more descriptive.

And if anyone stumbles upon this issue for another reason, here's our full CfnAccountPolicy:

new CfnAccountPolicy(this, "LogProcessorSubscriptionPolicy", {
	policyDocument: JSON.stringify({
		DestinationArn: this.functionArn,
		FilterPattern: `{ ($.level = "warn") || ($.level = "error") || ($.level = "fatal") || ($.errorMessage = *) }`,
		Distribution: "Random",
	}),
	policyName: "LogProcessorSubscriptionPolicy",
	policyType: "SUBSCRIPTION_FILTER_POLICY",
	scope: "ALL",
	selectionCriteria: `LogGroupName NOT IN ["${this.LogGroup.logGroupName}"]`,
});