[Bug]: aws_cloudwatch_metric_alarm with aws_oam_link and forbidden ValidationError

[KevinBrooke]

Terraform Core Version

v1.5.0

AWS Provider Version

v4.67.0

Affected Resource(s)

aws_cloudwatch_metric_alarm

Expected Behavior

During a new foundation deployment we configure an alerting account using OAM sink and links from member accounts to send their metrics to the alerting account.
The metrics and links are created, then the alarms fail to create with the error: ValidationError: One or more metrics in your request are Forbidden.

The deployment works with no changes after multiple applys.

I added in a null_resource to verify the metrics are available in the alerting account and added it as a dependency to the alerts resource, however the results are mixed, sometimes they all work, sometimes they all fail, other times a random number of them work.

The aws_cloudwatch_metric_alarm resource should be created with an option to ignore validation.
If this is not possible, it should have a "validation retry/wait/count" ability to wait for a defined time for the metric to become available.

Actual Behavior

ValidationError: One or more metrics in your request are Forbidden.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

main.tf

provider "aws" {
  alias  = "alarm"
  region = "eu-west-2"
  assume_role {
    role_arn = "arn:aws:iam::ACCOUNT_1_ID:role/ACCOUNT_1_ROLE_NAME"
  }
}

provider "aws" {
  alias  = "member"
  region = "eu-west-2"
  assume_role {
    role_arn = "arn:aws:iam::ACCOUNT_2_ID:role/ACCOUNT_2_ROLE_NAME"
  }
}

data "aws_caller_identity" "member" {
  provider = aws.member
}

locals {
  member_account_id         = data.aws_caller_identity.member.account_id
  sink_identifier           = "mysink"
  cloudwatch_log_group_name = "cloudwatch_logs" # Assumes CloudTrail is already configured to send logs to CloudWatch Logs
}

resource "aws_oam_sink" "this" {
  provider        = aws.alarm
  sink_identifier = local.sink_identifier
}

resource "aws_oam_sink_policy" "this" {
  provider        = aws.alarm
  sink_identifier = aws_oam_sink.this.sink_identifier
  policy = jsonencode(
    {
      Statement = [
        {
          Action = [
            "oam:CreateLink",
            "oam:UpdateLink",
          ]
          Condition = {
            "ForAllValues:StringEquals" = {
              "oam:ResourceTypes" = [
                "AWS::CloudWatch::Metric",
                "AWS::Logs::LogGroup",
              ]
            }
          }
          Effect = "Allow"
          Principal = {
            AWS = [
              data.aws_caller_identity.member.account_id
            ]
          }
          Resource = "*"
        },
      ]
      Version = "2012-10-17"
    }
  )
}

resource "aws_oam_link" "this" {
  provider        = aws.member
  sink_identifier = aws_oam_sink.this.arn
  label_template  = "$AccountName"
  resource_types  = ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup"]
}

resource "aws_cloudwatch_log_metric_filter" "this" {
  provider       = aws.member
  log_group_name = local.cloudwatch_log_group_name
  name           = "UnauthorizedAPICalls"
  pattern        = "{ ($.errorCode=\"*UnauthorizedOperation\") || ($.errorCode=\"AccessDenied*\") }"

  metric_transformation {
    default_value = "0"
    name          = format("%s_UnauthorizedAPICalls", local.member_account_id)
    namespace     = "CISBenchmark"
    unit          = "None"
    value         = "1"
  }
}

# Metric alarm in the child account
resource "aws_cloudwatch_metric_alarm" "member_account" {
  provider            = aws.member
  actions_enabled     = false
  alarm_description   = "Monitoring unauthorized API calls will help reveal application errors and may reduce time to detect malicious activity."
  alarm_name          = aws_cloudwatch_log_metric_filter.this.name
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = aws_cloudwatch_log_metric_filter.this.name
  namespace           = "CISBenchmark"
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  treat_missing_data  = "notBreaching"
}

# Metric alarm in alarm account
resource "aws_cloudwatch_metric_alarm" "this" {
  provider            = aws.alarm
  actions_enabled     = true
  alarm_description   = "Monitoring unauthorized API calls will help reveal application errors and may reduce time to detect malicious activity."
  alarm_name          = aws_cloudwatch_metric_alarm.member_account.alarm_name
  comparison_operator = "GreaterThanOrEqualToThreshold"
  datapoints_to_alarm = 1
  evaluation_periods  = 1
  threshold           = 1
  treat_missing_data  = "notBreaching"

  metric_query {
    account_id  = local.member_account_id
    id          = "m1"
    period      = 0
    return_data = true

    metric {
      metric_name = aws_cloudwatch_log_metric_filter.this.name
      namespace   = "CISBenchmark"
      period      = 300
      stat        = "Sum"
    }
  }
  depends_on = [aws_oam_link.this, aws_oam_sink.this]
}

Steps to Reproduce

Using 2 AWS accounts with CloudTrail configured to send its logs to their own log groups.
Create a role in each account to assume for the deployment.
Update locals, regions and the assume roles.

terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.