Error: InvalidParameterValueException: Task Settings CloudWatchLogGroup or CloudWatchLogStream cannot be set on create.

[vadzimkaredzinkokoba]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.9
on darwin_amd64

• provider registry.terraform.io/hashicorp/archive v2.2.0
• provider registry.terraform.io/hashicorp/aws v3.63.0
• provider registry.terraform.io/hashicorp/random v3.1.0
• provider registry.terraform.io/hashicorp/template v2.2.0
• provider registry.terraform.io/hashicorp/tls v3.1.0

Affected Resource(s)

• aws_dms_replication_task

Terraform Configuration Files

///////////////////////////////
// DMS endpoints
resource "aws_dms_endpoint" "******_dms_dump_source_endpoint" {
  count                       = local.******_dms_rds_dump_run
  database_name               = var.******_db_database_name
  endpoint_id                 = "${var.******_prefix}-****-dms-source-endpoint"
  endpoint_type               = var.******_dms_dump_source_endpoint_type
  engine_name                 = var.******_dms_replication_instance_engine_name
  username                    = var.******_db_database_username
  password                    = aws_db_instance.******_rds_replica_instance[count.index].password
  port                        = var.******_dms_replication_instance_port
  server_name                 = aws_db_instance.******_rds_replica_instance[count.index].endpoint #must be name without port!
  ssl_mode                    = var.******_dms_replication_instance_ssl_mode

  tags = {
    Name = "${var.******_prefix}-****-dms-source-enpoint-name"
  }
}

resource "aws_dms_endpoint" "******_dms_dump_target_endpoint" {
  count                       = local.******_dms_rds_dump_run
  database_name               = var.******_db_dump_database_name
  endpoint_id                 = "${var.******_prefix}-****-dms-target-endpoint"
  endpoint_type               = var.******_dms_dump_target_endpoint_type
  engine_name                 = var.******_dms_replication_instance_engine_name
  username                    = var.******_db_dump_database_username
  password                    = aws_db_instance.******_rds_dump_instance[count.index].password
  port                        = var.******_dms_replication_instance_port
  server_name                 = aws_db_instance.******_rds_dump_instance[count.index].endpoint #must be name without port!
  ssl_mode                    = var.******_dms_replication_instance_ssl_mode

  tags = {
    Name = "${var.******_prefix}-****-dms-target-enpoint-name"
  }
}

//DMS subnet group
resource "aws_dms_replication_subnet_group" "******_dms_replication_subnet_group" {
  count                                = local.******_dms_rds_dump_run
  replication_subnet_group_description = "Subnet group for replication instance"
  replication_subnet_group_id          = "${var.******_prefix}-****-dms-replication-subnet-group"

  subnet_ids = [
    aws_subnet.******_private_a_zone.id,
    aws_subnet.******_private_c_zone.id
  ]

  tags = {
    Name = "${var.******_prefix}-****-dms-replication-subnet-group-name"
  }
}

# DMS - replication instance
resource "aws_dms_replication_instance" "******_replication_instance" {
  count                        = local.******_dms_rds_dump_run
  allocated_storage            = var.******_dms_replication_instance_allocated_storage
  apply_immediately            = var.******_dms_replication_instance_apply_immediately
  auto_minor_version_upgrade   = var.******_dms_replication_instance_auto_minor_version_upgrade
  availability_zone            = var.******_dms_replication_instance_availability_zone
  engine_version               = var.******_dms_replication_instance_engine_version
  kms_key_arn                  = aws_kms_key.******_dms_key[count.index].arn
  multi_az                     = var.******_dms_replication_instance_multi_az
  preferred_maintenance_window = var.******_dms_replication_instance_maintenance_window
  publicly_accessible          = var.******_dms_replication_instance_publicly_accessible
  replication_instance_class   = var.******_dms_replication_instance_class
  replication_instance_id      = "${var.******_prefix}-****-dms-replication-instance"
  replication_subnet_group_id  = aws_dms_replication_subnet_group.******_dms_replication_subnet_group[count.index].id

  tags = {
    Name                       = "${var.******_prefix}-****-dms-replication-instance-name"
  }

  vpc_security_group_ids = [
   aws_security_group.******_rds_instance_sg.id
  ]

  depends_on = [
    aws_iam_role_policy_attachment.******_dms_cloudwatch_logs_role_policy_attachment,
    aws_iam_role_policy_attachment.******_dms_vpc_role_policy_attachment
  ]
}

# DMS replication task
resource "aws_dms_replication_task" "******_replication_task" {
  count                     = local.******_dms_rds_dump_run
  migration_type            = var.******_dms_replication_task_migration_type
  replication_instance_arn  = aws_dms_replication_instance.******_replication_instance[count.index].replication_instance_arn
  replication_task_id       = "${var.******_prefix}-****-dms-replication-task-id"
  replication_task_settings = jsonencode(
{
  "Logging": {
      "EnableLogging": true,
      "LogComponents": [
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "TRANSFORMATION"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "SOURCE_UNLOAD"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "IO"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "TARGET_LOAD"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "PERFORMANCE"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "SOURCE_CAPTURE"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "SORTER"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "REST_SERVER"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "VALIDATOR_EXT"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "TARGET_APPLY"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "TASK_MANAGER"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "TABLES_MANAGER"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "METADATA_MANAGER"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "FILE_FACTORY"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "COMMON"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "ADDONS"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "DATA_STRUCTURE"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "COMMUNICATION"
          },
          {
              "Severity": "LOGGER_SEVERITY_DEFAULT",
              "Id": "FILE_TRANSFER"
          }
      ],
      "CloudWatchLogGroup": "${aws_cloudwatch_log_group.******_dms_replication_task_cloudwatch_log_group[count.index].name}",
      "CloudWatchLogStream": "${aws_cloudwatch_log_stream.******_dms_replication_task_cloudwatch_log_stream[count.index].name}"
  },
  "StreamBufferSettings": {
      "StreamBufferCount": 3,
      "CtrlStreamBufferSizeInMB": 5,
      "StreamBufferSizeInMB": 8
  },
  "ErrorBehavior": {
      "FailOnNoTablesCaptured": true,
      "ApplyErrorUpdatePolicy": "LOG_ERROR",
      "FailOnTransactionConsistencyBreached": false,
      "RecoverableErrorThrottlingMax": 1800,
      "DataErrorEscalationPolicy": "SUSPEND_TABLE",
      "ApplyErrorEscalationCount": 0,
      "RecoverableErrorStopRetryAfterThrottlingMax": true,
      "RecoverableErrorThrottling": true,
      "ApplyErrorFailOnTruncationDdl": false,
      "DataTruncationErrorPolicy": "LOG_ERROR",
      "ApplyErrorInsertPolicy": "LOG_ERROR",
      "ApplyErrorEscalationPolicy": "LOG_ERROR",
      "RecoverableErrorCount": -1,
      "DataErrorEscalationCount": 0,
      "TableErrorEscalationPolicy": "STOP_TASK",
      "RecoverableErrorInterval": 5,
      "ApplyErrorDeletePolicy": "IGNORE_RECORD",
      "TableErrorEscalationCount": 0,
      "FullLoadIgnoreConflicts": true,
      "DataErrorPolicy": "LOG_ERROR",
      "TableErrorPolicy": "SUSPEND_TABLE"
  },
  "FullLoadSettings": {
      "CommitRate": var.******_dms_replication_task_replication_task_settings_commitrate,
      "StopTaskCachedChangesApplied": false,
      "StopTaskCachedChangesNotApplied": false,
      "MaxFullLoadSubTasks": var.******_dms_replication_task_replication_task_settings_maxfullloadsubtasks,
      "TransactionConsistencyTimeout": 600,
      "CreatePkAfterFullLoad": false,
      "TargetTablePrepMode": "DROP_AND_CREATE"
  },
  "TargetMetadata": {
      "ParallelApplyBufferSize": 0,
      "ParallelApplyQueuesPerThread": 0,
      "ParallelApplyThreads": 0,
      "TargetSchema": "",
      "InlineLobMaxSize": 0,
      "ParallelLoadQueuesPerThread": 0,
      "SupportLobs": true,
      "LobChunkSize": 0,
      "TaskRecoveryTableEnabled": false,
      "ParallelLoadThreads": 0,
      "LobMaxSize": 32,
      "BatchApplyEnabled": false,
      "FullLobMode": false,
      "LimitedSizeLobMode": true,
      "LoadMaxFileSize": 0,
      "ParallelLoadBufferSize": 0
  },
  "BeforeImageSettings": null,
  "ControlTablesSettings": {
      "historyTimeslotInMinutes": 5,
      "HistoryTimeslotInMinutes": 5,
      "StatusTableEnabled": false,
      "SuspendedTablesTableEnabled": false,
      "HistoryTableEnabled": false,
      "ControlSchema": "",
      "FullLoadExceptionTableEnabled": false
  },
  "LoopbackPreventionSettings": null,
  "CharacterSetSettings": null,
  "FailTaskWhenCleanTaskResourceFailed": false,
  "ChangeProcessingTuning": {
      "StatementCacheSize": 50,
      "CommitTimeout": 1,
      "BatchApplyPreserveTransaction": true,
      "BatchApplyTimeoutMin": 1,
      "BatchSplitSize": 0,
      "BatchApplyTimeoutMax": 30,
      "MinTransactionSize": 1000,
      "MemoryKeepTime": 60,
      "BatchApplyMemoryLimit": 500,
      "MemoryLimitTotal": 1024
  },
  "ChangeProcessingDdlHandlingPolicy": {
      "HandleSourceTableDropped": true,
      "HandleSourceTableTruncated": true,
      "HandleSourceTableAltered": true
  },
  "PostProcessingRules": null
}
)
  
  table_mappings            = jsonencode(
  {
  "rules": [
    {
      "rule-type": "selection",
      "rule-id": "1",
      "rule-name": "1",
      "object-locator": {
        "schema-name": var.******_db_database_name,
        "table-name": "%"
      },
      "rule-action": "include",
      "filters": []
    }
  ]
}
)
  tags = {
    Name = "${var.******_prefix}-****-dms-replication-task-name"
  }
  source_endpoint_arn       = aws_dms_endpoint.******_dms_dump_source_endpoint[count.index].endpoint_arn
  target_endpoint_arn       = aws_dms_endpoint.******_dms_dump_target_endpoint[count.index].endpoint_arn
}

_____________________VARS__________________________

// DMS task
variable "******_dms_rds_dump_run" {
  default = false
}

variable "******_dms_replication_task_migration_type" {
  default = "full-load"
}

variable "******_dms_replication_task_replication_task_settings_commitrate" {
  default = 10000
}

variable "******_dms_replication_task_replication_task_settings_maxfullloadsubtasks" {
  default = 8
}
_________________________CLOUD WATCH___________

//DMS replication task logs
resource "aws_cloudwatch_log_group" "*********_dms_replication_task_cloudwatch_log_group" {
  count = local.*********_dms_rds_dump_run
  name  = "${var.*********_prefix}-dms-replication-task-cloudwatch-log-group-name"
}

resource "aws_cloudwatch_log_stream" "*********_dms_replication_task_cloudwatch_log_stream" {
  count          = local.*********_dms_rds_dump_run
  name           = "${var.*********_prefix}-dms-replication-task-cloudwatch-log-stream-name"
  log_group_name = aws_cloudwatch_log_group.*********_dms_replication_task_cloudwatch_log_group[count.index].name
}

Debug Output

Error: InvalidParameterValueException: Task Settings CloudWatchLogGroup or CloudWatchLogStream cannot be set on create.

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

1. terraform apply

Important Factoids

If I comment this parameters and uncomment later

     #"CloudWatchLogGroup": "${aws_cloudwatch_log_group.******_dms_replication_task_cloudwatch_log_group[count.index].name}",
      #"CloudWatchLogStream":"${aws_cloudwatch_log_stream.******_dms_replication_task_cloudwatch_log_stream[count.index].name}"

InvalidParameterValueException: Task Settings CloudWatchLogGroup or CloudWatchLogStream cannot be modified.

If I comment this parameters and apply - replication task will be created with default CloudWatchLogGroup and CloudWatchLogStream

References

• aws_dms_replication_task replication_task_settings always reports as needing modification #1513

[philMarius]

Is there an update on this issue?

[chalmagr]

This does not seem to be a problem with terraform itself - I am getting the same using CloudFormation.
This is an AWS API limitation it seems...
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.Saving.html

[chalmagr]

This does not seem to be a problem with terraform itself - I am getting the same using CloudFormation. This is an AWS API limitation it seems... https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.Saving.html

And it cannot be updated after creation either...

Task Settings CloudWatchLogGroup or CloudWatchLogStream cannot be modified. (Service: AWSDatabaseMigrationService; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: #####; Proxy: null)

[chalmagr]

If you need to configure the log group what I found out that it works is creating the log group before using the name it will use in the task. The name of the log group is

dms-tasks-<replicationInstanceName>

[casey-robertson-paypal]

interesting - I need to confirm how mine is set up. I was defining them in a task json and saw similar behavior so I just left them out and let DMS create them. So it appears I'm running into the same bug but wasn't aware of it. But now that you say you can pre-create and reference maybe I'll give that a go. Does that result in a rebuild of the task or will it change it on the fly?

[casey-robertson-paypal]

If you need to configure the log group what I found out that it works is creating the log group before using the name it will use in the task. The name of the log group is

dms-tasks-

By creating before, do you mean using some kind of 'depends_on' or running the cloudwatch terraform first and then running any DMS terraform? If the latter that's really not ideal esp. if we've already created and run our tasks :-(

[alpha3b1]

The error is that you specified the log group in the definition of replication_task_settings

[samuelrcarvalho]

I have the same problem

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.46.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.