aws db parameter group ... converts keys and values to lower case and fails 'apply' due to aws_db_parameter_group changes

[ghost]

This issue was originally opened by @ebgc as hashicorp/terraform#9215. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

mac os x 10.12
Terraform v0.7.4

Affected Resource(s)

• resource_aws_db_parameter_group
• resource_aws_rds_cluster_parameter_group
• aws_db_instance

Terraform Configuration Files

resource "aws_db_instance" "default" {
  allocated_storage = "${var.db_storage}"
  allow_major_version_upgrade = false
  apply_immediately = false
  auto_minor_version_upgrade = false
  backup_retention_period = "${var.backup_retention_period}"
  backup_window = "${var.backup_window}"
  copy_tags_to_snapshot = "${var.copy_tags_to_snapshot}"
  db_subnet_group_name = "${aws_db_subnet_group.rds.id}"
  engine = "${var.db_engine}"
  engine_version = "${var.db_version}"
  final_snapshot_identifier = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.date}"
  identifier = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
  instance_class = "${var.db_instance}"
  maintenance_window = "${var.maintenance_window}"
  multi_az = "${var.multi_az}"
  name = "${var.db_name}"
  parameter_group_name = "${aws_db_parameter_group.default.id}"
  password = "${var.db_password}"
  publicly_accessible = "${var.db_public_accessible}"
  skip_final_snapshot = "${var.skip_final_snapshot}"
  snapshot_identifier = "${var.db_snapshot_id}"
  storage_type = "${var.db_storage_type}"
  username = "${var.db_username}"
  vpc_security_group_ids = ["${aws_security_group.rds.id}"]

  lifecycle {
    ignore_changes = ["final_snapshot_identifier"]
  }

  tags {
    Name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
    Description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
    Environment = "${var.environment}"
  }
}

resource "aws_db_parameter_group" "default" {
  name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
  description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
  family = "${data.template_file.db_family_dot.rendered}"

  # tf ignores them : timezone, client_encoding
  parameter {
    name = "lc_messages"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_monetary"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_numeric"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_time"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  tags {
    Name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
    Description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
    Environment = "${var.environment}"
  }
  lifecycle {
    ignore_changes = ["name", "description", "family", "tags"]
  }
}

Debug Output

1 error(s) occurred:

• aws_db_parameter_group.default: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.7.4
Resource ID: aws_db_parameter_group.default
Mismatch reason: attribute mismatch: parameter.1848500544.value
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"parameter.1848500544.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.2080321556.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.4193353794.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.3444491795.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Panic Output

n.a.

Expected Behavior

take what aws api returns without modification (dont use flattenParameters)
save what tf file defines at tfstate file without modification (dont use flattenParameters)

Actual Behavior

all keys and values will be down-cased on tfstate file even though on initial run they are set on aws as defined in tf file

Steps to Reproduce

have an aws_db_instance (postgres, 9.4.5) with resource_aws_db_parameter_group (postgres9.4) which sets 'lc_messages' to 'en_US.utf8', or 'timezone' to 'UTF'

even though we set 'en_US.utf8' in tf file (terraform during 'apply' will set 'en_US.utf8' on aws rds resource) - tfstate file will have 'en_us.utf8' set.
next run of terraform 'apply' will read resource from aws and convert it to lower case 'en_us.utf8' (see below 'flattenParameters' function) which will not fail compared to state file value (match). but because there is a diff with tf file (remember we set 'en_US.utf8') value it will try to modify the resource.
here another bug comes in play where terraform complains that db parameter group has a diff and reports: '* aws_db_parameter_group.default: diffs didn't match during apply.'

Important Factoids

---- terraform/builtin/providers/aws/resource_aws_db_parameter_group.go ----
func resourceAwsDbParameterGroupRead
...
d.Set("parameter", flattenParameters(describeParametersResp.Parameters))
...
---- terraform/builtin/providers/aws/resource_aws_db_parameter_group.go ----
func flattenParameters
...
r["name"] = strings.ToLower(_i.ParameterName)
...
r["value"] = strings.ToLower(_i.ParameterValue)
....
---- terraform/builtin/providers/aws/structure.go ----

---- terraform/builtin/providers/aws/structure.go ----

References

[breathingdust]

This also affects rds_cluster_parameter_group

[jochen-st]

Issue is still present with

Terraform v0.14.4

• provider registry.terraform.io/hashicorp/aws v3.24.0

In contrast to original report the tfstate file is now case sensitive. We have the following (shortened):

terraform file:

    {
      name  = "innodb_file_format"
      value = "Barracuda"
      apply_method = "pending-reboot"
    },

tfstate file:

              {
                "apply_method": "pending-reboot",
                "name": "innodb_file_format",
                "value": "Barracuda"
              },

Plan output:

      + parameter {
          + apply_method = "pending-reboot"
          + name         = "innodb_file_format"
          + value        = "Barracuda"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "innodb_file_format" -> null
          - value        = "barracuda" -> null
        }

Terraform trace log:

      - .parameter: planned set element cty.ObjectVal(map[string]cty.Value{"apply_method":cty.StringVal("pending-reboot"), "name":cty.StringVal("innodb_file_format"), "value":cty.StringVal("Barracuda")}) does not correlate with any element in actual

[bill-rich]

I've been looking into this and I believe I have a good idea of what is going on, but I would like to confirm I'm matching the cases that are being reported. @jochen-st or @ebgc, can you please post a complete Terraform config I can run that will replicate your issue?

[jochen-st]

@bill-rich Please find below the Terraform config. I was using RDS Module.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.27.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

data "aws_vpc" "default" {
  default = true
}

data "aws_subnet_ids" "all" {
  vpc_id = data.aws_vpc.default.id
}

resource "aws_security_group" "mysql" {
  name = "issue15583-sg"
  vpc_id      = data.aws_vpc.default.id

  ingress {
    from_port   = 3306
    to_port     = 3306
    protocol    = "tcp"
    description = "mysql"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  description = "Security group for example usage with Mysql instance"
}

module "db-mysql" {
  source  = "terraform-aws-modules/rds/aws"
  version = "~> 2.0"

  identifier = "mysql-issue15583"

  engine            = "mysql"
  engine_version    = "5.7.31"
  instance_class    = "db.t3.small"
  allocated_storage = 20

  username = "dbadmin"
  password = "dbpassword"
  port     = "3306"

  vpc_security_group_ids = [ aws_security_group.mysql.id ]

  maintenance_window = "Mon:00:00-Mon:03:00"
  backup_window      = "03:00-05:00"
  auto_minor_version_upgrade = false

  # disable backups to create DB faster
  backup_retention_period = 0

  subnet_ids = data.aws_subnet_ids.all.ids

  family = "mysql5.7"
  major_engine_version = "5.7"
  final_snapshot_identifier = "mysql-final-snapshot"
  deletion_protection = false

  # MySQL Parameters
  parameters = [
    {
      name  = "character_set_server"
      value = "utf8mb4"
      apply_method = "pending-reboot"
    },
    {
      name  = "innodb_default_row_format"
      value = "DYNAMIC"
      apply_method = "pending-reboot"
    },
    {
      name  = "innodb_large_prefix"
      value = 1
      apply_method = "pending-reboot"
    },
    {
      name  = "innodb_file_format"
      value = "Barracuda"
      apply_method = "pending-reboot"
    },
    {
      name  = "innodb_log_file_size"
      value = 2147483648
      apply_method = "pending-reboot"
    },
    {
      name  = "sql_mode"
      value = "STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION"
      apply_method = "pending-reboot"
    },
    {
      name  = "innodb_log_buffer_size"
      value = 268435456
      apply_method = "pending-reboot"
    },
    {
      name  = "max_allowed_packet"
      value = 268435456
      apply_method = "pending-reboot"
    },
  ]
}

After apply, the terraform plan looks like this:

Terraform will perform the following actions:

  # module.db-mysql.module.db_parameter_group.aws_db_parameter_group.this[0] will be updated in-place
  ~ resource "aws_db_parameter_group" "this" {
        id          = "mysql-issue15583-20210220180559128900000003"
        name        = "mysql-issue15583-20210220180559128900000003"
        tags        = {
            "Name" = "mysql-issue15583"
        }
        # (4 unchanged attributes hidden)

      + parameter {
          + apply_method = "pending-reboot"
          + name         = "innodb_default_row_format"
          + value        = "DYNAMIC"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "innodb_default_row_format" -> null
          - value        = "dynamic" -> null
        }
      + parameter {
          + apply_method = "pending-reboot"
          + name         = "innodb_file_format"
          + value        = "Barracuda"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "innodb_file_format" -> null
          - value        = "barracuda" -> null
        }
      + parameter {
          + apply_method = "pending-reboot"
          + name         = "sql_mode"
          + value        = "STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "sql_mode" -> null
          - value        = "strict_trans_tables,no_zero_in_date,no_zero_date,error_for_division_by_zero,no_auto_create_user,no_engine_substitution" -> null
        }
        # (5 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Note that parameters with mixed case (like innodb_default_row_format) will show as changed, where parameters with all lower case or numbers (like character_set_server or innodb_log_file_size) remain unchanged.

[ghost]

This has been released in version 3.33.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!