Lake Formation permissions: Permission changes not forcing new with IAM_ALLOWED_PRINCIPALS

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.1

• provider registry.terraform.io/hashicorp/aws v3.47.0

Affected Resource(s)

• aws_lakeformation_permissions

Terraform Configuration Files

Adding IAM_ALLOWED_PRINCIPALS permissions has no specific benefit but mimics what happens automatically in some accounts (e.g., not upgraded to Lake Formation model). Also, this can be done manually by practitioners. As such, it would be ideal if the AWS provider can work seamlessly with or without these extra permissions. Whether that is possible remains to be seen.

resource "aws_glue_catalog_database" "test" {
  name = "madagascar-database"
}

resource "aws_glue_catalog_table" "test" {
  name          = "barbados-table"
  database_name = aws_glue_catalog_database.test.name

  storage_descriptor {
    columns {
      name = "rating"
      type = "DOUBLE"
    }

    columns {
      name = "year"
      type = "STRING"
    }

    columns {
      name = "movie_title"
      type = "STRING"
    }
  }
}

resource "aws_lakeformation_data_lake_settings" "test" {
  admins = ["arn:aws:iam::<account_id>:role/<role_name>"]
}

resource "aws_iam_role" "test" {
  name = "odewissug-svalbard"
  path = "/"

  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "glue.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

# this permission replicates what AWS does in accounts that haven't (or can't?) be upgraded to Lake Formation security model
# https://docs.aws.amazon.com/lake-formation/latest/dg/upgrade-glue-lake-formation.html
resource "aws_lakeformation_permissions" "iap1" {
  permissions = ["ALL"]
  principal   = "IAM_ALLOWED_PRINCIPALS"

  database {
    name = aws_glue_catalog_database.test.name
  }
}

# this permission replicates what AWS does in accounts that haven't (or can't?) be upgraded to Lake Formation security model
# https://docs.aws.amazon.com/lake-formation/latest/dg/upgrade-glue-lake-formation.html
resource "aws_lakeformation_permissions" "iap2" {
  permissions = ["ALL"]
  principal   = "IAM_ALLOWED_PRINCIPALS"

  table {
    database_name = aws_glue_catalog_database.test.name
    name          = aws_glue_catalog_table.test.name
  }
}

resource "aws_lakeformation_permissions" "test" {
  permissions = ["SELECT"]
  principal   = aws_iam_role.test.arn

  table_with_columns {
    database_name = aws_glue_catalog_database.test.name
    name          = aws_glue_catalog_table.test.name
    column_names  = ["rating", "year"]

  }
}

Debug Output

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Expected Behavior

Permissions should be updated to fewer columns.

If it were doing ForceNew, it would look like this:

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Actual Behavior

You can add column permissions but you can't remove them:

• ✔️ rating, year 👉 rating, year, movie_title
• 🚫 rating, year, movie_title 👉 rating, year

Steps to Reproduce

1. terraform apply
2. Check console for permissions and principal odewissug-svalbard will now have 2 columns
3. Change aws_lakeformation_permissions.test.table_with_columns.column_names to ["rating", "year", "movie_title"]
4. terraform apply
5. Check console for permissions and principal odewissug-svalbard will now have the 3 columns
6. Change aws_lakeformation_permissions.test.table_with_columns.column_names to ["rating"]
7. terraform apply
8. Check console for permissions and principal odewissug-svalbard will still have the 3 columns but should be 1

Workaround

By manually removing the IAM_ALLOWED_PRINCIPALS permissions from affected tables and databases (e.g., using the AWS console), I have been able to get the AWS provider to correctly manage permissions.

Important Factoids

1. Without the IAM_ALLOWED_PRINCIPALS permissions, the provider works as intended
2. Without the IAM_ALLOWED_PRINCIPALS permissions, Terraform's plan includes ~ column_names = [ # forces replacement and Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
3. With the IAM_ALLOWED_PRINCIPALS permissions, Terraform's plan does not include ~ column_names = [ # forces replacement and shows Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
4. In other words, with the IAM_ALLOWED_PRINCIPALS permissions, the AWS provider is not doing forced replacement as it should, which is why the number of permissions cannot be reduced but only increased. Decreasing requires revoking existing permissions and then granting the new permissions (which is what happens without IAM_ALLOWED_PRINCIPALS).

References

• Lake Formation permissions: Delete problem with IAM_ALLOWED_PRINCIPALS #20047

[github-actions]

This functionality has been released in v3.49.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.