[Bug]: s3: cannot delete versioned bucket with DeleteMarkers only

[zarnovican]

Terraform Core Version

1.3.9

AWS Provider Version

5.29.0

Affected Resource(s)

• aws_s3_bucket

Expected Behavior

terraform destroy should delete the bucket

Actual Behavior

terraform destroy loops endlessly

aws_s3_bucket.main: Still destroying... [id=terraform-delete-versioned-20231207132331260400000001, 1m10s elapsed]
aws_s3_bucket.main: Still destroying... [id=terraform-delete-versioned-20231207132331260400000001, 1m20s elapsed]
aws_s3_bucket.main: Still destroying... [id=terraform-delete-versioned-20231207132331260400000001, 1m30s elapsed]
aws_s3_bucket.main: Still destroying... [id=terraform-delete-versioned-20231207132331260400000001, 1m40s elapsed]
..

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
#      version = "5.28.0"
      version = "5.29.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

resource "aws_s3_bucket" "main" {
  bucket_prefix = "terraform-delete-versioned-"
  force_destroy = true
}

resource "aws_s3_bucket_ownership_controls" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

resource "aws_s3_bucket_public_access_block" "main" {
  bucket = aws_s3_bucket.main.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "main" {
  bucket = aws_s3_bucket.main.id
  versioning_configuration {
    status = "Enabled"
  }
}

output "s3_bucket_name" {
  value = aws_s3_bucket.main.id
}

Steps to Reproduce

echo foo > foo.txt
terraform init -upgrade
terraform apply -auto-approve
aws s3 cp foo.txt s3://$(terraform output -raw s3_bucket_name)/foo.txt
aws s3 rm s3://$(terraform output -raw s3_bucket_name)/foo.txt
terraform destroy -auto-approve

Debug Output

No response

Panic Output

No response

Important Factoids

The problem seems to be triggered, when there are only DeleteMarkers on versioned bucket. In the reproducer, I have uploaded and deleted a single file.

It looks like, this issue was introduced in 5.29.0. My reproducer does not work on 5.28.0.

Workaround:

Delete the DeleteMarkers manually:

terraform refresh
aws s3api delete-objects --bucket $(terraform output -raw s3_bucket_name) --delete \
    "$(aws s3api list-object-versions --bucket $(terraform output -raw s3_bucket_name) --output json | \
    jq 'select(.DeleteMarkers != null) | {Objects: [.DeleteMarkers[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

References

Could be related to #34647 (based on reading the release-notes).

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

@zarnovican Thanks for raising this issue.
It has already been noticed in #34706. I'm going to close this one as a duplicate so that we can concentrate discussion in the linked issue.
Please add any additional comments there.

[zarnovican]

Before raising the issue, I did review existing tickets, but it didn't occur to me to review also closed ones. Sorry for the noise and thanks for fixing it.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.