AWS tests failing with "Error creating S3 bucket: TooManyBuckets"

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-05-07T02:39:59.262+0000

• Duration: 65 min 5 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2886
Skipped	181
Total	3067

Steps errors

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws

• Took 4 min 50 sec . View more details here
• Description: terraform apply -auto-approve

Terraform Apply on x-pack/metricbeat/module/aws

• Took 0 min 15 sec . View more details here
• Description: terraform apply -auto-approve

Error signal

• Took 0 min 0 sec . View more details here
• Description: startCloudTestEnv: terraform apply failed.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

[2023-05-07T02:58:46.611Z] │ Error: Error creating S3 bucket: TooManyBuckets: You have attempted to create more buckets than allowed
[2023-05-07T02:58:46.611Z] │ 	status code: 400, request id: AGZ2469TZCZQCJ5W, host id: umaCViHD/oClxR0bv29MIjK6kdZffE8Ydw/Wz5rdJ+VlRRDyxEaO3GLNhnOVZ6Yvt3G2jJemrOY=
[2023-05-07T02:58:46.611Z] │ 
[2023-05-07T02:58:46.611Z] │   with aws_s3_bucket.test,
[2023-05-07T02:58:46.611Z] │   on terraform.tf line 43, in resource "aws_s3_bucket" "test":
[2023-05-07T02:58:46.611Z] │   43: resource "aws_s3_bucket" "test" {

[cmacknz]

@jsoriano looks like you added the Terraform file here a long time ago, any idea how to fix this? #17480

CC @v1v

[jsoriano]

TooManyBuckets

Umm, I barely remember this, but is it possible that buckets are not being correctly cleaned up?

[v1v]

There is a last resort automation we built to delete cloud resources as long as they are tagged, @amannocci , can you confirm whether those s3 buckets are deleted with the existing cloud-reaper? Otherwise they are a good candidate for being added

[v1v]

Additionally, it is possible to request a higher quota, or at least we have done a similar kind of requests for some other aws resources, that's something everyone with admin request can request, just in case you wanna give a go, otherwise we can request it sometime tomorrow

[amannocci]

There is a last resort automation we built to delete cloud resources as long as they are tagged, @amannocci , can you confirm whether those s3 buckets are deleted with the existing cloud-reaper? Otherwise they are a good candidate for being added

We have automation in place to delete orphan S3 buckets related to Beats CI.
We reached the limit of 100 buckets on this AWS account. It seems related to various manual bucket creation.
I will reach out to the owners to check if they are all needed. However, we can ask for a raise of the quota.

[v1v]

Thanks @amannocci.

I just deleted some old S3 buckets manually and requested some higher quota. Unfortunately, mostly all of the existing S3 buckets don't have any TAGs or meaningful names so we can know who to ask whether those S3 buckets are needed or not :/