Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): garbage collect s3 assets (under --unstable flag) #31611

Merged
merged 85 commits into from
Oct 21, 2024
Merged

Conversation

kaizencc
Copy link
Contributor

@kaizencc kaizencc commented Oct 1, 2024

S3 Asset Garbage Collection

This PR introduces a new CLI command under the new --unstable flag. This flag ensures that users understand and opt-in to experimental or incomplete CLI features.

cdk gc will garbage collect unused assets in your bootstrapped S3 bucket. It goes through each object in the bucket, checks to see if the asset hash shows up in a cloudformation stack, and if not, tags the object as unused and/or deletes the object (depending on your configuration).

THIS COMMAND WILL DELETE OBJECTS IN YOUR BOOTSTRAPPED S3 BUCKET

basic garbage collection (immediately deletes objects that are unused):

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='s3'

garbage collection with a buffer (deletes unused objects > # of days specified):

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='s3' \
  --rollback-buffer-days=30

garbage collection with a created at buffer (deletes unused objects only if they have lived longer than this many days):

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='s3' \
  --created-buffer-days=5

garbage collect a specific bootstrap stack:

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='s3' \
  --bootstrap-stack-name=cdktest-0lc2i3vebi7-bootstrap-stack

before actually deleting your assets, you will be prompted one last time:

Found 1 objects to delete based off of the following criteria:
- objects have been isolated for > 0 days
- objects were created > 0 days ago

Delete this batch (yes/no/delete-all)?

To disable this, specify the --skip-delete-prompt option.

Todo in another PR

  • ECR asset collection

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team October 1, 2024 20:19
@github-actions github-actions bot added the p2 label Oct 1, 2024
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Oct 1, 2024
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Oct 1, 2024
mergify bot pushed a commit that referenced this pull request Oct 1, 2024
New protocol for `--unstable` is documented here and being used in #31611 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
packages/@aws-cdk-testing/cli-integ/lib/with-cdk-app.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/cdk-toolkit.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/garbage-collector.ts Outdated Show resolved Hide resolved
@kaizencc kaizencc self-assigned this Oct 2, 2024
@kaizencc kaizencc added the pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested label Oct 21, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review October 21, 2024 14:01

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

kaizencc and others added 3 commits October 21, 2024 10:10
Co-authored-by: Rico Hermans <rix0rrr@gmail.com>
Co-authored-by: Rico Hermans <rix0rrr@gmail.com>
@kaizencc kaizencc removed the pr/do-not-merge This PR should not be merged at this time. label Oct 21, 2024
Copy link
Contributor

mergify bot commented Oct 21, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 0c851ba
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 0a0e4ad into main Oct 21, 2024
11 of 12 checks passed
Copy link
Contributor

mergify bot commented Oct 21, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot deleted the conroy/gc branch October 21, 2024 19:22
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
contribution/core This is a PR that came from AWS. p2 pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested pr-linter/exempt-integ-test The PR linter will not require integ test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants