ecr: Garbage collection for ECR prints runs indefinitely #32742
Comments
BwL1289
added
bug
github-actions
bot
added
the
@aws-cdk/assets
khushail
added
investigating
|
Hi @BwL1289 , thanks for reaching out. Looks like a similar issue with garbage collector was reported few days ago and fix was also put in place last week- #32679 |
khushail
added
response-requested
|
Hi @khushail. I am using v2.174.0 which is the latest version. |
khushail
added
investigating
|
@BwL1289 , thanks for the confirmation. I tried to repro the scenario in my account with the latest version and here is what I am getting - 
Images are being tagged but not deleted. Trying to reproduce it in other region. |
|
@khushail I am seeing the same behavior, in addition to what I described originally. |
pahud
changed the title
|
This is strange @BwL1289, sorry that you are experiencing this. If this is a nextToken issue as we thought before, I believe a repro with 1000+ images in the ecr repo would help. However, before I get to that tomorrow, I just want to make sure -- you are on CDK CLI version 2.174.0, not just aws-cdk-lib 2.174.0 right? |
khushail
removed
the
investigating
|
@kaizencc all good. Appreciate the support. Yes, on CDK CLI version
|
|
@kaizencc also just to clarify, I don't believe it's only a nextToken issue (I experienced that before the latest fix). In the previous version in the verbose output I could see the same image ids being iterated over. Now I'm seeing images also get tagged but not deleted. What's also interesting is the verbose output stops after a period of time running. |
|
@kaizencc were you able to reproduce? |
|
@sakurai-ryo out of curiosity does #32985 resolve this? Thank you. |
|
Hi @BwL1289 |
|
Thanks @sakurai-ryo. Will do. |
…ction (#32985) ### Issue # (if applicable) Related to #32742 ### Reason for this change In the `garbageCollectS3` and `garbageCollectEcr` functions, a new `setInterval` is created in the `ProgressPrinter` class each time the for-await loop runs. As a result, the old `setInterval` can no longer be referenced via `this.setInterval`, and because it cannot be cleared, the messages continue to be displayed. https://github.com/aws/aws-cdk/blob/899965d6147829b8f8ac52ac8c1350de50d7b6d0/packages/aws-cdk/lib/api/garbage-collection/progress-printer.ts#L44 This behavior is the same as what occurs in the simple sample code below. ```ts const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) class Printer { private setInterval?: ReturnType<typeof setTimeout> public start() { this.setInterval = setInterval(() => { console.log("Hello, world!") }, 1000) } public stop() { clearInterval(this.setInterval) } } ;(async () => { const printer = new Printer() // print one message per second for 5 seconds printer.start() await sleep(5000) // print tow messages per second for 5 seconds printer.start() await sleep(5000) // clear the newer setInterval // the older one will still be running printer.stop() })() ``` ### Description of changes It didn't seem necessary to initialize `setInterval` each time the for-await loop runs, so I modified the code to call `.start()` before the loop begins. ### Describe any new or updated permissions being added Nothing. ### Description of how you validated changes ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
Confirm the newest fix works as expected. |
|
Comments on closed issues and PRs are hard for our team to see. |
Describe the bug
Seeing unexpected behavior, specifically in ECR.
In one account I see the following:
This continues seemingly indefinitely. It's also strange that the verbose flag stops working after some period of time.
In another account I see the following:
This also continues seemingly indefinitely. Same with verbose flag behavior not working as expected after some period of time.
I've left it running for 6 hours and still the same.
Regression Issue
Last Known Working CDK Version
No response
Expected Behavior
Delete all unused ecr and s3 cdk assets.
Current Behavior
Ecr and s3 assets that are unused are not being deleted and runs indefinitely.
Reproduction Steps
Run
cdk gc --unstable='gc' --type='all' -vvvv --rollback-buffer-days=1Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.174.0
Framework Version
No response
Node.js Version
22.12.0
OS
MacOS
Language
Python
Language Version
No response
Other information
No response
The text was updated successfully, but these errors were encountered: