Deploy fails due to log groups not being torn down

[patchwork01]

If you run a system test with buildDeployTest.sh, tear it down, then buildDeployTest.sh again, it fails to deploy due to log groups not being deleted during the tear down.

[m09526]

This one has got me too several times...really annoying.

[patchwork01]

One approach is to use a different instance ID each time, in case it's valuable to keep the log groups around. We should probably at least automate deleting the log groups as an optional part of the tear down scripts.

[patchwork01]

We'd probably want to be able to delete an instance of Sleeper, then deploy a new one with the same instance ID. That seems like something that should work.

We could add some random or date/time-based identifier to the log group name, but that would make it harder to read. That might also mean that each time you do cdk apply, it would generate a new log group name and detach the instance from the old one (assuming that will work).

If we optionally delete the log groups during tear down, that doesn't solve the general problem. You probably really want to retain the logs anyway.

[patchwork01]

There doesn't seem to be a good solution for now.

[patchwork01]

The tear down scripts already delete things that were configured to be retained. If that's what they do they could do that for log groups as well.

[patchwork01]

It doesn't look like there's a good way to find all the log groups that need to be torn down. They don't seem to be listed as resources in the CloudFormation stack, and tags set on the stack aren't propagated to the log groups.

We might be able to examine each Lambda to find the log group associated with it, and something similar for ECS. That would be a pretty complicated work-around though.

There's also this issue where when you flag an S3 bucket to auto-delete its objects when it's torn down, the Lambda function that does that gets a log group with a retention policy of never expire:

aws/aws-cdk#24815

There's also an option of automatically deleting all log groups that aren't associated with a resource deployed via CloudFormation, as seen in this tool:

https://gist.github.com/seeebiii/c91815200915a17131c9d908c525c357

That will only work in cases where all your Lambdas are deployed via CloudFormation, so we can't build it into Sleeper. We could automate something like that as part of a system test that would run regularly.