Description
TL;DR:
Do you constantly find yourself running cdk destroy and finding that you're left with 100s of LogGroups orphaned? Fed up of polluting your CloudWatch with the LogGroups of Lambdas long dead? I may have the solution for you!
Create your LogGroups named as they would be by CloudWatch! This will override the out-of-the-box ones and you can set retention & removal_policy!
An Example
I've found in the past that creating and destroying multiple stacks in a day, for multiple days, for multiple weeks results in 100s if not 1000s of orphaned LogGroups. This is annoying as it's noisy and you spend longer looking for the one LogGroup you need.
After some discussions with @JakeHendy, it was noted that creating a LogGroup explicitly, named the same as it would be implicitly, would allow us to:
- Set a custom retention policy for logs
- Set a custom removal policy for LogGroups
You can see an example below:
from aws_cdk import (
aws_lambda,
aws_lambda_python,
aws_logs,
core,
)
lambda_function = aws_lambda_python.PythonFunction(
self,
entry="function_entry",
handler="handler",
runtime=aws_lambda.Runtime.PYTHON_3_8,
memory_size=128,
timeout=core.Duration.minutes(5),
)
aws_logs.LogGroup(
self,
log_group_name=f"/aws/lambda/{lambda_function.function_name}",
removal_policy=core.RemovalPolicy.DESTROY,
retention=aws_logs.RetentionDays.ONE_DAY,
)This will then result in you being able to do cdk deploy and cdk destroy and no LogGroups being available under /aws/lambda/your_function_name* 🎉
A Gotcha
One issue I am still currently struggling with (and am losing hope on) is finding an easy way to not get orphaned LogGroups for CustomResource Lambda functions.
I have implemented the same pattern for explicit LogGroup creation for a Lambda-backed CustomResource and I always end up with an orphaned LogGroup... taunting me. 🤨 👎
I believe that this is due to how CloudFormation handles the deletion of the CustomResource, it probably looks like:
cdk destroy- Explicit LogGroup destroyed
- CustomResource invoked with
destroy- Lambda invoked, which creates a new (out of our domain) LogGroup
- CustomResource destroyed
- Lambda destroyed
cdk destroyends- Orphaned LogGroup taunts me 🧟♂️
I've posted in https://cdk.dev/ and as of yet, have no real luck. I've tried bouncing around with dependencies similar to how @alukach is doing it in the ss-orders project (Making a Construct explicitly depend on another) but that's not been too successful 😭
Jake suggested creating a Lambda that listens for DeleteStack in CloudTrail then deletes all LogGroups, but that's a Lambda for the sake of handling potentially one orphaned LogGroup, so in terms of effort vs. return, it's not a winner.
Happy for folks to jump in and give suggestions on how we could cover this gotcha off!