Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda): propagate tags to log group #26049

Closed
wants to merge 87 commits into from
Closed
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e5236c3
made custom log retention resource taggable and added new function to…
colifran May 31, 2023
cd69181
added new function to handle setting tags on log group
colifran May 31, 2023
24a4a75
wip
colifran Jun 1, 2023
ffb4e64
rico comments
colifran Jun 1, 2023
cf55357
Merge branch 'main' into colifran/log-group-tags
colifran Jun 16, 2023
d7798b3
made log propagation conditional and updated actions in policy to ref…
colifran Jun 16, 2023
0bc3bf4
unit tests for log group propagation from tags set on lambda function
colifran Jun 16, 2023
4a0b380
updated comment about log retention to also include propagate logs
colifran Jun 19, 2023
7dd325c
updated log retention integ test and lambda handler for log retention
colifran Jun 19, 2023
89e191c
updated log group tags logic
colifran Jun 19, 2023
e57178e
updated integ test and lambda handler logic to now set tags on log group
colifran Jun 19, 2023
1c7942c
updated lambda handler log propagation logic to only set tags on log …
colifran Jun 19, 2023
7d81dbe
added logic to remove tags from log group to lambda handler
colifran Jun 19, 2023
b9a33b6
updated logic for deleting tags
colifran Jun 19, 2023
72e68b5
updated lambda log-retention and aws-logs log-retention integ tests
colifran Jun 19, 2023
94115b5
added a check to only untag log group if tagsToDelete has a length gr…
colifran Jun 19, 2023
d94a372
lambda log-retention integ test and snapshots
colifran Jun 19, 2023
c8c74e1
updated lambda log retention integ test to check for lambda custom re…
colifran Jun 19, 2023
f965942
updated adding permissions for tagging and untagging log groups and u…
colifran Jun 19, 2023
eded477
added ability to have a policy statement to grant propagate tags to s…
colifran Jun 19, 2023
1d9072d
updated integ tests for log-retention for aws-lambda and aws-logs
colifran Jun 19, 2023
b9da1ee
updated log-retention integ test and snapshot for aws-logs
colifran Jun 19, 2023
59433e2
added clarifying comment to integ test
colifran Jun 19, 2023
021b3ce
added unit tests to test multiple log retention with log propagation
colifran Jun 19, 2023
b7e175e
aws-lambda log-retention integ test and snapshots
colifran Jun 20, 2023
8344040
updated aws-lambda log-retention integ test and snapshots
colifran Jun 20, 2023
d0063f9
Merge branch 'main' into colifran/log-group-tags
colifran Jun 20, 2023
9bfdcbb
readded single quotes to template properties
colifran Jun 20, 2023
c6f1ea5
updated README
colifran Jun 20, 2023
31810a5
Merge branch 'main' into colifran/log-group-tags
colifran Jun 20, 2023
10030f2
Merge branch 'main' into colifran/log-group-tags
colifran Jun 20, 2023
4f4314e
Merge branch 'main' into colifran/log-group-tags
colifran Jun 20, 2023
9a9c18b
added unit tests to log retention provider to verify log group creati…
colifran Jun 20, 2023
a08d8d1
removed unused import
colifran Jun 20, 2023
7d3594b
updated setLogGroupTags handler logic and added log retention provide…
colifran Jun 20, 2023
c438f5d
unit test for tag update and deletion
colifran Jun 20, 2023
32fd3d3
updated log-retention integ test and snapshots for aws-lambda to refl…
colifran Jun 20, 2023
a13d5b4
updated log retention integ and snapshots for aws-logs to reflect upd…
colifran Jun 20, 2023
632c3c9
Merge branch 'main' into colifran/log-group-tags
colifran Jun 21, 2023
5fc572b
updated logic to still execute setLogGroupTags when propagateTagsToLo…
colifran Jun 21, 2023
c4e52af
updated lambda log-retention integ test
colifran Jun 21, 2023
569231c
updated aws-lambda log-retention integ test snapshots
colifran Jun 21, 2023
1860953
updated aws-logs log-retention integ test
colifran Jun 21, 2023
a222af3
removed stale assets from aws-logs log-retention snapshots
colifran Jun 21, 2023
58ca4d2
updated handler logic to check if PropagateTags is 'true' instead of …
colifran Jun 21, 2023
0013ee4
updated aws-logs log-retention integ test and snapshots
colifran Jun 21, 2023
cbf7f86
updated aws-lambda log-retention integ test and snapshots
colifran Jun 21, 2023
0b914fa
updated README
colifran Jun 21, 2023
4b05f13
updated README
colifran Jun 21, 2023
17d4b9e
empty commit to rerun build
colifran Jun 21, 2023
5f0d7c8
updated tag propagation unit tests in log retention provider with Pro…
colifran Jun 21, 2023
850b7a0
Merge branch 'main' into colifran/log-group-tags
colifran Jun 24, 2023
b756e8a
Merge branch 'main' into colifran/log-group-tags
colifran Jul 5, 2023
b205e64
Merge branch 'main' into colifran/log-group-tags
colifran Jul 6, 2023
f75fa5d
Merge branch 'main' into colifran/log-group-tags
colifran Jul 11, 2023
142da43
Merge branch 'main' into colifran/log-group-tags
colifran Jul 12, 2023
da5bf94
updated README for aws-lambda updated == to === in handler helper fun…
colifran Jun 24, 2023
7bea21d
updated logic to pass log group ARN as a property in the event passed…
colifran Jun 24, 2023
492e73c
updated setLogGroupTags to use listTagsForResource, tagResource, and …
colifran Jun 24, 2023
6593335
updated log retention provider unit tests for tag propagation with ne…
colifran Jun 24, 2023
4320593
updated unit tests in log-retention-provider based on updates to clou…
colifran Jun 24, 2023
57136f1
removed unneeded comment in handler
colifran Jun 24, 2023
a7d4595
added comments to code
colifran Jun 26, 2023
8ea4092
added check for tagsToSet to have at most 50 key-value pairs before c…
colifran Jun 26, 2023
2128024
updated logGroupArn that is passed as event to resource provider lamb…
colifran Jul 5, 2023
509896a
added excludeResourceTypes to integ test for log retention
colifran Jul 5, 2023
9c77ac6
updated actions to be TagResource, UntagResource, and ListTagsForReso…
colifran Jul 5, 2023
ec0d9e4
refactored code used to for granting permissions and creating log res…
colifran Jul 5, 2023
760f615
added clarifying comment to event object property
colifran Jul 5, 2023
abc7b82
updated log-retention integ test and snapshots
colifran Jul 5, 2023
2979b30
updated README with an example of excluding a resource type to preven…
colifran Jul 5, 2023
26c7e4d
updated unit tests with updated base arn format and updated tagging a…
colifran Jul 5, 2023
fa4ea99
removed unclear comment
colifran Jul 5, 2023
319618a
updated log-retention integ test to add assertions
colifran Jul 5, 2023
d4b5b97
updated log-retention integ test
colifran Jul 6, 2023
87ab519
moved multiple log retention tag propagation unit tests under the mul…
colifran Jul 6, 2023
fbd3afa
removed awsApiCall
colifran Jul 11, 2023
ce4fc50
Merge branch 'main' into colifran/log-group-tags
mrgrain Jul 20, 2023
1b5fa57
Merge branch 'main' into colifran/log-group-tags
colifran Jul 22, 2023
595259f
made tag propagation props on event conditional
colifran Jul 22, 2023
b963f0c
added assertion to aws-lambda log-retention integ test
colifran Jul 22, 2023
2013faf
snapshots for aws-lambda log-retention integ test
colifran Jul 22, 2023
d5fc13d
assertions for aws-logs log-retention integ test and snapshots
colifran Jul 23, 2023
1bec5ad
readme update
colifran Jul 23, 2023
7ff5640
Merge branch 'main' into colifran/log-group-tags
colifran Jul 26, 2023
aa30d6c
Merge branch 'main' into colifran/log-group-tags
colifran Jul 28, 2023
23a30d1
Merge branch 'main' into colifran/log-group-tags
mergify[bot] Aug 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,64 @@ async function setRetentionPolicy(logGroupName: string, region?: string, options
} while (true); // exit happens on retry count check
}

/**
* Tags and untags a log group. This includes adding new tags and updating existing tags.
*
* @param logGroupName the name of the log group to create
* @param tags the tags to propagate to the log group
* @param region the region of the log group
* @param options CloudWatch API SDK options
*/
async function setLogGroupTags(logGroupName: string, tags: AWS.CloudWatchLogs.Tags[], region?: string, options?: SdkRetryOptions) {
// The same as in createLogGroupSafe(), here we could end up with the race
// condition where a log group is either already being created or its retention
// policy is being updated. This would result in an OperationAbortedException,
// which we will try to catch and retry the command a number of times before failing
let retryCount = options?.maxRetries == undefined ? 10 : options.maxRetries;
const delay = options?.retryOptions?.base == undefined ? 10 : options.retryOptions.base;
do {
try {
const cloudwatchlogs = new AWS.CloudWatchLogs({ apiVersion: '2014-03-28', region, ...options });
const tagsOnLogGroup = (await cloudwatchlogs.listTagsLogGroup({ logGroupName }).promise()).tags ?? {};

const tagsToSet: { [key: string]: string } = {};
const tagsKeys: string[] = [];
for (const tag of tags) {
if (tagsOnLogGroup[tag.Key] === undefined || tagsOnLogGroup[tag.Key] !== tag.Value) {
tagsToSet[tag.Key] = tag.Value;
}
tagsKeys.push(tag.Key);
}

const tagsToDelete = tagsOnLogGroup
? Object.keys(tagsOnLogGroup).filter(tag => !tagsKeys.includes(tag))
: [];

if (Object.keys(tagsToSet).length > 0) {
await cloudwatchlogs.tagLogGroup({ logGroupName, tags: tagsToSet }).promise();
}

if (tagsToDelete.length > 0) {
await cloudwatchlogs.untagLogGroup({ logGroupName, tags: tagsToDelete }).promise();
}

return;
} catch (error: any) {
if (error.code === 'OperationAbortedException') {
if (retryCount > 0) {
retryCount--;
await new Promise(resolve => setTimeout(resolve, delay));
continue;
} else {
// The log group is still being created by another execution but we are out of retries
throw new Error('Out of attempts to create a logGroup');
}
}
throw error;
}
} while (true);
}

export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent, context: AWSLambda.Context) {
try {
console.log(JSON.stringify({ ...event, ResponseURL: '...' }));
Expand All @@ -138,7 +196,11 @@ export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent
// Act on the target log group
await createLogGroupSafe(logGroupName, logGroupRegion, retryOptions);
await setRetentionPolicy(logGroupName, logGroupRegion, retryOptions, parseInt(event.ResourceProperties.RetentionInDays, 10));
if (event.ResourceProperties.PropagateTags === 'true') {
await setLogGroupTags(logGroupName, event.ResourceProperties.Tags ?? [], logGroupRegion, retryOptions);
}

// propagate tags to custom resource logs
if (event.RequestType === 'Create') {
// Set a retention policy of 1 day on the logs of this very function.
// Due to the async nature of the log group creation, the log group for this function might
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
{
"version": "31.0.0",
"version": "32.0.0",
"files": {
"0f8a80f5f2310ea7e3295258fdbb79c43a1e8c0a11e863e63dc82832599412ed": {
"03f4865a5a0af55ce1833e693c97d42df8d33908ddbf95f3c01cb19aa6cf3602": {
"source": {
"path": "asset.0f8a80f5f2310ea7e3295258fdbb79c43a1e8c0a11e863e63dc82832599412ed",
"path": "asset.03f4865a5a0af55ce1833e693c97d42df8d33908ddbf95f3c01cb19aa6cf3602",
"packaging": "zip"
},
"destinations": {
"current_account-current_region": {
"bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}",
"objectKey": "0f8a80f5f2310ea7e3295258fdbb79c43a1e8c0a11e863e63dc82832599412ed.zip",
"objectKey": "03f4865a5a0af55ce1833e693c97d42df8d33908ddbf95f3c01cb19aa6cf3602.zip",
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-${AWS::Region}"
}
}
},
"53d9f93777a59c056d47bcb74a136aae7c53c3e010fe62d984859dd0aa334457": {
"5837b4ebbd132ab8d64342d61a9901b7025321ebe297ec312c313e1c64161039": {
"source": {
"path": "aws-cdk-lambda-log-retention.template.json",
"packaging": "file"
},
"destinations": {
"current_account-current_region": {
"bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}",
"objectKey": "53d9f93777a59c056d47bcb74a136aae7c53c3e010fe62d984859dd0aa334457.json",
"objectKey": "5837b4ebbd132ab8d64342d61a9901b7025321ebe297ec312c313e1c64161039.json",
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-${AWS::Region}"
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,64 @@
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"logs:ListTagsLogGroup",
"logs:TagLogGroup",
"logs:UntagLogGroup"
],
"Effect": "Allow",
"Resource": [
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":log-group:/aws/lambda/",
{
"Ref": "OneMonth64E966BF"
},
":*"
]
]
},
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":log-group:/aws/lambda/",
{
"Ref": "OneYearA82EBDA9"
},
":*"
]
]
}
]
}
],
"Version": "2012-10-17"
Expand Down Expand Up @@ -145,7 +203,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "0f8a80f5f2310ea7e3295258fdbb79c43a1e8c0a11e863e63dc82832599412ed.zip"
"S3Key": "03f4865a5a0af55ce1833e693c97d42df8d33908ddbf95f3c01cb19aa6cf3602.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down Expand Up @@ -187,6 +245,16 @@
]
]
}
],
"Tags": [
{
"Key": "dept",
"Value": "sales"
},
{
"Key": "env",
"Value": "prod"
}
]
}
},
Expand All @@ -202,8 +270,19 @@
"Arn"
]
},
"FunctionName": "OneMonthFunction",
"Handler": "index.handler",
"Runtime": "nodejs14.x"
"Runtime": "nodejs14.x",
"Tags": [
{
"Key": "dept",
"Value": "sales"
},
{
"Key": "env",
"Value": "prod"
}
]
},
"DependsOn": [
"OneMonthServiceRoleFBD1064F"
Expand All @@ -229,7 +308,18 @@
]
]
},
"RetentionInDays": 30
"RetentionInDays": 30,
"PropagateTags": true,
"Tags": [
{
"Key": "dept",
"Value": "sales"
},
{
"Key": "env",
"Value": "prod"
}
]
}
},
"OneYearServiceRole24D47762": {
Expand Down Expand Up @@ -260,6 +350,12 @@
]
]
}
],
"Tags": [
{
"Key": "dept",
"Value": "eng"
}
]
}
},
Expand All @@ -275,8 +371,15 @@
"Arn"
]
},
"FunctionName": "OneYearFunction",
"Handler": "index.handler",
"Runtime": "nodejs14.x"
"Runtime": "nodejs14.x",
"Tags": [
{
"Key": "dept",
"Value": "eng"
}
]
},
"DependsOn": [
"OneYearServiceRole24D47762"
Expand All @@ -302,7 +405,14 @@
]
]
},
"RetentionInDays": 365
"RetentionInDays": 365,
"PropagateTags": true,
"Tags": [
{
"Key": "dept",
"Value": "eng"
}
]
}
}
},
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"version":"31.0.0"}
{"version":"32.0.0"}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "31.0.0",
"version": "32.0.0",
"testCases": {
"integ.log-retention": {
"stacks": [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "31.0.0",
"version": "32.0.0",
"artifacts": {
"aws-cdk-lambda-log-retention.assets": {
"type": "cdk:asset-manifest",
Expand All @@ -17,7 +17,7 @@
"validateOnSynth": false,
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-deploy-role-${AWS::AccountId}-${AWS::Region}",
"cloudFormationExecutionRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-cfn-exec-role-${AWS::AccountId}-${AWS::Region}",
"stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/53d9f93777a59c056d47bcb74a136aae7c53c3e010fe62d984859dd0aa334457.json",
"stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/5837b4ebbd132ab8d64342d61a9901b7025321ebe297ec312c313e1c64161039.json",
"requiresBootstrapStackVersion": 6,
"bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version",
"additionalDependencies": [
Expand Down
Loading