loggingBucketProps feature request

[arbzkhan]

https://github.com/awslabs/aws-solutions-constructs/blob/8ca5242530716c6401a3749ad2187f266e26cf4d/source/patterns/%40aws-solutions-constructs/core/lib/s3-bucket-helper.ts#LL64C1-L70C85

As per Code in this file(s3-bucket-helper.ts) from line 70, S3 bucket which has a lifecycle rule cannot be used on logging bucket according to code, because according to the code LifecycleConfiguration Rules are getting removed since because of this whatever LifecycleRules we are defining in the loggingBucketProps are getting removed.

I also replicated same scenario on my end. Replication steps:

1. I observed the behaviour in which, no CDK difference after updating the lifecycleRule to logging bucket.

2. In an attempt to resolve the issue, I tried to attach the lifecycle policy to the logging bucket explicitly using addLifecycleRule() method[1] as mentioned in the below code:

   Bucket appLogS3LogginBucket = appLog.getS3LoggingBucket();
    appLogS3LogginBucket.addLifecycleRule(loggingTransitionRule);
    appLogS3LogginBucket.addLifecycleRule(expiryRuleOnLoggingBucket);

Unfortunately, this too did not help me and I observed same behavior again. After analyzing the back-end code, I found that natively the logging bucket that's being created by 'aws-eventbridge-kinesisfirehose-s3' construct cannot have a lifecycle rule since in the line 70 of the repository LifecycleConfiguration Rules property is being removed from here.

3. To override this behavior, I used escape hatching concept to get an object of CfnBucket class and set the addOverride method as shown below:

CfnBucket cfnBucket = (CfnBucket)appLogS3LogginBucket.getNode().getDefaultChild();
cfnBucket.addOverride("Properties.LifecycleConfiguration.Rules.1.ExpirationInDays",50);

package com.myorg;
import software.amazon.awscdk.*;
import software.constructs.Construct;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.StackProps;
import software.amazon.awscdk.services.s3.*;
import software.amazon.awsconstructs.services.eventbridgekinesisfirehoses3.*;
import software.amazon.awscdk.services.events.*;
import java.util.*;  


public class s3bucketPolicy extends Stack {
    public s3bucketPolicy(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public s3bucketPolicy(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        // The code that defines your stack goes here

        int LOGGING_BUCKET_DATA_EXPIRY_IN_DAYS =30;
        String AWS_ACCOUNT = "xxxxxxxxxxxx";
        List serviceList = List.of("aws.ec2");
        Transition transitionData = Transition.builder()
    			.storageClass(StorageClass.INTELLIGENT_TIERING)
    			.transitionAfter(Duration.days(30))
    			.build();
    	
    	LifecycleRule TransitionOfDataRule = LifecycleRule.builder()
    			.enabled(true)
    			.id("TranstionToIntelTier")
    			.transitions(List.of(transitionData))
    			.build();
    	
    	LifecycleRule expiryRuleMainBucket = LifecycleRule.builder()
    			.enabled(true)
    			.id("MainBucketExpireObjects")
    			.expiration(Duration.days(60))
    			.build();

    	Transition transitionLogging = Transition.builder()
    			.storageClass(StorageClass.ONE_ZONE_INFREQUENT_ACCESS)
    			.transitionAfter(Duration.days(30))
    			.build();

    	LifecycleRule loggingTransitionRule = LifecycleRule.builder()
    			.enabled(true)
    			.id("TranstionToIntelTier")
    			.transitions(List.of(transitionLogging))
    			.build();

    	LifecycleRule expiryRuleOnLoggingBucket = LifecycleRule.builder()
    			.enabled(true)
    			.id("myname")
    			.expiration(Duration.days(LOGGING_BUCKET_DATA_EXPIRY_IN_DAYS))
    			.build();


         EventBus LogEventBus = EventBus.Builder.create(this, "bus")
         .eventBusName("MyEventBus")
         .build();
        


        EventbridgeToKinesisFirehoseToS3 appLog = new EventbridgeToKinesisFirehoseToS3(this, "appLog",
        new EventbridgeToKinesisFirehoseToS3Props.Builder()
        .existingEventBusInterface(LogEventBus)
        .bucketProps(BucketProps.builder()
                .bucketName("applog-data")
                .blockPublicAccess(BlockPublicAccess.BLOCK_ALL)
                .encryption(BucketEncryption.S3_MANAGED)     
                .autoDeleteObjects(true)
                .enforceSsl(true)
                .removalPolicy(RemovalPolicy.DESTROY)
                .lifecycleRules(List.of(TransitionOfDataRule, expiryRuleMainBucket))
                .build())
        .loggingBucketProps(BucketProps.builder()
                .bucketName("applog-logging")
                .blockPublicAccess(BlockPublicAccess.BLOCK_ALL)
                .encryption(BucketEncryption.S3_MANAGED)
                .autoDeleteObjects(true)
                .enforceSsl(true)
                .removalPolicy(RemovalPolicy.DESTROY)
                .versioned(false)
                // .lifecycleRules(List.of(loggingTransitionRule, expiryRuleOnLoggingBucket))
                .build())
        .eventRuleProps(new RuleProps.Builder()
                .description("Application Log to S3 via Kinesis Firehose")
                .ruleName("AppLogKinesisS3")
                .eventPattern(EventPattern.builder()
                        .account(Arrays.asList(AWS_ACCOUNT))
                        .detailType(List.of("ApplicationLogs"))
                        .source(serviceList)
                        //.detail(detail)
                        .build())
                .build())
        .build()); 

    
    // Bucket appLogS3Bucket = appLog.getS3Bucket();
    // appLogS3Bucket.addLifecycleRule(expiryRuleOnLoggingBucket);
    
    // .lifecycleRules(List.of(loggingTransitionRule, expiryRuleOnLoggingBucket))
    Bucket appLogS3LogginBucket = appLog.getS3LoggingBucket();
    appLogS3LogginBucket.addLifecycleRule(loggingTransitionRule);
    appLogS3LogginBucket.addLifecycleRule(expiryRuleOnLoggingBucket);

    CfnBucket cfnBucket = (CfnBucket)appLogS3LogginBucket.getNode().getDefaultChild();
    cfnBucket.addOverride("Properties.LifecycleConfiguration.Rules.1.ExpirationInDays",50);

    }
}

Expection:

As a feature request, Please consider lifecycle rules props for logging bucket as well.

[biffgaut]

Thanks for the input. It appears to us that the line of code removing lifecycle rules from the logging bucket is no longer appropriate (Perhaps it was never appropriate). We'll remove that line so that the lifecycle rules you specify in props will not be removed. We'll also expose existingLoggingBucket in the construct props for aws-eventbridge-kinesisfirehose-s3 so that clients can supply their own logging bucket. No ETA at this time - but it has a high priority for us (the timeframe won't be months).

[biffgaut]

Version 2.40.0 released today should address this issue.