Add a way to write properties from resources to a file in S3

[plumdog]

🚀 feature request

Some way of writing some of the values from resources to a file in S3.

Eg, something like:

const myBucket = new s3.Bucket(this, 'MyBucket');
new s3deploy.BucketDeployment(this, 'MyConfigFile', {
    source: s3deploy.Source.fromString(`{"thisBucketArn": "${myBucket.bucketArn}"}`),
    destinationBucket: myBucket,
    destinationKeyPrefix: 'config.json',
});

should deploy the bucket, then write the ARN of the bucket to the json file in the bucket at config.json.

I suspect there's a way to do this already by abusing assets sufficiently, but I haven't been able to make it work. The closest I can get is to write the variables into the source for a lambda and access them using API gateway, but that seems silly.

[plumdog]

Some more digging suggests to me that this might not be possible at present by hacking at assets, as assets are copied to a staging directory (session.assembly.outdir) at synthesize. Is there perhaps instead a way of handling this using Fn.sub?

[fimbulvetr]

This should help you get some of the way. It doesn't seem to support dynamically writing content, but it does support writing a file you have locally to an s3 bucket of your choice.

https://github.com/awslabs/aws-cdk/blob/master/packages/%40aws-cdk/aws-s3-deployment/README.md

If it were me, I would write the contents to the "Metadata" of a resource, which would give you the feature of allowing the metadata to be Fn::Sub'd (and therefore ${myBucket.bucketArn} would be evaluated). Then I'd have a lambda read from that resource's metadata and put it into an S3 bucket. This is, from a high level, exactly how cfn-init works - it reads the Fn::Sub'd text from a resource's Metadata.

[plumdog]

I did manage to handle this using the rather marvellous AwsCustomResource in @aws-cdk/custom-resources:

const config = {
  myBucketArn = myBucket.bucketArn,
};
new customResources.AwsCustomResource(this, 'WriteS3ConfigFile', {
  onUpdate: {
    service: 'S3',
    action: 'putObject',
    parameters: {
      Body: JSON.stringify(config),
      Bucket: myBucket.bucketName,
      Key: 'config.json',
    },
    physicalResourceId: Date.now().toString(),  // always write this file
  },
  // we need this because we're not doing conventional resource creation
  policyStatements: [
    new iam.PolicyStatement({
       actions: ['s3:PutObject'],
       resources: [`${myBucket.bucketArn}/config.json`],
    }),
  ],
});

Whether the right thing to do is:
a) bundling this up a reusable construct
b) writing something bespoke (I don't know enough about the internal to know what that would be)
c) documenting this a workaround
d) not bothering with either

I'm not sure. I suppose if this is functionality that lots of people might require, then a or b. If only a few, then c. If very very few (or just me), then d.

[rix0rrr]

It's not S3, so I don't know how hard your requirement is on that, but the SSM Parameter Store is intended exactly for storing this kind of configuration data.

[djorg83]

I created a class that allows you to provide a template for a JSON config file that you would like to be written to S3, for use with static website deployments.

How it works...

1. You provide it a template that can include placeholders for SSM parameters.

{
  apiUrl: '{{ssm:yourParameterName}}'
}

2. You provide it the source for the lambda function that produces the JSON config. You can find a working example here

3. It creates the CDK constructs for the lambda function, role, and policy. Grants permission to access the required ssm parameters. Grants permission to write the JSON file to S3. Grants permission to create a new invalidation in CloudFront. Then creates a custom resource to orchestrate the deployment.

4. When the stack is deployed, the lambda function is invoked. The SSM parameters are retrieved and substituted into the provided template. The resulting JSON is written to S3. A new invalidation for the created file is made to CloudFront.

Basic Example

new StaticWebsiteAppConfig(
  this,
  'UiAppConfig',
  {
    accountId,
    stage,
    region,
    providerFunctionCodePath: join(__dirname, '/lambda/app-config-producer'),
    bucketName: yourStaticWebsiteBucket.bucketName,
    distributionId: yourCloudFrontDistribution.distributionId,
    configTemplate: {
      apiUrl: '{{ssm:yourParameterName}}'
    },
  }
);

Full working example and source code for lambda function

https://gist.github.com/djorg83/11fc8cba5c37c9c7d89b108f9e42d2db

[iliapolo]

@djorg83 The StaticWebsiteAppConfig construct looks cool. I'm having second thoughts though about the place of such functionality in the core modules. The only module I feel might include this is @aws-cdk/aws-cloudfront, but even then, it seems a little too specific and involves a lot of moving parts.

We are currently actively working on redesigning (which we would love feedback on) the CloudFront module. As part of that work, we are considering vending an L3 library that makes it super easy to implement the StaticWebsite use-case. I think this construct will fit nicely in that library.

Until then, I suggest you publish this as a third-party reusable library. You already have the code ready, and it should be fairly easy to setup the publishing scaffolding, see Publishing Modules.

I'll also be happy to offer a review on that code and help in any way I can.

Thoughts?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[iliapolo]

Don't close

[SomayaB]

Closing this for now as there doesn't seem to be any more actionable steps. Here's a link to the Cloudfront tracking issue and redesign RFC. Feel free to reopen.

[rix0rrr]

Reopening. I've just ran into the need for this as well, and I actually feel that the @aws-cdk/aws-s3-deployment module would be a good location for it.

[jagregory]

+1 I think the s3 deployment would be a good place for this too. I'm uploading a bunch of files with a deployment, but I'd like one of them to be templated with information from the deployment (think environment name, S3 bucket URL etc).

[eladb]

I've been experimenting with something like this. Will report back in the coming weeks with some ideas.

[fjmacagno]

I have a similar use case, local directory of template files i want to substitute configs into and upload to S3.

There is also this ticket, but it is for a subtly different thing: #12903

[moltar]

I like the idea of s3deploy.Source.fromString( pattern, and that is what I was expecting to find in the intellisense.

And I think this sort-a exists already, but it's not a published package, just an example:

https://docs.aws.amazon.com/cdk/api/latest/docs/custom-resources-readme.html#provider-framework-examples