Statically typed CloudFormation Templates #401

rosenhouse · 2015-10-12T00:25:49Z

A draft, and not feature-complete. Feedback appreciated.

Context from my original issue #393:

I'm trying to programmatically generate CloudFormation templates.

cloudformation.CreateStackInput has its TemplateBody field typed as a *string. I'd like to be able to build up a strongly-typed Template in memory, call json.Marshal() and use the result as the TemplateBody.

I've roughly followed the approach outlined by @jasdel . One difference is that the intrinsic Fn::x and Ref functions are modeled as Go functions, not types. It makes the usage syntax a bit cleaner:

FnJoin("\n", "Stack ID is:", Ref("AWS::StackId") )

instead of

FnJoin{ "\n", []interface{}{ "Stack ID is:", Ref{ "AWS::StackId" } } }

But perhaps there are some drawbacks I haven't considered. And regardless, you'll probably want to dot-import this package.

What's missing:

Types for all the different resources
Types for all the different resource properties
Resource attributes including DependsOn which shows up a lot with VPCs

It would be great to auto-generate the resource and resource-property types somehow.

But I wanted to get feedback before I invested too much more time.

jasdel · 2015-10-19T19:38:22Z

Hi @rosenhouse thanks a lot for taking the time and putting this preliminary PR. I'm looking for predefined models of the template shapes to see if there is something we can use to generate code from. In the mean time I'll review the PR and get back to you with feedback.

Using helper functions instead of structs for Fn::x should work just as well too.

rosenhouse · 2015-11-05T04:21:48Z

@jasdel: Any update here? Would you be open to merging this PR as it currently stands? We could follow up later on with the generated resource & resource-property structs.

jasdel · 2015-11-05T20:57:11Z

service/cloudformation/template/template_test.go

+	assertEquivalentJSON(t, expectedJSON, string(actual))
+}
+
+func assertEquivalentJSON(t *testing.T, expected, actual string) {


Can assertEquivalentJSON be replaced with awstesting.AssertJSON? , they look similar.

jasdel · 2015-11-05T21:08:35Z

Thanks a lot for your hard work getting this PR together @rosenhouse. It looks good and I think we can pull it in.

I agree being able to generate the code would be the best plan, especially for the resources and property resources. The .NET SDK uses a JSON scheme for their Visual Studio's CloudFormation Template generator wizard. I'm investigating if it is possible for the Go SDK to reuse the scheme for our resource and resource-property structs.

- first pass, no tests or docs - aws#393

- with unit tests - backfill test of template structure

rosenhouse · 2015-11-05T21:42:59Z

Thanks for the review. I've rebased against master, replaced my assertEquivalentJSON with awstesting.AssertJSON and added a package doc header.

Add Statically typed CloudFormation Templates

jasdel · 2015-11-05T21:54:34Z

Looks great, thanks, and I merged the change in. This will be a great place to get started building templates programmatically. I'll update #393 when more information is available about auto generating the resources and resource-properties.

rosenhouse · 2015-11-05T23:17:44Z

Great, thank you!

References: #401, #425

…for Empty Collections (aws#401)

### Services * Synced the V2 SDK with latest AWS service API definitions. ### SDK Breaking changes * This update includes breaking changes to how the DynamoDB AttributeValue (un)marshier handles empty collections. ### Deprecations * `service/s3/s3crypto`: Deprecates the crypto client from the SDK ([aws#394](aws/aws-sdk-go-v2#394)) * s3crypto client is now deprecated and may be removed from the future versions of the SDK. * `aws`: Removes plugin credential provider ([aws#391](aws/aws-sdk-go-v2#391)) * Removing plugin credential provider from the v2 SDK developer preview. This feature may be made available as a separate module. * Removes support for deprecated Go versions ([aws#393](aws/aws-sdk-go-v2#393)) * Removes support for Go version specific files from the SDK. Also removes irrelevant build tags, and updates the README.md file. * Raises the minimum supported version to Go 1.11 for the SDK. Older versions may work, but are not actively supported ### SDK Features * `service/s3/s3manager`: Add Upload Buffer Provider ([aws#404](aws/aws-sdk-go-v2#404)) * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory. * Windows platforms will now default to buffering 1MB per part to reduce contention when uploading files. * Non-Windows platforms will continue to employ a non-buffering behavior. * `service/s3/s3manager`: Add Download Buffer Provider ([aws#404](aws/aws-sdk-go-v2#404)) * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory when copying from the http response body. * Windows platforms will now default to buffering 1MB per part to reduce contention when downloading files. * Non-Windows platforms will continue to employ a non-buffering behavior. * `service/dynamodb/dynamodbattribute`: New Encoder and Decoder Behavior for Empty Collections ([aws#401](aws/aws-sdk-go-v2#401)) * The `Encoder` and `Decoder` types have been enhanced to support the marshaling of empty structures, maps, and slices to and from their respective DynamoDB AttributeValues. * This change incorporates the behavior changes introduced via a marshal option in V1 ([aws#2834](aws#2834)) ### SDK Enhancements * `internal/awsutil`: Add suppressing logging sensitive API parameters ([aws#398](aws/aws-sdk-go-v2#398)) * Adds suppressing logging sensitive API parameters marked with the `sensitive` trait. This prevents the API type's `String` method returning a string representation of the API type with sensitive fields printed such as keys and passwords. * Related to [aws#2310](aws#2310) * Fixes [aws#251](aws/aws-sdk-go-v2#251) * `aws/request` : Retryer is now a named field on Request. ([aws#393](aws/aws-sdk-go-v2#393)) * `service/s3/s3manager`: Adds `sync.Pool` to allow reuse of part buffers for streaming payloads ([aws#404](aws/aws-sdk-go-v2#404)) * Fixes [aws#402](aws/aws-sdk-go-v2#402) * Uses the new behavior introduced in V1 [aws#2863](aws#2863) which allows the reuse of the sync.Pool across multiple Upload request that match part sizes. ### SDK Bugs * `service/s3/s3manager`: Fix index out of range when a streaming reader returns -1 ([aws#378](aws/aws-sdk-go-v2#378)) * Fixes the S3 Upload Manager's handling of an unbounded streaming reader that returns negative bytes read. * `internal/ini`: Fix ini parser to handle empty values [aws#406](aws/aws-sdk-go-v2#406) * Fixes incorrect modifications to the previous token value of the skipper. Adds checks for cases where a skipped statement should be marked as complete and not be ignored. * Adds tests for nested and empty field value parsing, along with tests suggested in [aws#2801](aws#2801)

jasdel added help wanted We are asking the community to submit a PR to resolve this issue. review needed labels Oct 19, 2015

jasdel reviewed Nov 5, 2015
View reviewed changes

jasdel removed the review needed label Nov 5, 2015

rosenhouse and others added 8 commits November 5, 2015 13:25

Strongly-typed CloudFormation templates

bf9430e

- first pass, no tests or docs - aws#393

Add Ref, FnBase64 and FnJoin

89bd234

- with unit tests - backfill test of template structure

Add FnFindInMap

a457eb9

Adds FnGetAtt, FnGetAZs, FnSelect CloudFormation functions

0d19d64

Split CloudFormation functions out into separate file

534dc6a

Add doc strings to CloudFormation template types and intrinsic functions

8920a96

Use awstesting.AssertJSON

6d143ba

Adds package doc header

af6c136

rosenhouse force-pushed the cloudformation-templates branch from c34316c to af6c136 Compare November 5, 2015 21:35

jasdel added a commit that referenced this pull request Nov 5, 2015

Merge pull request #401 from rosenhouse/cloudformation-templates

87173d6

Add Statically typed CloudFormation Templates

jasdel merged commit 87173d6 into aws:master Nov 5, 2015

rosenhouse deleted the cloudformation-templates branch November 5, 2015 22:44

jasdel added a commit that referenced this pull request Nov 10, 2015

Tag release v0.10.3

0d69769

References: #401, #425

skotambkar pushed a commit to skotambkar/aws-sdk-go that referenced this pull request May 20, 2021

service/dynamodb/dynamodbattribute: New Encoder and Decoder Behavior …

c7fa04b

…for Empty Collections (aws#401)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Statically typed CloudFormation Templates #401

Statically typed CloudFormation Templates #401

rosenhouse commented Oct 12, 2015

jasdel commented Oct 19, 2015

rosenhouse commented Nov 5, 2015

jasdel Nov 5, 2015

jasdel commented Nov 5, 2015

rosenhouse commented Nov 5, 2015

jasdel commented Nov 5, 2015

rosenhouse commented Nov 5, 2015

Statically typed CloudFormation Templates #401

Statically typed CloudFormation Templates #401

Conversation

rosenhouse commented Oct 12, 2015

jasdel commented Oct 19, 2015

rosenhouse commented Nov 5, 2015

jasdel Nov 5, 2015

Choose a reason for hiding this comment

jasdel commented Nov 5, 2015

rosenhouse commented Nov 5, 2015

jasdel commented Nov 5, 2015

rosenhouse commented Nov 5, 2015