Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(cli): synth fails if there was an error when synthesizing the stack #14613

Merged
merged 24 commits into from
May 10, 2021
Merged

fix(cli): synth fails if there was an error when synthesizing the stack #14613

merged 24 commits into from
May 10, 2021

Conversation

otaviomacedo
Copy link
Contributor

All stacks created inside a pipeline stage will be flagged for validation. After synth is done, the CLI will validate all flagged stacks plus the stacks that were explicitly specified.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Otavio Macedo and others added 21 commits April 30, 2021 15:10
* feat: cloudformation spec v35.1.0

* chore: attribute exception

Co-authored-by: AWS CDK Team <aws-cdk@amazon.com>
Co-authored-by: Nick Lynch <nlynch@amazon.com>
The combination of CodePipeline+CodeBuild will not properly extract
symlinks from a git repository. Instead, the symlink will be replaced
with a textual representation of its target.

To properly test the symlink behavior, we need to remove the symlink
from version control and re-create it at test time, after
CodePipeline+CodeBuild have extracted the archive.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…alized JSON (#14512)

Our logic for parsing `${}` expressions inside `Fn::Sub` incorrectly looked for closing braces everywhere in the string,
instead of doing it to the right of the opening `${`.
This would fail if the string passed to `Fn::Sub` was serialized JSON
(as those would contain closing braces completely unrelated to the `${`).

Fixes #14095

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
* deprecate MariaDB `10.0`, `10.1`
* add MariaDB `10.2.32`
* deprecate MySQL `5.5`, `5.6`
* add PostgreSQL `9.6.20`, `9.6.21`, `10.15`, `10.16`, `11.10`, `11.11`, `12.6`, `13.2`

fixes #14495.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Docker is only required when not bundling locally.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Ran npm-check-updates and yarn upgrade to keep the `yarn.lock` file up-to-date.
This reverts commit 9e421e5 (#14205).

Closes #14476


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
As a possible solution for #12597, refer to the 3rd-party module `cdk-ecr-deployment` which supports deploying image assets to a specific ECR repository.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Closes [#13845](#13845)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Closes #12555 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
A previous [PR](#13823) added a dependency from the stable module `aws-events-targets` to the experimental module `aws-batch`. Depending on experimental modules is no longer allowed.  In fact, this PR build should have failed when attempting to build `aws-cdk-lib`, preventing it from being merged. However, since the dependency was added but not used, the build succeeded, and the PR was merged.

### Why did we expect the PR build of #13823 to fail?

In the build step of `aws-cdk-lib` we strip all the L2 of experimental modules. This means that when compiling `aws-cdk-lib` the L2 types of experimental modules does not exists, if a stable module reference such a type, the build of `aws-cdk-lib` will fail. 

### Why did the PR build of #13823 succeeded?

It added a dependency on the experimental `aws-batch` but didn't use it.

### What are we doing to prevent this from happening in the future?
 #14249 adds a lint rule that disallow depending on an experimental module.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
> blocked on #14291

This PR adds a pkglint rule to disallow `@aws-cdk` modules to depend on experimental `@aws-cdk` modules. 

In v2 experimental modules will be released separately from `aws-cdk-lib`. This means that stable modules can not depend on experimental modules as it will create a cyclic dependency. The lint rule **also** ban exp->exp dependencies. The reasoning for the latter is that if an experimental module A, depends on experimental module B, in order to graduate module A, module B will also have to be graduated (to satisfy no stable->exp). Currently there are a few exp->exp dependencies, these might be removed as a part of the work to release the experimental modules separately from `aws-cdk-lib`. In the meantime we explicitly allow them by having an exclusion map.

The current implementation executes the check for each modules as a part of its `pkglint` phase, which is O(n^2) (for each module check the stability of all of its dependencies). This can be improved by executing the check only once, in the build of `aws-cdk-lib` (going over all the modules in a topological order and storing the stability). The disadvantage of executing it only on the build of `aws-cdk-lib` is that it will not be executed as a part of a typical developer flow, which is `buildup` from a specific package. I have testes it several times locally in the current implementation adds ~2 seconds to the root level `yarn pkglint`. Since it does not add significant time to our build, and gives better/earlier visibility to contributors I have decided to keep it simple and preform the check on each module build. Happy to discuss more if anyone have concerns or optimizations ideas I missed.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
introducing a higher level construct to define golang lambda functions. This was inspired by the
other high level constructs for Nodejs and Python

I'm setting a couple of things by default.

*Environment Variables*

* `GOOS=linux`
* `GOARCH=amd64`
* `GO111MODULE=on`

*AssetHashType*

* `AssetHashType.OUTPUT`
I've added some details in the README on when you could change this to `INPUT`. I'm not really sure what the majority use case would be though.

*Go Modules*

I am requiring the use of Go modules which were added in Go 1.11 (current version is Go 1.15). I think at this point most Go developers will probably be using Go modules (for a comparison the [aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) requires 1.15)

*Runtime*

* `Runtime.PROVIDED_AL2`

The `GO_1_X` runtime doesn't support newer Lambda features like Lambda extensions, and the [aws-lambda-go](https://github.com/aws/aws-lambda-go) library supports the provided runtimes out of the box.


Also, I copied over a lot of the boilerplate (package.json, LICENSE, etc) from the lambda-nodejs package and just did a search and replace, let me know if I got any of it wrong.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
)

Resolves #13294

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@gitpod-io
Copy link

gitpod-io bot commented May 10, 2021

@otaviomacedo otaviomacedo changed the title Stack synth validation fix(cli): synth fails if there was an error when synthesizing the stack May 10, 2021
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label May 10, 2021
@@ -12,7 +12,10 @@ Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to make these changes to the NOTICE files because I was getting errors like this:

In package package.json
- [license/3p-attributions] Unnecessary attribution found for dependency 'lru-cache' in NOTICE file. Attribution is determined from package.json (all "bundledDependencies" or the list in "pkglint.attribution")
- [license/3p-attributions] Unnecessary attribution found for dependency 'yallist' in NOTICE file. Attribution is determined from package.json (all "bundledDependencies" or the list in "pkglint.attribution")
- [license/3p-attributions] Unnecessary attribution found for dependency 'semver' in NOTICE file. Attribution is determined from package.json (all "bundledDependencies" or the list in "pkglint.attribution")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know why you were getting these errors, but the attribution is/was correct.

The problem here is that the tool we're using to determine which NPM dependencies are "bundled dependencies" is a bit shit, and it's not always detecting all dependencies.

I have a sense we need to replace this tool with something fixed.

See:

export class ThirdPartyAttributions extends ValidationRule {

return bundled.sync({ path: this.packageRoot });

const bundled = require('npm-bundled');

The package npm-bundled is just bad, and very dependent on how NPM chooses to lay out the package tree on whether it gets the right answer or not. (Note that the author of npm-bundled is in no way affiliated with NPM-the-package-manager, they just chose to name their package that way)

@@ -12,7 +12,10 @@ Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
of the Software, and to permit persons to whom t
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably unintended.

@@ -12,7 +12,10 @@ Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know why you were getting these errors, but the attribution is/was correct.

The problem here is that the tool we're using to determine which NPM dependencies are "bundled dependencies" is a bit shit, and it's not always detecting all dependencies.

I have a sense we need to replace this tool with something fixed.

See:

export class ThirdPartyAttributions extends ValidationRule {

return bundled.sync({ path: this.packageRoot });

const bundled = require('npm-bundled');

The package npm-bundled is just bad, and very dependent on how NPM chooses to lay out the package tree on whether it gets the right answer or not. (Note that the author of npm-bundled is in no way affiliated with NPM-the-package-manager, they just chose to name their package that way)

@@ -3,25 +3,6 @@ Copyright 2018-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

-------------------------------------------------------------------------------

The AWS CDK includes the following third-party software/licensing:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, these attributions should not be removed.

@@ -1,55 +1,2 @@
AWS Cloud Development Kit (AWS CDK)
Copyright 2018-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again

const stacksToValidate = await this.selectStacksForList(idsToValidate);
await this.validateStacks(stacksToValidate);
const allStacks = await this.selectStacksForList([]);
await this.validateStacks(allStacks.filter(art => art.validateOnSynth ?? false));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preferably we'd validate all stacks in one validateStacks() call. The difference is a more complete view of all errors, as synthesis will stop after this call if it encounters errors (and so wouldn't show any errors that would be encountered in the validateStacks() call below)

@rix0rrr rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label May 10, 2021
rix0rrr
rix0rrr previously approved these changes May 10, 2021
@mergify mergify bot dismissed rix0rrr’s stale review May 10, 2021 13:35

Pull request has been modified.

@otaviomacedo otaviomacedo removed the pr/do-not-merge This PR should not be merged at this time. label May 10, 2021
@mergify
Copy link
Contributor

mergify bot commented May 10, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 6eb63f6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 71c61e8 into aws:master May 10, 2021
@mergify
Copy link
Contributor

mergify bot commented May 10, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

eladb pushed a commit that referenced this pull request May 12, 2021
* chore(cloudfront): remove the use of calculateFunctionHash (#14583)

`calculateFunctionHash()` was used to compute the 'refresh token' of the
custom resource for the EdgeFunction construct.

This method is private to the lambda module and is deemed to be changed.
Instead, use the lambda function version's logical id.

The logical id of the version includes computing the function hash (among
others) and is a more reliable determinant of whether the underlying
function version changed.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* feat(cloudwatch): validate parameters for a metric dimensions (closes #3116) (#14365)

As per #3116, the changes in this PR validate metric dimension values (length, and checking if the value is null or undefined) and throw errors if the values are not valid. I've also corrected a comment in the metric-types.ts to use the correct method name

* feat(appmesh): change HealthChecks to use protocol-specific union-like classes (#14432)

BREAKING CHANGE: HealthChecks require use of static factory methods

fixes #11640

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* feat(msk): Cluster L2 Construct (#9908)

L2 Construct for a MSK Cluster. 

I wrote this for internal use and thought I'd share it. I tried to follow the [example resource](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/example-construct-library/lib/example-resource.ts) and [design guidelines](https://github.com/aws/aws-cdk/blob/master/DESIGN_GUIDELINES.md) as much as I could. Default properties were chosen either based on defaults when creating a cluster in the console or defaults set from CloudFormation.

Closes #9603

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* feat(kms): allow specifying key spec and key usage (#14478)

This allows specifying key spec and key usage, so you can create asymmetric keys.

closes #5639


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* chore: add `@types/jest` to a package that was missing it (#14609)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* chore: issue template for bugs to have an SSCCE example (#14615)

Ask customers to provide reproducible code snippets
to reduce the amount of triage time required.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* fix(cli): synth fails if there was an error when synthesizing the stack (#14613)

All stacks created inside a pipeline stage will be flagged for validation. After synth is done, the CLI will validate all flagged stacks plus the stacks that were explicitly specified.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* chore: annotate `aws-lambda-go` with `docker` requirement (#14618)

The `nozem` build tool needs to know that `docker` is required to
build/test this particular package.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* feat(elbv2): preserveClientIp for NetworkTargetGroup (#14589)

Allows users to configure client IP preservation for network target groups.

See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticloadbalancingv2-targetgroup-targetgroupattribute.html

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* chore: npm-check-updates && yarn upgrade (#14620)

Ran npm-check-updates and yarn upgrade to keep the `yarn.lock` file up-to-date.

* chore(mergify): add @BenChaimberg to team roster

* chore(release): 1.103.0

* chore: mark "otaviomacedo" as core contributor (#14619)

Co-authored-by: Otavio Macedo <otaviomacedo@protonmail.com>

* chore(cli): add npm command to upgrade notice (#14621)

A colleague had to go look up the command to update the CDK CLI. It occurred to me that is probably common with developers who don't work with NPM on a daily basis, such as anyone who isn't developing in TypeScript or JavaScript.

Put the necessary command right in the upgrade notice.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* fix(lambda): custom resource fails to connect to efs filesystem (#14431)

Fixes: #14430

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* feat(cfnspec): cloudformation spec v35.2.0 (#14610)

* feat: cloudformation spec v35.2.0

* add spec patches

Co-authored-by: AWS CDK Team <aws-cdk@amazon.com>
Co-authored-by: Elad Ben-Israel <benisrae@amazon.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* fix(lambda-nodejs): handler filename missing from error message (#14564)


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

* chore(build): clarify prerequisites in CONTRIBUTING and verify before build (#14642)

The build script at the top level of the repository performs a prerequisites check before beginning the build. This verification is not complete as it does not include the required checks for .NET and Python executables. Further, the prerequisites documentation in the contributing guide does not note the Docker requirement.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

Co-authored-by: Niranjan Jayakar <nija@amazon.com>
Co-authored-by: OksanaH <34384274+OksanaH@users.noreply.github.com>
Co-authored-by: Dominic Fezzie <fezzid@amazon.com>
Co-authored-by: Curtis <curtis.eppel@cultureamp.com>
Co-authored-by: Shinya Tsuda <shinya@dacci.org>
Co-authored-by: Rico Huijbers <rix0rrr@gmail.com>
Co-authored-by: Otavio Macedo <otaviomacedo@protonmail.com>
Co-authored-by: Griffin Byatt <byatt.griffin@gmail.com>
Co-authored-by: Ben Chaimberg <chaimber@amazon.com>
Co-authored-by: AWS CDK Team <aws-cdk@amazon.com>
Co-authored-by: Mitchell Valine <valinm@amazon.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jerry Kindall <52084730+Jerry-AWS@users.noreply.github.com>
Co-authored-by: Daniel Neilson <53624638+ddneilson@users.noreply.github.com>
Co-authored-by: Elad Ben-Israel <benisrae@amazon.com>
rix0rrr added a commit that referenced this pull request Jun 16, 2021
In #14613, we introduced validation so that `cdk synth` would guard
against incomplete context lookups. In the past, stacks with missing
context would have successfully completed synthesis, propagated down
the pipeline and caused confusing error messages during deployment.
The new behavior was to error out early in case there was missing
context.

This broke people who had resorted to resolving context in the pipeline
using multiple `synth` commands in `for`-loop: this used to work because
the `synths` would be incomplete but silently succeed, but with the new
validation the very first `cdk synth` would start failing and the `for`
loop would never complete.

This PR adds a `--no-validation` flag to `cdk synth` to stop the
additional validation, so the `for` loop can complete successfully.

The same behavior can be controlled with an environment variable,
by setting `CDK_VALIDATION=false`.

Fixes #15130.
rix0rrr added a commit that referenced this pull request Jun 16, 2021
In #14613, we introduced validation so that `cdk synth` would guard
against incomplete context lookups. In the past, stacks with missing
context would have successfully completed synthesis, propagated down
the pipeline and caused confusing error messages during deployment.
The new behavior was to error out early in case there was missing
context.

This broke people who had resorted to resolving context in the pipeline
using multiple `synth` commands in `for`-loop: this used to work because
the `synths` would be incomplete but silently succeed, but with the new
validation the very first `cdk synth` would start failing and the `for`
loop would never complete.

This PR adds a `--no-validation` flag to `cdk synth` to stop the
additional validation, so the `for` loop can complete successfully.

The same behavior can be controlled with an environment variable,
by setting `CDK_VALIDATION=false`.

Fixes #15130.
matthewsvu pushed a commit to matthewsvu/aws-cdk that referenced this pull request Jun 22, 2021
In aws#14613, we introduced validation so that `cdk synth` would guard
against incomplete context lookups. In the past, stacks with missing
context would have successfully completed synthesis, propagated down
the pipeline and caused confusing error messages during deployment.
The new behavior was to error out early in case there was missing
context.

This broke people who had resorted to resolving context in the pipeline
using multiple `synth` commands in `for`-loop: this used to work because
the `synths` would be incomplete but silently succeed, but with the new
validation the very first `cdk synth` would start failing and the `for`
loop would never complete.

This PR adds a `--no-validation` flag to `cdk synth` to stop the
additional validation, so the `for` loop can complete successfully.

The same behavior can be controlled with an environment variable,
by setting `CDK_VALIDATION=false`.

Fixes aws#15130.
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
…ck (aws#14613)

All stacks created inside a pipeline stage will be flagged for validation. After synth is done, the CLI will validate all flagged stacks plus the stacks that were explicitly specified.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
In aws#14613, we introduced validation so that `cdk synth` would guard
against incomplete context lookups. In the past, stacks with missing
context would have successfully completed synthesis, propagated down
the pipeline and caused confusing error messages during deployment.
The new behavior was to error out early in case there was missing
context.

This broke people who had resorted to resolving context in the pipeline
using multiple `synth` commands in `for`-loop: this used to work because
the `synths` would be incomplete but silently succeed, but with the new
validation the very first `cdk synth` would start failing and the `for`
loop would never complete.

This PR adds a `--no-validation` flag to `cdk synth` to stop the
additional validation, so the `for` loop can complete successfully.

The same behavior can be controlled with an environment variable,
by setting `CDK_VALIDATION=false`.

Fixes aws#15130.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

Successfully merging this pull request may close these issues.