feat(cli): --concurrency option #20345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
github-actions
bot
added
effort/medium
|
I have attempted to reproduce the failures on #20230 using these docs test/integ/run-against-repo test/integ/cli/test.shAll the tests pass but none of the tests listed in #20230 actually run I am able to run those those tests using: yarn jest --testMatch '**/integ/cli/cli.integtest.js'Add all tests pass with and without piping to @rix0rrr any suggestions for how to run the CLI integ tests and how to reproduce the errors? |
|
To save me some time checking, what did you change between the previous version of the PR and this one? |
Nothing - I haven't been able to reproduce the test failures so I'm not sure how to fix the previous issue. Can you help with running the integ tests and reproducing the failures? The documentation appears to be outdated or incorrect as these specific tests do not run with the above commands |
Oh no the problem is actually in |
|
The magic invocation to force detection of a non-TTY device might actually be |
relm923
force-pushed
the
relm/concurrent-deploys2
branch
from
3b35f6d
to
aa00342
Compare
|
Thanks @rix0rrr! I can successfully run the CLI integration tests with #20414 Unfortunately I still cannot actually reproduce any of the test failures. I've tried the following: $ test/integ/run-against-repo test/integ/cli/test.sh
$ test/integ/run-against-repo test/integ/cli/test.sh cli.integtest | less
$ test/integ/run-against-repo test/integ/cli/test.sh cli.integtest < /dev/null
$ true | (/opt/homebrew/opt/util-linux/bin/setsid ./test/integ/run-against-repo test/integ/cli/test.sh) 2>&1 | cat |
|
Is there anything someone needs help with to get this moved forward? |
|
we are really waiting for this one! is there anything that can be done to help? |
|
We have also been actively watching this for a while. I think it would help a lot of people if some feedback could be given on how this pr can progress. |
|
CLI integration tests that are failing: Looks to be error propagation: exceptions thrown in the concurrent deployment are not properly propagated to the main loop, but escape as unhandled promise rejections. |
relm923
commented
Jul 4, 2022
|
I'm trying to think about it, and I think await Promise.all(stackDeployPromises);
await queue.onIdle();Should be: await queue.onIdle();
await Promise.all(stackDeployPromises);No? Because in the current iteration, we will only await the If I'm reading this correctly, this is a very subtle bug and hard to find and reason about. I'm not very comfortable with the logic as currently written. Can you extract the logic of doing concurrent DAG operations (async tasks and dependencies between them) into a helper class, and add appropriate tests for it to increase confidence? Let's do at least the following tests:
I think if you're implementing this, we can even get rid of the |
rix0rrr
previously requested changes
Jul 18, 2022
|
Hi any news on that? so many people are waiting for that... @relm923 do you plan fixing the missing bits or need the community help? |
relm923
force-pushed
the
relm/concurrent-deploys2
branch
from
2737917
to
47bbcf3
Compare
ShadowDog007
reviewed
Aug 2, 2022
packages/aws-cdk/lib/deploy.ts
Outdated
|
|
||
| await enqueueStackDeploys(); | ||
|
|
||
| const results = await Promise.allSettled(deployPromises); |
There was a problem hiding this comment.
As rix0rrr mentioned await queue.onIdle(); should probably come first here.
Promise.allSettled is likely to be passed only promises for stack with no dependencies (changes to the array aren't tracked after it's called) so would resolve before all stacks have completed.
Calling await queue.onIdle(); first would resolve after everything is finished (only because deploy promises resolve after queuing stacks that become unblocked).
Also worth mentioning that enqueueStackDeploys is marked as async, but doesn't actually use await, same with the function passed to stacks.forEach
There was a problem hiding this comment.
Moved onIdle before allSettled and updated enqueueStackDeploys to sync
Still working through some strange PromiseRejectionHandledWarning warnings. Will take another look later this week but would welcome any help
There was a problem hiding this comment.
I think that warning is because we don't immediately await the tasks returned from queue.add
From the p-queue docs for .add
Note: If your items can potentially throw an exception, you must handle those errors from the returned Promise or they may be reported as an unhandled Promise rejection and potentially cause your process to exit immediately.
Perhaps it'd be better to have something like this:
try {
await queue.add(...);
} catch (err) {
deployErrors.push(err)
} There was a problem hiding this comment.
That will block/await on each individual stack deploy leading to a sequential deployment
There was a problem hiding this comment.
That code would be inside of stacks.forEach(async stack =>) so it wouldn't block. Array.prototype.forEach isn't promise aware so won't wait for completion, it'll effectively run up to the first await (starting the deployment, but not waiting for it)
Example:
There was a problem hiding this comment.
🤦♂️ thank you @ShadowDog007 I definitely was over thinking it. All appears to be working now
rix0rrr
previously approved these changes
Aug 9, 2022
relm923
force-pushed
the
relm/concurrent-deploys2
branch
from
2ece809
to
fa2369b
Compare
rix0rrr
approved these changes
Aug 10, 2022
|
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
|
@relm923 Hey. Are you aware of any problems that might result from expanding |
I've never actually used |
This was referenced Aug 14, 2022
mergify bot
pushed a commit
that referenced
this pull request
Aug 16, 2022
This PR adds #20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes #21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
rix0rrr
added a commit
that referenced
this pull request
Aug 18, 2022
The new concurrency logic introduced in #20345 was checking for all dependencies to a stack to have been deployed previously for the stack to be unblocked, but if not all stacks are selected, they will never become unblocked. Because there was no check on being "complete" at the end, this silently leads to no stacks being deployed and the CLI no-opping. Fix by only waiting for selected stacks among the dependencies, and adding a safeguard at the end to make sure we deployed all stacks successfully (or failed halfway through).
Naumel
added a commit
that referenced
this pull request
Aug 18, 2022
4 tasks
rix0rrr
pushed a commit
that referenced
this pull request
Aug 19, 2022
This PR adds #20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes #21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
rix0rrr
added a commit
that referenced
this pull request
Aug 19, 2022
The new concurrency logic introduced in #20345 was checking for all dependencies to a stack to have been deployed previously for the stack to be unblocked, but if not all stacks are selected, they will never become unblocked. Because there was no check on being "complete" at the end, this silently leads to no stacks being deployed and the CLI no-opping. Fix by only waiting for selected stacks among the dependencies, and adding a safeguard at the end to make sure we deployed all stacks successfully (or failed halfway through).
josephedward
pushed a commit
to josephedward/aws-cdk
that referenced
this pull request
Aug 30, 2022
Closes aws#1973 (reattempting aws#19378) Add `--concurrency` parameter to `cdk deploy` command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with `--progress events` due to the poor interaction of concurrent deployments and the progress bar rendering. ### Open Questions - [x] How best to write automated tests around this? Added unit and integration tests - [x] Should other commands (ex: `destroy`) support concurrency? Only supporting deploy command in this PR - [x] Any other concerns with this approach as it changes a key component of `cdk` - [x] How should this work with the `--exclusively` flag? Only check dependencies between requested stacks <details> <summary>Example Output (Success):</summary> <pre> $ yarn cdk deploy --all --require-approval "never" --concurrency 3 --progress bar yarn run v1.22.17 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.04s⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-test-1 relm-test-1: deploying... relm-test-2: deploying... relm-test-2: creating CloudFormation changeset... relm-test-1: creating CloudFormation changeset... relm-test-1 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-1 | 0/3 | 4:57:32 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 4:57:33 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-2 | 0/3 | 4:57:31 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2 | 1/3 | 4:57:32 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2 relm-test-1 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-1 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-1 ✅ relm-test-1 ✨ Deployment time: 28.39s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-1/ffcdb210-a6fd-11ec-b05c-0a02909bf4fb ✨ Total time: 44.43s ✅ relm-test-2 ✨ Deployment time: 28.45s Outputs: relm-test-2.ExportsOutputRefTempTopic9C0CBD7CC97C6BE5 = arn:aws:sns:us-east-1:XXXXXXXXXXXX:relm-test-2-TempTopic9C0CBD7C-2ETJWELY3MFD Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2/ffbc7400-a6fd-11ec-ac95-0ea201c9d581 ✨ Total time: 44.49s relm-test-2B relm-test-2B: deploying... relm-test-2B: creating CloudFormation changeset... relm-test-2B | 0/4 | 4:57:47 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:57:58 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 0/4 | 4:58:05 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) Resource creation Initiated relm-test-2B | 0/4 | 4:58:06 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2B | 1/4 | 4:58:07 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 2/4 | 4:58:19 PM | CREATE_COMPLETE | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 2/4 | 4:58:22 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 2/4 | 4:58:23 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) Resource creation Initiated relm-test-2B | 3/4 | 4:58:36 PM | CREATE_COMPLETE | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 4/4 | 4:58:37 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2B ✅ relm-test-2B ✨ Deployment time: 54.71s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2B/10b341d0-a6fe-11ec-a535-12d49b16bf9b ✨ Total time: 70.75s ✨ Done in 99.79s. </pre> </details> <details> <summary>Example Output (Failure):</summary> <pre> yarn run v1.22.18 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.18s⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-broken-1 relm-broken-1: deploying... relm-broken-2 relm-broken-2: deploying... relm-test-1: deploying... relm-test-1: creating CloudFormation changeset... relm-broken-1: creating CloudFormation changeset... relm-broken-2: creating CloudFormation changeset... relm-broken-2 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-2 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-broken-1 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-test-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:54 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-test-1 | 0/3 | 9:13:01 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-broken-2 | 0/3 | 9:13:02 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-broken-2 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 relm-broken-1 | 0/3 | 9:13:05 PM | CREATE_FAILED | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation cancelled new MetadataResource (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/metadata-resource.ts:22:24) \_ /Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:166:5 \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:231:5) \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:227:5) \_ injectMetadataResources (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:157:3) \_ Object.synthesize (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:18:3) \_ App.synth (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/stage.ts:180:23) \_ process.<anonymous> (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/app.ts:131:45) \_ Object.onceWrapper (node:events:646:26) \_ process.emit (node:events:526:28) \_ process.emit (node:domain:475:12) \_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/relm/Development/aws-cdk-test-stack/node_modules/@cspotcode/source-map-support/source-map-support.js:613:40) relm-broken-1 | 0/3 | 9:13:06 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 The following resource(s) failed to create: [TestTopic339EC197, CDKMetadata]. Rollback requested by user. relm-broken-1 | 1/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 2/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 3/3 | 9:13:11 PM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | relm-broken-1 relm-broken-2 | 2/3 | 9:13:11 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 3/3 | 9:13:12 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-broken-2 relm-test-1 | 2/3 | 9:13:10 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Failed resources: relm-broken-1 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 ❌ relm-broken-1 failed: Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307 throw new Error(`The stack named ${stackName} failed creation, it may need to be manually deleted from the AWS console: ${status}`); ^ Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. </pre> </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
josephedward
pushed a commit
to josephedward/aws-cdk
that referenced
this pull request
Aug 30, 2022
This PR adds aws#20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes aws#21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
josephedward
pushed a commit
to josephedward/aws-cdk
that referenced
this pull request
Aug 30, 2022
josephedward
pushed a commit
to josephedward/aws-cdk
that referenced
this pull request
Aug 30, 2022
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1973 (reattempting #19378)
Add
--concurrencyparameter tocdk deploycommand to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with--progress eventsdue to the poor interaction of concurrent deployments and the progress bar rendering.Open Questions
Added unit and integration tests
destroy) support concurrency?Only supporting deploy command in this PR
cdk--exclusivelyflag?Only check dependencies between requested stacks
Example Output (Success):
Example Output (Failure):
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license