feat(cli): `--concurrency` option #19378

relm923 · 2022-03-13T15:34:59Z

Closes #1973

Add --concurrency parameter to cdk deploy command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with --progress events due to the poor interaction of concurrent deployments and the progress bar rendering.

Open Questions

How best to write automated tests around this?
Added unit and integration tests
Should other commands (ex: destroy) support concurrency?
Only supporting deploy command in this PR
Any other concerns with this approach as it changes a key component of cdk
How should this work with the --exclusively flag?
Only check dependencies between requested stacks

Example output:

$ yarn cdk deploy --all --require-approval "never" --concurrency 3 --progress bar
yarn run v1.22.17
warning package.json: No license field
$ cdk deploy --all --require-approval never --concurrency 3 --progress bar

✨  Synthesis time: 16.04s

⚠️ The --concurrency flag only supports --progress "events". Switching to "events".
relm-test-1
relm-test-1: deploying...
relm-test-2: deploying...
relm-test-2: creating CloudFormation changeset...
relm-test-1: creating CloudFormation changeset...
relm-test-1 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-1 User Initiated
relm-test-1 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-1 User Initiated
relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) 
relm-test-2 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-2 User Initiated
relm-test-2 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-2 User Initiated
relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) 
relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated
relm-test-1 | 0/3 | 4:57:32 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
relm-test-1 | 1/3 | 4:57:33 PM | CREATE_COMPLETE      | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-2 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated
relm-test-2 | 0/3 | 4:57:31 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
relm-test-2 | 1/3 | 4:57:32 PM | CREATE_COMPLETE      | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-2 | 2/3 | 4:57:41 PM | CREATE_COMPLETE      | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) 
relm-test-2 | 3/3 | 4:57:43 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | relm-test-2 
relm-test-1 | 2/3 | 4:57:41 PM | CREATE_COMPLETE      | AWS::SNS::Topic    | TempTopic (TempTopic9C0CBD7C) 
relm-test-1 | 3/3 | 4:57:43 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | relm-test-1 

 ✅  relm-test-1

✨  Deployment time: 28.39s

Stack ARN:
arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-1/ffcdb210-a6fd-11ec-b05c-0a02909bf4fb

✨  Total time: 44.43s


 ✅  relm-test-2

✨  Deployment time: 28.45s

Outputs:
relm-test-2.ExportsOutputRefTempTopic9C0CBD7CC97C6BE5 = arn:aws:sns:us-east-1:XXXXXXXXXXXX:relm-test-2-TempTopic9C0CBD7C-2ETJWELY3MFD
Stack ARN:
arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2/ffbc7400-a6fd-11ec-ac95-0ea201c9d581

✨  Total time: 44.49s

relm-test-2B
relm-test-2B: deploying...
relm-test-2B: creating CloudFormation changeset...
relm-test-2B | 0/4 | 4:57:47 PM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-2B User Initiated
relm-test-2B | 0/4 | 4:57:58 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | relm-test-2B User Initiated
relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role     | TopicRole (TopicRole3526982D) 
relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-2B | 0/4 | 4:58:05 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role     | TopicRole (TopicRole3526982D) Resource creation Initiated
relm-test-2B | 0/4 | 4:58:06 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
relm-test-2B | 1/4 | 4:58:07 PM | CREATE_COMPLETE      | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) 
relm-test-2B | 2/4 | 4:58:19 PM | CREATE_COMPLETE      | AWS::IAM::Role     | TopicRole (TopicRole3526982D) 
relm-test-2B | 2/4 | 4:58:22 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy   | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) 
relm-test-2B | 2/4 | 4:58:23 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy   | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) Resource creation Initiated
relm-test-2B | 3/4 | 4:58:36 PM | CREATE_COMPLETE      | AWS::IAM::Policy   | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) 
relm-test-2B | 4/4 | 4:58:37 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | relm-test-2B 

 ✅  relm-test-2B

✨  Deployment time: 54.71s

Stack ARN:
arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2B/10b341d0-a6fe-11ec-a535-12d49b16bf9b

✨  Total time: 70.75s


✨  Done in 99.79s.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2022-03-13T15:35:02Z

relm923 · 2022-03-13T15:35:32Z

packages/aws-cdk/lib/cdk-toolkit.ts

@@ -153,11 +154,17 @@ export class CdkToolkit {
      warning('⚠️ It should only be used for development - never use it for your production Stacks!');
    }

+    const stacks = stackCollection.stackArtifacts;


This all works because stackArtifacts is pre-sorted based on stack dependencies

relm923 · 2022-03-17T14:19:57Z

@rix0rrr sorry for the direct tag but this PR didn't get auto-assigned and it looks like you've been on most of the PRs in these files. Any chance you'd be able to review or point me to the best person to get feedback?

rix0rrr

Thanks for taking initiative on this! I really appreciate it. However, things are rarely as simple as they seem:

I see why you made the exception for --require-approval... but this doesn't seem like the best way to go about things. A solution that allows for keeping the approval might toposort into levels, and do group approvals on each level before deploying them all in parallel.

That is not the biggest concern I have though, because it's something that allows for future extension. The bigger one is:

The stack monitor. We'll have multiple stack monitor outputs all trampling over each other. The stack monitor will need to be made multi-stack aware, and have output limitations placed on them, in order to have properly formatted output.

Also, the "total time" reporting seems to have disappeared.

Pull request has been modified.

relm923 · 2022-03-17T16:38:27Z

Thanks for the reply @rix0rrr!

I see why you made the exception for --require-approval... but this doesn't seem like the best way to go about things. A solution that allows for keeping the approval might toposort into levels, and do group approvals on each level before deploying them all in parallel.

I agree this is where we ultimately want to get to but what are your thoughts on leaving it as is for now and following up with a separate approval refactor?

The stack monitor. We'll have multiple stack monitor outputs all trampling over each other. The stack monitor will need to be made multi-stack aware, and have output limitations placed on them, in order to have properly formatted output.

I'll need to do some more digging around the monitors. Any suggestions around how to test the latest changes from the CLI level?

Also, the "total time" reporting seems to have disappeared.

Added back in the latest version. Although the calculation is a little odd as its total synth time + individual stack deploy time but that is pre-existing

relm923 · 2022-03-17T17:11:51Z

packages/aws-cdk/test/integ/cli/cli.integtest.ts

@@ -142,7 +150,17 @@ integTest('deploy', withDefaultFixture(async (fixture) => {
 integTest('deploy all', withDefaultFixture(async (fixture) => {
  const arns = await fixture.cdkDeploy('test-*', { captureStderr: false });

-  // verify that we only deployed a single stack (there's a single ARN in the output)


Pre-existing incorrect comment

relm923 · 2022-03-18T21:08:44Z

@rix0rrr as far as I can tell the Stack monitors work just fine concurrently but the monitor output can be confusing. I updated the logic to require --progress events when using concurrency to avoid this confusion. I've added an example of the logs in the description. Let me know what you think

relm923 · 2022-03-23T13:28:19Z

@rix0rrr how do you want to proceed here? Happy to make a separate PR to address the security approvals

laimonassutkus · 2022-05-03T15:28:26Z

In the meantime:
https://github.com/Biomapas/B.AwsCdkParallel

rix0rrr

All looks pretty good, after some minor changes I'm happy to merge this. I want everyone to know not concerned about this pull request itself: the code looks good and the feature makes sense.

I'm concerned about the inevitable follow-up requests this will bring:

Please make it work with confirmations (likelihood: high)
Now that more requests are executing in parallel, I'm getting more throttles and my deployments are failing as a result (likelihood: high)
Please make it work with the other progress reporter (likelihood: low-medium)

@relm923 will you be around to do follow-ups if issues come up?

One more thing that popped into my head: what happens if failures occur while a parallel deployment is happening? As best I can imagine, some red line will appear with an error, then a whole heap of SUCCESS will be printed as another stack that's deploying at the same time completes successfully, then we will print "Stack XYZ successfully deployed", and subsequently we'll exit with an error.

Can you tell me how confusing it will look when this happens?

rix0rrr · 2022-05-04T11:32:56Z

packages/aws-cdk/lib/cdk-toolkit.ts

+        .every((id) => !stacksAwaitingDeploy[id]);
+
+    const enqueueStackDeploys = async () => {
+      while (stacks[0] && isStackUnblocked(stacks[0])) {


We're leaving some concurrency on the table here by only considering the first element from the list.

If we had dependencies:

C -> A D -> B

And the list happened to toposort as [A, B, C, D], then with this implementation even if B finishes we still can't start D because we are forced to wait for C to start first.

This can be changed to a for loop -- on every iteration, consider all stacks.

It should probably be a forEach, otherwise we might run afoul of the "closing over a loop variable" gotcha.

Great catch! Updated to a forEach loop

rix0rrr · 2022-05-04T11:34:19Z

packages/aws-cdk/test/integ/cli/cli.integtest.ts

+}));
+
+integTest('deploy all concurrently', withDefaultFixture(async (fixture) => {
+  const arns = await fixture.cdkDeploy('test-*', {


I'm not sure this is safe, many stacks in the integ app were designed to be deployed in a "just-so" way. Please pick a well-defined subset of like 5 stacks or so.

I was mainly basing this off the pre-existing deploy-all test case -

aws-cdk/packages/aws-cdk/test/integ/cli/cli.integtest.ts

Line 151 in 9f57e25

integTest('deploy all', withDefaultFixture(async (fixture) => {

Happy to pick a different set if you'd prefer

rix0rrr · 2022-05-04T11:41:54Z

packages/aws-cdk/test/integ/cli/cli.integtest.ts

@@ -142,7 +150,17 @@ integTest('deploy', withDefaultFixture(async (fixture) => {
 integTest('deploy all', withDefaultFixture(async (fixture) => {
  const arns = await fixture.cdkDeploy('test-*', { captureStderr: false });

-  // verify that we only deployed a single stack (there's a single ARN in the output)


polothy · 2022-05-04T16:31:19Z

One more thing that popped into my head: what happens if failures occur while a parallel deployment is happening? As best I can imagine, some red line will appear with an error, then a whole heap of SUCCESS will be printed as another stack that's deploying at the same time completes successfully, then we will print "Stack XYZ successfully deployed", and subsequently we'll exit with an error.

In my custom tool that does parallel deployments, I had this same problem (especially when I was doing 50 in parallel). What I ended up doing was on error:

Stop starting any new deployments.
Let current deployments complete and print their results.
Once everything is done, re-print the failures at the end (EG: Re-printing failure 1 of 5).

This makes it really easy to find what failed in a wall of text.

Now that more requests are executing in parallel, I'm getting more throttles and my deployments are failing as a result (likelihood: high)

This can be a problem. I think the primary thing that causes throttling is the API call to check on the status of the CFN deployment. Think before this PR it was every 3 seconds which is too low for parallel deployments. Think for parallel deployments, we can just default to higher and then you can also use AWS SDK settings for controlling retries (EG: increase the number and the wait between retries).

relm923 · 2022-05-05T01:19:56Z

All looks pretty good, after some minor changes I'm happy to merge this. I want everyone to know not concerned about this pull request itself: the code looks good and the feature makes sense.

I'm concerned about the inevitable follow-up requests this will bring:

Please make it work with confirmations (likelihood: high)

Now that more requests are executing in parallel, I'm getting more throttles and my deployments are failing as a result (likelihood: high)

Please make it work with the other progress reporter (likelihood: low-medium)

@relm923 will you be around to do follow-ups if issues come up?

Happy to help with follow ups as they arise

One more thing that popped into my head: what happens if failures occur while a parallel deployment is happening? As best I can imagine, some red line will appear with an error, then a whole heap of SUCCESS will be printed as another stack that's deploying at the same time completes successfully, then we will print "Stack XYZ successfully deployed", and subsequently we'll exit with an error.

Can you tell me how confusing it will look when this happens?

Example with failure (colors stripped)

yarn run v1.22.18
warning package.json: No license field
$ cdk deploy --all --require-approval never --concurrency 3 --progress bar

✨ Synthesis time: 16.18s

⚠️ The --concurrency flag only supports --progress "events". Switching to "events".
relm-broken-1
relm-broken-1: deploying...
relm-broken-2
relm-broken-2: deploying...
relm-test-1: deploying...
relm-test-1: creating CloudFormation changeset...
relm-broken-1: creating CloudFormation changeset...
relm-broken-2: creating CloudFormation changeset...
relm-broken-2 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated
relm-broken-2 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated
relm-broken-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated
relm-broken-1 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated
relm-test-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated
relm-test-1 | 0/3 | 9:12:54 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated
relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197)
relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-test-1 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated
relm-test-1 | 0/3 | 9:13:01 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
relm-test-1 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197)
relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-broken-2 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated
relm-broken-2 | 0/3 | 9:13:02 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
relm-broken-2 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197)
relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237
new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22)
_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18)
_ Object. (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1)
_ Module._compile (node:internal/modules/cjs/loader:1103:14)
_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23)
_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
_ Object.require.extensions. [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12)
_ Module.load (node:internal/modules/cjs/loader:981:32)
_ Function.Module._load (node:internal/modules/cjs/loader:822:12)
_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12)
_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10)
_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10)
_ Object. (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3)
_ Module._compile (node:internal/modules/cjs/loader:1103:14)
_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
_ Module.load (node:internal/modules/cjs/loader:981:32)
_ Function.Module._load (node:internal/modules/cjs/loader:822:12)
_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
_ node:internal/main/run_main_module:17:47
relm-broken-1 | 0/3 | 9:13:05 PM | CREATE_FAILED | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation cancelled
new MetadataResource (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/metadata-resource.ts:22:24)
_ /Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:166:5
_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:231:5)
_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:227:5)
_ injectMetadataResources (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:157:3)
_ Object.synthesize (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:18:3)
_ App.synth (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/stage.ts:180:23)
_ process. (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/app.ts:131:45)
_ Object.onceWrapper (node:events:646:26)
_ process.emit (node:events:526:28)
_ process.emit (node:domain:475:12)
_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/relm/Development/aws-cdk-test-stack/node_modules/@cspotcode/source-map-support/source-map-support.js:613:40)
relm-broken-1 | 0/3 | 9:13:06 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 The following resource(s) failed to create: [TestTopic339EC197, CDKMetadata]. Rollback requested by user.
relm-broken-1 | 1/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
relm-broken-1 | 2/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197)
relm-broken-1 | 3/3 | 9:13:11 PM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | relm-broken-1
relm-broken-2 | 2/3 | 9:13:11 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197)
relm-broken-2 | 3/3 | 9:13:12 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-broken-2
relm-test-1 | 2/3 | 9:13:10 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197)

Failed resources:
relm-broken-1 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237
new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22)
_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18)
_ Object. (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1)
_ Module._compile (node:internal/modules/cjs/loader:1103:14)
_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23)
_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
_ Object.require.extensions. [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12)
_ Module.load (node:internal/modules/cjs/loader:981:32)
_ Function.Module._load (node:internal/modules/cjs/loader:822:12)
_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12)
_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10)
_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10)
_ Object. (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3)
_ Module._compile (node:internal/modules/cjs/loader:1103:14)
_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
_ Module.load (node:internal/modules/cjs/loader:981:32)
_ Function.Module._load (node:internal/modules/cjs/loader:822:12)
_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
_ node:internal/main/run_main_module:17:47

❌ relm-broken-1 failed: Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26)
at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24)
at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13
at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29)

/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307
throw new Error(The stack named ${stackName} failed creation, it may need to be manually deleted from the AWS console: ${status});
^
Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26)
at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24)
at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13
at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The failures are still reported here. In flight stacks will continue while remaining stacks are not updated

rittneje · 2022-05-05T01:32:15Z

@relm923 Just curious, do your changes support specifying concurrency within cdk.json, or only via a command line flag?

relm923 · 2022-05-05T02:06:05Z

@relm923 Just curious, do your changes support specifying concurrency within cdk.json, or only via a command line flag?

Only via CLI currently. Happy to (or have someone else) add settings/cdk.json support in a follow up PR

rix0rrr · 2022-05-05T07:39:33Z

Happy to help with follow ups as they arise

Thanks @relm923, that's what I needed to hear 😋. It's going in.

mergify · 2022-05-05T07:40:05Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-05-05T08:24:58Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: 8f1d06d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-05-05T08:25:12Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

This reverts commit 45a2d63.

rix0rrr · 2022-05-05T14:23:05Z

Unfortunately we've had to revert this: #20230

It breaks the integ tests (errors are in the other PR). Please run them, maybe piping the output into less to emulate the situation where we don't have a TTY.

relm923 · 2022-05-05T14:49:31Z

Unfortunately we've had to revert this: #20230

It breaks the integ tests (errors are in the other PR). Please run them, maybe piping the output into less to emulate the situation where we don't have a TTY.

Will take a look soon. I thought all the integration tests run on PRs automatically?

Reverts #19378 It breaks 4 integ tests: ``` ● security related changes without a CLI are expected to fail ... (node:7140) UnhandledPromiseRejectionWarning: Error: \"--require-approval\" is enabled and stack includes security-sensitive updates, but terminal (TTY) is not attached so we are unable to get a confirmation from the user ● update to stack in ROLLBACK_COMPLETE state will delete stack and create a new one expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: "" at expect (../node_modules/expect/build/index.js:128:15) at expect (cli.integtest.ts:267:9) at block (../helpers/cdk.ts:131:13) at runMicrotasks (<anonymous>) at ResourcePool.using (../helpers/resource-pool.ts:44:14) at ../helpers/test-helpers.ts:36:14 ● stack in UPDATE_ROLLBACK_COMPLETE state can be updated expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: "" at expect (../node_modules/expect/build/index.js:128:15) at expect (cli.integtest.ts:320:9) at runMicrotasks (<anonymous>) at ../helpers/cdk.ts:131:7 at ResourcePool.using (../helpers/resource-pool.ts:44:14) at ../helpers/test-helpers.ts:36:14 ● failed deploy does not hang ... (node:8336) UnhandledPromiseRejectionWarning: Error: The stack named cdktest-0m9jeq4rpbp-failed failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Property PolicyDocument cannot be empty. ``` Something around the handling of errors in the new async deployment.

rix0rrr · 2022-05-06T08:15:57Z

Unfortunately the term “integ test” is overloaded. We have:

cloudFormation snapshot tests
Tests that actually exercise the CLI on every commit

the second set is not run automatically in PRs since that has security implications: one needs to run them by hand

relm923 · 2022-05-06T12:39:26Z

@rix0rrr unless I'm doing something wrong (likely) test/integ/run-against-repo test/integ/cli/test.sh does not appear to run any of the tests in cli.integtest. The only way I can run those test is via:

yarn jest --testMatch '**/integ/cli/cli.integtest.js'

I've tried running with and without piping to less and all the tests pass for me. Any suggestions for how to reproduce the failures?

Reverts aws#19378 It breaks 4 integ tests: ``` ● security related changes without a CLI are expected to fail ... (node:7140) UnhandledPromiseRejectionWarning: Error: \"--require-approval\" is enabled and stack includes security-sensitive updates, but terminal (TTY) is not attached so we are unable to get a confirmation from the user ● update to stack in ROLLBACK_COMPLETE state will delete stack and create a new one expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: "" at expect (../node_modules/expect/build/index.js:128:15) at expect (cli.integtest.ts:267:9) at block (../helpers/cdk.ts:131:13) at runMicrotasks (<anonymous>) at ResourcePool.using (../helpers/resource-pool.ts:44:14) at ../helpers/test-helpers.ts:36:14 ● stack in UPDATE_ROLLBACK_COMPLETE state can be updated expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: "" at expect (../node_modules/expect/build/index.js:128:15) at expect (cli.integtest.ts:320:9) at runMicrotasks (<anonymous>) at ../helpers/cdk.ts:131:7 at ResourcePool.using (../helpers/resource-pool.ts:44:14) at ../helpers/test-helpers.ts:36:14 ● failed deploy does not hang ... (node:8336) UnhandledPromiseRejectionWarning: Error: The stack named cdktest-0m9jeq4rpbp-failed failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Property PolicyDocument cannot be empty. ``` Something around the handling of errors in the new async deployment.

OperationalFallacy · 2022-06-04T12:07:36Z

@rix0rrr

the second set is not run automatically in PRs since that has security implications: one needs to run them by hand

Isn't it the other way around? The manual runs have security implications.

This is a good feature, yet delayed by some obscure run by hand requirements. Do you have a process to improve this?

Closes #1973 (reattempting #19378) Add `--concurrency` parameter to `cdk deploy` command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with `--progress events` due to the poor interaction of concurrent deployments and the progress bar rendering. ### Open Questions - [x] How best to write automated tests around this? Added unit and integration tests - [x] Should other commands (ex: `destroy`) support concurrency? Only supporting deploy command in this PR - [x] Any other concerns with this approach as it changes a key component of `cdk` - [x] How should this work with the `--exclusively` flag? Only check dependencies between requested stacks <details> <summary>Example Output (Success):</summary> <pre> $ yarn cdk deploy --all --require-approval "never" --concurrency 3 --progress bar yarn run v1.22.17 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.04s ⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-test-1 relm-test-1: deploying... relm-test-2: deploying... relm-test-2: creating CloudFormation changeset... relm-test-1: creating CloudFormation changeset... relm-test-1 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-1 | 0/3 | 4:57:32 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 4:57:33 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-2 | 0/3 | 4:57:31 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2 | 1/3 | 4:57:32 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2 relm-test-1 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-1 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-1 ✅ relm-test-1 ✨ Deployment time: 28.39s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-1/ffcdb210-a6fd-11ec-b05c-0a02909bf4fb ✨ Total time: 44.43s ✅ relm-test-2 ✨ Deployment time: 28.45s Outputs: relm-test-2.ExportsOutputRefTempTopic9C0CBD7CC97C6BE5 = arn:aws:sns:us-east-1:XXXXXXXXXXXX:relm-test-2-TempTopic9C0CBD7C-2ETJWELY3MFD Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2/ffbc7400-a6fd-11ec-ac95-0ea201c9d581 ✨ Total time: 44.49s relm-test-2B relm-test-2B: deploying... relm-test-2B: creating CloudFormation changeset... relm-test-2B | 0/4 | 4:57:47 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:57:58 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 0/4 | 4:58:05 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) Resource creation Initiated relm-test-2B | 0/4 | 4:58:06 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2B | 1/4 | 4:58:07 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 2/4 | 4:58:19 PM | CREATE_COMPLETE | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 2/4 | 4:58:22 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 2/4 | 4:58:23 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) Resource creation Initiated relm-test-2B | 3/4 | 4:58:36 PM | CREATE_COMPLETE | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 4/4 | 4:58:37 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2B ✅ relm-test-2B ✨ Deployment time: 54.71s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2B/10b341d0-a6fe-11ec-a535-12d49b16bf9b ✨ Total time: 70.75s ✨ Done in 99.79s. </pre> </details> <details> <summary>Example Output (Failure):</summary> <pre> yarn run v1.22.18 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.18s ⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-broken-1 relm-broken-1: deploying... relm-broken-2 relm-broken-2: deploying... relm-test-1: deploying... relm-test-1: creating CloudFormation changeset... relm-broken-1: creating CloudFormation changeset... relm-broken-2: creating CloudFormation changeset... relm-broken-2 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-2 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-broken-1 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-test-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:54 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-test-1 | 0/3 | 9:13:01 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-broken-2 | 0/3 | 9:13:02 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-broken-2 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 relm-broken-1 | 0/3 | 9:13:05 PM | CREATE_FAILED | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation cancelled new MetadataResource (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/metadata-resource.ts:22:24) \_ /Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:166:5 \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:231:5) \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:227:5) \_ injectMetadataResources (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:157:3) \_ Object.synthesize (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:18:3) \_ App.synth (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/stage.ts:180:23) \_ process.<anonymous> (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/app.ts:131:45) \_ Object.onceWrapper (node:events:646:26) \_ process.emit (node:events:526:28) \_ process.emit (node:domain:475:12) \_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/relm/Development/aws-cdk-test-stack/node_modules/@cspotcode/source-map-support/source-map-support.js:613:40) relm-broken-1 | 0/3 | 9:13:06 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 The following resource(s) failed to create: [TestTopic339EC197, CDKMetadata]. Rollback requested by user. relm-broken-1 | 1/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 2/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 3/3 | 9:13:11 PM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | relm-broken-1 relm-broken-2 | 2/3 | 9:13:11 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 3/3 | 9:13:12 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-broken-2 relm-test-1 | 2/3 | 9:13:10 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Failed resources: relm-broken-1 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 ❌ relm-broken-1 failed: Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307 throw new Error(`The stack named ${stackName} failed creation, it may need to be manually deleted from the AWS console: ${status}`); ^ Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. </pre> </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

Closes aws#1973 (reattempting aws#19378) Add `--concurrency` parameter to `cdk deploy` command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with `--progress events` due to the poor interaction of concurrent deployments and the progress bar rendering. ### Open Questions - [x] How best to write automated tests around this? Added unit and integration tests - [x] Should other commands (ex: `destroy`) support concurrency? Only supporting deploy command in this PR - [x] Any other concerns with this approach as it changes a key component of `cdk` - [x] How should this work with the `--exclusively` flag? Only check dependencies between requested stacks <details> <summary>Example Output (Success):</summary> <pre> $ yarn cdk deploy --all --require-approval "never" --concurrency 3 --progress bar yarn run v1.22.17 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.04s ⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-test-1 relm-test-1: deploying... relm-test-2: deploying... relm-test-2: creating CloudFormation changeset... relm-test-1: creating CloudFormation changeset... relm-test-1 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-1 | 0/3 | 4:57:32 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 4:57:33 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-2 | 0/3 | 4:57:31 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2 | 1/3 | 4:57:32 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2 relm-test-1 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-1 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-1 ✅ relm-test-1 ✨ Deployment time: 28.39s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-1/ffcdb210-a6fd-11ec-b05c-0a02909bf4fb ✨ Total time: 44.43s ✅ relm-test-2 ✨ Deployment time: 28.45s Outputs: relm-test-2.ExportsOutputRefTempTopic9C0CBD7CC97C6BE5 = arn:aws:sns:us-east-1:XXXXXXXXXXXX:relm-test-2-TempTopic9C0CBD7C-2ETJWELY3MFD Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2/ffbc7400-a6fd-11ec-ac95-0ea201c9d581 ✨ Total time: 44.49s relm-test-2B relm-test-2B: deploying... relm-test-2B: creating CloudFormation changeset... relm-test-2B | 0/4 | 4:57:47 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:57:58 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 0/4 | 4:58:05 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) Resource creation Initiated relm-test-2B | 0/4 | 4:58:06 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2B | 1/4 | 4:58:07 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 2/4 | 4:58:19 PM | CREATE_COMPLETE | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 2/4 | 4:58:22 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 2/4 | 4:58:23 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) Resource creation Initiated relm-test-2B | 3/4 | 4:58:36 PM | CREATE_COMPLETE | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 4/4 | 4:58:37 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2B ✅ relm-test-2B ✨ Deployment time: 54.71s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2B/10b341d0-a6fe-11ec-a535-12d49b16bf9b ✨ Total time: 70.75s ✨ Done in 99.79s. </pre> </details> <details> <summary>Example Output (Failure):</summary> <pre> yarn run v1.22.18 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.18s ⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-broken-1 relm-broken-1: deploying... relm-broken-2 relm-broken-2: deploying... relm-test-1: deploying... relm-test-1: creating CloudFormation changeset... relm-broken-1: creating CloudFormation changeset... relm-broken-2: creating CloudFormation changeset... relm-broken-2 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-2 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-broken-1 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-test-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:54 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-test-1 | 0/3 | 9:13:01 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-broken-2 | 0/3 | 9:13:02 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-broken-2 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 relm-broken-1 | 0/3 | 9:13:05 PM | CREATE_FAILED | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation cancelled new MetadataResource (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/metadata-resource.ts:22:24) \_ /Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:166:5 \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:231:5) \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:227:5) \_ injectMetadataResources (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:157:3) \_ Object.synthesize (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:18:3) \_ App.synth (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/stage.ts:180:23) \_ process.<anonymous> (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/app.ts:131:45) \_ Object.onceWrapper (node:events:646:26) \_ process.emit (node:events:526:28) \_ process.emit (node:domain:475:12) \_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/relm/Development/aws-cdk-test-stack/node_modules/@cspotcode/source-map-support/source-map-support.js:613:40) relm-broken-1 | 0/3 | 9:13:06 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 The following resource(s) failed to create: [TestTopic339EC197, CDKMetadata]. Rollback requested by user. relm-broken-1 | 1/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 2/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 3/3 | 9:13:11 PM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | relm-broken-1 relm-broken-2 | 2/3 | 9:13:11 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 3/3 | 9:13:12 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-broken-2 relm-test-1 | 2/3 | 9:13:10 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Failed resources: relm-broken-1 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 ❌ relm-broken-1 failed: Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307 throw new Error(`The stack named ${stackName} failed creation, it may need to be manually deleted from the AWS console: ${status}`); ^ Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. </pre> </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

raymondKelly · 2023-11-22T02:19:03Z

@relm923 Out of interest does concurrency work down to the nested stack level or only the top level stack?

relm923 commented Mar 13, 2022

View reviewed changes

relm923 mentioned this pull request Mar 14, 2022

Deploy stacks in parallel where possible #1973

Closed

rix0rrr previously requested changes Mar 17, 2022

View reviewed changes

relm923 commented Mar 17, 2022

View reviewed changes

github-actions bot added the @aws-cdk/aws-cloudformation Related to AWS CloudFormation label Mar 18, 2022

github-actions bot assigned rix0rrr Mar 18, 2022

relm923 added 13 commits March 29, 2022 09:39

feat(aws-cdk): concurrent deployment support

e4839f8

feat(aws-cdk): respect stack dependencies

16a9bf2

chore(aws-cdk): add cli param

6f71f93

fix(aws-cdk): p-queue build error

be150d0

chore(aws-cdk): readme docs

c21b56c

fix(aws-cdk): downgrade p-queue to avoid esm

73f7048

fix(aws-cdk): ensure promises resolve

e0fb2f2

chore(aws-cdk): remove todo

eb82e90

chore(aws-cdk): throw error when concurrency used with require-approval

bbc7c84

chore(aws-cdk): restore total deploy time log

49a9c26

test(aws-cdk): integration tests for concurrent deploys

704021a

fix(aws-cdk): concurrency requires progress = events

c6e2bdf

chore(aws-cdk): update 3rd party licenses

443e24a

relm923 force-pushed the relm/concurrent-deploys branch from d10905e to 443e24a Compare March 29, 2022 13:45

chore(aws-cdk): third party license update

902e278

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 labels Mar 29, 2022

rix0rrr reviewed May 4, 2022

View reviewed changes

chore(aws-cdk): improve concurrency performance

e154879

rix0rrr changed the title ~~feat(aws-cdk): concurrent deployment support~~ feat(cli): --concurrency option May 5, 2022

rix0rrr approved these changes May 5, 2022

View reviewed changes

Merge branch 'master' into relm/concurrent-deploys

8f1d06d

mergify bot merged commit 45a2d63 into aws:master May 5, 2022

rix0rrr added a commit that referenced this pull request May 5, 2022

Revert "feat(cli): --concurrency option (#19378)"

722e508

This reverts commit 45a2d63.

rix0rrr mentioned this pull request May 5, 2022

chore(cli): revert concurrency PR #20230

Merged

relm923 mentioned this pull request May 14, 2022

feat(cli): --concurrency option #20345

Merged

4 tasks

mrgrain mentioned this pull request Aug 11, 2022

fix(cli): build assets before deploying any stacks #21513

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): `--concurrency` option #19378

feat(cli): `--concurrency` option #19378

relm923 commented Mar 13, 2022 •

edited

Loading

gitpod-io bot commented Mar 13, 2022

relm923 Mar 13, 2022

relm923 commented Mar 17, 2022

rix0rrr left a comment

relm923 commented Mar 17, 2022

relm923 Mar 17, 2022

rix0rrr May 4, 2022

relm923 commented Mar 18, 2022

relm923 commented Mar 23, 2022

laimonassutkus commented May 3, 2022

rix0rrr left a comment

rix0rrr May 4, 2022

relm923 May 5, 2022

rix0rrr May 4, 2022

relm923 May 5, 2022

rix0rrr May 4, 2022

polothy commented May 4, 2022

relm923 commented May 5, 2022

rittneje commented May 5, 2022

relm923 commented May 5, 2022

rix0rrr commented May 5, 2022

mergify bot commented May 5, 2022

aws-cdk-automation commented May 5, 2022

mergify bot commented May 5, 2022

rix0rrr commented May 5, 2022

relm923 commented May 5, 2022

rix0rrr commented May 6, 2022

relm923 commented May 6, 2022

OperationalFallacy commented Jun 4, 2022

raymondKelly commented Nov 22, 2023

feat(cli): --concurrency option #19378

feat(cli): --concurrency option #19378

Conversation

relm923 commented Mar 13, 2022 • edited Loading

Open Questions

gitpod-io bot commented Mar 13, 2022

Choose a reason for hiding this comment

relm923 commented Mar 17, 2022

rix0rrr left a comment

Choose a reason for hiding this comment

relm923 commented Mar 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

relm923 commented Mar 18, 2022

relm923 commented Mar 23, 2022

laimonassutkus commented May 3, 2022

rix0rrr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

polothy commented May 4, 2022

relm923 commented May 5, 2022

rittneje commented May 5, 2022

relm923 commented May 5, 2022

rix0rrr commented May 5, 2022

mergify bot commented May 5, 2022

aws-cdk-automation commented May 5, 2022

AWS CodeBuild CI Report

mergify bot commented May 5, 2022

rix0rrr commented May 5, 2022

relm923 commented May 5, 2022

rix0rrr commented May 6, 2022

relm923 commented May 6, 2022

OperationalFallacy commented Jun 4, 2022

raymondKelly commented Nov 22, 2023

feat(cli): `--concurrency` option #19378

feat(cli): `--concurrency` option #19378

relm923 commented Mar 13, 2022 •

edited

Loading