Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

codepipeline: x-account AND x-region deployments are a mess #24051

Open
rix0rrr opened this issue Feb 7, 2023 · 1 comment
Open

codepipeline: x-account AND x-region deployments are a mess #24051

rix0rrr opened this issue Feb 7, 2023 · 1 comment
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. effort/large Large work item – several weeks of effort p2

Comments

@rix0rrr
Copy link
Contributor

rix0rrr commented Feb 7, 2023

Describe the bug

A cross-region AND cross-account deployment where CodePipeline generates the support stacks for replication buckets and the support stacks for cross-account roles (which is the ideal situation), fails to deploy out of the box.

The cross-account and cross-region resources are created in separate stacks that don't have a dependency between them, but the resources on those stacks have bidirectional dependencies on each other:

  • The Bucket and Key policies reference the Action Role
  • The Role policy references the Bucket and Key (and deployment resources like CodeDeploy Deployment Groups)

Normally this wouldn't deploy, but we generate hard-coded resource names for the resources involved so that we can formulate policies anyway without having to have bidirectional cross-stack references.


Everything we're doing in this area right now is a patchwork of ad-hoc solutions, that needs to be rethought.

The bidirectional policy references between the support stacks should be replaced with some form of tag-based access control and clean dependencies between the stacks (and not the "we put some policies in both and hope it works" style we do today).

We should have a good think on what resources go into what stacks in what cases, keeping in mind:

  • The deployment resources can be owned or referenced (what stack are they int?)
  • The action role can be owned or referenced (what stack are they in, what if the user created them?)
    • If this goes into a support stack, what region does that support stack deploy into? us-east-1, $pipeline-region or $resources-region? Why is that the right choice?
  • The replication resources can be owned or referenced (what stack are they in, what if the user created them?)

Someone needs to go look at the current situation, come up with a good design, and cover the delta.

Expected Behavior

Cross-environment deployments to work with a minimum of fuss.

Current Behavior

Hard-to-debug failures or mysterious cyclic dependencies between stacks.

@rix0rrr rix0rrr added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 7, 2023
@github-actions github-actions bot added the @aws-cdk/aws-codepipeline Related to AWS CodePipeline label Feb 7, 2023
rix0rrr added a commit that referenced this issue Feb 7, 2023
…k dependency

In the case of a cross-account AND cross-region ECS deployment, a
dependency between the support stacks that is necessary is missing.

This cannot be fixed globally. Because the resources reference each
other bidirectionally (user -> bucket, bucket -> user), the only way to
fix this is to do it locally: in the CodePipeline module, where we can
have the knowledge that we use generated names and that everything will
work out if we deploy the role before the bucket.

All CodePipeline Actions must have this fix eventually, but since
people may have crazy stack setups in which addition of this dependency
may introduce a cyclic dependency (breaking the synth), we're rolling
this fix out with limited blast radius.

Follow-up in #24050, and suggest a good clean-up in #24051.
@rix0rrr rix0rrr added the p1 label Feb 7, 2023
@peterwoodworth peterwoodworth removed the needs-triage This issue or PR still needs to be triaged. label Feb 7, 2023
mergify bot pushed a commit that referenced this issue Feb 10, 2023
…ncy (#24053)

In the case of a cross-account AND cross-region ECS deployment, a dependency between the support stacks that is necessary is missing.

This cannot be fixed globally. Because the resources reference each other bidirectionally (user -> bucket, bucket -> user), the only way to fix this is to do it locally: in the CodePipeline module, where we can have the knowledge that we use generated names and that everything will work out if we deploy the role before the bucket.

All CodePipeline Actions must have this fix eventually, but since people may have crazy stack setups in which addition of this dependency may introduce a cyclic dependency (breaking the synth), we're rolling this fix out with limited blast radius.

Follow-up in #24050, and suggest a good clean-up in #24051.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@peterwoodworth peterwoodworth added the effort/large Large work item – several weeks of effort label Jun 26, 2023
@ldecaro
Copy link

ldecaro commented Jul 30, 2023

If there's a pipeline with a cross-region action and if this action needs to transfer files from one region to the other, the cross-region-stack will be created. In my use case, this is a CodeDeploy action. It requires an action role, that CDKPipelines may generate or you can use your own role. No matter what you choose, cdk will throw an error like the one below:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.0:java (default-cli) on project service: An exception occurred while executing the Java class. Error: 'DemoToolchain' depends on 'cross-region-stack-111111111111:us-east-2' ({DemoToolchain}.addDependency({cross-region-stack-111111111111:us-east-2})). Adding this dependency (replication bucket {cross-region-stack-111111111111:us-east-2/Default/CrossRegionCodePipelineReplicationBucket} to action role {DemoToolchain/AWSCodeDeployRoleForBlueGreen-UAT}) would create a cyclic reference.

[ERROR] @jsii/kernel.RuntimeError: Error: 'DemoToolchain' depends on 'cross-region-stack-111111111111:us-east-2' ({DemoToolchain}.addDependency({cross-region-stack-111111111111:us-east-2})). Adding this dependency (replication bucket {cross-region-stack-111111111111:us-east-2/Default/CrossRegionCodePipelineReplicationBucket} to action role {DemoToolchain/AWSCodeDeployRoleForBlueGreen-UAT}) would create a cyclic reference.

This problem will not happen if the deployment is cross-account (single or cross-region). If I have a pipeline that deploys a Service using CodeDeployAction in any region other than the region where the pipeline is deployed, the error above shows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. effort/large Large work item – several weeks of effort p2
Projects
None yet
Development

No branches or pull requests

4 participants