aws-s3-assets : Upgrading from LegacyStackSynthesizer to DefaultStackSynthesizer with cross-stack reference causes UpdateFailed

[robertluttrell]

Describe the bug

When upgrading from the LegacyStackSynthesizer to the DefaultStackSynthesizer, cross-stack asset dependencies cause deployment to fail with the following exception:

Export ProducingStack:ExportsOutputRefAssetParameters6547f4c9757e3c9a9d37d7a5d75004ce634cb9b01af7a7606412c36b947f248fS3BucketA01FB337B366B8FA cannot be deleted as it is in use by ConsumingStack

The issue of redeployment failing after removing generic cross-stack dependencies is described in #3414. In this specific case, CDK's legacy synthesizer creates an AssetParameters export in the producing stack so that it can be used in the consuming stack. However, the DefaultStackSynthesizer does not create this export. The result is that when upgrading from the DefaultStackSynthesizer to the LegacyStackSynthesizer, the deployment fails because the producing stack's export cannot be deleted while referenced by the consuming stack.

Expected Behavior

Deployment succeeds when updating an app with cross-stack asset dependencies from the LegacyStackSynthesizer to the DefaultStackSynthesizer.

Current Behavior

Deployment fails to update the producing stack

Reproduction Steps

Define stack classes as follows:

import os

from constructs import Construct

from aws_cdk import (
    App,
    Stack,
    CfnOutput,
    Fn,
    LegacyStackSynthesizer,
    StackProps
)

from aws_cdk.aws_s3 import (
    Bucket,
)

from aws_cdk.aws_s3_assets import (
    Asset,
)

from aws_cdk.aws_lambda import (
    Function,
    Code,
    Runtime
)

class ProducingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = Bucket(self, "My Bucket")
        self.bucket = bucket

        dirname = os.path.dirname(__file__)
        asset = Asset(self, "MyImageAsset", path=os.path.join(dirname, "asset.png"))
        self.asset=asset

class ConsumingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, bucket, asset,  **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        lambda_func = Function( self, "My Function", code=Code.from_inline("handler = lambda event, context : {}"), handler="index.handler", runtime=Runtime.PYTHON_3_7)
        bucket.grant_read_write(lambda_func)

        asset.grant_read(lambda_func)

app = App()

Deploy with stacks defined using LegacyStackSynthesizer:

producing_stack = ProducingStack(app, "ProducingStack", synthesizer=LegacyStackSynthesizer())
ConsumingStack(app, "ConsumingStack", producing_stack.bucket, producing_stack.asset, synthesizer=LegacyStackSynthesizer())
app.synth()

Deploy with stacks defined without synthesizer specified, defaulting to DefaultStackSynthesizer:

producing_stack = ProducingStack(app, "ProducingStack")
ConsumingStack(app, "ConsumingStack", producing_stack.bucket, producing_stack.asset)
app.synth()

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.61.0

Framework Version

No response

Node.js Version

v19.4.0

OS

Same behavior on MacOS and AL2

Language

Python

Language Version

3.10.9

Other information

No response

[khushail]

Hi @robertluttrell , Thanks for reaching out. I tried to reproduce the issue as stated but it succeeded when switching from LegalStackSynthesizer to DefaultStackSynthesizer.

Could you please share the code for repro? Thanks

[robertluttrell]

Hi @khushail, thanks for your response. I'm not sure I understand - did the code I included not reproduce this issue for you? I was able to reproduce it across several different environments. I wasn't totally clear in the way I structured the code in the repro steps - I broke out just the sample of code that should change between deployments, but that may have added to the confusion. Here are the steps a little more clearly:

1. Define the app as follows:

import os

from constructs import Construct

from aws_cdk import (
    App,
    Stack,
    CfnOutput,
    Fn,
    LegacyStackSynthesizer,
    StackProps
)

from aws_cdk.aws_s3 import (
    Bucket,
)

from aws_cdk.aws_s3_assets import (
    Asset,
)

from aws_cdk.aws_lambda import (
    Function,
    Code,
    Runtime
)

class ProducingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = Bucket(self, "My Bucket")
        self.bucket = bucket

        dirname = os.path.dirname(__file__)
        asset = Asset(self, "MyImageAsset", path=os.path.join(dirname, "asset.png"))
        self.asset=asset

class ConsumingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, bucket, asset,  **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        lambda_func = Function( self, "My Function", code=Code.from_inline("handler = lambda event, context : {}"), handler="index.handler", runtime=Runtime.PYTHON_3_7)
        bucket.grant_read_write(lambda_func)

        asset.grant_read(lambda_func)

app = App()

producing_stack = ProducingStack(app, "ProducingStack", synthesizer=LegacyStackSynthesizer())
ConsumingStack(app, "ConsumingStack", producing_stack.bucket, producing_stack.asset, synthesizer=LegacyStackSynthesizer())


app.synth()

2. cdk deploy. Note: CDK will throw an exception unless a file asset.png exists in the working directory

3. Change the app so the legacy stack synthesizers are not passed to the stack constructors. It should look like this:

import os

from constructs import Construct

from aws_cdk import (
    App,
    Stack,
    CfnOutput,
    Fn,
    LegacyStackSynthesizer,
    StackProps
)

from aws_cdk.aws_s3 import (
    Bucket,
)

from aws_cdk.aws_s3_assets import (
    Asset,
)

from aws_cdk.aws_lambda import (
    Function,
    Code,
    Runtime
)

class ProducingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = Bucket(self, "My Bucket")
        self.bucket = bucket

        dirname = os.path.dirname(__file__)
        asset = Asset(self, "MyImageAsset", path=os.path.join(dirname, "asset.png"))
        self.asset=asset

class ConsumingStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, bucket, asset,  **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        lambda_func = Function( self, "My Function", code=Code.from_inline("handler = lambda event, context : {}"), handler="index.handler", runtime=Runtime.PYTHON_3_7)
        bucket.grant_read_write(lambda_func)

        asset.grant_read(lambda_func)

app = App()

producing_stack = ProducingStack(app, "ProducingStack")
ConsumingStack(app, "ConsumingStack", producing_stack.bucket, producing_stack.asset)

app.synth()

4. cdk deploy Note: Doing a cdk destroy between steps 2 and 4 allows this step to succeed (and not reproduce the issue), but our use case prevents us from being able to destroy our customers' stacks.

Are you able to reproduce with these steps? Thanks

[khushail]

Thanks @robertluttrell for sharing and explaining the code steps. I was able to reproduce the issue.

I have marked this issue as p2, which means that we are unable to work on this immediately.
We use +1s to help prioritize our work, and are happy to re-evaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.