CrossAccountZoneDelegationRecord: IAM policy propogation delay causes intermittent failures

[lognoel]

Describe the bug

The CrossAccountZoneDelegationRecord intermittently fails to create a record with the following error:

Received response status [FAILED] from custom resource. Message returned: AccessDenied: User: [ROLE CREATED BY CrossAccountZoneDelegationRecord] is not authorized to perform: sts:AssumeRole on resource: [DNS DELEGATION ROLE IN OTHER ACCOUNT].

• despite a valid trust policy on the delegation role in the target account. Retrying the stack deployment once or twice always fixes the issue. I have noticed that it is more likely to occur in regions with higher latency to the IAM global control plane in us-east-1. In ca-west-1, I estimate this failure occurs when deploying a new CrossAccountZoneDelegationRecord approximately 30% of the time.

Expected Behavior

The CrossAccountZoneDelegationRecord successfully creates a Cross Account Zone Delegation record when given a delegation role with accurate trust policies.

Current Behavior

The CrossAccountZoneDelegationRecord intermittently fails, likely due to IAM policy propagation delay.

Reproduction Steps

Deploy a new CrossAccountZoneDelegationRecord several times in a non-us AWS region.

Possible Solution

In a similar issue using custom resources (#18237), retries was one of the suggested fixes, which seems appropriate here as well.

Additional Information/Context

No response

CDK CLI Version

2.122.0

Framework Version

No response

Node.js Version

18

OS

MacOs

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Thank you for the report. This could be related to #18237 (comment) and I am making it a p1 bug.

Can you share a small code snippets with very minimal props that we can just copy/paste in our IDE and reproduce this error in ca-west-1 and us-east-1?

[lognoel]

I'm not sure what code samples you want, the resource creation is very straightforward as specified by https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_route53.CrossAccountZoneDelegationRecord.html

[samson-keung]

On the CrossAccountZoneDelegationRecord construct, there is an assumeRoleRegion option to set which region the STS AssumeRole call should go to. Here is the link to the option details in the CDK doc.

Were you able to see if setting this option to us-east-1 helps with the problem?

[pahud]

Hi @lognoel

I was able to deploy from ca-west-1 using the sample below:

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const subZone = new route53.PublicHostedZone(this, 'SubZone', {
      zoneName: 'sub.example.com',
    });
    
    // import the delegation role by constructing the roleArn
    const delegationRoleArn = Stack.of(this).formatArn({
      region: '', // IAM is global in each partition
      service: 'iam',
      account: 'XXXXXXXXXXX',
      resource: 'role',
      resourceName: 'AdminRole4Switch',
    });
    const delegationRole = iam.Role.fromRoleArn(this, 'DelegationRole', delegationRoleArn);
    
    // create the record
    new route53.CrossAccountZoneDelegationRecord(this, 'delegate', {
      delegatedZone: subZone,
      parentHostedZoneName: 'example.com', // or you can use parentHostedZoneId
      delegationRole,
      assumeRoleRegion: 'us-east-1',
    });
  }
}

Let me know if it works for you.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.