❗ NOTICE (custom-resources): attribute error - vendor response doesn't contain endpointAddress

[grant-d]

Please add your +1 👍 to let us know you have encountered this

---

Status: In progress

Describe the bug

If I upgrade cdk & cdk-lib to 2.138.0 from 2.137.0, the following error occurs at deployment.
If I rollback, the error goes away.

CustomResource attribute error: Vendor response doesn't contain endpointAddress attribute in object ...|...IoTEndpoint

This was previously working for quite some time.

The specific resource is an AwsCustomResource related to IoT.
(I could not find a way to get the endpoint using aws-cdk-lib/aws-iot, so had to use a custom resource)

Code repro below.

Expected Behavior

Deployment succeeds

Current Behavior

Deployment fails with the following error:

8:34:21 AM | UPDATE_FAILED        | AWS::SSM::Parameter             | fooservice/Stage2...IotEndpointAddress
CustomResource attribute error: Vendor response doesn't contain endpointAddress attribute in object
arn:aws:cloudformation:eu-central-1:<account>:stack/<...>|...IoTEndpoint...

Reproduction Steps

    import * as cus from 'aws-cdk-lib/custom-resources'

    // https://stackoverflow.com/a/77092236
    const getIoTEndpoint = new cus.AwsCustomResource(this, 'IoTEndpoint', {
      onCreate: {
        service: 'Iot',
        action: 'describeEndpoint',
        // I am guessing the failure is on the following line (and/or below)
        physicalResourceId: cus.PhysicalResourceId.fromResponse('endpointAddress'),
        parameters: {
          endpointType: 'iot:Data-ATS'
        }
      },
      policy: cus.AwsCustomResourcePolicy.fromSdkCalls({ resources: cus.AwsCustomResourcePolicy.ANY_RESOURCE })
    })

    // I am guessing the failure is on the following line (and/or above)
    const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress') // eg, abc...xyz-ats.iot.us-west-2.amazonaws.com

Possible Solution

Rollback cdk-lib to 2.137.0

Additional Information/Context

No response

CDK CLI Version

2.119.0 (build 0392e71)

Framework Version

2.138.0

Node.js Version

v20.7.0

OS

MacOS

Language

TypeScript

Language Version

5.4.5

Other information

No response

[grant-d]

Related (I think): #25283
Though, once again, this issue only shows up in 2.138.0 and goes away in 2.137.0

Note that in both versions, the generated CFTs seem to look exactly the same.
And - per linked issue - neither has an Update property:

{
   "Type": "Custom::AWS",
   "Properties": {
    "ServiceToken": {
     "Fn::GetAtt": [
      "<redacted>",
      "Arn"
     ]
    },
    "Create": "{\"service\":\"Iot\",\"action\":\"describeEndpoint\",\"physicalResourceId\":{\"responsePath\":\"endpointAddress\"},\"parameters\":{\"endpointType\":\"iot:Data-ATS\"},\"logApiResponseData\":true}",
    "InstallLatestAwsSdk": false
   },
   "DependsOn": [
    "<redacted>"
   ],
   "UpdateReplacePolicy": "Delete",
   "DeletionPolicy": "Delete",
   "Metadata": {
    "aws:cdk:path": "<redacted>/IoTEndpoint/Resource/Default"
   }
  }

[pahud]

Looks like your response does not contain the endpointAddress.

const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress') 

Are you able to check the cloudwatch logs for the custom resource and see the full response object of that API call? I believe the full response object should be in the log.

[pahud]

And, can you check if this issue exists in 2.139.0 as it fixed some custom resource issues.

[grant-d]

Thanks @pahud . I deployed 2.139.0 and it also fails. Once again, rolling back to 137 works.
I will check logs and get back to you

[markusl]

@pahud I can also confirm that there is a breaking change in AwsCustomResource after 2.137.0.

export const getGroupId = (scope: Construct, identityStoreId: string, group: string, logGroup: logs.LogGroup) =>
  new cr.AwsCustomResource(scope, `getGroupId-${group}`, {
    onCreate: {
      service: 'IdentityStore',
      action: 'getGroupId',
      parameters: {
        IdentityStoreId: identityStoreId,
        AlternateIdentifier: {
          UniqueAttribute: {
            AttributePath: 'DisplayName',
            AttributeValue: group,
          },
        },
      },
      physicalResourceId: cr.PhysicalResourceId.fromResponse('GroupId'),
    },
    policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
      resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
    }),
    logGroup,
  });

The error message is CustomResource attribute error: Vendor response doesn't contain GroupId attribute in object arn:aws:cloudformation:us-east-8:999999999999:stack/stack-name. The GroupId does exist and is returned from the API when doing the API call.

Can you please have a look at this since it is quite critical and would block us from deploying AWS SSO Assignments?

Regards,
Markus

[markusl]

Is this related to the issue now that a new attribute logApiResponseData has been added?

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/cluster-resource.ts

Line 95 in c949076

// resource. Otherwise, CloudFormation will error with "Vendor response

[pahud]

Bumping this to p1 and I have escalated this to the team.

[colifran]

@markusl that would only be for the EKS cluster-resource. AwsCustomResource has its own custom resource it creates in its constructor:

aws-cdk/packages/aws-cdk-lib/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts

Line 520 in fd3a5e3

this.customResource = new cdk.CustomResource(this, 'Resource', {

[pahud]

Hi @grant-d

I can deploy the following code in 2.138.0 and 2.139.0 with exactly the same output as below:

Outputs:
dummy-stack2.EndpointOutput = a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const getIoTEndpoint = new cr.AwsCustomResource(this, 'IoTEndpoint', {
      onCreate: {
        service: 'Iot',
        action: 'describeEndpoint',
        physicalResourceId: cr.PhysicalResourceId.fromResponse('endpointAddress'),
        parameters: {
          endpointType: 'iot:Data-ATS'
        }
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({ resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE })
    })

    const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress')
    new CfnOutput(this, 'EndpointOutput', { value: IOT_ENDPOINT });
  }
}

I can't reproduce this issue on my end.

This is my CloudWatch Logs

[
    "INIT_START Runtime Version: nodejs:18.v28\tRuntime Version ARN: arn:aws:lambda:us-east-1::runtime:b475b23763329123d9e6f79f51886d0e1054f727f5b90ec945fcb2a3ec09afdd\n",
    "START RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051 Version: $LATEST\n",
    "2024-04-30T15:43:21.937Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\t{\"RequestType\":\"Create\",\"ServiceToken\":\"arn:aws:lambda:us-east-1:deducted:function:dummy-stack2-AWS679f53fac002430cb0da5b7982bd22872D-I2hig52uIdkt\",\"ResponseURL\":\"...\",\"StackId\":\"arn:aws:cloudformation:us-east-1:deducted:stack/dummy-stack2/435fcf60-0708-11ef-85cf-0ee587ffad51\",\"RequestId\":\"36187d1d-4add-4601-9764-b850ee234636\",\"LogicalResourceId\":\"IoTEndpoint9F0B923E\",\"ResourceType\":\"Custom::AWS\",\"ResourceProperties\":{\"ServiceToken\":\"arn:aws:lambda:us-east-1:deducted:function:dummy-stack2-AWS679f53fac002430cb0da5b7982bd22872D-I2hig52uIdkt\",\"InstallLatestAwsSdk\":\"false\",\"Create\":{\"service\":\"Iot\",\"action\":\"describeEndpoint\",\"physicalResourceId\":{\"responsePath\":\"endpointAddress\"},\"parameters\":{\"endpointType\":\"iot:Data-ATS\"},\"logApiResponseData\":true}}}\n",
    "2024-04-30T15:43:23.658Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\tAPI response { endpointAddress: 'a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com' }\n",
    "2024-04-30T15:43:23.696Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\tResponding {\"Status\":\"SUCCESS\",\"Reason\":\"OK\",\"PhysicalResourceId\":\"a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com\",\"StackId\":\"arn:aws:cloudformation:us-east-1:deducted:stack/dummy-stack2/435fcf60-0708-11ef-85cf-0ee587ffad51\",\"RequestId\":\"36187d1d-4add-4601-9764-b850ee234636\",\"LogicalResourceId\":\"IoTEndpoint9F0B923E\",\"NoEcho\":false,\"Data\":{\"region\":\"us-east-1\",\"endpointAddress\":\"a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com\"}}\n",
    "END RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051\n",
    "REPORT RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tDuration: 5154.74 ms\tBilled Duration: 5155 ms\tMemory Size: 128 MB\tMax Memory Used: 97 MB\tInit Duration: 181.34 ms\t\n"
]

Can you try my code snippets above and see if it works for you?

[pahud]

Looking at your provided code, I would not recommend specifying physicalResourceId like this because in this case your physicalResourceId would be subject to change from the response you receive and things may break unexpectedly if the SDK response does not contain that field in some edge cases.

physicalResourceId: cr.PhysicalResourceId.fromResponse('endpointAddress'),

Instead, I would recommend this statically.

physicalResourceId: cr.PhysicalResourceId.of('IoTEndpoint'),

Let me know if it works for you.

[pahud]

@markusl I was not able to execute getGroupId in my account but I have similar code

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const groupId = new cr.AwsCustomResource(this, 'getGroupId', {
      onCreate: {
        service: 'IdentityStore',
        action: 'describeGroup',
        parameters: {
          IdentityStoreId,
          GroupId,
        },
        physicalResourceId: cr.PhysicalResourceId.fromResponse('GroupId'),
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
        resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
    });

    const attrGroupId =  groupId.getResponseField('GroupId');
    new CfnOutput(this, 'GroupIdOutput', { value: attrGroupId });
  }
}

And I was able to get the GroupId from its response. And this works in 2.137.0, 2.138.0 and 2.139.0.

Are you able to successfully run my snippet above?

[pahud]

We should decouple the physicalResourceId from the API response as describe here to avoid potential error:

#30012 (comment)

[markusl]

In our case, with the help of AWS support, we were able to identify the problem, which was a missing onUpdate handler, which is required when performing a CDK version upgrade due to the added property.

We are now using the same handler for onCreate and onUpdate which returns the group name properly.

// AWS Custom Resource to dynamically fetch GroupId based on group name
export const getGroupId = (scope: Construct, identityStoreId: string, group: string, logGroup: logs.LogGroup) => new cr.AwsCustomResource(scope, `getGroupId-${group}`, {
  onCreate: getGroupIdSdkCall(identityStoreId, group),
  onUpdate: getGroupIdSdkCall(identityStoreId, group),
  policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
    resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
  }),
  logGroup,
});

I'm not sure if this is a "breaking change" or not, but I would suggest highlighting this properly in the changelog that the CDK version upgrade will cause an update to all Custom Resources.

[colifran]

@markusl Thanks for the follow-up and I agree, this feels like a documentation miss on our end. From the perspective of the custom resource it saw an update and didn't make an API call which is how it was configured. However, obviously you now don't have a groupId since your custom resource was configured to only make the API call on a create event. We'll work on improving the documentation for this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.