Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(route53): timeouts due to delete-existing-record-set-handler's wait time #27068

Merged
merged 5 commits into from
Sep 8, 2023

Conversation

kaizencc
Copy link
Contributor

@kaizencc kaizencc commented Sep 8, 2023

This PR updates the wait time of the delete-existing-record-set-handler from 60 to 1800 seconds.

Commit 399b6bb upgraded the handler to use the AWS SDK V3 (was V2) and made the following change:

-  await route53.waitFor('resourceRecordSetsChanged', { Id: changeResourceRecordSets.ChangeInfo.Id }).promise();
+  await waitUntilResourceRecordSetsChanged({ client: route53, maxWaitTime: 60 }, { Id: changeResourceRecordSets?.ChangeInfo?.Id })

The V2 route53.waitFor waits for 1800 seconds (see reference)

Waits for the resourceRecordSetsChanged state by periodically calling the underlying [Route53.getChange()](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Route53.html#getChange-property) operation every 30 seconds (at most 60 times).

The new explicit configuration was set to 60 seconds which is too short and caused timeouts and failure in our cloudformation stacks

Supersedes and closes #27060

Co-authored-by: Tom Roshko hi@pistazie.rocks


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

pistazie and others added 2 commits September 8, 2023 10:04
…onds to regression caused by AWS SDK V3 migration Regression introduced in 399b6bb
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@kaizencc kaizencc changed the title fix(route53): timeouts due to delete-existing-record-set-handler's wait time fix(route53): timeouts due to delete-existing-record-set-handler's wait time Sep 8, 2023
@kaizencc kaizencc changed the title fix(route53): timeouts due to delete-existing-record-set-handler's wait time fix(route53): timeouts due to delete-existing-record-set-handler's wait time Sep 8, 2023
rix0rrr
rix0rrr previously requested changes Sep 8, 2023
@@ -46,7 +46,7 @@ export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent
},
});

await waitUntilResourceRecordSetsChanged({ client: route53, maxWaitTime: 60 }, { Id: changeResourceRecordSets?.ChangeInfo?.Id });
await waitUntilResourceRecordSetsChanged({ client: route53, maxWaitTime: 1800 }, { Id: changeResourceRecordSets?.ChangeInfo?.Id });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be max 900, better yet 890 (to give Lambda 10s to send a response), and should match the defined Timeout of the Lambda function itself.

@kaizencc kaizencc changed the title fix(route53): timeouts due to delete-existing-record-set-handler's wait time fix(route53): timeouts due to delete-existing-record-set-handler's wait time Sep 8, 2023
@kaizencc kaizencc added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Sep 8, 2023
@aws-cdk-automation aws-cdk-automation dismissed their stale review September 8, 2023 17:10

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: f5eafec
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Sep 8, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit d730f7f into main Sep 8, 2023
10 checks passed
@mergify mergify bot deleted the update-delete-existing-record-set-handler-wait-time branch September 8, 2023 18:35
mikewrighton pushed a commit that referenced this pull request Sep 14, 2023
…it time (#27068)

This PR updates the wait time of the `delete-existing-record-set-handler` from 60 to 1800 seconds.

Commit 399b6bb upgraded the handler to use the AWS SDK V3 (was V2) and made the following change:

```
-  await route53.waitFor('resourceRecordSetsChanged', { Id: changeResourceRecordSets.ChangeInfo.Id }).promise();
+  await waitUntilResourceRecordSetsChanged({ client: route53, maxWaitTime: 60 }, { Id: changeResourceRecordSets?.ChangeInfo?.Id })
```

The V2 `route53.waitFor` waits for 1800 seconds (see [reference](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Route53.html#waitFor-property))
> `Waits for the resourceRecordSetsChanged state by periodically calling the underlying [Route53.getChange()](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Route53.html#getChange-property) operation every 30 seconds (at most 60 times).` 

The new explicit configuration was set to 60 seconds which is too short and caused timeouts and failure in our cloudformation stacks

Supersedes and closes #27060 

Co-authored-by: Tom Roshko [hi@pistazie.rocks](mailto:hi@pistazie.rocks)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS. p2 pr-linter/exempt-integ-test The PR linter will not require integ test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants