Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't increase lambda timeout on custom resource lambda #3272

Closed
1 of 5 tasks
abelmokadem opened this issue Jul 10, 2019 · 8 comments · Fixed by #4623
Closed
1 of 5 tasks

Can't increase lambda timeout on custom resource lambda #3272

abelmokadem opened this issue Jul 10, 2019 · 8 comments · Fixed by #4623
Assignees
Labels
@aws-cdk/custom-resources Related to AWS CDK Custom Resources feature-request A feature should be added or improved. in-progress This issue is being actively worked on.

Comments

@abelmokadem
Copy link
Contributor

  • I'm submitting a ...

    • 🪲 bug report
    • 🚀 feature request
    • 📚 construct library gap
    • ☎️ security issue or vulnerability => Please see policy
    • ❓ support request => Please see note at the top of this template.
  • What is the current behavior?
    If the current behavior is a 🪲bug🪲: Please provide the steps to reproduce
    The current timeout for custom resource lambda is set to 3 seconds. My lambda for the custom resource is taking a bit longer than 3 seconds to complete.

  • What is the expected behavior (or behavior of feature suggested)?
    I would like to be able to configure the lambda timeout for custom resource, or see an increase in the default timeout value.

  • What is the motivation / use case for changing the behavior or adding this feature?
    To support lambdas that might run into timeout issue. If the timeout happens 3 times, which it did for me, you will end up with a stack that is stuck for an hour.

  • Please tell us about your environment:

    • CDK CLI Version: 0.37.0
    • Module Version: 0.37.0
    • OS: [OSX Mojave]
    • Language: [TypeScript]
  • Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc)

I managed to overwrite the value with:

    const userPoolDomainNameLambda = (userPoolDomainNameCustomResource.node
      .findAll(ConstructOrder.PREORDER)
      .find(construct => {
        return construct instanceof lambda.SingletonFunction;
      }) as lambda.SingletonFunction)["lambdaFunction"].node.children.find((child: any) => {
      return child instanceof lambda.CfnFunction;
    }) as lambda.CfnFunction;

    userPoolDomainNameLambda.addPropertyOverride("Timeout", "300");

Now it is working for me. The execution time of the lambda ended up being 3400ms.

Cheers!

@abelmokadem abelmokadem added the needs-triage This issue or PR still needs to be triaged. label Jul 10, 2019
@NGL321 NGL321 added @aws-cdk/core Related to core CDK functionality investigating This issue is being investigated and/or work is in progress to resolve the issue. @aws-cdk/aws-lambda Related to AWS Lambda and removed needs-triage This issue or PR still needs to be triaged. labels Jul 10, 2019
@NGL321
Copy link
Contributor

NGL321 commented Jul 10, 2019

Hi @abelmokadem!

Thank you for contributing! After reviewing the feature request, I am wondering if this is what you are looking for.
Timeout is a prop for new lambda functions, even included in custom resources.

If not, let me know, and I will add this to the FR queue.

@NGL321 NGL321 added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 10, 2019
@robertd
Copy link
Contributor

robertd commented Jul 10, 2019

@abelmokadem Just like @NGL321 suggested.... cdk switched form number to cdk.Duration recently (#2857). I had the same issue and had to update all my cdk apps.

@abelmokadem
Copy link
Contributor Author

I see that I forgot to mention what my custom resource actually is. I'm talking about this resource https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_custom-resources.AwsCustomResource.html

@jogold
Copy link
Contributor

jogold commented Jul 12, 2019

Can you detail your use case? How is it that an API call takes more than 3 seconds to complete?

@eladb eladb assigned eladb and unassigned eladb Aug 12, 2019
@eladb eladb assigned nija-at and unassigned eladb Sep 3, 2019
@SomayaB
Copy link
Contributor

SomayaB commented Oct 17, 2019

Can you detail your use case? How is it that an API call takes more than 3 seconds to complete?

@abelmokadem

@nija-at nija-at added @aws-cdk/custom-resources Related to AWS CDK Custom Resources and removed @aws-cdk/core Related to core CDK functionality @aws-cdk/aws-lambda Related to AWS Lambda labels Oct 18, 2019
@nija-at nija-at assigned eladb and unassigned nija-at Oct 18, 2019
@dveijck
Copy link

dveijck commented Oct 21, 2019

@SomayaB I hope I can help out clarifying this issue a bit.

I had the exact same situation where I needed to provide a custom Cognito UserPool Domain through CDK. I created an AwsCustomResource just like @abelmokadem to invoke the CognitoIdentityServiceProvider#createUserPoolDomain API (CloudFormation has only provided direct support for this setup somewhere in the last two weeks, so the actual example is not relevant anymore, but this issue still is!).

I also experienced a Lambda execution time-out on this AWS API call. To me it is entirely unclear why this API call takes longer than 3 seconds to complete as there is no information available (either the Lambda function just get killed because of the execution time-out and that is what you see in the logs or it just works if the execution time-out is extended long enough), but I personally see no reason why an API call shouldn't take any longer than 3 seconds.

The case @abelmokadem makes is that since AwsCustomResource relies on a Lambda function to create the custom resource, it would be nice to at least provide the option to set the time-out property for this Lambda. Regardless of the short response times AWS tries to achieve for its API's, calls for custom resources just might take longer than expected for whatever reason (as in @abelmokadem's and my situation). The workaround @abelmokadem suggests works fine (it solved my issue as well), however as you can see, it requires the users to go through quite some 'hassle' to get a hold on the CfnFunction to be able to override the execution time-out.

Hope it helps in understanding this issue.

@jogold
Copy link
Contributor

jogold commented Oct 21, 2019

I'm of course OK to expose this prop in AwsCustomResource but how about increasing the timeout in AwsCustomResource to 6 seconds to avoid "stuck stacks"?

@SomayaB SomayaB added feature-request A feature should be added or improved. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Oct 21, 2019
@dveijck
Copy link

dveijck commented Oct 21, 2019

Since the AwsCustomResource is mainly a convenient way to call AWS API where CloudFormation support is lacking, a higher execution time-out sounds like a good option as well. I don't have any insights in usual durations of the AWS API calls (or what AWS strives for), so I don't know if 6 seconds is a good default.

The only concern to me in that case however is that this issue might pop up again in rare situations and people would have a hard time finding the solution for it.

So maybe both solutions, a higher default execution time-out to cover most situations and the option to override it with a short comment to state its purpose and a hint to help the few unlucky users when they run into this?

jogold added a commit to jogold/aws-cdk that referenced this issue Oct 22, 2019
@SomayaB SomayaB added the in-progress This issue is being actively worked on. label Oct 22, 2019
@mergify mergify bot closed this as completed in #4623 Oct 23, 2019
mergify bot pushed a commit that referenced this issue Oct 23, 2019
…rce (#4623)

* fix(custom-resources): increase and expose timeout for AwsCustomResource

Default timeout is now 6 seconds.

Closes #3272

* change default to 30s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/custom-resources Related to AWS CDK Custom Resources feature-request A feature should be added or improved. in-progress This issue is being actively worked on.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants