NodeJS SDK lambda client does not throw when reaching socketTimeout or Agent.timeout / NAT timeout

[kuhe]

Describe the bug

Discovered while testing #3624

const { NodeHttpHandler } = require("@aws-sdk/node-http-handler");
const { Agent } = require("http");

const agent = new Agent({
  keepAlive: true,
  timeout: 400 * 1000,
});

const lambdaClient = new LambdaClient({
  region: "us-east-1",
  logger: console,
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 8000,
    socketTimeout: 400 * 1000,
    httpAgent: agent,
  }),
});

When the SDK Client should time out upon breaching the node http Agent timeout setting or the NodeHttpHandler socketTimeout setting, instead nothing happens and the promise continues to be awaited.

Expected Behavior

When breaching the Agent's timeout value or the NodeHttpHandler's socketTimeout value, I would expect the SDK client to throw a timeout error to be handled by the client's caller.

Current Behavior

It may be restricted to this Lambda NAT timeout case, but the request from client.send(...) continues to be awaited.

Reproduction Steps

AWS resource stack:

• VPC (default is ok)
• private subnet with NAT gateway
• Lambda Invoker
  • place within private subnet and configure timeout to be greater than 420 seconds.
• Lambda B, returns after 360s to hit the 350s NAT gateway timeout
  • do not configure VPC, but configure timeout to be greater than 420 seconds.

Code for calling B from Invoker (modify function names as needed):

const { LambdaClient, InvokeCommand } = require("@aws-sdk/client-lambda");
const { NodeHttpHandler } = require("@aws-sdk/node-http-handler");
const { Agent } = require("http");

const agent = new Agent({
  keepAlive: true,
  // keepAliveMsecs: 999,
  // maxSockets: 100,
  // maxTotalSockets: 100,
  // maxFreeSockets: 256,
  timeout: 400 * 1000,
  // scheduling: "fifo",
});

const lambdaClient = new LambdaClient({
  region: "us-east-1",
  logger: console,
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 8000,
    socketTimeout: 400 * 1000,
    httpAgent: agent,
  }),
});

exports.handler = async function handler(event, context, callback) {
  let counter = 1;
  const timer = setInterval(() => {
    console.log(new Date(), "waiting...", counter++);
  }, 1000);

  const invokeParamsB = {
    FunctionName: "tcp-keepalive-test-B",
    Payload: "",
  };
  const commandB = new InvokeCommand(invokeParamsB);

  // should throw upon breach of timeout settings, but continues await until Lambda's own timeout.
  const dataB = await lambdaClient.send(commandB); 
  console.log(new TextDecoder("utf-8").decode(dataB.Payload));

  clearTimeout(timer);
};

Possible Solution

Race promise to throw timeout?

Additional Information/Context

This is not a solution for the timeout failing to throw, but incidental workaround to the timeout issue.

Making an SDK client request to a shorter lambda (or perhaps making any client request at all) first before calling the long-running lambda makes it succeed despite taking longer time overall.

Succeeds (!?)

  const invokeParamsA = {
    FunctionName: "tcp-keepalive-test-A",
    Payload: "",
  };
  const commandA = new InvokeCommand(invokeParamsA);

  const invokeParamsB = {
    FunctionName: "tcp-keepalive-test-B",
    Payload: "",
  };
  const commandB = new InvokeCommand(invokeParamsB);

  const dataA = await lambdaClient.send(commandA);
  console.log(new TextDecoder("utf-8").decode(dataA.Payload));

  const dataB = await lambdaClient.send(commandB);
  console.log(new TextDecoder("utf-8").decode(dataB.Payload));

SDK version used

3.110.0

Environment details (OS name and version, etc.)

AWS AL2

[ronak2121]

@kuhe is there any update on this issue? I'm seeing a similar issue with calls out to S3. I've set the socketTimeout value to 3s but I see my calls taking more than 6s and succeeding. This makes me think the socketTimeout value is not actually working

[kuhe]

No update, there is not enough information or priority available to work on this. It may be due to something specific to Lambda.

[kizaonline]

@kuhe This is effecting us in production and a ticket raised months ago with AWS Support who pointed us towards this issue, maybe due to the combination of NAT Gateway and Lambda?

[bbright024]

@kizaonline @ronak2121 and all others impacted by this bug -

there's a simple hack you can use to avoid this issue: complete a guaranteed fast request with the client, before attempting the >5 minute request.

e.g. if there's a lambda invoke that may take >5 minutes, first use the client to make an invoke request with the DryRun parameter set to true, then attempt the >5 minute invoke.

DryRun – Validate parameter values and verify that the user or role has permission to invoke the function.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html

this works because the keepalive parameters are not applied until after the first request is completed. you can see it in this code -

state.keepAliveTimeoutSet = true;

I highly recommend adding this workaround to your code - bnoordhuis mentions that this is intended behavior in node, so there's no guarantee it will ever be fixed:

The keepAliveMsecs option isn't applied until the socket is added to the connection pool, i.e., after the first request. That's by design, as far as I'm aware.
(2) is a policy change. Best way forward is to open a pull request and see how it's received.
nodejs/node#47137 (comment)

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.