DynamoDB and Throttling

[carlnordenfelt]

Hi,

This isn't so much an issue as a question regarding the implementation.
I wonder if and how exponential back offs are implemented in the sdk.

If I create a new dynamo object i see that maxRetries is undefined but I'm not sure exactly what that implies.
var AWS = require('aws'-sdk');
var dynamo = new AWS:DynamoDB();
console.log(dynamo);

When we get throttled on occasion I see that it takes a lot longer for our callback to be called, sometime up to 25 seconds. I haven't had the possibility to debug this so I'm not sure exactly what is happening which is why I am curious as to if and how the maxRetries is used, especially if it is not explicitly passed when creating the dynamo object.

I'm guessing that this might have something to do with this.
From: https://github.com/aws/aws-sdk-js/blob/master/lib/services/dynamodb.js

/**
    * @api private
*/
defaultRetryCount: 10,
/**
    * @api private
*/
retryDelays: function retryDelays() {
    var retryCount = this.numRetries();
    var delays = [];
    for (var i = 0; i < retryCount; ++i) {
        if (i === 0) {
            delays.push(0);
        } else {
            delays.push(50 * Math.pow(2, i - 1));
        }
    }
    return delays;
}

For arguments sake I will assume that the default retires are in fact 10 and that this is the logic that is applied for the exponential back off and have a follow up question on this:
Is there any way to control the number of retires for a specific call. I.e. I have my dynamo object with the default settings and I call putItem once and for that specific call I'd like to have a different maxRetries (in my case 0) but still use the same object. I suspect this is not feasible?

Looking forward to your response and some additional insight on this fine module :)

Br,
Carl

[lsegal]

Great questions, Carl.

1. Yes, the SDK implements exponential backoff (you can see this in the code snippet above).
2. Having maxRetries undefined means it defers to the default value for the given service. Most services have a default of 3, but DynamoDB has a default of 10.
3. You can configure the maxRetries parameter globally (AWS.config.maxRetries = 5) or per-service (new AWS.DynamoDB({maxRetries: 5})).

Note that setting a maxRetries value of 0 means the SDK will not retry throttling errors, which is probably not what you want. If the SDK is taking longer, it's usually because you are being throttled or there is some other retryable error being thrown.

If you want to debug how the SDK is retrying, you can add a handler to inspect these retries:

AWS.events.on('retry', function(resp) {
  if (resp.error && resp.error.retryable) {
    var date = new Date();
    console.log(date, '| Retrying request for the ' + resp.retryCount + 'th time.');
    console.log(date, '| Retry triggered by', err.code, err.message);
  }
});

That event fires whenever the SDK decides to retry. If retryable is set to true on the error, the SDK will retry with the retryDelay property (also on the error object). You can actually adjust either value on your own in that event, if you want more control over how retries work:

AWS.events.on('retry', function(resp) {
  // retry all requests with a 5sec delay (if they are retryable)
  if (resp.error) resp.error.retryDelay = 5000;

  // or alternatively, disable retries completely.
  // This is equivalent to setting maxRetries to 0.
  if (resp.error) resp.error.retryable = false;
});

Hope that helps!

[carlnordenfelt]

Hi,

Yes that helps a lot. It works pretty much as I thought it did :)
From the snippet I pasted I get that the sum of the delay of all retires would be 25550ms ~ 25 seconds which is consistent with the delays we are seeing.

I agree that in general you want the sdk to execute the retries but in our specific case we're not being throttled on the table but rather on a partition but that's another story.

Just so that I don't misunderstand, when you mention overriding the properties in AWS.events.on('retry', ...) I assume that doing so is still in the global scope and not possible to do for a specific operation, such as a putItem request?

Cheers

[lsegal]

Just so that I don't misunderstand, when you mention overriding AWS.events.on('retry', ...) I assume that doing so is still in the global scope and not possible to do for a specific operation, such as a putItem request?

You can add event hooks for individual requests, I was just trying to provide some simple debugging code. To attach the event to an individual request:

var req = dynamodb.putItem(params);
req.on('retry', function() { ... });
req.send(function(err, data) {
  console.log(err, data);
});

[carlnordenfelt]

Sorry, I completely misread that.
Thanks for your answers, this will help a lot.

Br,
Carl
On 5 Nov 2014 23:20, "Loren Segal" notifications@github.com wrote:

Just so that I don't misunderstand, when you mention overriding
AWS.events.on('retry', ...) I assume that doing so is still in the global
scope and not possible to do for a specific operation, such as a putItem
request?

You can add event hooks for individual requests, I was just trying to
provide some simple debugging code. To attach the event to an individual
request:

var req = dynamodb.putItem(params);
req.on('retry', function() { ... });
req.send(function(err, data) {
console.log(err, data);
});

—
Reply to this email directly or view it on GitHub
#402 (comment).

[lsegal]

Np. I'm going to mark this as closed. Feel free to open new issues for any other questions you have, or hop on our Gitter chat and we can discuss more of the technical features if you're up for it:

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.