[WIP] DynamoDB: autoscaling policy

[stephencoe]

Reasoning for this change

Adding autoscaling support for DynamoDB

Related issue

API Reference: http://docs.aws.amazon.com/ApplicationAutoScaling/latest/APIReference/API_PutScalingPolicy.html

Example configuration

resource "aws_appautoscaling_target" "dynamo_test" {
  service_namespace = "dynamodb"
  resource_id = "${aws_dynamodb_table.dynamodb_table_test.name}"
  scalable_dimension = "dynamodb:table:WriteCapacityUnits"
  role_arn = "${aws_iam_role.autoscale_role.arn}"
  min_capacity = 1
  max_capacity = 10
}

resource "aws_appautoscaling_policy" "dynamo_test" {
  name = "%s"
  service_namespace = "dynamodb"
  resource_id = "${aws_dynamodb_table.dynamodb_table_test.name}"
  scalable_dimension = "dynamodb:table:WriteCapacityUnits"
  adjustment_type = "ChangeInCapacity"
  cooldown = 60
  metric_aggregation_type = "Maximum"

  step_adjustment {
    metric_interval_lower_bound = 0
    scaling_adjustment = 1
  }
  
  customized_metric_specification = {
    metric_name = "foo"
    namespace = "dynamodb"
    statistic = "Sum"
  }

  predefined_metric_specification = {
    predefined_metric_type = "DynamoDBWriteCapacityUtilization"
  }
  
  scale_in_cooldown = 10
  scale_out_cooldown = 10
  target_value = 70

  depends_on = ["aws_appautoscaling_target.dynamo_test"]
}

[bflad]

aws-sdk-go was updated to 1.8.44 on master already: #900

[johnthedev97]

I am interested in this feature. I don't see any recent activity on this thread. Is this still being worked? If not I can give it a shot

[stephencoe]

I have pushed where I am with it, not had a lot of time but will try finish it up this week

[mcwqy9]

It would be nice if this patch also supported autoscaling on GSIs. You may run into a blocker in #671, the lack of any way to have terraform ignore changes to a GSI's capacity.

That brings up another point, the docs for aws_appautoscaling_policy should probably indicate how to have terraform ignore changes on the table capacity. Otherwise, assuming the table itself is captured in Terraform, every time you run an apply the capacity on the table will be reset to whatever you hardcoded in the config for the table itself.

[johnthedev97]

I don't think additional code changes are required to support autoscaling on GSI - just keep the scalabledimension as dynamodb:index:ReadCapacityUnits

[bflad]

Any update on this PR? We're excited to implement. 😄

[miesvanderrobot]

Looking forward to this...thanks for spinning it up! We've been running a non-native scaling function in AWS Lambda and are eager to migrate to the native scaling.

[radeksimko]

Hi @stephencoe
thanks for raising the PR.

I understand this is WIP, I left you some comments there. Other than what I mentioned I think we'll need to reshuffle the schema a little bit.

It's currently very flat with all StepScalingPolicyConfiguration-related fields in the 1st level, which makes it very difficult to implement this new scaling policy without conflicts as they're mutually exclusive.

I think the best way forward would be to deprecate adjustment_type, cooldown, metric_aggregation_type, step_adjustment and min_adjustment_magnitude in favour of a new TypeList & MaxItems: 0 field called step_scaling_policy_configuration under which we'd nest all those fields.

We can't entirely avoid dealing with long conditionals around those deprecated fields, but those will hopefully, eventually disappear.

Let me know what you think and/or whether you need help with anything.

I'm happy to make the mentioned changes if you don't have time to spend on this PR, or at the very least deal with the deprecation in a separate PR if you want me to.

[radeksimko]

I think this should be TypeMap instead?

[radeksimko]

I think we can't just introduce a required field - that would cause troubles to users with existing configs which don't use/need this field. In other words I don't believe it needs to be required.

[radeksimko]

As it's only possible to use a single scaling policy at a time would it make more sense to create target_tracking_scaling_policy_configuration and nest all these fields under it, so that we can then pass this object to the API optionally? e.g.

if v, ok := d.GetOk("target_tracking_scaling_policy_configuration"); ok {
  params.TargetTrackingScalingPolicyConfiguration = &applicationautoscaling.TargetTrackingScalingPolicyConfiguration{}
# ...
}

[stephencoe]

Hi @radeksimko,

Thanks for the feedback. Its unlikely that I am going to have a chance to pick this up again in the short term and I am aware there are a lot of people waiting for this to be implemented. Feel free to make the changes you have suggested if it helps to progress this towards being complete

[radeksimko]

@stephencoe Thanks for the quick response and honesty, I'll pick it up in upcoming days. 😉

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!