Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ecs] Step Scaling policy does not create all steps #10141

Closed
kyler-hyuna opened this issue Sep 2, 2020 · 11 comments · Fixed by #15345
Closed

[ecs] Step Scaling policy does not create all steps #10141

kyler-hyuna opened this issue Sep 2, 2020 · 11 comments · Fixed by #15345
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. effort/medium Medium work item – several days of effort needs-reproduction This issue needs reproduction. p1

Comments

@kyler-hyuna
Copy link

kyler-hyuna commented Sep 2, 2020

I'm using the scaleOnMetric method of the FargateService construct. My requirements need 2 policies.

construct.scaling.scaleOnMetric('scale-up-policy', {
  scalingSteps: [
    {
      lower: 65,
      change: +2,
    },
    {
      lower: 85,
      change: +3,
    },
  ],
  metric: construct.service.metricCpuUtilization({
    statistic: 'Maximum',
    period: cdk.Duration.minutes(1),
  }),
  cooldown: cdk.Duration.minutes(2),
});

construct.scaling.scaleOnMetric('scale-down-policy', {
  scalingSteps: [
    {
      upper: 50,
      change: -1,
    },
    {
      upper: 40,
      change: -2,
    },
  ],
  metric: construct.service.metricCpuUtilization({
    statistic: 'Average',
  }),
});

However it never creates the 65 step. It only creates >85 step.

Screenshot 2020-09-02 at 3 35 05 PM

Reproduction Steps

  • Create a FargateService
  • Add autoscaling
  • Make step scaling policies as outlined above

What did you expect to happen?

I should have 2 alarms created with the actions:

  • low alarm

    • remove 1 task < 50% CPU
    • remove 2 tasks < 40 % CPU
  • high alarm

    • add 2 tasks > 65% CPU
    • add 3 tasks > 85% CPU

What actually happened?

Environment

  • CLI Version : 1.61.1
  • Framework Version: 1.61.1
  • Node.js Version: v14.8.0
  • OS : mac os
  • Language (Version): typescript

Other


This is 🐛 Bug Report

@kyler-hyuna kyler-hyuna added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 2, 2020
@kyler-hyuna
Copy link
Author

Even worse I added in the 65 step twice to test, and now it works. This is definitely a bug

construct.scaling.scaleOnMetric('scale-up-policy', {
  scalingSteps: [
    {
      lower: 65,
      change: +2,
    },
    {
      lower: 65, // Duplicate of above
      change: +2,
    },
    {
      lower: 85,
      change: +3,
    },
  ],
  metric: construct.service.metricCpuUtilization({
    statistic: 'Maximum',
    period: cdk.Duration.minutes(1),
  }),
  cooldown: cdk.Duration.minutes(2),
});

Screenshot 2020-09-02 at 4 13 26 PM

@SomayaB SomayaB added the @aws-cdk/aws-ecs Related to Amazon Elastic Container label Sep 3, 2020
@olivier-schmitt
Copy link

Same issue in with the Python version; it's a very annoying bug.
Is there a workaround?

@kyler-hyuna
Copy link
Author

@olivier-schmitt try this

Even worse I added in the 65 step twice to test, and now it works. This is definitely a bug

construct.scaling.scaleOnMetric('scale-up-policy', {
  scalingSteps: [
    {
      lower: 65,
      change: +2,
    },
    {
      lower: 65, // Duplicate of above
      change: +2,
    },
    {
      lower: 85,
      change: +3,
    },
  ],
  metric: construct.service.metricCpuUtilization({
    statistic: 'Maximum',
    period: cdk.Duration.minutes(1),
  }),
  cooldown: cdk.Duration.minutes(2),
});

Screenshot 2020-09-02 at 4 13 26 PM

@SoManyHs SoManyHs added needs-reproduction This issue needs reproduction. and removed needs-triage This issue or PR still needs to be triaged. labels Nov 30, 2020
@SoManyHs
Copy link
Contributor

Hi @kyler-hyuna ,

What happens when you use the aws cli to describe the service/autoscaling policy?

@SoManyHs SoManyHs added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 30, 2020
@kyler-hyuna
Copy link
Author

Hey @SoManyHs

The code:

scaleOnMetric('scale-up-policy', {
    scalingSteps: [
      {
        lower: 40,
        change: +3,
      },
      {
        // FIXME: https://github.com/aws/aws-cdk/issues/10141
        lower: 40,
        change: +3,
      },
      {
        lower: 60,
        change: +5,
      },
    ],
    metric: construct.service.metricCpuUtilization({
      statistic: 'Maximum',
      period: cdk.Duration.minutes(1),
    }),
    cooldown: cdk.Duration.minutes(2),
  });

The describe (just kept the scaling part of it):

"StepScalingPolicyConfiguration": {
    "AdjustmentType": "ChangeInCapacity",
    "StepAdjustments": [
        {
            "MetricIntervalLowerBound": 0.0,
            "MetricIntervalUpperBound": 20.0,
            "ScalingAdjustment": 3
        },
        {
            "MetricIntervalLowerBound": 20.0,
            "ScalingAdjustment": 5
        }
    ],
    "Cooldown": 120,
    "MetricAggregationType": "Maximum"
}

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Dec 1, 2020
@coleman-c
Copy link

coleman-c commented Dec 4, 2020

We are seeing similar unexpected behaviors. I'm using the .netcore CDK version 1.76.0, and nodejs v10.19.0

The configuration

[ { "Lower": 300, "Change": 1 }, { "Lower": 600, "Change": 2 }, { "Lower": 900, "Change": 3 } ]

Results in the following:

[
  {
    "MetricIntervalLowerBound": 0,
    "MetricIntervalUpperBound": 300,
    "ScalingAdjustment": 2
  },
  {
    "MetricIntervalLowerBound": 300,
    "ScalingAdjustment": 3
  }
]

Wheras I think is should be:

[
  {
    "MetricIntervalLowerBound": 300,
    "MetricIntervalUpperBound": 600,
    "ScalingAdjustment": 1
  },
  {
    "MetricIntervalLowerBound": 600,
    "MetricIntervalUpperBound": 900,
    "ScalingAdjustment": 2
  },
  {
    "MetricIntervalLowerBound": 900,
    "ScalingAdjustment": 3
  }
]

@MrArnoldPalmer MrArnoldPalmer added effort/medium Medium work item – several days of effort p2 labels Feb 22, 2021
@YuryShchanouskiTR
Copy link

Any updates?

@matheushent
Copy link

Any update on it? I'm facing this.

Screen Shot 2021-06-24 at 09 21 41
Screen Shot 2021-06-24 at 09 22 01

@ruancomelli
Copy link
Contributor

ruancomelli commented Jun 28, 2021

Hi @SoManyHs
What should be the expected behaviour when the user does not provide any intervals starting from lower=0? Should we prepend a dummy scaling interval that takes no actions (that is, change=undefined)? If so, I can readily work on a PR for that.

@mergify mergify bot closed this as completed in #15345 Jul 8, 2021
mergify bot pushed a commit that referenced this issue Jul 8, 2021
…oes not start at 0 (#15345)

Before this fix, the first scaling step would be ignored whenever its lower bound was greater than zero (see linked issue).

Now, instead of *replacing* the first interval with a "dummy" one, we *prepend* the list of intervals. This behavior complements what was already implemented in case the last interval's upper bound was not Infinity: a dummy one was appended.

This fixes #10141


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Jul 8, 2021

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@kyler-hyuna
Copy link
Author

Thanks for fixing this @ruancomelli 🎉

hollanddd pushed a commit to hollanddd/aws-cdk that referenced this issue Aug 26, 2021
…oes not start at 0 (aws#15345)

Before this fix, the first scaling step would be ignored whenever its lower bound was greater than zero (see linked issue).

Now, instead of *replacing* the first interval with a "dummy" one, we *prepend* the list of intervals. This behavior complements what was already implemented in case the last interval's upper bound was not Infinity: a dummy one was appended.

This fixes aws#10141


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. effort/medium Medium work item – several days of effort needs-reproduction This issue needs reproduction. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants