feat(stepfunctions-tasks): add AutoTerminationPolicy.IdleTimeout to EmrCreateCluster #29954
Conversation
github-actions
bot
added
effort/medium
There was a problem hiding this comment.
The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.
A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.
aws-cdk-automation
dismissed
their stale review
✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
aws-cdk-automation
added
the
pr/needs-community-review
|
|
||
| if (this.props.autoTerminationPolicy !== undefined && !cdk.Token.isUnresolved(this.props.autoTerminationPolicy.idleTimeout)) { | ||
| if (this.props.autoTerminationPolicy.idleTimeout.toSeconds() < 60 || this.props.autoTerminationPolicy.idleTimeout.toSeconds() > 604800) { | ||
| throw new Error(`Idle Timeout must be between 60 and 604800 seconds but got ${this.props.autoTerminationPolicy.idleTimeout.toSeconds()}.`); |
There was a problem hiding this comment.
nit: can we update the error description to explicity mention the idle timeout is for autoTerminationPolicy.
| * | ||
| * @default None | ||
| */ | ||
| readonly idleTimeout: cdk.Duration; |
There was a problem hiding this comment.
Doesn't this property needs to be optional?
| @@ -1733,3 +1733,55 @@ test('Task throws if WAIT_FOR_TASK_TOKEN is supplied as service integration patt | |||
| }); | |||
| }).toThrow(/Unsupported service integration pattern. Supported Patterns: REQUEST_RESPONSE,RUN_JOB. Received: WAIT_FOR_TASK_TOKEN/); | |||
| }); | |||
|
|
|||
| test('Create Cluster with AutoTerminationPolicy', () => { | |||
There was a problem hiding this comment.
Can we also add a test to throw error when the IdleTimeout is given outside the accepted seconds?
|
|
||
| const app = new App(); | ||
|
|
||
| const stack = new Stack(app, 'aws-cdk-emr-create-cluster-with-on-demand-instance-fleet'); | ||
|
|
||
| const step = new EmrCreateCluster(stack, 'EmrCreateCluster', { | ||
| releaseLabel: 'emr-5.36.1', | ||
| releaseLabel: 'emr-6.14.0', |
There was a problem hiding this comment.
Any reason on why we need to update the version? As from the readme, it is mentioned emr release label above 5.30.0 should work.
| */ | ||
| export function AutoTerminationPolicyPropertyToJson(property: EmrCreateCluster.AutoTerminationPolicyProperty) { | ||
| return { | ||
| IdleTimeout: cdk.numberToCloudFormation(property.idleTimeout.toSeconds()), |
There was a problem hiding this comment.
IdleTimeout is an optional property in cloudformation. In that case shouldn't we handle it as optional?
|
This PR has been stale for more than 2 months. I am adding a response-requested label on it. @valentinlagunes do you have any updates on this? |
pahud
added
the
response-requested
|
This PR has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
github-actions
bot
added
closing-soon
Issue #29926
Closes #29926.
Reason for this change
Add support for AutoTerminationPolicy.IdleTimeout in emrCreateCluster task for Step Functions. This will terminate an EMR cluster automatically after it has been idle for a specified amount of time. This feature is currently already supported by CFN as shown here. The parameter is also available in the AWS Console for Step Functions. It is also available if the Step Function definition is specified entirely as ASL JSON in CDK.
Description of changes
This feature was previously supported by CDK. I essentially reverted this commit, but I added some extra checks. The feature was removed because Step Functions did not support the parameter at the time, but it has since been added.
I added the AutoTerminationPolicy as an optional parameter for the EmrCreateClusterProps object. The AutoTerminationPolicy has a required idleTimeout Duration variable. This Duration must be between 60 seconds and 7 days as specified by the CFN documentation above. I added a check for this in the constructor to throw an error if the Duration is outside of this range.
Description of how you validated changes
I added back a unit test that checks that the output JSON includes the AutoTerminationPolicy. I confirmed the existing unit tests all passed. I also confirmed the existing integration tests passed as well.
I also ran a sample integration test and deployed it in my personal account. I verified that a Step Function was created and that it had an EmrCreateCluster task. I executed the Step Function and verified that an EMR cluster was created with an idleTimeout of 2 minutes. I also verified that the EMR cluster was automatically terminated afterwards.
I added the idleTimeout parameter to an existing integration test and verified that the resulting Step Function had the correct parameters as well. I deployed that integration test in my personal account and verified the changes.
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license