Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): step functions task for cross-region AWS API call #30061

Merged
merged 11 commits into from
Jun 24, 2024

Conversation

tmokmss
Copy link
Contributor

@tmokmss tmokmss commented May 4, 2024

Issue # (if applicable)

Closes #29918.

Reason for this change

It would be useful if we could call AWS API across regions from a Step Functions state machine. Currently it is not officially supported even with AWS SDK integration tasks.

Our usecase is to automate a cross-region failover scenario in a multi-region application. This requires you to orchestrate multiple API calls for both active and standby regions (e.g. failover Aurora DB cluster, rewrite AppConfig parameter, etc), and it would be great if we can manage these operations in a single state machine.

Description of changes

This PR adds a new construct CallAwsServiceCrossRegion that deploys 1. a Lambda function to call AWS API in different regions 2. SFn task to call the function.

Because most properties are compatible with the existing CallAwsService construct, you can use the new construct by just adding the region property.

Additionally, it also allows to set endpoint to override AWS API endpoint, because some AWS APIs requires you to override it. (e.g. Route53 ARC)

Here's example code to define this new SFn task:

    const pollTable = new tasks.CallAwsServiceCrossRegion(this, 'DescribeTable', {
      service: 'dynamodb',
      action: 'describeTable',
      parameters: {
        TableName: tableName,
      },
      // You can set any region in the same partition
      region: 'ap-northeast-1',
    });

Description of how you validated changes

Added unit tests and integ tests.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 star-contributor [Pilot] contributed between 25-49 PRs to the CDK labels May 4, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team May 4, 2024 14:35
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@tmokmss tmokmss changed the title feat(aws-stepfunctions-tasks): Step Functions task for cross-region AWS API call feat(stepfunctions-tasks): step functions task for cross-region AWS API call May 4, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review May 4, 2024 14:38

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label May 4, 2024
Copy link
Contributor

@nmussy nmussy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, just some concerns and minor comments. I'm also wondering if it's better for maintainability not to create a new class, and to reuse the existing CallAwsService, and just to add an optional crossRegionTarget prop


try {
// esbuild-disable unsupported-require-call -- not esbuildable but that's fine
const pkg = require(`@aws-sdk/client-${event.service}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong, but normalizeServiceName is not being ran for your handler? It would allow us to keep the same pattern as other services props, see AwsSdkCall:

this.service = normalizeServiceName(service);

export function normalizeServiceName(service: string) {
service = service.toLowerCase(); // Lowercase
service = service.replace(/^@aws-sdk\/client-/, ''); // Strip the start of a V3 package name
service = v2ToV3Mapping()?.[service] ?? service; // Optionally map v2 name -> v3 name
return service;
}

Copy link
Contributor Author

@tmokmss tmokmss May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep this feature and API as simple as possible to lower the maintenance cost. AwsSdkCall code has become such complicated mostly for backward compatibility reason. Since this feature is new, we don't have to introduce the complexity at all.

Also the API is aimed to be compatible with the existing CallAwsService construct (and it should), which also does not support normalizing service names.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll let the maintainer weigh in, I think it's a nice piece of QoL, but I understand your maintainability concern

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help me understand, can you elaborate on your concern? What maintenance cost are you concerned with using AwsSdkCall code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AwsSdkCall's code is mainly to solve the v2-v3 compatibility problem. Since this construct does not require such compatibility, I think there is little advantage to using the existing code. After all, the new Lambda is only code under 100 lines. Also, if we want to reuse the existing code, we need to refactor it first (for example, currently it's only for CFn custom resource handler, but this one is for invocation from SFn). Given the ROI, I'd rather keep them independent.

*/
export interface CallAwsServiceCrossRegionProps extends sfn.TaskStateBaseProps {
/**
* The AWS service to call in AWS SDK for JavaScript v3 style.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment about normalizeServiceName

// esbuild-disable unsupported-require-call -- not esbuildable but that's fine
const pkg = require(`@aws-sdk/client-${event.service}`);
const Client = findV3ClientConstructor(pkg);
const Command = findCommandClass(pkg, event.action);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, you should be able to use normalizeActionName, see AwsSdkCall:

export function normalizeServiceName(service: string) {
service = service.toLowerCase(); // Lowercase
service = service.replace(/^@aws-sdk\/client-/, ''); // Strip the start of a V3 package name
service = v2ToV3Mapping()?.[service] ?? service; // Optionally map v2 name -> v3 name
return service;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as service name, I'd like to intentionally only support action names in camelCase.

/**
* The API action to call.
*
* Use camelCase.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment about normalizeActionName

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label May 6, 2024
@tmokmss
Copy link
Contributor Author

tmokmss commented May 7, 2024

Hi @nmussy, thanks for the comments.

reuse the existing CallAwsService, and just to add an optional crossRegionTarget prop

I considered this approach as well but chose the current approach, because with the current approach we can easily avoid from future breaking changes when SFn will officially start to support cross-region API call; we can just deprecate CallAwsServiceCrossRegion construct and add the official functionality to CallAwsService construct.

Copy link
Contributor

@nmussy nmussy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a couple of pending comments

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label May 7, 2024
Copy link
Contributor

@GavinZZ GavinZZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some minor feedbacks. Also have a high-level questions I want to clarify on. There's already a stepfunctions task for CallAwsService. Is the reason of building this cross-region-aws-sdk-handler using custom resource lambda function because the AWS SDK service integrations do not support cross region invocation?

*
* @default - service:action
*/
readonly iamAction?: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: does it only support one single action? What would users do if I want to support multiple actions?

Copy link
Contributor Author

@tmokmss tmokmss Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, you can use additionalIamStatements. I agree that this should ideally be iamActions?: string[], but then the API becomes less compatible with CallAwsService construct. I think it's okay to leave this as-is because most API only requires a single IAM action anyway. Wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you share an example how to use additionalIamStatements with another action? I also think it would be nice to add this example to the readme file.

Copy link
Contributor Author

@tmokmss tmokmss Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but that is same as CallAwsService, which is described here:

```ts
const detectLabels = new tasks.CallAwsService(this, 'DetectLabels', {
service: 'rekognition',
action: 'detectLabels',
iamResources: ['*'],
additionalIamStatements: [
new iam.PolicyStatement({
actions: ['s3:getObject'],
resources: ['arn:aws:s3:::my-bucket/*'],
}),
],
});
```

so I added a line as below:

Other properties such as additionalIamStatements can be used in the same way as the CallAwsService task.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, thank you!


try {
// esbuild-disable unsupported-require-call -- not esbuildable but that's fine
const pkg = require(`@aws-sdk/client-${event.service}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help me understand, can you elaborate on your concern? What maintenance cost are you concerned with using AwsSdkCall code.

@tmokmss
Copy link
Contributor Author

tmokmss commented Jun 19, 2024

Hi @GavinZZ, thank you so much for the review!

Is the reason of building this cross-region-aws-sdk-handler using custom resource lambda function because the AWS SDK service integrations do not support cross region invocation

Yes. According to the doc:

Currently, cross-Region AWS SDK integration and cross-Region AWS resource access aren't available in Step Functions.
https://docs.aws.amazon.com/step-functions/latest/dg/concepts-access-cross-acct-resources.html

GavinZZ
GavinZZ previously approved these changes Jun 21, 2024
Copy link
Contributor

mergify bot commented Jun 21, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 21, 2024
Copy link
Contributor

mergify bot commented Jun 21, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot dismissed GavinZZ’s stale review June 21, 2024 02:47

Pull request has been modified.

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 21, 2024
@tmokmss
Copy link
Contributor Author

tmokmss commented Jun 21, 2024

@GavinZZ Integ test failed due to a recent mainline change #30108. Looking forward to your re-approval 🙏 Thanks!

Copy link
Contributor

mergify bot commented Jun 24, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 24, 2024
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 6e6693e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 1397737 into aws:main Jun 24, 2024
12 checks passed
Copy link
Contributor

mergify bot commented Jun 24, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 star-contributor [Pilot] contributed between 25-49 PRs to the CDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stepfunctions-tasks: Step Functions task for cross-region AWS API call
4 participants