Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(stepfunctions): various JSONata query language fix #33404

Merged
merged 15 commits into from
Feb 19, 2025

Conversation

GavinZZ
Copy link
Contributor

@GavinZZ GavinZZ commented Feb 12, 2025

Issue # (if applicable)

Closes #33403 and #33374 and #33396.

Reason for this change

There are three issues here:

  1. For summary, the first issue is basically that assign property cannot be accessed with using Map.jsonata(...) but available if we directly create map through new Map(...) using JSONATA query language.
  2. For summary, the second issue is that JSONATA main PR added the outputs and assign property in the CatchProps interface for AddCatch functionality. But I don't think it's being used in the actual addCatch call https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/states/state.ts#L398.
  3. Result writer and item reader class do not support using JSONATA. Deployment will fails due to if SFN is set to use JSONATA, it expects Arguments in the ASL instead of Parameters.

Description of changes

Fix both issues by fixing the interface inheritance and added the props to AddCatch method.
Support JSONATA as the query language.

Description of how you validated changes

Added integ test and unit test to make sure that

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added bug This issue is a bug. p2 labels Feb 12, 2025
@aws-cdk-automation aws-cdk-automation requested a review from a team February 12, 2025 00:55
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Feb 12, 2025
Copy link

codecov bot commented Feb 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.16%. Comparing base (f4453c7) to head (c78af25).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #33404   +/-   ##
=======================================
  Coverage   82.16%   82.16%           
=======================================
  Files         119      119           
  Lines        6857     6857           
  Branches     1157     1157           
=======================================
  Hits         5634     5634           
  Misses       1120     1120           
  Partials      103      103           
Flag Coverage Δ
suite.unit 82.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
packages/aws-cdk ∅ <ø> (∅)
packages/aws-cdk-lib/core 82.16% <ø> (ø)

@GavinZZ GavinZZ added the pr/do-not-merge This PR should not be merged at this time. label Feb 12, 2025
@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 12, 2025

Marking this PR as pr/do-not-merge. Pending a review from @WinterYukky prior to merge this change.

Also update the PR label to correctly reflect the issue priority.

@GavinZZ GavinZZ added p1 and removed p2 labels Feb 12, 2025
@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 12, 2025
…#33423)

### Issue # (if applicable)

Closes #33403 and #33374 and #33396.

### Reason for this change

There are three issues here:
1. For summary, the first issue is basically that assign property cannot
be accessed with using Map.jsonata(...) but available if we directly
create map through new Map(...) using JSONATA query language.
2. For summary, the second issue is that JSONATA main PR added the
outputs and assign property in the CatchProps interface for AddCatch
functionality. But I don't think it's being used in the actual
`addCatch` call
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/states/state.ts#L398.
3. Result writer and item reader class do not support using JSONATA.
Deployment will fails due to if SFN is set to use JSONATA, it expects
`Arguments` in the ASL instead of `Parameters`.

### Description of changes

Fix both issues by fixing the interface inheritance and added the props
to `AddCatch` method.
Support `JSONATA` as the query language.

### Description of how you validated changes

Added integ test and unit test to make sure that 

### Checklist
- [ ] My code adheres to the [CONTRIBUTING
GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and
[DESIGN
GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license*
@github-actions github-actions bot added the effort/medium Medium work item – several days of effort label Feb 13, 2025
@GavinZZ GavinZZ changed the title fix(stepfunctions): support JSONATA for result writer and item reader fix(stepfunctions): various JSONata query language fix Feb 13, 2025
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This review is outdated)

@aws-cdk-automation aws-cdk-automation dismissed their stale review February 13, 2025 01:04

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Copy link
Contributor

@WinterYukky WinterYukky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thank you so much for fixing this bug. I'm very honored that you started working on this fix right away.

assign and _addCatch fix seems good! On the other hand, itemReader and itemWriter may need to be changes. In itemReader and itemWriter, whether to use Arguments or Parameters is determined only by the top-level or Map state queryLanguage. In this PR, users can use s3objectsitemReader.jsonata() and s3objectsitemReader.jsonPath(), but for example, s3objectsitemReader.jsonPath() (i.e. Parameters) can be used when the Map state uses JSONata we can't.

// invalid cofigure
 const distributedMap = sfn.DistributedMap.jsonPath(this, 'DistributedMap', {
    itemReader: sfn.S3ObjectsItemReader.jsonata({
      bucketNamePath: '{% $bucketName %}',
      prefix: '{% $prefix %}',
    }),
 });

This should probably be processed projectively with CDK without relying on user specifications. Also, since the properties used in JSONata and JSONPath are not different, there is probably no need to use different constructors.

 const distributedMap = sfn.DistributedMap.jsonPath(this, 'DistributedMap', {
    itemReader: new sfn.S3ObjectsItemReader({
      bucketName: '{% $bucketName %}',
      prefix: '{% $prefix %}',
    }),
 });

@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 13, 2025

@WinterYukky I addressed all your comments. Please give it another review once you're available.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 18, 2025
@aaythapa
Copy link
Contributor

@mergify update

Copy link
Contributor

mergify bot commented Feb 18, 2025

update

✅ Branch has been successfully updated

@GavinZZ GavinZZ dismissed WinterYukky’s stale review February 19, 2025 00:23

Dismiss to get this merged and released.

@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 19, 2025

@Mergifyio requeue

Copy link
Contributor

mergify bot commented Feb 19, 2025

requeue

❌ This pull request head commit has not been previously disembarked from queue.

@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 19, 2025

@mergify update

Copy link
Contributor

mergify bot commented Feb 19, 2025

update

✅ Branch has been successfully updated

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: c78af25
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@GavinZZ GavinZZ merged commit b16fac2 into main Feb 19, 2025
18 checks passed
@GavinZZ GavinZZ deleted the yuanhaoz/sfn-jsonata-fix branch February 19, 2025 01:49
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 19, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue is a bug. contribution/core This is a PR that came from AWS. effort/medium Medium work item – several days of effort p1 pr/do-not-merge This PR should not be merged at this time.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

step functions: Map.jsonata does not support assign
4 participants