stepfunctions: support variables and jsonata

[humanzz]

Describe the feature

Stepfunctions has recently released 2 major features: Variables and Jsonata

• https://aws.amazon.com/about-aws/whats-new/2024/11/aws-step-functions-variables-jsonata-transformations/
• https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/

This is a request for the relevant CDK constructs/utilities to support those features

Use Case

See feature description

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.170.0

Environment details (OS name and version, etc.)

macOS

[berenddeboer]

Please!

[pahud]

Thank you. Please help us 👍 on this issue to help the team prioritize. Meanwhile, we welcome any PRs from the community.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[snowe2010]

While we wait for WinterYukky's PR to be merged, is there any way to do this through Cfn? I have been unable to find any CFN mechanisms that correspond to the jsonata or variables.

[WtfJoke]

@snowe2010 its the same plain AWS::StepFunctions::StateMachine in cfn.

Just the Definition(String) has a different JSONata syntax. As soon as the definition has the property: "QueryLanguage": "JSONata" its not jsonpath anymore.

[snowe2010]

@WtfJoke it's supposed to be configurable without converting your entire machine, but only individual nodes, right? Where do you set QueryLanguage to JSONata on a per State/Task basis?

[wong-a]

+1 to what @WtfJoke said. The underlying resource/API just accepts an ASL JSON document for the definition. You can use DefinitionBody.fromFile or DefinitionBody.fromString to supply the entire definition in CDK.

@snowe2010 If you don't want to convert your entire state machine, you can add a CustomState which just accepts ASL.

[snowe2010]

ah ok. thank you two.

[WtfJoke]

Where do you set QueryLanguage to JSONata on a per State/Task basis
@snowe2010 yes exactly, on a per state/task basis.

Here is an example Echo StateMachine definition with one Step beeing in JsonPath and one in Jsonata:

Definition

On the second step there is also a variable

{
  "StartAt": "Echo Output",
  "States": {
    "Echo Output": {
      "Type": "Pass",
      "Comment": "This is my long running batch job",
      "Result": {
        "success": true
      },
      "ResultPath": null,
      "Next": "Echo Output Jsonata"
    },
    "Echo Output Jsonata": {
      "Type": "Pass",
      "End": true,
      "QueryLanguage": "JSONata",
      "Assign": {
        "time": "{% $states.context.Execution.StartTime %}"
      },
      "Output": {
        "success": true
      }
    }
  }
}

[snowe2010]

@WtfJoke thank you very much for the example. it's very helpful. I've pretty much touched the ASL language only when doing the workshop and since then haven't touched it at all, instead doing everything through CDK. So this is useful.

[JacekKosciesza]

I've just tested creating JSONata task to get object from a S3 bucket using CustomState. This object is actually an HTML file with an email body I want to send. It has a few %PLACEHOLDERS%, so I also tested some JSONata functions e.g. $replace. Works really good. Maybe this example will be helpful for someone.

this.task = new sfn.CustomState(this, "Get confirmation email from S3", {
  stateJson: {
    Type: "Task",
    Resource: "arn:aws:states:::aws-sdk:s3:getObject",
    QueryLanguage: "JSONata",
    Arguments: {
      Bucket: drive.bucket.bucketName,
      Key: "sc/newsletter/emails/confirm-newsletter-subscription.html",
    },
    Output: {
      body: `{%
        $replace($states.result.Body, "%CDN_DOMAIN%", "https://" & $states.input.environment.CDN_DOMAIN) ~>
        $replace("%YEAR%", $now("[Y]")) ~>
        $replace("%NEWSLETTER_SUBSCRIPTION_CONFIRMATION_URL%", $states.input.environment.CONFIRMATION_URL & "?id=" & $states.input.id)
      %}`,
    },
  },
});

[aquine-kujaruk]

Hello, I want to share a hack that I use to inject attributes not yet supported in the JSON generated by a Step Functions state.

The idea is to override the toStateJson() method of the State object to add additional properties to the base JSON. Here is an example implementation:

export const overrideStateJson = (state: State, props: Record<string, any>): void => {
  const originalToStateJson = state.toStateJson;

  state.toStateJson = function () {
    const baseStateJson = originalToStateJson.call(this);
    
    return {
      ...baseStateJson,
      ...props,
    };
  };
};

Now I can add any property to the state's JSON. For example, to add the Assign parameter to a LambdaInvoke:

const task = new LambdaInvoke(scope, 'MyTask', {
  lambdaFunction: myFunction,
});

overrideStateJson(task, {
  Assign: {
    "foo.$": "$.bar"
  }
});

[jvillane]

Hi @aquine-kujaruk,

I've managed to accomplished the same objective by using CfnStateMachine and the definitionSubstitutions attribute:

...
let definition = fs.readFileSync('./lib/scraper/asl/legislaturas.asl.json', 'utf8');

new CfnStateMachine(this, `${id}-sm`, {
  roleArn: smRole.roleArn,
  definitionString: definition,
  definitionSubstitutions: {
    events_connection_arn: connection.connectionArn,
    legislaturas_table_name: legislaturasTable.tableName,
  },
  stateMachineName: `${id}-sm`,
  stateMachineType: StateMachineType.EXPRESS,
  tracingConfiguration: {
    enabled: true
  },
  loggingConfiguration: {
    destinations: [{
      cloudWatchLogsLogGroup: {
        logGroupArn: logGroup.logGroupArn
      },
    }],
    includeExecutionData: true,
    level: 'ALL',
  }
});
...

And in the state machine definition you must use ${events_connection_arn}:

...
    "Get Legislaturas": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Arguments": {
        "ApiEndpoint": "https://web-back.senado.cl/api/legislatures",
        "Method": "GET",
        "InvocationConfig": {
          "ConnectionArn": "${events_connection_arn}"
        }
      },
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "BackoffRate": 2,
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "JitterStrategy": "FULL"
        }
      ],
      "Next": "Foreach Legislatura",
      "Output": "{% $states.result.ResponseBody.data %}"
    },
...

Hope this helps someone else...

[snowe2010]

the linked PR has been closed and comments restricted, numerous times. #32343

is there any chance of getting that PR merged? Or will a new PR need to be opened and reviews solicited again?

[pruthvirajksuresh]

hope this feature gets released soon. waiting from a long time. any help would be greatly appreciated . thank you

[cshields236]

export const overrideStateJson = (state: State, props: Record<string, any>): void => {
const originalToStateJson = state.toStateJson;

state.toStateJson = function () {
const baseStateJson = originalToStateJson.call(this);

return {
  ...baseStateJson,
  ...props,
};

};
};

Thanks for this, works great!

Hopefully gets a proper release soon

[joerino96]

@GavinZZ @gracelu0 any update for the linked CR? My team is eagerly waiting on jsonata support in CDK.

[dguisinger]

Looks like this PR is getting the same lack of attention as others have that I keep watching. Its unfortunate, I like CDK, but it doesn't seem AWS is willing to staff up the required levels to actually keep up with the AWS services released. OpenSource contributions are great, until the gate keepers never let anything in....

[GavinZZ]

The PR for this highly requested feature has been approved by a maintainer. It's currently in the pr/do-not-merge state while we await any final feedback from other maintainers (though we expect none). We anticipate merging the PR soon, with the targeted release date set for Wednesday, February 5.

I understand the frustration of the wait, but I want to clarify that this PR has not lacked attention. There have been many productive internal conversations, and it has been actively reviewed. We appreciate everyone's patience, and we're working hard to ensure the feature is thoroughly vetted before merging. Thank you for your continued support of CDK!

A big shoutout to @WinterYukky for his relentless work on this change and for addressing all of our feedback! The contribution is much appreciated!

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.