[aws-stepfunctions] Support for ResultSelector

[kennu]

A few weeks ago Step Functions added the ResultSelector Payload Template to the Task, Map and Parallel states. It's still missing from AWS CDK. Some other stuff was added too.

It's possible to use CustomState to write your own state, but that is very complicated to do in practice because many things are no longer automatically generated. It would be helpful if state objects allowed you to easily add custom attributes to the state definition generated by toStateJson().

My temporary solution is to extend the state class with something like this:

class CustomLambdaInvoke extends LambdaInvoke {
  public toStateJson(): object {
    const obj: any = super.toStateJson()
    obj.ResultSelector = {
      'Error.$': '$.Payload.Error',
    }
    return obj
  }
}

[shivlaks]

@kennu thanks for opening the issue. I think it's duplicated by #9821 and we're working on it!
let's use that issue to track our efforts to get these new features included into the CDK.

[michaelwiles]

@shivlaks so I kinda went ahead and narrowed the focus of #9821 to only be conditions because I wanted a 1 to 1 between pull rqeuest and issue. So this ticket is not duplicated.

I'm happy to adjust that direction if you think otherwise...

[metgrahamr]

It's not clear from #9821 whether the scope is going to be widened to include ResultSelector. If not could this issue be re-opened as this would be a really useful feature to cut down on the amount of state information that we are passing around.

[kennu]

I would definitely like to get rid of my CustomState objects and use ResultSelector directly. And it would be cool to have support for adding any custom attributes to all the regular state objects, in case more features are added to Step Functions.

[shivlaks]

reopening so we can track this feature in this issue rather than lump it into the issue for Conditions

[michaelwiles]

So for ResultSelector this is my proposal:

We add a resultSelector entry to State - I know that ResultSelector is not relevant for all state types (only Parallel, Task and Map) the current design is to include all the feature and let the sub classes use and provide what they do.

A Result Selector's values are simply key value pairs so I'm not sure if we should create a ResultSelector type or just make the type of the resultSelector field some kind of key:value store so definition: resultSelector?: { [name: string]: string }; I do think this would be adequate.

Then we add a method to the relevant types: addResultSelector which takes this type { [name:string]: string }. This method gets added only to the "states" that support the ResultSelector - TaskStateBase (for tasks) Parallel and Map.

Then we add a method renderResultSelector to State which is then called from the toStateJson method in Map, TaskStateBase and Parallel.

We could potentially choose to create a new ResultSelector type - not entirely sure if this is necessary or appropriate - though I do not see many places where they're using a key -> value structure (which is what you'd need here).

[shivlaks]

@michaelwiles - I had a similar initial thought (start with adding it into State and expose it in concrete implementations where it can be used. Will let you know once I get into the implementation this week, but those seem like a sound set of steps towards landing this feature.

[michaelwiles]

Thanks - not being fluent in typescript I didn't feel confident to start this. So glad it's being taken up.

[cameronsstone]

Expanding on @kennu's solution - this is what I'm using in the meantime:

import {LambdaInvoke, LambdaInvokeProps} from "@aws-cdk/aws-stepfunctions-tasks";
import {Construct} from "@aws-cdk/core";

export interface LambdaInvokeWithResultSelectorProps extends LambdaInvokeProps {
    readonly resultSelector?: object
}

/**
 * Custom lambda invoker so we can use ResultSelector to filter out all the Lambda Task metadata.
 * See https://github.com/aws/aws-cdk/issues/9904
 */
export class LambdaInvokeWithResultSelector extends LambdaInvoke {
    private readonly resultSelector?: object

    constructor(scope: Construct, id: string, props: LambdaInvokeWithResultSelectorProps) {
        super(scope, id, props);

        this.resultSelector = props.resultSelector
    }

    public toStateJson(): object {
        const stateJson: any = super.toStateJson();
        if (this.resultSelector !== undefined) {
            stateJson.ResultSelector = this.resultSelector
        }
        return stateJson
    }
}

FYI: I'm a TS noob, so please be gentle. :-)

[dardanbekteshi]

Would be awesome to have this included in CDK.

[thantos]

Will be great to see this. Here is my shorter and messier hack.

export class LambdaInvokeResultSelector extends LambdaInvoke {
  constructor(
    construct: Construct,
    id: string,
    private xProps: LambdaInvokeProps & {
      resultSelector?: { [key: string]: any };
    }
  ) {
    super(construct, id, xProps);
  }

  // eslint-disable-next-line @typescript-eslint/ban-types
  public toStateJson(): object {
    return {
      ...super.toStateJson(),
      ResultSelector: this.xProps.resultSelector || undefined
    };
  }
}

[eikeon]

@thantos @cameronsstone @kennu Thank you for the work around. Are we still missing an ingredient to cook up the following

 resultSelector: "$.Payload",
 resultPath: "$.Items",

AKA... trying to get an array of items that's returned in the payload merged in with the state input under Items.

@shivlaks Is this a use case that the one you're working on will handle?

Looking forward to having all the ingredients for the Input and Output processing dance.

EDIT: without an extra level $.Items.Items or clobbering the state input

Yeah; it looks like https://states-language.net/#payload-template must be an object and can't be a json path to an array or string. Would be nice if it could be expanded to not squash the non object value cases. Unless there's some other way of processing the input and output without adding an extra level?

[thantos]

Note that the requirement to return an object is in the result selector API on Sfn, not a limitation of CDK or Cfn.

+1 on greater control of outputs. Merging an object (multiple named fields) with the state, setting primatives and arrays as outputs, omitting state fields without listing current fields.

[wong-a]

@thantos @eikeon The Step Functions team is always looking for ways to improve Amazon States Language. JSON processing is a common pain point. If you have suggestions or feature requests, feel free to cut us an issue on the ASL spec's GitHub repo. Syntax proposals are welcome!

https://github.com/awslabs/states-language/issues

[eikeon]

@wong-a, thank you! will do. I think think there's a fairly small addition that could be made to fill in these non object cases. For now I added an extra object wrapper and carried on... but continue to lose chucks of time when hitting such use cases.
This time the addition of the resultSelector got me thinking it was the cure all. A welcome addition but there's at least one or two more tasty ingredients befor

[tomaszdudek7]

Still waiting.

[shivlaks]

I'm going to be picking this one up this weekend. shooting for a PR next week.

[Vinith1994]

@shivlaks Can you please provide an update for this? Thanks!

[aaronbfagan]

Any update on this?

[wong-a]

@aaronbfagan I put up a PR up for this yesterday. Waiting for review from CDK maintainers.

[shivlaks]

@Vinith1994 @aaronbfagan - @wong-a beat me to it and i'm reviewing the PR. we'll try to get this merged soon

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.