StepFunctions - RunLambdaTask with Payload not working

[sacag]

I am seeing more issues with RunLambdaTask:
below is the code which works with InvokeFunction..$.Payload has my model in json string

 Task = new InvokeFunction(lambdaFunction),                
                InputPath = "$.Payload",
                ResultPath = "$.lambdaResult",
                OutputPath = "$.lambdaResult"

The same code when using RunLambdaTask...not sure how to access $.Payload in the Payload property, so that it can be passed over to the Lambda function? This is what I have

Task = new RunLambdaTask(lambdaFunction,
                            new RunLambdaTaskProps 
                                        { Payload = new Dictionary<string, object>() 
                                                { { "lambdaResult", Context.StringAt("$$.Payload") } } }),
                InputPath = "$.Payload",
                ResultPath = "$.lambdaResult",
                OutputPath = "$.lambdaResult"

When I do Context.StringAt(...) in RunLambdaTask, i think it takes the execution's input and not the task input..i think, maybe there is another way to access the TaskState input?

Basically, the model sent to my lambda is NULL in case of RunLambdaTask.

Environment

• CLI Version :
• Framework Version: 1.31
• OS : Windows
• Language : C#

Other

---

This is 🐛 Bug Report

[pavelhlushchanka]

I confirm the issue. Experienced with just simple replace of InvokeFunction to RunLambdaTask.

[wqzoww]

Hi sacag@,

When you use RunLambdaTask, the "payload" in RunLambdaTaskProps is actually the input you want to pass to your lambda function.
See https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html#API_Invoke_RequestParameters

If you want to pass your $.Payload to Lambda, you should include it inside "payload".
Example: search "lambda example" in https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions/README.md

Calling Context.StringAt(...) is different as it extracts the variable stored in Context Object, the "input" in the block "execution" refers to the input for your execution. The main usage of Context Object is to extract the task token if you want to invoke your lambda function with the pattern ".waitForTaskToken".

[sacag]

@wqzoww @shivlaks - are you suggesting something like this? I think I tried it, but let try again.

Task = new RunLambdaTask(lambdaFunction,
                            new RunLambdaTaskProps 
                                        { Payload = new Dictionary<string, object>() 
                                                { { "lambdaResult", "$.Payload" } } }),
                InputPath = "$.Payload",
                ResultPath = "$.lambdaResult",
                OutputPath = "$.lambdaResult"

[sacag]

@shivlaks - the above code snippet did not work...parameter received in lambda is null

[shivlaks]

@sacag - here's a gist that includes a couple of small inline Lambdas to illustrate how the $.Payload is passed and read between them.

one of the things you're missing in your line from the looks of it is is the .$ as the suffix to the input key. That indicates to Step Functions that the value is a path. It should look something like this

"lambdaResult.$", "$.Payload"

I don't believe you need to specify InputPath, ResultPath, OutputPath

The input will always include the payload in the request body when calling subsequent lambdas and will need to be pointed at specifically. You can read more about the request parameters here

I'm going to mark this as guidance as this isn't quite a bug, but does represent an area where we can improve documentation, samples, and providing direction to users who want to migrate from InvokeFunction to RunLambdaTask

[pavelhlushchanka]

@shivlaks @wqzoww is it possible to get exactly the same behaviour as it was before? I can add to payload something like this "input.$" -> TaskInput.fromContextAt("$$.Execution.Input"). But such change introduces extra layer in my json input that goes to lambda. Is it possible wire payload to input of execution? I tried to create step function manually and having "Payload.$": "$$.Execution.Input" in Parameters block works exactly how it was and what i need, but it's not possible to do the same with cdk.

[sacag]

@shivlaks @wqzoww - suggestion doesn't seem to work. The json layout changes per task state (TaskStateEntered(1st image) and TaskStateScheduled(2nd image))seems like. See the attached images. So, the below code cannot find the correct jsonpath as described in the code below and sends NULL to lambda function

 var saveMsgToDbTask = new Task(scope, "taskId", new TaskProps
            {                
                Task = new RunLambdaTask(props.SaveMessageLambda,
                            new RunLambdaTaskProps
                            {
                                Payload = new Dictionary<string, object>() { { "lambdaResult.$", "$.Payload" } }
                            }), 
            });

[sacag]

@shivlaks - So, my Lambda's FunctionHandler looks like below:

public MessageModel FunctionHandler(MessageModel input, ILambdaContext context)
{}

I changed my input parameter in the RunLambdaTask to this:

Payload = new Dictionary<string, object>() { { "input.$", "$.Payload" } }

Still I get NULL in the lambda's function-handler

[shivlaks]

@codenamestif - Glad you were able to get it working. I'm looking into whether there's a way to preserve the old flow by reaching out to the Step Functions team. AFAIK, the new service integration pattern is structurally different. Some of the historical context is captured here

@sacag are you reading the input field (or whatever you're naming the field) in your handler?

i.e. in the above example: InvokeFunction would have produced

{
  "statusCode": "200",
  "body": "hello, world!"
}

the output, and hence input for subsequent Lambda's in your Step Functions workflow with using RunLambdaTask is now

{
  "ExecutedVersion": "$LATEST",
  "Payload": {
    "statusCode": "200",
    "body": "hello, world!"
  },
  "SdkHttpMetadata": {
    "HttpHeaders": {
      "Connection": "keep-alive",
      "Content-Length": "43",
      "Content-Type": "application/json",
      "Date": "Thu, 16 Apr 2020 17:58:15 GMT",
      "X-Amz-Executed-Version": "$LATEST",
      "x-amzn-Remapped-Content-Length": "0",
      "x-amzn-RequestId": "88fba57b-adbe-467f-abf4-daca36fc9028",
      "X-Amzn-Trace-Id": "root=1-5e989cb6-90039fd8971196666b022b62;sampled=0"
    },
    "HttpStatusCode": 200
  },
  "SdkResponseMetadata": {
    "RequestId": "88fba57b-adbe-467f-abf4-daca36fc9028"
  },
  "StatusCode": 200
}

[sacag]

So, I see how we can get it to work in C# as well. I was able to proof it out. But, it is no where close to an optimal solution for us. We will have to make un-necessary changes to our Model just to incorporate an additional layer of "input field" added to the payload. It's not like Typescript/Nodejs. It's not a dynamic model. So, hopefully there is a better way to do this in C# if not now then sometime soon. Maybe before InvokeFunction is retired. @shivlaks

[shivlaks]

@codenamestif - as you've mentioned, specifying the key as Payload.$ and the value as $.Payload is how the old flow is maintained. The limitation of not being able to specify that in the CDK is a bug. Classifying it as a p1 so we can fix the native experience for all users to support it. Copy @sacag

[DerkSchooltink]

So, I see how we can get it to work in C# as well. I was able to proof it out. But, it is no where close to an optimal solution for us. We will have to make un-necessary changes to our Model just to incorporate an additional layer of "input field" added to the payload. It's not like Typescript/Nodejs. It's not a dynamic model. So, hopefully there is a better way to do this in C# if not now then sometime soon. Maybe before InvokeFunction is retired. @shivlaks

I realise that this may be off-topic, apologies in advance.

You could implement a DTO that consists of a Lambda-compatible structure and map it to your original model so you don't have to mutate your model. While there may be constraints to the setup of mapping payloads using Stepfunctions/CDK (especially in statically typed languages like C#), you could easily abstract away the input models from your internal modes. In general I think this is good practise :)

[pavelhlushchanka]

@shivlaks cool! But i'm not sure about specifying $. Payload as a value. In my case i start execution using sdk with some dynamic input. So basically the first task of my chain has to retrieve the data from $$.Execution.Input and the others take the payload from the output of the previous task. Am i wrong?

[shivlaks]

@shivlaks cool! But i'm not sure about specifying $. Payload as a value. In my case i start execution using sdk with some dynamic input. So basically the first task of my chain has to retrieve the data from $$.Execution.Input and the others take the payload from the output of the previous task. Am i wrong?

@codenamestif Nope, you're right. We need to cover all cases. The $.Payload example was specific to the example that @sacag provided (to get exactly the output of the previous Lambda's execution)

[wong-a]

TL;DR RunLambdaTask should allow users to set Payload to $ or any reference path. "Payload.$": "$" is the way to send the state input as the Invoke Payload, but isn't supported right now.

Step Functions has two ways to represent Lambda Tasks, which is reflected in the two different constructs RunLambdaTask and InvokeFunction. It may help to look at the underlying Amazon States Language code and highlight the difference in behavior.

InvokeFunction (a.k.a "old style" Lambda Task)

The deprecated InvokeFunction used an older way to call Lambda Invoke from Step Functions by setting the Resource field of the Task state to a Lambda function ARN. By default, the input to the state is passed to the Lambda function as the Payload. Here's an example. If the input to the Invoke Function state is { "foo": "bar" }, the Payload send to the Lambda function is also { "foo": "bar" }.

"Invoke Function": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
    "Next": "NEXT_STATE"
}

Optionally, you can specify Parameters to override the Payload that is sent to the Lambda function. In CDK this was set in InvokeFunctionProps.payload.

"Invoke Function - with parameters": {
    "Type": "Task",
    "Parameters": {
      "OriginalExecutionInput.$": "$$.Execution.Input",
      "StateInput.$": "$",
      "HardcodedValue": 123
    },
    "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
    "Next": "NEXT_STATE"
}

In this example, the input to the state is { "foo": "bar" } and the input to the execution is {}. The value of Parameters after resolving the reference paths becomes the Payload sent to Lambda:

{
    "OriginalExecutionInput": {},
    "StateInput": { "foo": "bar" },
    "HardcodedValue": 123
}

RunLambdaTask

RunLambdaTask uses the newer syntax that is consistent with other service integrations. The Resource field is set to arn:aws:states:::lambda:invoke and the FunctionName and Payload need to be specified in the Parameters field. Here Parameters represents parameters to the Lambda Invoke API. Another difference is the output of the state which looks like @shivlaks's example.

To send the entire state input to the Lambda function, "Payload.$": "$" must be set in Parameters. The ASL looks like this:

"Run Lambda Task": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {
        "FunctionName": "FUNCTION_NAME",
        "Payload.$": "$"
    },
    "Next": "NEXT_STATE"
}

In CDK, this is configured in RunLambdaTaskProps.payload but currently, it can't be set to specify reference paths like $ (the entire input), $.foo (a field named foo in the input), $$ (the entire context object), or $$.Execution.Input (a field in the context object - @codenamestif's use case). Currently, the Payload is empty if this property is omitted. In most cases I think users will use $ to pass the state input (or effective input after InputPath filtering like @sacag). Making that the default if RunLambdaTaskProps.payload is not provided sounds sensible to me.

Some edge cases to consider:

• If the user actually wants to pass an empty Payload to Invoke
• How this works with .waitForTaskToken, since Context.taskToken also needs to be injected into the Payload somewhere

[fabio-vitali]

I wonder if with this fix we could take only the "Payload" part of the task result (the lambda return value) and mount it on the ResultPath... I really can't find a way to do this

[wong-a]

I wonder if with this fix we could take only the "Payload" part of the task result (the lambda return value) and mount it on the ResultPath... I really can't find a way to do this

@fabio-vitali Not directly, but you can chain a Pass state with InputPath and ResultPath to the Task to accomplish this. This example below writes the result of the Lambda task to a field named LambdaResult. The Pass state replaces the value of LambdaResult with the value of Payload.

const lambdaTask = new stepfunctions.Task(this, 'CallLambda', {
  task: new tasks.RunLambdaTask(lambdaFn),
  resultPath: '$.LambdaResult'
})

const extractPayload = new stepfunctions.Pass(this, 'ExtractPayload', {
  inputPath: '$.LambdaResult.Payload',
  resultPath: '$.LambdaResult'
})

new stepfunctions.StateMachine(this, 'DemoStateMachine', {
  definition: lambdaTask.next(extractPayload)
})

[fabio-vitali]

@wong-a thank you!!

[shivlaks]

@fabio-vitali related issue (#7709) - workaround if the behaviour is desired to add a Pass state as @wong-a mentioned

I think I included another full sample here feel free to add any other thoughts or considerations you come across in the related issue

cheers!