Added Lambda function block

[dominictarro]

Closes #354

Implements a LambdaFunction block.

Example

from prefect_aws import LambdaFunction

lf = LambdaFunction.load("my-lambda")
lf.invoke(
  payload={"foo": "bar"}
)

{
  "Payload": <StreamingBody>,
  "ResponseMetadata": {...},
  "StatusCode": 200
}

Screenshots

Checklist

• References any related issue by including "Closes #" or "Closes ".
  • If no issue exists and your change is not a small fix, please create an issue first.
• Includes tests or only affects documentation.
• Passes pre-commit checks.
  • Run pre-commit install && pre-commit run --all locally for formatting and linting.
• Includes screenshots of documentation updates.
  • Run mkdocs serve view documentation locally.
• Summarizes PR's changes in CHANGELOG.md

[dominictarro]

1. I noticed that the new style doesn't highlight links at all. That should be addressed.
2. I didn't process the AWS response in LambdaFunction.invoke. I think this is the right choice, but I am open to handling error codes and immediately fetching the data response from the StreamingBody in the response["Payload"].
3. The failed tests are overwhelmingly coming from the ECS modules. The lambda module is passing locally.

[zzstoatzz]

hey @dominictarro - thanks so much for the contribution 🙏🏼 just a couple superficial comments, code is looking good to me!

I will try to get this QA'd as soon as possible

edit: its also curious we have those "empty JSON"-looking errors on windows 🧐

[zzstoatzz]

nit, but I think we generally do examples like this instead of the >>> syntax. is there specific reason you did that?

[dominictarro]

No particular reason. Will clean it up.

[desertaxle]

Thanks so much for the PR @dominictarro! For this block, I would recommend implementing the JobBlock interface. Implementing that interface will make it easy for users to either fire-and-forget or wait for execution to finish.

[dominictarro]

hey @dominictarro - thanks so much for the contribution 🙏🏼 just a couple superficial comments, code is looking good to me!

I will try to get this QA'd as soon as possible

edit: its also curious we have those "empty JSON"-looking errors on windows 🧐

@zzstoatzz

So it seems related to needing Docker to execute the Lambda. One of two ways we can go about this

1. Just patch the boto client's .invoke (lambda_client.invoke)
2. Set up a test docker engine and cache the images.

I see almost no reason to go as far as option 2 wants us to go, but maybe you can think of one that I'm missing.

[zzstoatzz]

@dominictarro your first option

Just patch the boto client's .invoke (lambda_client.invoke)

seems reasonable to me, doing some assert_called_with to make sure we're passing the expected things.

I'll also reiterate @desertaxle's point, that for consistency and to allow flexibility for the user, we should try to implement the JobBlock interface here. Any questions about that?

[dominictarro]

@desertaxle @zzstoatzz

... that for consistency and to allow flexibility for the user, we should try to implement the JobBlock interface here. Any questions about that?

Only issue is this is Lambda's 'Event' (asynchronous) endpoint (1, 2) doesn't return any job ID or result fetch URL. The JobBlock interface seems more natural for external execution environments where that is returned by the API like dbt.

If we're intent on JobBlock, what I can try is to create a coroutine that performs the invocation and add it to the running loop, then place its future in the JobRun result. wait_for_completion and fetch_result can just await that future. Maybe this is what you were looking for? I haven't done something like it but I think it should work.

[desertaxle]

The JobBlock interface will work well with the RequestResponse invocation type you currently have as the default. I recommend implementing the JobBlock interface where trigger uses the RequestResponse invocation type and add another method where the Lambda function is invoked with the 'Event' invocation type. That way the 'Event' invocation style doesn't need to return a JobRun and it's clear that method shouldn't be used when the user wants to wait for the run to finish.

[dominictarro]

The JobBlock interface will work well with the RequestResponse invocation type.

I'll change it up so that LambdaFunction implements the JobBlock interface.

I think I'm a bit confused by the use of "fire-and-forget". The boto3.client("lambda").invoke with RequestResponse completely blocks until the Lambda has completed (see examples below). If the point of conforming this block to a JobBlock is the ability to invoke the job and do other things until a result in JobRun is accessed, then this approach doesn't work. It'll just block and then return a JobRun once the Lambda finishes, which seems to defeat the point of using the interface. Is there something that I'm misunderstanding?

I tried these with invoke_with_response_stream as well, no dice.

[desertaxle]

The boto3.client("lambda").invoke with RequestResponse completely blocks until the Lambda has completed

Ah, I didn't realize boto3 blocks until the Lambda function execution completes. Your original intuition that the Lambda function doesn't fit the JobBlock interface is correct. I'd like to see if we can run the invocation as a coroutine or in another thread, but we don't need to block this PR for that. I can give this a review once the tests are passing. Thanks for your patience while I wrap my head around this!

[dominictarro]

@desertaxle Ready for review whenever you have time.

[desertaxle]

Looks great! Just a few nits from me.

[dominictarro]

This is ready for review whenever you have the time.

[zzstoatzz]

this is looking good to me - thanks so much for the contribution / working through the feedback!