AWS Bedrock LLM and Embeddings

[mlconnor]

This PR is to include the AWS LLM and Embeddings for Amazon Bedrock. A couple of callouts and limitations...

1. The embeddings implementation does not do batched of embeddings. Batch size is set to one. Trying to follow up with the Bedrock team to get the API for how to batch, not sure if its available yet.
2. I'm assuming Node 18 with fetch included by the system. If we need to support < Node 18 then we will have to import fetch.
3. Currently, I'm hard coding the available models in the component. Long term, we should probably call this dynamically.

[HenryHengZJ]

feel free to Mark it as Ready for Review when the refractor is done!

[mlconnor]

Hey Henry, my understanding is that the project is MIT licensed. I don't understand why you're asking us to sign an SAP contributor license. Can you clarify? Also, the MIT license is a liberal license, so the rights being requested in that agreement are (from what I can tell) already granted by MIT. I'd like to be able to contribute to this project without getting lawyers involved.

[HenryHengZJ]

Hey Henry, my understanding is that the project is MIT licensed. I don't understand why you're asking us to sign an SAP contributor license. Can you clarify? Also, the MIT license is a liberal license, so the rights being requested in that agreement are (from what I can tell) already granted by MIT. I'd like to be able to contribute to this project without getting lawyers involved.

When setting up the project, we just thought of following what others open source project has done with the CLA. But as we have more PR coming in, we also started to realize there is no real needs for that since this is already a MIT licensed project. This was raised a couple of times, and now we have finally taken this CLA off. It will only be reflected for new incoming PR, so you'll probably have to open a new PR, or we can stick to this one, just ignore the CLA message.

Edit: CLA has been removed, and no longer appears as one of the checks for PR

[arneschreuder]

@mlconnor is there a reason to not use import { .... } from 'langchain/llms/bedrock'?
Docs here

I am also interested and keen to get this node into the repo. How can I help?

[mlconnor]

@mlconnor is there a reason to not use import { .... } from 'langchain/llms/bedrock'? Docs here

I am also interested and keen to get this node into the repo. How can I help?

Hey @arneschreuder, a friend of mine @brnkrygs created a new branch in my repo (link below) that uses the underlying langchain class as you suggest. I agree that it's the right way to go. We found a bug that we are tracking down in that code that results in the Claude Model results being cut off. It's possible that it's because Flowing uses v128 instead of the newer v135 but we need to determine that. If you'd like to help, we'd love to have you help us track down why that is. The Titan models seem to work well so if you are good with Titan than you might be ok with Brian's branch. Once we get this fixed we will issue another pull request using the langchain base class.

https://github.com/mlconnor/Flowise/tree/feature/langchain-bedrock-llm

[arneschreuder]

@mlconnor and @brnkrygs apologies for only getting back to this now. I was on leave for a bit. Should I look at @brnkrygs's branch to see if we can replace the bedrock components from langchain to this PR or how else can I help?

[brnkrygs]

Hi folks!

Thanks for the teamwork on getting Bedrock support into Flowise!

Looks like recent updates to Langchain have gotten the right bugs fixed, and include Bedrock GA SDK updates.

I have a separate fork at the moment off of the head of Flowise:main that drops this support into place.

Rather than splinter off into new work, I'm going to see if I can get this fork/branch cleaned up/updated to make a clean merge possible. Expect an update on this PR today!

[arneschreuder]

That's great news @brnkrygs. Please do not hesitate to reach out if we can help with anything. We are quite excited for this.

[HenryHengZJ]

Thanks for the effort! Hopefully this can get it in before the webinar on 8th!

[brnkrygs]

Alright, the cleaned up commits here should add:

1. Support for AWS credentials. Note that these should be optional... the AWS SDK can pick up authentication from the executing environment too. For example, when you run the server on an EC2 instance, the SDK can use the instance's role to get credentials, you don't need to set up long-term IAM keys.
2. Support for Bedrock as an LLM, using langchain's implementation.
3. Support for Bedrock as a chatmodel, again using langchain's implementation.

Hopefully the langchain hooks will help Flowise stay updated without too much extra work!

My local tests are working, but I don't have enough access to all supported integrations to do a full manual regression.

Also, note that stopSequences are not (yet) implemented. I haven't had time to figure out the right way to parse the Flowise json setting to map them into the Bedrock parameter.

If that's a blocker, feel free to grab this and augment. Otherwise I'll look for time later in the week to work on it.

Happy to chase stopSequences as a separate PR as well.

[arneschreuder]

Ill review for us. I think we can add stopSequences in a next iteration. I think it is more important to just get Bedrock capabilities in.

[arneschreuder]

These look correct to me. At some later point we can read the regions, models and different dynamic elements from the AWS API/Site. But this is good enough for now in my opinion.

[brnkrygs]

A note for future updates too: Embeddings.

Langchainjs doesn't yet have Bedrock Embeddings support. Chasing that down to see what progress might be underway. Might want to rename this PR to reflect this fact.

[HenryHengZJ]

This is great! We can have a separate PR to update the stopSequences param. Thanks a lot guys

[arneschreuder]

Thank you very much @mlconnor and @brnkrygs for this! Much appreciated.