AWS Bedrock Document/Image Support

[MNorgren]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Amazon Bedrock supports text, images and documents in their Converse API.

Currently, langchainjs only supports the following MessageContent:

• text
• image_url

It appears that the langchain-aws (the python version), has added support for both of these message content types:

• Link to the Issue
• Link to the Pull Request

I propose to add support for these message content types in the ChatBedrockConverse.

Motivation

My motivation(s) to add support for these message content types in the ChatBedrockConverse are:

1. To maintain consistency between python and js libraries as recommended in your CONTRIBUTION.md
2. To keep up with the Amazon Bedrock features

Proposal (If applicable)

Step 1: Add Support for document and image

In the file libs/langchain-aws/src/common.ts, add support for block.type === 'document' || block.type === 'image'

See this commit for full file details

        // Code starting on line 126: https://github.com/langchain-ai/langchainjs/blob/e065486267136b4c47946d33c60c8b4ff76f4465/libs/langchain-aws/src/common.ts#L126
        if (typeof msg.content === "string" && msg.content !== "") {
          return {
            role: "user" as const,
            content: [
              {
                text: msg.content,
              },
            ],
          };
        } else if (Array.isArray(msg.content)) {
          const contentBlocks: ContentBlock[] = msg.content.flatMap((block) => {
            if (block.type === "image_url") {
              const base64: string =
                typeof block.image_url === "string"
                  ? block.image_url
                  : block.image_url.url;
              return extractImageInfo(base64);
            } else if (block.type === "text") {
              return {
                text: block.text,
              };
            } else if (block.type === 'document' && block.document !== undefined) {
              return {
                document: block.document,
              };
            } else if (block.type === 'image' && block.image !== undefined) {
              return {
                image: block.image,
              };
            } else {
              throw new Error(`Unsupported content block type: ${block.type}`);
            }
          });
          return {
            role: "user" as const,
            content: contentBlocks,
          };
        }

Step 2: Invoke client with Human Messages

Document Example: (working)

const client = new ChatBedrockConverse(clientOptions);

// Convert PDF to bytes
const invoiceUrl = 'https://example.com/invoice.pdf';
const arrayBuffer = await fetch(pdfUrl).then((res) => res.arrayBuffer());
const bytes = new Uint8Array(arrayBuffer);

// Create HumanMessage with document
const userPrompt = new HumanMessage({
  content: [
    {
      type: 'text',
      text: 'Please parse this invoice for me and return the total amount',
    },
    {
      type: 'document',
      document: {
        name: 'Example Invoice',
        format: 'pdf',
        source: {
          bytes,
        },
      },
    },
  ],
});

// Invoke request
const pdfResponse = await client.invoke([userPrompt]);

Image Example: (working)

const client = new ChatBedrockConverse(clientOptions);

// Convert image to bytes
const imageUrl = 'https://example.com/house.png';
const arrayBuffer = await fetch(pdfUrl).then((res) => res.arrayBuffer());
const bytes = new Uint8Array(arrayBuffer);

const userPrompt = new HumanMessage({
  content: [
    {
      type: 'text',
      text: 'What does this house look like?',
    },
    {
      type: 'image',
      document: {
        name: 'Example House',
        format: 'png',
        source: {
          bytes,
        },
      },
    },
  ],
});
const imageResponse = await client.invoke([userPrompt]);

[MNorgren]

I've created Pull Request #6956 to resolve this conversation.

[MNorgren]

Closing this, as it has been resolved with #6956 - Just waiting for the next release.

Thank you!