feat(anthropic): langchain-anthropic support custom create anthropic client

[rxliuli]

Support for custom Anthropic clients using @langchain/anthropic, which will allow the use of @anthropic-ai/vertex-sdk and @anthropic-ai/bedrock-sdk, and also support custom Anthropic, such as supporting a modified version of the edge runtime.

import { expect, it } from 'vitest'
import { authenticate } from '../authenticate'
import { ChatAnthropic } from '@langchain/anthropic'
import { AnthropicVertex } from '@anthropic-ai/vertex-sdk'
import { GoogleAuth } from 'google-auth-library'
import { AnthropicVertexWeb } from '../AnthropicVertex'

it('Call Anthropic Vertex on nodejs', async () => {
  const client = new AnthropicVertex({
    region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
    projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
    googleAuth: new GoogleAuth({
      credentials: {
        client_email: import.meta.env
          .VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
        private_key: import.meta.env
          .VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
      },
      scopes: ['https://www.googleapis.com/auth/cloud-platform'],
    }),
  })
  const chat = new ChatAnthropic({
    apiKey: 'test',
    model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
    createClient: (() => client) as any,
  })
  const response = await chat.invoke([['human', 'Hello!']])
  console.log(response)
  expect(response).not.undefined
})
it('Call Anthropic Vertex on edge runtime', async () => {
  const token = await authenticate({
    clientEmail: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
    privateKey: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
  })
  const client = new AnthropicVertexWeb({
    region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
    projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
    accessToken: token.access_token,
  })
  const chat = new ChatAnthropic({
    apiKey: 'test',
    model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
    createClient: (() => client) as any,
  })
  const response = await chat.invoke([['human', 'Hello!']])
  console.log(response)
  expect(response).not.undefined
}, 10_000)

Fixes #5196

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		Sep 4, 2024 7:53pm

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			Sep 4, 2024 7:53pm

[jacoblee93]

Ah very nice!

[jacoblee93]

Thank you for this!

[jacoblee93]

This is live in @langchain/anthropic@0.2.16!

[rossanodr]

Support for custom Anthropic clients using @langchain/anthropic, which will allow the use of @anthropic-ai/vertex-sdk and @anthropic-ai/bedrock-sdk, and also support custom Anthropic, such as supporting a modified version of the edge runtime.

import { expect, it } from 'vitest'
import { authenticate } from '../authenticate'
import { ChatAnthropic } from '@langchain/anthropic'
import { AnthropicVertex } from '@anthropic-ai/vertex-sdk'
import { GoogleAuth } from 'google-auth-library'
import { AnthropicVertexWeb } from '../AnthropicVertex'

it('Call Anthropic Vertex on nodejs', async () => {
  const client = new AnthropicVertex({
    region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
    projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
    googleAuth: new GoogleAuth({
      credentials: {
        client_email: import.meta.env
          .VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
        private_key: import.meta.env
          .VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
      },
      scopes: ['https://www.googleapis.com/auth/cloud-platform'],
    }),
  })
  const chat = new ChatAnthropic({
    apiKey: 'test',
    model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
    createClient: (() => client) as any,
  })
  const response = await chat.invoke([['human', 'Hello!']])
  console.log(response)
  expect(response).not.undefined
})
it('Call Anthropic Vertex on edge runtime', async () => {
  const token = await authenticate({
    clientEmail: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
    privateKey: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
  })
  const client = new AnthropicVertexWeb({
    region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
    projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
    accessToken: token.access_token,
  })
  const chat = new ChatAnthropic({
    apiKey: 'test',
    model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
    createClient: (() => client) as any,
  })
  const response = await chat.invoke([['human', 'Hello!']])
  console.log(response)
  expect(response).not.undefined
}, 10_000)

Fixes #5196

Hey, I'm a little nooby.
How should I use it to make it owrk with vertex ai?
Can you help me? thanks

[jacoblee93]

Hey @rossanodr, do these docs help?

https://js.langchain.com/v0.2/docs/integrations/chat/anthropic/#custom-clients

[rossanodr]

Hey @rossanodr, do these docs help?

https://js.langchain.com/v0.2/docs/integrations/chat/anthropic/#custom-clients

Thanks, but I can't seem to find a way to make it work with Vertex. I'm still getting the error:
Error: Unable to verify model params: {"lc":1,"type":"constructor","id":["langchain","chat_models","chat_integration","ChatVertexAI"],
even though I've configured access in the Vertex dashboard.

[rxliuli]

Hey @rossanodr, do these docs help?
https://js.langchain.com/v0.2/docs/integrations/chat/anthropic/#custom-clients

Thanks, but I can't seem to find a way to make it work with Vertex. I'm still getting the error: Error: Unable to verify model params: {"lc":1,"type":"constructor","id":["langchain","chat_models","chat_integration","ChatVertexAI"], even though I've configured access in the Vertex dashboard.

The model name should be similar to claude-3-5-sonnet@20240620 rather than claude-3-5-sonnet-20240620, refer to https://docs.anthropic.com/en/docs/about-claude/models#model-names

---

Additionally, you should first create an AnthropicVertex client and test it without using langchain to determine if the error is caused by langchain. For your reference, I also encountered many authentication-related errors when I initially used AnthropicVertex.

const client = new AnthropicVertex({
  region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
  projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
  googleAuth: new GoogleAuth({
    credentials: {
      client_email: import.meta.env
        .VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
      private_key: import.meta.env
        .VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
    },
    scopes: ['https://www.googleapis.com/auth/cloud-platform'],
  }),
})
const chat = new ChatAnthropic({
  apiKey: 'test',
  model: 'claude-3-5-sonnet@20240620',
  createClient: (() => client) as any,
})
const response = await chat.invoke([['human', 'Hello!']])
console.log(response)

---

Strange, I just noticed that I've been passing a fake apiKey to ChatAnthropic all along. I will submit another PR to handle ignoring the check for API key existence when createClient exists.

[rossanodr]

Hey @rossanodr, do these docs help?
https://js.langchain.com/v0.2/docs/integrations/chat/anthropic/#custom-clients

Thanks, but I can't seem to find a way to make it work with Vertex. I'm still getting the error: Error: Unable to verify model params: {"lc":1,"type":"constructor","id":["langchain","chat_models","chat_integration","ChatVertexAI"], even though I've configured access in the Vertex dashboard.

The model name should be similar to claude-3-5-sonnet@20240620 rather than claude-3-5-sonnet-20240620, refer to https://docs.anthropic.com/en/docs/about-claude/models#model-names

Additionally, you should first create an AnthropicVertex client and test it without using langchain to determine if the error is caused by langchain. For your reference, I also encountered many authentication-related errors when I initially used AnthropicVertex.

const client = new AnthropicVertex({
  region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
  projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
  googleAuth: new GoogleAuth({
    credentials: {
      client_email: import.meta.env
        .VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
      private_key: import.meta.env
        .VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
    },
    scopes: ['https://www.googleapis.com/auth/cloud-platform'],
  }),
})
const chat = new ChatAnthropic({
  apiKey: 'test',
  model: 'claude-3-5-sonnet@20240620',
  createClient: (() => client) as any,
})
const response = await chat.invoke([['human', 'Hello!']])
console.log(response)

Strange, I just noticed that I've been passing a fake apiKey to ChatAnthropic all along. I will submit another PR to handle ignoring the check for API key existence when createClient exists.

I am having authentication issues [llm/error] [1:llm
] [96.96s] LLM run errored with error: "Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n\nError: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n
Probably because it is a Next.js application.
I usually use "@langchain/google-vertexai-web", but here there isn't a similar option available.

[rxliuli]

I am having authentication issues [llm/error] [1:llm ] [96.96s] LLM run errored with error: "Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n\nError: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n Probably because it is a Next.js application. I usually use "@langchain/google-vertexai-web", but here there isn't a similar option available.

Similarly, for testing, I suggest using only NodeJS instead of Edge runtime, as they have some annoying inconsistencies. You can try to complete basic examples in NodeJS first, then consider Edge Runtime, otherwise you may run into issues where you don't know where the error is coming from (in fact, Google cloud services' authentication method itself is very complex, and compared to the original LLM provider that only needs an API_KEY, it's downright nasty). Additionally, I've also submitted a PR for @anthropic-ai/vertex-sdk to support edge runtime (in my scenario, it's cloudflare workers), but it hasn't been merged yet. Reference: anthropics/anthropic-sdk-typescript#509

[rossanodr]

I am having authentication issues [llm/error] [1:llm ] [96.96s] LLM run errored with error: "Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n\nError: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.\n Probably because it is a Next.js application. I usually use "@langchain/google-vertexai-web", but here there isn't a similar option available.

Similarly, for testing, I suggest using only NodeJS instead of Edge runtime, as they have some annoying inconsistencies. You can try to complete basic examples in NodeJS first, then consider Edge Runtime, otherwise you may run into issues where you don't know where the error is coming from (in fact, Google cloud services' authentication method itself is very complex, and compared to the original LLM provider that only needs an API_KEY, it's downright nasty). Additionally, I've also submitted a PR for @anthropic-ai/vertex-sdk to support edge runtime (in my scenario, it's cloudflare workers), but it hasn't been merged yet. Reference: anthropics/anthropic-sdk-typescript#509

Unfortunately, I can't opt for the NodeJS environment at the moment. I hope it will be possible to use it in the edge environment soon.
Thank you for the assistance.