How to use Langchain VertexAI for self-deployed model in gcp?

[Storkki]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

new ChatVertexAI({
          ...this.modelConfig,
         endpoint: 'myDeployedModelEndpointId',
         authOptions: {
            authClient: client as JSONClient,
          },
        })

Description

It thros error like below:

call to gauth.request {
  "input": "https://myDeployedModelEndpointId/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-pro:generateContent",
  "code": "ERR_INVALID_URL",
  ...
}

Is there posible to make next endpoint in langchain js? (projects/PROJECT_ID/locations/us-central1/endpoints/myDeployedModelEndpointId).

Or how to use self-deployed models/other models from model garden with ChatVertexAI?

System Info

Nodejs 18
Windows 11

"@langchain/google-vertexai": "^0.1.0",
"langchain": "^0.2.0",
"@langchain/community": "^0.2.1",

[dosubot[bot]]

I found a similar unsolved discussion: Integrating langchainJS with fine tuned gemini model ^[1].

To use Langchain VertexAI for a self-deployed model in GCP and configure the endpoint in Langchain JS to avoid the ERR_INVALID_URL error, follow these steps:

1. Install the necessary package:

   For Node.js environments:

   npm install @langchain/google-vertexai @langchain/core

   For web environments:

   npm install @langchain/google-vertexai-web @langchain/core

2. Set up your credentials:

   For Node.js:

   export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-file.json"

   For web environments, you can set the credentials directly in your code:

   import { ChatVertexAI } from "@langchain/google-vertexai-web";
   
   const model = new ChatVertexAI({
     authOptions: {
       credentials: {"type":"service_account","project_id":"YOUR_PROJECT-12345", ...},
     },
   });

3. Instantiate the model:

   import { ChatVertexAI } from "@langchain/google-vertexai-web";
   
   const llm = new ChatVertexAI({
     model: "gemini-1.5-pro",
     temperature: 0,
     maxRetries: 2,
     // other params...
   });

4. Configure the endpoint:

   You can configure the endpoint by extending the GoogleVertexAILLMConnection class and overriding the buildUrl method to ensure the URL is correctly formatted:

   import { GoogleVertexAILLMConnection } from "@langchain/google-vertexai-web";
   
   class CustomVertexAIConnection extends GoogleVertexAILLMConnection {
     async buildUrl(): Promise<string> {
       const method = this.streaming ? "serverStreamingPredict" : "predict";
   
       if (this.customModelURL.trim() !== "") {
         return `${this.customModelURL}:${method}`;
       }
   
       const projectId = await this.client.getProjectId();
       return `https://${this.endpoint}/v1/projects/${projectId}/locations/${this.location}/publishers/google/models/${this.model}:${method}`;
     }
   }
   
   const customConnection = new CustomVertexAIConnection({
     endpoint: "your-custom-endpoint",
     location: "your-location",
     model: "your-model",
     authOptions: {
       credentials: {"type":"service_account","project_id":"YOUR_PROJECT-12345", ...},
     },
   });

5. Invoke the model:

   const inputText = "VertexAI is an AI company that ";
   const completion = await llm.invoke(inputText);
   console.log(completion);

By following these steps, you should be able to configure the endpoint correctly and avoid the ERR_INVALID_URL error. For more detailed documentation, you can refer to the LangChain API reference ^[2][3][4][5].

To continue talking to Dosu, mention @dosu.