GoogleGenerativeAIError: [400 Bad Request] * BatchEmbedContentsRequest.requests: at most 100 requests can be in one batch

[ajaykrupalk]

How do I resolve this error when using Google Gemini Pro along with GithubRepoLoader?

GoogleGenerativeAIError: [400 Bad Request] * BatchEmbedContentsRequest.requests: at most 100 requests can be in one batch

[dosubot]

🤖

Hey there, @ajaykrupalk! Great to see you back. Running into a bit of a snag, aren't we? Let's crack this one together.

To resolve the "GoogleGenerativeAIError: [400 Bad Request] * BatchEmbedContentsRequest.requests: at most 100 requests can be in one batch" error when using Google Gemini Pro with GithubRepoLoader, you need to set the maxConcurrency option to a value less than or equal to 100 when instantiating your model. This is because Google Gemini Pro only allows a maximum of 100 requests in one batch.

Here is how you can do it:

import { GoogleGeminiProEmbeddings } from "langchain/embeddings/google";

const model = new GoogleGeminiProEmbeddings({ maxConcurrency: 100 });

This code will ensure that LangChain only sends 100 requests to Google Gemini Pro at a time, thus avoiding the error. This solution was also applied in a similar issue in the langchainjs repository: GoogleGenerativeAIEmbeddings cannot embed more than 100 documents, which was resolved by a pull request: google-genai[patch]: Extend GoogleGenerativeAIEmbeddings to embed more than 100 documents.

As for the default value of maxConcurrency in the GoogleGeminiProEmbeddings class in the langchainjs codebase, I wasn't able to find an answer within the repository. It might be defined elsewhere or I could have missed it. If you need further assistance, please provide more details.

Sources

• GoogleGenerativeAIEmbeddings cannot embed more than 100 documents
• docs/docs/modules/data_connection/text_embedding/how_to/rate_limits.mdx

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[ajaykrupalk]

@dosu-bot these are the changes I implemented yet I am seeing the error:

const embeddings = new GoogleGenerativeAIEmbeddings({
modelName: "embedding-001", // 768 dimensions
taskType: TaskType.RETRIEVAL_DOCUMENT,
title: "Retrieval Document",
maxBatchSize: 100
});

[dosubot]

🤖

Based on the information you've provided, it seems like you've correctly set the maxBatchSize to 100 in your GoogleGenerativeAIEmbeddings configuration. However, the error message you're encountering indicates that more than 100 requests are being sent in a single batch, which exceeds the limit set by the Google Generative AI API.

This could be due to the way the chunkArray function is splitting your documents array into chunks. If the chunkArray function is not working as expected, it could be creating chunks with more than 100 documents.

Here is the implementation of the chunkArray function in the langchainjs codebase:

export const chunkArray = <T>(arr: T[], chunkSize: number) =>
  arr.reduce((chunks, elem, index) => {
    const chunkIndex = Math.floor(index / chunkSize);
    const chunk = chunks[chunkIndex] || [];
    // eslint-disable-next-line no-param-reassign
    chunks[chunkIndex] = chunk.concat([elem]);
    return chunks;
  }, [] as T[][]);

This function is supposed to split the documents array into chunks of size maxBatchSize or less. If this function is not working correctly, it could be causing the error you're seeing.

You can find the chunkArray function in the langchain/src/util/chunk.ts file.

To resolve this issue, you should ensure that the chunkArray function is correctly splitting your documents array into chunks of 100 or less. If the function is not working as expected, you may need to debug or replace it.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• GoogleGenerativeAIEmbeddings cannot embed more than 100 documents
• libs/langchain-google-genai/src/embeddings.ts
• langchain/src/util/chunk.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[RitikSinha]

hello @ajaykrupalk, I have also faced this issue. I used a little hack to bypass the issue. I processed the texts in batches. and we hit API every minute for one batch.

• The code is designed to process an array of items in batches asynchronously.

• processArray divides the input array into smaller batches and processes each batch using processBatch.

• processBatch handles the actual processing logic, which includes database operations (connecting, creating clients), generating embeddings, and logging the generated vectors.

• There's also a delay between processing batches to throttle the rate of processing.

• This approach helps manage resource consumption and prevents overwhelming external services like databases or APIs by limiting the number of simultaneous requests.

async function processBatch(batch, i) {
  console.log("processing " + i);

  try {
    // connecting to the database
    const sbApiKey = process.env.SB_API_KEY;
    const sbUrl = process.env.SB_URL;
    const client = createClient(sbUrl, sbApiKey);
    const embeddings = new GoogleGenerativeAIEmbeddings({
      modelName: "embedding-001", // 768 dimensions
      taskType: TaskType.RETRIEVAL_DOCUMENT,
      title: "data",
    });
    //vector store
    const vector = await SupabaseVectorStore.fromDocuments(batch, embeddings, {
      client,
      tableName: "documents",
    });
    console.log(vector);
  } catch (err) {
    console.log(err);
  }
}

export const processArray = async (array) => {
  const batchSize = 50;
  const delayInMilliseconds = 60 * 1000; // 1 minute in milliseconds
  for (let i = 0; i < array.length; i += batchSize) {
    const batch = array.slice(i, i + batchSize);

    await processBatch(batch, i);

    if (i + batchSize < array.length) {
      // Delay for 1 minute before processing the next batch
      await new Promise((resolve) => setTimeout(resolve, delayInMilliseconds));
    }
  }
};

[ajaykrupalk]

Hey @RitikSinha thank you, this will be really helpful. I will definitely try it.

[ajaykrupalk]

This can be fixed using the following code. The below code is used to send 50 documents at a time to the vector store.

const batchSize = 50;
const totalDocuments = documents.length;

for (let i = 0; i < totalDocuments; i += batchSize) {
     const batch = documents.slice(i, i + batchSize);
      await vectorstore.addDocuments(batch);
}