How to set MimeType on gemini calls?

[rossanodr]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const generationConfig = {
  temperature: 1,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

Description

How can I set the responseMimeType on gemini calls using langchain?

System Info

"yarn info langchain"

[dosubot[bot]]

Para configurar o responseMimeType em chamadas Gemini usando LangChain, você pode especificar o responseMimeType diretamente na configuração generationConfig da sua solicitação. Aqui está um exemplo de como fazer isso:

import { GoogleAIModelRequestParams, GoogleAIResponseMimeType } from 'path-to-langchain-google-common/src/types';

const requestParams: GoogleAIModelRequestParams = {
  generationConfig: {
    responseMimeType: 'application/json' as GoogleAIResponseMimeType,
    temperature: 1,
    topP: 0.95,
    topK: 64,
    maxOutputTokens: 8192,
    // outras opções de configuração
  },
  // outros parâmetros de solicitação
};

// Use requestParams na sua chamada Gemini

Alternativamente, se você estiver usando a classe GoogleBaseLLM, pode definir o responseMimeType diretamente na instância:

import { GoogleBaseLLM, GoogleAIResponseMimeType } from 'path-to-langchain-google-common/src/llms';

const llm = new GoogleBaseLLM({
  responseMimeType: 'application/json' as GoogleAIResponseMimeType,
  temperature: 1,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  // outras opções de configuração
});

// Use a instância llm na sua chamada Gemini

Isso configurará a instância GoogleBaseLLM para usar o responseMimeType especificado para todas as suas chamadas.

Referências

• libs/langchain-google-common/src/utils/gemini.ts
• libs/langchain-google-common/src/types.ts
• libs/langchain-google-common/src/llms.ts

Details

Sources

• libs/langchain-google-common/src/utils/gemini.ts
• libs/langchain-google-common/src/types.ts
• libs/langchain-google-common/src/llms.ts
• libs/langchain-google-gauth/src/llms.ts
• libs/langchain-community/src/llms/googlevertexai/common.ts
• libs/langchain-google-webauth/src/llms.ts
• langchain-core/src/tracers/base.ts
• langchain-core/src/callbacks/manager.ts
• examples/src/guides/deployment/error_handling.ts
• libs/langchain-google-common/src/utils/failed_handler.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[rossanodr]

@dosu this does not work

[dosubot[bot]]

To correctly set the responseMimeType for Gemini calls using LangChain, ensure that the mimeType is extracted and set properly when handling media content. Here is a detailed example:

1. Extracting and Setting MIME Type:

   const extractMimeType = (
     str: string
   ): { mimeType: string; data: string } | null => {
     if (str.startsWith("data:")) {
       return {
         mimeType: str.split(":")[1].split(";")[0],
         data: str.split(",")[1],
       };
     }
     return null;
   };
   
   function messageContentImageUrl(
     content: MessageContentImageUrl
   ): GeminiPartInlineData | GeminiPartFileData {
     const url: string =
       typeof content.image_url === "string"
         ? content.image_url
         : content.image_url.url;
     if (!url) {
       throw new Error("Missing Image URL");
     }
   
     const mineTypeAndData = extractMimeType(url);
     if (mineTypeAndData) {
       return {
         inlineData: mineTypeAndData,
       };
     } else {
       return {
         fileData: {
           mimeType: "image/png",
           fileUri: url,
         },
       };
     }
   }
   
   function messageContentMedia(
     content: Record<string, any>
   ): GeminiPartInlineData | GeminiPartFileData {
     if ("mimeType" in content && "data" in content) {
       return {
         inlineData: {
           mimeType: content.mimeType,
           data: content.data,
         },
       };
     } else if ("mimeType" in content && "fileUri" in content) {
       return {
         fileData: {
           mimeType: content.mimeType,
           fileUri: content.fileUri,
         },
       };
     }
   
     throw new Error("Invalid media content");
   }

2. Setting Up Parameters for Gemini Call:

   import { copyAIModelParamsInto } from 'libs/langchain-google-common/src/utils/common';
   
   const params = {
     // other parameters
     responseMimeType: 'application/json',
   };
   
   const options = {
     // other options
   };
   
   const target = {};
   
   const requestParams = copyAIModelParamsInto(params, options, target);

3. Using the GoogleLLM Class:

   import { GoogleLLM } from 'path-to-langchain-google-common/src/llms';
   
   const model = new GoogleLLM({
     responseMimeType: 'application/json',
     temperature: 1,
     topP: 0.95,
     topK: 64,
     maxOutputTokens: 8192,
     // other options
   });
   
   const response = await model.invoke("Your prompt here");

Make sure to replace 'path-to-langchain-google-common/src/llms' with the actual path to the GoogleLLM class in your project.

These steps should help you correctly set the responseMimeType for your Gemini calls using LangChain ^[1][2].

To continue talking to Dosu, mention @dosu.

[buddypia]

@rossanodr
In my use case

        let mimetype = mimeType.lookup(urlFileExtension) || "";
        if (urlFileExtension === "mp3") {
          mimetype = "audio/mp3";
        } else if (urlFileExtension === "mp4") {
          mimetype = "video/mp4";
        }

        const model = "gemini-1.5-pro-preview-0409";
        const generativeModel = new VertexAI({
          location: "asia-northeast1",
        }).getGenerativeModel({
          model: model,
        });

        const instructionPrompt = `
      Write a Japanese summary
      # Instruction
      - Write in Japanese
      - Write within 4000 characters
      `;

        const req = {
          contents: [
            {
              role: "user",
              parts: [
                {
                  fileData: {
                    mimeType: mimetype,
                    fileUri: gsUri,
                  },
                },
                { text: instructionPrompt },
              ],
            },
          ],
        };

        const res = await generativeModel.generateContent(req);
        return (
          res.response.candidates?.[0].content?.parts?.[0].text || "No response"
        );

[rossanodr]

@rossanodr In my use case

        let mimetype = mimeType.lookup(urlFileExtension) || "";
        if (urlFileExtension === "mp3") {
          mimetype = "audio/mp3";
        } else if (urlFileExtension === "mp4") {
          mimetype = "video/mp4";
        }

        const model = "gemini-1.5-pro-preview-0409";
        const generativeModel = new VertexAI({
          location: "asia-northeast1",
        }).getGenerativeModel({
          model: model,
        });

        const instructionPrompt = `
      Write a Japanese summary
      # Instruction
      - Write in Japanese
      - Write within 4000 characters
      `;

        const req = {
          contents: [
            {
              role: "user",
              parts: [
                {
                  fileData: {
                    mimeType: mimetype,
                    fileUri: gsUri,
                  },
                },
                { text: instructionPrompt },
              ],
            },
          ],
        };

        const res = await generativeModel.generateContent(req);
        return (
          res.response.candidates?.[0].content?.parts?.[0].text || "No response"
        );

What I need is to set the response mime type. something like const model = new ChatVertexAI({
model: "gemini-1.5-flash",
maxOutputTokens: 8000,
responseMimeType: 'application/json'
});