[Bug] gemini-1.5-pro-latest returns error when combining tools and output schema #703
Comments
MichaelDoyle
changed the title
|
We will take a look at this. Thanks for reporting! Changed the title for readability as the exact error message(s) is/are repeated in the bug reports itself. |
Thank you @MichaelDoyle 🙏 |
|
Looking into this, just reproduced it. Seems to happen with gemini 1.5 flash too, so unlikely the model |
|
An issue where they're getting the same message, linking just for reference google-gemini/generative-ai-swift#195 |
|
I believe this is a limitation of the Gemini API at the moment, I will try to follow up and get it passed on to them |
Can confirm @cabljac, I just came across this same bug using 1.5 Flash. |
This is a good example of why I'm so curious about the philosophy behind when to use tools vs. custom services: https://github.com/firebase/genkit/discussions/731. This seems like a use-case where services might be necessary, at least for the time being, if we want to do things like compose model responses, resolve promises, etc., while also validating output? |
Okay, thank you. |
In my opinion genkit is such a "service", it does parsing, validation, generation, has an interface for tools, etc... I believe the end user shouldn't have to build their own abstraction on top of these building blocks, just like I don't have to build any abstraction layers when I'm using firestore, firebase storage, callable functions or any other firebase service. |
|
That more or less aligns with my understanding of Genkit's goals @lazakrisz, and I agree. I initially developed a "service" abstraction for my own use-case because I was just stubbing things in until I got up to speed using tools. It was only when I went to refactor and replace the service abstraction with Genkit's tools that I hit up against this limitation. Specifically, I'm identifying the names/types of fields in a form given a form name. I then use the field type to determine the corresponding candidate field attributes, which get used to dynamically generate an output schema. The current inefficiency is that I'm essentially using this service abstraction to fan-out model calls for every field/attribute pair. I'm working on creating one mega-prompt that asks the model to select from the candidate attributes in one go, which would get it down to just two API calls.
Okay, looking at @MichaelDoyle's response in this other discussion, I don't think this is actually true. In this case, even though I just need a tool to retrieve a static value from a local object or JSON file, I suppose there would still need to be two API calls. |
|
Are you two in the genkit discord? I feel like this and any other feedback you have would be highly valued there |
|
Just joined @cabljac, thanks for the suggestion! |
|
My suggestion for this is to check for the condition of tools + json mode in the Gemini plugins (Google/Vertex) and simply don't set JSON mode when that's the case. Gemini output was pretty good before we even had JSON mode implemented, it should work in the majority of cases. We can separately suggest it as a fix at the Gemini API layer. |
|
Note this probably only occurs the Google AI plugin, as that's the only one which supports strict json mode currently |
Describe the bug
When using Gemini 1.5 Pro with tool calling, (function calling ) and specifying output schema within
generate, the generate call returns an error:[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-latest:generateContent: [400 Bad Request] Function calling with a response mime type: 'application/json' is unsupported.What's interesting is upon removal of the
output.schemaproperty the generate function does output valid JSON. (see screenshot below)and here is the topmost part of the response:
for clarity here is the entire response / logs cycle:
I've also tried the vertex ai plugin but got a similar esoteric error message.
Tried the following manual function invocation however generate once again throws an error:
the original source code which throws an error:
here is the quizTool, which is the same in both cases:
If I remove the
output.schema, all works and I get an object as a response, but the object takes an arbitrary shape instead of the desired shape that I'd like, which makes parsing things quite tedious.here is an example:
and the Genkit local dashboard output:
input:
{ "messages": [ { "role": "user", "content": [ { "text": "\n Respond as JSON only.\n You are a helpful assistant. Using the tools available, try to generate questions and appropriate answers for a quiz.\n You should only generate n number of questions, n is the number of questions that should be generated based on the tool's response.\n You should take the following criteria in mind:\n * generate questions and answers based on the topics that the user has chosen, you may use the appropriate tools to get this information, you may use any distribution to generate the questions from the provided topics randomly\n * generate questions and answers based on the questions that the user has chosen, you may use the appropriate tools to get this information, you must include these questions and you must only provide potential asnwers to these questions\n * you may also generate more than one potential answers to a given question\n \n QuizId: 0qfxuNZiTgBsQ1ZXYpQ5\n " } ] } ], "tools": [ { "name": "quizTool", "description": "\n This tool is useful for getting information about a quiz. The information includes the following:\n * the number of questions that should be generated\n * the topics that the user has chosen\n * the user questions that should be included in the questions\n ", "outputSchema": { "type": "object", "properties": { "numberOfQuestions": { "type": "number", "description": "The amount of questions that should be generated" }, "userQuestions": { "type": "array", "items": { "type": "string" }, "description": "The questions that were chosen by the user and must be included in the generated question set with the provided answers" }, "userTopics": { "type": "array", "items": { "type": "string" }, "description": "The topics that were chosen by the user for which you must generate questions and the appropriate answers." } }, "required": [ "numberOfQuestions", "userQuestions", "userTopics" ], "additionalProperties": true, "description": "JavaScript object which contains the numberOfQuestions property, userQuestions and userTopics properties.", "$schema": "http://json-schema.org/draft-07/schema#" }, "inputSchema": { "type": "object", "properties": { "quizId": { "type": "string", "description": "The ID of the quiz about which you require more information." } }, "required": [ "quizId" ], "additionalProperties": true, "description": "JavaScript object which contains the quiz id property, which should be the id of the quiz that you require the information about.", "$schema": "http://json-schema.org/draft-07/schema#" } } ], "output": { "format": "text" } }output:
{ "candidates": [ { "index": 0, "message": { "role": "model", "content": [ { "toolRequest": { "name": "quizTool", "input": { "quizId": "0qfxuNZiTgBsQ1ZXYpQ5" } } } ] }, "finishReason": "stop", "custom": { "safetyRatings": [ { "category": "HARM_CATEGORY_DANGEROUS_CONTENT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_HATE_SPEECH", "probability": "NEGLIGIBLE" } ] } } ], "custom": { "candidates": [ { "content": { "parts": [ { "functionCall": { "name": "quizTool", "args": { "quizId": "0qfxuNZiTgBsQ1ZXYpQ5" } } } ], "role": "model" }, "finishReason": "STOP", "index": 0, "safetyRatings": [ { "category": "HARM_CATEGORY_DANGEROUS_CONTENT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "probability": "NEGLIGIBLE" }, { "category": "HARM_CATEGORY_HATE_SPEECH", "probability": "NEGLIGIBLE" } ] } ], "usageMetadata": { "promptTokenCount": 287, "candidatesTokenCount": 29, "totalTokenCount": 316 } }, "usage": { "inputCharacters": 975, "inputImages": 0, "inputVideos": 0, "inputAudioFiles": 0, "outputCharacters": 0, "outputImages": 0, "outputVideos": 0, "outputAudioFiles": 0, "inputTokens": 287, "outputTokens": 29, "totalTokens": 316 }, "latencyMs": 2156.761416912079 }here is the input for the one that Errors out:
{ "messages": [ { "role": "user", "content": [ { "text": "\n Respond as JSON only.\n You are a helpful assistant. Using the tools available, try to generate questions and appropriate answers for a quiz.\n You should only generate n number of questions, n is the number of questions that should be generated based on the tool's response.\n You should take the following criteria in mind:\n * generate questions and answers based on the topics that the user has chosen, you may use the appropriate tools to get this information, you may use any distribution to generate the questions from the provided topics randomly\n * generate questions and answers based on the questions that the user has chosen, you may use the appropriate tools to get this information, you must include these questions and you must only provide potential asnwers to these questions\n * you may also generate more than one potential answers to a given question\n \n QuizId: 0qfxuNZiTgBsQ1ZXYpQ5\n " }, { "text": "\n\nOutput should be in JSON format and conform to the following schema:\n\n```\n{\"type\":\"object\",\"properties\":{\"questions\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"question\":{\"type\":\"string\",\"description\":\"The questions for the given quiz question.\"},\"answer\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"The array of answers to the question.\"}},\"required\":[\"question\",\"answer\"],\"additionalProperties\":true},\"description\":\"An array of objects that contains the question and answers to that question.\"}},\"required\":[\"questions\"],\"additionalProperties\":true,\"description\":\"JavaScript Object which contains the questions property that indicates the questions for the user\",\"$schema\":\"http://json-schema.org/draft-07/schema#\"}\n```\n", "metadata": { "purpose": "output", "source": "default" } } ] } ], "tools": [ { "name": "quizTool", "description": "\n This tool is useful for getting information about a quiz. The information includes the following:\n * the number of questions that should be generated\n * the topics that the user has chosen\n * the user questions that should be included in the questions\n ", "outputSchema": { "type": "object", "properties": { "numberOfQuestions": { "type": "number", "description": "The amount of questions that should be generated" }, "userQuestions": { "type": "array", "items": { "type": "string" }, "description": "The questions that were chosen by the user and must be included in the generated question set with the provided answers" }, "userTopics": { "type": "array", "items": { "type": "string" }, "description": "The topics that were chosen by the user for which you must generate questions and the appropriate answers." } }, "required": [ "numberOfQuestions", "userQuestions", "userTopics" ], "additionalProperties": true, "description": "JavaScript object which contains the numberOfQuestions property, userQuestions and userTopics properties.", "$schema": "http://json-schema.org/draft-07/schema#" }, "inputSchema": { "type": "object", "properties": { "quizId": { "type": "string", "description": "The ID of the quiz about which you require more information." } }, "required": [ "quizId" ], "additionalProperties": true, "description": "JavaScript object which contains the quiz id property, which should be the id of the quiz that you require the information about.", "$schema": "http://json-schema.org/draft-07/schema#" } } ], "output": { "format": "json", "schema": { "type": "object", "properties": { "questions": { "type": "array", "items": { "type": "object", "properties": { "question": { "type": "string", "description": "The questions for the given quiz question." }, "answer": { "type": "array", "items": { "type": "string" }, "description": "The array of answers to the question." } }, "required": [ "question", "answer" ], "additionalProperties": true }, "description": "An array of objects that contains the question and answers to that question." } }, "required": [ "questions" ], "additionalProperties": true, "description": "JavaScript Object which contains the questions property that indicates the questions for the user", "$schema": "http://json-schema.org/draft-07/schema#" } } }To Reproduce
Expected behavior
I would expect to be able to call functions / tools via the model and the output should be JSON / zod parsable. I thought this was entirely possible according to the document / sample code given on the Genkit site + the model description site.
where it is stated that the model supports:
Screenshots
please see the screenshot above, thank you.
Runtime (please complete the following information):
** Node version
Additional context
Should I cross post this into: https://github.com/google-gemini/generative-ai-js as that seems to be the root cause of the issue? 🤔 Is this something that is not currently supported? 🤔 It seems weird that I'm unable to specify a concrete schema for the generate function, but the model itself is capable of returning JSON / object responses, am I missing something? Thank you 🙏
The text was updated successfully, but these errors were encountered: