Extracting Function Names and Arguments for ToolExecutor Not Working with ChatVertexAI, Despite Success with ChatOpenAI and ChatAnthropicTools

[jorge2najo]

Issue Description:

I'm attempting to extract the function_name and associated arguments using a specific structure that integrates seamlessly with ChatOpenAI or ChatAnthropicTools. This process facilitates the utilization of the ToolExecutor module from langgraph.prebuilt.tool_executor to execute the extracted function and arguments.

However, when trying to apply the same methodology with ChatVertexAI, I encounter difficulties. Could you confirm if this functionality is currently supported or implemented for ChatVertexAI? If so, guidance on how to properly achieve this would be greatly appreciated.

Script I would like to use:

class RecordDog(BaseModel):
    """Record some identifying information about a dog."""

    name: str = Field(..., description="The dog's name")
    color: str = Field(..., description="The dog's color")
    fav_food: Optional[str] = Field(None, description="The dog's favorite food")

class RecordDogTool(BaseTool):
    name = "record_dog_tool"
    description = "useful for when you need to record information about a dog"
    args_schema: Type[BaseModel] = RecordDog
    return_direct: bool = False
    
    def _run(
        self, 
        name: str = "Harry", 
        color: str = "brown", 
        fav_food: Optional[str] = "chicken"
    ) -> str:
    
        """Use the tool.""" 

        return f"Recorded {name} who is {color} and loves {fav_food}."


class RecordPerson(BaseModel):
    """Record some identifying information about a person."""

    name: str = Field(..., description="The person's name")
    age: int = Field(..., description="The person's age")
    fav_food: Optional[str] = Field(None, description="The person's favorite food")


class RecordPersonTool(BaseTool):
    name = "record_person_tool"
    description = "useful for when you need to record information about a person"
    args_schema: Type[BaseModel] = RecordPerson
    return_direct: bool = False
    
    def _run(
        self, 
        name: str = "Pedro", 
        age: int = 30, 
        fav_food: Optional[str] = "Pizza"
    ) -> str:
    
        """Use the tool.""" 

        return f"Recorded {name} who is {age} years old and loves {fav_food}."

# Creation of the list of tools
tools = [RecordDogTool(), RecordPersonTool()]
tools = [RecordDogTool(), RecordPersonTool()]
functions = [convert_to_openai_function(t) for t in tools]
tool_executor = ToolExecutor(tools)

# # For ChatOpenAI
# llm = ChatOpenAI(model="gpt-4-turbo-preview", temperature=0)
# llm_with_tools = llm.bind_functions(functions)

# # For ChatAnthropicTools
# llm = ChatAnthropicTools(model_name="claude-3-opus-20240229", temperature=0)
# llm_with_tools = llm.bind_tools(tools=tools)

# For ChatVertexAI
llm = ChatVertexAI(model_name="gemini-pro", temperature=0)
llm_with_tools = llm.bind(functions)

result = llm_with_tools.invoke("Record a dog named Fido who is black and loves bones.")
print(result)
        

Expected output:

Output ChatOpenAI
'''
content='' additional_kwargs={'function_call': {'arguments': '{"name":"Fido","color":"black","fav_food":"bones"}', 'name': 'record_dog_tool'}} response_metadata={'finish_reason': 'function_call', 'logprobs': None}
'''

Output ChatAnthropicTools
'''
content='' additional_kwargs={'tool_calls': [{'function': {'name': 'record_dog_tool', 'arguments': '{"name": "Fido", "color": "black", "fav_food": "bones"}'}, 'type': 'function'}]}

'''

For ChatVertexAI should be something similar like this:
Output ChatVertexAI

content= additional_kwargs={name: 'record_dog_tool', 'arguments':{'name': 'Fido', 'color': 'black', 'fav_food': 'bones', 'type': function'}}