langgraph/tutorials/sql-agent/ #879
giscus[bot]
bot
Replies: 22 comments 51 replies
-
|
Hello, thanks for this amazing explanation. I'm trying to convert this sql agent to gemini llm and BigQuery but in the following step I'm receiving an error: query_check_system = """You are a SQL expert with a strong attention to detail.
Double check the BigQuery query for common mistakes, including:
- Using NOT IN with NULL values
- Using UNION when UNION ALL should have been used
- Using BETWEEN for exclusive ranges
- Data type mismatch in predicates
- Properly quoting identifiers
- Using the correct number of arguments for functions
- Casting to the correct data type
- Using the proper columns for joins
If there are any of the above mistakes, rewrite the query. If there are no mistakes, just reproduce the original query.
You will call the appropriate tool to execute the query after running this check."""
query_check_prompt = ChatPromptTemplate.from_messages(
[("system", query_check_system), ("placeholder", "{messages}")]
)
query_check = query_check_prompt | ChatGoogleGenerativeAI(temperature=0, model="gemini-pro").bind_tools(
[db_query_tool], tool_choice="required"
)
query_check.invoke({"messages": [("user", "SELECT COUNT(id_cliente) FROM my_project.my_dataset.cliente;")]})Produces this error: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/proto/message.py", line 729, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py", line 235, in to_proto
pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
File "/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py", line 36, in to_proto
return self._descriptor(**value)
ValueError: unknown enum label "any"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/sqlchain/app_langgraph_base.py", line 112, in <module>
query_check.invoke({"messages": [("user", "SELECT COUNT(id_cliente) FROM my_project.my_dataset.cliente;")]})
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2507, in invoke
input = step.invoke(input, config)
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4588, in invoke
return self.bound.invoke(
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 248, in invoke
self.generate_prompt(
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 677, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 534, in generate
raise e
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 524, in generate
self._generate_with_cache(
File "/home/user/.local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 749, in _generate_with_cache
result = self._generate(
File "/home/user/.local/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 758, in _generate
request = self._prepare_request(
File "/home/user/.local/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 907, in _prepare_request
formatted_tool_config = ToolConfig(
File "/usr/local/lib/python3.10/dist-packages/proto/message.py", line 757, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py", line 235, in to_proto
pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
File "/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py", line 36, in to_proto
return self._descriptor(**value)
ValueError: unknown enum label "any"What am I missing? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Can't we just directly import tools from |
Beta Was this translation helpful? Give feedback.
-
|
I would say that technically yes. But with doing that, you are adding the tools to the agent and allowing it to self-select the tool to use. Here, with LangGraph, you are able to better control the trajectory of the tools used |
Beta Was this translation helpful? Give feedback.
-
|
There might be confusion here. For example, the |
Beta Was this translation helpful? Give feedback.
-
|
I think it's possible, but you will need to build the tools by hand, practically the same as SQLDatabaseToolkit does (source code) |
Beta Was this translation helpful? Give feedback.
-
|
I encountered an issue where the from langchain_core.prompts import ChatPromptTemplate
query_check_system = """You are a SQL expert with a strong attention to detail.
Double check the SQLite query for common mistakes, including:
- Using NOT IN with NULL values
- Using UNION when UNION ALL should have been used
- Using BETWEEN for exclusive ranges
- Data type mismatch in predicates
- Properly quoting identifiers
- Using the correct number of arguments for functions
- Casting to the correct data type
- Using the proper columns for joins
If there are any of the above mistakes, rewrite the query. If there are no mistakes, just reproduce the original query.
You will call the appropriate tool to execute the query after running this check."""
query_check_prompt = ChatPromptTemplate.from_messages(
[("human", query_check_system), ("placeholder", "{messages}")] # here I changed the 'system' to 'human'
)
query_check = query_check_prompt | ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(
[db_query_tool], tool_choice="required"
)
query_check.invoke({"messages": [("user", "SELECT * FROM Artist LIMIT 10;")]}) |
Beta Was this translation helpful? Give feedback.
-
|
i was experience a slight problem while using this code, which query gen node return results from get_schema_tool without creating a query, solution for this problem was adding db_query_tool as a tool to query_gen!! 🤖🤖 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm encountered the same issue. Can you show what you mean by "...adding db_query_tool as a tool to query_gen..."? Thanks in advance! |
Beta Was this translation helpful? Give feedback.
-
|
what i noticed is query gen just use the context coming from sample data. So I did bind the db_query_tool into query gen like this, and also I removed all the tools mentioned like list_tables_tool, model_get_schema, get_schema_tool. i gave the same details using those in the prompt in using normal Pythonic way. reason to remove the mentioned tools with some nodes to achieve speed and generate the correct query within one iteration! i mean I customize this a lot to achieve what I want.. hope this will help. |
Beta Was this translation helpful? Give feedback.
-
|
I also found this problem, I think I can start from should continue, I don't quite understand why there are toolcalls and direct output, so I want to change the judgment condition there and let it go correct and then execute that way, do you think?I also think it's okay for you to do this, and then submit it there and change it to END, or not? |
Beta Was this translation helpful? Give feedback.
-
|
For me, it was a bit of a pain to solve that without modifying the structure too much. I detected that the issue comes from the So, instead of binding a tool, I just changed the prompt, the conditional function, and the I'll leave the changes here if somebody has trouble following the tutorial, but there are better ways to create a SQL Agent. QUERY_GEN_INSTRUCTION = """You are a SQL expert with a strong attention to detail.
You can define SQL queries, analyze queries results and interpretate query results to response an answer.
Read the messages bellow and identify the user question, table schemas, query statement and query result, or error if they exist.
1. If there's not any query result that make sense to answer the question, create a syntactically correct SQLite query to answer the user question. DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.
2. If you create a query, response ONLY the query statement. For example, "SELECT id, name FROM pets;"
3. If a query was already executed, but there was an error. Response with the same error message you found. For example: "Error: Pets table doesn't exist"
4. If a query was already executed successfully interpretate the response and answer the question following this pattern: Answer: <<question answer>>. For example: "Answer: There three cats registered as adopted"
"""
# Chain
query_gen_prompt = ChatPromptTemplate.from_messages([("system", QUERY_GEN_INSTRUCTION), ("placeholder", "{messages}")])
query_gen = query_gen_prompt | ChatOpenAI(model="gpt-4o-mini")
# Node
def query_gen_node(state: State):
message = query_gen_chain.invoke(state)
return {"messages": [message]}
# Conditional Function
def should_continue(state: State) -> Literal[END, "correct_query", "query_gen"]:
messages = state["messages"]
last_message = messages[-1]
if last_message.content.startswith("Answer:"):
return END
if last_message.content.startswith("Error:"):
return "query_gen"
else:
return "correct_query" |
Beta Was this translation helpful? Give feedback.
-
|
I think that the approach advised in the tutorial provided on the langchain docs is more straight forward and more efficiently exploits the agent / graph architecture. Using create_react_agent allows to handle more complex db schemas and tool_calls in an efficient way. Also, it relies on a scientific framework described in literature, and it's easier to integrate in multi-agent applications. Despite seeing why someone would choose to follow a more fixed implementation like this one for some use cases, I don't think it should be the go-to and I don't understand why the tutorials for SQL-agents change so often |
Beta Was this translation helpful? Give feedback.
-
|
NEVER get the right results. I have tried more than 30 times and optimized prompts for all nodes. What can I do to fix it? |
Beta Was this translation helpful? Give feedback.
-
|
There is a mistake in the tutorial. It gave me a headache too, but I have the solution. You need to add model_check_query when you initialize the query_gen variable, so the AI can call that tool. Like this: query_gen = query_gen_prompt | ChatOpenAI(model="gpt-4o", temperature=0).bind_tools( |
Beta Was this translation helpful? Give feedback.
-
|
@patrickcteng Thanks for your reply. After binding the tool From LangSmith Trace and the log output, after three attempts, the generated SQL code is still unusable, and the Graph execution is blocked without any known reason. Have you encountered this situation before? ================================= Tool Message =================================
Error: The wrong tool was called: sql_db_schema. Please fix your mistakes. Remember to only call SubmitFinalAnswer to submit the final answer. Generated queries should be outputted WITHOUT a tool call.
================================= Tool Message =================================
Error: The wrong tool was called: model_check_query. Please fix your mistakes. Remember to only call SubmitFinalAnswer to submit the final answer. Generated queries should be outputted WITHOUT a tool call.
================================== Ai Message ==================================
To find out which sales agent made the most in sales in 2009, we need to join the `Employee`, `Customer`, and `Invoice` tables. We will filter the invoices by the year 2009 and group the results by the sales agent to sum up the total sales.
Here is the query to achieve this:
\```sql
SELECT
e.FirstName || ' ' || e.LastName AS SalesAgent,
SUM(i.Total) AS TotalSales
FROM
Employee e
JOIN
Customer c ON e.EmployeeId = c.SupportRepId
JOIN
Invoice i ON c.CustomerId = i.CustomerId
WHERE
strftime('%Y', i.InvoiceDate) = '2009'
GROUP BY
SalesAgent
ORDER BY
TotalSales DESC
LIMIT 1;
\```
Let's execute this query.
Tool Calls:
SubmitFinalAnswer (call_3SH6DxnnCP2iqGtp8ABHCsFO)
Call ID: call_3SH6DxnnCP2iqGtp8ABHCsFO
Args:
final_answer: SELECT
e.FirstName || ' ' || e.LastName AS SalesAgent,
SUM(i.Total) AS TotalSales
FROM
Employee e
JOIN
Customer c ON e.EmployeeId = c.SupportRepId
JOIN
Invoice i ON c.CustomerId = i.CustomerId
WHERE
strftime('%Y', i.InvoiceDate) = '2009'
GROUP BY
SalesAgent
ORDER BY
TotalSales DESC
LIMIT 1; |
Beta Was this translation helpful? Give feedback.
-
|
It's amazing. Without modifying the code, after restarting the Jupyter Kernel and running it again, the last AI message was not blocked; instead, it executed the SQL query and got the correct result. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, how did you fix it? I binded the tool |
Beta Was this translation helpful? Give feedback.
-
|
pls answer, how did you fix it ? |
Beta Was this translation helpful? Give feedback.
-
|
I am using the code from https://langchain-ai.github.io/langgraph/tutorials/sql-agent/ tutorial as it is with my SQL database. But I am facing error as In the below edge definitions, it is able to list all the tables and also getting the right table for my prompt. It is using How to fix this? Let me know if any inputs or other details needed. Thank you in advance! Below is the event(printed) when
|
Beta Was this translation helpful? Give feedback.
-
|
@madhubandru this cuz your customization isn't complete. your tool binding returns a tool message so the next step is to call that tool. if you don't want to call the tool do not bind those and get SQL query as model output. one option to get the only thing you want from llm is to use structured output. to do so you need to use a structured output support chat model from langchain create a pedantic class and use it with |
Beta Was this translation helpful? Give feedback.
-
|
@ashen007 Below is the code for my customization of the graph and the error I am encountering. Please suggest to me what is wrong here. Graph Flow: ERROR: When retrying (at 4PM) with same code got this error: it's weird |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I also faced an issue that originated from an if statement in the query gen node! my approach was to generate SQL -> check if it is accurate -> if not re-direct to gen node -> if correct go to execution node -> get results node. this was done using a state graph and inserting each node output as a message in the graph state. in this approach, we do not need an if statement and tool call to get the final answer. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for looking into my issue. The code you provided executes without any errors, but the SQL statement generated by |
Beta Was this translation helpful? Give feedback.
-
|
@madhubandru If you're encountering issues with query generation, possibly due to the complexity of the table schema, I recommend using a dynamic few-shot approach by providing examples that reflect the complexity of the queries. One of our team members implemented this method, and it led to a significant improvement in results. :3 |
Beta Was this translation helpful? Give feedback.
-
|
BadRequestError: Error code: 400 - {'error': {'message': 'invalid role: tool, role must be one of [system, user, assistant]', 'type': 'api_error', 'param': None, 'code': None}} anyone ever encountered this error and have solved it? thanks. |
Beta Was this translation helpful? Give feedback.
-
|
@Adhikasurya use
tool calling is available! |
Beta Was this translation helpful? Give feedback.
-
|
thanks!!, it has binding tools now, but it seems to produce another error, i'll look it up later. AttributeError: 'ModelPrivateAttr' object has no attribute 'chat' |
Beta Was this translation helpful? Give feedback.
-
|
i guess it always works well by using OpenAI models, but always encountered errors while using local llm |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @Adhikasurya, I also encountered the same problem: "AttributeError: 'ModelPrivateAttr' object has no attribute 'chat'", have you solved it? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @Wijipedia, sorry i have stopped working on it. I have migrated to CrewAI and it works just fine and easy to customize |
Beta Was this translation helpful? Give feedback.
-
|
Some questions about the exaple: Why you need to return to It's execute_query necessary? maybe you can summarize correct_query and execute_query. Please help me to clarify my indeas :3
|
Beta Was this translation helpful? Give feedback.
-
|
something like this? credits to madhubandru post from this thread |
Beta Was this translation helpful? Give feedback.
-
|
I was thinking on something like this:
|
Beta Was this translation helpful? Give feedback.
-
|
Nice! For our use case we made some like this: Aquí tienes una versión mejorada del post en inglés, paso a paso, con una redacción más clara y profesional: Steps:
Here’s a simplified version of my Pydantic class: class QueryReviewer(BaseModel):
"""
Stores the evaluation of the query generated by the agent, including the final answer and the observations.
"""
question: Annotated[str, Field(..., description="The main question that the query must answer.")]
query: Annotated[str, Field(..., description="The SQL query, must be in valid MS SQL syntax.")]
opinion: Annotated[
str,
Field(
...,
description="The opinion about the query, to know whether or not it meets the requirement of the question",
),
]
errors: Annotated[
Optional[List[str]],
Field(
default_factory=list,
description="A list of errors found in the query, if any. Can be null if there are no errors.",
),
] = None
steps_to_fix: Annotated[
Optional[List[str]],
Field(
default_factory=list,
description="A list of steps to fix the errors found in the query, if any. Can be null if no steps are needed.",
),
] = None
df_name: Annotated[
str, Field(..., description="The snake_case name for the dataframe that will hold the query result.")
]
conclusion: Optional[Literal["correct", "incorrect"]] = Field(
None,
description="The conclusion about the query. Must be 'correct', 'incorrect', or 'none'. Can be null if the conclusion is not determined.",
)
@field_validator("df_name")
def validate_snake_case(cls, value):
"""
Validates that df_name is in snake_case format.
"""
if not re.match(r"^[a-z_][a-z0-9_]*$", value):
raise ValueError("df_name must be in snake_case, starting with a lowercase letter or underscore.")
return valueThe
I'm completely open to feedback and constructive criticism! |
Beta Was this translation helpful? Give feedback.
-
|
I'm not totally sure about your use case, but that approach makes sense to me, we actually think on something similar. 🙌 Here is my feedback
In the approach I'm working on I'm not using any tool. The happy path is working but it is progress yet... Maybe it could give you some ideas, feel free to comment. 😊 -> Notebook: Custom SQL Agent |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate your feedback! Thank you very much. |
Beta Was this translation helpful? Give feedback.
-
|
Guys I'm having a problem, the agent create the correct query (I confirm it in dbeaver) but I keep receiving this error "content: "Error: Query failed. Please rewrite your query and try again.", and it start looping again. Can somebody help ? |
Beta Was this translation helpful? Give feedback.
-
|
Two posible options, your query is always bad or you are not updating the corrected query in the state or tool. |
Beta Was this translation helpful? Give feedback.
-
|
I tested the query on dbeaver and it's working perfectly, then I test with this: |
Beta Was this translation helpful? Give feedback.
-
|
you should add some time-related SQL queries as a few shot examples |
Beta Was this translation helpful? Give feedback.
-
|
If you have complex query's maybe you need to give some few shots, also don't forget to pass the table schema and some useful data that helps the llm to create a correct query. |
Beta Was this translation helpful? Give feedback.
-
|
Does somebody test it with 4o-mini ? because when using it, the agent doesn't execute the query, only answer with the query itself |
Beta Was this translation helpful? Give feedback.
-
|
I had a similar problem... I could fix it as I explained here ... You could also see the full fix in the SQL Agent (LangGraph) -Fixed version Notebook. |
Beta Was this translation helpful? Give feedback.
-
|
I tryied your soution and didn't work, maybe our use case are a bit different |
Beta Was this translation helpful? Give feedback.
-
|
Hey, so I found the issue in this one and was able to correct it. In the query_gen_node when it tries to create a query, it notices that it doesnt have the schema for the other tables it needs inorder to join and fetch the results. So I simple did this while binding the tools to the model.- llm_gen_tools = llm.bind_tools( ) query_gen = query_gen_prompt | llm_gen_tools And in the system instruction I changed this part - to " Call SubmitFinalAnswer to submit the final answer." It worked for me! Hope it helps you as well! |
Beta Was this translation helpful? Give feedback.
-
|
mine works perfectly fine, what error did you fix ? |
Beta Was this translation helpful? Give feedback.
-
|
UnprocessableEntityError: status_code: 422, body: data=None message='No valid tool call or response generated' {'first_tool_call': {'messages': [AIMessage(content='', additional_kwargs={}, response_metadata={}, id='056cef8e-59c8-4b41-bf40-c8493d756c66', tool_calls=[{'name': 'sql_db_list_tables', 'args': {}, 'id': 'tool_abcd123', 'type': 'tool_call'}])]}} |
Beta Was this translation helpful? Give feedback.
-
|
Does this not work with GroqAPI ? face this error - " { File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\pregel\init.py:1551, in Pregel.invoke(self, input, config, stream_mode, output_keys, interrupt_before, interrupt_after, debug, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\pregel\init.py:1290, in Pregel.stream(self, input, config, stream_mode, output_keys, interrupt_before, interrupt_after, debug, subgraphs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\pregel\runner.py:56, in PregelRunner.tick(self, tasks, reraise, timeout, retry_policy, get_waiter) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\pregel\retry.py:29, in run_with_retry(task, retry_policy) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\utils\runnable.py:385, in RunnableSeq.invoke(self, input, config, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langgraph\utils\runnable.py:167, in RunnableCallable.invoke(self, input, config, **kwargs) Cell In[52], line 12, in (state) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\runnables\base.py:5354, in RunnableBindingBase.invoke(self, input, config, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\language_models\chat_models.py:286, in BaseChatModel.invoke(self, input, config, stop, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\language_models\chat_models.py:786, in BaseChatModel.generate_prompt(self, prompts, stop, callbacks, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\language_models\chat_models.py:643, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\language_models\chat_models.py:633, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_core\language_models\chat_models.py:851, in BaseChatModel._generate_with_cache(self, messages, stop, run_manager, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\langchain_groq\chat_models.py:473, in ChatGroq._generate(self, messages, stop, run_manager, **kwargs) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\groq\resources\chat\completions.py:287, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, parallel_tool_calls, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\groq\_base_client.py:1244, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\groq\_base_client.py:936, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls) File c:\Users\VaishakBhuvanMR\Desktop\Comparechat\compare\Lib\site-packages\groq\_base_client.py:1039, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls) BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '{"tool\\_call":{"id":"pending","type":"function","function":{"name":"sql\\_db\\_schema"},"parameters":{"table\\_names":"Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track"}}}'}}" |
Beta Was this translation helpful? Give feedback.
-
|
this is common case with groq failed to call the correct tool, what you can do is handle the error or switch to better model |
Beta Was this translation helpful? Give feedback.
-
|
I was getting the same error about calling the wrong tool so I rewrote the graph slightly to make it easier to debug. (Relevant code is shown as the last code block in this message.) While looking at the message generated by the query generator chain that is bound with the SubmitFinalAnswer tool, I'm starting to wonder if there is a problem with the way LangChain handles tool bindings. The chat model is clearly calling a tool that is not bound to it. Btw, I'm using Anthropic model Claude 3.5 Sonnet v2. I have my breakpoint set at the statement right after the line where the query-generating chain is invoked, which is the statement starting with
I can tell there was only one tool bound to the chat model in the chain stored in query_generator by printing the kwargs field on the second step in the chain, as follows: print((query_generator.steps[1]).kwargs)Output: However, the message that was output from query_generator.invoke() contains a totally different tool call which is not mentioned anywhere in my code: To make it easier to see, here is the tool_calls property of that AIMessage: pprint.pprint(message.tool_calls)Output: How is it possible this tool # Configure a chat model instance with ability to call a tool that gets the schema of a table.
model_get_schema = ChatBedrock(model_id=awsutil.default_chat_model_id, client=awsutil.get_client('bedrock-runtime')).bind_tools([get_schema_tool])
# Configure a chat model instance with ability to call a tool that checks and runs a query.
chat_model_with_query_tool = ChatBedrock(model_id=awsutil.default_chat_model_id, client=awsutil.get_client('bedrock-runtime')).bind_tools([db_query_tool], tool_choice="db_query_tool")
# Assemble query-checking chain to be used as a Tool
query_checker = p.query_check_prompt | chat_model_with_query_tool
#test_query_checker(query_checker)
def model_check_query(state: ChatQueryState) -> dict[str, list[AIMessage]]:
"""
Use this tool to double-check the correctness of your query and execute it when deemed correct.
"""
return {
"messages": [
query_checker.invoke(
{
"messages": [state["messages"][-1]],
"database_service": "AWS Athena",
"dialect": "Trino"
}
)
]
}
# Assemble query-generating chain.
chat_model_with_end_tool = ChatBedrock(model_id=awsutil.default_chat_model_id, client=awsutil.get_client('bedrock-runtime')).bind_tools(
[tool.SubmitFinalAnswer]
)
query_generator = p.query_gen_prompt | chat_model_with_end_tool
def query_gen_node(state: ChatQueryState):
message = query_generator.invoke(
{
'messages': state_messages_excluding_type(state, SystemMessage)['messages'],
'database_service': "AWS Athena",
'dialect': "Trino"
}
)
# Sometimes the LLM will hallucinate and call the wrong tool. We need to catch this and return an error message.
tool_messages = [] # <<<BREAKPOINT SET HERE>>>
if message.tool_calls:
for tc in message.tool_calls:
if tc["name"] != "SubmitFinalAnswer":
tool_messages.append(
ToolMessage(
content=f"Error: The wrong tool was called: {tc['name']}. Please fix your mistakes. Remember to only call SubmitFinalAnswer to submit the final answer. Generated queries should be outputted WITHOUT a tool call.",
tool_call_id=tc["id"],
)
)
else:
tool_messages = []
return {"messages": [message] + tool_messages}
workflow.add_node("first_tool_call", first_tool_call)
# Add nodes for the first two tools
workflow.add_node("list_tables_tool", tool_util.create_tool_node_with_fallback([list_tables_tool]))
# Add a node for a model to choose the relevant tables based on the question and available tables
workflow.add_node(
"model_get_schema",
lambda state: {
"messages": [model_get_schema.invoke(state["messages"])]
},
)
# A node to run the get_schema_tool if requested.
workflow.add_node("get_schema_tool", tool_util.create_tool_node_with_fallback([get_schema_tool]))
workflow.add_node("query_gen", query_gen_node)
# # Add a node for the model to check the query before executing it
# workflow.add_node("correct_query", model_check_query)
# # Add node for executing the query
# workflow.add_node("execute_query", tool_util.create_tool_node_with_fallback([db_query_tool]))
# Specify the edges between the nodes
workflow.add_edge(START, "first_tool_call")
workflow.add_edge("first_tool_call", "list_tables_tool")
workflow.add_edge("list_tables_tool", "model_get_schema")
workflow.add_edge("model_get_schema", "get_schema_tool")
workflow.add_edge("get_schema_tool", "query_gen")
# workflow.add_conditional_edges(
# "query_gen",
# should_continue,
# )
# workflow.add_edge("correct_query", "execute_query")
# workflow.add_edge("execute_query", "query_gen")
# Compile the workflow into a runnable
compiled_graph: "CompiledStateGraph" = workflow.compile() |
Beta Was this translation helpful? Give feedback.
-
|
One issue I encountered with the proposed approach is that the Here is my implementation using a simple ReAct Agent design to call the tools. I realized the This could be a good start for simpler use cases.
from langchain_community.agent_toolkits import SQLDatabaseToolkit
from dotenv import load_dotenv
from langchain_community.utilities import SQLDatabase
from sqlalchemy import create_engine
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import ToolNode, tools_condition
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_openai import ChatOpenAI
from typing_extensions import TypedDict
from typing import Annotated
from langgraph.graph import END, StateGraph, START
from langgraph.graph.message import add_messages
# Load environment variables
load_dotenv()
db_file = "chinook.db"
engine = create_engine(f"sqlite:///{db_file}")
db = SQLDatabase(engine=engine)
# SQL Manipulation Tools
toolkit = SQLDatabaseToolkit(db=db, llm=ChatOpenAI(model="gpt-4o"))
sql_db_toolkit_tools = toolkit.get_tools()
query_gen_system = """
You are an agent designed to interact with SQL database.
Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.
<instructions>
1. Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 10 results.
2. You can order the results by a relevant column to return the most interesting examples in the database. Never query for all the columns from a specific table, only ask for the relevant columns given the question.
3. You have access to tools for interacting with the database. Use your tools to fetch the database scheema, so that you can generate the query based on the schema. Only use the information returned by the tools to construct your final answer.
4. You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.
5. Once you are able to provide an answer from the data fectched from the database, don't call any tools again.
</instructions>
<Restrictions>
1. DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.
2. DO NOT MAKE UP ANSWER.
</Restrictions>
"""
query_gen_prompt = ChatPromptTemplate.from_messages(
[
("system",
query_gen_system
),
MessagesPlaceholder(
variable_name="messages"),
]
)
query_gen_model = query_gen_prompt | ChatOpenAI(
model="gpt-4o", temperature=0).bind_tools(tools=sql_db_toolkit_tools)
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
def query_gen_node(state: State):
return {"messages": [query_gen_model.invoke(state["messages"])]}
graph_builder.add_node("query_gen", query_gen_node)
query_gen_tools_node = ToolNode(tools=sql_db_toolkit_tools)
graph_builder.add_node("query_gen_tools", query_gen_tools_node)
graph_builder.add_conditional_edges(
"query_gen",
tools_condition,
{"tools": "query_gen_tools", END: END},
)
graph_builder.add_edge("query_gen_tools", "query_gen")
graph_builder.set_entry_point("query_gen")
graph = graph_builder.compile()
# print(graph.get_graph().draw_mermaid())
initial_query = str("Which sales agent made the most in sales in 2009?")
for event in graph.stream(
{"messages": [("user", initial_query)]}
):
print(event)Results: |
Beta Was this translation helpful? Give feedback.
-
|
If a database has 100 tables then this approach isnt the right one. You dont want to pass on details of 100 tables as prompt. Also we are simply relying on table names or columnnames. thats not always the case, in real scenario. your names can vary. So instead of using these pre built tools. use agents wiht rag to fetch the relevant things |
Beta Was this translation helpful? Give feedback.
-
|
@fnusatvik07 what do u suggest then ? |
Beta Was this translation helpful? Give feedback.
-
|
you can add few extra nodes to the graph to decide which tables to use for particular query and load schema description only for that tables into sql agent graph. yet we have to give description to router to get better decision on selecting tables. i'm not sure what type of rag @fnusatvik07 mentioned but if we need to get data from sql database we have to use sql someway or another i think 🤷🏻♂️ |
Beta Was this translation helpful? Give feedback.
-
|
Great example, thanks. It worked fine with OpenAi, but when I switched to Ollama I did not get it working. Seems that it's not for some reason doing the schema call at all. Not sure what is the best way to debug this. Do I need to have different prompts for Ollama. (llama3.2). I only have one table, promo_prices, which contains column store. Here's what LangSmith says to me at that point: AI {} TOOL Rendered Output |
Beta Was this translation helpful? Give feedback.
-
|
Guys, I want to add memory to this agent, but I am not able to. Can you please suggest to me the way to move forward? I am Using OPENAI API KEY for my model. |
Beta Was this translation helpful? Give feedback.
-
|
Can someone please explain how the SubmitFinalAnswer tool works? I don't understand how it can be a tool when it's not a function that returns anything: ` ` |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the example, I am working with something similar: |
Beta Was this translation helpful? Give feedback.
-
|
I am getting this error BadRequestError Traceback (most recent call last) 17 frames BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': 'db_query_tool{"query": "SELECT SUM(amount) FROM transactions"}'}} I am using Groq llama-3.3-70b-versatile (supports tool use) https://console.groq.com/docs/tool-use Please help |
Beta Was this translation helpful? Give feedback.
and others
-
langgraph/tutorials/sql-agent/
Build language agents as graphs
https://langchain-ai.github.io/langgraph/tutorials/sql-agent/
Beta Was this translation helpful? Give feedback.
All reactions