Not routing to correct agent in langgraph

[palaklanger]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

llm=load_llm()

members = ["Jira_userstory_analyst","Product Quality Engineer","General_agent"]
system_prompt = (
    "You are a supervisor tasked with managing"
    " following workers:  {members}. Product Quality Engineer worker has details about failure cases, serial numbers affected and their issue classifications. Given the following user request,"
    " respond with the worker to act next. Each worker will perform a"
    " task and respond with their results and status. When finished,"
    " respond with FINISH."
)



# Our team supervisor is an LLM node. It just picks the next agent to process
# and decides when the work is completed
options = ["FINISH"] + members


# Using openai function calling can make output parsing easier for us
function_def = {
    "name": "route",
    "description": "Select the next role.",
    "parameters": {
        "title": "routeSchema",
        "type": "object",
        "properties": {
            "next": {
                "title": "Next",
                "anyOf": [
                    {"enum": options},
                ],
            }
        },
        "required": ["next"],
    },
}


prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder(variable_name="messages"),
        (
            "system",
            "Given the conversation above, who should act next?"
            " Or should we FINISH? Select one of: {options}",
        ),
    ]
).partial(options=str(options), members=", ".join(members))


supervisor_chain = (
    prompt
    | llm.bind_functions(functions=[function_def], function_call="route")
    | JsonOutputFunctionsParser()
)

Description

There are three members that we are using in Langgraph:["Jira_userstory_analyst","Product Quality Engineer","General_agent"]

Supervisor prompt: You are a supervisor tasked with managing following workers: Jira_userstory_analyst, Product Quality Engineer, General_agent. Product Quality Engineer worker has details about failure cases, serial numbers affected and their issue classifications. Given the following user request, respond with the worker to act next. Each worker will perform a task and respond with their results and status. When finished, respond with FINISH. System: Given the conversation above, who should act next? Or should we FINISH? Select one of: ['FINISH', 'Jira_userstory_analyst', 'Product Quality Engineer', 'General_agent']"

Question prompt: What are serial number affected by issue classification by x in y?

In this scenario, it should go to Product Quality Engineer , but it goes to Jira_userstory_analyst.
Please advise

System Info

openai==1.35.3
langchain==0.2.5
langchain-openai==0.1.9
langchain-community==0.2.5
langchain_experimental==0.0.60
tabulate==0.9.0
snowflake-sqlalchemy==1.5.1

[dosubot[bot]]

The issue you're encountering might be due to the way the JsonOutputFunctionsParser is handling the output from the LLM. Here are a few steps to troubleshoot and potentially resolve the issue:

1. Ensure Correct Function Binding: Verify that the function definition and binding are correctly set up. The function_call should match the function name in the function_def.

2. Check the Supervisor Prompt: Ensure that the supervisor prompt is clear and unambiguous. The prompt should clearly indicate that the "Product Quality Engineer" should handle the request related to serial numbers and issue classifications.

3. Debug the Output: Add logging or print statements to debug the output from the LLM and the parsed result. This can help identify if the issue is with the LLM's response or the parsing logic.

4. Update to Latest Versions: Ensure you are using the latest versions of langchain, openai, and related packages, as there might be bug fixes or improvements that address this issue.

Here is a revised version of your code with added debug statements:

import json
from langchain import load_llm, ChatPromptTemplate, MessagesPlaceholder, JsonOutputFunctionsParser

llm = load_llm()

members = ["Jira_userstory_analyst", "Product Quality Engineer", "General_agent"]
system_prompt = (
    "You are a supervisor tasked with managing"
    " following workers:  {members}. Product Quality Engineer worker has details about failure cases, serial numbers affected and their issue classifications. Given the following user request,"
    " respond with the worker to act next. Each worker will perform a"
    " task and respond with their results and status. When finished,"
    " respond with FINISH."
)

options = ["FINISH"] + members

function_def = {
    "name": "route",
    "description": "Select the next role.",
    "parameters": {
        "title": "routeSchema",
        "type": "object",
        "properties": {
            "next": {
                "title": "Next",
                "anyOf": [
                    {"enum": options},
                ],
            }
        },
        "required": ["next"],
    },
}

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder(variable_name="messages"),
        (
            "system",
            "Given the conversation above, who should act next?"
            " Or should we FINISH? Select one of: {options}",
        ),
    ]
).partial(options=str(options), members=", ".join(members))

supervisor_chain = (
    prompt
    | llm.bind_functions(functions=[function_def], function_call="route")
    | JsonOutputFunctionsParser()
)

# Debugging: Print the output from the LLM and the parsed result
def debug_supervisor_chain(input_messages):
    result = supervisor_chain(input_messages)
    print("LLM Output:", result)
    try:
        parsed_result = json.loads(result)
        print("Parsed Result:", parsed_result)
    except json.JSONDecodeError as e:
        print("Failed to parse JSON:", e)

# Example usage
input_messages = [
    {"role": "user", "content": "What are serial number affected by issue classification by x in y?"}
]
debug_supervisor_chain(input_messages)

This code includes a debug_supervisor_chain function that prints the output from the LLM and attempts to parse it as JSON. This can help you identify if the issue is with the LLM's response or the parsing logic.

Additionally, ensure that the JsonOutputFunctionsParser is correctly handling the output. If the JSON is not well-formed, it might raise a JSONDecodeError or TypeError ^[1][2].

If the issue persists, consider reviewing the known compatibility issues between the versions of openai, langchain, and langchain-openai that could affect the routing logic ^[3].

To continue talking to Dosu, mention @dosu.