Possible design issue with create_retrieval_chain from create_history_aware_retriever

[jlchereau]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Full code at https://github.com/digillia/Digillia-Colab/blob/main/tools/langchain.ipynb

from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain

condense_prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content=SYSTEM_PROMPT),
    HumanMessagePromptTemplate.from_template(CONDENSE_PROMPT),
])
history_aware_retriever = create_history_aware_retriever(llm, retriever, condense_prompt)

context_prompt = ChatPromptTemplate.from_messages([
    SystemMessagePromptTemplate.from_template(SYSTEM_PROMPT + '\n' + CONTEXT_PROMPT),
    HumanMessagePromptTemplate.from_template('{input}')
])
question_answer_chain = create_stuff_documents_chain(llm, context_prompt)

rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

Description

I am currently evaluating various frameworks for a RAG project with small prototypes. Code available at https://github.com/digillia/Digillia-Colab/blob/main/tools/langchain.ipynb. Click the Google Colab button to run in Colab.

I am using create_history_aware_retriever, create_stuff_documents_chain and create_retrieval_chain as described at https://python.langchain.com/docs/use_cases/question_answering/chat_history/#chain-with-chat-history.

The issue I am facing is when I ask a follow-up question like Where?, it does not use the history to understand the question and replies that it cannot answer the question.

I have run the code with set_debug(True) which shows a possible design issue in the way create_history_aware_retriever, create_stuff_documents_chain and create_retrieval_chain are supposed to work together:

1. When I first ask When was Napoleon 1st born?, there is no history, the question is not rephrased and the relevant documents are pulled by history_aware_retriever. The question is answered by question_answer_chain based on these documents added to the context.

2. Then when I ask **Where?", history_aware_retriever properly rephrases the question to Where was Napoleon 1st born? considering the chat history and pulls the relevant documents. But then question_answer_chain tries to answer the original question Where? based on these documents added to the context, but fails.

The problem is question_answer_chain receives the original question instead of the rephrased question based on the chat history. In fact history_aware_retriever should pass both the documents and the rephrased question to question_answer_chain.

I have not found a way to fix this with the LangChain APIs, so this might be a design issue. Please let me know your opinion.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103
Python Version: 3.9.6 (default, Feb 3 2024, 15:58:27)
[Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.1.40
langchain: 0.1.14
langchain_community: 0.0.31
langsmith: 0.1.40
langchain_openai: 0.0.2.post1
langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

Hey @jlchereau! Great to see you diving into the depths of LangChain again. Looking forward to unpacking this issue with you 🚀

Based on your description, it seems like the issue lies in the interaction between the create_history_aware_retriever, create_stuff_documents_chain, and create_retrieval_chain functions. Specifically, the question_answer_chain is not receiving the rephrased question from the history_aware_retriever.

To address this, you could modify the create_history_aware_retriever function to return both the retrieved documents and the rephrased question when there is chat history. Here's a possible modification:

def create_history_aware_retriever(
    llm: LanguageModelLike,
    retriever: RetrieverLike,
    prompt: BasePromptTemplate,
) -> RetrieverOutputLike:
    """Modified to return both documents and rephrased question when there's chat history."""
    if "input" not in prompt.input_variables:
        raise ValueError(
            "Expected `input` to be a prompt variable, "
            f"but got {prompt.input_variables}"
        )

    def retrieve_and_rephrase(input_data):
        # Rephrase the question using the LLM and prompt
        rephrased_question = prompt | llm | StrOutputParser()(input_data)
        # Retrieve documents based on the rephrased question
        documents = retriever(rephrased_question)
        # Return both the rephrased question and the documents
        return {"rephrased_question": rephrased_question, "documents": documents}

    retrieve_documents: RetrieverOutputLike = RunnableBranch(
        (
            # Both empty string and empty list evaluate to False
            lambda x: not x.get("chat_history", False),
            # If no chat history, then we just pass input to retriever and return original question
            (lambda x: {"rephrased_question": x["input"], "documents": retriever(x["input"])}),
        ),
        # If chat history, then we rephrase and retrieve
        retrieve_and_rephrase,
    ).with_config(run_name="chat_retriever_chain")

    return retrieve_documents

This modification ensures that when there is chat history, the question is first rephrased using the provided language model and prompt. The rephrased question is then used to retrieve documents. Both the rephrased question and the retrieved documents are returned. When there is no chat history, the original question is passed directly to the retriever, and both the original question and the retrieved documents are returned.

For create_stuff_documents_chain and create_retrieval_chain, you would need to ensure they are capable of handling the output structure ({"rephrased_question": ..., "documents": ...}) from create_history_aware_retriever. This might involve modifying these functions to extract the rephrased_question and use it in subsequent steps of the chain.

Please note that this is a suggested modification and might not perfectly fit your use case. You might need to adjust it according to your specific requirements. Also, remember to thoroughly test any changes to ensure they work as expected.

Regarding the current implementation of create_history_aware_retriever, create_stuff_documents_chain, and create_retrieval_chain in the LangChain codebase, I'm afraid I don't have that information. However, you can check the source code of LangChain on GitHub to understand their current implementation.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• How to answer prompt with the previous questions context, using langchain.
• Can't make the chat to understand previous context
• Langchain Agent rephrasing questions incorrectly
• libs/langchain/langchain/chains/history_aware_retriever.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jlchereau]

Really impressed by the answering bot, but the suggestion of rewriting langchain's code for create_history_aware_retriever seems like a design issue to me.

[ilyasch199]

Really impressed by the answering bot, but the suggestion of rewriting langchain's code for create_history_aware_retriever seems like a design issue to me.

Did you find a solution for the problem , as i started debugging i found the same problem ?