Using tools with ConversationalRetrievalChain

[anup-a]

I have a custom chatbot setup using ConversationalRetrievalChain, but so far I'm struggling to integrate tools within it. If I go with RAG approach I don't get as accurate results as my previous setup. I would appreciate any help here. Thanks.

This is my current attempt, and it does normal qa seemlessly, but for the tasks which requires tools, it sorts of hangs up. I'm not sure if this is even the correct approach.

      self.retriever = self.load_retriever()
      _template = """Given the following conversation and a follow-up, 
      understand and rephrase the follow-up input to be a standalone question if needed.
      If it's not a question return "Hmm, I'm not sure".
      Preserve the original question in the answer sentiment during rephrasing.
      =========
      Chat History:
      {chat_history}
      =========
      Follow Up Input: {question}
      Standalone question:"""
      self.condense_question_prompt = PromptTemplate.from_template(_template)


      self.template = """{base_prompt}
      
      Given a question, try to answer it using the context below, and if you cannot answer, 
      or If the question is not actually a question then, politely respond or say "Hmm, I'm not sure" but don't fabricate answer.

      Lastly answer the question as if you were a support person who loves helping customers.
      Question: {question}
      =========
      {context}
      =========
      Answer in Markdown:"""
  
      self.prompt = PromptTemplate(
          input_variables=["context", "question"],
          template=self.template,
          partial_variables={"base_prompt": self.chatbot.base_prompt} 
      )

      logger.info("Completed setup for qa_over_sources bot")

      search = SerpAPIWrapper(serpapi_api_key=settings.SERPAPI_API_KEY)

      self.tools = [
          Tool(
              name="Search",
              func=search.run,
              description="useful for when you need to answer questions about current events. You should ask targeted questions"
          ),
          WorldNewsTool()
      ]

      llm_with_tools = self.llm.bind_tools(self.tools)

      self.chain = ConversationalRetrievalChain.from_llm(
          llm=llm_with_tools,
          memory=self.memory,
          retriever=self.retriever,
          return_source_documents=True,
          verbose=True,
          condense_question_llm = ChatOpenAI(temperature=0, model_name="gpt-4-1106-preview"),
          condense_question_prompt=self.condense_question_prompt,
          combine_docs_chain_kwargs={'prompt': self.prompt}
      )

[jjfeore]

I'm not entirely sure what you're trying to do. Are you trying to create a retrieval chain that uses tools to adjust the questions the user answers, OR are you trying to allow the user to ask a question and have that question be answered by either the search tool, retrieval, or the WorldNewsTool as appropriate?

If it's the latter (i.e. letting the user ask a question and have it answered by the most appropriate of those three sources), you should set up an agent instead. This is fairly simple to do and the docs on it are pretty straightforward and should be enough to help you through it. You could bind the search tool, WorldNewsTool, and a retrieval tool (you can generate this easily with create_retriever_tool) to the LLM, and then have the agent run queries through that LLM.

If it's the former (i.e. you want the tools to modify possibly modify the user's question and then perform retrieval), then I suspect the issue is that you need to adjust the output of your tools. For instance, the Search tool as it's presently written, would just return an answer (as opposed to a prompt that might be helpful for retrieval) and the description is also confusing enough that the LLM probably doesn't understand what it's meant to use it for. You'd also probably need to bind the tools to the condense_question_llm or just write some custom chain to handle this.

Even if you're just trying to do the former, using an agent is probably still the easiest way to do this. The only difficulty would be that the agent wouldn't necessarily always use the retriever, if that was a goal of yours.

[anup-a]

Thanks for quick response @jjfeore. My requirement is to use the tools to improve the answers while always using chat history and context (retriever), which is why the agent with create_retriever_tool did not work well for me.

After debugging with chain logs (with my current code) , I realised that tool is being detected, and can be found in tool_calls object, but is not invoked automatically. I found this doc which mentions how to pass results to model, but I'm not sure at which point I should make that call.

[anup-a]

Is there a way to make sure the agent always retriever, alongside other tools if necessary ? cc @dosu

[jjfeore]

I don't think ConversationalRetrievalChain is going to work the way you want it to work. It's always going to do two things: condense the question with chat history and then chain that into a retrieval chain that retrieves documents related to the question and then prompts the LLM with those documents and the question.

There is a way to force tool calls if you're using one of the models that supports the "tool_choice" parameter. I'd still just recommend not forcing the retriever tool call and instead using an agent and including something like, "You must ALWAYS use the retriever tool when answering questions" in the system instructions. That will still generally result in it using retriever, but not using it when prompted with something completely irrelevant, e.g. "Hi. How are you?"