Getting prompt token count before calling invoke method on agent.

[Sagar0070]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Sequence

from langchain_community.tools.convert_to_openai import format_tool_to_openai_tool
from langchain_core.language_models import BaseLanguageModel
from langchain_core.prompts.chat import ChatPromptTemplate
from langchain_core.runnables import Runnable, RunnablePassthrough
from langchain_core.tools import BaseTool

from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser


def create_openai_tools_agent(
    llm: BaseLanguageModel, tools: Sequence[BaseTool], prompt: ChatPromptTemplate
) -> Runnable:
    """Create an agent that uses OpenAI tools.

    Args:
        llm: LLM to use as the agent.
        tools: Tools this agent has access to.
        prompt: The prompt to use. See Prompt section below for more on the expected
            input variables.

    Returns:
        A Runnable sequence representing an agent. It takes as input all the same input
        variables as the prompt passed in does. It returns as output either an
        AgentAction or AgentFinish.

    Example:

        .. code-block:: python

            from langchain import hub
            from langchain_community.chat_models import ChatOpenAI
            from langchain.agents import AgentExecutor, create_openai_tools_agent

            prompt = hub.pull("hwchase17/openai-tools-agent")
            model = ChatOpenAI()
            tools = ...

            agent = create_openai_tools_agent(model, tools, prompt)
            agent_executor = AgentExecutor(agent=agent, tools=tools)

            agent_executor.invoke({"input": "hi"})

            # Using with chat history
            from langchain_core.messages import AIMessage, HumanMessage
            agent_executor.invoke(
                {
                    "input": "what's my name?",
                    "chat_history": [
                        HumanMessage(content="hi! my name is bob"),
                        AIMessage(content="Hello Bob! How can I assist you today?"),
                    ],
                }
            )

    Prompt:

        The agent prompt must have an `agent_scratchpad` key that is a
            ``MessagesPlaceholder``. Intermediate agent actions and tool output
            messages will be passed in here.

        Here's an example:

        .. code-block:: python

            from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

            prompt = ChatPromptTemplate.from_messages(
                [
                    ("system", "You are a helpful assistant"),
                    MessagesPlaceholder("chat_history", optional=True),
                    ("human", "{input}"),
                    MessagesPlaceholder("agent_scratchpad"),
                ]
            )
    """
    missing_vars = {"agent_scratchpad"}.difference(prompt.input_variables)
    if missing_vars:
        raise ValueError(f"Prompt missing required variables: {missing_vars}")

    llm_with_tools = llm.bind(
        tools=[format_tool_to_openai_tool(tool) for tool in tools]
    )

    agent = (
        RunnablePassthrough.assign(
            agent_scratchpad=lambda x: format_to_openai_tool_messages(
                x["intermediate_steps"]
            )
        )
        | prompt
        | llm_with_tools
        | OpenAIToolsAgentOutputParser()
    )
    return agent

Description

Say i created a agent_executor using the below code:
agent = create_openai_tools_agent(chat_llm, tools, prompt)
agent_executor = AgentExecutor(
agent=agent, tools=tools, verbose=False, memory=memory, return_intermediate_steps=True, early_stopping_method="generate",
max_iterations=CUSTOMER_BOT_CONFIG["max_iterations"],
max_execution_time=CUSTOMER_BOT_CONFIG["max_execution_time"],
)

with get_openai_callback() as cb:  gets us the token usage details post the agent_executor.ainvoke call. 

Is there any way we can get the prompt_tokens only, before even invoking the agent_executor using packages like tiktoken. 

System Info

langchain==0.1.3
langchain-community==0.0.15
langchain-core==0.2.0
langchain-experimental==0.0.49
langchain-openai==0.0.3

[dosubot[bot]]

Hey @Sagar0070! I'm here to assist you with any bugs, questions, or contributions you have in mind. Let's tackle this together!

Yes, you can get the prompt_tokens before invoking the agent_executor using the tiktoken package. The get_num_tokens method can be used to achieve this. Here is the relevant code snippet:

def import_tiktoken() -> Any:
    """Import tiktoken for counting tokens for OpenAI models."""
    return guard_import("tiktoken")

def get_num_tokens(string: str, openai_model_name: str) -> int:
    """Calculate num tokens for OpenAI with tiktoken package.

    Official documentation: https://github.com/openai/openai-cookbook/blob/main
                            /examples/How_to_count_tokens_with_tiktoken.ipynb
    """
    tiktoken = import_tiktoken()

    encoding = tiktoken.encoding_for_model(openai_model_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

Additionally, the LangChain framework provides methods to get token IDs and count tokens for messages using the tiktoken package. Here is an example of how you can use these methods:

def get_token_ids(self, text: str) -> List[int]:
    """Get the tokens present in the text with tiktoken package."""
    if sys.version_info[1] <= 7:
        return super().get_token_ids(text)
    _, encoding_model = self._get_encoding_model()
    return encoding_model.encode(text)

def get_num_tokens_from_messages(self, messages: List[BaseMessage]) -> int:
    """Calculate num tokens for gpt-3.5-turbo and gpt-4 with tiktoken package."""
    if sys.version_info[1] <= 7:
        return super().get_num_tokens_from_messages(messages)
    model, encoding = self._get_encoding_model()
    if model.startswith("gpt-3.5-turbo-0301"):
        tokens_per_message = 4
        tokens_per_name = -1
    elif model.startswith("gpt-3.5-turbo") or model.startswith("gpt-4"):
        tokens_per_message = 3
        tokens_per_name = 1
    else:
        raise NotImplementedError(
            f"get_num_tokens_from_messages() is not presently implemented "
            f"for model {model}."
        )
    num_tokens = 0
    messages_dict = [convert_message_to_dict(m) for m in messages]
    for message in messages_dict:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(str(value)))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3
    return num_tokens

These methods use the tiktoken package to encode the text and return the number of tokens, which represent the prompt_tokens ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[Sagar0070]

The one mentioned above gets us the token only if we have the final prompt. How do i get the final prompt from the agent_executor or the agent.

[dosubot[bot]]

To extract the final prompt from the agent_executor or the agent before invoking it, you can construct the prompt using the PREFIX, FORMAT_INSTRUCTIONS, and SUFFIX provided in the ConversationalChatAgent class. Here is an example of how you can do it:

from tiktoken import encoding_for_model
from langchain.agents.conversational_chat.prompt import PREFIX, SUFFIX
from langchain.agents.conversational_chat.base import ConversationalChatAgent
from langchain_core.language_models import BaseLanguageModel
from langchain_core.tools import BaseTool

# Define the components of the prompt
FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question"""

# Example values for placeholders
tool_names = "search, calculator"
input_question = "What is the capital of France?"
agent_scratchpad = " I need to find the capital of France."

# Construct the final prompt
final_prompt = f"{PREFIX}\n\n{FORMAT_INSTRUCTIONS.format(tool_names=tool_names)}\n\n{SUFFIX.format(input=input_question, agent_scratchpad=agent_scratchpad)}"

# Calculate the prompt tokens using tiktoken
encoding = encoding_for_model("gpt-3.5-turbo")  # Replace with the appropriate model name
prompt_tokens = encoding.encode(final_prompt)

print(f"Final Prompt: {final_prompt}")
print(f"Prompt Tokens: {prompt_tokens}")
print(f"Number of Prompt Tokens: {len(prompt_tokens)}")

This code constructs the final prompt by filling in the placeholders in the PREFIX, FORMAT_INSTRUCTIONS, and SUFFIX strings. It then uses the tiktoken package to encode the prompt and calculate the number of tokens. The PREFIX and SUFFIX are imported from the langchain.agents.conversational_chat.prompt module, ensuring consistency with the LangChain framework ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[Sagar0070]

I want to create a agent using create_openai_tools_agent() provided by langchain.
agent = create_openai_tools_agent(chat_llm, tools, prompt)