Agent API doesn't return token usage information

[erik-balfe]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

typescript
import { ChatGroq } from "npm:@langchain/groq@0.0.12";
import { ChatPromptTemplate } from "npm:@langchain/core@0.2.8/prompts";
import { HumanMessage, SystemMessage } from "npm:@langchain/core@0.2.8/messages";

import { WikipediaQueryRun } from "npm:@langchain/community@0.2.12/tools/wikipedia_query_run";
import { AgentExecutor, createToolCallingAgent } from "npm:langchain@0.2.5/agents";
const GROQ_API_KEY = 'exampleKey'

async function exampleAgentRequest() {
  const llmClient = new ChatGroq({ apiKey: GROQ_API_KEY });
  const chatMessages = [
    new SystemMessage({ content: "You are a helpful assistant." }),
    new HumanMessage({ content: "Who is the current president of France?" }),
  ];
  const wikipediaTool = new WikipediaQueryRun({
    topKResults: 3,
    maxDocContentLength: 20000,
  });
  const tools = [wikipediaTool];
  const promptTemplate = ChatPromptTemplate.fromMessages(chatMessages);
  const agent = createToolCallingAgent({ llm: llmClient, tools, prompt: promptTemplate });
  const agentExecutor = new AgentExecutor({ agent, tools, returnIntermediateSteps: true });
  const result = await agentExecutor.invoke({
    template: promptTemplate,
  });
  console.log(result);

  return result;
}

await exampleAgentRequest()

Provided code returns {input, output} object with text values.
But the important part from API is lost - I can not get usage data.

### Description

I'm attempting to use LangChain's agent functionality to get answers from an LLM that can also utilize tools (e.g., Wikipedia). However, I've encountered an issue where I can't retrieve the token usage data, which is crucial for managing costs with paid API services.

In the current LangChain library (version 0.2.5), the agent implementation seems to omit or not provide the usage data that is typically returned from API requests in agent mode.

When making standard calls via llmClient.invoke('some question'), the returned value includes usage information. However, agents only return "input" and "output" texts. I've thoroughly examined the API reference but couldn't find any information about accessing the usage data. The API reference doesn't provide clear guidance on this matter, and querying an LLM chatbot for assistance hasn't been helpful.

I've attached a code example that demonstrates the default way to use agents, as shown in the documentation. This code is similar to what an LLM chatbot provides for a comprehensive question about agent usage.

Could someone please clarify:
1. Is the usage information hidden somewhere in the agent response?
2. Is this an unintentional omission in the implementation?
3. Does the library intentionally hide this information from users in the agent API?
4. Is there a workaround to retrieve the LLM API results, including usage data?

I haven't found any relevant information in the documentation. While I plan to examine the source code to locate where this data might be omitted and attempt to fix it, I believe it would be more efficient and productive if addressed by the library's original developers.

This issue is significant as it affects a basic developer need: accessing important data from the API response, specifically token usage information for cost management.

### System Info

The system information is not particularly relevant in this case. The described behavior appears to be a characteristic of the library that doesn't meet a fundamental developer need: accessing important data from the API response.

[dosubot[bot]]

Hey there, @erik-balfe! I'm here to help you with your coding adventures. Let's squash those bugs and unravel the mysteries of software together! 🤖

To retrieve token usage data from LangChain's agent functionality, you can use the usage_metadata property from the response object. Here is an example of how you can implement this:

import { ChatGroq } from "npm:@langchain/groq@0.0.12";
import { ChatPromptTemplate } from "npm:@langchain/core@0.2.8/prompts";
import { HumanMessage, SystemMessage } from "npm:@langchain/core@0.2.8/messages";
import { WikipediaQueryRun } from "npm:@langchain/community@0.2.12/tools/wikipedia_query_run";
import { AgentExecutor, createToolCallingAgent } from "npm:langchain@0.2.5/agents";

const GROQ_API_KEY = 'exampleKey';

async function exampleAgentRequest() {
  const llmClient = new ChatGroq({ apiKey: GROQ_API_KEY });
  const chatMessages = [
    new SystemMessage({ content: "You are a helpful assistant." }),
    new HumanMessage({ content: "Who is the current president of France?" }),
  ];
  const wikipediaTool = new WikipediaQueryRun({
    topKResults: 3,
    maxDocContentLength: 20000,
  });
  const tools = [wikipediaTool];
  const promptTemplate = ChatPromptTemplate.fromMessages(chatMessages);
  const agent = createToolCallingAgent({ llm: llmClient, tools, prompt: promptTemplate });
  const agentExecutor = new AgentExecutor({ agent, tools, returnIntermediateSteps: true });
  
  const result = await agentExecutor.invoke({
    template: promptTemplate,
  });

  // Assuming the response includes usage metadata
  console.log(result.usage_metadata);

  return result;
}

await exampleAgentRequest();

This code will log the token usage data, including input_tokens, output_tokens, and other relevant metadata, to the console ^[1][2].

To continue talking to Dosu, mention @dosu.

[erik-balfe]

const agentExecutor = new AgentExecutor({ agent, tools, returnIntermediateSteps: true });

const result = await agentExecutor.invoke({
template: promptTemplate,
});

// Assuming the response includes usage metadata
console.log(result.usage_metadata);

// Unfortunately, usage metadata is not available in the result

It would indeed be good if accessing result.usage_metadata worked as suggested! However, the current implementation doesn't provide this functionality. The result object from agentExecutor.invoke() only contains:

1. An input string (the original query)
2. An output string (the final answer)
3. Optionally, intermediateSteps array (if enabled) with data about the execution steps

None of these include the token usage information we're seeking. This lack of usage data in the agent's response is the core issue we're facing, as it prevents effective monitoring and management of API usage and associated costs when working with LangChain agents.

To my mind the usage data is supposed to present for every step object in intermediateSteps array.
Now it has something like that:

 {
    "action": {
      "tool": "wikipedia-api",
      "toolInput": {
        "input": "president of france"
      },
      "toolCallId": "call_e52k",
      "log": "Invoking \"wikipedia-api\" with {\"input\":\"president of france\"}\n",
      "messageLog": [
        {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain_core",
            "messages",
            "AIMessageChunk"
          ],
          "kwargs": {
            "content": "",
            "additional_kwargs": {
              "tool_calls": [
                {
                  "id": "call_e52k",
                  "type": "function",
                  "function": {
                    "name": "wikipedia-api",
                    "arguments": "{\"input\":\"president of france\"}"
                  }
                }
              ]
            },
            "tool_call_chunks": [
              {
                "name": "wikipedia-api",
                "args": "{\"input\":\"president of france\"}",
                "id": "call_e52k",
                "index": 0
              }
            ],
            "tool_calls": [
              {
                "name": "wikipedia-api",
                "args": {
                  "input": "president of france"
                },
                "id": "call_e52k"
              }
            ],
            "invalid_tool_calls": [],
            "response_metadata": {}
          }
        }
      ]
    },
    "observation": "text"
}

[erik-balfe]

I've discovered that this issue is specific to the Groq provider. When using other providers, such as Anthropic, the usage data is indeed available. For example, with Anthropic, the usage information can be found in:

intermediateSteps[0].action.messageLog.kwargs.additional_kwargs.usage

This suggests that the problem likely lies with either the Groq library or with Groq's backend, which may not return usage data specifically for tool calls.

To clarify for other users and developers:

1. The issue of missing usage data appears to be isolated to the Groq provider.
2. Other providers, like Anthropic, do include usage information in the agent's response.
3. When using non-Groq providers, developers can access usage data through the intermediateSteps array enabling it in advance in AgentExecutor constructor.

Given this new information, it might be worth investigating:

1. Whether this is a known limitation of the Groq API for tool calls.
2. If there's a way to enable or request usage data from Groq specifically for agent/tool interactions.
3. Whether the LangChain implementation for Groq can be updated to include usage data if it's available from the API but not currently being passed through.

This update narrows down the scope of the original issue and provides a potential workaround for users who have the flexibility to switch to a different provider. However, for those specifically using or requiring Groq, the problem of accessing usage data in agent calls remains unresolved.

Suggestion to LangChain library developers:
It would be extremely helpful if you could add an example in the documentation that demonstrates how to retrieve usage data when working with agents. The current method of enabling intermediateSteps and then navigating through its complex structure to find information in kwargs, messageLog, etc., is not intuitive or user-friendly. Consider adding a more straightforward method to access this data, such as a dedicated getUsageData() method on the agent executor or including usage information directly in the top-level result object. This would greatly improve the developer experience and make it easier for users to monitor and manage their API usage across different providers.