How do we specify a custom callback or project name for LangSmith to report the tool selection?

[thomaschristopherking]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def get_chain():
    # this works for putting most logs into LangSmith under the project name provided    
    return LLMChain(
            llm=self._chat_model,
            prompt=prompt,
            verbose=True,
            callback_manager=self.get_callback_manager("some_project_name"),
        )

def get_tools(self) -> List[Callable]:
    # This is how we specify tools, and this doesn't seem to work for logging the tool selection
    return [
            StructuredTool.from_function(
                func=self.rewrite_query,
                name="rewrite_query",
                description="...",
                callbacks=self._llm_operator.get_callback_manager("some_project_name"),
            )
    ]

Description

We are trying to use LangChain in a web service that should dynamically send logs under different projects to LangSmith based on the endpoint (in our service) being called.

However because it requires setting the project name in environment variables, this means the project name becomes globally scoped to the whole webserver whereas we want to scope it per-request.

We looked at the LangChain source code and discovered that callbacks are used to send data to LangSmith, and we can specify the LangChain callback with a specific project name before we invoke a chain. However, we can't seem to specify the LangSmith project name for recording the tool decision process.

This is what we expect to see in LangSmith:

This is what we actually see:

System Info

System Information

OS: Windows
OS Version: 10.0.22621
Python Version: 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]

Package Information

langchain_core: 0.2.15
langchain: 0.2.7
langsmith: 0.1.85
langchain_openai: 0.1.8
langchain_text_splitters: 0.2.2

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

Hey there, @thomaschristopherking! I'm Dosu, a bot here to help you with bug fixes, answer your questions, and guide you on how to become a contributor to the repository. I see you've encountered an issue, and I'm here to assist you while we wait for a human maintainer to step in.

To specify a custom callback or project name for LangSmith to report the tool selection, you can use the RunnableConfig class. Here is an example of how to set it up:

from langchain_core.runnables import RunnableConfig

# Define your custom callback or project name
config = RunnableConfig(callbacks=["your_custom_callback"], project_name="your_project_name")

# Use the config in the invoke_tool function
def invoke_tool(
    tool_call_request: ToolCallRequest, config: Optional[RunnableConfig] = None
):
    """A function that we can use to perform a tool invocation.

    Args:
        tool_call_request: a dict that contains the keys name and arguments.
            The name must match the name of a tool that exists.
            The arguments are the arguments to that tool.
        config: This is configuration information that LangChain uses that contains
            things like callbacks, metadata, etc. See LCEL documentation about RunnableConfig.

    Returns:
        output from the requested tool
    """
    tool_name_to_tool = {tool.name: tool for tool in tools}
    name = tool_call_request["name"]
    requested_tool = tool_name_to_tool[name]
    return requested_tool.invoke(tool_call_request["arguments"], config=config)

# Example usage
invoke_tool({"name": "rewrite_query", "arguments": {"query": "example"}}, config=config)

This setup allows you to specify custom callbacks and project names for LangSmith to report the tool selection. Additionally, LangChain provides a comprehensive callbacks system that allows you to hook into various stages of your LLM application, which can be useful for logging, monitoring, streaming, and other tasks. You can subscribe to these events by using the callbacks argument available throughout the API.

For example, you can use the LabelStudioCallbackHandler to send predictions to Label Studio for human evaluation, feedback, and annotation:

from langchain_community.callbacks import LabelStudioCallbackHandler

handler = LabelStudioCallbackHandler(
    api_key='<your_key_here>',
    url='http://localhost:8080',
    project_name='LangChain-%Y-%m-%d',
    mode='prompt'
)

config = RunnableConfig(callbacks=[handler], project_name="your_project_name")

# Example usage with LabelStudioCallbackHandler
invoke_tool({"name": "rewrite_query", "arguments": {"query": "example"}}, config=config)

This example demonstrates how to integrate a custom callback handler like LabelStudioCallbackHandler to report tool selections and other events to Label Studio ^[1][2][3].

To continue talking to Dosu, mention @dosu.