Advancing Tool Support in Spring AI

[ThomasVitale]

This issue presents ideas and suggestions for consolidating and advancing tool support in Spring AI, before reaching the first GA version.

Tool Registration

Tools are modelled via the ToolCallback API. By default, Spring AI supports registering tools from methods and from functional objects.

Methods

In the context of Java, a tool can naturally be modelled as a method. Spring AI support registering methods as tools via the MethodToolCallback API.

Declarative tool registration is supported via the @Tool annotation on methods.

class MyTools {

    @Tool(description = "Get the list of books written by the given author available in the library")
    List<Book> booksByAuthor(String author) {
        return bookService.getBooksByAuthor(new Author(author));
    }

}

@GetMapping("/chat")
String chat(String authorName) {
    return chatClient.prompt()
            .user("What books written by %s are available in the library?".formatted(authorName))
            .tools(new MyTools())
            .call()
            .content();
}

Functions

In special cases, there might be a wish for modelling tools as functional objects (Function/BiFunction/Supplier/Consumer). Spring AI support registering functional objects as tools via the FunctionToolCallback API.

@GetMapping("/chat")
String chat(String authorName) {
    return chatClient.prompt()
            .user("What books written by %s are available in the library?".formatted(authorName))
            .tools(FunctionToolCallback.builder("booksByAuthor", author -> bookService().getBooksByAuthor(author))
                        .description("Get the list of books written by the given author available in the library")
                        .inputType(BookService.Author.class)
                        .build())
            .call()
            .content();
}

Tool Execution

By default, tool execution is performed transparently by a ChatModel object using the ToolCallingManager API. In case of errors while executing tools, a ToolCallExceptionConverter is used to process the exception.

If you need full control over the tool execution, for example by asking the user permission before each tool execution, you can disable the internal tool execution feature and manage the tool calling workflow yourself using the ToolCallingManager API directly.

ChatOptions chatOptions = ToolCallingChatOptions.builder()
			.toolCallbacks(ToolCallbacks.from(tools))
			.internalToolExecutionEnabled(false)
			.build();

Prompt prompt = new Prompt("What's the weather in Copenhagen?", chatOptions);

ChatResponse chatResponse = chatModel.call(prompt);

if (chatResponse.hasToolCalls()) {
    List<Message> messages = toolCallingManager.executeToolCalls(prompt, chatResponse);
}

Prompt secondPrompt = new Prompt(messages, chatOptions);

ChatResponse secondChatResponse = chatModel.call(secondPrompt);

Tool Resolution

Instead of providing tools as ToolCallbacks to ChatModel or ChatClient, you have the option to pass just tool names and let the framework resolve them into ToolCallbacks dynamically.

By default, Spring AI support dynamic tool resolution from the following:

• Functional objects registered as beans with the Spring context (Function/BiFunction/Supplier/Consumer).
• ToolCallbacks registered as beans with the Spring context.

Additional resolution strategies can be provided via the ToolCallbackResolver API.

Example:

@Configuration(proxyBeanMethods = false)
class Functions {

    @Bean
    @Description("Get the list of books written by the given author available in the library")
    public Function<Author, List<Book>> booksByAuthor(BookService bookService) {
        return author -> bookService.getBooksByAuthor(author);
    }

}

Tasks

• Advancing Tool Support - Part 1 #2064
• Advancing Tool Support - Part 2 #2085
• Advancing Tool Support - Part 3 #2121
• Advancing Tool Support - Part 4 #2148
• Advancing Tool Support - Part 5 #2162
• Advancing Tool Support - Part 6 #2169

[hansdesmet]

This would make tooling great !
LangChain4J also has a @Tool annotation with the same purpose:
https://docs.langchain4j.dev/tutorials/tools/#high-level-tool-api

[Yougoss]

Is there a plan to support Function parameters properties description? to make ai better understand how to get the corresponding parameters from the context.

From openai doc, there is a description field for each function parameters property

[ogbozoyan]

When a chat model determines that a certain tool needs to be called

It's not quite clear how the model realizes that it's “time” to call in the tool
Is it possible to predefine the flow, when model is going to call a method, it checks with @Description from the user if he wants it and on positive answer starts to execute the method ?

[ThomasVitale]

@Yougoss there is already some partial support based on whether you're using methods or functions, and whether it's a nested object or not. I've been working on improving that logic to ensure consistency of behaviour across all those scenarios, and writing documentation for it. You can see the draft PR here: #2162 Feel free to share your feedback on the solution!

The idea implemented in that PR is such that descriptions and required properties can be specified using one of the following supported annotations:

• @ToolParam(required = ..., description = ...) (from Spring AI)
• @JsonProperty(required = ...) (from Jackson)
• @Schema(required = ..., description = ...) (from Swagger)
• @Nullable (from Spring Framework)

Notice that this is just a proposal. The final solution might differ.

[ThomasVitale]

All the work included in this issue has been delivered. It's available in 1.0.0-SNAPSHOT and will be included in the upcoming 1.0.0-M6 version.

Full documentation of the new tool calling features: https://docs.spring.io/spring-ai/reference/api/tools.html