Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for tools returning images #2367

Open
alexandreroman opened this issue Mar 3, 2025 · 0 comments
Open

Add support for tools returning images #2367

alexandreroman opened this issue Mar 3, 2025 · 0 comments

Comments

@alexandreroman
Copy link

Expected Behavior

When using tools, users might want to generate images with the Java API (leveraging BufferedImage, Graphics2D, ImageIO, etc.).

In this case, Spring AI should take care of converting those images to a text based structure.
LLM usually supports base64 encoded images.

@Tool
BufferedImage generateImage() {
  var img = BufferedImage(64, 64, BufferedImage.TYPE_4BYTE_ABGR);
  var g = img.createGraphics();
  g.setColor(Color.WHITE);
  g.fillRect(0, 0, 64, 64);
  g.dispose();
  return img;
}

Spring AI should identify the return type, and convert the source image to a JSON structure holding the base64 encoded image. The PNG format might be used by the default implementation.

This should be the default behavior.

In any case, the user can override this mechanism by providing a ToolCallResultConverter implementation in the Tool annotation.

Current Behavior

Spring AI doesn't support tools returning images.

There is a pluggable mechanism available in the Tool annotation with ToolCallResultConverter.

A default implementation also exists: DefaultToolCallResultConverter.

However, this implementation currently ignores images.

Context

As a workaround, users have to change their tool signatures, in order to explicitly return a String value,
taking care of the base64 encoding in the tool implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant