You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using tools, users might want to generate images with the Java API (leveraging BufferedImage, Graphics2D, ImageIO, etc.).
In this case, Spring AI should take care of converting those images to a text based structure.
LLM usually supports base64 encoded images.
@Tool
BufferedImage generateImage() {
var img = BufferedImage(64, 64, BufferedImage.TYPE_4BYTE_ABGR);
var g = img.createGraphics();
g.setColor(Color.WHITE);
g.fillRect(0, 0, 64, 64);
g.dispose();
return img;
}
Spring AI should identify the return type, and convert the source image to a JSON structure holding the base64 encoded image. The PNG format might be used by the default implementation.
This should be the default behavior.
In any case, the user can override this mechanism by providing a ToolCallResultConverter implementation in the Tool annotation.
Current Behavior
Spring AI doesn't support tools returning images.
There is a pluggable mechanism available in the Tool annotation with ToolCallResultConverter.
A default implementation also exists: DefaultToolCallResultConverter.
However, this implementation currently ignores images.
Context
As a workaround, users have to change their tool signatures, in order to explicitly return a String value,
taking care of the base64 encoding in the tool implementation.
The text was updated successfully, but these errors were encountered:
Expected Behavior
When using tools, users might want to generate images with the Java API (leveraging
BufferedImage
,Graphics2D
,ImageIO
, etc.).In this case, Spring AI should take care of converting those images to a text based structure.
LLM usually supports base64 encoded images.
Spring AI should identify the return type, and convert the source image to a JSON structure holding the base64 encoded image. The PNG format might be used by the default implementation.
This should be the default behavior.
In any case, the user can override this mechanism by providing a
ToolCallResultConverter
implementation in theTool
annotation.Current Behavior
Spring AI doesn't support tools returning images.
There is a pluggable mechanism available in the
Tool
annotation withToolCallResultConverter
.A default implementation also exists:
DefaultToolCallResultConverter
.However, this implementation currently ignores images.
Context
As a workaround, users have to change their tool signatures, in order to explicitly return a
String
value,taking care of the base64 encoding in the tool implementation.
The text was updated successfully, but these errors were encountered: