Skip to content

Commit

Permalink
feat: add imageData option to KurtMessage
Browse files Browse the repository at this point in the history
Now it is possible to send inline image data to multi-modal LLMs.
  • Loading branch information
jemc committed Sep 27, 2024
1 parent 6fb7df8 commit a8a2222
Showing 1 changed file with 21 additions and 0 deletions.
21 changes: 21 additions & 0 deletions packages/kurt/src/Kurt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,27 @@ export type KurtMessage = {
*/
text: string

/**
* When present, this is an image data message, with a base64-encoded image.
* This is often used with "multi-modal" LLMs that support image mode input.
*
* Not all LLM providers or underlying models support this kind of message.
* Check your LLM provider's documentation for confirmaton.
*/
imageData: {
/**
* The IANA standard MIME type of the inline image data.
*
* Not all MIME types are supported by all LLM providers.
* "image/png" and "image/jpeg" are the most commonly supported.
* Check your LLM provider's documentation for the right list.
*/
mimeType: string

/** Base64-encoded image data, as a string. */
base64Data: string
}

/**
* When present, this is a tool call message, with structured data input
* in the `args` object, and structured data output in the `result` object.
Expand Down

0 comments on commit a8a2222

Please sign in to comment.