This repo is meant to help developers build integrations between tldraw's canvas and AI tools. It contains several resources that can be used to get information out of tldraw (in order to prompt a model) and to create changes in tldraw based on some generated instructions.
- Join the Discord channel
- Learn more about the tldraw SDK
The best way to get started is to clone this repository and experiment with its example project.
The module is also distributed as an NPM package, @tldraw/ai. It is meant to be used together with the tldraw SDK.
This repository is a pnpm monorepo. It has three parts:
/packagecontains the ai module package itself/example/clientcontains the example's frontend (a Vite app)/example/workercontains the example's backend (a Cloudflare Worker)
-
Clone this repository.
-
Enable
corepack
corepack enable- Install dependencies using pnpm.
pnpm i- Create a
/example/.dev.varsfile. Add any environment variables required by the server to the/example/.dev.varsfile. By default, our example project requires an OpenAI API Key so your/example/.dev.varsfile should look something like this:
OPENAI_API_KEY=sk-proj-rest-of-your-key
ANY_OTHER_KEY_YOU_ARE_USING=here
If you need to use public-friendly API keys on the frontend, create a /example/.env file and put them there. See this guide for more information about environment variables in Vite.
VITE_LEAKABLE_OPENAI_API_KEY=sk-proj-rest-of-your-key
VITE_SOME_PUBLIC_KEY=sk-proj-rest-of-your-key
- Start the development server.
pnpm run dev- Open localhost:5173 in your browser.
You can now make any changes you wish to the example project.
You can now make any changes you wish to the example project.
7. Start hacking
There are a few things you can do right away:
- Tweak the example's system prompt at
./worker/do/openai/system-prompt.ts. - Make the example's system capable of creating new shapes.
- Make the example's system capable of creating new events.
See the README in example/worker/do/openai/README.md for more information on the example's backend.
Note: If you find yourself needing to make changes to the package code, please let us know on the tldraw discord channel. Your changes would be very useful information as we continue to develop the module!
For production, install the @tldraw/ai package in a new repository, such as a fork of tldraw's Vite template. See the tldraw repository for more resources.
Install the @tldraw/ai package from NPM or your package manager of choice.
npm i @tldraw/aiUse the useTldrawAi hook in your code.
function App() {
return (
<div style={{ position: 'fixed', inset: 0 }}>
<Tldraw persistenceKey="example">
<MyPromptUi />
</Tldraw>
</div>
)
}
const TLDRAW_AI_OPTIONS = {
transforms: [],
generate: async ({ editor, prompt, signal }) => {
// todo, return changes
return []
},
stream: async function* ({ editor, prompt, signal }) {
// todo, yield each change as it is ready
},
}
function MyPromptUi() {
const ai = useTldrawAi(TLDRAW_AI_OPTIONS)
return (
<div style={{ position: 'fixed', bottom: 100, left: 16 }}>
<button onClick={() => ai.prompt('draw a unicorn')}>Unicorn</button>
</div>
)
}Read on for more tips about using and configuring the hook.
The fundamental unit in tldraw's ai module is the "change". A change is an instruction to the tldraw editor to do one of the following:
createShapecreates a shapeupdateShapeupdates a shapedeleteShapedeletes a shapecreateBindingcreates a bindingupdateBindingupdates a bindingdeleteBindingdeletes a binding
See package/src/types.ts for more about each change.
Changes should be generated by the generate or stream methods of the useTldrawAi configuration. You can do generate the changes using whichever method you wish, however the expectation is that you will send information to an LLM or other model to generate changes for you.
You may find that models are bad at generating changes directly. In our example project, we communicate with the LLM using a "simplified" format, parsing the response before sending back the actual changes expected by the ai module.
The ai module has support for "transforms" (TldrawAiTransform). This feature allows you to modify the user's prompt (the data extracted from the canvas) and then use those modifications again later when handling changes. These are useful when preparing the data into a format that is easier for an LLM to work with.
Our example project includes several of these. When extracting data from the canvas, SimpleIds transform replaces tldraw's regular ids (which look like shape:some_long_uuid) with simplified ids (like 1 or 2). Later, when handling changes, the transform replaces the simplified ids with their original ids (or creates new ones).
To create a transform, extend the TldrawAiTransform class:
export class MyCustomTransform extends TldrawAiTransform {
override transformPrompt = (prompt: TLAiPrompt) => {
// modify the prompt when it's first created
}
override transformChange = (change: TLAiChange) => {
// modify each change when it's being handled
}
}Pass your new class to the useTldrawAi hook's configuration.
const MY_STATIC_CONFIG: TldrawAiOptions = {
transforms: [MyCustomTransform],
...etc,
}When the user creates a new prompt, the ai module will create a new instance of each transform to be used for that prompt only. This means that you can stash whatever data you wish on the instance. See the examples in example/client/transforms as a reference.
The TldrawAiModule class is responsible for
- Getting information about the current tldraw editor's canvas
- Incorporating transforms before and after changes are generated
- Applying "changes" to the tldraw editor's canvas
The package exports a hook, useTldrawAiModule, that creates an instance of the TldrawAiModule class for you to use in React. This class handles tasks such as getting information out of the tldraw canvas and applying changes to the tldraw canvas.
The useTldrawAi hook adds an extra layer of convenience around the ai module. This hook handles many of the standard behaviors for you. While we expect to expand this hook to support more configration, you may find it necessary to create your own version of this hook (based on its source code) in order to customize it further.
The hook responds with three methods: prompt, repeat, and cancel.
promptaccepts either a string or a configuration object withmessagesandstream. By default, thepromptmethod will call your configuration'sgeneratemethod. Ifstreamis true, then it will call your configuration'sstreammethod.cancelwill cancel any currently running generation.repeatwill apply the same changes that were generated last time. This is useful for debugging.
Generate vs. Stream
You don't need to define both generate and stream, though you should define one of them. If you call ai.prompt with the stream flag set to true, but don't have stream implemented, then you'll get an error; likewise, if you call ai.prompt without the stream flag and without generate, then you'll get an error. Just be sure to implement one or both.
Static configuration
If you're using the useTldrawAi hook, we recommend creating a custom hook that passes static options to the useTldrawAi hook. See the useTldrawAiExample hook in our example project as a reference.
export function useMyCustomAiHook() {
const ai = useTldrawAi(MY_STATIC_OPTIONS)
}
const MY_STATIC_OPTIONS: TldrawAiOptions = {
transforms: [],
generate: async ({ editor, prompt, signal }) => {
// todo, return changes
return []
},
stream: async function* ({ editor, prompt, signal }) {
// todo, yield each change as it is ready
},
}If you must define the options inside of a React component, it's important that you memoize the options correctly.
export function MyPromptUi() {
const myMemoizedOptions = useMemo<TldrawAiOptions>(() => {
return {
transforms: [],
generate: async ({ editor, prompt, signal }) => {
return []
},
stream: async function* ({ editor, prompt, signal }) {},
}
}, [])
const ai = useTldrawAi(myMemoizedOptions)
return <etc />
}Calling the hook
The ai module relies on the tldraw editor at runtime. You should use the useTldrawAi hook inside of the tldraw editor's React context, or else provide it with the current editor instance as a prop.
You can do that via a child component:
function App() {
return (
<div style={{ position: 'fixed', inset: 0 }}>
<Tldraw persistenceKey="example">
<MyPromptUi />
</Tldraw>
</div>
)
}
function MyPromptUi() {
const ai = useMyCustomAiHook()
return (
<div style={{ position: 'fixed', bottom: 100, left: 16 }}>
<button onClick={() => ai.prompt('draw a unicorn')}>Unicorn</button>
</div>
)
}Or via the Tldraw component's components prop:
const components: TLComponents = {
InFrontOfTheCanvas: () => {
const ai = useMyCustomAiHook()
return (
<div>
<button onClick={() => ai.prompt('draw a unicorn')}>Unicorn</button>
</div>
)
},
}
function App() {
return (
<div style={{ position: 'fixed', inset: 0 }}>
<Tldraw persistenceKey="example" components={components} />
</div>
)
}If this is inconvenient—or if you like a challenge—you can also pass the editor as an argument to useTldrawAi. While this involves some "juggling", it may be useful when you wish to place the ai module into a global context or necessary if you need to use it in different parts of your document tree.
function App() {
const [editor, setEditor] = useState<Editor | null>(null)
return (
<div style={{ position: 'fixed', inset: 0 }}>
<TldrawBranch onEditorMount={setEditor} />
{editor && <MyPromptUi editor={editor} />}
</div>
)
}
function TldrawBranch({ onMount }: { onMount: (editor: Editor) => void }) {
return (
<div style={{ position: 'fixed', inset: 0 }}>
<Tldraw persistenceKey="example" onMount={onEditorMount} />
</div>
)
}
function MyPromptUi({ editor }: Editor) {
const ai = useTldrawAi({ editor, ...MY_STATIC_OPTIONS })
return (
<div style={{ position: 'fixed', bottom: 100, left: 16 }}>
<button onClick={() => ai.prompt('draw a unicorn')}>Unicorn</button>
</div>
)
}This project is provided under the MIT license found here. The tldraw SDK is provided under the tldraw license.
Copyright (c) 2024-present tldraw Inc. The tldraw name and logo are trademarks of tldraw. Please see our trademark guidelines for info on acceptable usage.
You can find the @tldraw/ai package on npm here. You can find tldraw on npm here.
Found a bug? Please submit an issue.
Have questions, comments or feedback? Join our discord. For the latest news and release notes, visit tldraw.dev.
Find us on Twitter/X at @tldraw or email us at mailto:hello@tldraw.com.