Language model access API

[jrieken]

This issue is about allowing extensions to use language model that are contributed by other extensions. The API proposal is https://github.com/microsoft/vscode/blob/main/src/vscode-dts/vscode.proposed.languageModels.d.ts and its scope is to allow to use the chat feature of language models.

Remaining finalisation todos

• consolidate different LanguageModelChatXYZMessage-type into one type and a role-enum
• move System message/role into its own proposal
• have a select-function that's the only way of getting to a chat model
• consider moving token counting into its own proposal (some model don't do token counting e.g anthropic or ollama)
• consider to have more sturcture on LanguageModelChatResponse, e.g have a fake reply message with role like others (ollama, open ai, anthropic)
  • I renamed LanguageModelChatResponse#stream to text and I will continue to explore a stream of structured objects. That would be additive to text and plugged in as its foundation.
  • add something like modelResult: Thenable<{ [name: string]: any }>
• add a StopReason-enum that's also used in LanguageModelChat

[isidorn]

Docs: https://code.visualstudio.com/api/extension-guides/language-model

[jrieken]

With #206358 there will be no more language model access object. Instead all requests are made directly via vscode.lm.sendChatRequest(model, messages, options, token)

[worksofliam]

Hi @jrieken

Is there any plans to add a registerLanguageModel, so I can implement my own calls to other models other than the Copilot provided models?

Thanks!

[jrieken]

Yeah, that exists. It is the src/vscode-dts/vscode.proposed.chatProvider.d.ts which is still very rough and currently not yet scheduled for finalisation. Tho, feel free to give it a spin and report what's missing/wrong. For instance, locally I have a dummy extension that uses ollama to drive this API

[worksofliam]

@jrieken I assume you are referring to the createChatParticipant API?

So if I use createChatParticipant, I can then use the id I give it with sendChatMessage?

[jrieken]

@jrieken I assume you are referring to the createChatParticipant API?

No, I mean the proposal that I have mentioned

[jrieken]

FYI that I made the following changes to the languageModels proposal

• there is no more vscode.lm.languageModels and generally no more referring of language model by string-identifier.
• there is vscode.lm.selectChatModels(selector) that takes a selector and returns matching chat models or nothing (undefined)

These changes allow for a better implementation of the providing-end (chatProvider-proposal) and most importantly make it more obvious that models can (and will) change. The following pattern had become popular'ish vscode.lm.sendChatRequest('hardcoded.model.id, ....)`. This is brittle because extensions break as soon as the model id isn't known anymore. Instead models need to be selected, making it hopefully more clear that a desired model doesn't exist. The new selector-mechanism allows for new flexibility as well. A couple of samples

// select a model by vendor and family
await vscode.lm.selectChatModels({ vendor: 'copilot', family: 'gpt-3.5-turbo' })

// select all models
await vscode.lm.selectChatModels()

// hardcode an identifier (previous usage model, NOT recommended because brittle)
await vscode.lm.selectChatModels({ id: 'someHardcodedModelId' })

To restate, the selectChatModels-function can return undefined which means no model was found. That's because the selector is to strict (using id) or because no models are known (missing extensions). The shape of language model selectors is this

export interface LanguageModelChatSelector {
  vendor?: string;
  family?: string;
  version?: string;
  id?: string;
}

Note that

• property order represents how specific it is
• properties are matched based on equality, when present it must match
• we will likely add some comparison selector for input and/or output tokens

[InTheCloudDan]

I see mention of moving the System prompt to a separate Proposal, can that be linked to? I am not finding it and we use it in our extension.

Also is this section of the guide still accurate? https://code.visualstudio.com/api/extension-guides/language-model#prompt-crafting It's not showing those commands as imports anymore and from my reading of the types it looks like it should be closer to:

vscode.LanguageModelChatMessage(vscode.LanguageModelChatMessageRole.User, message)

[jrieken]

This is place of all API proposals: https://github.com/microsoft/vscode/tree/main/src/vscode-dts

Also is this section of the guide still accurate

Likely outdated

[bwateratmsft]

Really confusing... In my sample remote-setup (which isn't codespaces) this is working just fine and I see so smoking guns for why this wouldn't work in codespaces. For now, I have added some logging that should help us get to the bottom of this

Where should I look for the additional logs?

[jrieken]

Enable trace logging, show the window-log and check for messages starting with [LM]. There should be some for registering and some for selection

[jrieken]

@bwateratmsft I did find something after @roblourens running into a similar issues. Can you confirm that a later/second invocation yields models and/or that the vscode.lm.onDidChangeChatModels-event fires and that after that chat models are selected?

[bwateratmsft]

I found some log lines for selection but nothing for registration:

[LM] selected language models {"vendor":"copilot","extension":{"value":"ms-azuretools.vscode-azure-github-copilot","_lower":"ms-azuretools.vscode-azure-github-copilot"}} []

We end up doing language model selection several times and each time it returns undefined.

I even tried doing the following and still there were no log lines about registration:

1. Disable all 3 extensions (Copilot Chat, Copilot, and our extension)
2. Reload browser window
3. Enable trace window logging
4. Enable extensions
5. Execute a query with regular Copilot (success)
6. Execute a query with our extension (failure)

However, when using VSCode Desktop as the frontend, I do see log lines about language model registration:

[LM] registering language model chat github.copilot-chat/1/gpt-3.5-turbo {"extension":{"value":"GitHub.copilot-chat","_lower":"github.copilot-chat"},"id":"gpt-3.5-turbo","vendor":"copilot","name":"GPT 3.5 Turbo","family":"gpt-3.5-turbo","version":"gpt-3.5-turbo-0613","maxInputTokens":4000,"maxOutputTokens":2000,"auth":{"providerLabel":"GitHub Copilot Chat","accountLabel":"bwateratmsft"}}
...
[LM] registering language model chat github.copilot-chat/2/gpt-4 {"extension":{"value":"GitHub.copilot-chat","_lower":"github.copilot-chat"},"id":"gpt-4","vendor":"copilot","name":"GPT 4","family":"gpt-4","version":"gpt-4-0613","maxInputTokens":4000,"maxOutputTokens":2000,"auth":{"providerLabel":"GitHub Copilot Chat","accountLabel":"bwateratmsft"}}

[jrieken]

I even tried doing the following and still there were no log lines about registration

Thanks for the update. That would really mean the language model API doesn't register. Can you check the chat log (Output > GH Copilot Chat) for messages starting with registerChatResponseProvider

[bwateratmsft]

I don't see any messages like that from either browser or desktop frontend on the Codespace. However, on the browser session there was this message which did not appear on the desktop session--might be spurious but I'm not sure:

2024-05-15 17:47:09.424 [info] [RemoteAgents] Invalid remote agent response: bad request: unknown integration (SyntaxError: Unexpected token 'b', "bad reques"... is not valid JSON)

[isidorn]

We have finalized the Language Model and Chat Participant API 🚀

Thus, you can now publish your extensions to the VS Marketplace. This API is finalized in VS Code Insiders, and will be finalized in VS Code Stable in July. Thus we suggest that you use an engines: ^1.90.0 property in your package.json, and VS Code Stable will gracefully handle your extensions and ignore your usage of Language Model and Chat API until the API gets finalized in Stable in July.

Full docs https://code.visualstudio.com/api/extension-guides/language-model

We have decided to only finalise the User message. And System message might be finalised in one of the future iterations. Main reason for this is general API consideration - as we are still learning how the System prompt API would fit with some of the non-GPT models. Also we believe that the System prompt API requires more responsible AI testing from our team.
If your extension would like to use the System prompt API - please let us know if you hit issues with using the User prompt as a replacement.

We are excited to see what you build ✨

[worksofliam]

@isidorn I am still wondering if there will be a lm.registerModel() API so I can use other models inside of the chat API. Is this something coming in the future?

For now, it can be shimmed by wrapping lm.sendChatRequest and handling other entries (perhaps I want to point to my local Ollama instance).

[jrieken]

@worksofliam https://github.com/microsoft/vscode/blob/main/src/vscode-dts/vscode.proposed.chatProvider.d.ts

[jrieken]

Still a proposal and not yet planned for finalisation