Use a chat model instead of LLM for codestral completion

[brichet]

Langchainjs warns against using the text completion model, and advises using the chat model instead, which may be more up-to-date (see caution at https://js.langchain.com/docs/integrations/llms/).

This PR replace the text completion model by a chat model in codestral completion.

[brichet]

The completion works better when using a MistralAI model designed for code completion, like codestral-latest.

[jtpio]

Looks like we can now update this PR to use the default prompts from #28?

[jtpio]

Testing this locally, completions appear to be quite fast, although the response often includes code block delimiters:

Over on #27 there is some logic to not discard responses that may include the code block delimiters, but instead process the string to keep the suggestion without the backticks and language name.

So maybe we could isolate that to some util functions so it can be reused across providers.

[jtpio]

We can also improve the default completion prompt to mention that responses should be formatted without code block delimiters.

[brichet]

We can also improve the default completion prompt to mention that responses should be formatted without code block delimiters.

I did some tests with these kind of prompts and it doesn't work, looks like it doesn't care of these instructions.
But it worked better on my side with the codestral-latest model, is it the one you tested ?

[jtpio]

But it worked better on my side with the codestral-latest model, is it the one you tested ?

Yeah strangely it seems to be returning code blocks quite often?

jupyterlite-ai-codestral-codeblocks.mov

Network response:

{
    "object": "chat.completion",
    "created": 1738938116,
    "model": "codestral-latest",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "tool_calls": null,
                "content": "```python\ndf = pd.DataFrame({\n    'A': [1, 2, 3],\n    'B': [4, 5, 6]\n})\n```"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 88,
        "total_tokens": 125,
        "completion_tokens": 37
    }
}

[jtpio]

Playing around with it a bit more locally, adding more instructions to the prompt like this one seems to be helping a bit:

ai/src/llm-models/chrome-completer.ts

Lines 11 to 20 in 640f525

	/**
	* The initial prompt to use for the completion.
	* Add extra instructions to get better results.
	*/
	const CUSTOM_SYSTEM_PROMPT = `${COMPLETION_SYSTEM_PROMPT}
	Only give raw strings back, do not format the response using backticks!
	The output should be a single string, and should correspond to what a human users
	would write.
	Do not include the prompt in the output, only the string that should be appended to the current input.
	`;

But there still seems to be some issues with completions being a little bit "off" by a few characters?

jupyterlite-ai-codestral-test-2.mov

[brichet]

Playing around with it a bit more locally, adding more instructions to the prompt like this one seems to be helping a bit

Thanks for digging into it. Feel free to push in the branch with your changes. Otherwise I can do it.

[jtpio]

I pushed a change to improve the default prompt used for completions, so it can also be useful for other providers.

[jtpio]

@brichet if this looks good to you, happy to merge as it is right now.

[brichet]

@brichet if this looks good to you, happy to merge as it is right now.

Let's merge it, thank you for the prompt update.

I still have some unexpected suggestions from MistralAI , but it may be more an issue with the model.

Peek.2025-02-10.11-51.webm