How to prefix the data generation

[lfoppiano]

I've got a question about generation using a prefix, meaning providing the first words to the LLM. Is it possible?

So for example, the prompt in plain text is:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert on .... blablabla.<|eot_id|><|start_header_id|>user<|end_header_id|>
Usser prompt balblabla

Question: {question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>Explanation: 

So we prefix the assistant: "Explanation: " to be the next token to provide a first token

I'm using llama-server via curl / openai python library

[foldl]

Have a check on 'user_suffix'?

[lfoppiano]

Sorry, I could not find it, is it documented somewhere?

[foldl]

Sorry. It is --in-suffix.

I have implemented this feature in chatllm.cpp. I have tried to make a PR to llama.cpp on this, but found --in-suffix which seemed to be something alike. Correct me if I am wrong.

[lfoppiano]

I cannot find the option in llama-server, are you sure is there? Can you please link it in the documentation?

[foldl]

llama.cpp/examples/main/README.md

Line 115 in 4ddd199

The `--in-suffix` flag is used to add a suffix after your input. This is useful for adding an "Assistant:" prompt after the user's input. It's added after the new-line character (`\n`) that's automatically added to the end of the user's input. Here's an example of how to use the `--in-suffix` flag in conjunction with the `--reverse-prompt` flag: