Add an infilling DSL #182
Labels
enhancement
help wanted
text
Linked to text generation
transformers
Linked to the `transformers` integration
Workflow with LLMs often imply recursively calling models where at each step we concatenate the result of the previous call to the prompt. Consider the following example:
The fact that we have to do the concatenation manually can quickly become cumbersome. Worse, we loose the KV cache across generations. I thus suggest to add a thin infilling DSL to outlines with the new API
A few quick points:
stop_at
kwarg from the template;The text was updated successfully, but these errors were encountered: