Beginnging of response is cut out / missing

[onusai]

Describe the bug

Ive tested several alpaca models and iv'e noticed that start of the response is almost always missing.

Might be similar to this issue: #300

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

Load Alpaca model https://huggingface.co/elinas/alpaca-30b-lora-int4/tree/main and generate using this prompt:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Write a python function that adds two numbers

### Response:

Screenshot

Logs

Output generated in 4.61 seconds (4.12 tokens/s, 19 tokens)

System Info

Windows 11, RTX3090

[onusai]

After further testing, it appears that issue occurs when max_new_tokens is >= 1992. This issue does not occur when max_new_tokens is less than 1992, but it does duplicate the last token of the input.

[oobabooga]

This is being caused by your prompt being truncated. Try reducing max_new_tokens.

[onusai]

Yep that seems to fix it. Is there a reason this isnt automatically calculated? something like max_new_tokens = contex_size - len(prompt)?

[oobabooga]

For generating long text (like a long novel) in several iterations (clicking on Continue multiple times), it is convenient to truncate the beginning of the text to make room for more tokens. This is related to #498 btw