[BUG] finetune w/o lora, inference get error: Negative code found: {codes}

[sairin1202]

不使用lora训练的ckpt，load后出现Negative code found: {codes}， 生成codes出现0，但是训练loss正常

如果使用lora训练后merge的模型，一切正常

[yy524]

我也遇到同样问题，请问您找到原因了吗？

[sairin1202]

还没有。。

[Jielin-Qiu]

Same issue here. Pretrain from scratch or finetune (not lora) on new dataset --> Negative code found: {codes}

So far:

• official weight --> ok
• official weight + lora --> ok
• our pretrain weight --> error
• our pretrain weight + lora --error
• official weight + finetune on our data (not lora) --> error

Some examples:

• Pretrain from scratch - Step 50k: Audio
• Finetune (not lora) - Step 10k: Audio

I am able to detect that there are indeed negative values:
Negative values found: tensor([-1, -1, -1, -1, -1, -1, -1, -1], device='cuda:0', dtype=torch.int32)

Not sure if it is an error in the training pipeline, guessing -1 shouldn't be decoded as output?

[liulangdeyeshou]

我前几天遇到了和你一样的问题，后来我改了config.json中的n_local_heads后就好了。我猜测是因为我改了网络结构后，部分参数对不上，导致了推理结果错误。我是在使用lora去重新finetune我训练的pretrain权重的时候发现的这个问题。你可以去尝试check下你的config.json里的参数。

[github-actions]

This issue is stale because it has been open for 30 days with no activity.