Add some info on dtypes

[osanseviero]

No description provided.

[pcuenca]

Thanks a lot, I like the idea to introduce this section! My suggestions are totally optional, but I wouldn't state that float16 is the expected way to run the model.

[pcuenca]

Suggested change

As mentioned above, `transformers` loads using `float32` (no matter with which precision the models are stored), so it's important to specify the dtype when loading the models. If you want to fine-tune Code Llama, it's recommended to use `bfloat16`. Using `float16` can lead to overflows and NaNs. Similarly, if you run inference, it's recommended to use `float16`. Using `bfloat16` can lead to a slower inference.

As mentioned above, `transformers` loads weights using `float32` (no matter with which precision the models are stored), so it's important to specify the desired `dtype` when loading the models. If you want to fine-tune Code Llama, it's recommended to use `bfloat16`, as using `float16` can lead to overflows and NaNs. If you run inference, we recommended to use `float16` because `bfloat16` can be slower.

[pcuenca]

One additional question I have is: why do we convert the original weights to float16? If they were produced in bfloat16, shouldn't we keep them like that and still recommend to run inference in float16? I understand the difference is small in this case, but what's the drawback in keeping the original precision?

[osanseviero]

Maybe @ArthurZucker can shed some light on conventions for this

[pcuenca]

Yes, I'd love to hear his thoughts!

[ArthurZucker]

Actually the convention is more historic (we kind of always uploaded the checkpoints of Llama1 to fp16, same with other models) and there are no real degradation if you are training again the model. Though uploading to original dtype is better I guess, we did not really care since what is important is to respect the dtype of training. And given how people were saying "Inference is slow" I'd rather have fast inference for people not used to code, and training issues for people who are able to train and just need to cast the model to the correct dtype 😉

Would be great to be able to store both versions on the hub @osanseviero, with a revision or tag this way people could choose from either 👍🏻

[pcuenca]

@ArthurZucker that makes a lot of sense. We use variants in diffusers, i.e.: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main/unet (see the fp16 vs normal safetensors), and the same mechanism exists in transformers: huggingface/transformers#21332

[pcuenca]

Thank you!