Loading XGLM triggers warning "Some weights of .. were not initialized from" for a module with no params

[MattYoon]

System Info

• transformers version: 4.32.0
• Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
• Python version: 3.9.16
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.1
• Accelerate version: 0.20.3
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.0+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@ArthurZucker @younesbelkada
But probably a general issue in the PretrainedModel class.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('facebook/xglm-564M')
# Some weights of XGLMForCausalLM were not initialized from the model checkpoint 
# at facebook/xglm-564M and are newly initialized: ['model.embed_positions.weights']
# You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Expected behavior

The warning should not be triggered.

The positional embedding for XGLM uses XGLMSinusoidalPositionalEmbedding which does not have actual trainable parameters. My guess is that since there is no parameters, the key is not actually stored in the checkpoint and triggers the warning. This issue might exist in other models that similarly have nn.Module with no trainable parameters.

[younesbelkada]

Thanks for raising the issue @MattYoon
@ArthurZucker can confirm, I think the fix would be to make the weight of that module non-persistent (persistent=False) so that they won't get saved in the state_dict: https://github.com/huggingface/transformers/blob/main/src/transformers/models/xglm/modeling_xglm.py#L179
Indeed there is not need to save them in the state dict (or to consider them when loading the weights) as they are created on the fly. If that fixes the issue, would you be happy to open a PR with the fix?