-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added error when sequence length is bigger than max_position_embeddings #32156
Conversation
Hi, It would be nice If you give some time to reviewing this pull request. @ArthurZucker . It relates to this issue. |
max_position_embedding = self.position_embedding.weight.shape[0] | ||
|
||
if seq_length > max_position_embedding: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- No need to create a new variable if it's only used once
- We should instead store a variable from the config to avoid expensive checks on the weight size for every forward pass
max_position_embedding = self.position_embedding.weight.shape[0] | |
if seq_length > max_position_embedding: | |
if seq_length > self.max_position_embeddings: |
And in the init
def __init__(self, config: CLIPTextConfig):
super().__init__()
embed_dim = config.hidden_size
self.max_position_embeddings = config.max_position_embeddings
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realised I asked for this change -- coming back to it, I think that we shouldn't set self.max_position_embeddings
as the user can change the embeddings with set_input_embeddings
--> the value from the config will become out of date. In this case, checking the shapes as before is less brittle, could you change it back to this? Apologies for flip-flopping here.
Hi! Can you take a look at this pull request again please? Thank you. @amyeroberts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I checked that all config have this attribute
This is gonna break remote code, as they won'strore the self.max_position embedding |
Fine for me @amyeroberts feel free to merge |
@amyeroberts |
LGTM and we have reviews already. Thanks for the helpful error message! |
Fixes # 32154