Token indices sequence length is longer than the specified maximum sequence length for this model.

[faisal-hearth]

How to reproduce the behaviour

This problem arose when I switched to spacy 3.0 and started using 'en_core_web_trf' instead of 'en_core_web_lg'

# Here text is 1031 tokens long
nlp = spacy.load('en_core_web_trf')
doc = nlp(text)

After executing, it's resulting in the error below:
Token indices sequence length is longer than the specified maximum sequence length for this model (1313 > 512). Running this sequence through the model will result in indexing errors

The problem seems to be in the transformer Pipeline

Your Environment

• Operating System: Window 10
• Python Version Used: 3.9.1
• spaCy Version Used: 3.0.1
• Environment Information: en_core_web_trf 3.0.0

[adrianeboyd]

Hi, your script doesn't stop and you still end up with a processed doc, though, right? I think this is just a warning (as opposed to an error that stops further processing) from the transformers tokenizer. Internally, spacy-transformers runs the tokenizer and then truncates sequences that are too long so that the actual processing with the transformer model proceeds without errors.

Obviously the final annotation can be affected by the fact that some sequences are truncated, but this allows arbitrary texts to be processed without errors and then aligned back with the spacy tokens.

There can also be some issues with some model types that don't provide their model_max_length correctly, but that shouldn't be an issue for roberta-base in en_core_web_trf.

[faisal-hearth]

Yes, the script is still running.
Thanks for clarifying.

[github-actions]

This issue has been automatically closed because it was answered and there was no follow-up discussion.

[baivabdash]

Hi, I have a follow-up question. I am facing this 512 tokens warning while using a distilbert model. But my script runs fairly well and the model is able to reach a decent score. So my question is : Is there a chance that just resolving this warning gives me a boost in score? It would be very much helpful if you could explain in detail..

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.