Bert model with AlbertTokenizer problem

[ganbaaelmer]

Rasa Open Source version

2.8.16

Rasa SDK version

2.8.3

Rasa X version

0.42.6

Python version

3.8

What operating system are you using?

Linux

What happened?

the error came from a HF model using a different tokenizer (e.g. BertModel with AlbertTokenizer), breaks due to the hard coded mapping in registry.py.

https://forum.rasa.com/t/how-to-import-huggingface-models-to-rasa/50238

Command / Request

No response

Relevant log output

No response

[sara-tagger]

Thanks for raising this issue, @aeshky will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[ganbaaelmer]

@aeshky please help us

[mleimeister]

This is not a bug but rather a known limitation of the current implementation. The documentation has been updated to clarify which models work with the current version of LanguageModelFeaturizer here: #10385
Exploratory work on how it can be improved and allow loading arbitrary models is already happening (#6307, #10624), therefore this ticket could be closed imo.

[aeshky]

Thanks for looking into this @mleimeister. I'm closing this issue based on your comment.

[mleimeister]

@ganbaaelmer Please follow this ticket for further updates on the progress.