We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hi
I want to use Roberta Tokenizer. In the following, there is an example that shows how we can do this.
from transformers import AutoTokenizer model_name = "HooshvareLab/roberta-fa-zwnj-base" tokenizer = AutoTokenizer.from_pretrained(model_name) string = "این یک سند است" tokenized_string = tokenizer.tokenize(string) print(tokenized_string)
from transformers import AutoTokenizer
model_name = "HooshvareLab/roberta-fa-zwnj-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
string = "این یک سند است"
tokenized_string = tokenizer.tokenize(string)
print(tokenized_string)
The result of the above code is as follows: ['اÛĮÙĨ', 'ĠÛĮÚ©', 'ĠسÙĨد', 'Ġاست'] However, it should be: ["این", "یک", "سند" , "است"] What is your idea to solve this issue?
['اÛĮÙĨ', 'ĠÛĮÚ©', 'ĠسÙĨد', 'Ġاست']
["این", "یک", "سند" , "است"]
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Hi
I want to use Roberta Tokenizer. In the following, there is an example that shows how we can do this.
from transformers import AutoTokenizermodel_name = "HooshvareLab/roberta-fa-zwnj-base"tokenizer = AutoTokenizer.from_pretrained(model_name)string = "این یک سند است"tokenized_string = tokenizer.tokenize(string)print(tokenized_string)The result of the above code is as follows:
['اÛĮÙĨ', 'ĠÛĮÚ©', 'ĠسÙĨد', 'Ġاست']However, it should be:
["این", "یک", "سند" , "است"]What is your idea to solve this issue?
The text was updated successfully, but these errors were encountered: