New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

leading space for the first token #282

Closed

monsieurzhang opened this issue Jan 11, 2019 · 1 comment

monsieurzhang commented Jan 11, 2019

I notice there is a leading space for the first token in the output.
e.g. ['▁', 'New', '▁York']

Can I ask what is the purpose of the first spacer?
It seams there is no extra benefit.
Thank you.

Collaborator

taku910 commented Jan 12, 2019

See https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto#L201

You can disable the dummy prefix with --add_dummy_prefix=false of spm_train.

taku910 closed this as completed

taku910 mentioned this issue

better candidate for unk_surface #280

Closed

ivanstepanovftw mentioned this issue

Avoid hardcoding a space at the beginning of the prompt. ggerganov/llama.cpp#1315

Closed

crasm mentioned this issue

[feature] Make space insertion optional in SentencePiece tokenizer ggerganov/llama.cpp#3664

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment