Doc bug: wrong token argument name for Tokenizer.from_pretrained()

[rravenel]

System Info

N/A for doc bug.

Who can help?

@stevhliu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

https://huggingface.co/docs/tokenizers/api/tokenizer#tokenizers.Tokenizer.from_pretrained

from_pretrained
( identifierrevision = 'main'auth_token = None ) → Tokenizer

Parameters

identifier (str) — The identifier of a Model on the Hugging Face Hub, that contains a tokenizer.json file
revision (str, defaults to main) — A branch or commit id
auth_token (str, optional, defaults to None) — An optional auth token used to access private repositories on the Hugging Face Hub

Expected behavior

'auth_token' is incorrect.

'use_auth_token' works, but with the following warning:

/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/modeling_utils.py:3220: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.

'token' also works.

[LysandreJik]

Ah, indeed! The docs should be changd! Would you be down to open a PR to update this?

[rravenel]

I am brand spanking new here and don't know Rust.

Is this the right file: https://github.com/huggingface/tokenizers/blob/main/bindings/python/src/tokenizer.rs?

And I'm just removing 'auth_' between lines 572 and 613? Not sure about the instance of 'token' on line 605 - ignore?

I'm also not setup on your doc build tools. I can take a stab at this next week.

[stevhliu]

Thanks for catching this! For the docs, you can edit the docstring here 🙂

[rravenel]

PR submitted.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[rravenel]

Nudge.

…

On Sat, Sep 28, 2024, 01:03 github-actions[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md> are likely to be ignored. — Reply to this email directly, view it on GitHub <#33183 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATIILVIBU5M2FSEGGPBAYDZYZPGRAVCNFSM6AAAAABNJO2HAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBQGU2TQNZYGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>