Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix example int8_inference_huggingface.py #1

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

alexrs
Copy link
Owner

@alexrs alexrs commented May 19, 2023

What

When running the example int8_inference_huggingface.py I was getting the error:

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported

when using the latest transformers=4.29.2. According to this issue:

the tokenizer in the config on the hub points to LLaMATokenizer. However, the tokenizer in the library is LlamaTokenizer.
This is likely due to the configuration files being created before the final PR was merged in.

(See comment)

Fix

Use LlamaForCausalLM and LlamaTokenizer instead of Auto... classes works as expected.

Test plan

  • Run python examples/int8_inference_huggingface.py
  • Obtained the following output:
Hamburg is in which country?
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.

alexrs and others added 2 commits May 19, 2023 15:23
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant