[LLM Runtime] Extend API for GGUF

[Zhenzhong1]

Type of Change

[LLM Runtime] Extend API for GGUF

Description

Add the model_type arg:
model_type can be set manually or empty.
If the model_type is empty, it will be set according to the HF configuration automatically.

model_name = "TheBloke/Mistral-7B-v0.1-GGUF"
model_file = "mistral-7b-v0.1.Q4_0.gguf" 
tokenizer_name = "mistralai/Mistral-7B-v0.1"

prompt = "Once upon a time"
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
model = AutoModelForCausalLM.from_pretrained(model_name, model_file = model_file, model_type = "llama")
outputs = model.generate(inputs, max_new_tokens=300)

Error tests:

model = AutoModelForCausalLM.from_pretrained(model_name, model_file = model_file, model_type = "PIG")

Correct tests:

model = AutoModelForCausalLM.from_pretrained(model_name, model_file = model_file)

Expected Behavior & Potential Risk

N/A

How has this PR been tested?

Manualluy

Dependency Change?

N/A

[a32543254]

LGTM

[hshen14]

Suggest removing streamer in the sample code to simplify the example further. We've done in the main page.

[Zhenzhong1]

Suggest removing streamer in the sample code to simplify the example further. We've done in the main page.

Fixed.

[Zhenzhong1]

Suggest removing streamer in the sample code to simplify the example further. We've done in the main page.

@hshen14 @a32543254
I have checked it. If we removed streamer, it will be no text output.

Need to print output.

outputs = model.generate(inputs, max_new_tokens=300)
print(outpus)

But as the picutre shown, the output is logit_id. It means we still need to add the deocder if we want the ouput to be text.

outputs = model.generate(inputs,max_new_tokens=300)
print(outputs)
for i in outputs:
    print(tokenizer.decode(i))