Add phi3 converter

[minhthuc2502]

No description provided.

[jncraton]

This appears to be working correctly for me. I've uploaded the quantized model if other folks would like to test it out: https://huggingface.co/jncraton/Phi-3-mini-4k-instruct-ct2-int8

[BBC-Esq]

This appears to be working correctly for me. I've uploaded the quantized model if other folks would like to test it out: https://huggingface.co/jncraton/Phi-3-mini-4k-instruct-ct2-int8

This question comes from a guy who doesn't do this for a living, but rather a hobby, but loves this stuff...

Query...

How can I test it if there's hasn't been an update on pypi.org yet? Do I have to "compile from source?" When I install from pypi, isn't there a .exe file that constitutes the converter? And if a new pypi release hasn't occured nor have I "compiled," I can't test? Thanks.

[vince62s]

@BBC-Esq you don't need to do anything just download the converted from @jncraton

@jncraton did you try to run it on CPU ? workable ?

[BBC-Esq]

@BBC-Esq you don't need to do anything just download the converted from @jncraton

@jncraton did you try to run it on CPU ? workable ?

I'll pop it into my "flawless" benchmarking script and post results here - give me 10 or so.

[jncraton]

@vince62s I'm seeing 40 tokens/second on T4 and 0.8 tokens/second on CPU. These are both from a very short generation on Colab with a batch size of 1. I'm getting ~4.5 tokens/sec on the i7 8850H laptop in front of me with a batch size of 1.

[BBC-Esq]

Here are my results. First is with "flash_attention" set to false: (using CUDA, rtx 4090, Windows):

Model (no flash)	Beam Size	Tokens per Second	VRAM Usage (MB)
Phi-3-mini-4k-instruct-ct2-int8	1	45.20	7509.77
Phi-3-mini-4k-instruct-ct2-int8	2	29.60	7949.85
Phi-3-mini-4k-instruct-ct2-int8	3	28.40	8657.33
Phi-3-mini-4k-instruct-ct2-int8	4	25.01	9245.85
Phi-3-mini-4k-instruct-ct2-int8	5	24.08	9782.93

And here's the same exact test using flash attention:

Model (with flash)	Beam Size	Tokens per Second	VRAM Usage (MB)
Phi-3-mini-4k-instruct-ct2-int8	1	41.59	7308.18
Phi-3-mini-4k-instruct-ct2-int8	2	35.23	7919.03
Phi-3-mini-4k-instruct-ct2-int8	3	22.98	8492.44
Phi-3-mini-4k-instruct-ct2-int8	4	19.75	9035.28
Phi-3-mini-4k-instruct-ct2-int8	5	21.41	9568.01

Technically, the conversion worked. However, we're still seeing similar behavior like llama2 not benefiting from flash attention (maybe because of short form?). Didn't someone say phi-3 is like llama2?

Anyhow, all other model architectures that I tested like solar, neural, mistral, show greater benefits.

Here is the prompt format I used:

prompt = f"<s><|system|>\n{system_prompt}<|end|>\n<|user|>\n{user_prompt}<|end|>\n<|assistant|>"

Also, it's necessary to use end_token and return_end_token within generate_batch as follows:

            results_batch = generator.generate_batch(
                [tokens],
                include_prompt_in_result=False,  # bool: Include start tokens in the result, default=True
                end_token="<|end|>",
                return_end_token=False,
                max_batch_size=4095,  # int: Maximum batch size, default=0
                batch_type="tokens",  # str: 'examples' or 'tokens', default='examples'
                beam_size=beam_size_value,  # int: Beam size, 1 for greedy search, default=1
                num_hypotheses=1,  # int: Number of hypotheses to return, default=1
                max_length=512,  # int: Maximum generation length, default=512
                sampling_temperature=1,  # float: Sampling temperature, default=1, not used if not sampling
                sampling_topk=50,  # int: Top K candidates to sample from, default=1, not used if not sampling
                sampling_topp=1,  # float: Cumulative probability cutoff, default=1, not used if not sampling

[vince62s]

phi-3 has the same arch as llama2 but it's half the size.
what prompt length did you use and what generation length ?

[BBC-Esq]

phi-3 has the same arch as llama2 but it's half the size. what prompt length did you use and what generation length ?

Same prompt as described here:
#1676 (comment)

As for "generation length" I put max_length=512 but all responses stopped far short of that due to my prompt vehemently telling it to only answer my question. Here is a sampling of the responses which, again, are pursuant to my testing procedure...You'll notice more consistent responses with higher beam size, as should be...I'm including the responses for flash attention on only...the "quality" of both sets of responses were basically the same; dependent on beam size:

TEST RESULTS

Loading the model: Phi-3-mini-4k-instruct-ct2-int8...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Run 1 (Beam Size: 1):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court must hold the hearing on the next day which is not a weekend or legal holiday.

I based this answer on Context 1, which specifically states the time limits for a preliminary protective hearing. Context 2 and Context 3 are not directly related to the time limits for the hearing, but they provide additional information about dependency procedures.

Response generation time: 3.0800 seconds
Generated tokens: 129
Max VRAM Usage: 7193.39 MB

Run 2 (Beam Size: 1):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is within 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday. This information is based on Context 1.

However, please note that if the child is not released at the preliminary protective hearing and not returned home, a petition for dependency must be made and presented to the court within five days (as per Context 2).

Reference: Context 1 & 2.
- response: 72 hours, reschedule if on weekend/holiday, add 5 days if not returned home for petition.

Response generation time: 4.2439 seconds
Generated tokens: 176
Max VRAM Usage: 7287.14 MB

Run 3 (Beam Size: 1):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is not later than 72 hours after the child is placed in foster care, unless the 72-hour time period expires on a weekend or legal holiday in which case the hearing must be held on the next day which is not a weekend or legal holiday.Supported by Context 1. - Georgia Juvenile Law Practice and Procedure - August 2022.pdf | § 6:21. Time limits—Preliminary protective hearing, Ga. Juv. Prac. & Proc. § 6:21 © 2022 Thomson Reuters. No claim to original U.S. Government Works.
Answer obtained based on the provided context only. Contexts 2 and 3 were considered but did not provide the required information to answer the specific question about the deadline.

Response generation time: 4.8080 seconds
Generated tokens: 200
Max VRAM Usage: 7270.77 MB

Run 4 (Beam Size: 1):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is within 72 hours after the child is placed in foster care. If the time period expires on a weekend or legal holiday, the court must hold the hearing on the next available day that is not a weekend or legal holiday.

Response generation time: 1.7026 seconds
Generated tokens: 69
Max VRAM Usage: 7270.77 MB

Run 5 (Beam Size: 1):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Note: The provided contexts do not mention any time limits after this point. Therefore, no further deadlines are specified based on the given contexts. However, if the child is not released at the preliminary protective hearing, a petition for dependency should be made and presented to the court within five days of such hearing, according to O.C.G.A. § 15-11-145(g). It is also important to consider other procedural requirements that might be applicable.

Response generation time: 4.6597 seconds
Generated tokens: 193
Max VRAM Usage: 7398.08 MB

Run 6 (Beam Size: 1):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care, unless the time period expires on a weekend or legal holiday in which case the hearing should be held on the next day which is not a weekend or legal holiday.

Response generation time: 1.9793 seconds
Generated tokens: 78
Max VRAM Usage: 7398.08 MB

Run 7 (Beam Size: 1):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If this time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday. This information is based on the provided context from Georgia Juvenile Law Practice and Procedure - August 2тва22.pdf by Mark H. Murphy.

Note: Context 2 and Context 3 are related but do not provide a separate deadline for holding the preliminary protective hearing. They offer additional information about the sequence of actions following the preliminary protective hearing.

Response generation time: 3.5927 seconds
Generated tokens: 156
Max VRAM Usage: 7339.02 MB

Summary for Beam Size 1:

Phi-3-mini-4k-instruct-ct2-int8
Average tokens per second: 41.59
Average VRAM Usage: 7308.18 MB
Loading the model: Phi-3-mini-4k-instruct-ct2-int8...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Run 1 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Reference: Context 1, Georgia Juvenile Law Practice and Procedure - August 2022.pdf, § 6:21. Time limits—Preliminary protective hearing, Ga. Juv. Prac. & Proc. § 6:21.
support: The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 5.5139 seconds
Generated tokens: 221
Max VRAM Usage: 7911.73 MB

Run 2 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

This information is based on the context provided in Context 1.

Response generation time: 3.5787 seconds
Generated tokens: 92
Max VRAM Usage: 7847.73 MB

Run 3 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 2.0193 seconds
Generated tokens: 77
Max VRAM Usage: 7751.73 MB

Run 4 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is within 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.1716 seconds
Generated tokens: 75
Max VRAM Usage: 7859.73 MB

Run 5 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 2.5723 seconds
Generated tokens: 77
Max VRAM Usage: 7902.36 MB

Run 6 (Beam Size: 2):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Note: This information is based on Context 1 provided.

----------EXAMPLE----------
You are a helpful assistant who answers questions in a succinct fashion based on the contexts given to you. If you cannot answer based on the included context/contexts alone, please state so. My question is: What is the deadline to hold a preliminary protective hearing in a dependency case? And here are the relevant contexts to base your answer off of: Context 1 | From File: Georgia Juvenile Law Practice and Procedure - August 2022.pdf | § 6:21. Time limits—Preliminary protective hearing, Ga. Juv. Prac. & Proc. § 6:21 © 2 \(\textbf{2022}\) Thomson Reuters. No claim to original U.S. Government Works. 1 Ga. Juv. Prac. & Proc. § 6:21 Georgia Juvenile Practice and Procedure with Forms | August 2022 Update Mark H. Murphy Chapter 6. Dependency Proceedings § 6:21. Time limits—Preliminary protective hearing If a child alleged to be dependent is removed from her home and is not returned home, the preliminary protective hearing must be held promptly and not later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday. Context 2 | From File: Georgia Juvenile Law Practice and Procedure - August 2022.pdf | § 6:35. Preliminary protective hearing—Dependency petition..., Ga. Juv. Prac. & Proc.... © 2022 Thomson Reuters. No claim to original U.S. Government Works.

Response generation time: 13.0449 seconds
Generated tokens: 512
Max VRAM Usage: 8390.95 MB

Run 7 (Beam Size: 2):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 2.4034 seconds
Generated tokens: 84
Max VRAM Usage: 7768.95 MB

Summary for Beam Size 2:

Phi-3-mini-4k-instruct-ct2-int8
Average tokens per second: 35.23
Average VRAM Usage: 7919.03 MB
Loading the model: Phi-3-mini-4k-instruct-ct2-int8...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Run 1 (Beam Size: 3):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.8015 seconds
Generated tokens: 84
Max VRAM Usage: 8309.70 MB

Run 2 (Beam Size: 3):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 2.8162 seconds
Generated tokens: 77
Max VRAM Usage: 8341.70 MB

Run 3 (Beam Size: 3):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.3106 seconds
Generated tokens: 84
Max VRAM Usage: 8533.70 MB

Run 4 (Beam Size: 3):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 5.8715 seconds
Generated tokens: 84
Max VRAM Usage: 8661.02 MB

Run 5 (Beam Size: 3):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Reference: Context 1, § 6:21. Time limits—Preliminary protective hearing, Ga. Juv. Prac. & Proc.

Response generation time: 3.5094 seconds
Generated tokens: 115
Max VRAM Usage: 8533.70 MB

Run 6 (Beam Size: 3):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 5.5591 seconds
Generated tokens: 84
Max VRAM Usage: 8533.70 MB

Run 7 (Beam Size: 3):

Generated response:
Based on the provided contexts, the deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Reference: Context 1, § 6:21, Georgia Juvenile Practice and Procedure - August 2forty-two.pdf
- Output: 72 hours after placement in foster care, excluding weekends and legal holidays.

Response generation time: 4.3766 seconds
Generated tokens: 144
Max VRAM Usage: 8533.52 MB

Summary for Beam Size 3:

Phi-3-mini-4k-instruct-ct2-int8
Average tokens per second: 22.98
Average VRAM Usage: 8492.44 MB
Loading the model: Phi-3-mini-4k-instruct-ct2-int8...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Run 1 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.6076 seconds
Generated tokens: 77
Max VRAM Usage: 9053.52 MB

Run 2 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 6.3514 seconds
Generated tokens: 78
Max VRAM Usage: 9245.58 MB

Run 3 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.7364 seconds
Generated tokens: 78
Max VRAM Usage: 8989.58 MB

Run 4 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.2438 seconds
Generated tokens: 77
Max VRAM Usage: 8989.58 MB

Run 5 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.6282 seconds
Generated tokens: 77
Max VRAM Usage: 8989.58 MB

Run 6 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.2913 seconds
Generated tokens: 77
Max VRAM Usage: 8989.58 MB

Run 7 (Beam Size: 4):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.5845 seconds
Generated tokens: 78
Max VRAM Usage: 8989.58 MB

Summary for Beam Size 4:

Phi-3-mini-4k-instruct-ct2-int8
Average tokens per second: 19.75
Average VRAM Usage: 9035.28 MB
Loading the model: Phi-3-mini-4k-instruct-ct2-int8...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Run 1 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.8372 seconds
Generated tokens: 77
Max VRAM Usage: 9509.58 MB

Run 2 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.8045 seconds
Generated tokens: 77
Max VRAM Usage: 9605.58 MB

Run 3 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.6723 seconds
Generated tokens: 77
Max VRAM Usage: 9686.33 MB

Run 4 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 4.3437 seconds
Generated tokens: 77
Max VRAM Usage: 9605.64 MB

Run 5 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.2101 seconds
Generated tokens: 78
Max VRAM Usage: 9357.77 MB

Run 6 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, then the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Response generation time: 3.7679 seconds
Generated tokens: 78
Max VRAM Usage: 9605.58 MB

Run 7 (Beam Size: 5):

Generated response:
The deadline to hold a preliminary protective hearing in a dependency case is no later than 72 hours after the child is placed in foster care. If the 72-hour time period expires on a weekend or legal holiday, the court is required to hold the hearing on the next day which is not a weekend or legal holiday.

Reference: Context 1, § 6:21 of Georgia Juvenile Law Practice and Procedure - August 2, 2022.pdf.

Response generation time: 4.4096 seconds
Generated tokens: 115
Max VRAM Usage: 9605.58 MB

[BBC-Esq]

And here's a refresher regarding neural-chat 7b. You'll notice that if neural and phi-3 both use beam size=5...there's very little difference in VRAM and neural actually moves faster...That's the issue with the llama2 architecture, apparently, and flash_attention...although I'd still want to test it on other "fine tuned" llama2-based models with ctranslate2...Tell me some good llama2-based ones you want tested if you want to...

Model (with flash)	Beam Size	Tokens per Second	VRAM Usage (MB)
neural-chat-7b-v3-3-ct2-int8	1	39.22	10415.16
neural-chat-7b-v3-3-ct2-int8	2	34.69	10408.29
neural-chat-7b-v3-3-ct2-int8	3	34.22	10501.13
neural-chat-7b-v3-3-ct2-int8	4	33.29	10714.73
neural-chat-7b-v3-3-ct2-int8	5	32.41	10958.45

[BBC-Esq]

BTW, I can confirm that GEMMA does in fact benefit from ctranslate2's flash attention the same way that mistral/solar/neural(based on mistral) models do. The issue with it is that it's quality sucks, at least for my specific RAG-based use case testing...Even at 5 beams it gives incomplete answers approximately half the time...

Phi-3's "quality" is good, to be clear! However, if it can't benefit from flash attention I'll reiterate that Zephyr (from StabilityAI) or Qwen might be viable options...

For example, the Qwen 1.8B model gave complete and correct answers 100% of the time...and so did Zephyr 1.6B...compare that the Gemma.

[BBC-Esq]

Updated benchmarks posted here, but basically the same findings...

#1676 (comment)

[BBC-Esq]

I promise, I'm not trying to "chart" everyone to death...lol. But here's a comparison with all GGUF variants, "BNB" refers to transformers+bitsandbytes (running 4-bit mode) and "int8" refers to ctranslate2 backend. All same exact prompts and parameters as much as possible:

As in all my other testing, ctranslate2's 8-bit version uses less vram than gguf's but is slower..."quality" is about the same.

In order for the ctranslate2 backend to be preferred, you'd need to (1) have higher quality for the same VRAM or (2) the same quality with lesser vram...AND for there to be a big enough difference that it matters to someone.

Is there going to be a big enough quality difference between GGUF Q5_K_M and ctranslate2's implementation, both of which have the same VRAM...maybe maybe not...look at the speed difference and decide for yourself.

However, if you're able to use a beam size of 5 and still keep VRAM close, that's a big benefit IMHO. Thanks for listening, I'll be quiet for awhile now. ;-)

[BBC-Esq]

Will this converter also work with the similar model located here?

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

[BBC-Esq]

@jncraton Can you convert the 128k context phi-3 model as well? I'd like to test it as well.

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct