Performance on Apple silicon

[marella]

Context: #1 (comment)

@bgonzalezfractal did you notice any performance improvement just by changing the threads parameter?

If you don't have the latest quantized models, you can go back to the previous commit using:

git checkout e707f99
git submodule update

Here you can run the build commands and check:

cmake -S . -B build
cmake --build build

[bgonzalezfractal]

@marella I haven't been able to use the llm again, once we manage to solve the starcoder quantize issue, I can post the M1 Pro 64GB performance.

[marella]

Hi, I'm guessing the issue might be related to performance - it is running too slow so taking time to print the output.

By default print(llm(...)) function doesn't show any output until all the text is generated. So you can try the new stream option to see text as soon as a token is generated:

for token in llm('def fibo(', max_new_tokens=5, stream=True):
    print(token, end='', flush=True)

---

The code used by starcoder example and this library is same. The only difference is in the way it is built. So building the library from source can help validate if building locally improves performance and may prevent the library from being stuck:

git clone --recurse-submodules https://github.com/marella/ctransformers
cd ctransformers
./scripts/build.sh

llm = AutoModelForCausalLM.from_pretrained(..., lib='/path/to/ctransformers/build/lib/libctransformers.dylib')

[marella]

See #8