It is possible compile gpt-neox with cuda ???

[ArturK-85]

When i trying compile with -DGGML_CUBLAS=ON gpt-neox example run only on cpu.

[ggerganov]

It's possible. You have to offload the tensors used for matrix multiplication to the GPU.
Something like this:

https://github.com/ggerganov/llama.cpp/blob/905d87b70aa189623d500a28602d7a3a755a4769/llama.cpp#L1030-L1056

[ArturK-85]

Thank you for your quick response, I was very interested in your project, it opens up completely new possibilities in AI optimization. I will use the tips. Thank you for your help and explanation