perf: parallelize quantization

[jon-chuang]

llama.cpp/llama.cpp

Line 1558 in 8b67998

static void llama_model_quantize_internal(const std::string & fname_inp, const std::string & fname_out, enum llama_ftype ftype) {

Is currently single threaded. Quantization is quite slow (vicuna 7B: 65156.31 ms, vicuna 13B: 129902.48 ms).

[sw]

@ikawrakow did that in #896, see kQuantizeQ4 in ggml_extra.cpp, but that's for a new quantization scheme.

llama.cpp/ggml_extra.cpp

Lines 287 to 291 in 6bfb00a

	int nthread = std::min(nchunk, int(std::thread::hardware_concurrency()));
	std::vector<std::thread> workers(nthread-1);
	for (auto& w : workers) w = std::thread(compute);
	compute();
	for (auto& w : workers) w.join();

It did indeed speed things up. This could probably be integrated into llama_model_quantize_internal so that a new cpp module isn't necessary.

[jon-chuang]

Is the new quantization scheme the one that minimizes MSE against the original weights?

[sw]

Resolved by #1075