Support batched generation for llama.cpp

[the-crypt-keeper]

Batch generation has landed: ggerganov/llama.cpp#3228

This should make our test suite ~10x faster on GGUF models.

[the-crypt-keeper]

Looks like the example parallel.cpp has 3 problems:

1. there's a hard-coded input template

2. there's a hard-coded reverse prompt (maybe less of an issue):

3. there is a system prompt evaluated separately from the other prompts and this cannot be disabled

[the-crypt-keeper]

Turns out performance of lammacpp batched generation on CUDA devices is pretty awful, top speed-up is ~2x on an A6000. Not going to bother for so little gain.