Making Auto-Llama more powerful with a LoRA trainig step

[maddes8cht]

The biggest problem of Auto-Llama-cpp seems to be the small context size of the Llama models.

Since Llama-cpp now has the ability to import LoRAs that - depending on the size of the training material - can be trained themselves in a reasonable amount of time, an optional intermediate step for Llama-cpp could be an interesting feature:

In the first working mode, Auto-Llama-cpp collects all data that can be seen as relevant for its task and prepares them as training data for its own model.

In the second step, a LoRA is trained with exactly this traings data.

In the third step, the Llama model is started with the trained Lora - now the Llama is an expert on the topic to be processed.
Only in the third step Auto-LLama-cpp solves the given tasks.

In my opinion, this would make Auto-Llama-cpp much more powerful, possibly more powerful than Auto-GPT itself.
Of course, the whole thing would be extremely slow - on the other hand, steps 1 and 2 might actually be processed very autonomously, unattended, since Auto-Llama-cpp only collects information in the process.