Combining Inference and PEFT Tokens in a Batch

[jiazhihao]

Description of changes:

This PR combines LLM inference and PEFT tokens in a batch to enable kernel-level optimizations.

TODOs (first release):

• Fix fusion issue
• Optimizers (PR Peft optimizer #1290)
• Align cross entropy loss
• Implement code to initialize a new PEFT from scratch (instead of continuing the finetuning of existing ones)
• Support upload back to Huggingface (PR HuggingFace Upload Feature #1300)
• Python interface, similar to PEFT
• Add option to add LoRA layers to every linear layer
• Demo!

TODOs (remaining ones):

• Move QKV and output projections from attention into separate operators, to enable adding LoRA to them (PR modify constructor and allocate weights #1428).
• Update implementation, debug and merge token-level finetuning (https://arxiv.org/abs/2402.18789, draft PR Peft token level fwd/bwd #1233)
• Align models other than LLAMA (opt, falcon, etc...)
• Add support for LLAMA-3, Phi, Mistral
• group gemm kernels for multi-tenant lora
• HIP support
• Add softmax automatically (for categorical cross-entropy loss)

Related Issues:

Linked Issues:

• Issue #

Issues closed by this PR:

• Closes #

---

This change is 

[goliaro]

@jiazhihao why is this change required?

[jiazhihao]

Because we used softmax cross entropy loss, so the last layer of the LLM must be softmax