This repository contains the open-source code and benchmark results for the paper - Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward.
The benchmark assesses the performance of various compression and inference methods.

Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
Arnav Chavan, Raghav Magazine, Shubham Kushwaha, Deepak Gupta, Merouane Debbah
Nyun AI, Transmute AI Lab, KU 6G Research Cente

All the experiments are performed in isolated Python3.10 environments with method-specific requirements such as package & library versions. The exact repository and branch details can be inferred from .gitmodules

The repository follows a structured format with a branch naming convention of "A100<method>", where <method> denotes the specific evaluation method. The organization within each branch is outlined as follows:

• engine/: This directory contains the implementation of engine methods along with setup, generation, and bench scripts.

• prune/: Here, you'll find the implementation of pruning methods along with associated setup, generation, and bench scripts.

• quant/: This directory houses the implementation of quantization methods, complete with setup, generation, and bench scripts.

• exports/: A shared export folder is structured as exports/MODEL_NAME/METHOD_TYPE/METHOD_NAME/.

• experiments/: This section holds formatted benchmarking results in notebooks, featuring metrics such as RM (run memory), WM (weight memory), and various GPU utilization graphs.

Please note that all setup, generation, and benchmarking scripts (.sh) strive to be kept up-to-date with the latest runs and are tailored to Python 3.10 with CUDA 12.1 (or the version necessary for the method). Adjustments to the scripts may be required, or a different script should be utilized.

Discover various branches dedicated to evaluated methods within the repository:

• A100Exllama: Investigates the Exllama engine with GPTQ quantization.

• A100Exllamav2: Explores the latest ExllamaV2 featuring EXL2 & GPTQ quantizations.

• A100Llamacpp: Examines the CPP implementation of the Llama architecture for enhanced speed.

• A100MLCLLM: Explores MLCLLM, offering extensive hardware and platform support.

• A100TGI: Investigates the Text Generation Inference toolkit, employed for LLM inferences in production.

• A100VLLM: Explores VLLM.

• A100TensorRTLLM: Investigates NVIDIA's TensorRTLLM inference engine.

• A100GPTQ: Explores the GPTQ quantization method through AutoGPTQ.

• A100HF: Investigates multiple quantization methods, alongside baseline generation speeds for each method.

• A1000mniquant: Explores the OmniQuant quantization method.

Note: Each branch is equipped with its own set of updated scripts, which may or may not be synchronized with other branches. Additionally, specific quantization methods might lack dedicated branches; however, the corresponding scripts can be directly referenced in the respective branches or from the main branch. Models and scales directly obtained from the HF Hub were also utilized as needed.

* with fine-tuning

We extend our gratitude to the following repositories and sources for providing essential methods, engines, and datasets utilized in our benchmarking project:

1. Llama-2-7b-hf - Hugging Face model repository for Llama-2-7b.
2. llama.cpp - Source for llama.cpp, a key engine method used in our benchmarks.
3. exllama - Repository for the ExLlama engine method.
4. exllamav2 - Source for ExLlamaV2 engine method.
5. alpaca-cleaned - Alpaca dataset on Hugging Face, utilized in our benchmarks.
6. squeezellm - Repository for SqueezeLLM quantization method.
7. squeezellmgradients - Repository for SqueezeLLM-gradients.
8. omniquant - Source for OmniQuant quantization method.
9. mlcllm - Repository for the MLC-LLM engine method.
10. llmpruner - Source for LLM-Pruner pruning method.
11. tensorrtllm - Source for TensorRT-LLM engine method (branch: release/0.5.0).
12. autogptq - Repository for AutoGPTQ, offering quantization package based on the GPTQ algorithm.
13. autoawq - Repository for AutoAWQ, implementing the AWQ algorithm for 4-bit quantization.
14. vllm - Source for vllm package offering the inference and serving engine

These resources have been instrumental in conducting the benchmarks and evaluations. We appreciate the creators and maintainers of these repositories for their valuable contributions.