Comparative Evaluation of Energy Efficiency in Large Language Models: Analyzing Improvements Across Incremental Versions in Inference Tasks

This experiment is a group project for the GreenLab course, under the Computer Science Master's programme at VU Amsterdam.

Experiment Candidates

1. Alibaba Cloud’s QWen:

The versions that are going to be tested are incrementally as follows:

• Qwen1-7B
• Qwen1.5-7B
• Qwen2-7B
• Qwen2.5-7B

2. Google’s Gemma

The versions that are going to be tested are incrementally as follows:

• Gemma-2B-it
• Gemma-1.1-2B-it
• Gemma-2-2B-it

These versions are all instruct versions of the Gemma model. This will not affect our study because we are not drawing any comparative conclusions on the model performance between the LLM candidates.

3. Mistralai’s Mistral

The versions that are going to be tested are incrementally as follows:

• Mistral-7B-Instruct-v0.1
• Mistral-7B-Instruct-v0.2
• Mistral-7B-Instruct-v0.3

These versions are all instruct versions of the open-source Mistral model model. Just as for Gemma, this will not affect our study because we are not drawing any comparative conclusions on the model performance between the LLM candidates.

Tool Selection

Experiment Automation

We automated the experiment using the following framework: Experiment-Runner

Metrics Extraction

• Energy Consumption (CPU/GPU):
  • PowerJoular :
    • CPU Consumption (Joules)
    • GPU Consumption (Joules)
• Resource Utilization:
  • top : CPU Utilization (%), Memory Utilization (Bytes/%)
  • nvidia-smi : GPU Utilization (%), GPU VMemory (Bytes/%)
• Model Performance:
  • DeepEval:
    • Contextual Relevancy (0-1 Score)
    • Answer Correctness (0-1 Score)
    • Summarization Completeness (0-1 Score)

Running the Experiment

Installation

git clone --recursive https://github.com/andrei-calin-dragomir/greenlab-course-project.git
cd ./greenlab-course-project
python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
cd ./experiment-runner
pip install -r requirements.txt

Execution

python3 -m venv venv
cd ./experiment-runner
python experiment-runner/ ../RunnerConfig.py

Execution Flow

The workflow of the experiment is defined as:

1. BEFORE_EXPERIMENT
2. BEFORE_RUN
3. START_RUN
4. START_MEASUREMENT
5. INTERACT
6. STOP_MEASUREMENT
7. STOP_RUN
8. POPULATE_RUN_DATA
9. AFTER_EXPERIMENT