1_native_gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "652ea6c8-8d13-4228-853e-fad46db470f5",
   "metadata": {},
   "source": [
    "# Inference using Llamacpp on Intel GPUs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e0aeac-58b1-4114-95f1-7d3a7a4c34f2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how to run an LLM inference on Windows with Intel GPUs. It applies to Intel Core Ultra and Core 11 - 14 gen integrated GPUs (iGPUs), as well as Intel Arc Series GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cf7db8-9529-47dd-b41d-81b22c8d5848",
   "metadata": {},
   "source": [
    "## What is an AIPC\n",
    "\n",
    "What is an AI PC you ask?\n",
    "\n",
    "Here is an [explanation](https://www.intel.com/content/www/us/en/newsroom/news/what-is-an-ai-pc.htm#gs.a55so1) from Intel:\n",
    "\n",
    "”An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud. The GPU and CPU can also process these workloads, but the NPU is especially good at low-power AI calculations. The AI PC represents a fundamental shift in how our computers operate. It is not a solution for a problem that didn’t exist before. Instead, it promises to be a huge improvement for everyday PC usages.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4682eb3e-540b-4814-8142-c54efc32f31b",
   "metadata": {},
   "source": [
    "## Install Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f8b6d2-34af-44ad-8363-dea57660bc00",
   "metadata": {},
   "source": [
    "### Step 1: System Preparation\n",
    "\n",
    "To set up your AIPC for running with Intel iGPUs, follow these essential steps:\n",
    "\n",
    "1. Update Intel GPU Drivers: Ensure your system has the latest Intel GPU drivers, which are crucial for optimal performance and compatibility. You can download these directly from Intel's [official website](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) . Once you have installed the official drivers, you could also install Intel ARC Control to monitor the gpu:\n",
    "\n",
    "   <img src=\"Assets/gpu_arc_control.png\">\n",
    "\n",
    "\n",
    "2. Install Visual Studio 2022 Community edition with C++: Visual Studio 2022, along with the “Desktop Development with C++” workload, is required. This prepares your environment for C++ based extensions used by the intel SYCL backend that powers accelerated Ollama. You can download VS 2022 Community edition from the official site, [here](https://visualstudio.microsoft.com/downloads/).\n",
    "\n",
    "3. Install conda-forge: conda-forge will manage your Python environments and dependencies efficiently, providing a clean, minimal base for your Python setup. Visit conda-forge's [installation site](https://conda-forge.org/download/) to install for windows.\n",
    "\n",
    "   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8040fd21-7782-4b97-a0eb-327816328f17",
   "metadata": {},
   "source": [
    "## Step 2: Setup the environment and install required libraries\n",
    "\n",
    "### After installation of conda-forge, open the Miniforge Prompt, and create a new python environment:\n",
    "  ```\n",
    "  conda create -n llm-cpp python=3.11\n",
    "\n",
    "  ```\n",
    "\n",
    "### Activate the new environment\n",
    "```\n",
    "conda activate llm-cpp\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm4.png\">\n",
    "\n",
    "### With the llm-cpp environment active, use pip to install required libraries for suppport. \n",
    "\n",
    "```\n",
    "pip install --pre --upgrade ipex-llm[cpp]\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm5.png\">\n",
    "\n",
    "### Create llama-cpp directory\n",
    "\n",
    "```\n",
    "mkdir llama-cpp\n",
    "cd llama-cpp\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm6.png\">\n",
    "\n",
    "### Please run the following command with administrator privilege in Miniforge Prompt. We should see many soft links of llama.cpp’s executable files in current directory.\n",
    "```\n",
    "init-llama-cpp.bat\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm7.png\">\n",
    "\n",
    "### Set the following environment variables according to your device to use GPU acceleration\n",
    "For Intel iGPU:\n",
    "```\n",
    "set SYCL_CACHE_PERSISTENT=1\n",
    "\n",
    "```\n",
    "### Below shows a simple example to show how to run a community GGUF model\n",
    "* Download and run the model for example as below \n",
    "\n",
    "```\n",
    "main -m mistral-7b-instruct-v0.1.Q4_K_M.gguf -n 32 --prompt \"What is AI\" -t 8 -e -ngl 33 --color\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm8.png\">\n",
    "\n",
    "### Below is an example output\n",
    "\n",
    "<img src=\"Assets/llm9.png\">\n",
    "\n",
    "\n",
    "<img src=\"Assets/llm10.png\">\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65e5cd95-18a4-4879-9d3d-05e302448ff6",
   "metadata": {},
   "outputs": [],
   "source": [
    "! C:\\workshop\\llama-cpp\\main.exe -m ../models/llama-2-7b-chat.Q5_K_M.gguf -n 100 --prompt \"What is AI\" -t 16 -ngl 999 --color -e "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92387fa9-2376-49a7-a94b-a29f254a0471",
   "metadata": {},
   "source": [
    "* Reference: https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac73234-1851-42ad-9b6c-67ba9562db32",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}