7_llm_sycl_gpu_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "652ea6c8-8d13-4228-853e-fad46db470f5",
   "metadata": {},
   "source": [
    "# Inference using SYCL backend on AI PC using Llamacpp Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e0aeac-58b1-4114-95f1-7d3a7a4c34f2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how to install LLamacpp for SYCL on Windows with Intel GPUs. It applies to Intel Core Ultra and Core 11 - 14 gen integrated GPUs (iGPUs), as well as Intel Arc Series GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cf7db8-9529-47dd-b41d-81b22c8d5848",
   "metadata": {},
   "source": [
    "## What is an AIPC\n",
    "\n",
    "What is an AI PC you ask?\n",
    "\n",
    "Here is an [explanation](https://www.intel.com/content/www/us/en/newsroom/news/what-is-an-ai-pc.htm#gs.a55so1) from Intel:\n",
    "\n",
    "”An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud. The GPU and CPU can also process these workloads, but the NPU is especially good at low-power AI calculations. The AI PC represents a fundamental shift in how our computers operate. It is not a solution for a problem that didn’t exist before. Instead, it promises to be a huge improvement for everyday PC usages.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4682eb3e-540b-4814-8142-c54efc32f31b",
   "metadata": {},
   "source": [
    "## Install Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f8b6d2-34af-44ad-8363-dea57660bc00",
   "metadata": {},
   "source": [
    "### Step 1: System Preparation\n",
    "\n",
    "To set up your AIPC for running with Intel iGPUs, follow these essential steps:\n",
    "\n",
    "1. Update Intel GPU Drivers: Ensure your system has the latest Intel GPU drivers, which are crucial for optimal performance and compatibility. You can download these directly from Intel's [official website](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) . Once you have installed the official drivers, you could also install Intel ARC Control to monitor the gpu:\n",
    "\n",
    "   <img src=\"Assets/gpu_arc_control.png\">\n",
    "\n",
    "\n",
    "2. Install Visual Studio 2022 Community edition with C++: Visual Studio 2022, along with the “Desktop Development with C++” workload, is required. This prepares your environment for C++ based extensions used by the intel SYCL backend that powers accelerated Ollama. You can download VS 2022 Community edition from the official site, [here](https://visualstudio.microsoft.com/downloads/).\n",
    "\n",
    "3. Install conda-forge: conda-forge will manage your Python environments and dependencies efficiently, providing a clean, minimal base for your Python setup. Visit conda-forge's [installation site](https://conda-forge.org/download/) to install for windows.\n",
    "\n",
    "4. Install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html)\n",
    "\n",
    "   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8040fd21-7782-4b97-a0eb-327816328f17",
   "metadata": {},
   "source": [
    "## Step 2: Install Llamacpp python for SYCL\n",
    "The llama.cpp SYCL backend is designed to support Intel GPU firstly. Based on the cross-platform feature of SYCL.\n",
    "\n",
    "### After installation of conda-forge, open the Miniforge Prompt, and create a new python environment:\n",
    "  ```\n",
    "  conda create -n llm-sycl python=3.11\n",
    "\n",
    "  ```\n",
    "\n",
    "### Activate the new environment\n",
    "```\n",
    "conda activate llm-sycl\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm4.png\">\n",
    "\n",
    "### With the llm-sycl environment active, enable oneAPI environment. \n",
    "Type oneapi in the windows search and then open the Intel oneAPI command prompt for Intel 64 for Visual Studio 2022 App.\n",
    "\n",
    "<img src=\"Assets/oneapi1.png\">\n",
    "\n",
    "#### Run the below command in the VS command prompt and you should see the below sycl devices displayed in the console\n",
    "There should be one or more level-zero GPU devices displayed as ext_oneapi_level_zero:gpu.\n",
    "\n",
    "```\n",
    "sycl-ls\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/oneapi2.png\">\n",
    "\n",
    "### Install build tools\n",
    "\n",
    "* Download & install [cmake for Windows](https://cmake.org/download/):\n",
    "* The new Visual Studio will install Ninja as default. (If not, please install it manually: https://ninja-build.org/)\n",
    "\n",
    "### Install llama.cpp Python\n",
    "\n",
    "  \n",
    "* On the oneAPI command line window, step into the llama.cpp main directory and run the following:\n",
    "  \n",
    "  ```\n",
    "  @call \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64 --force\n",
    "\n",
    "    Open a new terminal and perform the following steps:\n",
    "\n",
    "\n",
    "# Set the environment variables\n",
    "    set CMAKE_GENERATOR=Ninja\n",
    "    set CMAKE_C_COMPILER=cl\n",
    "    set CMAKE_CXX_COMPILER=icx\n",
    "    set CXX=icx\n",
    "    set CC=cl\n",
    "    set CMAKE_ARGS=\"-DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DCMAKE_CXX_COMPILER=icx -DCMAKE_C_COMPILER=cl\"\n",
    "    \n",
    "    pip install llama-cpp-python -U --force --no-cache-dir –-verbose\n",
    "\n",
    "### Below shows a simple example to show how to run a community GGUF model with llama.cpp for SYCL\n",
    "* Download the model from huggingface and prepare the model for inference\n",
    "* Run the model as below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32f188eb-4cbf-49cd-8e88-0f3c6da19894",
   "metadata": {},
   "outputs": [],
   "source": [
    "!@call \"C:\\\\Program Files (x86)\\\\Intel\\\\oneAPI\\\\setvars.bat\" intel64 --force"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0f21fb7-cb4a-421e-97f1-d763d989cebd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_cpp import Llama\n",
    "prompt = \"Write a story about Pandas\"\n",
    "prompt_template=f'''SYSTEM: You are a helpful, respectful and honest assistant. Always answer as helpfully.\n",
    "\n",
    "USER: {prompt}\n",
    "\n",
    "ASSISTANT:\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01173f7a-0725-4b34-aabc-7e6582b87da4",
   "metadata": {},
   "source": [
    "## Run the inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c71841b-7f2f-4907-a2e7-ac112ea9b0d7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "llm = Llama(\n",
    "      model_path=\".\\models\\phi-2.Q5_K_M.gguf\",\n",
    "      chat_format=\"llama-2\",\n",
    "      n_gpu_layers=-1, # use GPU acceleration\n",
    "      seed=1337, # set a specific seed\n",
    "      n_ctx=2048, # set the context window\n",
    "      n_threads=16,\n",
    "      f16_kv=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ba9c09-568c-4535-8bb1-0bcba22c31b6",
   "metadata": {},
   "source": [
    "The below code creates a chat completion object specifies the input messages and tells the model to generate text in a streaming fashion.\n",
    "Then we iterates over the generated chunks of text to generate streaming response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bedf4abc-9837-4d6f-90f2-aa4a5d0f6eec",
   "metadata": {},
   "outputs": [],
   "source": [
    "output = llm.create_chat_completion(\n",
    "    messages=[\n",
    "        { \"role\": \"system\", \"content\": \"You are a story writing assistant.\" },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": prompt\n",
    "        }\n",
    "    ],\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "for chunk in output:\n",
    "    delta = chunk['choices'][0]['delta']\n",
    "    if 'content' in delta:   \n",
    "        print(delta['content'], end='', flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e65241-f87d-42b0-9b6d-41826a6583a0",
   "metadata": {},
   "source": [
    "## Pulling models from Huggingface hub\n",
    "\n",
    "The below code loads the pre-trained Llama model from huggingface repository specified by the repository ID, filename, and other parameters for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c777d38-6990-47d9-9333-5470cd80b68b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from llama_cpp import Llama\n",
    "\n",
    "prompt = \"Write a story about Pandas\"\n",
    "\n",
    "llm = Llama.from_pretrained(repo_id=\"TheBloke/phi-2-GGUF\",\n",
    "                                filename=\"*Q5_K_M.gguf\",\n",
    "                                chat_format=\"llama-2\",\n",
    "                                n_gpu_layers=-1, # Uncomment to use GPU acceleration\n",
    "                                seed=1337, # Uncomment to set a specific seed\n",
    "                                n_ctx=2048, # Uncomment to increase the context window\n",
    "                                n_threads=16,\n",
    "                                f16_kv=True,\n",
    "                               )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4785f66e-a0f1-43fb-8195-7314c41e0438",
   "metadata": {},
   "source": [
    "The below code creates a chat completion object specifies the input messages and tells the model to generate text in a streaming fashion.\n",
    "Then we iterates over the generated chunks of text to generate streaming response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2843365-7c12-412f-ba93-3dd89ec7f857",
   "metadata": {},
   "outputs": [],
   "source": [
    "output = llm.create_chat_completion(\n",
    "    messages=[\n",
    "        { \"role\": \"system\", \"content\": \"You are a story writing assistant.\" },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": prompt\n",
    "        }\n",
    "    ],\n",
    "    max_tokens=256,\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "for chunk in output:\n",
    "    delta = chunk['choices'][0]['delta']\n",
    "    if 'content' in delta:   \n",
    "        print(delta['content'], end='', flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061ce086-54e6-41b7-8283-e1e66bcd4a47",
   "metadata": {},
   "source": [
    "## Example output\n",
    "\n",
    "<img src=\"Assets/output_latest.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92387fa9-2376-49a7-a94b-a29f254a0471",
   "metadata": {},
   "source": [
    "* Reference:https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac73234-1851-42ad-9b6c-67ba9562db32",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}