example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2ab4ccad-6e8d-4c14-af1c-485b24f7d2a0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Using LlamaIndex and llamafile to build a local, private research assistant\n",
    "\n",
    "[llamafile](https://github.com/Mozilla-Ocho/llamafile), an open source project from Mozilla, is one of the simplest ways to run a large language model (LLM) on your laptop. All you have to do is download a llamafile from [HuggingFace](https://huggingface.co/models?sort=trending&search=llamafile) then run the file. That's it. **On most computers, you won't need to install anything.**\n",
    "\n",
    "There are a few reasons why you might want to run an LLM on your laptop, including:\n",
    "\n",
    "1. Privacy: Running locally means you won't have to share your data with third parties.\n",
    "2. High availability: Run your LLM-based app without an internet connection.\n",
    "3. Bring your own model: You can easily test many different open-source LLMs (anything available on HuggingFace) and see which one works best for your task.\n",
    "4. Free debugging/testing: Local LLMs allow you to test many parts of an LLM-based system without paying for API calls.\n",
    "\n",
    "In this blog post, we'll show how to set up a llamafile and use it to run a local LLM on your computer. Then, we'll show how to use LlamaIndex with your llamafile as the LLM & embedding backend for a local RAG-based research assistant. You won't have to sign up for any cloud service or send your data to any third party--everything will just run on your laptop.\n",
    "\n",
    "## Download and run a llamafile\n",
    "\n",
    "First, what is a llamafile? A llamafile is an executable LLM that you can run on your own computer. It contains the weights for a given open source LLM, as well as everything needed to actually run that model on your computer. There's nothing to install or configure (with a few caveats, discussed [here](https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#gotchas)). \n",
    "\n",
    "Each llamafile bundles 1) model weights & metadata in gguf format + 2) a copy of [`llama.cpp`](https://github.com/ggerganov/llama.cpp) specially compiled using [Cosmopolitan Libc](https://github.com/jart/cosmopolitan). This allows the models to run on most computers without additional installation. llamafiles also come with a ChatGPT-like browser interface, a CLI, and an OpenAI-compatible REST API for chat models.\n",
    "\n",
    "There are only 2 steps to setting up a llamafile:\n",
    "\n",
    "1. Download a llamafile\n",
    "2. Make the llamafile executable\n",
    "\n",
    "We'll go through each step in detail below.\n",
    "\n",
    "### Step 1: Download a llamafile\n",
    "\n",
    "There are many llamafiles available on the [HuggingFace model hub](https://huggingface.co/models?sort=trending&search=llamafile) (just search for 'llamafile') but for the purpose of this walkthrough, we'll use [TinyLlama-1.1B](https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile?download=true) (0.67 GB, [model info](https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-GGUF)). To download the model, you can either click this download link: [TinyLlama-1.1B](https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile?download=true) or open a terminal and use something like `wget`. The download should take 5-10 minutes depending on the quality of your internet connection. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ded722d-dcdf-41fd-b1a9-02b7df6d9d36",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wget https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eee98ff-6391-4295-9cfc-b9b8051689fb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "This model is small and won't be very good at actually answering questions but, since it's a relatively quick download and its inference speed will allow you to index your vector store in just a few minutes, it's good enough for the examples below. For a higher-quality LLM, you may want to use a larger model like [Mistral-7B-Instruct](https://huggingface.co/Mozilla/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile?download=true) (5.15 GB, [model info](https://huggingface.co/Mozilla/Mistral-7B-Instruct-v0.2-llamafile))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3528f7d8-afed-4c99-8cae-6e07df1772ee",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Step 2: Make the llamafile executable\n",
    "\n",
    "If you didn't download the llamafile from the command line, figure out where your browser stored your downloaded llamafile. \n",
    "\n",
    "Now, open your computer's terminal and, if necessary, go to the directory where your llamafile is stored: `cd path/to/downloaded/llamafile`\n",
    "\n",
    "**If you're using macOS, Linux, or BSD**, you'll need to grant permission for your computer to execute this new file. (You only need to do this once.):\n",
    "\n",
    "**If you're on Windows, instead just rename the file by adding \".exe\" on the end** e.g. rename `TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile` to `TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile.exe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da266cff-506c-445d-a007-936e2f7f96f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92db82f0-58db-47a2-9eba-a0bc1950a1ee",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Kick the tires\n",
    "\n",
    "Now, your llamafile should be ready to go. First, you can check which version of the llamafile library was used to build the llamafile binary you should downloaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "604811a4-d77a-4c41-bfce-61353960cdbe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-01T20:10:34.045269Z",
     "start_time": "2024-04-01T20:10:33.916275Z"
    },
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llamafile v0.7.0\r\n"
     ]
    }
   ],
   "source": [
    "!./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2789678-a0f6-47a0-b028-ae8092a701f7",
   "metadata": {},
   "source": [
    "The post was written using a model built with `llamafile v0.7.0`. If your llamafile displays a different version and some of the steps below don't work as expected, please [post an issue on the llamafile issue tracker](https://github.com/Mozilla-Ocho/llamafile/issues).\n",
    "\n",
    "The easiest way to use your llamafile is via its built-in chat interface. In a terminal, run "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "333cdc4a3174804d",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea2921640c49968",
   "metadata": {
    "collapsed": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your browser should open automatically and display a chat interface. (If it doesn't, just open your browser and point it at http://localhost:8080). When you're done chatting, return to your terminal and hit `Control-C` to shut down llamafile. If you're running these commands inside a notebook, just interrupt the notebook kernel to stop the llamafile."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49119268-5c6f-4ecf-998c-503ef4d07fb8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "In the rest of this walkthrough, we'll be using the llamafile's built-in inference server instead of the browser interface. The llamafile's server provides a REST API for interacting with the TinyLlama LLM via HTTP. Full server API documentation is available [here](https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/README.md#api-endpoints). To start the llamafile in server mode, run:\n",
    "\n",
    "```bash\n",
    "./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser --embedding\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c909ec1-0a86-48a2-8411-f038788662d0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Summary: Download and run a llamafile\n",
    "\n",
    "```bash\n",
    "# 1. Download the llamafile-ized model\n",
    "wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n",
    "\n",
    "# 2. Make it executable (you only need to do this once)\n",
    "chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n",
    "\n",
    "# 3. Run in server mode\n",
    "./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser --embedding\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee137781-9ff6-4558-8eff-5d272cfe169e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Build a research assistant using LlamaIndex and llamafile\n",
    "\n",
    "Now, we'll show how to use LlamaIndex with your llamafile to build a research assistant to help you learn about some topic of interest--for this post, we chose [homing pigeons](https://en.wikipedia.org/wiki/Homing_pigeon). We'll show how to prepare your data, index into a vector store, then query it. \n",
    "\n",
    "One of the nice things about running an LLM locally is privacy. You can mix both \"public data\" like Wikipedia pages and \"private data\" without worrying about sharing your data with a third party. Private data could include e.g. your private notes on a topic or PDFs of classified content. As long as you use a local LLM (and a local vector store), you won't have to worry about leaking data. Below, we'll show how to combine both types of data. Our vector store will include Wikipedia pages, an Army manual on caring for homing pigeons, and some brief notes we took while we were reading about this topic. \n",
    "\n",
    "To get started, download our example data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31c8fd24-6953-446d-82bf-5547baa534fd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "\n",
    "# Download 'The Homing Pigeon' manual from Project Gutenberg\n",
    "!wget https://www.gutenberg.org/cache/epub/55084/pg55084.txt -O data/The_Homing_Pigeon.txt\n",
    "\n",
    "# Download some notes on homing pigeons\n",
    "!wget https://gist.githubusercontent.com/k8si/edf5a7ca2cc3bef7dd3d3e2ca42812de/raw/24955ee9df819e21975b1dd817938c1bfe955634/homing_pigeon_notes.md -O data/homing_pigeon_notes.md"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2489e13a-b7cc-42be-a29b-ad4be43e5608",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Next, we'll need to install LlamaIndex and a few of its integrations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cb0a872-57d2-437b-ba7f-0dc5d47d02b0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Install llama-index\n",
    "!pip install llama-index-core\n",
    "# Install llamafile integrations and SimpleWebPageReader\n",
    "!pip install llama-index-embeddings-llamafile llama-index-llms-llamafile llama-index-readers-web"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ac2f188-d6a0-4adc-9ec1-39d980a4c549",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Start your llamafile server and configure LlamaIndex\n",
    "\n",
    "In this example, we'll use the same llamafile to both produce the embeddings that will get indexed in our vector store and as the LLM that will answer queries later on. (However, there is no reason you can't use one llamafile for the embeddings and separate llamafile for the LLM functionality--you would just need to start the llamafile servers on different ports.)\n",
    "\n",
    "To start the llamafile server, open a terminal and run:\n",
    "\n",
    "```bash\n",
    "./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser --embedding --port 8080\n",
    "```\n",
    "\n",
    "Now, we'll configure LlamaIndex to use this llamafile:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0bb2a20d-29a0-4d25-b9d4-f4e81db755fb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Configure LlamaIndex\n",
    "from llama_index.core import Settings\n",
    "from llama_index.embeddings.llamafile import LlamafileEmbedding\n",
    "from llama_index.llms.llamafile import Llamafile\n",
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "\n",
    "Settings.embed_model = LlamafileEmbedding(base_url=\"http://localhost:8080\")\n",
    "\n",
    "Settings.llm = Llamafile(\n",
    "    base_url=\"http://localhost:8080\",\n",
    "    temperature=0,\n",
    "    seed=0\n",
    ")\n",
    "\n",
    "# Also set up a sentence splitter to ensure texts are broken into semantically-meaningful chunks (sentences) that don't take up the model's entire\n",
    "# context window (2048 tokens). Since these chunks will be added to LLM prompts as part of the RAG process, we want to leave plenty of space for both \n",
    "# the system prompt and the user's actual question.\n",
    "Settings.transformations = [\n",
    "    SentenceSplitter(\n",
    "        chunk_size=256, \n",
    "        chunk_overlap=5\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bf428ce-ae45-4fa2-b42d-521a048ae190",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Prepare your data and build a vector store\n",
    "\n",
    "Now, we'll load our data and index it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "39296b12-a3cb-4ed5-9d1a-68fab3954a31",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 139.65file/s]\n"
     ]
    }
   ],
   "source": [
    "# Load local data\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "local_doc_reader = SimpleDirectoryReader(input_dir='./data')\n",
    "docs = local_doc_reader.load_data(show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a4b84695-bce8-4875-a73f-cbb05b61a1ea",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We'll load some Wikipedia pages as well\n",
    "from llama_index.readers.web import SimpleWebPageReader\n",
    "urls = [\n",
    "    'https://en.wikipedia.org/wiki/Homing_pigeon',\n",
    "    'https://en.wikipedia.org/wiki/Magnetoreception',\n",
    "]\n",
    "web_reader = SimpleWebPageReader(html_to_text=True)\n",
    "docs.extend(web_reader.load_data(urls))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b69bd884-bd9d-4a9e-8eab-ab75b4609460",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "392ec29acf0f4f83982c54f37521d309",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48555bdfff504957b43284af4e6f4d2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/459 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Build the index\n",
    "from llama_index.core import VectorStoreIndex\n",
    "\n",
    "index = VectorStoreIndex.from_documents(\n",
    "    docs,\n",
    "    show_progress=True,\n",
    ")\n",
    "# Save the index\n",
    "index.storage_context.persist(persist_dir=\"./storage\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7d9b83-e6be-428a-a660-9f0b10b41e5a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Query your research assistant\n",
    "\n",
    "Finally, we're ready to ask some questions about homing pigeons.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f8ad6852-ba6c-4b3e-b2dd-903baedfd5b3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6ee1c0a4-44c9-4f3f-90e4-89beb8e75f81",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ksilverstein/dev/llamafile-examples/integrations/llama-index/venv/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py:58: RuntimeWarning: invalid value encountered in scalar divide\n",
      "  return product / norm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Homing pigeons were used for a variety of purposes, including military reconnaissance, communication, and transportation. They were also used for scientific research, such as studying the behavior of birds in flight and their migration patterns. In addition, they were used for religious ceremonies and as a symbol of devotion and loyalty. Overall, homing pigeons played an important role in the history of aviation and were a symbol of the human desire for communication and connection.\n"
     ]
    }
   ],
   "source": [
    "print(query_engine.query(\"What were homing pigeons used for?\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0a65ef47-cee5-4ecd-b63e-507a4586f674",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The context information provided in the given context is that homing pigeons were first used in the 19th century. However, prior knowledge would suggest that homing pigeons have been used for navigation and communication for centuries.\n"
     ]
    }
   ],
   "source": [
    "print(query_engine.query(\"When were homing pigeons first used?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a56709b-8e67-4ea0-b9a4-50761243b826",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "In this post, we've shown how to download and set up an LLM running locally via llamafile. Then, we showed how to use this LLM with LlamaIndex to build a simple RAG-based research assistant for learning about homing pigeons. Your assistant ran 100% locally: you didn't have to pay for API calls or send data to a third party. \n",
    "\n",
    "As a next step, you could try running the examples above with a better model like [Mistral-7B-Instruct](https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile?download=true). You could also try building a research assistant for different topic like \"semiconductors\" or \"how to bake bread\".\n",
    "\n",
    "To find out more about llamafile, check out the project on [GitHub](https://github.com/Mozilla-Ocho/llamafile), read this [blog post](https://justine.lol/oneliners/) on bash one-liners using LLMs, or say hi to the community on [Discord](https://discord.com/invite/teDuGYVTB2)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv-llamaindex-blog",
   "language": "python",
   "name": "venv-llamaindex-blog"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}