Ministral_8B_Instruct_2410_HF_GGUF_TEST.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-Comparison-Colab-UI/blob/main/Ministral_8B_Instruct_2410_HF_GGUF_TEST.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## Ministral-8B-Instruct-2410-HF-GGUF-TEST WebUI\n",
        "\n",
        "1. Run the following cell, takes ~5 min\n",
        "(You may need to confirm to proceed by typing \"Y\")\n",
        "2. Click the gradio link at the bottom\n",
        "3. Instruction Template:\n",
        "\n",
        "```\n",
        "<s>[INST] {prompt}[/INST] </s>\n",
        "```\n",
        "\n",
        "Original model: https://huggingface.co/prince-canuma/Ministral-8B-Instruct-2410-HF\n",
        "\n",
        "Quantized model: https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-HF-GGUF-TEST\n",
        "\n",
        "Want to try other local LLMs? Check out this repo: https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D-MiHp_bveP6",
        "outputId": "2aecbfaa-cc81-475d-8f4a-1820e1c63c49"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 123629 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link\n",
            "\n",
            "Looking in indexes: https://download.pytorch.org/whl/cu121\n",
            "Collecting torch==2.2.2\n",
            "  Downloading https://download.pytorch.org/whl/cu121/torch-2.2.2%2Bcu121-cp310-cp310-linux_x86_64.whl (757.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m757.3/757.3 MB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting torchvision==0.17.2\n",
            "  Downloading https://download.pytorch.org/whl/cu121/torchvision-0.17.2%2Bcu121-cp310-cp310-linux_x86_64.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m83.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting torchaudio==2.2.2\n",
            "  Downloading https://download.pytorch.org/whl/cu121/torchaudio-2.2.2%2Bcu121-cp310-cp310-linux_x86_64.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m59.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (1.13.3)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.4.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (2024.6.1)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m50.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m38.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m103.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-curand-cu12==10.3.2.106 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m13.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nccl-cu12==2.19.3 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.0/166.0 MB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nvtx-cu12==12.1.105 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting triton==2.2.0 (from torch==2.2.2)\n",
            "  Downloading https://download.pytorch.org/whl/triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (167.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m167.9/167.9 MB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision==0.17.2) (1.26.4)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision==0.17.2) (10.4.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.2) (12.6.77)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.2.2) (3.0.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.2.2) (1.3.0)\n",
            "Installing collected packages: triton, nvidia-nvtx-cu12, nvidia-nccl-cu12, nvidia-cusparse-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, nvidia-cusolver-cu12, nvidia-cudnn-cu12, torch, torchvision, torchaudio\n",
            "  Attempting uninstall: nvidia-nccl-cu12\n",
            "    Found existing installation: nvidia-nccl-cu12 2.23.4\n",
            "    Uninstalling nvidia-nccl-cu12-2.23.4:\n",
            "      Successfully uninstalled nvidia-nccl-cu12-2.23.4\n",
            "  Attempting uninstall: nvidia-cusparse-cu12\n",
            "    Found existing installation: nvidia-cusparse-cu12 12.5.4.2\n",
            "    Uninstalling nvidia-cusparse-cu12-12.5.4.2:\n",
            "      Successfully uninstalled nvidia-cusparse-cu12-12.5.4.2\n",
            "  Attempting uninstall: nvidia-curand-cu12\n",
            "    Found existing installation: nvidia-curand-cu12 10.3.7.77\n",
            "    Uninstalling nvidia-curand-cu12-10.3.7.77:\n",
            "      Successfully uninstalled nvidia-curand-cu12-10.3.7.77\n",
            "  Attempting uninstall: nvidia-cufft-cu12\n",
            "    Found existing installation: nvidia-cufft-cu12 11.3.0.4\n",
            "    Uninstalling nvidia-cufft-cu12-11.3.0.4:\n",
            "      Successfully uninstalled nvidia-cufft-cu12-11.3.0.4\n",
            "  Attempting uninstall: nvidia-cuda-runtime-cu12\n",
            "    Found existing installation: nvidia-cuda-runtime-cu12 12.6.77\n",
            "    Uninstalling nvidia-cuda-runtime-cu12-12.6.77:\n",
            "      Successfully uninstalled nvidia-cuda-runtime-cu12-12.6.77\n",
            "  Attempting uninstall: nvidia-cuda-cupti-cu12\n",
            "    Found existing installation: nvidia-cuda-cupti-cu12 12.6.80\n",
            "    Uninstalling nvidia-cuda-cupti-cu12-12.6.80:\n",
            "      Successfully uninstalled nvidia-cuda-cupti-cu12-12.6.80\n",
            "  Attempting uninstall: nvidia-cublas-cu12\n",
            "    Found existing installation: nvidia-cublas-cu12 12.6.3.3\n",
            "    Uninstalling nvidia-cublas-cu12-12.6.3.3:\n",
            "      Successfully uninstalled nvidia-cublas-cu12-12.6.3.3\n",
            "  Attempting uninstall: nvidia-cusolver-cu12\n",
            "    Found existing installation: nvidia-cusolver-cu12 11.7.1.2\n",
            "    Uninstalling nvidia-cusolver-cu12-11.7.1.2:\n",
            "      Successfully uninstalled nvidia-cusolver-cu12-11.7.1.2\n",
            "  Attempting uninstall: nvidia-cudnn-cu12\n",
            "    Found existing installation: nvidia-cudnn-cu12 9.5.0.50\n",
            "    Uninstalling nvidia-cudnn-cu12-9.5.0.50:\n",
            "      Successfully uninstalled nvidia-cudnn-cu12-9.5.0.50\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.4.1+cu121\n",
            "    Uninstalling torch-2.4.1+cu121:\n",
            "      Successfully uninstalled torch-2.4.1+cu121\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.19.1+cu121\n",
            "    Uninstalling torchvision-0.19.1+cu121:\n",
            "      Successfully uninstalled torchvision-0.19.1+cu121\n",
            "  Attempting uninstall: torchaudio\n",
            "    Found existing installation: torchaudio 2.4.1+cu121\n",
            "    Uninstalling torchaudio-2.4.1+cu121:\n",
            "      Successfully uninstalled torchaudio-2.4.1+cu121\n",
            "Successfully installed nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvtx-cu12-12.1.105 torch-2.2.2+cu121 torchaudio-2.2.2+cu121 torchvision-0.17.2+cu121 triton-2.2.0\n",
            "Cloning into 'text-generation-webui'...\n",
            "remote: Enumerating objects: 19214, done.\u001b[K\n",
            "remote: Counting objects: 100% (188/188), done.\u001b[K\n",
            "remote: Compressing objects: 100% (94/94), done.\u001b[K\n",
            "remote: Total 19214 (delta 132), reused 132 (delta 94), pack-reused 19026 (from 1)\u001b[K\n",
            "Receiving objects: 100% (19214/19214), 27.91 MiB | 11.10 MiB/s, done.\n",
            "Resolving deltas: 100% (13655/13655), done.\n",
            "Note: switching to 'd1af7a41ade7bd3c3a463bfa640725edb818ebaf'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by switching back to a branch.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -c with the switch command. Example:\n",
            "\n",
            "  git switch -c <new-branch-name>\n",
            "\n",
            "Or undo this operation with:\n",
            "\n",
            "  git switch -\n",
            "\n",
            "Turn off this advice by setting config variable advice.detachedHead to false\n",
            "\n",
            "/content/text-generation-webui\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python==0.3.1+cpuavx2 (from -r requirements.txt (line 38))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.3.1+cpuavx2-cp310-cp310-linux_x86_64.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m78.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda==0.3.1+cu121 (from -r requirements.txt (line 46))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.3.1+cu121-cp310-cp310-linux_x86_64.whl (449.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m449.5/449.5 MB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda-tensorcores==0.3.1+cu121 (from -r requirements.txt (line 52))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda_tensorcores-0.3.1+cu121-cp310-cp310-linux_x86_64.whl (486.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m486.2/486.2 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllamav2==0.2.3+cu121.torch2.4.1 (from -r requirements.txt (line 58))\n",
            "  Downloading https://github.com/oobabooga/exllamav2/releases/download/v0.2.3/exllamav2-0.2.3+cu121.torch2.4.1-cp310-cp310-linux_x86_64.whl (137.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m137.1/137.1 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine != \"x86_64\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting flash-attn==2.6.3+cu123torch2.4cxx11abiFALSE (from -r requirements.txt (line 63))\n",
            "  Downloading https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu123torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl (187.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m187.3/187.3 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate==0.33.* (from -r requirements.txt (line 1))\n",
            "  Downloading accelerate-0.33.0-py3-none-any.whl.metadata (18 kB)\n",
            "Collecting bitsandbytes==0.44.* (from -r requirements.txt (line 2))\n",
            "  Downloading bitsandbytes-0.44.1-py3-none-manylinux_2_24_x86_64.whl.metadata (3.5 kB)\n",
            "Collecting colorama (from -r requirements.txt (line 3))\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)\n",
            "Collecting datasets (from -r requirements.txt (line 4))\n",
            "  Downloading datasets-3.0.1-py3-none-any.whl.metadata (20 kB)\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (0.8.0)\n",
            "Collecting fastapi==0.112.4 (from -r requirements.txt (line 6))\n",
            "  Downloading fastapi-0.112.4-py3-none-any.whl.metadata (27 kB)\n",
            "Collecting gradio==4.26.* (from -r requirements.txt (line 7))\n",
            "  Downloading gradio-4.26.0-py3-none-any.whl.metadata (15 kB)\n",
            "Requirement already satisfied: jinja2==3.1.4 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (3.1.4)\n",
            "Collecting lm_eval==0.3.0 (from -r requirements.txt (line 9))\n",
            "  Downloading lm_eval-0.3.0-py3-none-any.whl.metadata (6.8 kB)\n",
            "Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (3.7)\n",
            "Collecting numba==0.59.* (from -r requirements.txt (line 11))\n",
            "  Downloading numba-0.59.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)\n",
            "Requirement already satisfied: numpy==1.26.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (1.26.4)\n",
            "Collecting optimum==1.17.* (from -r requirements.txt (line 13))\n",
            "  Downloading optimum-1.17.1-py3-none-any.whl.metadata (18 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (2.2.2)\n",
            "Collecting peft==0.12.* (from -r requirements.txt (line 15))\n",
            "  Downloading peft-0.12.0-py3-none-any.whl.metadata (13 kB)\n",
            "Requirement already satisfied: Pillow>=9.5.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (10.4.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (5.9.5)\n",
            "Collecting pydantic==2.8.2 (from -r requirements.txt (line 18))\n",
            "  Downloading pydantic-2.8.2-py3-none-any.whl.metadata (125 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m125.2/125.2 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (6.0.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (2.32.3)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 21)) (13.9.2)\n",
            "Requirement already satisfied: safetensors==0.4.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 22)) (0.4.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (1.13.1)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 24)) (0.2.0)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 25)) (2.17.0)\n",
            "Collecting transformers==4.45.* (from -r requirements.txt (line 26))\n",
            "  Downloading transformers-4.45.2-py3-none-any.whl.metadata (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.4/44.4 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 27)) (4.66.5)\n",
            "Requirement already satisfied: wandb in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 28)) (0.18.3)\n",
            "Collecting SpeechRecognition==3.10.0 (from -r requirements.txt (line 31))\n",
            "  Downloading SpeechRecognition-3.10.0-py2.py3-none-any.whl.metadata (28 kB)\n",
            "Collecting flask_cloudflared==0.0.14 (from -r requirements.txt (line 32))\n",
            "  Downloading flask_cloudflared-0.0.14-py3-none-any.whl.metadata (4.6 kB)\n",
            "Collecting sse-starlette==1.6.5 (from -r requirements.txt (line 33))\n",
            "  Downloading sse_starlette-1.6.5-py3-none-any.whl.metadata (6.7 kB)\n",
            "Collecting tiktoken (from -r requirements.txt (line 34))\n",
            "  Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.33.*->-r requirements.txt (line 1)) (24.1)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.33.*->-r requirements.txt (line 1)) (2.2.2+cu121)\n",
            "Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.33.*->-r requirements.txt (line 1)) (0.24.7)\n",
            "Collecting starlette<0.39.0,>=0.37.2 (from fastapi==0.112.4->-r requirements.txt (line 6))\n",
            "  Downloading starlette-0.38.6-py3-none-any.whl.metadata (6.0 kB)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from fastapi==0.112.4->-r requirements.txt (line 6)) (4.12.2)\n",
            "Collecting aiofiles<24.0,>=22.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl.metadata (9.7 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (4.2.2)\n",
            "Collecting ffmpy (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading ffmpy-0.4.0-py3-none-any.whl.metadata (2.9 kB)\n",
            "Collecting gradio-client==0.15.1 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading gradio_client-0.15.1-py3-none-any.whl.metadata (7.1 kB)\n",
            "Collecting httpx>=0.24.1 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading httpx-0.27.2-py3-none-any.whl.metadata (7.1 kB)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (6.4.5)\n",
            "Collecting markupsafe~=2.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (3.7.1)\n",
            "Collecting orjson~=3.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading orjson-3.10.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.6/50.6 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydub (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl.metadata (1.4 kB)\n",
            "Collecting python-multipart>=0.0.9 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading python_multipart-0.0.12-py3-none-any.whl.metadata (1.9 kB)\n",
            "Collecting ruff>=0.2.2 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading ruff-0.7.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (25 kB)\n",
            "Collecting semantic-version~=2.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl.metadata (9.7 kB)\n",
            "Collecting tomlkit==0.12.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading tomlkit-0.12.0-py3-none-any.whl.metadata (2.7 kB)\n",
            "Requirement already satisfied: typer<1.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 7)) (0.12.5)\n",
            "Collecting uvicorn>=0.14.0 (from gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading uvicorn-0.32.0-py3-none-any.whl.metadata (6.6 kB)\n",
            "Collecting jsonlines (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading jsonlines-4.0.0-py3-none-any.whl.metadata (1.6 kB)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 9)) (2.10.1)\n",
            "Collecting openai>=0.6.4 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading openai-1.52.0-py3-none-any.whl.metadata (24 kB)\n",
            "Collecting pybind11>=2.6.2 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pybind11-2.13.6-py3-none-any.whl.metadata (9.5 kB)\n",
            "Collecting pycountry (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting pytablewriter (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pytablewriter-1.2.0-py3-none-any.whl.metadata (37 kB)\n",
            "Collecting rouge-score>=0.0.4 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sacrebleu==1.5.0 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading sacrebleu-1.5.0-py3-none-any.whl.metadata (1.3 kB)\n",
            "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 9)) (1.5.2)\n",
            "Collecting sqlitedict (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tqdm-multiprocess (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl.metadata (5.7 kB)\n",
            "Collecting zstandard (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)\n",
            "Collecting llvmlite<0.43,>=0.42.0dev0 (from numba==0.59.*->-r requirements.txt (line 11))\n",
            "  Downloading llvmlite-0.42.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)\n",
            "Collecting coloredlogs (from optimum==1.17.*->-r requirements.txt (line 13))\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from optimum==1.17.*->-r requirements.txt (line 13)) (1.13.3)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic==2.8.2->-r requirements.txt (line 18)) (0.7.0)\n",
            "Collecting pydantic-core==2.20.1 (from pydantic==2.8.2->-r requirements.txt (line 18))\n",
            "  Downloading pydantic_core-2.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.45.*->-r requirements.txt (line 26)) (3.16.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.45.*->-r requirements.txt (line 26)) (2024.9.11)\n",
            "Collecting tokenizers<0.21,>=0.20 (from transformers==4.45.*->-r requirements.txt (line 26))\n",
            "  Downloading tokenizers-0.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
            "Requirement already satisfied: Flask>=0.8 in /usr/local/lib/python3.10/dist-packages (from flask_cloudflared==0.0.14->-r requirements.txt (line 32)) (2.2.5)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 7)) (2024.6.1)\n",
            "Collecting websockets<12.0,>=10.0 (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Collecting portalocker (from sacrebleu==1.5.0->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading portalocker-2.10.1-py3-none-any.whl.metadata (8.5 kB)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (16.1.0)\n",
            "Collecting dill<0.3.9,>=0.3.0 (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting xxhash (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading multiprocess-0.70.17-py310-none-any.whl.metadata (7.2 kB)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (3.10.10)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 14)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 14)) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 14)) (2024.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 20)) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 20)) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 20)) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 20)) (2024.8.30)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 21)) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 21)) (2.18.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (1.64.1)\n",
            "Requirement already satisfied: protobuf!=4.24.0,<5.0.0,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 25)) (3.0.4)\n",
            "Requirement already satisfied: click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (8.1.7)\n",
            "Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (0.4.0)\n",
            "Requirement already satisfied: gitpython!=3.1.29,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (3.1.43)\n",
            "Requirement already satisfied: platformdirs in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (4.3.6)\n",
            "Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (2.16.0)\n",
            "Requirement already satisfied: setproctitle in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 28)) (1.3.3)\n",
            "Collecting diskcache>=5.6.1 (from llama-cpp-python==0.3.1+cpuavx2->-r requirements.txt (line 38))\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n",
            "Collecting ninja (from exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 58))\n",
            "  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)\n",
            "Collecting fastparquet (from exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 58))\n",
            "  Downloading fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (4.23.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.12.1)\n",
            "Requirement already satisfied: itsdangerous>=2.0 in /usr/local/lib/python3.10/dist-packages (from Flask>=0.8->flask_cloudflared==0.0.14->-r requirements.txt (line 32)) (2.2.0)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (2.4.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (24.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (6.1.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.15.2)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (4.0.3)\n",
            "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 28)) (4.0.11)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (3.7.1)\n",
            "Collecting httpcore==1.* (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading httpcore-1.0.6-py3-none-any.whl.metadata (21 kB)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (1.3.1)\n",
            "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7))\n",
            "  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 21)) (0.1.2)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (1.3.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (4.54.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (1.4.7)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (3.2.0)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.7.0)\n",
            "Collecting jiter<1,>=0.4.0 (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading jiter-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (3.8.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 9)) (3.5.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (3.4.1)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate==0.33.*->-r requirements.txt (line 1)) (12.6.77)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 7)) (1.5.4)\n",
            "\u001b[33mWARNING: typer 0.12.5 does not provide the extra 'all'\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting humanfriendly>=9.1 (from coloredlogs->optimum==1.17.*->-r requirements.txt (line 13))\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)\n",
            "Collecting cramjam>=2.3 (from fastparquet->exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 58))\n",
            "  Downloading cramjam-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 4))\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",
            "Collecting DataProperty<2,>=1.0.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading DataProperty-1.0.1-py3-none-any.whl.metadata (11 kB)\n",
            "Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading mbstrdecoder-1.1.3-py3-none-any.whl.metadata (4.0 kB)\n",
            "Collecting pathvalidate<4,>=2.3.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pathvalidate-3.2.1-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting tabledata<2,>=1.3.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tabledata-1.3.3-py3-none-any.whl.metadata (3.7 kB)\n",
            "Collecting tcolorpy<1,>=0.0.5 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tcolorpy-0.1.6-py3-none-any.whl.metadata (6.4 kB)\n",
            "Collecting typepy<2,>=1.3.2 (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading typepy-1.3.2-py3-none-any.whl.metadata (9.3 kB)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->optimum==1.17.*->-r requirements.txt (line 13)) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (1.2.2)\n",
            "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from gitdb<5,>=4.0.1->gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 28)) (5.0.1)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (2024.10.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.20.0)\n",
            "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9)) (5.2.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp->datasets->-r requirements.txt (line 4)) (0.2.0)\n",
            "Downloading accelerate-0.33.0-py3-none-any.whl (315 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m315.1/315.1 kB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bitsandbytes-0.44.1-py3-none-manylinux_2_24_x86_64.whl (122.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m122.4/122.4 MB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fastapi-0.112.4-py3-none-any.whl (93 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.9/93.9 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gradio-4.26.0-py3-none-any.whl (17.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.1/17.1 MB\u001b[0m \u001b[31m92.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lm_eval-0.3.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.7/178.7 kB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading numba-0.59.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.7/3.7 MB\u001b[0m \u001b[31m85.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading optimum-1.17.1-py3-none-any.whl (407 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m407.1/407.1 kB\u001b[0m \u001b[31m30.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading peft-0.12.0-py3-none-any.whl (296 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m296.4/296.4 kB\u001b[0m \u001b[31m26.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pydantic-2.8.2-py3-none-any.whl (423 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m423.9/423.9 kB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading transformers-4.45.2-py3-none-any.whl (9.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.9/9.9 MB\u001b[0m \u001b[31m115.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading SpeechRecognition-3.10.0-py2.py3-none-any.whl (32.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m32.8/32.8 MB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading flask_cloudflared-0.0.14-py3-none-any.whl (6.4 kB)\n",
            "Downloading sse_starlette-1.6.5-py3-none-any.whl (9.6 kB)\n",
            "Downloading gradio_client-0.15.1-py3-none-any.whl (313 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m313.6/313.6 kB\u001b[0m \u001b[31m26.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pydantic_core-2.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m82.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading sacrebleu-1.5.0-py3-none-any.whl (65 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.6/65.6 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tomlkit-0.12.0-py3-none-any.whl (37 kB)\n",
            "Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Downloading datasets-3.0.1-py3-none-any.whl (471 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m471.6/471.6 kB\u001b[0m \u001b[31m35.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m66.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Downloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httpx-0.27.2-py3-none-any.whl (76 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.4/76.4 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httpcore-1.0.6-py3-none-any.whl (78 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.0/78.0 kB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading llvmlite-0.42.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.8/43.8 MB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n",
            "Downloading openai-1.52.0-py3-none-any.whl (386 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m386.9/386.9 kB\u001b[0m \u001b[31m30.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading orjson-3.10.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m144.5/144.5 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pybind11-2.13.6-py3-none-any.whl (243 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.3/243.3 kB\u001b[0m \u001b[31m22.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_multipart-0.0.12-py3-none-any.whl (23 kB)\n",
            "Downloading ruff-0.7.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m120.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Downloading starlette-0.38.6-py3-none-any.whl (71 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.5/71.5 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tokenizers-0.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m80.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading uvicorn-0.32.0-py3-none-any.whl (63 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.7/63.7 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ffmpy-0.4.0-py3-none-any.whl (5.8 kB)\n",
            "Downloading jsonlines-4.0.0-py3-none-any.whl (8.7 kB)\n",
            "Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.2/307.2 kB\u001b[0m \u001b[31m25.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m108.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Downloading pytablewriter-1.2.0-py3-none-any.whl (111 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.1/111.1 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tqdm_multiprocess-0.0.11-py3-none-any.whl (9.8 kB)\n",
            "Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m103.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading cramjam-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m85.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading DataProperty-1.0.1-py3-none-any.whl (27 kB)\n",
            "Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading jiter-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (325 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m325.2/325.2 kB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading mbstrdecoder-1.1.3-py3-none-any.whl (7.8 kB)\n",
            "Downloading pathvalidate-3.2.1-py3-none-any.whl (23 kB)\n",
            "Downloading tabledata-1.3.3-py3-none-any.whl (11 kB)\n",
            "Downloading tcolorpy-0.1.6-py3-none-any.whl (8.1 kB)\n",
            "Downloading typepy-1.3.2-py3-none-any.whl (31 kB)\n",
            "Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)\n",
            "Building wheels for collected packages: rouge-score, sqlitedict\n",
            "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24935 sha256=ddcf9e6ed2c52613bf44622c6d8e1b8cf15d81bb38f1a4cc88c5e33049c28f25\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16864 sha256=a7f403cfb6832ff6ebccd91845b50baf0f00f2791a77773505eca8478d38767e\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
            "Successfully built rouge-score sqlitedict\n",
            "Installing collected packages: sqlitedict, pydub, ninja, zstandard, xxhash, websockets, tomlkit, tcolorpy, semantic-version, ruff, python-multipart, pydantic-core, pycountry, pybind11, portalocker, pathvalidate, orjson, mbstrdecoder, markupsafe, llvmlite, jsonlines, jiter, humanfriendly, h11, ffmpy, diskcache, dill, cramjam, colorama, aiofiles, uvicorn, typepy, tqdm-multiprocess, tiktoken, starlette, SpeechRecognition, sacrebleu, rouge-score, pydantic, numba, multiprocess, httpcore, coloredlogs, tokenizers, sse-starlette, llama-cpp-python-cuda-tensorcores, llama-cpp-python-cuda, llama-cpp-python, httpx, fastparquet, fastapi, transformers, openai, gradio-client, flask_cloudflared, flash-attn, exllamav2, DataProperty, bitsandbytes, accelerate, tabledata, peft, gradio, datasets, pytablewriter, optimum, lm_eval\n",
            "  Attempting uninstall: pydantic-core\n",
            "    Found existing installation: pydantic_core 2.23.4\n",
            "    Uninstalling pydantic_core-2.23.4:\n",
            "      Successfully uninstalled pydantic_core-2.23.4\n",
            "  Attempting uninstall: markupsafe\n",
            "    Found existing installation: MarkupSafe 3.0.1\n",
            "    Uninstalling MarkupSafe-3.0.1:\n",
            "      Successfully uninstalled MarkupSafe-3.0.1\n",
            "  Attempting uninstall: llvmlite\n",
            "    Found existing installation: llvmlite 0.43.0\n",
            "    Uninstalling llvmlite-0.43.0:\n",
            "      Successfully uninstalled llvmlite-0.43.0\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 2.9.2\n",
            "    Uninstalling pydantic-2.9.2:\n",
            "      Successfully uninstalled pydantic-2.9.2\n",
            "  Attempting uninstall: numba\n",
            "    Found existing installation: numba 0.60.0\n",
            "    Uninstalling numba-0.60.0:\n",
            "      Successfully uninstalled numba-0.60.0\n",
            "  Attempting uninstall: tokenizers\n",
            "    Found existing installation: tokenizers 0.19.1\n",
            "    Uninstalling tokenizers-0.19.1:\n",
            "      Successfully uninstalled tokenizers-0.19.1\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.44.2\n",
            "    Uninstalling transformers-4.44.2:\n",
            "      Successfully uninstalled transformers-4.44.2\n",
            "  Attempting uninstall: accelerate\n",
            "    Found existing installation: accelerate 0.34.2\n",
            "    Uninstalling accelerate-0.34.2:\n",
            "      Successfully uninstalled accelerate-0.34.2\n",
            "Successfully installed DataProperty-1.0.1 SpeechRecognition-3.10.0 accelerate-0.33.0 aiofiles-23.2.1 bitsandbytes-0.44.1 colorama-0.4.6 coloredlogs-15.0.1 cramjam-2.9.0 datasets-3.0.1 dill-0.3.8 diskcache-5.6.3 exllamav2-0.2.3+cu121.torch2.4.1 fastapi-0.112.4 fastparquet-2024.5.0 ffmpy-0.4.0 flash-attn-2.6.3 flask_cloudflared-0.0.14 gradio-4.26.0 gradio-client-0.15.1 h11-0.14.0 httpcore-1.0.6 httpx-0.27.2 humanfriendly-10.0 jiter-0.6.1 jsonlines-4.0.0 llama-cpp-python-0.3.1+cpuavx2 llama-cpp-python-cuda-0.3.1+cu121 llama-cpp-python-cuda-tensorcores-0.3.1+cu121 llvmlite-0.42.0 lm_eval-0.3.0 markupsafe-2.1.5 mbstrdecoder-1.1.3 multiprocess-0.70.16 ninja-1.11.1.1 numba-0.59.1 openai-1.52.0 optimum-1.17.1 orjson-3.10.9 pathvalidate-3.2.1 peft-0.12.0 portalocker-2.10.1 pybind11-2.13.6 pycountry-24.6.1 pydantic-2.8.2 pydantic-core-2.20.1 pydub-0.25.1 pytablewriter-1.2.0 python-multipart-0.0.12 rouge-score-0.1.2 ruff-0.7.0 sacrebleu-1.5.0 semantic-version-2.10.0 sqlitedict-2.1.0 sse-starlette-1.6.5 starlette-0.38.6 tabledata-1.3.3 tcolorpy-0.1.6 tiktoken-0.8.0 tokenizers-0.20.1 tomlkit-0.12.0 tqdm-multiprocess-0.0.11 transformers-4.45.2 typepy-1.3.2 uvicorn-0.32.0 websockets-11.0.3 xxhash-3.5.0 zstandard-0.23.0\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "98ae7f|\u001b[1;32mOK\u001b[0m  |   154MiB/s|/content/text-generation-webui/models//Ministral-8B-Instruct-2410-HF-Q8_0.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "\u001b[2;36m04:47:57-118758\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Starting Text generation web UI                                            \n",
            "\u001b[2;36m04:47:57-593216\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading \u001b[32m\"Ministral-8B-Instruct-2410-HF-Q8_0.gguf\"\u001b[0m                          \n",
            "\u001b[2;36m04:47:57-969629\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m llama.cpp weights detected:                                                \n",
            "\u001b[2;36m                \u001b[0m         \u001b[32m\"models/Ministral-8B-Instruct-2410-HF-Q8_0.gguf\"\u001b[0m                           \n",
            "llama_model_loader: loaded meta data with 36 key-value pairs and 327 tensors from models/Ministral-8B-Instruct-2410-HF-Q8_0.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.type str              = model\n",
            "llama_model_loader: - kv   2:                               general.name str              = Ministral 8B Instruct 2410\n",
            "llama_model_loader: - kv   3:                            general.version str              = 2410\n",
            "llama_model_loader: - kv   4:                           general.finetune str              = Instruct\n",
            "llama_model_loader: - kv   5:                           general.basename str              = Ministral\n",
            "llama_model_loader: - kv   6:                         general.size_label str              = 8B\n",
            "llama_model_loader: - kv   7:                          llama.block_count u32              = 36\n",
            "llama_model_loader: - kv   8:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   9:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 12288\n",
            "llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 100000000.000000\n",
            "llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 128\n",
            "llama_model_loader: - kv  16:               llama.attention.value_length u32              = 128\n",
            "llama_model_loader: - kv  17:                          general.file_type u32              = 7\n",
            "llama_model_loader: - kv  18:                           llama.vocab_size u32              = 131072\n",
            "llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv  20:            tokenizer.ggml.add_space_prefix bool             = false\n",
            "llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = gpt2\n",
            "llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = tekken\n",
            "llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,131072]  = [\"<unk>\", \"<s>\", \"</s>\", \"[INST]\", \"[...\n",
            "llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n",
            "Exception ignored on calling ctypes callback function: <function llama_log_callback at 0x7fa9dfcd05e0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/llama_cpp_cuda/_logger.py\", line 39, in llama_log_callback\n",
            "    print(text.decode(\"utf-8\"), end=\"\", flush=True, file=sys.stderr)\n",
            "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 128: invalid continuation byte\n",
            "llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  30:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  31:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - kv  32:                      quantize.imatrix.file str              = /models_out/Ministral-8B-Instruct-241...\n",
            "llama_model_loader: - kv  33:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt\n",
            "llama_model_loader: - kv  34:             quantize.imatrix.entries_count i32              = 252\n",
            "llama_model_loader: - kv  35:              quantize.imatrix.chunks_count i32              = 128\n",
            "llama_model_loader: - type  f32:   73 tensors\n",
            "llama_model_loader: - type q8_0:  254 tensors\n",
            "llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect\n",
            "llm_load_vocab: special tokens cache size = 1000\n",
            "llm_load_vocab: token to piece cache size = 0.8498 MB\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = BPE\n",
            "llm_load_print_meta: n_vocab          = 131072\n",
            "llm_load_print_meta: n_merges         = 269443\n",
            "llm_load_print_meta: vocab_only       = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_layer          = 36\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_swa            = 0\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 12288\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 100000000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_ctx_orig_yarn  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: ssm_dt_b_c_rms   = 0\n",
            "llm_load_print_meta: model type       = 8B\n",
            "llm_load_print_meta: model ftype      = Q8_0\n",
            "llm_load_print_meta: model params     = 8.02 B\n",
            "llm_load_print_meta: model size       = 7.94 GiB (8.50 BPW) \n",
            "llm_load_print_meta: general.name     = Ministral 8B Instruct 2410\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 2 '</s>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 1196 'Ä'\n",
            "llm_load_print_meta: EOG token        = 2 '</s>'\n",
            "llm_load_print_meta: max token length = 150\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\n",
            "ggml_cuda_init: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5, VMM: yes\n",
            "llm_load_tensors: ggml ctx size =    0.31 MiB\n",
            "llm_load_tensors: offloading 36 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 37/37 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =   544.00 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  7583.14 MiB\n",
            ".........................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 8224\n",
            "llama_new_context_with_model: n_batch    = 512\n",
            "llama_new_context_with_model: n_ubatch   = 512\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 100000000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  1156.50 MiB\n",
            "llama_new_context_with_model: KV self size  = 1156.50 MiB, K (f16):  578.25 MiB, V (f16):  578.25 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.50 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   562.07 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =    24.07 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1158\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
            "Model metadata: {'quantize.imatrix.dataset': '/training_dir/calibration_datav3.txt', 'quantize.imatrix.entries_count': '252', 'llama.attention.head_count_kv': '8', 'llama.feed_forward_length': '12288', 'llama.embedding_length': '4096', 'tokenizer.ggml.add_bos_token': 'true', 'general.size_label': '8B', 'general.type': 'model', 'general.version': '2410', 'quantize.imatrix.chunks_count': '128', 'llama.context_length': '32768', 'general.name': 'Ministral 8B Instruct 2410', 'tokenizer.ggml.bos_token_id': '1', 'general.basename': 'Ministral', 'general.architecture': 'llama', 'llama.rope.freq_base': '100000000.000000', 'llama.block_count': '36', 'llama.attention.head_count': '32', 'llama.attention.key_length': '128', 'llama.attention.value_length': '128', 'general.finetune': 'Instruct', 'general.file_type': '7', 'tokenizer.ggml.pre': 'tekken', 'llama.vocab_size': '131072', 'quantize.imatrix.file': '/models_out/Ministral-8B-Instruct-2410-HF-GGUF/Ministral-8B-Instruct-2410-HF.imatrix', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.add_space_prefix': 'false', 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.model': 'gpt2', 'general.quantization_version': '2', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'tokenizer.ggml.eos_token_id': '2', 'tokenizer.ggml.unknown_token_id': '0'}\n",
            "Using fallback chat format: llama-2\n",
            "\u001b[2;36m04:48:34-223825\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[32m\"Ministral-8B-Instruct-2410-HF-Q8_0.gguf\"\u001b[0m in \u001b[1;36m36.63\u001b[0m seconds.         \n",
            "\u001b[2;36m04:48:34-225839\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m LOADER: \u001b[32m\"llama.cpp\"\u001b[0m                                                        \n",
            "\u001b[2;36m04:48:34-226858\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m TRUNCATION LENGTH: \u001b[1;36m8196\u001b[0m                                                    \n",
            "\u001b[2;36m04:48:34-227807\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m INSTRUCTION TEMPLATE: \u001b[32m\"Alpaca\"\u001b[0m                                             \n",
            "\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "\n",
            "Running on public URL: https://75548c5a8f797ac941.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "llama_perf_context_print:        load time =     538.96 ms\n",
            "llama_perf_context_print: prompt eval time =       0.00 ms /    41 tokens (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:        eval time =       0.00 ms /    62 runs   (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:       total time =    3438.78 ms /   103 tokens\n",
            "Output generated in 3.92 seconds (15.82 tokens/s, 62 tokens, context 41, seed 1549028511)\n",
            "\u001b[2;36m04:55:40-336014\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Deleted \u001b[32m\"logs/instruct/20241021-04-54-54.json\"\u001b[0m.                            \n",
            "Llama.generate: 22 prefix-match hit, remaining 21 prompt tokens to eval\n",
            "llama_perf_context_print:        load time =     538.96 ms\n",
            "llama_perf_context_print: prompt eval time =       0.00 ms /    21 tokens (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:        eval time =       0.00 ms /   101 runs   (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:       total time =    4849.70 ms /   122 tokens\n",
            "Output generated in 5.32 seconds (18.99 tokens/s, 101 tokens, context 43, seed 1282938247)\n",
            "\u001b[2;36m04:56:03-344862\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Deleted \u001b[32m\"logs/instruct/20241021-04-55-40.json\"\u001b[0m.                            \n",
            "Llama.generate: 22 prefix-match hit, remaining 22 prompt tokens to eval\n",
            "llama_perf_context_print:        load time =     538.96 ms\n",
            "llama_perf_context_print: prompt eval time =       0.00 ms /    22 tokens (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:        eval time =       0.00 ms /   715 runs   (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:       total time =   37233.68 ms /   737 tokens\n",
            "Output generated in 37.70 seconds (18.97 tokens/s, 715 tokens, context 44, seed 1720010199)\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "!pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121\n",
        "\n",
        "!git clone -b V20241021 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-HF-GGUF-TEST/resolve/main/Ministral-8B-Instruct-2410-HF-Q8_0.gguf?download=true -d /content/text-generation-webui/models/ -o Ministral-8B-Instruct-2410-HF-Q8_0.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 100000 --model Ministral-8B-Instruct-2410-HF-Q8_0.gguf --n_ctx 8196"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyORAP4O/JL48ESNVg2A1Gda",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}