Gemma_The_Writer_N_Restless_Quill_10B_Uncensored_GGUF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-Comparison-Colab-UI/blob/main/Gemma_The_Writer_N_Restless_Quill_10B_Uncensored_GGUF.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## Gemma-The-Writer-N-Restless-Quill-10B-Uncensored-GGUF WebUI\n",
        "\n",
        "WARNING: Uncensored. Cursing, Swearing, Horror, Graphic Details\n",
        "This is a text-generation model for uncensored storytelling.\n",
        "\n",
        "1. Run the following cell, takes ~5 min\n",
        "(You may need to confirm to proceed by typing \"Y\")\n",
        "2. Click the gradio link at the bottom\n",
        "3. Example prompt, use Notebook mode:\n",
        "\n",
        "```\n",
        "Science Fiction: The Last Transmission - Write a story that takes place entirely within a spaceship's cockpit as the sole surviving crew member attempts to send a final message back to Earth before the ship's power runs out. The story should explore themes of isolation, sacrifice, and the importance of human connection in the face of adversity. 800-1000 words.\n",
        "```\n",
        "\n",
        "Quantized model: https://huggingface.co/DavidAU/Gemma-The-Writer-N-Restless-Quill-10B-Uncensored-GGUF\n",
        "\n",
        "Want to try other local LLMs? Check out this repo: https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D-MiHp_bveP6",
        "outputId": "03337bbe-c35e-4599-ecae-419c7a3eddce"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Looking in indexes: https://download.pytorch.org/whl/cu121\n",
            "Requirement already satisfied: torch==2.2.2 in /usr/local/lib/python3.10/dist-packages (2.2.2+cu121)\n",
            "Requirement already satisfied: torchvision==0.17.2 in /usr/local/lib/python3.10/dist-packages (0.17.2+cu121)\n",
            "Requirement already satisfied: torchaudio==2.2.2 in /usr/local/lib/python3.10/dist-packages (2.2.2+cu121)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (2024.9.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.2) (2.2.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision==0.17.2) (1.26.4)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision==0.17.2) (10.4.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.2) (12.6.77)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.2.2) (2.1.5)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.2.2) (1.3.0)\n",
            "fatal: destination path 'text-generation-webui' already exists and is not an empty directory.\n",
            "/content/text-generation-webui\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python==0.3.1+cpuavx2 (from -r requirements.txt (line 36))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.3.1+cpuavx2-cp310-cp310-linux_x86_64.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda==0.3.1+cu121 (from -r requirements.txt (line 44))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.3.1+cu121-cp310-cp310-linux_x86_64.whl (449.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m449.5/449.5 MB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda-tensorcores==0.3.1+cu121 (from -r requirements.txt (line 50))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda_tensorcores-0.3.1+cu121-cp310-cp310-linux_x86_64.whl (486.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m486.2/486.2 MB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllamav2==0.2.3+cu121.torch2.4.1 (from -r requirements.txt (line 56))\n",
            "  Downloading https://github.com/oobabooga/exllamav2/releases/download/v0.2.3/exllamav2-0.2.3+cu121.torch2.4.1-cp310-cp310-linux_x86_64.whl (137.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m137.1/137.1 MB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine != \"x86_64\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting flash-attn==2.6.3+cu123torch2.4cxx11abiFALSE (from -r requirements.txt (line 61))\n",
            "  Downloading https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu123torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl (187.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m187.3/187.3 MB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: accelerate==1.0.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 1)) (1.0.1)\n",
            "Requirement already satisfied: bitsandbytes==0.44.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 2)) (0.44.1)\n",
            "Requirement already satisfied: colorama in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 3)) (0.4.6)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (3.1.0)\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (0.8.0)\n",
            "Requirement already satisfied: fastapi==0.112.4 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 6)) (0.112.4)\n",
            "Requirement already satisfied: gradio==4.26.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 7)) (4.26.0)\n",
            "Requirement already satisfied: jinja2==3.1.4 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (3.1.4)\n",
            "Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (3.7)\n",
            "Requirement already satisfied: numba==0.59.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (0.59.1)\n",
            "Requirement already satisfied: numpy==1.26.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (1.26.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (2.2.2)\n",
            "Requirement already satisfied: peft==0.12.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (0.12.0)\n",
            "Requirement already satisfied: Pillow>=9.5.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (10.4.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (5.9.5)\n",
            "Requirement already satisfied: pydantic==2.8.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (2.8.2)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (6.0.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 18)) (2.32.3)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (13.9.4)\n",
            "Requirement already satisfied: safetensors==0.4.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (0.4.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 21)) (1.13.1)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 22)) (0.2.0)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (2.17.1)\n",
            "Requirement already satisfied: transformers==4.46.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 24)) (4.46.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 25)) (4.66.6)\n",
            "Requirement already satisfied: wandb in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 26)) (0.18.7)\n",
            "Requirement already satisfied: SpeechRecognition==3.10.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 29)) (3.10.0)\n",
            "Requirement already satisfied: flask_cloudflared==0.0.14 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 30)) (0.0.14)\n",
            "Requirement already satisfied: sse-starlette==1.6.5 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 31)) (1.6.5)\n",
            "Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 32)) (0.8.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==1.0.*->-r requirements.txt (line 1)) (24.2)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==1.0.*->-r requirements.txt (line 1)) (2.2.2+cu121)\n",
            "Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==1.0.*->-r requirements.txt (line 1)) (0.26.2)\n",
            "Requirement already satisfied: starlette<0.39.0,>=0.37.2 in /usr/local/lib/python3.10/dist-packages (from fastapi==0.112.4->-r requirements.txt (line 6)) (0.38.6)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from fastapi==0.112.4->-r requirements.txt (line 6)) (4.12.2)\n",
            "Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (23.2.1)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (4.2.2)\n",
            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.4.0)\n",
            "Requirement already satisfied: gradio-client==0.15.1 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.15.1)\n",
            "Requirement already satisfied: httpx>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.27.2)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (6.4.5)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (2.1.5)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (3.8.0)\n",
            "Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (3.10.11)\n",
            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.25.1)\n",
            "Requirement already satisfied: python-multipart>=0.0.9 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.0.19)\n",
            "Requirement already satisfied: ruff>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.8.1)\n",
            "Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (2.10.0)\n",
            "Requirement already satisfied: tomlkit==0.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.12.0)\n",
            "Requirement already satisfied: typer<1.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 7)) (0.13.0)\n",
            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 7)) (0.32.1)\n",
            "Requirement already satisfied: llvmlite<0.43,>=0.42.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba==0.59.*->-r requirements.txt (line 10)) (0.42.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic==2.8.2->-r requirements.txt (line 16)) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic==2.8.2->-r requirements.txt (line 16)) (2.20.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.46.*->-r requirements.txt (line 24)) (3.16.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.46.*->-r requirements.txt (line 24)) (2024.9.11)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers==4.46.*->-r requirements.txt (line 24)) (0.20.3)\n",
            "Requirement already satisfied: Flask>=0.8 in /usr/local/lib/python3.10/dist-packages (from flask_cloudflared==0.0.14->-r requirements.txt (line 30)) (3.0.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 7)) (2024.9.0)\n",
            "Requirement already satisfied: websockets<12.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 7)) (11.0.3)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (17.0.0)\n",
            "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (0.3.8)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (3.5.0)\n",
            "Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (0.70.16)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 4)) (3.11.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 12)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 12)) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 12)) (2024.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 18)) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 18)) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 18)) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 18)) (2024.8.30)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 19)) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 19)) (2.18.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (1.68.0)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (4.25.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 23)) (3.1.3)\n",
            "Requirement already satisfied: click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (8.1.7)\n",
            "Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (0.4.0)\n",
            "Requirement already satisfied: gitpython!=3.1.29,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (3.1.43)\n",
            "Requirement already satisfied: platformdirs in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (4.3.6)\n",
            "Requirement already satisfied: sentry-sdk>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (2.18.0)\n",
            "Requirement already satisfied: setproctitle in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 26)) (1.3.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.3.1+cpuavx2->-r requirements.txt (line 36)) (5.6.3)\n",
            "Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 56)) (1.11.1.2)\n",
            "Requirement already satisfied: fastparquet in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 56)) (2024.11.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (4.23.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.12.1)\n",
            "Requirement already satisfied: itsdangerous>=2.1.2 in /usr/local/lib/python3.10/dist-packages (from Flask>=0.8->flask_cloudflared==0.0.14->-r requirements.txt (line 30)) (2.2.0)\n",
            "Requirement already satisfied: blinker>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from Flask>=0.8->flask_cloudflared==0.0.14->-r requirements.txt (line 30)) (1.9.0)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (2.4.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (24.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (6.1.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (0.2.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (1.17.2)\n",
            "Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 4)) (4.0.3)\n",
            "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 26)) (4.0.11)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (3.7.1)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (1.0.7)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (1.3.1)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (0.14.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 19)) (0.1.2)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (4.55.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (1.4.7)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 7)) (3.2.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (3.4.2)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (12.6.77)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 7)) (1.5.4)\n",
            "\u001b[33mWARNING: typer 0.13.0 does not provide the extra 'all'\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: cramjam>=2.3 in /usr/local/lib/python3.10/dist-packages (from fastparquet->exllamav2==0.2.3+cu121.torch2.4.1->-r requirements.txt (line 56)) (2.9.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 7)) (1.2.2)\n",
            "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from gitdb<5,>=4.0.1->gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 26)) (5.0.1)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (2024.10.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 7)) (0.21.0)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate==1.0.*->-r requirements.txt (line 1)) (1.3.0)\n",
            "\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "29bd80|\u001b[1;32mOK\u001b[0m  |       0B/s|/content/text-generation-webui/models//Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "\u001b[2;36m09:41:43-870648\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Starting Text generation web UI                                            \n",
            "\u001b[2;36m09:41:44-384640\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading \u001b[32m\"Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m           \n",
            "\u001b[2;36m09:41:44-903132\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m llama.cpp weights detected:                                                \n",
            "\u001b[2;36m                \u001b[0m         \u001b[32m\"models/Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m            \n",
            "llama_model_loader: loaded meta data with 35 key-value pairs and 508 tensors from models/Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = gemma2\n",
            "llama_model_loader: - kv   1:                               general.type str              = model\n",
            "llama_model_loader: - kv   2:                               general.name str              = Gemma The Writer Exp4 3 Bf16 X4\n",
            "llama_model_loader: - kv   3:                           general.finetune str              = x4\n",
            "llama_model_loader: - kv   4:                           general.basename str              = Gemma-The-Writer-exp4-3\n",
            "llama_model_loader: - kv   5:                         general.size_label str              = 10B\n",
            "llama_model_loader: - kv   6:                   general.base_model.count u32              = 0\n",
            "llama_model_loader: - kv   7:                               general.tags arr[str,2]       = [\"mergekit\", \"merge\"]\n",
            "llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192\n",
            "llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 3584\n",
            "llama_model_loader: - kv  10:                         gemma2.block_count u32              = 46\n",
            "llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 16\n",
            "llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001\n",
            "llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256\n",
            "llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256\n",
            "llama_model_loader: - kv  17:                          general.file_type u32              = 14\n",
            "llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000\n",
            "llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000\n",
            "llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096\n",
            "llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default\n",
            "llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = [\"<pad>\", \"<eos>\", \"<bos>\", \"<unk>\", ...\n",
            "llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...\n",
            "llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n",
            "llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2\n",
            "llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1\n",
            "llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3\n",
            "llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...\n",
            "llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false\n",
            "llama_model_loader: - kv  34:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:  185 tensors\n",
            "llama_model_loader: - type q4_K:  313 tensors\n",
            "llama_model_loader: - type q5_K:    9 tensors\n",
            "llama_model_loader: - type q6_K:    1 tensors\n",
            "llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect\n",
            "llm_load_vocab: special tokens cache size = 217\n",
            "llm_load_vocab: token to piece cache size = 1.6014 MB\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = gemma2\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 256000\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: vocab_only       = 0\n",
            "llm_load_print_meta: n_ctx_train      = 8192\n",
            "llm_load_print_meta: n_embd           = 3584\n",
            "llm_load_print_meta: n_layer          = 46\n",
            "llm_load_print_meta: n_head           = 16\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_rot            = 256\n",
            "llm_load_print_meta: n_swa            = 4096\n",
            "llm_load_print_meta: n_embd_head_k    = 256\n",
            "llm_load_print_meta: n_embd_head_v    = 256\n",
            "llm_load_print_meta: n_gqa            = 2\n",
            "llm_load_print_meta: n_embd_k_gqa     = 2048\n",
            "llm_load_print_meta: n_embd_v_gqa     = 2048\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-06\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 2\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 10000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_ctx_orig_yarn  = 8192\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: ssm_dt_b_c_rms   = 0\n",
            "llm_load_print_meta: model type       = 27B\n",
            "llm_load_print_meta: model ftype      = Q4_K - Small\n",
            "llm_load_print_meta: model params     = 10.03 B\n",
            "llm_load_print_meta: model size       = 5.51 GiB (4.72 BPW) \n",
            "llm_load_print_meta: general.name     = Gemma The Writer Exp4 3 Bf16 X4\n",
            "llm_load_print_meta: BOS token        = 2 '<bos>'\n",
            "llm_load_print_meta: EOS token        = 1 '<eos>'\n",
            "llm_load_print_meta: UNK token        = 3 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<pad>'\n",
            "llm_load_print_meta: LF token         = 227 '<0x0A>'\n",
            "llm_load_print_meta: EOT token        = 107 '<end_of_turn>'\n",
            "llm_load_print_meta: EOG token        = 1 '<eos>'\n",
            "llm_load_print_meta: EOG token        = 107 '<end_of_turn>'\n",
            "llm_load_print_meta: max token length = 48\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\n",
            "ggml_cuda_init: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5, VMM: yes\n",
            "llm_load_tensors: ggml ctx size =    0.45 MiB\n",
            "llm_load_tensors: offloading 46 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 47/47 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =   717.77 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  5644.80 MiB\n",
            "................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 1024\n",
            "llama_new_context_with_model: n_batch    = 512\n",
            "llama_new_context_with_model: n_ubatch   = 512\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 10000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =   368.00 MiB\n",
            "llama_new_context_with_model: KV self size  =  368.00 MiB, K (f16):  184.00 MiB, V (f16):  184.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.98 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   507.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =    11.01 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1850\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
            "Model metadata: {'general.quantization_version': '2', 'tokenizer.ggml.add_space_prefix': 'false', 'gemma2.attention.head_count': '16', 'gemma2.feed_forward_length': '14336', 'gemma2.block_count': '46', 'tokenizer.ggml.pre': 'default', 'general.base_model.count': '0', 'gemma2.attn_logit_softcapping': '50.000000', 'general.type': 'model', 'gemma2.embedding_length': '3584', 'general.basename': 'Gemma-The-Writer-exp4-3', 'tokenizer.ggml.padding_token_id': '0', 'gemma2.context_length': '8192', 'tokenizer.chat_template': \"{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\\n' + message['content'] | trim + '<end_of_turn>\\n' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\\n'}}{% endif %}\", 'general.architecture': 'gemma2', 'gemma2.attention.key_length': '256', 'gemma2.attention.value_length': '256', 'gemma2.attention.layer_norm_rms_epsilon': '0.000001', 'general.finetune': 'x4', 'general.file_type': '14', 'gemma2.attention.sliding_window': '4096', 'gemma2.final_logit_softcapping': '30.000000', 'gemma2.attention.head_count_kv': '8', 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.model': 'llama', 'general.name': 'Gemma The Writer Exp4 3 Bf16 X4', 'tokenizer.ggml.bos_token_id': '2', 'tokenizer.ggml.eos_token_id': '1', 'tokenizer.ggml.unknown_token_id': '3', 'general.size_label': '10B', 'tokenizer.ggml.add_bos_token': 'true'}\n",
            "Available chat formats from metadata: chat_template.default\n",
            "Using gguf chat template: {{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\n",
            "' + message['content'] | trim + '<end_of_turn>\n",
            "' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\n",
            "'}}{% endif %}\n",
            "Using chat eos_token: <eos>\n",
            "Using chat bos_token: <bos>\n",
            "\u001b[2;36m09:42:15-025470\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[32m\"Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m in \u001b[1;36m30.64\u001b[0m   \n",
            "\u001b[2;36m                \u001b[0m         seconds.                                                                   \n",
            "\u001b[2;36m09:42:15-026980\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m LOADER: \u001b[32m\"llama.cpp\"\u001b[0m                                                        \n",
            "\u001b[2;36m09:42:15-028023\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m TRUNCATION LENGTH: \u001b[1;36m1024\u001b[0m                                                    \n",
            "\u001b[2;36m09:42:15-029039\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m INSTRUCTION TEMPLATE: \u001b[32m\"Custom \u001b[0m\u001b[32m(\u001b[0m\u001b[32mobtained from model metadata\u001b[0m\u001b[32m)\u001b[0m\u001b[32m\"\u001b[0m              \n",
            "\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "\n",
            "Running on public URL: https://97dad45a290cf35ccf.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "\u001b[2;36m09:44:04-869807\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;31mERROR   \u001b[0m Failed to build the chat prompt. The input is too long for the available   \n",
            "\u001b[2;36m                \u001b[0m         context length.                                                            \n",
            "\u001b[2;36m                \u001b[0m                                                                                    \n",
            "\u001b[2;36m                \u001b[0m         Truncation length: \u001b[1;36m1024\u001b[0m                                                    \n",
            "\u001b[2;36m                \u001b[0m         max_new_tokens: \u001b[1;36m2048\u001b[0m \u001b[1m(\u001b[0mis it too high?\u001b[1m)\u001b[0m                                     \n",
            "\u001b[2;36m                \u001b[0m         Available context length: \u001b[1;36m-1024\u001b[0m                                            \n",
            "\u001b[2;36m                \u001b[0m                                                                                    \n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/queueing.py\", line 566, in process_events\n",
            "    response = await route_utils.call_process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py\", line 261, in call_process_api\n",
            "    output = await app.get_blocks().process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1786, in process_api\n",
            "    result = await self.call_function(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1350, in call_function\n",
            "    prediction = await utils.async_iteration(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 583, in async_iteration\n",
            "    return await iterator.__anext__()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 576, in __anext__\n",
            "    return await anyio.to_thread.run_sync(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py\", line 33, in run_sync\n",
            "    return await get_asynclib().run_sync_in_worker_thread(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 877, in run_sync_in_worker_thread\n",
            "    return await future\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 807, in run\n",
            "    result = context.run(func, *args)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 559, in run_sync_iterator_async\n",
            "    return next(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 742, in gen_wrapper\n",
            "    response = next(iterator)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 436, in generate_chat_reply_wrapper\n",
            "    for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True, for_ui=True)):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 403, in generate_chat_reply\n",
            "    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message, for_ui=for_ui):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 348, in chatbot_wrapper\n",
            "    prompt = generate_chat_prompt(text, state, **kwargs)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 236, in generate_chat_prompt\n",
            "    raise ValueError\n",
            "ValueError\n",
            "\u001b[2;36m09:44:54-291695\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading \u001b[32m\"Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m           \n",
            "\u001b[2;36m09:44:54-810881\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m llama.cpp weights detected:                                                \n",
            "\u001b[2;36m                \u001b[0m         \u001b[32m\"models/Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m            \n",
            "llama_model_loader: loaded meta data with 35 key-value pairs and 508 tensors from models/Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = gemma2\n",
            "llama_model_loader: - kv   1:                               general.type str              = model\n",
            "llama_model_loader: - kv   2:                               general.name str              = Gemma The Writer Exp4 3 Bf16 X4\n",
            "llama_model_loader: - kv   3:                           general.finetune str              = x4\n",
            "llama_model_loader: - kv   4:                           general.basename str              = Gemma-The-Writer-exp4-3\n",
            "llama_model_loader: - kv   5:                         general.size_label str              = 10B\n",
            "llama_model_loader: - kv   6:                   general.base_model.count u32              = 0\n",
            "llama_model_loader: - kv   7:                               general.tags arr[str,2]       = [\"mergekit\", \"merge\"]\n",
            "llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192\n",
            "llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 3584\n",
            "llama_model_loader: - kv  10:                         gemma2.block_count u32              = 46\n",
            "llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 16\n",
            "llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001\n",
            "llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256\n",
            "llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256\n",
            "llama_model_loader: - kv  17:                          general.file_type u32              = 14\n",
            "llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000\n",
            "llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000\n",
            "llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096\n",
            "llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default\n",
            "llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = [\"<pad>\", \"<eos>\", \"<bos>\", \"<unk>\", ...\n",
            "llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...\n",
            "llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n",
            "llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2\n",
            "llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1\n",
            "llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3\n",
            "llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...\n",
            "llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false\n",
            "llama_model_loader: - kv  34:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:  185 tensors\n",
            "llama_model_loader: - type q4_K:  313 tensors\n",
            "llama_model_loader: - type q5_K:    9 tensors\n",
            "llama_model_loader: - type q6_K:    1 tensors\n",
            "llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect\n",
            "llm_load_vocab: special tokens cache size = 217\n",
            "llm_load_vocab: token to piece cache size = 1.6014 MB\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = gemma2\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 256000\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: vocab_only       = 0\n",
            "llm_load_print_meta: n_ctx_train      = 8192\n",
            "llm_load_print_meta: n_embd           = 3584\n",
            "llm_load_print_meta: n_layer          = 46\n",
            "llm_load_print_meta: n_head           = 16\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_rot            = 256\n",
            "llm_load_print_meta: n_swa            = 4096\n",
            "llm_load_print_meta: n_embd_head_k    = 256\n",
            "llm_load_print_meta: n_embd_head_v    = 256\n",
            "llm_load_print_meta: n_gqa            = 2\n",
            "llm_load_print_meta: n_embd_k_gqa     = 2048\n",
            "llm_load_print_meta: n_embd_v_gqa     = 2048\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-06\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 2\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 10000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_ctx_orig_yarn  = 8192\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: ssm_dt_b_c_rms   = 0\n",
            "llm_load_print_meta: model type       = 27B\n",
            "llm_load_print_meta: model ftype      = Q4_K - Small\n",
            "llm_load_print_meta: model params     = 10.03 B\n",
            "llm_load_print_meta: model size       = 5.51 GiB (4.72 BPW) \n",
            "llm_load_print_meta: general.name     = Gemma The Writer Exp4 3 Bf16 X4\n",
            "llm_load_print_meta: BOS token        = 2 '<bos>'\n",
            "llm_load_print_meta: EOS token        = 1 '<eos>'\n",
            "llm_load_print_meta: UNK token        = 3 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<pad>'\n",
            "llm_load_print_meta: LF token         = 227 '<0x0A>'\n",
            "llm_load_print_meta: EOT token        = 107 '<end_of_turn>'\n",
            "llm_load_print_meta: EOG token        = 1 '<eos>'\n",
            "llm_load_print_meta: EOG token        = 107 '<end_of_turn>'\n",
            "llm_load_print_meta: max token length = 48\n",
            "llm_load_tensors: ggml ctx size =    0.45 MiB\n",
            "llm_load_tensors: offloading 46 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 47/47 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =   717.77 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  5644.80 MiB\n",
            "................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 2048\n",
            "llama_new_context_with_model: n_batch    = 512\n",
            "llama_new_context_with_model: n_ubatch   = 512\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 10000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =   736.00 MiB\n",
            "llama_new_context_with_model: KV self size  =  736.00 MiB, K (f16):  368.00 MiB, V (f16):  368.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.98 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   507.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =    15.01 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1850\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
            "Model metadata: {'general.quantization_version': '2', 'tokenizer.ggml.add_space_prefix': 'false', 'gemma2.attention.head_count': '16', 'gemma2.feed_forward_length': '14336', 'gemma2.block_count': '46', 'tokenizer.ggml.pre': 'default', 'general.base_model.count': '0', 'gemma2.attn_logit_softcapping': '50.000000', 'general.type': 'model', 'gemma2.embedding_length': '3584', 'general.basename': 'Gemma-The-Writer-exp4-3', 'tokenizer.ggml.padding_token_id': '0', 'gemma2.context_length': '8192', 'tokenizer.chat_template': \"{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\\n' + message['content'] | trim + '<end_of_turn>\\n' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\\n'}}{% endif %}\", 'general.architecture': 'gemma2', 'gemma2.attention.key_length': '256', 'gemma2.attention.value_length': '256', 'gemma2.attention.layer_norm_rms_epsilon': '0.000001', 'general.finetune': 'x4', 'general.file_type': '14', 'gemma2.attention.sliding_window': '4096', 'gemma2.final_logit_softcapping': '30.000000', 'gemma2.attention.head_count_kv': '8', 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.model': 'llama', 'general.name': 'Gemma The Writer Exp4 3 Bf16 X4', 'tokenizer.ggml.bos_token_id': '2', 'tokenizer.ggml.eos_token_id': '1', 'tokenizer.ggml.unknown_token_id': '3', 'general.size_label': '10B', 'tokenizer.ggml.add_bos_token': 'true'}\n",
            "Available chat formats from metadata: chat_template.default\n",
            "Using gguf chat template: {{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\n",
            "' + message['content'] | trim + '<end_of_turn>\n",
            "' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\n",
            "'}}{% endif %}\n",
            "Using chat eos_token: <eos>\n",
            "Using chat bos_token: <bos>\n",
            "\u001b[2;36m09:44:57-631574\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[32m\"Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\"\u001b[0m in \u001b[1;36m3.34\u001b[0m    \n",
            "\u001b[2;36m                \u001b[0m         seconds.                                                                   \n",
            "\u001b[2;36m09:44:57-633363\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m LOADER: \u001b[32m\"llama.cpp\"\u001b[0m                                                        \n",
            "\u001b[2;36m09:44:57-634600\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m TRUNCATION LENGTH: \u001b[1;36m2048\u001b[0m                                                    \n",
            "\u001b[2;36m09:44:57-635849\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m INSTRUCTION TEMPLATE: \u001b[32m\"Custom \u001b[0m\u001b[32m(\u001b[0m\u001b[32mobtained from model metadata\u001b[0m\u001b[32m)\u001b[0m\u001b[32m\"\u001b[0m              \n",
            "\u001b[2;36m09:45:12-268566\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Deleted \u001b[32m\"logs/instruct/20241205-09-42-35.json\"\u001b[0m.                            \n",
            "\u001b[2;36m09:45:15-962905\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;31mERROR   \u001b[0m Failed to build the chat prompt. The input is too long for the available   \n",
            "\u001b[2;36m                \u001b[0m         context length.                                                            \n",
            "\u001b[2;36m                \u001b[0m                                                                                    \n",
            "\u001b[2;36m                \u001b[0m         Truncation length: \u001b[1;36m2048\u001b[0m                                                    \n",
            "\u001b[2;36m                \u001b[0m         max_new_tokens: \u001b[1;36m2048\u001b[0m \u001b[1m(\u001b[0mis it too high?\u001b[1m)\u001b[0m                                     \n",
            "\u001b[2;36m                \u001b[0m         Available context length: \u001b[1;36m0\u001b[0m                                                \n",
            "\u001b[2;36m                \u001b[0m                                                                                    \n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/queueing.py\", line 566, in process_events\n",
            "    response = await route_utils.call_process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py\", line 261, in call_process_api\n",
            "    output = await app.get_blocks().process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1786, in process_api\n",
            "    result = await self.call_function(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1350, in call_function\n",
            "    prediction = await utils.async_iteration(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 583, in async_iteration\n",
            "    return await iterator.__anext__()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 576, in __anext__\n",
            "    return await anyio.to_thread.run_sync(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py\", line 33, in run_sync\n",
            "    return await get_asynclib().run_sync_in_worker_thread(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 877, in run_sync_in_worker_thread\n",
            "    return await future\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 807, in run\n",
            "    result = context.run(func, *args)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 559, in run_sync_iterator_async\n",
            "    return next(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 742, in gen_wrapper\n",
            "    response = next(iterator)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 436, in generate_chat_reply_wrapper\n",
            "    for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True, for_ui=True)):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 403, in generate_chat_reply\n",
            "    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message, for_ui=for_ui):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 348, in chatbot_wrapper\n",
            "    prompt = generate_chat_prompt(text, state, **kwargs)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 236, in generate_chat_prompt\n",
            "    raise ValueError\n",
            "ValueError\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/queueing.py\", line 566, in process_events\n",
            "    response = await route_utils.call_process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py\", line 261, in call_process_api\n",
            "    output = await app.get_blocks().process_api(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1786, in process_api\n",
            "    result = await self.call_function(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 1350, in call_function\n",
            "    prediction = await utils.async_iteration(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 583, in async_iteration\n",
            "    return await iterator.__anext__()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 576, in __anext__\n",
            "    return await anyio.to_thread.run_sync(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py\", line 33, in run_sync\n",
            "    return await get_asynclib().run_sync_in_worker_thread(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 877, in run_sync_in_worker_thread\n",
            "    return await future\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py\", line 807, in run\n",
            "    result = context.run(func, *args)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 559, in run_sync_iterator_async\n",
            "    return next(iterator)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/utils.py\", line 742, in gen_wrapper\n",
            "    response = next(iterator)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 436, in generate_chat_reply_wrapper\n",
            "    for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True, for_ui=True)):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 403, in generate_chat_reply\n",
            "    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message, for_ui=for_ui):\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 348, in chatbot_wrapper\n",
            "    prompt = generate_chat_prompt(text, state, **kwargs)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 195, in generate_chat_prompt\n",
            "    prompt = make_prompt(messages)\n",
            "  File \"/content/text-generation-webui/modules/chat.py\", line 151, in make_prompt\n",
            "    prompt = renderer(messages=messages)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/jinja2/environment.py\", line 1304, in render\n",
            "    self.environment.handle_exception()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/jinja2/environment.py\", line 939, in handle_exception\n",
            "    raise rewrite_traceback_stack(source=source)\n",
            "  File \"<template>\", line 1, in top-level template code\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/jinja2/sandbox.py\", line 304, in getitem\n",
            "    return obj[argument]\n",
            "jinja2.exceptions.UndefinedError: list object has no element 0\n",
            "llama_perf_context_print:        load time =     695.80 ms\n",
            "llama_perf_context_print: prompt eval time =       0.00 ms /    76 tokens (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:        eval time =       0.00 ms /  1028 runs   (    0.00 ms per token,      inf tokens per second)\n",
            "llama_perf_context_print:       total time =   71381.36 ms /  1104 tokens\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "!pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121\n",
        "\n",
        "!git clone -b V20241028 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/DavidAU/Gemma-The-Writer-N-Restless-Quill-10B-Uncensored-GGUF/resolve/main/Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf?download=true -d /content/text-generation-webui/models/ -o Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 100000 --model Gemma-The-Writer-N-Restless-Quill-10B-D_AU-Q4_k_s.gguf --n_ctx 2048"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMRfPCN+hHa9JPd4/DwuB9x",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}