L5_Chat_with_any_LLM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c5753a0c",
   "metadata": {},
   "source": [
    "# L5: Chat with any LLM! 💬"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a01a3724",
   "metadata": {},
   "source": [
    "Load your HF API key and relevant Python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0fa6fa00-6bd1-4839-bcaf-8bae9267ee79",
   "metadata": {
    "height": 199
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import IPython.display\n",
    "from PIL import Image\n",
    "import base64 \n",
    "import requests \n",
    "requests.adapters.DEFAULT_TIMEOUT = 60\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "hf_api_key = os.environ['HF_API_KEY']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "095da8fe-24aa-4dc7-8e08-aa2f949ae21f",
   "metadata": {
    "height": 167
   },
   "outputs": [],
   "source": [
    "# Helper function\n",
    "import requests, json\n",
    "from text_generation import Client\n",
    "\n",
    "#FalcomLM-instruct endpoint on the text_generation library\n",
    "client = Client(os.environ['HF_API_FALCOM_BASE'],\n",
    "                headers={\"Authorization\": f\"Basic {hf_api_key}\"},\n",
    "                timeout=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfe6fc97",
   "metadata": {},
   "source": [
    "## Building an app to chat with any LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "745a3c9b",
   "metadata": {},
   "source": [
    "Here we'll be using an [Inference Endpoint](https://huggingface.co/inference-endpoints) for `falcon-40b-instruct` , the best ranking open source LLM on the [🤗 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a7065860-3c0b-490d-9e7c-22e5b79fc004",
   "metadata": {
    "height": 46
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nMath has been both invented and discovered. It is a human invention in the sense that it is a system of rules and concepts that we have created to help us understand the world around us. However, it is also a discovery in the sense that it is a fundamental aspect of the universe that we have uncovered through our observations and experiments.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = \"Has math been invented or discovered?\"\n",
    "client.generate(prompt, max_new_tokens=256).generated_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0dcb659e-b71b-46da-b9d2-6ee62498995f",
   "metadata": {
    "height": 284
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.9/site-packages/gradio_client/documentation.py:105: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'\n",
      "  warnings.warn(f\"Could not get documentation group for {cls}: {exc}\")\n",
      "/usr/local/lib/python3.9/site-packages/gradio_client/documentation.py:105: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'\n",
      "  warnings.warn(f\"Could not get documentation group for {cls}: {exc}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  https://0.0.0.0:7860\n",
      "IMPORTANT: You are using gradio version 3.37.0, however version 4.29.0 is available, please upgrade.\n",
      "--------\n",
      "\n",
      "Could not create share link. Missing file: /usr/local/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2. \n",
      "\n",
      "Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps: \n",
      "\n",
      "1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64\n",
      "2. Rename the downloaded file to: frpc_linux_amd64_v0.2\n",
      "3. Move the file to this location: /usr/local/lib/python3.9/site-packages/gradio\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://s172-30-155-157p7860.lab-aws-production.deeplearning.ai/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Back to Lesson 2, time flies!\n",
    "import gradio as gr\n",
    "def generate(input, slider):\n",
    "    output = client.generate(input, max_new_tokens=slider).generated_text\n",
    "    return output\n",
    "\n",
    "demo = gr.Interface(fn=generate, \n",
    "                    inputs=[gr.Textbox(label=\"Prompt\"), \n",
    "                            gr.Slider(label=\"Max new tokens\", \n",
    "                                      value=20,  \n",
    "                                      maximum=1024, \n",
    "                                      minimum=1)], \n",
    "                    outputs=[gr.Textbox(label=\"Completion\")])\n",
    "\n",
    "gr.close_all()\n",
    "demo.launch(share=True, server_port=int(os.environ['PORT1']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e5f55e2",
   "metadata": {},
   "source": [
    "## `gr.Chatbot()`\n",
    "\n",
    "- `gr.Chatbot()` allows you to save the chat history (between the user and the LLM) as well as display the dialogue in the app.\n",
    "- Define your `fn` to take in a `gr.Chatbot()` object.  \n",
    "  - Within your defined `fn` function, append a tuple (or a list) containing the user message and the LLM's response:\n",
    "`chatbot_object.append( (user_message, llm_message) )`\n",
    "\n",
    "- Include the chatbot object in both the inputs and the outputs of the app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "43beebb7-40a6-4af5-a701-882821b6ed36",
   "metadata": {
    "height": 386
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 7860\n"
     ]
    },
    {
     "ename": "OSError",
     "evalue": "Cannot find empty port in range: 7861-7861. You can specify a different port by setting the GRADIO_SERVER_PORT environment variable or passing the `server_port` parameter to `launch()`.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 22\u001b[0m\n\u001b[1;32m     19\u001b[0m     msg\u001b[38;5;241m.\u001b[39msubmit(respond, inputs\u001b[38;5;241m=\u001b[39m[msg, chatbot], outputs\u001b[38;5;241m=\u001b[39m[msg, chatbot]) \u001b[38;5;66;03m#Press enter to submit\u001b[39;00m\n\u001b[1;32m     21\u001b[0m gr\u001b[38;5;241m.\u001b[39mclose_all()\n\u001b[0;32m---> 22\u001b[0m \u001b[43mdemo\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlaunch\u001b[49m\u001b[43m(\u001b[49m\u001b[43mshare\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mserver_port\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mint\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mos\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menviron\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mPORT2\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/usr/local/lib/python3.9/site-packages/gradio/blocks.py:1840\u001b[0m, in \u001b[0;36mBlocks.launch\u001b[0;34m(self, inline, inbrowser, share, debug, enable_queue, max_threads, auth, auth_message, prevent_thread_lock, show_error, server_name, server_port, show_tips, height, width, encrypt, favicon_path, ssl_keyfile, ssl_certfile, ssl_keyfile_password, ssl_verify, quiet, show_api, file_directories, allowed_paths, blocked_paths, root_path, _frontend, app_kwargs)\u001b[0m\n\u001b[1;32m   1832\u001b[0m     wasm_utils\u001b[38;5;241m.\u001b[39mregister_app(app)\n\u001b[1;32m   1833\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1834\u001b[0m     (\n\u001b[1;32m   1835\u001b[0m         server_name,\n\u001b[1;32m   1836\u001b[0m         server_port,\n\u001b[1;32m   1837\u001b[0m         local_url,\n\u001b[1;32m   1838\u001b[0m         app,\n\u001b[1;32m   1839\u001b[0m         server,\n\u001b[0;32m-> 1840\u001b[0m     ) \u001b[38;5;241m=\u001b[39m \u001b[43mnetworking\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstart_server\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1841\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1842\u001b[0m \u001b[43m        \u001b[49m\u001b[43mserver_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1843\u001b[0m \u001b[43m        \u001b[49m\u001b[43mserver_port\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1844\u001b[0m \u001b[43m        \u001b[49m\u001b[43mssl_keyfile\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1845\u001b[0m \u001b[43m        \u001b[49m\u001b[43mssl_certfile\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1846\u001b[0m \u001b[43m        \u001b[49m\u001b[43mssl_keyfile_password\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1847\u001b[0m \u001b[43m        \u001b[49m\u001b[43mapp_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mapp_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1848\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1849\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mserver_name \u001b[38;5;241m=\u001b[39m server_name\n\u001b[1;32m   1850\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlocal_url \u001b[38;5;241m=\u001b[39m local_url\n",
      "File \u001b[0;32m/usr/local/lib/python3.9/site-packages/gradio/networking.py:169\u001b[0m, in \u001b[0;36mstart_server\u001b[0;34m(blocks, server_name, server_port, ssl_keyfile, ssl_certfile, ssl_keyfile_password, app_kwargs)\u001b[0m\n\u001b[1;32m    167\u001b[0m         \u001b[38;5;28;01mpass\u001b[39;00m\n\u001b[1;32m    168\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 169\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mOSError\u001b[39;00m(\n\u001b[1;32m    170\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot find empty port in range: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mmin\u001b[39m(server_ports)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m-\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mmax\u001b[39m(server_ports)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. You can specify a different port by setting the GRADIO_SERVER_PORT environment variable or passing the `server_port` parameter to `launch()`.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    171\u001b[0m     )\n\u001b[1;32m    173\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m ssl_keyfile \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    174\u001b[0m     path_to_local_server \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://\u001b[39m\u001b[38;5;132;01m{\u001b[39;00murl_host_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m:\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mport\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m/\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
      "\u001b[0;31mOSError\u001b[0m: Cannot find empty port in range: 7861-7861. You can specify a different port by setting the GRADIO_SERVER_PORT environment variable or passing the `server_port` parameter to `launch()`."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMPORTANT: You are using gradio version 3.37.0, however version 4.29.0 is available, please upgrade.\n",
      "--------\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def respond(message, chat_history):\n",
    "        #No LLM here, just respond with a random pre-made message\n",
    "        bot_message = random.choice([\"Tell me more about it\", \n",
    "                                     \"Cool, but I'm not interested\", \n",
    "                                     \"Hmmmm, ok then\"]) \n",
    "        chat_history.append((message, bot_message))\n",
    "        return \"\", chat_history\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "    chatbot = gr.Chatbot(height=240) #just to fit the notebook\n",
    "    msg = gr.Textbox(label=\"Prompt\")\n",
    "    btn = gr.Button(\"Submit\")\n",
    "    clear = gr.ClearButton(components=[msg, chatbot], \n",
    "                           value=\"Clear console\")\n",
    "\n",
    "    btn.click(respond, inputs=[msg, chatbot], outputs=[msg, chatbot])\n",
    "    msg.submit(respond, inputs=[msg, chatbot], outputs=[msg, chatbot]) #Press enter to submit\n",
    "\n",
    "gr.close_all()\n",
    "demo.launch(share=True, server_port=int(os.environ['PORT2']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8646d777-c211-4d31-9426-7b5d78b533ae",
   "metadata": {},
   "source": [
    "#### Format the prompt with the chat history\n",
    "\n",
    "- You can iterate through the chatbot object with a for loop.\n",
    "- Each item is a tuple containing the user message and the LLM's message.\n",
    "\n",
    "```Python\n",
    "for turn in chat_history:\n",
    "    user_msg, bot_msg = turn\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "55bae99d-7a63-4a40-bab7-de7d10b8ab1b",
   "metadata": {
    "height": 471
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 7860\n",
      "Running on local URL:  https://0.0.0.0:7862\n",
      "IMPORTANT: You are using gradio version 3.37.0, however version 4.29.0 is available, please upgrade.\n",
      "--------\n",
      "\n",
      "Could not create share link. Missing file: /usr/local/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2. \n",
      "\n",
      "Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps: \n",
      "\n",
      "1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64\n",
      "2. Rename the downloaded file to: frpc_linux_amd64_v0.2\n",
      "3. Move the file to this location: /usr/local/lib/python3.9/site-packages/gradio\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://s172-30-155-157p7862.lab-aws-production.deeplearning.ai/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def format_chat_prompt(message, chat_history):\n",
    "    prompt = \"\"\n",
    "    for turn in chat_history:\n",
    "        user_message, bot_message = turn\n",
    "        prompt = f\"{prompt}\\nUser: {user_message}\\nAssistant: {bot_message}\"\n",
    "    prompt = f\"{prompt}\\nUser: {message}\\nAssistant:\"\n",
    "    return prompt\n",
    "\n",
    "def respond(message, chat_history):\n",
    "        formatted_prompt = format_chat_prompt(message, chat_history)\n",
    "        bot_message = client.generate(formatted_prompt,\n",
    "                                     max_new_tokens=1024,\n",
    "                                     stop_sequences=[\"\\nUser:\", \"<|endoftext|>\"]).generated_text\n",
    "        chat_history.append((message, bot_message))\n",
    "        return \"\", chat_history\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "    chatbot = gr.Chatbot(height=240) #just to fit the notebook\n",
    "    msg = gr.Textbox(label=\"Prompt\")\n",
    "    btn = gr.Button(\"Submit\")\n",
    "    clear = gr.ClearButton(components=[msg, chatbot], value=\"Clear console\")\n",
    "\n",
    "    btn.click(respond, inputs=[msg, chatbot], outputs=[msg, chatbot])\n",
    "    msg.submit(respond, inputs=[msg, chatbot], outputs=[msg, chatbot]) #Press enter to submit\n",
    "\n",
    "gr.close_all()\n",
    "demo.launch(share=True, server_port=int(os.environ['PORT3']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22b8de8",
   "metadata": {},
   "source": [
    "### Adding other advanced features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9e4fff81-a3d1-4cb8-8d6e-d152ab39065a",
   "metadata": {
    "height": 131
   },
   "outputs": [],
   "source": [
    "def format_chat_prompt(message, chat_history, instruction):\n",
    "    prompt = f\"System:{instruction}\"\n",
    "    for turn in chat_history:\n",
    "        user_message, bot_message = turn\n",
    "        prompt = f\"{prompt}\\nUser: {user_message}\\nAssistant: {bot_message}\"\n",
    "    prompt = f\"{prompt}\\nUser: {message}\\nAssistant:\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ee9bc5-fce7-44b1-af2a-e69bc7c598b6",
   "metadata": {},
   "source": [
    "### Streaming\n",
    "\n",
    "- If your LLM can provide its tokens one at a time in a stream, you can accumulate those tokens in the chatbot object.\n",
    "- The `for` loop in the following function goes through all the tokens that are in the stream and appends them to the most recent conversational turn in the chatbot's message history."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "700eb3bc-b63a-4ccb-94c4-70ec2e54bcda",
   "metadata": {
    "height": 437
   },
   "outputs": [],
   "source": [
    "def respond(message, chat_history, instruction, temperature=0.7):\n",
    "    prompt = format_chat_prompt(message, chat_history, instruction)\n",
    "    chat_history = chat_history + [[message, \"\"]]\n",
    "    stream = client.generate_stream(prompt,\n",
    "                                      max_new_tokens=1024,\n",
    "                                      stop_sequences=[\"\\nUser:\", \"<|endoftext|>\"],\n",
    "                                      temperature=temperature)\n",
    "                                      #stop_sequences to not generate the user answer\n",
    "    acc_text = \"\"\n",
    "    #Streaming the tokens\n",
    "    for idx, response in enumerate(stream):\n",
    "            text_token = response.token.text\n",
    "\n",
    "            if response.details:\n",
    "                return\n",
    "\n",
    "            if idx == 0 and text_token.startswith(\" \"):\n",
    "                text_token = text_token[1:]\n",
    "\n",
    "            acc_text += text_token\n",
    "            last_turn = list(chat_history.pop(-1))\n",
    "            last_turn[-1] += acc_text\n",
    "            chat_history = chat_history + [last_turn]\n",
    "            yield \"\", chat_history\n",
    "            acc_text = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "09873dfd-5b6c-41d6-9479-12e8c8894295",
   "metadata": {
    "height": 267
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 7860\n",
      "Running on local URL:  https://0.0.0.0:7863\n",
      "IMPORTANT: You are using gradio version 3.37.0, however version 4.29.0 is available, please upgrade.\n",
      "--------\n",
      "\n",
      "Could not create share link. Missing file: /usr/local/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2. \n",
      "\n",
      "Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps: \n",
      "\n",
      "1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64\n",
      "2. Rename the downloaded file to: frpc_linux_amd64_v0.2\n",
      "3. Move the file to this location: /usr/local/lib/python3.9/site-packages/gradio\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://s172-30-155-157p7863.lab-aws-production.deeplearning.ai/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.9/site-packages/requests/models.py\", line 971, in json\n",
      "    return complexjson.loads(self.text, **kwargs)\n",
      "  File \"/usr/local/lib/python3.9/json/__init__.py\", line 346, in loads\n",
      "    return _default_decoder.decode(s)\n",
      "  File \"/usr/local/lib/python3.9/json/decoder.py\", line 340, in decode\n",
      "    raise JSONDecodeError(\"Extra data\", s, end)\n",
      "json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/routes.py\", line 442, in run_predict\n",
      "    output = await app.get_blocks().process_api(\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/blocks.py\", line 1389, in process_api\n",
      "    result = await self.call_function(\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/blocks.py\", line 1108, in call_function\n",
      "    prediction = await utils.async_iteration(iterator)\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/utils.py\", line 347, in async_iteration\n",
      "    return await iterator.__anext__()\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/utils.py\", line 340, in __anext__\n",
      "    return await anyio.to_thread.run_sync(\n",
      "  File \"/usr/local/lib/python3.9/site-packages/anyio/to_thread.py\", line 56, in run_sync\n",
      "    return await get_async_backend().run_sync_in_worker_thread(\n",
      "  File \"/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py\", line 2144, in run_sync_in_worker_thread\n",
      "    return await future\n",
      "  File \"/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py\", line 851, in run\n",
      "    result = context.run(func, *args)\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/utils.py\", line 323, in run_sync_iterator_async\n",
      "    return next(iterator)\n",
      "  File \"/usr/local/lib/python3.9/site-packages/gradio/utils.py\", line 692, in gen_wrapper\n",
      "    yield from f(*args, **kwargs)\n",
      "  File \"/tmp/ipykernel_21/693938904.py\", line 11, in respond\n",
      "    for idx, response in enumerate(stream):\n",
      "  File \"/usr/local/lib/python3.9/site-packages/text_generation/client.py\", line 235, in generate_stream\n",
      "    raise parse_error(resp.status_code, resp.json())\n",
      "  File \"/usr/local/lib/python3.9/site-packages/requests/models.py\", line 975, in json\n",
      "    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)\n",
      "requests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)\n"
     ]
    }
   ],
   "source": [
    "with gr.Blocks() as demo:\n",
    "    chatbot = gr.Chatbot(height=240) #just to fit the notebook\n",
    "    msg = gr.Textbox(label=\"Prompt\")\n",
    "    with gr.Accordion(label=\"Advanced options\",open=False):\n",
    "        system = gr.Textbox(label=\"System message\", lines=2, \n",
    "                            value=\"A conversation between a user and an LLM-based AI assistant. The assistant gives helpful and honest answers.\")\n",
    "        temperature = gr.Slider(label=\"temperature\", minimum=0.1, maximum=1, value=0.7, step=0.1)\n",
    "    btn = gr.Button(\"Submit\")\n",
    "    clear = gr.ClearButton(components=[msg, chatbot], value=\"Clear console\")\n",
    "\n",
    "    btn.click(respond, inputs=[msg, chatbot, system], outputs=[msg, chatbot])\n",
    "    msg.submit(respond, inputs=[msg, chatbot, system], outputs=[msg, chatbot]) #Press enter to submit\n",
    "\n",
    "gr.close_all()\n",
    "demo.queue().launch(share=True, server_port=int(os.environ['PORT4']))    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4a51a07",
   "metadata": {},
   "source": [
    "Notice, in the cell above, you have used `demo.queue().launch()` instead of `demo.launch()`. \"queue\" helps you to boost up the performance for your demo. You can read [setting up a demo for maximum performance](https://www.gradio.app/guides/setting-up-a-demo-for-maximum-performance) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8d9ec80a-39ad-4f58-b79e-4f413c5074c0",
   "metadata": {
    "height": 29
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 7860\n"
     ]
    }
   ],
   "source": [
    "gr.close_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12cf9b3a-4202-4e3a-9c6b-941fa1290ab8",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}