05-Adding_Memory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01a8b5c0-87cb-4302-8e3c-dc809d0039fb",
   "metadata": {},
   "source": [
    "# Understanding Memory in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f73380-6395-4e9f-9c83-3f47a5d7e292",
   "metadata": {},
   "source": [
    "In the previous Notebooks, we successfully explored how OpenAI models can enhance the results from Azure AI Search queries. \n",
    "\n",
    "However, we have yet to discover how to engage in a conversation with the LLM. With [Bing Chat](http://chat.bing.com/), for example, this is possible, as it can understand and reference the previous responses.\n",
    "\n",
    "There is a common misconception that LLMs (Large Language Models) have memory. This is not true. While they possess knowledge, they do not retain information from previous questions asked to them.\n",
    "\n",
    "In this Notebook, our goal is to illustrate how we can effectively \"endow the LLM with memory\" by employing prompts and context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "733c782e-204c-47d0-8dae-c9df7091ab23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from langchain_community.chat_message_histories import ChatMessageHistory, CosmosDBChatMessageHistory\n",
    "from langchain_core.chat_history import BaseChatMessageHistory\n",
    "from langchain_core.runnables import ConfigurableFieldSpec\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain_openai import AzureOpenAIEmbeddings\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from operator import itemgetter\n",
    "from typing import List\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import CustomAzureSearchRetriever, get_answer\n",
    "from common.prompts import DOCSEARCH_PROMPT\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "import logging\n",
    "\n",
    "# Get the root logger\n",
    "logger = logging.getLogger()\n",
    "# Set the logging level to a higher level to ignore INFO messages\n",
    "logger.setLevel(logging.WARNING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bc63c55-a57d-49a7-b6c7-0f18bca8199e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc72b22-11c2-4df0-91b8-033d01829663",
   "metadata": {},
   "source": [
    "### Let's start with the basics\n",
    "Let's use a very simple example to see if the GPT model of Azure OpenAI have memory. We again will be using langchain to simplify our code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3eef5dc9-8b80-4085-980c-865fa41fa1f6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me some use cases for reinforcement learning\"\n",
    "FOLLOW_UP_QUESTION = \"What was my prior question?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a00181d5-bd76-4ce4-a256-75ac5b58c60f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 1000\n",
    "# Create an OpenAI instance\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT4oMINI_DEPLOYMENT_NAME\"], \n",
    "                      temperature=0.5, max_tokens=COMPLETION_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9502d0f1-fddf-40d1-95d2-a1461dcc498a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We create a very simple prompt template, just the question as is:\n",
    "output_parser = StrOutputParser()\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are an assistant that give thorough responses to users.\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c5c9903e-15c7-4e05-87a1-58e5a7917ba2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement Learning (RL) is a powerful area of machine learning where an agent learns to make decisions by taking actions in an environment to maximize cumulative rewards. Here are some notable use cases across various domains:\n",
       "\n",
       "### 1. **Robotics**\n",
       "   - **Autonomous Navigation**: Robots can learn to navigate through complex environments, avoiding obstacles and optimizing paths.\n",
       "   - **Manipulation Tasks**: RL can be used to train robots to perform tasks like grasping objects or assembling parts, adapting to different shapes and weights.\n",
       "\n",
       "### 2. **Gaming**\n",
       "   - **Game AI**: RL has been used to develop AI that can play games at superhuman levels, such as AlphaGo for Go and OpenAI Five for Dota 2.\n",
       "   - **Procedural Content Generation**: RL can be used to adapt game environments dynamically based on player behavior, enhancing engagement.\n",
       "\n",
       "### 3. **Healthcare**\n",
       "   - **Personalized Treatment Plans**: RL can optimize treatment strategies for chronic diseases by learning from patient responses to different interventions.\n",
       "   - **Drug Discovery**: It can help in identifying promising drug candidates by optimizing molecular structures through simulation.\n",
       "\n",
       "### 4. **Finance**\n",
       "   - **Algorithmic Trading**: RL can be employed to develop trading strategies that adapt to market conditions by learning from historical data.\n",
       "   - **Portfolio Management**: It can optimize asset allocation by learning the best strategies based on market movements and risk factors.\n",
       "\n",
       "### 5. **Natural Language Processing**\n",
       "   - **Dialogue Systems**: RL can enhance chatbots and virtual assistants by optimizing responses based on user satisfaction and engagement metrics.\n",
       "   - **Text Summarization**: RL can help in generating concise summaries by maximizing relevance and coherence from the original text.\n",
       "\n",
       "### 6. **Autonomous Vehicles**\n",
       "   - **Driving Policies**: RL can be used to develop driving algorithms that learn from real-world driving scenarios, improving safety and efficiency.\n",
       "   - **Traffic Management**: It can optimize traffic signal timings and routing to reduce congestion and improve flow.\n",
       "\n",
       "### 7. **Recommendation Systems**\n",
       "   - **Dynamic Recommendations**: RL can optimize recommendations in real-time based on user interactions, improving user engagement and satisfaction.\n",
       "   - **Content Personalization**: It can be used to personalize content delivery on platforms like streaming services, adapting to user preferences over time.\n",
       "\n",
       "### 8. **Energy Management**\n",
       "   - **Smart Grids**: RL can optimize energy distribution in smart grids, balancing supply and demand efficiently.\n",
       "   - **Building Energy Management**: It can learn to manage heating, ventilation, and air conditioning (HVAC) systems to minimize energy consumption while maintaining comfort.\n",
       "\n",
       "### 9. **Manufacturing**\n",
       "   - **Supply Chain Optimization**: RL can help in managing inventory levels and production schedules, reducing costs and improving efficiency.\n",
       "   - **Quality Control**: It can be used to optimize inspection processes, learning to identify defects in products through trial and error.\n",
       "\n",
       "### 10. **Education**\n",
       "   - **Adaptive Learning Systems**: RL can tailor educational content to individual learning styles and paces, enhancing student engagement and effectiveness.\n",
       "   - **Game-based Learning**: It can optimize learning pathways in educational games, adapting to the player’s progress and difficulties.\n",
       "\n",
       "### Conclusion\n",
       "These use cases demonstrate the versatility and potential of reinforcement learning across various sectors. By leveraging the trial-and-error learning process, RL can optimize complex decision-making tasks, leading to improved efficiency, effectiveness, and user satisfaction in numerous applications."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see what the GPT model responds\n",
    "chain = prompt | llm | output_parser\n",
    "response_to_initial_question = chain.invoke({\"input\": QUESTION})\n",
    "display(Markdown(response_to_initial_question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "99acaf3c-ce68-4b87-b24a-6065b15ff9a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I'm unable to access previous interactions or questions. However, I'm here to help you with any new questions or topics you'd like to discuss! What can I assist you with today?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Now let's ask a follow up question\n",
    "printmd(chain.invoke({\"input\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e1c143-c95f-4566-a8b4-af8c42f08dd2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As you can see, it doesn't remember what it just responded, sometimes it responds based only on the system prompt, or just randomly. This proof that the LLM does NOT have memory and that we need to give the memory as a a conversation history as part of the prompt, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0946ce71-6285-432e-b011-9c2dc1ba7b8a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hist_prompt = ChatPromptTemplate.from_template(\n",
    "\"\"\"\n",
    "    {history}\n",
    "    Human: {question}\n",
    "    AI:\n",
    "\"\"\"\n",
    ")\n",
    "chain = hist_prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6d088e51-e5eb-4143-b87d-b2be429eb864",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Conversation_history = \"\"\"\n",
    "Human: {question}\n",
    "AI: {response}\n",
    "\"\"\".format(question=QUESTION, response=response_to_initial_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d99e34ad-5539-44dd-b080-3ad05efd2f01",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was asking for some use cases for reinforcement learning."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain.invoke({\"history\":Conversation_history, \"question\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e5af6-55d6-4353-b3f6-3275c95db00a",
   "metadata": {},
   "source": [
    "**Bingo!**, so we now know how to create a chatbot using LLMs, we just need to keep the state/history of the conversation and pass it as context every time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafd1694-0077-4aa8-bd01-e9f763ce36a3",
   "metadata": {},
   "source": [
    "## Now that we understand the concept of memory via adding history as a context, let's go back to our GPT Smart Search engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9787ffb6-2b11-4b03-92fc-9443cd1f2ab9",
   "metadata": {},
   "source": [
    "From Langchain website:\n",
    "    \n",
    "A memory system needs to support two basic actions: reading and writing. Recall that every chain defines some core execution logic that expects certain inputs. Some of these inputs come directly from the user, but some of these inputs can come from memory. A chain will interact with its memory system twice in a given run.\n",
    "\n",
    "    AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.\n",
    "    AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.\n",
    "    \n",
    "So this process adds delays to the response, but it is a necessary delay :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f36e8f14-e566-4ae9-a7d4-6dee7f469dad",
   "metadata": {},
   "source": [
    "![image](./images/memory_diagram.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ef9f459b-e8b8-40b9-a94d-80c079968594",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "index1_name = \"srch-index-files\"\n",
    "index2_name = \"srch-index-csv\"\n",
    "index3_name = \"srch-index-books\"\n",
    "indexes = [index1_name, index2_name, index3_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b01852c2-6192-496c-adff-4270f9380469",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize our custom retriever \n",
    "retriever = CustomAzureSearchRetriever(indexes=indexes, topK=10, reranker_threshold=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633937e8-18e6-43f2-b4d5-fc36157a4d97",
   "metadata": {},
   "source": [
    "If you check closely in prompts.py, there is an optional variable in the `DOCSEARCH_PROMPT` called `history`. Now it is the time to use it. It is basically a place holder were we will inject the conversation in the prompt so the LLM is aware of it before it answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035fa6e6-226c-400f-a504-30255385f43b",
   "metadata": {},
   "source": [
    "**Now let's add memory to it:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "3c8c9381-08d0-4808-9ab1-78156ca1be6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "store = {} # Our first memory will be a dictionary in memory\n",
    "\n",
    "# We have to define a custom function that takes a session_id and looks somewhere\n",
    "# (in this case in a dictionary in memory) for the conversation\n",
    "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
    "    if session_id not in store:\n",
    "        store[session_id] = ChatMessageHistory()\n",
    "    return store[session_id]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "48ff51e1-2b1e-4c67-965d-1c2e2f55e005",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We use our original chain with the retriever but removing the StrOutputParser\n",
    "chain = (\n",
    "    {\n",
    "        \"context\": itemgetter(\"question\") | retriever, \n",
    "        \"question\": itemgetter(\"question\"),\n",
    "        \"history\": itemgetter(\"history\")\n",
    "    }\n",
    "    | DOCSEARCH_PROMPT\n",
    "    | llm\n",
    ")\n",
    "\n",
    "## Then we pass the above chain to another chain that adds memory to it\n",
    "\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0e582915-243f-42cb-bb1e-c35a20ee0b9f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is where we configure the session id\n",
    "config={\"configurable\": {\"session_id\": \"abc123\"}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ff493b1-b133-4880-a040-e80f7460e7af",
   "metadata": {},
   "source": [
    "Notice below, that we are adding a `history` variable in the call. This variable will hold the chat historywithin the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d91a7ff4-6148-459d-917c-37302805dd09",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has a variety of applications across different domains. Here are some notable use cases:\n",
       "\n",
       "1. **Epidemic Modeling and Control:** RL can be utilized to model the spread of infectious diseases and to formulate optimal intervention strategies. For instance, a multi-agent epidemic model allows individual agents to make decisions that affect disease transmission, which can be optimized using game theory and reinforcement learning techniques [[1]](https://arxiv.org/pdf/2004.12959v1.pdf).\n",
       "\n",
       "2. **Lockdown Decision Making During Pandemics:** In the context of the COVID-19 pandemic, RL algorithms can automatically compute lockdown decisions for specific cities or regions. These policies are based on various disease parameters and population characteristics, balancing health and economic considerations [[2]](https://arxiv.org/pdf/2003.14093v2.pdf).\n",
       "\n",
       "3. **Preventive Strategies for Influenza:** A deep reinforcement learning approach has been developed to learn prevention strategies for pandemic influenza. This involves a meta-population model that captures the infection process and uses RL to learn effective mitigation policies across interconnected districts [[3]](https://arxiv.org/pdf/2003.13676v1.pdf).\n",
       "\n",
       "4. **Personalized Recommendation Systems:** RL can enhance recommendation systems by predicting user preferences and adapting recommendations based on user interactions. For example, a hybrid recommendation algorithm uses reinforcement learning to recommend song sequences that better match listeners' evolving preferences [[4]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/).\n",
       "\n",
       "5. **Fairness in Interactive Recommender Systems:** To address bias and discrimination in recommendations, an RL-based framework has been proposed to maintain a balance between accuracy and fairness dynamically. This approach allows the system to adapt to changing user preferences and fairness considerations over time [[5]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206277/).\n",
       "\n",
       "6. **Job Scheduling in Data Centers:** RL methods can be applied to optimize job scheduling in data centers, where multi-dimensional resources need to be allocated efficiently. A specific approach called A2cScheduler employs deep reinforcement learning to improve scheduling performance in complex computing environments [[6]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/).\n",
       "\n",
       "7. **Automatic Feature Engineering in Machine Learning:** Reinforcement learning can also be used to automate the feature engineering process, which is often time-consuming and requires expert knowledge. A framework called CAFEM employs RL to optimize feature transformation strategies across different datasets [[7]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/).\n",
       "\n",
       "These use cases illustrate the versatility and effectiveness of reinforcement learning in solving complex problems across various fields, from healthcare to technology and beyond."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "25dfc233-450f-4671-8f1c-0b446e46f048",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was, \"Tell me some use cases for reinforcement learning.\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c67073c2-9a82-4e44-a9e2-48fe868c1634",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Empty Search Response\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "You're welcome! If you have more questions in the future, feel free to ask. Goodbye and take care!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": \"Thank you! Good bye\"},config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87405173",
   "metadata": {},
   "source": [
    "## Using CosmosDB as persistent memory\n",
    "\n",
    "In previous cell we have added local RAM memory to our chatbot. However, it is not persistent, it gets deleted once the app user's session is terminated. It is necessary then to use a Database for persistent storage of each of the bot user conversations, not only for Analytics and Auditing, but also if we wish to provide recommendations in the future. \n",
    "\n",
    "Here we will store the conversation history into CosmosDB for future auditing purpose.\n",
    "We will use a class in LangChain use CosmosDBChatMessageHistory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d87cc7c6-5ef1-4492-b133-9f63a392e223",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create the function to retrieve the conversation\n",
    "\n",
    "def get_session_history(session_id: str, user_id: str) -> CosmosDBChatMessageHistory:\n",
    "    cosmos = CosmosDBChatMessageHistory(\n",
    "        cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "        cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "        cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "        connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "        session_id=session_id,\n",
    "        user_id=user_id\n",
    "        )\n",
    "\n",
    "    # prepare the cosmosdb instance\n",
    "    cosmos.prepare_cosmos()\n",
    "    return cosmos\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "94f4179b-c1c7-49da-9c80-a42c275ed4d6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    "    history_factory_config=[\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"user_id\",\n",
    "            annotation=str,\n",
    "            name=\"User ID\",\n",
    "            description=\"Unique identifier for the user.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"session_id\",\n",
    "            annotation=str,\n",
    "            name=\"Session ID\",\n",
    "            description=\"Unique identifier for the conversation.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "    ],\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8cf1f1f0-6e46-4136-9f33-4e46617b7d4f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is where we configure the session id and user id\n",
    "random_session_id = \"session\"+ str(random.randint(1, 1000))\n",
    "ramdom_user_id = \"user\"+ str(random.randint(1, 1000))\n",
    "\n",
    "config={\"configurable\": {\"session_id\": random_session_id, \"user_id\": ramdom_user_id}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0b20c00c-4098-4970-84e5-f71ea7615c65",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'configurable': {'session_id': 'session988', 'user_id': 'user220'}}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7e3c32f4-f883-4045-91f9-ca317c2d01fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has a wide range of applications across various fields. Here are some notable use cases:\n",
       "\n",
       "1. **Epidemic Modeling**: RL can be utilized to model and predict the spread of infectious diseases. For instance, a multi-agent epidemic model allows agents to make decisions that affect disease transmission. By applying game theory and reinforcement learning, optimal decisions can be derived to predict disease spread and necessitate external interventions for better regulation of agent behaviors [[1]](https://arxiv.org/pdf/2004.12959v1.pdf).\n",
       "\n",
       "2. **Lockdown Policy Optimization**: In the context of pandemics like COVID-19, RL algorithms can compute lockdown decisions for cities or regions. These policies are learned automatically based on disease parameters and population characteristics, balancing health and economic considerations while accounting for the realities of imperfect lockdowns [[2]](https://arxiv.org/pdf/2003.14093v2.pdf).\n",
       "\n",
       "3. **Prevention Strategies for Infectious Diseases**: RL techniques can be applied to learn prevention strategies in complex epidemiological models, such as pandemic influenza. By using deep reinforcement learning, effective mitigation policies can be developed to control the spread of diseases across multiple districts [[3]](https://arxiv.org/pdf/2003.13676v1.pdf).\n",
       "\n",
       "4. **Music Recommendation Systems**: A personalized hybrid recommendation algorithm based on RL can enhance music recommendations by simulating the interaction process of listeners. This approach captures subtle changes in listener preferences, improving the recommendation of song sequences [[4]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/).\n",
       "\n",
       "5. **Fairness in Interactive Recommender Systems**: RL frameworks can maintain a balance between accuracy and fairness in recommendation systems by dynamically adapting to changes in user preferences and fairness status. This ensures that recommendations are both fair and of high quality [[5]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206277/).\n",
       "\n",
       "6. **Job Scheduling in Data Centers**: An RL-based approach called A2cScheduler can be used for efficient job scheduling in data centers. This method employs deep reinforcement learning to manage resource allocation effectively, adapting to complex computing environments [[6]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206316/).\n",
       "\n",
       "7. **Automatic Feature Engineering**: RL can also play a role in feature engineering for machine learning projects. A framework called Cross-data Automatic Feature Engineering Machine (CAFEM) utilizes RL to optimize feature generation across different datasets, improving the efficiency and performance of machine learning models [[7]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206177/).\n",
       "\n",
       "These examples illustrate the versatility of reinforcement learning in addressing complex decision-making problems across various domains."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7e29643b-a531-4117-8e85-9c88a625cf02",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was about the use cases for reinforcement learning."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "50146f05-5ef6-484f-a8ec-9631643054f2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "We discussed various use cases for reinforcement learning, including applications in epidemic modeling, lockdown policy optimization, music recommendation systems, and job scheduling in data centers."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Can you tell me a one line summary of our conversation?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "8bc02369-904c-4063-93e1-fff24fe6a3ab",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Empty Search Response\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "You're very welcome! If you have any more questions or need assistance, feel free to ask. Enjoy your day!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Thank you very much!\"},\n",
    "    config=config))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "87d60faa-1446-4c07-8970-0f9712c33b2f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Empty Search Response\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "I provided a one-line summary in response to your request for a concise recap of our conversation about the use cases for reinforcement learning. If you have any further questions or need clarification, feel free to ask!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"I do have one more question, why did you give me a one line summary?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "cfe748aa-6116-4a7a-97e6-f1c680dd23ad",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Empty Search Response\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "I aimed to keep it brief and focused, as you specifically requested a one-line summary. However, if you prefer a two-line summary or more detail, I can certainly provide that! Would you like me to expand on it?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"why not 2?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc5ac98",
   "metadata": {},
   "source": [
    "#### Let's check our Azure CosmosDB to see the whole conversation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e30694-ae2a-47bb-a5c7-db51ecdbba1e",
   "metadata": {},
   "source": [
    "![CosmosDB Memory](./images/cosmos-chathistory.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6789cada-23a3-451a-a91a-0906ceb0bd14",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### Adding memory to our application allows the user to have a conversation, however this feature is not something that comes with the LLM, but instead, memory is something that we must provide to the LLM in form of context of the question.\n",
    "\n",
    "We added persitent memory using CosmosDB.\n",
    "\n",
    "We also can notice that the current chain that we are using is smart, but not that much. Although we have given memory to it, many times it searches for similar docs everytime, regardless of the input. This doesn't seem efficient, but regardless, we are very close to finish our first RAG-talk to your data Bot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c629ebf4-aced-45b7-a6a2-315810d37d48",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We know now how to do a Smart Search Engine that can power a chatbot!! great!\n",
    "\n",
    "In the next notebook 6, we are going to build our first RAG bot. In order to do this we will introduce the concept of Agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f53a8f7a-5e28-4d5f-9a33-0a3be0536b0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}