all_agents_tutorials/simple_data_analysis_agent_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis Simple Agent\n",
    "\n",
    "## Overview\n",
    "This tutorial guides you through creating an AI-powered data analysis agent that can interpret and answer questions about a dataset using natural language. It combines language models with data manipulation tools to enable intuitive data exploration.\n",
    "\n",
    "## Motivation\n",
    "Data analysis often requires specialized knowledge, limiting access to insights for non-technical users. By creating an AI agent that understands natural language queries, we can democratize data analysis, allowing anyone to extract valuable information from complex datasets without needing to know programming or statistical tools.\n",
    "\n",
    "## Key Components\n",
    "1. Language Model: Processes natural language queries and generates human-like responses\n",
    "2. Data Manipulation Framework: Handles dataset operations and analysis\n",
    "3. Agent Framework: Connects the language model with data manipulation tools\n",
    "4. Synthetic Dataset: Represents real-world data for demonstration purposes\n",
    "\n",
    "## Method\n",
    "1. Create a synthetic dataset representing car sales data\n",
    "2. Construct an agent that combines the language model with data analysis capabilities\n",
    "3. Implement a query processing function to handle natural language questions\n",
    "4. Demonstrate the agent's abilities with example queries\n",
    "\n",
    "## Conclusion\n",
    "This approach to data analysis offers significant benefits:\n",
    "- Accessibility for non-technical users\n",
    "- Flexibility in handling various query types\n",
    "- Efficient ad-hoc data exploration\n",
    "\n",
    "By making data insights more accessible, this method has the potential to transform how organizations leverage their data for decision-making across various fields and industries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries and set environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent\n",
    "from langchain.agents import AgentType\n",
    "from langchain_openai import ChatOpenAI\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "# Load environment variables\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "# Load environment variables and set OpenAI API key\n",
    "load_dotenv()\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "# Set a random seed for reproducibility\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Sample Data\n",
    "\n",
    "In this section, we create a sample dataset of car sales. This includes generating dates, car makes, models, colors, and other relevant information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "First few rows of the generated data:\n",
      "        Date       Make      Model  Color  Year     Price  Mileage  \\\n",
      "0 2022-01-01   Mercedes      Sedan  Green  2022  57952.65   5522.0   \n",
      "1 2022-01-02  Chevrolet  Hatchback    Red  2021  58668.22  94238.0   \n",
      "2 2022-01-03       Audi      Truck  White  2019  69187.87   7482.0   \n",
      "3 2022-01-04     Nissan  Hatchback  Black  2016  40004.44  43846.0   \n",
      "4 2022-01-05   Mercedes  Hatchback    Red  2016  63983.07  52988.0   \n",
      "\n",
      "   EngineSize  FuelEfficiency SalesPerson  \n",
      "0         2.0            24.7       Alice  \n",
      "1         1.6            26.2         Bob  \n",
      "2         2.0            28.0       David  \n",
      "3         3.5            24.8       David  \n",
      "4         2.5            24.1       Alice  \n",
      "\n",
      "DataFrame info:\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000 entries, 0 to 999\n",
      "Data columns (total 10 columns):\n",
      " #   Column          Non-Null Count  Dtype         \n",
      "---  ------          --------------  -----         \n",
      " 0   Date            1000 non-null   datetime64[ns]\n",
      " 1   Make            1000 non-null   object        \n",
      " 2   Model           1000 non-null   object        \n",
      " 3   Color           1000 non-null   object        \n",
      " 4   Year            1000 non-null   int32         \n",
      " 5   Price           1000 non-null   float64       \n",
      " 6   Mileage         1000 non-null   float64       \n",
      " 7   EngineSize      1000 non-null   float64       \n",
      " 8   FuelEfficiency  1000 non-null   float64       \n",
      " 9   SalesPerson     1000 non-null   object        \n",
      "dtypes: datetime64[ns](1), float64(4), int32(1), object(4)\n",
      "memory usage: 74.3+ KB\n",
      "\n",
      "Summary statistics:\n",
      "                      Date         Year         Price       Mileage  \\\n",
      "count                 1000  1000.000000   1000.000000   1000.000000   \n",
      "mean   2023-05-15 12:00:00  2018.445000  51145.360800  48484.643000   \n",
      "min    2022-01-01 00:00:00  2015.000000  20026.570000     19.000000   \n",
      "25%    2022-09-07 18:00:00  2017.000000  36859.940000  23191.500000   \n",
      "50%    2023-05-15 12:00:00  2018.000000  52215.155000  47506.000000   \n",
      "75%    2024-01-20 06:00:00  2020.000000  65741.147500  73880.250000   \n",
      "max    2024-09-26 00:00:00  2022.000000  79972.640000  99762.000000   \n",
      "std                    NaN     2.256117  17041.610861  29103.404593   \n",
      "\n",
      "        EngineSize  FuelEfficiency  \n",
      "count  1000.000000     1000.000000  \n",
      "mean      2.744500       29.688500  \n",
      "min       1.600000       20.000000  \n",
      "25%       2.000000       24.500000  \n",
      "50%       2.500000       29.700000  \n",
      "75%       3.500000       34.700000  \n",
      "max       4.000000       40.000000  \n",
      "std       0.839389        5.896316  \n"
     ]
    }
   ],
   "source": [
    "# Generate sample data\n",
    "n_rows = 1000\n",
    "\n",
    "# Generate dates\n",
    "start_date = datetime(2022, 1, 1)\n",
    "dates = [start_date + timedelta(days=i) for i in range(n_rows)]\n",
    "\n",
    "# Define data categories\n",
    "makes = ['Toyota', 'Honda', 'Ford', 'Chevrolet', 'Nissan', 'BMW', 'Mercedes', 'Audi', 'Hyundai', 'Kia']\n",
    "models = ['Sedan', 'SUV', 'Truck', 'Hatchback', 'Coupe', 'Van']\n",
    "colors = ['Red', 'Blue', 'Black', 'White', 'Silver', 'Gray', 'Green']\n",
    "\n",
    "# Create the dataset\n",
    "data = {\n",
    "    'Date': dates,\n",
    "    'Make': np.random.choice(makes, n_rows),\n",
    "    'Model': np.random.choice(models, n_rows),\n",
    "    'Color': np.random.choice(colors, n_rows),\n",
    "    'Year': np.random.randint(2015, 2023, n_rows),\n",
    "    'Price': np.random.uniform(20000, 80000, n_rows).round(2),\n",
    "    'Mileage': np.random.uniform(0, 100000, n_rows).round(0),\n",
    "    'EngineSize': np.random.choice([1.6, 2.0, 2.5, 3.0, 3.5, 4.0], n_rows),\n",
    "    'FuelEfficiency': np.random.uniform(20, 40, n_rows).round(1),\n",
    "    'SalesPerson': np.random.choice(['Alice', 'Bob', 'Charlie', 'David', 'Eva'], n_rows)\n",
    "}\n",
    "\n",
    "# Create DataFrame and sort by date\n",
    "df = pd.DataFrame(data).sort_values('Date')\n",
    "\n",
    "# Display sample data and statistics\n",
    "print(\"\\nFirst few rows of the generated data:\")\n",
    "print(df.head())\n",
    "\n",
    "print(\"\\nDataFrame info:\")\n",
    "df.info()\n",
    "\n",
    "print(\"\\nSummary statistics:\")\n",
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Data Analysis Agent\n",
    "\n",
    "Here, we create a Pandas DataFrame agent using LangChain. This agent will be capable of analyzing our dataset and answering questions about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Analysis Agent is ready. You can now ask questions about the data.\n"
     ]
    }
   ],
   "source": [
    "# Create the Pandas DataFrame agent\n",
    "agent = create_pandas_dataframe_agent(\n",
    "    ChatOpenAI(model=\"gpt-4o\", temperature=0),\n",
    "    df,\n",
    "    verbose=True,\n",
    "    allow_dangerous_code=True,\n",
    "    agent_type=AgentType.OPENAI_FUNCTIONS,\n",
    ")\n",
    "print(\"Data Analysis Agent is ready. You can now ask questions about the data.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Question-Asking Function\n",
    "\n",
    "This function allows us to easily ask questions to our data analysis agent and display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ask_agent(question):\n",
    "    \"\"\"Function to ask questions to the agent and display the response\"\"\"\n",
    "    response = agent.run({\n",
    "        \"input\": question,\n",
    "        \"agent_scratchpad\": f\"Human: {question}\\nAI: To answer this question, I need to use Python to analyze the dataframe. I'll use the python_repl_ast tool.\\n\\nAction: python_repl_ast\\nAction Input: \",\n",
    "    })\n",
    "    print(f\"Question: {question}\")\n",
    "    print(f\"Answer: {response}\")\n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Questions\n",
    "\n",
    "Here are some example questions you can ask the data analysis agent. You can modify these or add your own questions to analyze the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\N7\\AppData\\Local\\Temp\\ipykernel_16872\\610968568.py:3: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use invoke instead.\n",
      "  response = agent.run({\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `python_repl_ast` with `{'query': 'df.columns.tolist()'}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3m['Date', 'Make', 'Model', 'Color', 'Year', 'Price', 'Mileage', 'EngineSize', 'FuelEfficiency', 'SalesPerson']\u001b[0m\u001b[32;1m\u001b[1;3mThe column names in the dataset are:\n",
      "1. Date\n",
      "2. Make\n",
      "3. Model\n",
      "4. Color\n",
      "5. Year\n",
      "6. Price\n",
      "7. Mileage\n",
      "8. EngineSize\n",
      "9. FuelEfficiency\n",
      "10. SalesPerson\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "Question: What are the column names in this dataset?\n",
      "Answer: The column names in the dataset are:\n",
      "1. Date\n",
      "2. Make\n",
      "3. Model\n",
      "4. Color\n",
      "5. Year\n",
      "6. Price\n",
      "7. Mileage\n",
      "8. EngineSize\n",
      "9. FuelEfficiency\n",
      "10. SalesPerson\n",
      "---\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `python_repl_ast` with `{'query': 'df.shape[0]'}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3m1000\u001b[0m\u001b[32;1m\u001b[1;3mThe dataset contains 1000 rows.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "Question: How many rows are in this dataset?\n",
      "Answer: The dataset contains 1000 rows.\n",
      "---\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `python_repl_ast` with `{'query': \"df['Price'].mean()\"}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3m51145.360799999995\u001b[0m\u001b[32;1m\u001b[1;3mThe average price of cars sold is approximately $51,145.36.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "Question: What is the average price of cars sold?\n",
      "Answer: The average price of cars sold is approximately $51,145.36.\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# Example questions\n",
    "ask_agent(\"What are the column names in this dataset?\")\n",
    "ask_agent(\"How many rows are in this dataset?\")\n",
    "ask_agent(\"What is the average price of cars sold?\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}