zzz-nb001-data-loaders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "465c003a-29cf-4cfe-a0c6-d36c04ae2b37",
   "metadata": {},
   "source": [
    "# Data Loaders\n",
    "* Load all kinds of data and then ask the LLM questions about it.\n",
    "* Connect with data sources and load private documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8332b6e9-8164-4859-879c-f021a4dfd69d",
   "metadata": {},
   "source": [
    "## LangChain built-in data loaders.\n",
    "* Labeled as \"integrations\".\n",
    "* Most of them require to install the corresponding libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a2f9501-fa9e-4830-95e2-537dff951cf1",
   "metadata": {},
   "source": [
    "## LangChain documentation on Document Loaders\n",
    "* See the documentation page [here](https://python.langchain.com/v0.1/docs/modules/data_connection/document_loaders/).\n",
    "* See the list of built-in document loaders [here](https://python.langchain.com/v0.1/docs/integrations/document_loaders/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be46161e-45e9-46d7-8214-bcbea10aff2e",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "871e0018-cba4-4959-881a-0a65093d202d",
   "metadata": {},
   "source": [
    "#### After you download the code from the github repository in your computer\n",
    "In terminal:\n",
    "* cd project_name\n",
    "* pyenv local 3.11.4\n",
    "* poetry install\n",
    "* poetry shell"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4065e336-d054-412c-8a3f-1fbec63e1bcd",
   "metadata": {},
   "source": [
    "#### To open the notebook with Jupyter Notebooks\n",
    "In terminal:\n",
    "* jupyter lab\n",
    "\n",
    "Go to the folder of notebooks and open the right notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62dda8d4-80cf-4b8f-9981-94edda5e9911",
   "metadata": {},
   "source": [
    "#### To see the code in Virtual Studio Code or your editor of choice.\n",
    "* open Virtual Studio Code or your editor of choice.\n",
    "* open the project-folder\n",
    "* open the 001-data-loaders.py file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af743328-1bc8-4b01-85fb-fcb21c6499c2",
   "metadata": {},
   "source": [
    "## Create your .env file\n",
    "* In the github repo we have included a file named .env.example\n",
    "* Rename that file to .env file and here is where you will add your confidential api keys. Remember to include:\n",
    "* OPENAI_API_KEY=your_openai_api_key\n",
    "* LANGCHAIN_TRACING_V2=true\n",
    "* LANGCHAIN_ENDPOINT=https://api.smith.langchain.com\n",
    "* LANGCHAIN_API_KEY=your_langchain_api_key\n",
    "* LANGCHAIN_PROJECT=your_project_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "863dd299-0780-49ad-a1b7-b76e249350da",
   "metadata": {},
   "source": [
    "We will call our LangSmith project **001-data-loaders**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40e766aa-f3e2-491f-be99-d0c6b700d47a",
   "metadata": {},
   "source": [
    "## Track operations\n",
    "From now on, we can track the operations **and the cost** of this project from LangSmith:\n",
    "* [smith.langchain.com](https://smith.langchain.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f99504a-1b8f-4360-b342-0b81ffa06aff",
   "metadata": {},
   "source": [
    "## Connect with the .env file located in the same directory of this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39e5789-5bde-42e1-88dd-92dc8e363c24",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c5514113-ddca-4ae9-9de6-0b9225b18f3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fecd39d0-e72e-4bc2-8a68-2fa4008ea365",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv())\n",
    "openai_api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f4a923-b19e-498e-9be5-e47ec4a77d80",
   "metadata": {},
   "source": [
    "#### Install LangChain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fef1e5c-b7e2-4a04-96c5-8f64377b8eba",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c1cf94ae-6c39-4475-9c5b-4b74d8d78753",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install langchain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189e9e17-dfb0-4fd3-85b9-1fba83771941",
   "metadata": {},
   "source": [
    "## Connect with an LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b21d23f4-61f5-4227-8a75-7eefde6680ee",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "148df8e0-361d-4ddd-8709-af48fa1648d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install langchain-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1998155-91de-4cbc-bc88-8d77beefb51b",
   "metadata": {},
   "source": [
    "* NOTE: Since right now is the best LLM in the market, we will use OpenAI by default. You will see how to connect with other Open Source LLMs like Llama3 or Mistral in a next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f639b09-de0e-43ff-826c-03deb85605b9",
   "metadata": {},
   "source": [
    "## Simple data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c5afd8-9c16-4d5e-97da-4c59f1460b15",
   "metadata": {},
   "source": [
    "#### Loading a .txt file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b14d27c3-0b1b-4b11-a883-8da2734f21a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "chatModel = ChatOpenAI(model=\"gpt-3.5-turbo-0125\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cafba46-3c43-457e-9a9a-19978f4fdab9",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2175d878-2274-4852-ae62-b3b9f704f31c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install langchain-community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "438c93f3-a1ff-41de-af09-bc67729ad271",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "\n",
    "loader = TextLoader(\"./data/be-good.txt\")\n",
    "\n",
    "loaded_data = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daa0d08b-8ae6-4276-b2c1-70c0cd8d8bde",
   "metadata": {},
   "source": [
    "* If you uncomment and execute the next cell you will see the contents of the loaded document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8e6bc045-56d3-4e1d-b822-85c40a2e9609",
   "metadata": {},
   "outputs": [],
   "source": [
    "#loaded_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39dc7c6-3386-43ff-a0a6-c3abe54897c5",
   "metadata": {},
   "source": [
    "#### Loading a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4fa559db-8f5a-4ffd-a9c2-541f6099c8cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import CSVLoader\n",
    "\n",
    "loader = CSVLoader('./data/Street_Tree_List.csv')\n",
    "\n",
    "loaded_data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "864441c6-4a4c-4b38-ab38-f51517b2af45",
   "metadata": {},
   "outputs": [],
   "source": [
    "#loaded_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfb26c9d-0813-4877-846d-ea022a345709",
   "metadata": {},
   "source": [
    "#### Loading an .html file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce0455f-d8ae-43a8-9e19-de45bd623393",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "74cf0ae9-8f7f-484e-b2d2-cc8db2c9f4e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install bs4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b62f5922-747d-48e3-94d3-319cb0fbb96c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredHTMLLoader\n",
    "\n",
    "loader = UnstructuredHTMLLoader('./data/100-startups.html')\n",
    "\n",
    "loaded_data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ac9b265a-4b91-420a-a9c1-b9f352095458",
   "metadata": {},
   "outputs": [],
   "source": [
    "#loaded_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9612386a-60df-4e83-9804-a4f0a6b4086e",
   "metadata": {},
   "source": [
    "#### Loading a .pdf file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b5a3725-07af-4be0-a6f0-fe6271a8e060",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "587e4158-d2f3-423f-a481-438aa0822432",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pypdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fb50f45f-e6b7-4623-84d5-d121e33a4387",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader('./data/5pages.pdf')\n",
    "\n",
    "loaded_data = loader.load_and_split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ca3c2412-1de3-4224-ac11-21765dc2e723",
   "metadata": {},
   "outputs": [],
   "source": [
    "#loaded_data[0].page_content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b168178e-7730-4427-b43c-4e2321d216d7",
   "metadata": {},
   "source": [
    "#### Loading a Wikipedia page and asking questions about it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a570d68-31b4-448e-bc93-948df3092223",
   "metadata": {},
   "source": [
    "If you are using the pre-loaded poetry shell, you do not need to install the following package because it is already pre-loaded for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b6b67dd6-19d6-4b15-a153-12b5297cffa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c4af0a00-1afa-456e-951d-15fe8ba2c763",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import WikipediaLoader\n",
    "\n",
    "name = \"JFK\"\n",
    "\n",
    "loader = WikipediaLoader(query=name, load_max_docs=1)\n",
    "\n",
    "loaded_data = loader.load()[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "74c6b0be-9dd0-43b9-bd3e-d0a551987998",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "chat_template = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"human\", \"Answer this {question}, here is some extra {context}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "messages = chat_template.format_messages(\n",
    "    question=\"What was the full name of JFK\",\n",
    "    context=loaded_data\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e618700d-8a5a-4837-adbe-818f3d639497",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = chatModel.invoke(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85bd5fa3",
   "metadata": {},
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc24f4f-d23c-4202-992d-91b0623136ae",
   "metadata": {},
   "source": [
    "## How to execute the code from Visual Studio Code\n",
    "* In Visual Studio Code, see the file 001-data-loaders.py\n",
    "* In terminal, make sure you are in the directory of the file and run:\n",
    "    * python 001-data-loaders.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1af248e-6069-44b3-a2cd-a20aa3259874",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}