notebook/walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "31cad07c",
   "metadata": {},
   "source": [
    "# 1. Introduction\n",
    "\n",
    "In this project, we'll build a multilabel classifier to classify Disaster event related messages into appropriate categories. This can can help tell us about the nature of the event so that the message can be routed to the correct organizations, enabling faster mobilization of resources.\n",
    "\n",
    "The project will be divided into two separate modules: \"Extract, Transformer, Load\" and Machine Learning. As you'll see, the initial data for the project is not clean. We'll take this opportunity to showcase some basic ETL skills, and save the clean data to a Sqlite database, which can then be loaded by the ML pipeline. Finally, we will train an XGBoost classifier to predict the labels fo new messages.\n",
    "\n",
    "This is a walkthrough notebook of the project. The finished scripts, and a web app with a user interface to allow message classification can be found [here](https://github.com/sajal2692/disaster-message-classifier/blob/master/data/process_data.py).\n",
    "\n",
    "*Note: This project is available as part of some Udacity nanodegrees.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "79997438",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib\n",
    "from sqlalchemy import create_engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcc67d46",
   "metadata": {},
   "source": [
    "# 2. Data\n",
    "\n",
    "Data is available to us in two different files: `disaster_messages.csv` and `disaster_categories.csv`.\n",
    "\n",
    "Let's load these two datasets and take a look at their first few rows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a525f371",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = pd.read_csv('../data/disaster_categories.csv')\n",
    "messages = pd.read_csv('../data/disaster_messages.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "bbc5f80e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Weather update - a cold front from Cuba that c...</td>\n",
       "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "      <td>Cyclone nan fini osinon li pa fini</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>says: west side of Haiti, rest of the country ...</td>\n",
       "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                            message  \\\n",
       "0   2  Weather update - a cold front from Cuba that c...   \n",
       "1   7            Is the Hurricane over or is it not over   \n",
       "2   8                    Looking for someone but no name   \n",
       "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
       "4  12  says: west side of Haiti, rest of the country ...   \n",
       "\n",
       "                                            original   genre  \n",
       "0  Un front froid se retrouve sur Cuba ce matin. ...  direct  \n",
       "1                 Cyclone nan fini osinon li pa fini  direct  \n",
       "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct  \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct  \n",
       "4  facade ouest d Haiti et le reste du pays aujou...  direct  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d62564c9",
   "metadata": {},
   "source": [
    "We're only interested in the `message` column from the `disaster_messages` dataset. We'll use that column to train our message classifier, and ignore the other columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2eb2b0ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>categories</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-1;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>related-1;request-1;offer-0;aid_related-1;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                         categories\n",
       "0   2  related-1;request-0;offer-0;aid_related-0;medi...\n",
       "1   7  related-1;request-0;offer-0;aid_related-1;medi...\n",
       "2   8  related-1;request-0;offer-0;aid_related-0;medi...\n",
       "3   9  related-1;request-1;offer-0;aid_related-1;medi...\n",
       "4  12  related-1;request-0;offer-0;aid_related-0;medi..."
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "77872d4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.iloc[0]['categories']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192d9e4a",
   "metadata": {},
   "source": [
    "The `disaster_categories` dataset contains the category labels for our messages, but in a serialized fashion. We have also taken a closer look at the category for the first row. We'll need to convert the categories to a format better suited for our ML model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "253ba1f6",
   "metadata": {},
   "source": [
    "# 3. Extract, Transform, Load!\n",
    "\n",
    "Let's start by merging the two datasets so that the messages and categories are present in the same dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "d6e39343",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = messages.merge(categories, on='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "01091bc1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "      <th>categories</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Weather update - a cold front from Cuba that c...</td>\n",
       "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
       "      <td>direct</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "      <td>Cyclone nan fini osinon li pa fini</td>\n",
       "      <td>direct</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-1;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
       "      <td>direct</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>direct</td>\n",
       "      <td>related-1;request-1;offer-0;aid_related-1;medi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>says: west side of Haiti, rest of the country ...</td>\n",
       "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
       "      <td>direct</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                            message  \\\n",
       "0   2  Weather update - a cold front from Cuba that c...   \n",
       "1   7            Is the Hurricane over or is it not over   \n",
       "2   8                    Looking for someone but no name   \n",
       "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
       "4  12  says: west side of Haiti, rest of the country ...   \n",
       "\n",
       "                                            original   genre  \\\n",
       "0  Un front froid se retrouve sur Cuba ce matin. ...  direct   \n",
       "1                 Cyclone nan fini osinon li pa fini  direct   \n",
       "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct   \n",
       "4  facade ouest d Haiti et le reste du pays aujou...  direct   \n",
       "\n",
       "                                          categories  \n",
       "0  related-1;request-0;offer-0;aid_related-0;medi...  \n",
       "1  related-1;request-0;offer-0;aid_related-1;medi...  \n",
       "2  related-1;request-0;offer-0;aid_related-0;medi...  \n",
       "3  related-1;request-1;offer-0;aid_related-1;medi...  \n",
       "4  related-1;request-0;offer-0;aid_related-0;medi...  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cffad1e",
   "metadata": {},
   "source": [
    "### Cleaning and transforming the categories column \n",
    "\n",
    "Let's start by splitting the  categories column into separate columns for each category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "2b838b19",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = df['categories'].str.split(';',expand=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "a4250808",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-1</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-1</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-1</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-1</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0          1        2              3               4   \\\n",
       "0  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "1  related-1  request-0  offer-0  aid_related-1  medical_help-0   \n",
       "2  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "3  related-1  request-1  offer-0  aid_related-1  medical_help-0   \n",
       "4  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "\n",
       "                   5                    6           7           8   \\\n",
       "0  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "1  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "2  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "3  medical_products-1  search_and_rescue-0  security-0  military-0   \n",
       "4  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "\n",
       "              9   ...             26                      27  \\\n",
       "0  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "1  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "2  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "3  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "4  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "\n",
       "                  28        29       30      31            32      33  \\\n",
       "0  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "1  weather_related-1  floods-0  storm-1  fire-0  earthquake-0  cold-0   \n",
       "2  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "3  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "4  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "\n",
       "                34               35  \n",
       "0  other_weather-0  direct_report-0  \n",
       "1  other_weather-0  direct_report-0  \n",
       "2  other_weather-0  direct_report-0  \n",
       "3  other_weather-0  direct_report-0  \n",
       "4  other_weather-0  direct_report-0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08420380",
   "metadata": {},
   "source": [
    "Extracting the column names from a row of the categories dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "d466d958",
   "metadata": {},
   "outputs": [],
   "source": [
    "category_column_names = categories.iloc[0].apply(lambda x: x.split(\"-\")[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "11ca23fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['related', 'request', 'offer', 'aid_related', 'medical_help', 'medical_products', 'search_and_rescue', 'security', 'military', 'child_alone', 'water', 'food', 'shelter', 'clothing', 'money', 'missing_people', 'refugees', 'death', 'other_aid', 'infrastructure_related', 'transport', 'buildings', 'electricity', 'tools', 'hospitals', 'shops', 'aid_centers', 'other_infrastructure', 'weather_related', 'floods', 'storm', 'fire', 'earthquake', 'cold', 'other_weather', 'direct_report']\n"
     ]
    }
   ],
   "source": [
    "# Let's check which categories we have\n",
    "print([name for name in category_column_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba08140",
   "metadata": {},
   "source": [
    "Using the extracted column names as the header of the categories dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6f218d17",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories.columns = category_column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "6825a601",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>search_and_rescue</th>\n",
       "      <th>security</th>\n",
       "      <th>military</th>\n",
       "      <th>child_alone</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-1</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-1</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-1</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-1</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "0    related    request    offer    aid_related    medical_help  \\\n",
       "0  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "1  related-1  request-0  offer-0  aid_related-1  medical_help-0   \n",
       "2  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "3  related-1  request-1  offer-0  aid_related-1  medical_help-0   \n",
       "4  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "\n",
       "0    medical_products    search_and_rescue    security    military  \\\n",
       "0  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "1  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "2  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "3  medical_products-1  search_and_rescue-0  security-0  military-0   \n",
       "4  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "\n",
       "0    child_alone  ...    aid_centers    other_infrastructure  \\\n",
       "0  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "1  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "2  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "3  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "4  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
       "\n",
       "0    weather_related    floods    storm    fire    earthquake    cold  \\\n",
       "0  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "1  weather_related-1  floods-0  storm-1  fire-0  earthquake-0  cold-0   \n",
       "2  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "3  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "4  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "\n",
       "0    other_weather    direct_report  \n",
       "0  other_weather-0  direct_report-0  \n",
       "1  other_weather-0  direct_report-0  \n",
       "2  other_weather-0  direct_report-0  \n",
       "3  other_weather-0  direct_report-0  \n",
       "4  other_weather-0  direct_report-0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a470067",
   "metadata": {},
   "source": [
    "Next, we'll fix the values of the above dataset so that they are binary. 1 indicates that a message belonging to a given category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "af784265",
   "metadata": {},
   "outputs": [],
   "source": [
    "for column in categories:\n",
    "    # the last character of a value indicates its binary label\n",
    "    categories[column] = categories[column].str[-1].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26bec9a0",
   "metadata": {},
   "source": [
    "The `related` column contains values other than 1 or 2, so we will convert the 2s to 1s by testricting the maximum value of a category as 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "ac745b07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    20042\n",
       "0     6140\n",
       "2      204\n",
       "Name: related, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories['related'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a9640468",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories['related'] = categories['related'].clip(0,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "190bf815",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    20246\n",
       "0     6140\n",
       "Name: related, dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories['related'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836a808d",
   "metadata": {},
   "source": [
    "Let's concatenate the categories dataframe back to our original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "82939f09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the original categories column from `df`\n",
    "df.drop('categories',axis=1,inplace=True)\n",
    "\n",
    "# concatenate the original dataframe with the new `categories` dataframe\n",
    "df = pd.concat([df,categories], axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "aa3ef0dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Weather update - a cold front from Cuba that c...</td>\n",
       "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "      <td>Cyclone nan fini osinon li pa fini</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>says: west side of Haiti, rest of the country ...</td>\n",
       "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                            message  \\\n",
       "0   2  Weather update - a cold front from Cuba that c...   \n",
       "1   7            Is the Hurricane over or is it not over   \n",
       "2   8                    Looking for someone but no name   \n",
       "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
       "4  12  says: west side of Haiti, rest of the country ...   \n",
       "\n",
       "                                            original   genre  related  \\\n",
       "0  Un front froid se retrouve sur Cuba ce matin. ...  direct        1   \n",
       "1                 Cyclone nan fini osinon li pa fini  direct        1   \n",
       "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct        1   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct        1   \n",
       "4  facade ouest d Haiti et le reste du pays aujou...  direct        1   \n",
       "\n",
       "   request  offer  aid_related  medical_help  medical_products  ...  \\\n",
       "0        0      0            0             0                 0  ...   \n",
       "1        0      0            1             0                 0  ...   \n",
       "2        0      0            0             0                 0  ...   \n",
       "3        1      0            1             0                 1  ...   \n",
       "4        0      0            0             0                 0  ...   \n",
       "\n",
       "   aid_centers  other_infrastructure  weather_related  floods  storm  fire  \\\n",
       "0            0                     0                0       0      0     0   \n",
       "1            0                     0                1       0      1     0   \n",
       "2            0                     0                0       0      0     0   \n",
       "3            0                     0                0       0      0     0   \n",
       "4            0                     0                0       0      0     0   \n",
       "\n",
       "   earthquake  cold  other_weather  direct_report  \n",
       "0           0     0              0              0  \n",
       "1           0     0              0              0  \n",
       "2           0     0              0              0  \n",
       "3           0     0              0              0  \n",
       "4           0     0              0              0  \n",
       "\n",
       "[5 rows x 40 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a5efa8",
   "metadata": {},
   "source": [
    "### Removing duplicates and NaN rows, if any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "9d3bb930",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "171"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking for duplicate rows in df\n",
    "df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "49d2aae7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop duplicates\n",
    "df.drop_duplicates(keep='first', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "53df4966",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop NaN messages\n",
    "df.dropna(subset=[\"message\"], axis=0, inplace=True) # drop the row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "07856481",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the id and original columns as they are not useful for the learning problem\n",
    "df.drop([\"id\", \"original\"], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97a68a1",
   "metadata": {},
   "source": [
    "### Saving to a database\n",
    "\n",
    "As part of this project's simulated scenario, we'll use sqlite and save the dataframe to a db."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "220d3a98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26215"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "database_filename = \"disaster_db\"\n",
    "\n",
    "engine = create_engine('sqlite:///'+ database_filename)\n",
    "df.to_sql('messages', engine, if_exists=\"replace\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b83eb05",
   "metadata": {},
   "source": [
    "# 4. Machine Learning Pipeline\n",
    "\n",
    "Let's start by loading the data back form our database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "da77a0f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "import nltk\n",
    "# uncomment below if NLTK data needs to be downloaded\n",
    "# nltk.download(['punkt', 'wordnet'])\n",
    "# nltk.download('stopwords')\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "from sklearn.multioutput import MultiOutputClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "7d9a8813",
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = create_engine('sqlite:///' + database_filename)\n",
    "df = pd.read_sql_query('select * from messages', engine)\n",
    "# the training data is a numpy array of all messages\n",
    "X = df['message'].values\n",
    "# the labels are all the different categories\n",
    "Y = df.drop(columns=['message','genre'], axis=1)\n",
    "category_names = Y.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "41d74b65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26215"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "2b08224c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Weather update - a cold front from Cuba that could pass over Haiti',\n",
       "       'Is the Hurricane over or is it not over',\n",
       "       'Looking for someone but no name',\n",
       "       'UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperately.',\n",
       "       'says: west side of Haiti, rest of the country today and tonight'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "0c8e49a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>search_and_rescue</th>\n",
       "      <th>security</th>\n",
       "      <th>military</th>\n",
       "      <th>child_alone</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   related  request  offer  aid_related  medical_help  medical_products  \\\n",
       "0        1        0      0            0             0                 0   \n",
       "1        1        0      0            1             0                 0   \n",
       "2        1        0      0            0             0                 0   \n",
       "3        1        1      0            1             0                 1   \n",
       "4        1        0      0            0             0                 0   \n",
       "\n",
       "   search_and_rescue  security  military  child_alone  ...  aid_centers  \\\n",
       "0                  0         0         0            0  ...            0   \n",
       "1                  0         0         0            0  ...            0   \n",
       "2                  0         0         0            0  ...            0   \n",
       "3                  0         0         0            0  ...            0   \n",
       "4                  0         0         0            0  ...            0   \n",
       "\n",
       "   other_infrastructure  weather_related  floods  storm  fire  earthquake  \\\n",
       "0                     0                0       0      0     0           0   \n",
       "1                     0                1       0      1     0           0   \n",
       "2                     0                0       0      0     0           0   \n",
       "3                     0                0       0      0     0           0   \n",
       "4                     0                0       0      0     0           0   \n",
       "\n",
       "   cold  other_weather  direct_report  \n",
       "0     0              0              0  \n",
       "1     0              0              0  \n",
       "2     0              0              0  \n",
       "3     0              0              0  \n",
       "4     0              0              0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc519dbe",
   "metadata": {},
   "source": [
    "### Splitting the dataset into train/test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "c99c7052",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "d74eed0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20972 5243\n"
     ]
    }
   ],
   "source": [
    "print(len(X_train), len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ef94ea",
   "metadata": {},
   "source": [
    "### Preprocessing the messages texts\n",
    "\n",
    "For the classifier to perform well, we need to preprocess the disaster message texts to standardize and tokenize them. We can define a function to do this, which can be used later when defining our model pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "3fe3bc48",
   "metadata": {},
   "outputs": [],
   "source": [
    "punctuation_regex = re.compile(r\"[^\\w\\s]\")\n",
    "stopwords = stop_words = stopwords.words('english')\n",
    "wordnet_lemmatizer = WordNetLemmatizer()\n",
    "pos_tags_to_lemmatize = [\"n\", \"v\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "8076981a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"\n",
    "    Tokenizes a given text.\n",
    "    Args:\n",
    "        text: text string\n",
    "    Returns:\n",
    "        tokens: list of tokens\n",
    "    \"\"\"\n",
    "    # lowercase string and remove punctuation\n",
    "    text = punctuation_regex.sub(\" \", text.lower()).strip()\n",
    "    # tokenize text\n",
    "    tokens = [token for token in word_tokenize(text)]\n",
    "    # lemmatize text based on pos tags\n",
    "    for pos_tag in pos_tags_to_lemmatize:\n",
    "        tokens = [wordnet_lemmatizer.lemmatize(token, pos=pos_tag) for token in word_tokenize(text)]\n",
    "    # remove stopwords\n",
    "    tokens = [token for token in tokens if token not in stopwords]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b287b42",
   "metadata": {},
   "source": [
    "### Building the model pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "198a73e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model():\n",
    "    \"\"\"Builds classification model \"\"\"\n",
    "\n",
    "    pipeline = Pipeline([\n",
    "        ('vect', CountVectorizer(tokenizer=tokenize)),\n",
    "        ('tfidf', TfidfTransformer()),\n",
    "        ('clf', MultiOutputClassifier(XGBClassifier(learning_rate=0.1)))\n",
    "    ])\n",
    "\n",
    "    parameters = {\n",
    "        \"clf__estimator__max_depth\": [8, 16],\n",
    "        \"clf__estimator__colsample_bytree\":[0.5, 0.75]\n",
    "    }\n",
    "\n",
    "    cv = GridSearchCV(pipeline, cv=3, param_grid=parameters, n_jobs=1, scoring=\"f1_micro\")\n",
    "    return cv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c50976",
   "metadata": {},
   "source": [
    "### Fitting the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "5f2ec635",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "efde03a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=3,\n",
       "             estimator=Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                                        CountVectorizer(tokenizer=&lt;function tokenize at 0x2b0626dd0&gt;)),\n",
       "                                       (&#x27;tfidf&#x27;, TfidfTransformer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        MultiOutputClassifier(estimator=XGBClassifier(base_score=None,\n",
       "                                                                                      booster=None,\n",
       "                                                                                      callbacks=None,\n",
       "                                                                                      colsample_bylevel=None,\n",
       "                                                                                      colsample_bynode=None,\n",
       "                                                                                      colsample_bytree=None,\n",
       "                                                                                      early_stopping_rounds=None,\n",
       "                                                                                      enable_categorical...\n",
       "                                                                                      max_delta_step=None,\n",
       "                                                                                      max_depth=None,\n",
       "                                                                                      max_leaves=None,\n",
       "                                                                                      min_child_weight=None,\n",
       "                                                                                      missing=nan,\n",
       "                                                                                      monotone_constraints=None,\n",
       "                                                                                      n_estimators=100,\n",
       "                                                                                      n_jobs=None,\n",
       "                                                                                      num_parallel_tree=None,\n",
       "                                                                                      predictor=None,\n",
       "                                                                                      random_state=None,\n",
       "                                                                                      reg_alpha=None,\n",
       "                                                                                      reg_lambda=None, ...)))]),\n",
       "             n_jobs=1,\n",
       "             param_grid={&#x27;clf__estimator__colsample_bytree&#x27;: [0.5, 0.75],\n",
       "                         &#x27;clf__estimator__max_depth&#x27;: [8, 16]},\n",
       "             scoring=&#x27;f1_micro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=3,\n",
       "             estimator=Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                                        CountVectorizer(tokenizer=&lt;function tokenize at 0x2b0626dd0&gt;)),\n",
       "                                       (&#x27;tfidf&#x27;, TfidfTransformer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        MultiOutputClassifier(estimator=XGBClassifier(base_score=None,\n",
       "                                                                                      booster=None,\n",
       "                                                                                      callbacks=None,\n",
       "                                                                                      colsample_bylevel=None,\n",
       "                                                                                      colsample_bynode=None,\n",
       "                                                                                      colsample_bytree=None,\n",
       "                                                                                      early_stopping_rounds=None,\n",
       "                                                                                      enable_categorical...\n",
       "                                                                                      max_delta_step=None,\n",
       "                                                                                      max_depth=None,\n",
       "                                                                                      max_leaves=None,\n",
       "                                                                                      min_child_weight=None,\n",
       "                                                                                      missing=nan,\n",
       "                                                                                      monotone_constraints=None,\n",
       "                                                                                      n_estimators=100,\n",
       "                                                                                      n_jobs=None,\n",
       "                                                                                      num_parallel_tree=None,\n",
       "                                                                                      predictor=None,\n",
       "                                                                                      random_state=None,\n",
       "                                                                                      reg_alpha=None,\n",
       "                                                                                      reg_lambda=None, ...)))]),\n",
       "             n_jobs=1,\n",
       "             param_grid={&#x27;clf__estimator__colsample_bytree&#x27;: [0.5, 0.75],\n",
       "                         &#x27;clf__estimator__max_depth&#x27;: [8, 16]},\n",
       "             scoring=&#x27;f1_micro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 CountVectorizer(tokenizer=&lt;function tokenize at 0x2b0626dd0&gt;)),\n",
       "                (&#x27;tfidf&#x27;, TfidfTransformer()),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 MultiOutputClassifier(estimator=XGBClassifier(base_score=None,\n",
       "                                                               booster=None,\n",
       "                                                               callbacks=None,\n",
       "                                                               colsample_bylevel=None,\n",
       "                                                               colsample_bynode=None,\n",
       "                                                               colsample_bytree=None,\n",
       "                                                               early_stopping_rounds=None,\n",
       "                                                               enable_categorical=False,\n",
       "                                                               eval_metric=None,\n",
       "                                                               gamm...\n",
       "                                                               grow_policy=None,\n",
       "                                                               importance_type=None,\n",
       "                                                               interaction_constraints=None,\n",
       "                                                               learning_rate=0.1,\n",
       "                                                               max_bin=None,\n",
       "                                                               max_cat_to_onehot=None,\n",
       "                                                               max_delta_step=None,\n",
       "                                                               max_depth=None,\n",
       "                                                               max_leaves=None,\n",
       "                                                               min_child_weight=None,\n",
       "                                                               missing=nan,\n",
       "                                                               monotone_constraints=None,\n",
       "                                                               n_estimators=100,\n",
       "                                                               n_jobs=None,\n",
       "                                                               num_parallel_tree=None,\n",
       "                                                               predictor=None,\n",
       "                                                               random_state=None,\n",
       "                                                               reg_alpha=None,\n",
       "                                                               reg_lambda=None, ...)))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer(tokenizer=&lt;function tokenize at 0x2b0626dd0&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfTransformer</label><div class=\"sk-toggleable__content\"><pre>TfidfTransformer()</pre></div></div></div><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">clf: MultiOutputClassifier</label><div class=\"sk-toggleable__content\"><pre>MultiOutputClassifier(estimator=XGBClassifier(base_score=None, booster=None,\n",
       "                                              callbacks=None,\n",
       "                                              colsample_bylevel=None,\n",
       "                                              colsample_bynode=None,\n",
       "                                              colsample_bytree=None,\n",
       "                                              early_stopping_rounds=None,\n",
       "                                              enable_categorical=False,\n",
       "                                              eval_metric=None, gamma=None,\n",
       "                                              gpu_id=None, grow_policy=None,\n",
       "                                              importance_type=None,\n",
       "                                              interaction_constraints=None,\n",
       "                                              learning_rate=0.1, max_bin=None,\n",
       "                                              max_cat_to_onehot=None,\n",
       "                                              max_delta_step=None,\n",
       "                                              max_depth=None, max_leaves=None,\n",
       "                                              min_child_weight=None,\n",
       "                                              missing=nan,\n",
       "                                              monotone_constraints=None,\n",
       "                                              n_estimators=100, n_jobs=None,\n",
       "                                              num_parallel_tree=None,\n",
       "                                              predictor=None, random_state=None,\n",
       "                                              reg_alpha=None, reg_lambda=None, ...))</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: XGBClassifier</label><div class=\"sk-toggleable__content\"><pre>XGBClassifier(base_score=None, booster=None, callbacks=None,\n",
       "              colsample_bylevel=None, colsample_bynode=None,\n",
       "              colsample_bytree=None, early_stopping_rounds=None,\n",
       "              enable_categorical=False, eval_metric=None, gamma=None,\n",
       "              gpu_id=None, grow_policy=None, importance_type=None,\n",
       "              interaction_constraints=None, learning_rate=0.1, max_bin=None,\n",
       "              max_cat_to_onehot=None, max_delta_step=None, max_depth=None,\n",
       "              max_leaves=None, min_child_weight=None, missing=nan,\n",
       "              monotone_constraints=None, n_estimators=100, n_jobs=None,\n",
       "              num_parallel_tree=None, predictor=None, random_state=None,\n",
       "              reg_alpha=None, reg_lambda=None, ...)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">XGBClassifier</label><div class=\"sk-toggleable__content\"><pre>XGBClassifier(base_score=None, booster=None, callbacks=None,\n",
       "              colsample_bylevel=None, colsample_bynode=None,\n",
       "              colsample_bytree=None, early_stopping_rounds=None,\n",
       "              enable_categorical=False, eval_metric=None, gamma=None,\n",
       "              gpu_id=None, grow_policy=None, importance_type=None,\n",
       "              interaction_constraints=None, learning_rate=0.1, max_bin=None,\n",
       "              max_cat_to_onehot=None, max_delta_step=None, max_depth=None,\n",
       "              max_leaves=None, min_child_weight=None, missing=nan,\n",
       "              monotone_constraints=None, n_estimators=100, n_jobs=None,\n",
       "              num_parallel_tree=None, predictor=None, random_state=None,\n",
       "              reg_alpha=None, reg_lambda=None, ...)</pre></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=3,\n",
       "             estimator=Pipeline(steps=[('vect',\n",
       "                                        CountVectorizer(tokenizer=<function tokenize at 0x2b0626dd0>)),\n",
       "                                       ('tfidf', TfidfTransformer()),\n",
       "                                       ('clf',\n",
       "                                        MultiOutputClassifier(estimator=XGBClassifier(base_score=None,\n",
       "                                                                                      booster=None,\n",
       "                                                                                      callbacks=None,\n",
       "                                                                                      colsample_bylevel=None,\n",
       "                                                                                      colsample_bynode=None,\n",
       "                                                                                      colsample_bytree=None,\n",
       "                                                                                      early_stopping_rounds=None,\n",
       "                                                                                      enable_categorical...\n",
       "                                                                                      max_delta_step=None,\n",
       "                                                                                      max_depth=None,\n",
       "                                                                                      max_leaves=None,\n",
       "                                                                                      min_child_weight=None,\n",
       "                                                                                      missing=nan,\n",
       "                                                                                      monotone_constraints=None,\n",
       "                                                                                      n_estimators=100,\n",
       "                                                                                      n_jobs=None,\n",
       "                                                                                      num_parallel_tree=None,\n",
       "                                                                                      predictor=None,\n",
       "                                                                                      random_state=None,\n",
       "                                                                                      reg_alpha=None,\n",
       "                                                                                      reg_lambda=None, ...)))]),\n",
       "             n_jobs=1,\n",
       "             param_grid={'clf__estimator__colsample_bytree': [0.5, 0.75],\n",
       "                         'clf__estimator__max_depth': [8, 16]},\n",
       "             scoring='f1_micro')"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99a13186",
   "metadata": {},
   "source": [
    "### Evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "1704888e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        precision    recall  f1-score   support\n",
      "\n",
      "               related       0.96      0.82      0.89      4707\n",
      "               request       0.55      0.79      0.65       622\n",
      "                 offer       0.00      0.00      0.00         0\n",
      "           aid_related       0.66      0.77      0.71      1881\n",
      "          medical_help       0.25      0.59      0.35       186\n",
      "      medical_products       0.27      0.64      0.38       114\n",
      "     search_and_rescue       0.18      0.72      0.29        40\n",
      "              security       0.03      0.33      0.06         9\n",
      "              military       0.32      0.57      0.41        95\n",
      "           child_alone       0.00      0.00      0.00         0\n",
      "                 water       0.71      0.82      0.76       294\n",
      "                  food       0.76      0.80      0.78       579\n",
      "               shelter       0.60      0.75      0.67       375\n",
      "              clothing       0.48      0.73      0.58        51\n",
      "                 money       0.22      0.69      0.34        42\n",
      "        missing_people       0.13      0.80      0.23        10\n",
      "              refugees       0.24      0.66      0.35        62\n",
      "                 death       0.46      0.79      0.58       136\n",
      "             other_aid       0.15      0.66      0.25       158\n",
      "infrastructure_related       0.05      0.40      0.09        43\n",
      "             transport       0.25      0.72      0.37        85\n",
      "             buildings       0.38      0.72      0.49       137\n",
      "           electricity       0.26      0.71      0.38        41\n",
      "                 tools       0.00      0.00      0.00         0\n",
      "             hospitals       0.05      0.75      0.10         4\n",
      "                 shops       0.00      0.00      0.00         0\n",
      "           aid_centers       0.04      0.29      0.06         7\n",
      "  other_infrastructure       0.03      0.32      0.05        19\n",
      "       weather_related       0.71      0.85      0.77      1206\n",
      "                floods       0.58      0.87      0.70       289\n",
      "                 storm       0.63      0.73      0.68       416\n",
      "                  fire       0.28      0.75      0.41        20\n",
      "            earthquake       0.81      0.88      0.84       446\n",
      "                  cold       0.38      0.71      0.49        56\n",
      "         other_weather       0.12      0.39      0.19        76\n",
      "         direct_report       0.45      0.73      0.56       624\n",
      "\n",
      "             micro avg       0.61      0.79      0.69     12830\n",
      "             macro avg       0.33      0.60      0.40     12830\n",
      "          weighted avg       0.72      0.79      0.74     12830\n",
      "           samples avg       0.53      0.66      0.54     12830\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sajal/.pyenv/versions/3.10.0/envs/disaster-message-classifier/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/sajal/.pyenv/versions/3.10.0/envs/disaster-message-classifier/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/sajal/.pyenv/versions/3.10.0/envs/disaster-message-classifier/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in samples with no predicted labels. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/sajal/.pyenv/versions/3.10.0/envs/disaster-message-classifier/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in samples with no true labels. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "y_preds = model.predict(X_test)\n",
    "print(classification_report(y_preds, y_test.values, target_names=category_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "8f3c21c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "related                   0.813656\n",
      "request                   0.898722\n",
      "offer                     0.995232\n",
      "aid_related               0.774747\n",
      "medical_help              0.923899\n",
      "medical_products          0.955369\n",
      "search_and_rescue         0.973107\n",
      "security                  0.982071\n",
      "military                  0.970437\n",
      "child_alone               1.000000\n",
      "water                     0.970818\n",
      "food                      0.949647\n",
      "shelter                   0.946786\n",
      "clothing                  0.989701\n",
      "money                     0.978257\n",
      "missing_people            0.989701\n",
      "refugees                  0.970818\n",
      "death                     0.970818\n",
      "other_aid                 0.881556\n",
      "infrastructure_related    0.935342\n",
      "transport                 0.959756\n",
      "buildings                 0.961472\n",
      "electricity               0.982262\n",
      "tools                     0.994278\n",
      "hospitals                 0.989319\n",
      "shops                     0.995804\n",
      "aid_centers               0.988747\n",
      "other_infrastructure      0.956132\n",
      "weather_related           0.885180\n",
      "floods                    0.957849\n",
      "storm                     0.944688\n",
      "fire                      0.991799\n",
      "earthquake                0.972344\n",
      "cold                      0.984360\n",
      "other_weather             0.949838\n",
      "direct_report             0.861911\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# collect accuracy scores in a dict\n",
    "category_name_2_accuracy_score = {}\n",
    "for i in range(len(category_names)):\n",
    "    category_name_2_accuracy_score[y_test.columns[i]] = accuracy_score(y_test.values[:,i],y_preds[:,i])\n",
    "print(pd.Series(category_name_2_accuracy_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be303d7",
   "metadata": {},
   "source": [
    "This is an imbalanced dataset, as most categories for a given message will be 0. The f1-score is a better metric of the model's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e2ab629",
   "metadata": {},
   "source": [
    "### Predicting categories for new messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "1169ca61",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text):\n",
    "    \"\"\"Returns a list of predicted categories for the given text\"\"\"\n",
    "    preds = model.predict([text])\n",
    "    predicted_categories = [category for i, category in enumerate(y_test.columns) if preds[0][i] == 1]\n",
    "    return predicted_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "765296ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['related',\n",
       " 'request',\n",
       " 'aid_related',\n",
       " 'food',\n",
       " 'shelter',\n",
       " 'weather_related',\n",
       " 'floods',\n",
       " 'direct_report']"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(\"after the floods in our area we are trapped. we need food and shelter \")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dba0ac",
   "metadata": {},
   "source": [
    "# 5. Further Improvements\n",
    "\n",
    "\n",
    "This notebook sticks to the basics in order to provide a good baseline model for this classification task. There is plenty of room for improvement here, including but not limited to:\n",
    "\n",
    "1. Using word embeddings (GloVe, word2vec) or even sentence embeddings (Univsersal Sentgence Encoder) to transform message text, instead of using a CountVectorizer. This should allow the model to generalize well to similar/unseen words and improve the model accuracy too.\n",
    "2. Some of the category column values like `related` and `child_alone` are highly skewed. We can look at adding more data for these categories.\n",
    "3. A Neural Network can be trained for this task. Better yet, a pre-trained state-of-the-art transformer based network can be fine tuned using the data available to us.\n",
    "4. Certain classification categories are \"noisy\" such as \"related\" or \"child alone\" (no positive examples). These can either be removed, or more data can be acquired that provides enough missing examples for such cases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a6ab476",
   "metadata": {},
   "source": [
    "# 6. Summary\n",
    "\n",
    "In this project, we built an ETL pipeline to load messy and unsuitable-for-training data, clean and transform it, and save it to a database. We also built a ML pipeline to tokenize message text, and train an XGBoost classifier to classify messages into different categories. \n",
    "\n",
    "We stuck to the basics for building this classifier, and there's plenty of room for improvement in the future using modern NLP architectures."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}