FML1w7_clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering\n",
    "#### Part of the course on \"Foundations of machine learning\", Department of Mathematics and Statistics, University of Turku, Finland\n",
    "#### Lectures available on YouTube: https://youtube.com/playlist?list=PLbkSohdmxoVAZ9DEHEWHjeGK7Ei-DjKHI&si=Msu74_I0qhLrRWcu\n",
    "#### Code available on GitHub: https://github.com/ionpetre/FoundML_course_assignments\n",
    "\n",
    "#### This notebook is based on the following sources: \n",
    "> https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html#sphx-glr-auto-examples-cluster-plot-cluster-comparison-py\n",
    "\n",
    "> https://www.kaggle.com/code/karnikakapoor/customer-segmentation-clustering (KARNIKA KAPOOR)\n",
    "\n",
    "> https://www.kaggle.com/code/mihirjhaveri/wholesale-customer-retail-uci (MIHIR JHAVERI)\n",
    "\n",
    "> https://www.kaggle.com/code/kautumn06/yellowbrick-clustering-evaluation-examples (KRISTENMCINTYRE)\n",
    "\n",
    "Clustering is a fundamental unsupervised learning technique used to group similar data points together based on their intrinsic characteristics. The objective of clustering is to identify patterns and structures within the dataset without any predefined labels. The algorithm segregates the data into distinct clusters, with data points within a cluster sharing common traits and features. This grouping allows for a better understanding of the underlying relationships and similarities in the data, aiding in data analysis, pattern recognition, and decision-making. Clustering algorithms, such as Gaussian mixture models, K-means, agglomerative clustering, hierarchical clustering, and DBSCAN, play a crucial role in various applications, including customer segmentation, image recognition, anomaly detection, and recommendation systems. Effective clustering helps reveal hidden insights and patterns within large datasets, contributing to more informed business strategies and improved model performance in a wide array of real-world applications.\n",
    "\n",
    "We demonstrate in this notebook the following methods:\n",
    " 1. GMM\n",
    " 2. K-means\n",
    " 3. Agglomerative clustering\n",
    " 4. Hierarchical clustering\n",
    " 5. DBSCAN\n",
    "\n",
    "We first demo these methods using a labeled dataset with a known number of desired clusters so that we can see the quality of the clustering. We then demo them on an unlabled dataset where the optimal number of clusters is to be determined and their significance identified. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:11.060373Z",
     "iopub.status.busy": "2021-09-09T17:34:11.059927Z",
     "iopub.status.idle": "2021-09-09T17:34:12.616363Z",
     "shell.execute_reply": "2021-09-09T17:34:12.615487Z",
     "shell.execute_reply.started": "2021-09-09T17:34:11.06034Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import seaborn as sns\n",
    "import matplotlib.colors as mcolors\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN\n",
    "from sklearn.mixture import GaussianMixture\n",
    "from sklearn.metrics import homogeneity_score, completeness_score, v_measure_score, adjusted_rand_score\n",
    "from sklearn.metrics import adjusted_mutual_info_score, silhouette_score "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the seed of the random number generator, for reproducibility purposes\n",
    "\n",
    "import os\n",
    "\n",
    "def reset_seed(SEED = 0):\n",
    "    \"\"\"Reset the seed for every random library in use (System, numpy)\"\"\"\n",
    "\n",
    "    os.environ['PYTHONHASHSEED']=str(SEED)\n",
    "    np.random.seed(SEED)\n",
    "\n",
    "\n",
    "reset_seed(150)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the style of the plots\n",
    "\n",
    "plt.style.use(\"seaborn-whitegrid\")\n",
    "plt.rc(\"figure\", autolayout=True)\n",
    "plt.rc(\n",
    "    \"axes\",\n",
    "    labelweight=\"bold\",\n",
    "    labelsize=\"large\",\n",
    "    titleweight=\"bold\",\n",
    "    titlesize=14,\n",
    "    titlepad=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Demo clustering on the Iris dataset (labels known, number of clusters known)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Iris Dataset: https://en.wikipedia.org/wiki/Iris_flower_data_set\n",
    "\n",
    "This dataset consists of data collected on 150 Iris flowers, 50 from each of three types: Setosa, Versicolour, and Virginica. For each flower in the dataset we have its Iris type, its petal and sepal\n",
    "length and width. The goal of this assignment is to create a model that learns the type of Iris based on its petal and sepal length and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Iris dataset from the sklearn library. \n",
    "# The organisation of the data in the sklearn library is described at https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html. \n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "iris_X = load_iris(return_X_y=False, as_frame=True)['frame']\n",
    "\n",
    "#Check the dataset\n",
    "print(iris_X.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_X.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualise the data in 2D, color the datapoints depending on their labels. This suggests the clusters we look for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize dataset for easier parameter selection\n",
    "iris_X[['sepal length (cm)',\n",
    "       'sepal width (cm)',\n",
    "       'petal length (cm)',\n",
    "       'petal width (cm)']] = pd.DataFrame(\n",
    "    StandardScaler().fit_transform(iris_X[['sepal length (cm)',\n",
    "                                           'sepal width (cm)',\n",
    "                                           'petal length (cm)',\n",
    "                                           'petal width (cm)']]), \n",
    "    columns=['sepal length (cm)',\n",
    "             'sepal width (cm)',\n",
    "             'petal length (cm)',\n",
    "             'petal width (cm)']\n",
    ")\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15,5))\n",
    "true_colors = np.array(['red', 'green', 'blue'])  # map categorical values to colors\n",
    "\n",
    "# Plot the dataset in 2D based on its sepal characteristics\n",
    "\n",
    "iris_X.plot(kind=\"scatter\", \n",
    "            x=\"sepal length (cm)\", \n",
    "            y=\"sepal width (cm)\", \n",
    "            color=true_colors[iris_X['target']],\n",
    "            title = 'Sepal characteristics',\n",
    "            ax = axes [0],\n",
    "            #figsize=(5,3),\n",
    "           )\n",
    "\n",
    "# Plot the dataset in 2D based on its petal characteristics\n",
    "\n",
    "iris_X.plot(kind=\"scatter\", \n",
    "            x=\"petal length (cm)\", \n",
    "            y=\"petal width (cm)\", \n",
    "            color=true_colors[iris_X['target']],\n",
    "            title = 'Petal characteristics',\n",
    "            ax = axes [1],\n",
    "            #figsize=(5,3),\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**As we knew from other assignments, the petal characteristics provide a much better separation of the data**\n",
    "\n",
    "Our clustering algorithms will separate the data using all features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run on the Iris data, without the target values (finding the clusters is the objective here)\n",
    "X = iris_X[['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============\n",
    "# Create the cluster objects\n",
    "# ============\n",
    "\n",
    "three_means = KMeans(\n",
    "    n_clusters=3,\n",
    "    n_init=10,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state=2023,\n",
    ")\n",
    "\n",
    "agg_ward = AgglomerativeClustering(\n",
    "    n_clusters=3, \n",
    "    linkage=\"ward\", # ‘ward’ minimizes the variance of the clusters being merged.\n",
    "    metric = 'euclidean'\n",
    ")\n",
    "\n",
    "agg_average = AgglomerativeClustering(\n",
    "    n_clusters=3,\n",
    "    linkage=\"average\", # ‘average’ uses the average of the distances of each observation of the two sets.\n",
    "    metric=\"euclidean\",\n",
    ")\n",
    "\n",
    "'''\n",
    "    DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is \n",
    "    a popular density-based clustering algorithm used in data mining and machine learning. \n",
    "    It groups together data points that are close to each other in space, \n",
    "    identifying clusters based on their density and separating noise or outliers.\n",
    "    It identifies itself the optimal number of clusters for a dataset. \n",
    "\n",
    "'''\n",
    "\n",
    "dbscan = DBSCAN(\n",
    "    eps=0.5,\n",
    "    min_samples = 5,\n",
    "    metric=\"euclidean\",\n",
    "    n_jobs = -1,\n",
    ")\n",
    "\n",
    "gmm = GaussianMixture(\n",
    "    n_components=3,\n",
    "    covariance_type=\"full\",\n",
    "    random_state=2023,\n",
    "    max_iter = 1000,\n",
    "    tol = 1e-3, # The convergence threshold. \n",
    "                # EM iterations will stop when the lower bound average gain is below this threshold.\n",
    ")\n",
    "\n",
    "clustering_algorithms = (\n",
    "    (\"KMeans\", three_means),\n",
    "    (\"Agglomerative Ward\", agg_ward),\n",
    "    (\"Agglomerative Average\", agg_average),\n",
    "    (\"DBSCAN\", dbscan),\n",
    "    (\"Gaussian Mixture\", gmm),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = np.array(['orange', 'olive', 'purple'])  # map categorical values to colors\n",
    "\n",
    "\n",
    "fig, axes = plt.subplots(nrows=10, ncols=2, figsize=(15,70))\n",
    "\n",
    "plot_num = 0\n",
    "\n",
    "for name, algorithm in clustering_algorithms:\n",
    "    t0 = time.time()\n",
    "    algorithm.fit(X)\n",
    "    t1 = time.time()\n",
    "    \n",
    "    if hasattr(algorithm, \"labels_\"):\n",
    "        y_pred = algorithm.labels_.astype(int)\n",
    "    else:\n",
    "        y_pred = algorithm.predict(X)\n",
    "\n",
    "    iris_X.plot(kind=\"scatter\", \n",
    "                x=\"sepal length (cm)\", \n",
    "                y=\"sepal width (cm)\", \n",
    "                color=colors[y_pred],\n",
    "                title = name,\n",
    "                ax=axes[plot_num,0],\n",
    "                #figsize=(8,15),\n",
    "               )\n",
    "    \n",
    "    iris_X.plot(kind=\"scatter\", \n",
    "                x=\"sepal length (cm)\", \n",
    "                y=\"sepal width (cm)\", \n",
    "                color=true_colors[iris_X['target']],\n",
    "                title = 'True labels',\n",
    "                ax = axes [plot_num,1],\n",
    "                #figsize=(5,3),\n",
    "               )\n",
    "\n",
    "    plot_num += 1\n",
    "    \n",
    "    iris_X.plot(kind=\"scatter\", \n",
    "                x=\"petal length (cm)\", \n",
    "                y=\"petal width (cm)\", \n",
    "                color=colors[y_pred],\n",
    "                title = name,\n",
    "                ax=axes[plot_num,0],\n",
    "                #figsize=(8,15),\n",
    "               )\n",
    "    \n",
    "    iris_X.plot(kind=\"scatter\", \n",
    "            x=\"petal length (cm)\", \n",
    "            y=\"petal width (cm)\", \n",
    "            color=true_colors[iris_X['target']],\n",
    "            title = 'True labels',\n",
    "                ax=axes[plot_num,1],\n",
    "            #figsize=(5,3),\n",
    "           )\n",
    "    \n",
    "    plot_num += 1\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE.** AggloemrativeAverage and DBSCAN seem a little worse on this dataset than the other methods. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Since we know the true labels for the Iris dataset, we can calculate several different metrics for the quality of the clustering results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_scores = pd.DataFrame(np.nan, \n",
    "                  index=['KMeans',\n",
    "                         'Agglomerative Ward',\n",
    "                         'Agglomerative Average',\n",
    "                         'DBSCAN',\n",
    "                         'Gaussian Mixture'\n",
    "                        ], \n",
    "                  columns=['Homogeneity', \n",
    "                           'Completeness',\n",
    "                           'V-measure',\n",
    "                           'Adjusted Rand Index',\n",
    "                           'Adjusted Mutual Information',\n",
    "                           'Silhouette Coefficient',\n",
    "                          ]\n",
    "                 )\n",
    "\n",
    "for name, algorithm in clustering_algorithms:\n",
    "    if hasattr(algorithm, \"labels_\"):\n",
    "        y_pred = algorithm.labels_.astype(int)\n",
    "    else:\n",
    "        y_pred = algorithm.predict(X)\n",
    "        \n",
    "    cluster_scores.loc[name] = [\n",
    "        homogeneity_score(iris_X['target'], y_pred),\n",
    "        completeness_score(iris_X['target'],  y_pred),\n",
    "        v_measure_score(iris_X['target'],y_pred),\n",
    "        adjusted_rand_score(iris_X['target'], y_pred),\n",
    "        adjusted_mutual_info_score(iris_X['target'], y_pred),\n",
    "        silhouette_score(X, y_pred)                                \n",
    "    ]    \n",
    "    \n",
    "cluster_scores.style.highlight_max(color = 'lightgreen', axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion: GaussianMixture outperforms the other methods on this dataset. On the silhouette coefficient, agglomerative average does better. \n",
    "Note: on dataset with unknown labels, the silhouette coefficients is the only one of these metrics that we can use to measure the quality of the clustering results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Demo clustering on a dataset with unknwown labels, unknown number of clusters\n",
    "\n",
    "### We use the UCI Wholesale customers dataset https://archive.ics.uci.edu/dataset/292/wholesale+customers\n",
    "\n",
    "Features: \n",
    "\n",
    "1)\tFRESH: annual spending (m.u.) on fresh products (Continuous);\n",
    "2)\tMILK: annual spending (m.u.) on milk products (Continuous);\n",
    "3)\tGROCERY: annual spending (m.u.)on grocery products (Continuous);\n",
    "4)\tFROZEN: annual spending (m.u.)on frozen products (Continuous)\n",
    "5)\tDETERGENTS_PAPER: annual spending (m.u.) on detergents and paper products (Continuous) \n",
    "6)\tDELICATESSEN: annual spending (m.u.)on and delicatessen products (Continuous); \n",
    "7)\tCHANNEL: customersâ€™ Channel - Horeca (Hotel/Restaurant/CafÃ©) or Retail channel (Nominal)\n",
    "8)\tREGION: customersâ€™ Region â€“ Lisnon, Oporto or Other (Nominal)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "wholesale_X, wholesale_y = fetch_openml(\n",
    "    data_id=1511,\n",
    "    as_frame=True,\n",
    "    return_X_y=True,\n",
    "    parser = 'auto'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wholesale_y.value_counts()\n",
    "\n",
    "# 1 encodes 'hotel' customers and 2 encodes 'retail' customers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wholesale_X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The data seems to have had its columns renamed in OpenML. We bring them to their original format. \n",
    "\n",
    "wholesale_X.rename(\n",
    "    columns={\"V2\": \"Fresh\", \n",
    "             \"V3\": \"Milk\",\n",
    "             \"V4\": \"Grocery\",\n",
    "             \"V5\": \"Frozen\",\n",
    "             \"V6\": \"Detergents_Paper\",\n",
    "             \"V7\": \"Delicatessen\",\n",
    "            },\n",
    "    inplace = True,\n",
    ")\n",
    "\n",
    "# Add the channels to the dataset\n",
    "wholesale_X = pd.concat([wholesale_X, wholesale_y.to_frame()], axis=1)\n",
    "\n",
    "# V1 is identical to \"Region\" and we drop it\n",
    "wholesale_X.drop(['V1'], axis=1, inplace=True)\n",
    "\n",
    "wholesale_X.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: no missing values!** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the data without the region and the channel\n",
    "X = wholesale_X[[\"Fresh\", \"Milk\",\"Grocery\", \"Frozen\", \"Detergents_Paper\", \"Delicatessen\"]]\n",
    "\n",
    "# normalize dataset for easier parameter selection\n",
    "X = pd.DataFrame(StandardScaler().fit_transform(X), columns=X.columns)\n",
    "X.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualise each data category per channel (hotel/retail) and per region (three of them)\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Fresh\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Item - Fresh')\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Milk\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Item - Milk')\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Grocery\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Item - Grocery')\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Frozen\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Item - Frozen')\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Detergents_Paper\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Item - Detergents_Paper')\n",
    "\n",
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "sns.catplot(x=\"Channel\", y=\"Delicatessen\", hue =\"Region\", kind=\"bar\", errorbar=None, data=wholesale_X)\n",
    "plt.title('Delicatessen')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============\n",
    "# Create the cluster objects\n",
    "# ============\n",
    "\n",
    "\n",
    "k_means = KMeans(\n",
    "    n_init=10,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state=2023,\n",
    ")\n",
    "\n",
    "agg_ward = AgglomerativeClustering(\n",
    "    linkage=\"ward\", # ‘ward’ minimizes the variance of the clusters being merged.\n",
    "    metric = 'euclidean',\n",
    ")\n",
    "\n",
    "agg_average = AgglomerativeClustering(\n",
    "    linkage=\"average\", # ‘average’ uses the average of the distances of each observation of the two sets.\n",
    "    metric=\"euclidean\",\n",
    ")\n",
    "\n",
    "\n",
    "# The GMM is a little problematic in this context. It needs a small hack, done with this weird code below. \n",
    "from sklearn.base import ClusterMixin\n",
    "class GaussianMixtureCluster(GaussianMixture, ClusterMixin):\n",
    "    \"\"\"Subclass of GaussianMixture to make it a ClusterMixin.\"\"\"\n",
    "\n",
    "    def fit(self, X):\n",
    "        super().fit(X)\n",
    "        self.labels_ = self.predict(X)\n",
    "        return self\n",
    "\n",
    "    def get_params(self, **kwargs):\n",
    "        output = super().get_params(**kwargs)\n",
    "        output[\"n_clusters\"] = output.get(\"n_components\", None)\n",
    "        return output\n",
    "\n",
    "    def set_params(self, **kwargs):\n",
    "        kwargs[\"n_components\"] = kwargs.pop(\"n_clusters\", None)\n",
    "        return super().set_params(**kwargs)\n",
    "\n",
    "\n",
    "\n",
    "gmm = GaussianMixtureCluster(\n",
    "    covariance_type=\"full\",\n",
    "    random_state=2023,\n",
    "    max_iter = 1000,\n",
    "    tol = 1e-3, # The convergence threshold. \n",
    "                # EM iterations will stop when the lower bound average gain is below this threshold.\n",
    ")\n",
    "\n",
    "clustering_algorithms = (\n",
    "    (\"KMeans\", k_means),\n",
    "    (\"Agglomerative Ward\", agg_ward),\n",
    "    (\"Agglomerative Average\", agg_average),\n",
    "    (\"Gaussian Mixture\", gmm),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Maximize the silhouette score to find the optimal numbers of clusters.\n",
    "The silhouette coefficient/score is a metric used to calculate the goodness of a clustering technique. Its value ranges from -1 to 1.\n",
    "- 1: clusters are well apart from each other and clearly distinguished.\n",
    "- 0: clusters are indifferent, or we can say that the distance between clusters is not significant.\n",
    "- -1: clusters are assigned in the wrong way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yellowbrick.cluster import KElbowVisualizer\n",
    "\n",
    "\n",
    "for name, algorithm in clustering_algorithms:\n",
    "    visualizer = KElbowVisualizer(algorithm, \n",
    "                              k = (2,11), \n",
    "                              metric = 'silhouette', \n",
    "                              timings = False,\n",
    "                              locate_elbow = False,\n",
    "                              force_model=True,\n",
    "                             )\n",
    "\n",
    "    visualizer.fit(X)    # Fit the data to the visualizer\n",
    "    visualizer.show()    # Draw the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbscan = DBSCAN(\n",
    "    eps=0.5,\n",
    "    min_samples = 5,\n",
    "    metric=\"euclidean\",\n",
    "    p=2,\n",
    "    n_jobs = -1,\n",
    ")\n",
    "\n",
    "dbscan.fit(X)\n",
    "unique, counts = np.unique(dbscan.labels_, return_counts=True)\n",
    "print(np.asarray((unique, counts)).T)\n",
    "print(\"Number of clusters (without the outliers) identified by DBSCAN:\", \n",
    "      np.sum(np.array(unique) >= 0, axis=0)\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The algorithms agree in their suggestions for the optimal number of clusters: 2. \n",
    "\n",
    "> The next step would be to validate these clusters, either by evaluating some quality merics (say their silhouette score profile), or by identifying their meaning somehow. This is often difficult and it requirers domain-specific knowledge about the dataset.\n",
    "\n",
    "> In this notebook instead, we will only visualize the clusters to see how they differ. \n",
    "\n",
    "> Our data is multi-dimensional, which makes visualising it difficult. We will use PCA to project it onto the first 2 principal components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X_2comp = pd.DataFrame(\n",
    "    PCA(n_components = 2, random_state = 2023).fit_transform(X),\n",
    "    columns = ['PC1', 'PC2']\n",
    ")\n",
    "\n",
    "sns.relplot(\n",
    "    x=\"PC1\", \n",
    "    y=\"PC2\", \n",
    "    data=X_2comp, \n",
    "    height=6,\n",
    "    palette = sns.color_palette(palette = 'tab10'),\n",
    ").fig.suptitle(\"UCI Wholesale\", fontsize=14)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_means = KMeans(\n",
    "    n_clusters=2,\n",
    "    n_init=10,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state=2023,\n",
    ")\n",
    "\n",
    "agg_ward = AgglomerativeClustering(\n",
    "    n_clusters=2, \n",
    "    linkage=\"ward\", # ‘ward’ minimizes the variance of the clusters being merged.\n",
    "    metric = 'euclidean',\n",
    ")\n",
    "\n",
    "agg_average = AgglomerativeClustering(\n",
    "    n_clusters=2,\n",
    "    linkage=\"average\", # ‘average’ uses the average of the distances of each observation of the two sets.\n",
    "    metric=\"euclidean\",\n",
    ")\n",
    "\n",
    "\n",
    "gmm = GaussianMixture(\n",
    "    n_components=2,\n",
    "    covariance_type=\"full\",\n",
    "    random_state=2023,\n",
    "    max_iter = 1000,\n",
    "    tol = 1e-3, # The convergence threshold. \n",
    "                # EM iterations will stop when the lower bound average gain is below this threshold.\n",
    ")\n",
    "\n",
    "optimal_clustering_algorithms = (\n",
    "    (\"KMeans\", k_means),\n",
    "    (\"Agglomerative Ward\", agg_ward),\n",
    "    (\"Agglomerative Average\", agg_average),\n",
    "    (\"Gaussian Mixture\", gmm),\n",
    "    (\"DBSCAN\", dbscan)\n",
    ")\n",
    "\n",
    "\n",
    "for name, model in optimal_clustering_algorithms:\n",
    "    model.fit(X)    # Fit the data to the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "for name, model in optimal_clustering_algorithms:\n",
    "    print(name)\n",
    "    if hasattr(model, \"labels_\"):\n",
    "        y_pred = model.labels_.astype(int)\n",
    "    else:\n",
    "        y_pred = model.predict(X)\n",
    "        \n",
    "    X_2comp['cluster'] = y_pred.reshape(-1,1)\n",
    "\n",
    "    sns.relplot(\n",
    "        x=\"PC1\", \n",
    "        y=\"PC2\", \n",
    "        hue=\"cluster\", \n",
    "        data=X_2comp, \n",
    "        height=6,\n",
    "        palette = sns.color_palette(palette = 'tab10'),\n",
    "    ).fig.suptitle(name, fontsize=12)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge: cluster the California housing dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset from sklearn, add the target to the main dataset\n",
    "\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "calif_X, calif_y = fetch_california_housing(return_X_y=True, as_frame=True)\n",
    "\n",
    "calif_X = pd.concat([calif_X, calif_y.to_frame()], axis=1)\n",
    "del calif_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q1: How many features do you have in the dataset? \n",
    "#### Q2: How many datapoints do you have in the dataset? \n",
    "#### Q3: Are there missing values in the dataset? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize the dataset on all features, except Latitude and Longitude\n",
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q4: Cluster the housing dataset using only the \"HouseAge\" feature. What is the optimal number of clusters (from 3 to 10) suggested by the silhouette score? \n",
    "\n",
    "Hint: to train on a single feature, some changes in the datastructure may be needed. Try applying to_numpy() and reshape(-1, 1) to your single-feature data, it may help. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will use K-means in this assignment and we will use the silhouette to find the optimal number of clusters\n",
    "# The elbow method is sometimes difficult to interpret: several values may be just as well selected\n",
    "# Instead, we will select the number of custers that offer the maximal silhouette score.\n",
    "# Below we set \"locate_elbow\" to False and we check the maximum in the plot. \n",
    "\n",
    "# Your code here to setup the K-means model and the KElbowVisualizer\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here to fit KElbow and visualize the results\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the K-means model using the optimal number of clusters. \n",
    "Visualise the clusters using the code below.\n",
    "\n",
    "#### Q5. How old are the houses in the cluster with the most recent houses? (0-5, 0-8, 0-12, 0-15, 0-21) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_means = KMeans(\n",
    "    n_clusters = HERE_THE_NUMBER_OF_CLUSTERS,\n",
    "    n_init = 5,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state = 2023,\n",
    ")\n",
    "\n",
    "k_means.fit(X['HouseAge'].to_numpy().reshape(-1, 1)) \n",
    "y_pred = k_means.labels_.astype(int)\n",
    "\n",
    "calif_X['cluster'] = y_pred\n",
    "\n",
    "fig = plt.figure()\n",
    "\n",
    "\n",
    "sns.stripplot(\n",
    "    data = calif_X, \n",
    "    x = 'HouseAge',\n",
    "    hue=\"cluster\", \n",
    ")\n",
    "\n",
    "plt.show()\n",
    "\n",
    "del k_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q6: Cluster the housing dataset using only the \"MedInc\" feature. What is the optimal number of clusters (from 3 to 10) suggested by the silhouette score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "k_means = KMeans(\n",
    "    n_clusters = HERE_THE_NUMBER_OF_CLUSTERS,\n",
    "    n_init = 5,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state = 2023,\n",
    ")\n",
    "\n",
    "k_means.fit(X['MedInc'].to_numpy().reshape(-1, 1)) \n",
    "y_pred = k_means.labels_.astype(int)\n",
    "\n",
    "calif_X['cluster'] = y_pred\n",
    "\n",
    "sns.relplot(\n",
    "    x=\"Longitude\", \n",
    "    y=\"Latitude\", \n",
    "    hue=\"cluster\", \n",
    "    data=calif_X, \n",
    "    height=6,\n",
    "    palette = 'tab10', #palette = ['red', 'blue', 'green'], #sns.color_palette(\"Paired\"),\n",
    ");\n",
    "\n",
    "plt.show()\n",
    "\n",
    "del k_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q7: Cluster the housing dataset using only the \"MedHouseVal\" feature. What is the optimal number of clusters (from 3 to 10) suggested by the silhouette score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "k_means = KMeans(\n",
    "    n_clusters = HERE_THE_NUMBER_OF_CLUSTERS,\n",
    "    n_init = 5,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state = 2023,\n",
    ")\n",
    "\n",
    "k_means.fit(X['MedHouseVal'].to_numpy().reshape(-1, 1)) \n",
    "y_pred = k_means.labels_.astype(int)\n",
    "\n",
    "calif_X['cluster'] = y_pred\n",
    "\n",
    "sns.relplot(\n",
    "    x=\"Longitude\", \n",
    "    y=\"Latitude\", \n",
    "    hue=\"cluster\", \n",
    "    data=calif_X, \n",
    "    height=6,\n",
    "    palette = sns.color_palette(palette = 'tab10'),\n",
    ")\n",
    "\n",
    "plt.show()\n",
    "del k_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q8: Cluster the housing dataset using all features except latitude and longitude. What is the optimal number of clusters (from 3 to 10) suggested by the silhouette score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "k_means = KMeans(\n",
    "    n_clusters = HERE_THE_NUMBER_OF_CLUSTERS,\n",
    "    n_init = 5,  # Number of times the k-means algorithm is run with different centroid seeds. \n",
    "                # The final results is the best output of n_init consecutive runs in terms of inertia. \n",
    "    random_state = 2023,\n",
    ")\n",
    "\n",
    "k_means.fit(X['MedHouseVal'].to_numpy().reshape(-1, 1)) \n",
    "y_pred = k_means.labels_.astype(int)\n",
    "\n",
    "calif_X['cluster'] = y_pred\n",
    "\n",
    "sns.relplot(\n",
    "    x=\"Longitude\", \n",
    "    y=\"Latitude\", \n",
    "    hue=\"cluster\", \n",
    "    data=calif_X, \n",
    "    height=6,\n",
    "    palette = sns.color_palette(palette = 'tab10'),\n",
    ")\n",
    "\n",
    "plt.show()\n",
    "del k_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}