005_machine_learning_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Disclaimer**: Some examples and some text in this lesson are modified from Jake VanderPlas's [Python Data Science Handbook](http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/Index.ipynb). It is superb starting point for anyone who wants to learn more about machine learning and data science in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to machine learning\n",
    "\n",
    "Machine learning (ML) is a huge area of research and application, and it is currently expanding very quickly. We will look at the fundamentals principles of machine learning and try out some of the algorithms.\n",
    "\n",
    "ML can be broadly defined as building models of data, usually with a purpose of transfomation (e.g. dimensionality reduction) or prediction (e.g. classification or regression), but sometimes just to further our understanding of the data.\n",
    "\n",
    "\"Learning\" appears when we give these models **tunable parameters** that can be **adapted** to observed data; in this way the algorithm can be considered to be \"learning\" from the data.\n",
    "\n",
    "# Categories of ML\n",
    "\n",
    "### Supervised learning\n",
    "Involves modeling the relationship between measured **features of data** and some **label** associated with the data; once this model is determined, it can be used to apply labels to new, unknown data. This is further subdivided into **classification** tasks and **regression** tasks: in classification, the labels are discrete categories, while in regression, the labels are continuous quantities.\n",
    "\n",
    "### Unsupervised learning\n",
    "Involves modeling the features of a dataset **without** reference to any label. These models include tasks such as clustering and dimensionality reduction. Clustering algorithms identify distinct groups of data, while dimensionality reduction algorithms search for new representations of the data in a space with fewer dimensions.\n",
    "\n",
    "In addition, there are so-called *semi-supervised* learning methods, which falls somewhere between supervised learning and unsupervised learning. Semi-supervised learning methods are often useful when only incomplete labels are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = 10 * np.random.rand(50)\n",
    "y = 2 * x - 1 + np.random.randn(50)\n",
    "plt.scatter(x, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = LinearRegression(fit_intercept=True)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = x[:, np.newaxis]\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xfit = np.linspace(-1, 11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Xfit = xfit[:, np.newaxis]\n",
    "yfit = model.predict(Xfit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(x, y)\n",
    "plt.plot(xfit, yfit);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris = pd.read_csv('data/iris.csv')\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.pairplot(iris, hue='species', size=2.5);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "X_iris = iris.drop('species', axis='columns')\n",
    "X_iris.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y_iris = iris['species']\n",
    "y_iris.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 1. import a model class\n",
    "from sklearn.naive_bayes import GaussianNB \n",
    "\n",
    "# 2. instantiate model\n",
    "model = GaussianNB()                       \n",
    "\n",
    "# 3. fit model to data\n",
    "model.fit(X_iris, y_iris)                  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_model = model.predict(X_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame({'true_labels':y_iris, 'predicted_labels':y_model})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.mean(y_iris == y_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection  import train_test_split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_iris, y_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 2. instantiate model\n",
    "model = GaussianNB()            \n",
    "\n",
    "# 3. fit model to data\n",
    "model.fit(Xtrain, ytrain)                  \n",
    "\n",
    "# 4. predict on new data\n",
    "y_model = model.predict(Xtest)             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dimensionality reduction / transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 1. import the model class\n",
    "from sklearn.decomposition import PCA  \n",
    "\n",
    "# 2. instantiate the model with hyperparameters\n",
    "model = PCA(n_components=2)            \n",
    "\n",
    "# 3. fit to data. Notice y is not specified!\n",
    "model.fit(X_iris)                      \n",
    "\n",
    "# 4. transform the data to two dimensions\n",
    "X_2D = model.transform(X_iris)         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris['PCA1'] = X_2D[:, 0]\n",
    "iris['PCA2'] = X_2D[:, 1]\n",
    "sns.lmplot(\"PCA1\", \"PCA2\", hue='species', data=iris, fit_reg=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 1. import the model class\n",
    "from sklearn.mixture import GaussianMixture     \n",
    "\n",
    "# 2. instantiate the model with hyperparameters\n",
    "model = GaussianMixture (n_components=3,\n",
    "            covariance_type='full')  \n",
    "\n",
    "# 3. fit to data. Notice y is not specified!\n",
    "model.fit(X_iris)                    \n",
    "\n",
    "# 4. determine cluster labels\n",
    "iris['cluster'] = model.predict(X_iris)\n",
    "\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.lmplot(\"PCA1\", \"PCA2\", data=iris, hue='species',\n",
    "           col='cluster', fit_reg=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(digits.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "digits.images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "digits.images[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.imshow(digits.images[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(10, 10, figsize=(8, 8),\n",
    "                         subplot_kw={'xticks':[], 'yticks':[]},\n",
    "                         gridspec_kw=dict(hspace=0.1, wspace=0.1))\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(digits.images[i])\n",
    "    ax.text(0.05, 0.05, str(digits.target[i]),\n",
    "            transform=ax.transAxes, color='green')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = digits.data\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = digits.target\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import Isomap\n",
    "iso = Isomap(n_components=2)\n",
    "iso.fit(digits.data)\n",
    "data_projected = iso.transform(digits.data)\n",
    "data_projected.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(data_projected[:, 0], data_projected[:, 1], c=digits.target,\n",
    "            edgecolor='none', alpha=0.5,\n",
    "            cmap=plt.cm.get_cmap('nipy_spectral', 10))\n",
    "plt.colorbar(label='digit label', ticks=range(10))\n",
    "plt.clim(-0.5, 9.5);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "model = GaussianNB()\n",
    "model.fit(Xtrain, ytrain)\n",
    "y_model = model.predict(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "mat = confusion_matrix(ytest, y_model)\n",
    "\n",
    "sns.heatmap(mat, square=True, annot=True, cbar=False)\n",
    "plt.xlabel('predicted value')\n",
    "plt.ylabel('true value');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(10, 10, figsize=(8, 8),\n",
    "                         subplot_kw={'xticks':[], 'yticks':[]},\n",
    "                         gridspec_kw=dict(hspace=0.1, wspace=0.1))\n",
    "\n",
    "test_images = Xtest.reshape(-1, 8, 8)\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(test_images[i], cmap='binary', interpolation='nearest')\n",
    "    ax.text(0.05, 0.05, str(y_model[i]),\n",
    "            transform=ax.transAxes,\n",
    "            color='green' if (ytest[i] == y_model[i]) else 'red')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}