Rede Neural.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sumário"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Funções de Ativação](#Funções-de-Ativação)\n",
    "\n",
    "[Funções Auxiliares](#Funções-Auxiliares)\n",
    "\n",
    "[Funções de Custo](#Funções-de-Custo)\n",
    "\n",
    "[Inicialização de Pesos](#Inicialização-de-Pesos)\n",
    "\n",
    "[Regularização](#Regularização)\n",
    "\n",
    "[Learning Rate Decay](#Learning-Rate-Decay)\n",
    "\n",
    "[Batch Normalization](#Batch-Normalization)\n",
    "\n",
    "[Batch Generator](#Batch-Generator)\n",
    "\n",
    "[Implementação](#Implementação)\n",
    "\n",
    "[Testes da Implementação](#Testes-da-Implementação)\n",
    "\n",
    "- [Exemplos do Notebook da Intuição](#Exemplos-do-Notebook-da-Intuição)\n",
    "\n",
    "- [Regressão](#Regressão)\n",
    "    - [Regressão Linear Simples](#Regressão-Linear-Simples---Exemplo-do-Perceptron)\n",
    "    - [Regressão Linear Multivariada](#Regressão-Linear-Multivariada---Exerc%C3%ADcio-de-Regressão-do-Perceptron)\n",
    "    - [Regressão Quadrática](#Regressão-Quadrática)\n",
    "    - [Regressão Cúbica](#Regressão-Cúbica)\n",
    "    - [Regressão Logarítimica](#Regressão-Logar%C3%ADtimica)\n",
    "    - [Regressão Exponencial](#Regressão-Exponencial)\n",
    "\n",
    "- [Classificação Binária](#Classificação-Binária)\n",
    "    - [Porta AND/OR](#Porta-AND/OR)\n",
    "    - [Porta XOR](#Porta-XOR)\n",
    "    - [2 Clusters](#2-Clusters)\n",
    "    - [4 Clusters](#4-Clusters)\n",
    "    - [Círculos](#C%C3%ADrculos)\n",
    "    - [Moons](#Moons)\n",
    "    - [Espiral](#Espiral)\n",
    "    \n",
    "- [Classificação Multiclasse](#Classificação-Multiclasse)\n",
    "    - [3 Clusters Multiclasse](#3-Clusters-Multiclasse)\n",
    "    - [4 Clusters Multiclasse](#4-Clusters-Multiclasse)\n",
    "    - [Espiral - 5 Classes](#Espiral---5-Classes)\n",
    "    - [Make Classification - 4 Classes](#Make-Classification---4-Classes)\n",
    "    - [Iris Dataset](#Iris-Dataset)\n",
    "\n",
    "[Referências](#Referências)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports and Configurações"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import _pickle as pkl\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.datasets import make_blobs, make_circles, make_moons, make_classification\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import MinMaxScaler, OneHotEncoder\n",
    "from utils import plot\n",
    "from utils.samples_generator import make_spiral, make_square, make_cubic, make_exp, make_log10\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Funções de Ativação"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear(x, derivative=False):\n",
    "    return np.ones_like(x) if derivative else x\n",
    "\n",
    "def sigmoid(x, derivative=False):\n",
    "    if derivative:\n",
    "        y = sigmoid(x)\n",
    "        return y*(1 - y)\n",
    "    return 1.0/(1.0 + np.exp(-x))\n",
    "\n",
    "def tanh(x, derivative=False):\n",
    "    if derivative:\n",
    "        y = tanh(x)\n",
    "        return 1 - y**2\n",
    "    return (np.exp(x) - np.exp(-x))/(np.exp(x) + np.exp(-x))\n",
    "\n",
    "def relu(x, derivative=False):\n",
    "    if derivative:\n",
    "        return np.where(x <= 0, 0, 1)\n",
    "    return np.maximum(0, x)\n",
    "\n",
    "def leaky_relu(x, derivative=False):\n",
    "    alpha = 0.1\n",
    "    if derivative:\n",
    "        return np.where(x <= 0, alpha, 1)\n",
    "    return np.where(x <= 0, alpha*x, x)\n",
    "\n",
    "def elu(x, derivative=False):\n",
    "    alpha = 1.0\n",
    "    if derivative:\n",
    "        y = elu(x)\n",
    "        return np.where(x <= 0, y + alpha, 1)\n",
    "    return np.where(x <= 0, alpha*(np.exp(x) - 1), x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Funções Auxiliares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x, y_oh=None, derivative=False):\n",
    "    if derivative:\n",
    "        y_pred = softmax(x)\n",
    "        k = np.nonzero(y_pred * y_oh)\n",
    "        pk = y_pred[k]\n",
    "        y_pred[k] = pk * (1.0 - pk)\n",
    "        return y_pred\n",
    "    exp = np.exp(x)\n",
    "    return exp / np.sum(exp, axis=1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Funções de Custo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Para Regressão"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Para Classificação Binária"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Para Classificação Multiclasse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neg_log_likelihood(y_oh, y_pred, derivative=False):\n",
    "    k = np.nonzero(y_pred * y_oh)\n",
    "    pk = y_pred[k]\n",
    "    if derivative:\n",
    "        y_pred[k] = (-1.0 / pk)\n",
    "        return y_pred\n",
    "    return np.mean(-np.log(pk))\n",
    "\n",
    "def softmax_neg_log_likelihood(y_oh, y_pred, derivative=False):\n",
    "    y_softmax = softmax(y_pred)\n",
    "    if derivative:\n",
    "        return -(y_oh - y_softmax) / y_oh.shape[0]    \n",
    "    return neg_log_likelihood(y_oh, y_softmax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inicialização de Pesos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularização"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Rate Decay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementação "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exemplos do Notebook da Intuição"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exemplo 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([[0.05, 0.10]])\n",
    "y = np.array([[0.01, 0.99]])\n",
    "\n",
    "w1 = np.array([[0.15, 0.20], [0.25, 0.30]])\n",
    "b1 = np.array([[0.35]]) \n",
    "w2 = np.array([[0.40, 0.45], [0.50, 0.55]])\n",
    "b2 = np.array([[0.60]])\n",
    "\n",
    "# insira sua rede aqui!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exemplo 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([[0.1, 0.2, 0.7]])\n",
    "y = np.array([[1, 0, 0]])\n",
    "D_in, D_out = x.shape[1], y.shape[1]\n",
    "\n",
    "w1 = np.array([[0.1, 0.2, 0.3], [0.3, 0.2, 0.7], [0.4, 0.3, 0.9]])\n",
    "b1 = np.ones((1,3))\n",
    "w2 = np.array([[0.2, 0.3, 0.5], [0.3, 0.5, 0.7], [0.6, 0.4, 0.8]])\n",
    "b2 = np.ones((1,3))\n",
    "w3 = np.array([[0.1, 0.4, 0.8], [0.3, 0.7, 0.2], [0.5, 0.2, 0.9]])\n",
    "b3 = np.ones((1,3))\n",
    "\n",
    "# insira sua rede aqui!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Gradient Checking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regressão"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Linear Simples - Exemplo do Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.loadtxt('data/medidas.csv', delimiter=',', skiprows=1)\n",
    "print(data.shape)\n",
    "\n",
    "x, y = data[:,0].reshape(-1,1), data[:,1].reshape(-1,1)\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(), x.max())\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, nn.predict(x), c='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Linear Multivariada - Exercício de Regressão do Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.loadtxt('data/notas.csv', delimiter=',', skiprows=1)\n",
    "print(data.shape)\n",
    "\n",
    "x, y = data[:,:-1], data[:,-1].reshape(-1,1)\n",
    "print(x.shape, y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(axis=0), x.max(axis=0))\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Quadrática"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_square(n_samples=100, x_min=-10, x_max=10, a=1, b=1, c=1, noise=10)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(axis=0), x.max(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, nn.predict(x), c='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Cúbica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_cubic(n_samples=100, x_min=-4, x_max=4, a=1, b=0, c=-10, d=0, noise=3)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(axis=0), x.max(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, nn.predict(x), c='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Logarítimica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_log10(n_samples=100, x_min=1, x_max=100, noise=0.3)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(axis=0), x.max(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, nn.predict(x), c='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regressão Exponencial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_exp(n_samples=100, x_min=0, x_max=5, noise=10)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = MinMaxScaler(feature_range=(-1, 1))\n",
    "x = minmax.fit_transform(x.astype(np.float64))\n",
    "\n",
    "print(x.min(axis=0), x.max(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, nn.predict(x), c='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classificação Binária"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Porta AND/OR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
    "y = np.array([0, 0, 0, 1]).reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Predições:', y_pred, sep='\\n')\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Porta XOR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
    "y = np.array([0, 1, 1, 0]).reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Predições:', y_pred, sep='\\n')\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 Clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_blobs(n_samples=100, n_features=2, centers=2, random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "\n",
    "threshold = 0 if nn.layers[-1].activation == linear else 0.5\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred >= threshold)))\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, threshold=threshold, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 Clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_blobs(n_samples=500, n_features=2, cluster_std=0.9, centers=[(-3, -3), (3, 3), (-3, 3), (3, -3)], random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "y = np.where(y >= 2, 1, 0)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, threshold=0.5, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Círculos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_circles(n_samples=500, noise=0.1, factor=0.4, random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, threshold=0.5, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_moons(200, noise=0.20)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, threshold=0.5, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Espiral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_spiral(n_samples=100, n_class=2, radius=5, laps=1.75)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap='bwr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = nn.predict(x)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred > 0.5)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=True, nn=nn, threshold=0.5, cmap='bwr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classificação Multiclasse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 Clusters Multiclasse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_blobs(n_samples=300, n_features=2, centers=[(0, -3), (-3, 3), (3, 3)], random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap=plt.cm.viridis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onehot = OneHotEncoder(sparse=False)\n",
    "y_onehot = onehot.fit_transform(y)\n",
    "print(y_onehot[::60])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "    \n",
    "y_pred = np.argmax(nn.predict(x), axis=1)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=False, nn=nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 Clusters Multiclasse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_blobs(n_samples=400, n_features=2, centers=[(-3, 0), (3, 0), (0, 3), (0, -3)], random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap=plt.cm.viridis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onehot = OneHotEncoder(sparse=False)\n",
    "y_onehot = onehot.fit_transform(y)\n",
    "print(y_onehot[::70])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = np.argmax(nn.predict(x), axis=1)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=False, nn=nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Espiral - 5 Classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_spiral(n_samples=100, n_class=5, radius=1, laps=0.5)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap=plt.cm.viridis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "onehot = OneHotEncoder(sparse=False)\n",
    "y_onehot = onehot.fit_transform(y)\n",
    "print(y_onehot[::100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = np.argmax(nn.predict(x), axis=1)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=False, nn=nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Classification - 4 Classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = make_classification(n_samples=100, n_classes=4, n_features=2, n_clusters_per_class=1, n_redundant=0, n_repeated=0, random_state=1234)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap=plt.cm.viridis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onehot = OneHotEncoder(sparse=False)\n",
    "y_onehot = onehot.fit_transform(y)\n",
    "print(y_onehot[::10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = np.argmax(nn.predict(x), axis=1)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=False, nn=nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iris Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_iris()\n",
    "x, y = data.data[:, 2:], data.target.reshape(-1,1)\n",
    "\n",
    "print(data.feature_names)\n",
    "print(data.target_names)\n",
    "print(x.shape, y.shape)\n",
    "plt.scatter(x[:,0], x[:,1], c=list(np.array(y).ravel()), s=15, cmap=plt.cm.viridis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onehot = OneHotEncoder(sparse=False)\n",
    "y_onehot = onehot.fit_transform(y)\n",
    "print(y_onehot[::20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim, output_dim = x.shape[1], y.shape[1]\n",
    "\n",
    "# insira sua rede aqui!\n",
    "\n",
    "y_pred = np.argmax(nn.predict(x), axis=1)\n",
    "print('Acurácia: {:.2f}%'.format(100*accuracy_score(y, y_pred)))\n",
    "\n",
    "plot.classification_predictions(x, y, is_binary=False, nn=nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Link para o Questionário](https://docs.google.com/forms/d/e/1FAIpQLSf1B3JKsPPPc1oYD_3FyX4UvNLpcXnTW2I177A_WE0_B56Elw/viewform?usp=sf_link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Referências"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Tensorflow Playground](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.94880&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false)\n",
    "\n",
    "- [Linear Classification](http://cs231n.github.io/linear-classify/)\n",
    "\n",
    "__Regularization__:\n",
    "- [Implementing Neural Network L1 Regularization](https://jamesmccaffrey.wordpress.com/2017/06/27/implementing-neural-network-l1-regularization/)\n",
    "- [Implementing Neural Network L2 Regularization](https://jamesmccaffrey.wordpress.com/2017/06/29/implementing-neural-network-l2-regularization/)\n",
    "- [Neural Network L2 Regularization Using Python](https://visualstudiomagazine.com/articles/2017/09/01/neural-network-l2.aspx)\n",
    "\n",
    "__Momentum__:\n",
    "- [Neural Network Momentum Using Python](https://visualstudiomagazine.com/articles/2017/08/01/neural-network-momentum.aspx)\n",
    "- [Beyond SGD: Gradient Descent with Momentum and Adaptive Learning Rate](https://wiseodd.github.io/techblog/2016/06/22/nn-optimization/)\n",
    "\n",
    "__Learning Rate Decay__:\n",
    "- [Using Learning Rate Schedules for Deep Learning Models in Python with Keras](https://machinelearningmastery.com/using-learning-rate-schedules-deep-learning-models-python-keras/)\n",
    "- [Learning Rate Schedules and Adaptive Learning Rate Methods for Deep Learning](https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1)\n",
    "- [Tensorflow Exponential Decay Documentation](https://www.tensorflow.org/api_docs/python/tf/train/exponential_decay)\n",
    "\n",
    "__Batch Normalization__:\n",
    "- [Understanding the backward pass through Batch Normalization Layer](https://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html)\n",
    "- [Deriving the Gradient for the Backward Pass of Batch Normalization](https://kevinzakka.github.io/2016/09/14/batch_normalization/)\n",
    "- [Implementing BatchNorm in Neural Net](https://wiseodd.github.io/techblog/2016/07/04/batchnorm/)\n",
    "- [What does the gradient flowing through batch normalization looks like ?](http://cthorey.github.io./backpropagation/)\n",
    "- [GRADIENTS, BATCH NORMALIZATION AND LAYER NORMALIZATION](https://theneuralperspective.com/2016/10/27/gradient-topics/)\n",
    "\n",
    "__Gradient Checking__:\n",
    "- [cs231n](https://github.com/cs231n/cs231n.github.io/blob/master/neural-networks-3.md#gradcheck)\n",
    "- [Gradient checking and advanced optimization](http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)\n",
    "- [BatchNormalization fails gradient checking](https://github.com/torch/nn/issues/744)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}