5.4-MNIST_ComparacaoAlgoritmos.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPJP3E8kSgRq2KxKKpfxTXi"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Comparação entre os modelos utilizando a base de dados MNIST\n",
        "\n",
        "MNIST é uma base de dados de imagens de dígitos manuscritos que é amplamente utilizada em testes e benchmarks de modelos de aprendizado de máquina. A base de dados MNIST foi criada pelo National Institute of Standards and Technology (NIST) dos EUA e contém 60.000 imagens de treinamento e 10.000 imagens de teste de dígitos manuscritos de 0 a 9. As imagens são em preto e branco e têm 28x28 pixels cada.\n",
        "\n",
        "MNIST é uma base de dados popular entre os pesquisadores e desenvolvedores de modelos de aprendizado de máquina, pois é de fácil acesso e contém um conjunto de dados bem definido e de fácil utilização. É comumente usada como um benchmark para avaliar o desempenho de diferentes modelos de aprendizado de máquina em tarefas de classificação de imagens."
      ],
      "metadata": {
        "id": "-O6TQN7bb40Y"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PXhCPOD6ryw2"
      },
      "outputs": [],
      "source": [
        "import joblib\n",
        "\n",
        "from sklearn.metrics import confusion_matrix,ConfusionMatrixDisplay\n",
        "from sklearn.metrics import classification_report\n",
        "import numpy as np\n",
        "\n",
        "from io import BytesIO\n",
        "import requests\n",
        "import time"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.datasets import mnist\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
        "\n",
        "X_test = (X_test.astype(np.float32))\n",
        "X_test = X_test.reshape(10000, 784)\n"
      ],
      "metadata": {
        "id": "IXF6uzcwAvP4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Algoritmo KNN"
      ],
      "metadata": {
        "id": "T5p0Jx5Lb34j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mLink = 'https://github.com/diegonogare/MachineLearning/blob/main/modelo_knn_mnist.pkl?raw=true'\n",
        "mfile = BytesIO(requests.get(mLink).content)\n",
        "modelo_knn = joblib.load(mfile)"
      ],
      "metadata": {
        "id": "9tBZIiR-Mp-q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(modelo_knn)\n",
        "\n",
        "inicio = time.time()\n",
        "predicao_knn = modelo_knn.predict(X_test)\n",
        "termino = time.time()\n",
        "print(\"[%s segundos para fazer o scoring do modelo]\" % (termino - inicio))\n",
        "\n",
        "predicao_knn = predicao_knn.astype(np.int64)\n",
        "matriz_confusao = confusion_matrix(y_test, predicao_knn)\n",
        "print(matriz_confusao) \n",
        "print('-'*40)\n",
        "print(classification_report(y_test, predicao_knn))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b1tYCkdQWHfi",
        "outputId": "fc53a4ae-3760-419b-cd13-bccf0f75fce8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GridSearchCV(cv=StratifiedKFold(n_splits=10, random_state=42, shuffle=True),\n",
            "             estimator=KNeighborsClassifier(),\n",
            "             param_grid=[{'metric': ['euclidean'], 'n_neighbors': [3, 5, 7, 9],\n",
            "                          'weights': ['uniform', 'distance']}],\n",
            "             verbose=3)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but KNeighborsClassifier was fitted with feature names\n",
            "  \"X does not have valid feature names, but\"\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[38.627347469329834 segundos para fazer o scoring do modelo]\n",
            "[[ 980    0    0    0    0    0    0    0    0    0]\n",
            " [   0 1135    0    0    0    0    0    0    0    0]\n",
            " [   0    0 1032    0    0    0    0    0    0    0]\n",
            " [   0    0    0 1010    0    0    0    0    0    0]\n",
            " [   0    0    0    0  982    0    0    0    0    0]\n",
            " [   0    0    0    0    0  892    0    0    0    0]\n",
            " [   0    0    0    0    0    0  958    0    0    0]\n",
            " [   0    0    0    0    0    0    0 1028    0    0]\n",
            " [   0    0    0    0    0    0    0    0  974    0]\n",
            " [   0    0    0    0    0    0    0    0    0 1009]]\n",
            "----------------------------------------\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00       980\n",
            "           1       1.00      1.00      1.00      1135\n",
            "           2       1.00      1.00      1.00      1032\n",
            "           3       1.00      1.00      1.00      1010\n",
            "           4       1.00      1.00      1.00       982\n",
            "           5       1.00      1.00      1.00       892\n",
            "           6       1.00      1.00      1.00       958\n",
            "           7       1.00      1.00      1.00      1028\n",
            "           8       1.00      1.00      1.00       974\n",
            "           9       1.00      1.00      1.00      1009\n",
            "\n",
            "    accuracy                           1.00     10000\n",
            "   macro avg       1.00      1.00      1.00     10000\n",
            "weighted avg       1.00      1.00      1.00     10000\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Algoritmo Decision Tree"
      ],
      "metadata": {
        "id": "cVLU35kZgLeh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mLink = 'https://github.com/diegonogare/MachineLearning/blob/main/modelo_dt_mnist.pkl?raw=true'\n",
        "mfile = BytesIO(requests.get(mLink).content)\n",
        "modelo_dt = joblib.load(mfile)"
      ],
      "metadata": {
        "id": "jfU6Ta3oT89k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(modelo_dt)\n",
        "\n",
        "inicio = time.time()\n",
        "predicao_dt = modelo_dt.predict(X_test)\n",
        "termino = time.time()\n",
        "print(\"[%s segundos para fazer o scoring do modelo]\" % (termino - inicio))\n",
        "\n",
        "predicao_dt = predicao_dt.astype(np.int64)\n",
        "matriz_confusao = confusion_matrix(y_test, predicao_dt)\n",
        "print(matriz_confusao) \n",
        "print('-'*40)\n",
        "print(classification_report(y_test, predicao_dt))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_R69_tNzT9ic",
        "outputId": "4b77d72b-e853-40bf-8e57-605f83452722"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GridSearchCV(cv=StratifiedKFold(n_splits=10, random_state=42, shuffle=True),\n",
            "             estimator=DecisionTreeClassifier(),\n",
            "             param_grid=[{'criterion': ['gini', 'entropy'],\n",
            "                          'max_depth': [10, 50, 80]}],\n",
            "             verbose=3)\n",
            "[0.01206064224243164 segundos para fazer o scoring do modelo]\n",
            "[[ 980    0    0    0    0    0    0    0    0    0]\n",
            " [   0 1135    0    0    0    0    0    0    0    0]\n",
            " [   0    0 1032    0    0    0    0    0    0    0]\n",
            " [   0    0    0 1010    0    0    0    0    0    0]\n",
            " [   0    0    0    0  982    0    0    0    0    0]\n",
            " [   0    0    0    0    0  892    0    0    0    0]\n",
            " [   0    0    0    0    0    0  958    0    0    0]\n",
            " [   0    0    0    0    0    0    0 1028    0    0]\n",
            " [   0    0    0    0    0    0    0    0  974    0]\n",
            " [   0    0    0    0    0    0    0    0    0 1009]]\n",
            "----------------------------------------\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00       980\n",
            "           1       1.00      1.00      1.00      1135\n",
            "           2       1.00      1.00      1.00      1032\n",
            "           3       1.00      1.00      1.00      1010\n",
            "           4       1.00      1.00      1.00       982\n",
            "           5       1.00      1.00      1.00       892\n",
            "           6       1.00      1.00      1.00       958\n",
            "           7       1.00      1.00      1.00      1028\n",
            "           8       1.00      1.00      1.00       974\n",
            "           9       1.00      1.00      1.00      1009\n",
            "\n",
            "    accuracy                           1.00     10000\n",
            "   macro avg       1.00      1.00      1.00     10000\n",
            "weighted avg       1.00      1.00      1.00     10000\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but DecisionTreeClassifier was fitted with feature names\n",
            "  \"X does not have valid feature names, but\"\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Algoritmo Multi-Layer Perceptron"
      ],
      "metadata": {
        "id": "urtchp28gPzN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mLink = 'https://github.com/diegonogare/MachineLearning/blob/main/modelo_mlp_mnist.pkl?raw=true'\n",
        "mfile = BytesIO(requests.get(mLink).content)\n",
        "modelo_mlp = joblib.load(mfile)"
      ],
      "metadata": {
        "id": "yCMNDP4C99aF",
        "outputId": "d7041715-04cf-4dca-811c-002cae86de41",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator MLPClassifier from version 1.1.3 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
            "https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations\n",
            "  UserWarning,\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator LabelBinarizer from version 1.1.3 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
            "https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations\n",
            "  UserWarning,\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator GridSearchCV from version 1.1.3 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
            "https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations\n",
            "  UserWarning,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(modelo_mlp)\n",
        "\n",
        "inicio = time.time()\n",
        "predicao_mlp = modelo_mlp.predict(X_test)\n",
        "termino = time.time()\n",
        "print(\"[%s segundos para fazer o scoring do modelo]\" % (termino - inicio))\n",
        "\n",
        "predicao_mlp = predicao_mlp.astype(np.int64)\n",
        "matriz_confusao = confusion_matrix(y_test, predicao_mlp)\n",
        "print(matriz_confusao) \n",
        "print('-'*40)\n",
        "print(classification_report(y_test, predicao_mlp))"
      ],
      "metadata": {
        "id": "f4sbvd0B99Ti",
        "outputId": "0b180134-6469-444b-9ba5-b6f5001f90fd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GridSearchCV(cv=StratifiedKFold(n_splits=10, random_state=42, shuffle=True),\n",
            "             estimator=MLPClassifier(),\n",
            "             param_grid=[{'activation': ['tanh', 'relu'],\n",
            "                          'hidden_layer_sizes': [100, (10, 10), (15, 15),\n",
            "                                                 (20, 20)],\n",
            "                          'solver': ['sgd', 'adam']}],\n",
            "             verbose=3)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but MLPClassifier was fitted with feature names\n",
            "  \"X does not have valid feature names, but\"\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0.10040473937988281 segundos para fazer o scoring do modelo]\n",
            "[[ 979    0    0    0    0    0    0    1    0    0]\n",
            " [   0 1134    0    0    0    0    0    1    0    0]\n",
            " [   0    3 1024    1    1    0    0    1    2    0]\n",
            " [   0    0    1 1000    0    1    0    2    0    6]\n",
            " [   0    0    0    0  975    0    0    1    0    6]\n",
            " [   0    0    0    1    0  888    0    0    3    0]\n",
            " [   0    0    0    0    0    1  955    0    2    0]\n",
            " [   1    1    1    0    0    0    0 1024    0    1]\n",
            " [   0    0    5    2    0    1    1    2  959    4]\n",
            " [   0    0    0    4    0    0    0    0    0 1005]]\n",
            "----------------------------------------\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00       980\n",
            "           1       1.00      1.00      1.00      1135\n",
            "           2       0.99      0.99      0.99      1032\n",
            "           3       0.99      0.99      0.99      1010\n",
            "           4       1.00      0.99      1.00       982\n",
            "           5       1.00      1.00      1.00       892\n",
            "           6       1.00      1.00      1.00       958\n",
            "           7       0.99      1.00      0.99      1028\n",
            "           8       0.99      0.98      0.99       974\n",
            "           9       0.98      1.00      0.99      1009\n",
            "\n",
            "    accuracy                           0.99     10000\n",
            "   macro avg       0.99      0.99      0.99     10000\n",
            "weighted avg       0.99      0.99      0.99     10000\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Referências\n",
        "\n",
        "LeCun, Yann, et al. \"MNIST handwritten digit database.\" AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist/. [Acessado em 05-Dezembro-2022].\n",
        "\n",
        "Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: concepts, tools, and techniques to build intelligent systems. O'Reilly Media, 2019."
      ],
      "metadata": {
        "id": "GMYUV4rdfNT_"
      }
    }
  ]
}