CreateMNISTDataAsset.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Opprette tabluar data asset (MLTable) i SDK v2"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[MLTable](https://learn.microsoft.com/en-us/azure/machine-learning/concept-data#mltable) er Azure ML Studio sitt forsøk på å forenkle konsumering av tabluære data. MLTable erstatter det som heter TabularDataset i V1. Målet er å abstrahere vekk mest mulig ansvar for definisjonen av datasettet vekk fra consumenten og flytte det til der dataene ligger.\n",
        "\n",
        "Noen features fra MLTable:\n",
        "- Lazy loading av data\n",
        "- Definere schema der data ligger\n",
        "- Filtrere hvilke filer som er relevante\n",
        "- Filtrere vekk kolonner\n",
        "- Enkle datatransformasjoner\n",
        "\n",
        "Vi begynner med å hente MNIST fra azureml sin opendatasets pakke og fjerner kolonne for radnummer."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.opendatasets import MNIST\n",
        "mnist_df = MNIST.get_tabular_dataset().to_pandas_dataframe()\n",
        "mnist_df.head()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1666872305958
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vi gjør en random 20/80 split og lagrer lokalt som csv."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "train_data_folder = os.path.join(os.getcwd() + '/data/train')\n",
        "test_data_folder = os.path.join(os.getcwd() + '/data/test')\n",
        "\n",
        "os.makedirs(train_data_folder, exist_ok=True)\n",
        "os.makedirs(test_data_folder, exist_ok=True)\n",
        "\n",
        "train_data_df, test_data_df = train_test_split(mnist_df, train_size=0.8, shuffle=True)\n",
        "train_data_df.to_csv(train_data_folder + '/MNIST_Train.csv', index=False)\n",
        "test_data_df.to_csv(test_data_folder + '/MNIST_Test.csv', index=False)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1666872314130
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(train_data_df.shape)\n",
        "print(test_data_df.shape)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1666872314221
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Oppretter [MLTable](https://learn.microsoft.com/en-us/azure/machine-learning/concept-data#mltable) filer som ligger sammen med sine respektive datasett.\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile data/train/MLTable\n",
        "type: mltable\n",
        "\n",
        "paths:\n",
        "  - pattern: ./*.csv\n",
        "transformations:\n",
        "  - read_delimited:\n",
        "      delimiter: ','\n",
        "      encoding: ascii\n",
        "      header: all_files_same_headers"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile data/test/MLTable\n",
        "type: mltable\n",
        "\n",
        "paths:\n",
        "  - pattern: ./*.csv\n",
        "transformations:\n",
        "  - read_delimited:\n",
        "      delimiter: ','\n",
        "      encoding: ascii\n",
        "      header: all_files_same_headers"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Deretter brukes Azure ML SDKv2 for å registrere datasettet."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install azure-ai-ml"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azure.ai.ml.entities import Data\n",
        "from azure.ai.ml.constants import AssetTypes\n",
        "from azure.ai.ml import MLClient\n",
        "from azure.identity import DefaultAzureCredential\n",
        "\n",
        "ml_client = MLClient(\n",
        "    credential=DefaultAzureCredential(),\n",
        "    subscription_id = \"INSERT_SUBSCRIPTION_ID\",\n",
        "    resource_group_name = \"INSERT_RESOURCE_GROUP\",\n",
        "    workspace_name = \"INSERT_WORKSPACE\",\n",
        ")\n",
        "\n",
        "train_data = Data(\n",
        "    path='./data/train',\n",
        "    type=AssetTypes.MLTABLE,\n",
        "    description=\"\",\n",
        "    name=\"MNIST_Train\",\n",
        "    version='1'\n",
        ")\n",
        "\n",
        "ml_client.data.create_or_update(train_data)\n",
        "\n",
        "test_data = Data(\n",
        "    path='./data/test',\n",
        "    type=AssetTypes.MLTABLE,\n",
        "    description=\"\",\n",
        "    name=\"MNIST_Test\",\n",
        "    version='1',\n",
        ")\n",
        "\n",
        "ml_client.data.create_or_update(test_data)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1666861103862
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the mltable dataset like this when it is received in a component receiving the data."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Re-import dataset in a component\n",
        "\n",
        "import mltable\n",
        "\n",
        "tbl_test = mltable.load(\"data/test\")\n",
        "df_test = tbl_test.to_pandas_dataframe()\n",
        "\n",
        "X_test_flattened = df.drop(\"label\", axis=1).to_numpy().astype('uint8')\n",
        "X_test = X_test_flattened.reshape(X_test_flattened.shape[0], 28, 28)\n",
        "y_test = df[\"label\"].to_numpy().astype('uint8')\n",
        "X_train"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1666861104680
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python38-azureml",
      "language": "python",
      "display_name": "Python 3.8 - AzureML"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "kernel_info": {
      "name": "python38-azureml"
    },
    "microsoft": {
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}