02a_Importing_Data_(Console).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Data (with the AWS Console)\n",
    "\n",
    "In this notebook, we'll create our project workspace in Amazon Personalize and import the prepared data - using the [Amazon Personalize console UI](https://console.aws.amazon.com/personalize/home).\n",
    "\n",
    "> For an **alternative** approach to the same steps *programmatically* using [Boto3, the AWS SDK for Python](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) - see Notebook [02b_Importing_Data_(Python_SDK).ipynb](02b_Importing_Data_(Python_SDK).ipynb) instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Dataset Group (DSG)\n",
    "\n",
    "You can think of the **dataset group** as your **project workspace**: It's the container within which your datasets, models and deployments/inferences will be created.\n",
    "\n",
    "A dataset group can contain multiple solutions (models) and campaigns (deployments), but **only one instance of each dataset type** (interactions, items, and users), so:\n",
    "\n",
    "- You can experiment with different *algorithms and models* **within** one dataset group - which we'll do in this example... But\n",
    "- For comparing results with *different datasets/schemas*, you'll usually need to work with **multiple** dataset groups.\n",
    "\n",
    "Since all these steps can be performed through the SDKs/API too, it's absolutely possible to automate pipelines for setting up multiple dataset groups and experiments within them. We'd recommend referring to the MLOps samples in the [official Amazon Personalize samples repository](https://github.com/aws-samples/amazon-personalize-samples) for examples on how to do this.\n",
    "\n",
    "Since we'll experiment only with different model configurations, we'll create a single dataset group in this example.\n",
    "\n",
    "▶️ **Open** the [Amazon Personalize console](https://console.aws.amazon.com/personalize/home) in the region where you wish to work (i.e. the one your data is in)\n",
    "\n",
    "▶️ **Click** to *Create Dataset Group* (or *Get started*, if you see that button instead)\n",
    "\n",
    "- Enter a valid Dataset Group name (like `personalize-poc-lab`) and click **Next**\n",
    "\n",
    "![](static/imgs/create_dataset_group1.png \"Create Dataset Group screenshot\")\n",
    "\n",
    "That's it, your dataset group is now created!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Interactions Dataset Schema\n",
    "\n",
    "You should be automatically prompted to continue to Step 2 (*Create User-Item Interaction Data*). If not, you can access the same workflow by:\n",
    "\n",
    "- Opening the *Dataset groups* list from the left sidebar menu\n",
    "- Selecting your DSG by clicking on the name hyperlink\n",
    "- Clicking the *User-Item Interaction Data Import* button from the dashboard\n",
    "\n",
    "In this step, we'll need to **define the structure** of our interactions CSV using a JSON schema language - referring to the [Datasets and Schemas](https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html) section of the developer guide.\n",
    "\n",
    "> ⚠️ **NOTE** that:\n",
    ">\n",
    "> - The columns list in the JSON must **exactly match** the data file, **including the order of columns**\n",
    "> - Any fields with **missing values** *must* include `null` in their `type` entry to be correctly treated by the model\n",
    "> - Watch out also for the `categorical` attribute, which must be set on string fields where appropriate\n",
    "\n",
    "A comprehensive example schema is provided on the [Interactions Dataset doc page](https://docs.aws.amazon.com/personalize/latest/dg/interactions-datasets.html) for reference.\n",
    "\n",
    "For our case, the schema should be:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Interactions\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"EVENT_VALUE\",\n",
    "            \"type\": [\"float\", \"null\"]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"TIMESTAMP\",\n",
    "            \"type\": \"long\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"EVENT_TYPE\",\n",
    "            \"type\": \"string\"\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "```\n",
    "\n",
    "▶️ **Check** this schema matches your `interactions.csv` file as we created previously\n",
    "\n",
    "▶️ **Configure** your dataset with:\n",
    "\n",
    "- *Dataset name* as `personalize-movielens-interactions`\n",
    "- A *New schema* named as `personalize-movielens-interactions-schema`, using the JSON above ^\n",
    "\n",
    "![](static/imgs/create-dataset-interactions.png \"Create Dataset screenshot showing specified names and schema\")\n",
    "\n",
    "Once you've checked your dataset setup, go ahead and click **Next**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the Interactions Data\n",
    "\n",
    "You should be automatically prompted to continue to Step 3 (*Import User-Item Interaction Data*). If not, you can access the same workflow by:\n",
    "\n",
    "- Opening your Dataset Group from the *Dataset groups* list in the sidebar menu\n",
    "- Clicking the *Datasets* submenu that now becomes enabled in the sidebar menu\n",
    "- Selecting the interactions dataset, and clicking the *Create dataset import job* button\n",
    "\n",
    "In this step, we'll create a **dataset import job** to read our interactions data from S3, validate and load it into our Amazon Personalize dataset group.\n",
    "\n",
    "▶️ **Name** your import job, like: `personalize-movielens-interactions-01`\n",
    "\n",
    "▶️ **Select** the **role** we set up previously with access to the S3 bucket. If it doesn't show in the dropdown, you can select *Enter a custom IAM role ARN* and enter the ARN manually.\n",
    "\n",
    "If you went through the setup via [Notebook 0](00_Environment_Setup.ipynb), you should be able to get your role ARN by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r\n",
    "print(personalize_role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...If you set up manually, find your role in the [AWS IAM Console 'Roles' list](https://console.aws.amazon.com/iam/home?#/roles) and click on its name to show the details page. The ARN will be near the top, with a format something like `arn:aws:iam::123456789012:role/PersonalizeRolePOC`\n",
    "\n",
    "▶️ **Enter** the **S3 URI** of your uploaded interactions CSV.\n",
    "\n",
    "Assuming you prepared your data via [Notebook 1](01_Preparing_Input_Data.ipynb), you should be able to get your data URI by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r\n",
    "print(interactions_s3uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...If you set up manually, your URI should follow a similar structure to the placeholder text.\n",
    "\n",
    "your import job configuration should look something like:\n",
    "\n",
    "![](static/imgs/create-interactions-import.png \"Create dataset import screenshot\")\n",
    "\n",
    "When you've checked your configuration, click **Finish** to start the import. You'll typically be directed back to the *dashboard* overview screen for your dataset group:\n",
    "\n",
    "![](static/imgs/dashboard-importing-interactions.png \"Dataset group dashboard showing importing interactions data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ⏰ Importing the data can take some time - which for our small datasets like our movielens-100k extract is often dominated by **overheads** of starting up and shutting down required processing infrastructure and jobs... Rather than scaling with the number of records.\n",
    ">\n",
    "> For this small sample, the import should take something like 15 minutes.\n",
    "\n",
    "We'll start off the next (item metadata) import job in parallel, and wait for both to complete in a later section.\n",
    "\n",
    "Note that:\n",
    "\n",
    "- We **cannot use the dataset until the import job is complete** (e.g. in `ACTIVE` status)... But also,\n",
    "- **Only the interactions dataset is mandatory** - so it would be possible to skip over the item metadata sections below and start building solutions, as soon as this import job completes.\n",
    "- The **Datasets tab is now enabled** in the dashboard: From this tab you can view more details about your three datasets (interactions, users, and items) and in-progress import jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Item Metadata Schema\n",
    "\n",
    "We'll follow similar general steps for our item metadata set as for the core interactions dataset: First, creating a dataset schema using the [Datasets and Schemas](https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html) docs and the [sample items schema](https://docs.aws.amazon.com/personalize/latest/dg/items-datasets.html#schema-examples-items) as a guide.\n",
    "\n",
    "▶️ **Start** the workflow by clicking the **Item** data *Import* button from the dashboard, or selecting the Item slot on the *Datasets* list (from sidebar tab) and clicking *Create dataset*\n",
    "\n",
    "▶️ **CHECK** the schema below exactly matches the `item-meta.csv` we created earlier:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Items\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"GENRES\",\n",
    "            \"type\": \"string\",\n",
    "            \"categorical\": true\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"YEAR\",\n",
    "            \"type\": [\"int\", \"null\"]\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "```\n",
    "\n",
    "▶️ **Configure** your dataset as follows:\n",
    "\n",
    "- **Name** as `personalize-movielens-items`\n",
    "- Create a **new schema** named `personalize-movielens-items-schema` using the JSON above ^\n",
    "\n",
    "![](static/imgs/create-items-schema.png \"Screenshot of items schema creation page with configured options\")\n",
    "\n",
    "After checking your configuration, go ahead and click **Next**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the Items Metadata\n",
    "\n",
    "In this step, we'll create a **dataset import job** to read our item metadata from S3, validate and load it into our Amazon Personalize dataset group.\n",
    "\n",
    "▶️ **Name** your import job, like: `personalize-movielens-items-01`\n",
    "\n",
    "▶️ **Select** the same **role** we used for the interactions import (see above)\n",
    "\n",
    "▶️ **Enter** the **S3 URI** of your uploaded *items* CSV.\n",
    "\n",
    "Assuming you prepared your data via Notebook 1, you should be able to get your data URI by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r\n",
    "print(items_s3uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...If you set up manually, your URI should follow a similar structure to the placeholder text.\n",
    "\n",
    "your import job configuration should look something like:\n",
    "\n",
    "![](static/imgs/create-items-import.png \"Screenshot of items dataset import configuration\")\n",
    "\n",
    "Once you've checked your setup, go ahead and click **Start import** to kick off the job.\n",
    "\n",
    "> ⏰ As with interactions, importing the data can take some time - which for our small datasets like our movielens-100k extract is often dominated by **overheads** of starting up and shutting down required processing infrastructure and jobs... Rather than scaling with the number of records.\n",
    ">\n",
    "> For this small sample, the import should take something like 15 minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Imports to Complete\n",
    "\n",
    "We'll need to wait until our dataset imports are completed before training models.\n",
    "\n",
    "You can check up on the summary status of your initial imports on the **Dashboard** for your dataset group\n",
    "\n",
    "...Or from the **details page for each dataset**, accessed through the **Datasets** tab.\n",
    "\n",
    "> ⚠️ **Note:** It's the status of the *import* (not the *dataset*) that matters. You'll notice that when going through the *Datasets* tab - a dataset may show as \"Active\" (the schema is defined & ready) even though it has import jobs still \"In progress\" (records are not yet imported). The dashboard screen shows a simplified summary of the status.\n",
    "\n",
    "![](static/imgs/datasets-list.png \"Screenshot of datasets list view\")\n",
    "\n",
    "![](static/imgs/importing-items-incomplete.png \"Screenshot of incomplete items import\")\n",
    "\n",
    "▶️ **Wait** for both your datasets to have **'Active'** (completed) status before continuing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Record your setup\n",
    "\n",
    "Although we did this setup in the AWS console, we'll want to use our models here in the SageMaker notebook later.\n",
    "\n",
    "To facilitate this, we need to store some values!\n",
    "\n",
    "▶️ **Check** the configurations in the cell below match your setup, and run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHECK:\n",
    "dataset_group_name = \"personalize-poc-lab\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Run** the below, which will automatically determine and store the 'ARN' of your dataset group, from the above and the other values we stored in previous notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r\n",
    "\n",
    "# Use the personalize_role_arn as a base, but adjust:\n",
    "# - service iam->personalize\n",
    "# - region none->region (since IAM is global)\n",
    "# - resource to the Personalize dataset group\n",
    "header, partition, svc, reg, acct_id, resource = personalize_role_arn.split(\":\")\n",
    "svc = \"personalize\"\n",
    "reg = region\n",
    "resource = f\"dataset-group/{dataset_group_name}\"\n",
    "\n",
    "dataset_group_arn = \":\".join((header, partition, svc, region, acct_id, resource))\n",
    "print(dataset_group_arn)\n",
    "%store dataset_group_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All set!\n",
    "\n",
    "We've now created our dataset group (project) in Amazon Personalize and imported our source datasets.\n",
    "\n",
    "In the next notebook we'll create and evaluate some recommendation models based on this data:\n",
    "\n",
    "- Follow along in the **AWS Console** with the instructions and screenshots in [03a_Creating_and_Evaluating_Solutions_(Console).ipynb](03a_Creating_and_Evaluating_Solutions_(Console).ipynb), *OR*\n",
    "- Run the same steps in code with the **AWS SDK for Python (Boto3)** by following [03b_Creating_and_Evaluating_Solutions_(Python_SDK).ipynb](03b_Creating_and_Evaluating_Solutions_(Python_SDK).ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}