Jupyter/Episode II – Attack of the Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie Wars\n",
    "## ~ Episode II – Attack of the Data  ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we should set the notebook so that it outputs all results of each cell and not only the last one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And import all the python libraries needed for this data analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import statsmodels\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's define a **extension for the standard deviation** provided by the statistics package. The problem is that if we try to compute the standard deviation of a set with **less than two elements, we'll get an error**. \n",
    "\n",
    "Theorically, this error is correct, but in this case is appropiate to **define the standard deviation** of a set whose size is less than 2 **as 0**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def std_mod(data):\n",
    "    return statistics.stdev(data) if (len(data) > 1) else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, we state where our data sources for the experiments will be.\n",
    "\n",
    "In our case, they are 3 files about movies, users and ratings that our company **data engineer** has created for us containing a significant sample of our full production database of 26 million ratings, according to our criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder_path = 'data\\\\'\n",
    "\n",
    "movies_file_path = data_folder_path + 'movies.csv'\n",
    "users_file_path = data_folder_path + 'users.csv'\n",
    "rating_file_path = data_folder_path + 'ratings.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_movies = pd.read_csv(movies_file_path, sep = ';', index_col = 'ID')\n",
    "all_users = pd.read_csv(users_file_path, sep = ';', index_col = 'Id')\n",
    "all_ratings = pd.read_csv(rating_file_path, sep = ',')\n",
    "\n",
    "movie_genres = ['Action', 'Adventure', 'Animation', \"Children's\", 'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir', 'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to start with the data analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic analysis - Glance at the raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's describe what information do these datasets have and in what amounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "f\"Movies dataset shape: {all_movies.shape}\"\n",
    "f\"Users dataset shape: {all_users.shape}\"\n",
    "f\"Ratings dataset shape: {all_ratings.shape}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "all_movies.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset contains information about the **title**, **year** and the **genres** asocciated with each movie;\n",
    "but we will need to transform this information into useful columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_users.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset has features related with the **gender**, **age** and **occupation** of the different users, but: \n",
    "- The **Age** column is not showing the real age of users but a mapping to different groups. \n",
    "- The **Occupation** column is filled with integers which are a mapping to their real occupations. \n",
    "\n",
    "We will need additional information on these two columns in order to preprocess them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ratings.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "all_ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ratings are **integers** between 1 and 5, mostly centered around 3 and 4, that have asociated a unique pair of **user and movie ids**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to obtain an approximate rating for each missing user and movie pairs. So the first natural questions that arise are:\n",
    "    \n",
    "   - **Do we have ratings for all the movies?**\n",
    "   - **Do we have ratings for all the users?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_with_votes = list(all_ratings['movie'].unique())\n",
    "users_with_votes = list(all_ratings['user'].unique())\n",
    "\n",
    "f\"Movies with ratings: {len(movies_with_votes)} ({round(len(movies_with_votes)/len(all_movies) * 100, 2)}%)\"\n",
    "f\"Active users: {len(users_with_votes)} ({round(len(users_with_votes)/len(all_users) * 100, 2)}%)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost all the movies has been rated, but **almost half** of the users have **not rated any movie** *(pasive users)*.\n",
    "\n",
    "How we choose to handle this will have deep implications on the model selection stage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deeper analysis - Understanding the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consulting with the domain experts *(or business analysts)* of area where the data comes from, we got back the following information about the **Occupation** and **Age**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_ages = {1: '0 - 17', 18: '18 - 24', 25: '25 - 34', 35: '35 - 44', 45: '45 - 49', 50: '50 - 55', 56: '56+'}\n",
    "\n",
    "user_occupations = { 0:  \"other\", 1:  \"academic/educator\", 2:  \"artist\",  3:  \"clerical/admin\", 4:  \"college/grad student\",\n",
    "                    5:  \"customer service\", 6:  \"doctor/health care\", 7:  \"executive/managerial\", 8:  \"farmer\", 9:  \"homemaker\",\n",
    "                     10:  \"K-12 student\", 11:  \"lawyer\", 12:  \"programmer\", 13:  \"retired\", 14:  \"sales/marketing\", 15:  \"scientist\",\n",
    "                     16:  \"self-employed\", 17:  \"technician/engineer\", 18:  \"tradesman/craftsman\", 19:  \"unemployed\", 20:  \"writer\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a deeper look at the different interesting columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movies: Genre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use **one-hot encoding**, adding a column per genre with 1 if the movie belongs to that genre or 0 otherwise, to create boolean features for the movies' genres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for genre in movie_genres:\n",
    "    all_movies[genre] = all_movies['Genre'].apply(lambda x: 1 if genre in x.split(sep = '|') else 0)\n",
    "\n",
    "all_movies.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it's distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_distribution = pd.DataFrame({\n",
    "    'Genre': movie_genres,\n",
    "    'Proportion': list(map(lambda x : sum(all_movies[x]) / len(all_movies), movie_genres))\n",
    "})\n",
    "\n",
    "shorted_genres = genre_distribution.sort_values(by = 'Proportion', ascending = False)['Genre']\n",
    "\n",
    "genre_barplot = sns.barplot(x = 'Genre', y = 'Proportion', data = genre_distribution, order = shorted_genres)\n",
    "genre_barplot.set_xticklabels(genre_barplot.get_xticklabels(), rotation = 45, horizontalalignment = 'right')\n",
    "genre_barplot.set_title(\"Movies' genres distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that we have a 'dramatic' **umbalance** in the genres distribution, this would be true if if each movie was associated with a **single** genre; but it is not the case. For example: \n",
    "\n",
    "> The American President (1995) belongs to *Comedy*, *Drama* and *Romance* genres. \n",
    "\n",
    "This means that we will need to **normalize** the genre features taking in account the number of genres associated with each movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for genre in movie_genres:\n",
    "    all_movies[genre + '_norm'] = all_movies['Genre'].apply(lambda x: 1/len(x.split(sep = '|')) if genre in x.split(sep = '|') else 0)\n",
    "    all_movies['genre_num'] = all_movies['Genre'].apply(lambda x: len(x.split(sep = '|')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is the resulting normalized genre distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_distribution['Normalized'] = list(map(lambda x : sum(all_movies[x + '_norm']) / len(all_movies), movie_genres))\n",
    "genre_distribution = genre_distribution.rename(columns={\"Proportion\": 'Absolute'})\n",
    "genre_distribution = pd.melt(genre_distribution, id_vars = 'Genre', value_vars= ['Absolute', 'Normalized'], var_name='Type', value_name='Proportion')\n",
    "\n",
    "norm_genre_barplot = sns.barplot(x = 'Genre', y = 'Proportion', data = genre_distribution, hue = 'Type', order = shorted_genres)\n",
    "norm_genre_barplot.set_xticklabels(norm_genre_barplot.get_xticklabels(), rotation = 45, horizontalalignment = 'right')\n",
    "norm_genre_barplot.set_title(\"Movies' genres distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now comparing the distributions, we can see that the previous representation was not fitting the real situation. This second representation of the distribution is sightly different. For a better understanding of the movies genre, we would need more information about the movies like summary, tags and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movies: Year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **year of release** for each movie can be easily extracted as a feature from the title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "all_movies['Year'] = all_movies['Title'].apply(lambda x : int(x[-5:-1]))\n",
    "\n",
    "year_barplot = sns.distplot(all_movies.Year)\n",
    "year_barplot.set_title(\"Movies' years distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that almost all the movies were recorded between **1980** and **2000**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Users: Age"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing the mapping to the users' age ranges, we can take a look at its distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_users['Age interval'] = all_users['Age'].map(lambda age_code: user_ages[age_code])\n",
    "\n",
    "age_distribution = pd.DataFrame({\n",
    "    'Age': list(user_ages.values()),\n",
    "    'Count': [dict(all_users.Age.value_counts())[key] for key in user_ages.keys()],\n",
    "    'Proportion': [dict(all_users.Age.value_counts())[key]/len(all_users) for key in user_ages.keys()]\n",
    "})\n",
    "\n",
    "age_barplot = sns.barplot(x = 'Age', y = 'Proportion', data = age_distribution)\n",
    "age_barplot.set_title(\"Users' age distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The users are centered around **20** to **40** years old, this can cause problems when maiking prediction for the  **old** and **young**. \n",
    "\n",
    "It is also worth mentioning that the intervals have not the same size and that the class mark are the minimun age, when the midpoint could be more representative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Users: Occupation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing the mapping to the users' occupations, we can take a look at its distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_users['Occupation_decoded'] = all_users['Occupation'].map(lambda occupation_code: user_occupations[occupation_code])\n",
    "\n",
    "occupation_distribution = pd.DataFrame({\n",
    "    'Occupation': list(Counter(all_users.Occupation_decoded).keys()),\n",
    "    'Count': list(Counter(all_users.Occupation_decoded).values())\n",
    "})\n",
    "occupations_order = occupation_distribution.sort_values(by = 'Count', ascending = False)['Occupation']\n",
    "\n",
    "occupation_barplot = sns.barplot(x = 'Occupation', y = 'Count', data = occupation_distribution, order = occupations_order)\n",
    "occupation_barplot.set_xticklabels(occupation_barplot.get_xticklabels(), rotation = 45, horizontalalignment = 'right')\n",
    "occupation_barplot.set_title(\"Users' occupation distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's quite **broad** having 20 different occupations, and very **unbalanced** with the 5 most common categories having more than half of the users.\n",
    "\n",
    "This may not be very helpful, so we will have to redesign the occupation feature **by grouping** them to obtain a more balanced and reduced set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Users: Gender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gender of the users is also provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gender_distribution = pd.DataFrame({\n",
    "    'Gender': list(Counter(all_users.Gender).keys()),\n",
    "    'Count': list(Counter(all_users.Gender).values()),\n",
    "    'Proportion': [Counter(all_users.Gender)[key]/len(all_users) for key in Counter(all_users.Gender).keys()]\n",
    "})\n",
    "\n",
    "gender_barplot = sns.barplot(x = 'Gender', y = 'Proportion', data = gender_distribution)\n",
    "gender_barplot.set_title(\"Users' gender distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The majority of our users are males, depending of the type of model chosen we may need to balance the data to avoid a gender bias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ratings [Target]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To recommend movies, we will predict unknown ratings based on known ones, so let's study its distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "rating_distribution = pd.DataFrame({\n",
    "    'Rating': list(Counter(all_ratings.rating).keys()),\n",
    "    'Count': list(Counter(all_ratings.rating).values()),\n",
    "    'Proportion': [Counter(all_ratings.rating)[key]/len(all_ratings) for key in Counter(all_ratings.rating).keys()]\n",
    "})\n",
    "\n",
    "rating_barplot = sns.barplot( x = 'Rating', y = 'Proportion', data = rating_distribution)\n",
    "rating_barplot.set_title('Ratings distribution');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neutral and positive ratings (3, 4, & 5)represents an 86% of the total. This tell us that people tend to rate more the movies they like, so in a sense **low ratings are more valuable**. \n",
    "\n",
    "Ratings are heavily centered around 3.6, but extreme values are more important from a business perspective. This could cause a loss of effectiveness for the models, it should be taken in consideration when choosing the **metrics** to compare the models performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Marginal distribution: Mean ratings by movie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the relation between the **mean** of the ratings associated with each movie and its **deviations**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "movies_ratings = list(map(lambda movie: list(all_ratings.loc[all_ratings['movie'] == movie]['rating']), all_ratings['movie'].unique()))\n",
    "\n",
    "ratings_by_movie = pd.DataFrame({\n",
    "    'Movie': list(all_ratings['movie'].unique()),\n",
    "    'Mean rating': list(map(lambda ratings: sum(ratings)/len(ratings), movies_ratings)),\n",
    "    'Rating deviation': list(map(lambda ratings: std_mod(ratings), movies_ratings)),\n",
    "    'Rating count': list(map(lambda ratings: len(ratings), movies_ratings))\n",
    "})\n",
    "\n",
    "ratings_by_movie_regplot = sns.regplot(x = 'Mean rating', y = 'Rating deviation', data = ratings_by_movie, order = 2 , line_kws = {'color': 'red'})\n",
    "ratings_by_movie_regplot.set_title('Mean vs deviation of ratings by movie');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some movies with **extreme** values of deviation in its ratings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Marginal distribution: Mean ratings by user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also examine the relation between the **mean** of the ratings associated with each user and its **deviations**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "users_ratings = list(map(lambda user: list(all_ratings.loc[all_ratings['user'] == user]['rating']), all_ratings['user'].unique()))\n",
    "\n",
    "ratings_by_user = pd.DataFrame({\n",
    "    'User': list(all_ratings['user'].unique()),\n",
    "    'Mean rating': list(map(lambda ratings: sum(ratings)/len(ratings), users_ratings)),\n",
    "    'Rating deviation': list(map(lambda ratings: std_mod(ratings), users_ratings)),\n",
    "    'Rating count': list(map(lambda ratings: len(ratings), users_ratings))\n",
    "})\n",
    "\n",
    "ratings_by_user_regplot = sns.regplot(x = 'Mean rating', y = 'Rating deviation', data = ratings_by_user, order = 2 , line_kws = {'color': 'red'})\n",
    "ratings_by_user_regplot.set_title('Mean vs deviation of ratings by user');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some **infrequent extreme behaviours** among the users, such as: users who only give negative or positive feedback, (they have low and high mean rating but low deviations). \n",
    "\n",
    "These extreme user behaviours can be very harmful for modelling, we should treat them as **outliers**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Marginal distribution: Acumulated ratings by user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's check the **amount** of movies that users rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accumulated_ratings = sns.kdeplot(ratings_by_user['Rating count'], cumulative = True, cut = 0)\n",
    "accumulated_ratings.axvline(30, color = 'red')\n",
    "accumulated_ratings.set_title('Acumulated distribution of movies rated count');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are very few users that have rated less than 30 movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At last but not at least, let's look at the full picture to draw some preliminary conclusions:\n",
    "- The dataset is extensive enough for our experiments.\n",
    "- We have movies and users with and without ratings.\n",
    "- All categories (genres, ages, and occupations) are represented.\n",
    "- Most movies have scores that aggreed upon most users, with a deviation going from 0.5 to 1.25.\n",
    "- Few movies have a very well defined score, and there are such movies for each possible score.\n",
    "- Few movies have very conflicting scores.\n",
    "- Users tends to be rate more the movies that they love than the ones that they hate.\n",
    "- There exists some groups of especial interest, such as \"Haters\" and \"Lovers\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And save the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_movies.to_csv(data_folder_path + 'movies_with_genre_and_year.csv', sep = ';')\n",
    "all_users.to_csv(data_folder_path + 'users_with_age_interval_and_occupation.csv', sep = ';')\n",
    "\n",
    "ratings_by_movie.to_csv(data_folder_path + 'ratings_by_movie_data.csv', sep = ';')\n",
    "ratings_by_user.to_csv(data_folder_path + 'ratings_by_user_data.csv', sep = ';')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, looks like a representative sample of our production dataset and in acordance with what we would expect of our company users."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}