Recommendation Engine .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Project - Build a recommendation engine which will predict the products to be purchased by an Instacart consumer again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Whether you shop from meticulously planned grocery lists or let whimsy guide your grazing, our unique food rituals define who we are. Instacart, a grocery ordering and delivery app aim to make it easy to fill your refrigerator and pantry with your personal favorites and staples when you need them. After selecting products through the Instacart app, personal shoppers review your order and do the in-store shopping and delivery for you.\n",
    "\n",
    "## Instacart’s data science team plays a big part in providing this delightful shopping experience. Currently, they use transactional data to develop models that predict which products a user will buy again, try for the first time, or add to their cart next during a session. Recently, Instacart open-sourced this data - see their blog post on 3 Million Instacart Orders, Open Sourced.\n",
    "\n",
    "## In this data science project, we are going to use this anonymized data on customer orders over time to predict which previously purchased products will be in a user’s next order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Datasets\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/aisles.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/departments.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/sample_submission.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/order_products__prior.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/order_products__train.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/orders.csv\n",
    "#https://s3.amazonaws.com/hackerday.datascience/87/products.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This problem can be solved using three different approaches:\n",
    "\n",
    "#1. Standard MBA or Market basket analysis (association rules or arules)\n",
    "\n",
    "#2. using a predictive model to estimate the demand for a particular product\n",
    "\n",
    "#3. Product recommendation engine using collaborative filtering\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Step we are going to follow in this hackerday\n",
    "\n",
    "#1. read all the csv files\n",
    "\n",
    "#2. join the relevant files\n",
    "\n",
    "#3. doing exploratory data analysis\n",
    "\n",
    "#4. getting into the solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#1 = dry fruits\n",
    "#2 = coffee\n",
    "#1 = soups\n",
    "#2 = milk\n",
    "#3 = vegetables\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transaction set\n",
    "\n",
    "#Invoice 1 = {apple, oranges, rice, wheat, milk}\n",
    "#Invoice 2 = {rice, milk, butter, bread, fruits , apple}\n",
    ".\n",
    ".\n",
    ".\n",
    "#Invoice 20 = {...........}\n",
    "\n",
    "#total transation is 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ARULES\n",
    "\n",
    "# Support = proportion of transactions in the invoice data which contains an iten set\n",
    "# X = {milk, rice}\n",
    "# Support of X = 2/20\n",
    "\n",
    "# Confidence = confidence of a rule\n",
    "# rule of buying apple(Y) given that the user has already added X\n",
    "# Confidence of (X > Y) = Sup(X U Y) / Sup(X)\n",
    "# X = LHS and Y = RHS\n",
    "# Confidence = 2/2 = 100\n",
    "\n",
    "# Lift = lift of a specific rule\n",
    "# Lift (X > Y) = Sup(X U Y) / Sup(X) * Sup(Y)\n",
    "# Lift = 2/(2*2) = 2/4 = 1/2 = 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Steps in creating Association rules\n",
    "# Apply the minimum support criteria to identify most frequent item set\n",
    "# these frequent item sets and the minimum confidence constraint are used to form rules\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# eclat algorithm\n",
    "# apriori algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# algorithmic definition\n",
    "# if a set of products are most frequent in a dataset, then the constituents of the most\n",
    "# frequent set are also called most frequent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------\n",
      "You have loaded plyr after dplyr - this is likely to cause problems.\n",
      "If you need functions from both plyr and dplyr, please load plyr first, then dplyr:\n",
      "library(plyr); library(dplyr)\n",
      "------------------------------------------------------------------------------\n",
      "\n",
      "Attaching package: ‘plyr’\n",
      "\n",
      "The following objects are masked from ‘package:dplyr’:\n",
      "\n",
      "    arrange, count, desc, failwith, id, mutate, rename, summarise,\n",
      "    summarize\n",
      "\n"
     ]
    },
    {
     "ename": "ERROR",
     "evalue": "Error in library(Hmisc): there is no package called ‘Hmisc’\n",
     "output_type": "error",
     "traceback": [
      "Error in library(Hmisc): there is no package called ‘Hmisc’\n"
     ]
    },
    {
     "ename": "ERROR",
     "evalue": "Error in library(arules): there is no package called ‘arules’\n",
     "output_type": "error",
     "traceback": [
      "Error in library(arules): there is no package called ‘arules’\n"
     ]
    },
    {
     "ename": "ERROR",
     "evalue": "Error in library(arulesViz): there is no package called ‘arulesViz’\n",
     "output_type": "error",
     "traceback": [
      "Error in library(arulesViz): there is no package called ‘arulesViz’\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Attaching package: ‘data.table’\n",
      "\n",
      "The following objects are masked from ‘package:dplyr’:\n",
      "\n",
      "    between, last\n",
      "\n"
     ]
    }
   ],
   "source": [
    "library(dplyr)\n",
    "library(ggplot2)\n",
    "library(plyr)\n",
    "library(Hmisc)\n",
    "library(readr)\n",
    "library(arules)\n",
    "library(arulesViz)\n",
    "library(stringr)\n",
    "library(data.table)\n",
    "library(methods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "orders <- fread(\"https://s3.amazonaws.com/hackerday.datascience/87/orders.csv\")\n",
    "orders\n",
    "\n",
    "products<-fread('https://s3.amazonaws.com/hackerday.datascience/87/products.csv')\n",
    "products\n",
    "\n",
    "order_products_prior <- fread('https://s3.amazonaws.com/hackerday.datascience/87/order_products__prior.csv')\n",
    "order_products_prior\n",
    "\n",
    "order_products_train <- fread('https://s3.amazonaws.com/hackerday.datascience/87/order_products__train.csv')\n",
    "order_products_train\n",
    "\n",
    "aisles <- fread('https://s3.amazonaws.com/hackerday.datascience/87/aisles.csv')\n",
    "aisles\n",
    "\n",
    "departments <- fread('https://s3.amazonaws.com/hackerday.datascience/87/departments.csv')\n",
    "departments\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# which day of the week the company received most orders?\n",
    "\n",
    "orders$day_week_name <- \n",
    "  ifelse(orders$order_dow == 0,\n",
    "         'Sunday',\n",
    "         ifelse(orders$order_dow == 1,\n",
    "                'Monday',\n",
    "                ifelse(orders$order_dow == 2,\n",
    "                       'Tuesday',\n",
    "                       ifelse(orders$order_dow == 3,\n",
    "                              'Wednesday',\n",
    "                              ifelse(orders$order_dow == 4,\n",
    "                                     'Thursday',\n",
    "                                     ifelse(orders$order_dow == 5,\n",
    "                                            'Friday',\n",
    "                                            ifelse(orders$order_dow == 6,\n",
    "                                                   'Saturday',\"\")))))))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "orders$day_ordered <- factor(orders$day_week_name,levels = c(\"Sunday\",\n",
    "                                                             \"Monday\",\n",
    "                                                             \"Tuesday\",\n",
    "                                                             \"Wednesday\",\n",
    "                                                             \"Thursday\",\n",
    "                                                             \"Friday\",\n",
    "                                                             \"Saturday\"))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# visualization of orders placed by different days of the week\n",
    "\n",
    "dow_graph <- barplot(\n",
    "  table(orders$day_ordered),\n",
    "  main = \"Total Orders by Day\",\n",
    "  xlab = 'Days',\n",
    "  ylab = 'Number of Orders',\n",
    "  col = 'blue')\n",
    "\n",
    "text(\n",
    "  x = dow_graph,\n",
    "  y = table(orders$day_ordered),\n",
    "  labels = table(orders$day_ordered),\n",
    "  pos = 1,\n",
    "  cex = 1.0,\n",
    "  col = 'white'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "##########EDA##############\n",
    "orders <- fread(\"https://s3.amazonaws.com/hackerday.datascience/87/orders.csv\")\n",
    "orders\n",
    "\n",
    "products<-fread('https://s3.amazonaws.com/hackerday.datascience/87/products.csv')\n",
    "products\n",
    "\n",
    "order_products_prior <- fread('https://s3.amazonaws.com/hackerday.datascience/87/order_products__prior.csv')\n",
    "order_products_prior\n",
    "\n",
    "order_products <- fread('https://s3.amazonaws.com/hackerday.datascience/87/order_products__train.csv')\n",
    "order_products\n",
    "\n",
    "\n",
    "aisles <- fread('https://s3.amazonaws.com/hackerday.datascience/87/aisles.csv')\n",
    "aisles\n",
    "\n",
    "departments <- fread('https://s3.amazonaws.com/hackerday.datascience/87/departments.csv')\n",
    "departments\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(knitr)\n",
    "kable(head(orders,12))\n",
    "\n",
    "kable(head(order_products,10))\n",
    "\n",
    "kable(head(products,10))\n",
    "\n",
    "kable(head(order_products_prior,10))\n",
    "\n",
    "kable(head(aisles,12))\n",
    "\n",
    "kable(head(departments,12))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Recoding the variables\n",
    "\n",
    "orders <- orders %>%\n",
    "  mutate(order_hour_of_day = as.numeric(order_hour_of_day),\n",
    "         eval_set = as.factor(eval_set))\n",
    "\n",
    "products <- products %>%\n",
    "  mutate(product_name = as.factor(product_name))\n",
    "\n",
    "aisles <- aisles %>%\n",
    "  mutate(aisle = as.factor(aisle))\n",
    "\n",
    "departments <- departments %>%\n",
    "  mutate(department = as.factor(department))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###graphs/visualization for orders by hour of the day\n",
    "orders %>%\n",
    "  ggplot(aes(x=order_hour_of_day)) +\n",
    "  geom_histogram(stat = 'count',fill='blue')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# orders by the day of the week\n",
    "\n",
    "#Hypothesis\n",
    "# Is there any effect of day of the week on orders\n",
    "\n",
    "orders %>%\n",
    "  ggplot(aes(x=order_dow)) +\n",
    "  geom_histogram(stat = 'count',fill='red')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#conclusion: most orders are placed on Sunday==0, and Monday==1\n",
    "\n",
    "#Hypothesis\n",
    "# Do people order more often after exactly 1 week?\n",
    "orders %>%\n",
    "  ggplot(aes(x=days_since_prior_order)) +\n",
    "  geom_histogram(stat='count',fill='orange')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#conclusion: yes they do order more often after exactly 1 week\n",
    "\n",
    "# Question: how many prior orders placed?\n",
    "orders %>% filter(eval_set=='prior') %>% count_(orders,'order_number') %>%\n",
    "  ggplot(aes(order_number,n)) + geom_line(color='red',size=1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from the training set\n",
    "\n",
    "order_products %>%\n",
    "  group_by(order_id) %>%\n",
    "  summarise(n_items=last(add_to_cart_order)) %>%\n",
    "  ggplot(aes(x=n_items)) +\n",
    "  geom_histogram(stat = 'count',fill='red') +\n",
    "  geom_rug() #+\n",
    "  #coord_cartesian(xlim = c(0,80))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "order_products_prior %>%\n",
    "  group_by(order_id) %>%\n",
    "  summarise(n_items=last(add_to_cart_order)) %>%\n",
    "  ggplot(aes(x=n_items)) +\n",
    "  geom_histogram(stat = 'count',fill='red') +\n",
    "  geom_rug() #+\n",
    "#coord_cartesian(xlim = c(0,80))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "######################################\n",
    "order_products_prior[1:10,]\n",
    "products[1:10,]\n",
    "\n",
    "mydata <- order_products_prior[,1:2]\n",
    "mydata <- merge(mydata,products,by='product_id')\n",
    "\n",
    "mydata <- arrange(mydata,order_id)\n",
    "head(mydata)\n",
    "\n",
    "mydata<- mydata[,c(2,3)]\n",
    "head(mydata)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dataset is disjoint\n",
    "# for market basket anlysis we would need transactional dataset\n",
    "# how to convert the available information to a transactional dataset\n",
    "\n",
    "dt <- split(mydata$product_name,mydata$order_id)\n",
    "\n",
    "dt2 = as(dt,'transactions')\n",
    "\n",
    "summary(dt2)\n",
    "\n",
    "inspect(dt2)[[5]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#visualiza the most frequent item sets in this dataset\n",
    "itemFrequency(dt2,type='relative')\n",
    "itemFrequencyPlot(dt2,topN=20,type='relative')\n",
    "itemFrequencyPlot(dt2,topN=50,type='absolute')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#create rules\n",
    "rule_1 = apriori(dt2,parameter = list(support=0.00001,\n",
    "                                      confidence=0.90))\n",
    "library(RColorBrewer)\n",
    "plot(rule_1,control = list(col=brewer.pal(11,\"Spectral\")),main=\"\")\n",
    "\n",
    "\n",
    "rule_2 = apriori(dt2,parameter = list(support=0.0001,\n",
    "                                      confidence=0.90))\n",
    "plot(rule_2,control = list(col=brewer.pal(11,\"Spectral\")),main=\"\")\n",
    "\n",
    "\n",
    "rule_3 = apriori(dt2,parameter = list(support=0.001,\n",
    "                                      confidence=0.90))\n",
    "plot(rule_3,control = list(col=brewer.pal(11,\"Spectral\")),main=\"\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary(rule_3)\n",
    "\n",
    "rule_4 <- apriori(dt2,\n",
    "                  parameter = list(support=0.001,\n",
    "                                   confidence=0.8,\n",
    "                                   minlen=3))\n",
    "\n",
    "rule_5 <- apriori(dt2,\n",
    "                  parameter = list(support=0.001,\n",
    "                                   confidence=0.8,\n",
    "                                   maxlen=4))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# converting the rules into a data frame\n",
    "rules3 = as(rule_3,'data.frame')\n",
    "\n",
    "inspect(subset(rule_3,subset= rhs %pin% 'Banana'))\n",
    "\n",
    "# before recommending the products to the company you can sort the rules\n",
    "inspect(head(sort(rule_3,by='lift'),5))\n",
    "\n",
    "summary(rule_3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "plot(rule_3,method = 'graph',control = list(type='items',main=''))\n",
    "\n",
    "subrule3 <- head(sort(rule_3,by='lift'),10)\n",
    "\n",
    "plot(subrule3,method = 'graph',control = list(type='items',main=''))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# shall we continue with the existing set of rules?\n",
    "\n",
    "# NO, because we need to clear the redundant rules from the set\n",
    "\n",
    "# identify the unnecessary rules\n",
    "subset.matrix = is.subset(rule_3,rule_3)\n",
    "subset.matrix[lower.tri(subset.matrix,diag = T)] <- NA\n",
    "\n",
    "redundant = colSums(subset.matrix, na.rm = T) >= 1\n",
    "which(redundant)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "rule3_pruned <- rule_3[!redundant]\n",
    "rules<-rule3_pruned\n",
    "\n",
    "# clean the rules\n",
    "inspect(rules)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda3]",
   "language": "python",
   "name": "conda-env-Anaconda3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}