1-web-scrape-images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "\n",
    "**Objective**\n",
    "\n",
    "Scrape images from Kickstarter from a random subset of the web robots kickstarter dataset which conatains mostly meta data but also has image url information. We will use this url information to download the full images from the selected subset. The images will be saved to a local directory with the image id as the file name. These files will later be analyzed using custom python function to extract low-level structral features form the images.\n",
    "\n",
    "**Resources**\n",
    "\n",
    "Web_robots_data metadata for kickstarter.com campaigns - source: \n",
    "\n",
    "https://webrobots.io/kickstarter-datasets/\n",
    "\n",
    "\n",
    "Author: Nicholas Mostovych"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## for data\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "import pickle\n",
    "\n",
    "# for webscraping\n",
    "import requests as r\n",
    "import nltk\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "import lxml\n",
    "\n",
    "# other\n",
    "import time\n",
    "import random\n",
    "\n",
    "## Interactive env\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from IPython.core.display import clear_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parse html objects using BeautifulSoup\n",
    "def parse(scraped_html):\n",
    "    \"\"\"\n",
    "    Use the BeautifulSoup library to parse the scraped HTML of a project \n",
    "    using an lxml parser\n",
    "    \n",
    "    Args:\n",
    "        scraped_html (response object): the unparsed response object collected\n",
    "        by the web scraper\n",
    "    \n",
    "    Returns:\n",
    "        a soup object containing the parsed HTML\n",
    "    \"\"\"\n",
    "    return BeautifulSoup(scraped_html.text, 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paid proxy information stored in local file, not shared on github. Retrieve this info.\n",
    "def get_proxy_data(column):\n",
    "    \"\"\"\n",
    "    Function to use with get_proxy() function to extract specfific proxy data\n",
    "    \n",
    "    Args:\n",
    "        ip = 0\n",
    "        port_http = 1\n",
    "        port_socks = 2\n",
    "        login = 3\n",
    "        password = 4\n",
    "        \n",
    "    \n",
    "    Returns:\n",
    "        proxy data in list form\n",
    "    \"\"\"\n",
    "    ### directory information\n",
    "    f = open('/home/mosto/Documents/insight/kickstarter-project/proxylist.csv', \"r\")\n",
    "    \n",
    "    ### get proxy data\n",
    "    lines = f.readlines()\n",
    "    result=[]\n",
    "    for x in lines:\n",
    "        result.append(x.split('\\t')[column])\n",
    "    f.close()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grab my proxy data\n",
    "def get_proxy():\n",
    "    \"\"\"\n",
    "    Function to pick a random proxy and assemble the correct url\n",
    "    \n",
    "    Args:\n",
    "        none\n",
    "        \n",
    "    \n",
    "    Returns:\n",
    "        random proxy url \n",
    "    \"\"\"\n",
    "    \n",
    "    ### Retrieve proxy information \n",
    "    proxy_ip = get_proxy_data(0)\n",
    "    del proxy_ip[0]\n",
    "    port_http = get_proxy_data(1)\n",
    "    port_socks = get_proxy_data(2)\n",
    "    login = get_proxy_data(3)\n",
    "    password = get_proxy_data(4)\n",
    "\n",
    "    ### Pick a random proxy to use\n",
    "    in_proxy_ip = random.choice(proxy_ip)\n",
    "    user = login[1]\n",
    "    passwd = password[1]\n",
    "    passwd = passwd.rstrip()\n",
    "    ip = in_proxy_ip\n",
    "    port = port_http[1]\n",
    "    \n",
    "    ### Assemble the url\n",
    "    proxy_url = \"http://\" + user + \":\" + passwd + \"@\" + ip + \":\" + port + \"/\"\n",
    "    return {\n",
    "        \"http\": proxy_url,\n",
    "        \"https\": proxy_url\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Data and Organize Data Subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Webscrapped meta-data\n",
    "datafile = '/home/mosto/Documents/insight/kickstarter-project/web_robots_data_to_08-2020_processed.pkl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load table containing Web Robots data\n",
    "df = joblib.load(datafile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 17)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Obtain subset of images that were scrapped from the large web robots dataset\n",
    "'''\n",
    "A subset of images was randomly sampled from the large web robots dataset and then\n",
    "the images were downloaded from Kickstarter.com. These randomly selected images are\n",
    "specified using the random seed number of 74 and then this dataset looks at the first\n",
    "10000 images of this dataset\n",
    "'''\n",
    "\n",
    "# Select projects in USD\n",
    "df_USD = df[df['currency'] == 'USD']\n",
    "# Take a random sample of the Web Robots data using a seed value to ensure repeatability\n",
    "seed = np.random.seed(74)\n",
    "df_sample = df_USD.sample(50000)\n",
    "df1_sample=df_sample.iloc[:10000]\n",
    "df1_sample.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Request: 2; Row ID: 52182; Frequency: 0.17123741158750425 requests/sec\n",
      "\n",
      "Run time: 17.52166175842285\n",
      "Average rate: 0.17121663694699893\n",
      "# of images downloaded: 3\n"
     ]
    }
   ],
   "source": [
    "# This will scrape images to the local directory with the image file names = image id from web robots dataset\n",
    "\n",
    "# Rename dataframe for scraping\n",
    "df = df1_sample\n",
    "\n",
    "# Initalize an empty DataFrame to store scraped HTML names as a reference\n",
    "images_collection = pd.DataFrame(columns=['image'])\n",
    "\n",
    "# Record the start time\n",
    "start_time = time.time()\n",
    "\n",
    "# Initialize the number of requests\n",
    "request_count = 0\n",
    "\n",
    "# Select which projects to scrape via its index. If scrapper fails for some reason we can pick up where we left off\n",
    "start = 0\n",
    "stop = 3\n",
    "\n",
    "# Set up for loop to scrape all images in df\n",
    "for row in range(start, stop, 1):\n",
    "    \n",
    "    # instantiate a new random proxy\n",
    "    current_proxy = get_proxy()     \n",
    "    # make index variable \n",
    "    index = df.index[row] \n",
    "    \n",
    "    # Scrape HTML url\n",
    "    try:\n",
    "        \n",
    "        scraped_html = r.get(df.iloc[row]['url'],proxies=current_proxy)\n",
    "        soup = parse(scraped_html)\n",
    "        # Get image urls\n",
    "        img = soup.find_all('img',src=True)        \n",
    "        # Select first image which is the cover photo\n",
    "        img_url = img[0]['src']\n",
    "        # Wait to lower chances of getting blocked by website during scraping \n",
    "        time.sleep(random.uniform(1,2))\n",
    "        \n",
    "        # Open image\n",
    "        res=r.get(img_url,proxies=current_proxy)\n",
    "        con=res.content\n",
    "        # Save image \n",
    "        out=open(str(index),'wb')\n",
    "        out.write(con)\n",
    "        out.close()\n",
    "    \n",
    "        # Record scraped HTML name to reference dataframe\n",
    "        images_collection.loc[index, 'image'] = str(index)\n",
    "    \n",
    "    # Error handling    \n",
    "    except IndexError:\n",
    "        #If there are no images, we will get an index error in the img[0] expression, so we skip it\n",
    "        continue\n",
    "    except ProxyError:\n",
    "        # Sometimes proxy fails, so we skip it\n",
    "        continue\n",
    "    except OSError:\n",
    "        # Sometimes proxy fails, so we skip it\n",
    "        continue\n",
    "    \n",
    "    # Wait again because we pinged the same site twice\n",
    "    time.sleep(random.uniform(2,4)) \n",
    "    \n",
    "    \n",
    "    # Monitor the requests by clearing the output and displaying current \n",
    "    # progress\n",
    "    elapsed_time = time.time() - start_time\n",
    "    clear_output(wait = True)\n",
    "    print(\n",
    "        'Request: {}; Row ID: {}; Frequency: {} requests/sec'.format(\n",
    "            request_count + start,\n",
    "            index,\n",
    "            (request_count + 1) / elapsed_time\n",
    "        )\n",
    "    )\n",
    "    request_count += 1\n",
    "    \n",
    "    \n",
    "# Display the overall time, average scraping speed and total number of scraped\n",
    "# project pages\n",
    "run_time = time.time() - start_time\n",
    "print()\n",
    "print('Run time:', run_time)\n",
    "print('Average rate:', len(images_collection) / run_time)\n",
    "print('# of images downloaded:', len(images_collection))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/home/mosto/Documents/insight/kickstarter-project/df1_dynamic_scrape_scraped_collection_0-2.pkl']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save reference dataframe to know which images you have downloarded to the directory\n",
    "# Serialize the data table containing the scraped HTML for each project\n",
    "joblib.dump(\n",
    "    images_collection, '/home/mosto/Documents/insight/kickstarter-project/df1_dynamic_scrape_scraped_collection_{}-{}.pkl'.format(\n",
    "        start,\n",
    "        stop - 1\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}