Data Science Workshop 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Real Data from the Web"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Semestral Project*: build a statistical model from predicting real estate prices based on property features from data placed on ad site(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Curriculum\n",
    "\n",
    "## Getting the data\n",
    "### - Setting up the environement\n",
    "### - Anatomy of a Spider\n",
    "### - Anatomy of a Web Page\n",
    "### - Scrapy - a scraping framework\n",
    "### - Beautiful Soup\n",
    "### - Managing the Crawling Frontier\n",
    "### - Crawling Ethical Aspects \n",
    "## Processing the data\n",
    "### - Descriptive Analytics\n",
    "### - Feature Engineering\n",
    "### - Price Determining Factors\n",
    "## Price Prediction\n",
    "### - Training the model\n",
    "### - Model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Spider* - a program that:\n",
    "- visits web pages from a list called *the frontier*\n",
    "- parses them to extract information - data and links\n",
    "- manages the _crawling frontier_ (links to be visited)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://packtpub.com/packt/offers/free-learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://docs.scrapy.org/en/latest/topics/spiders.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You start by generating the initial Requests to crawl the first URLs, and specify a callback function to be called with the response downloaded from those requests.\n",
    "\n",
    "In the callback function, you parse the response (web page) and return either dicts with extracted data, Item objects, Request objects, or an iterable of these objects.\n",
    "\n",
    "In callback functions, you parse the page contents, typically using Selectors \n",
    "\n",
    "Finally, the items returned from the spider will be typically persisted to a database or written to a file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrapy\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "class PythonEventsSpider(scrapy.Spider):\n",
    "    name = 'pythoneventsspider'\n",
    "\n",
    "    start_urls = ['https://www.python.org/events/python-events/',]\n",
    "    found_events = []\n",
    "\n",
    "    def parse(self, response):\n",
    "        for event in response.xpath('//ul[contains(@class, \"list-recent-events\")]/li'):\n",
    "            event_details = dict()\n",
    "            event_details['name'] = event.xpath('h3[@class=\"event-title\"]/a/text()').extract_first()\n",
    "            event_details['location'] = event.xpath('p/span[@class=\"event-location\"]/text()').extract_first()\n",
    "            event_details['time'] = event.xpath('p/time/text()').extract_first()\n",
    "            self.found_events.append(event_details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'PyCon Odessa', 'location': 'BC «Solnechnyiy», 5, Sonyachna St., Odessa, Ukraine', 'time': '16 March'}\n",
      "{'name': 'IndyPy Automate Conf 2019', 'location': 'Indiana, USA', 'time': '22 March'}\n",
      "{'name': 'PyCon SK 2019', 'location': 'Bratislava, Slovakia', 'time': '22 March – 24 March '}\n",
      "{'name': 'Moscow Python Conf++ 2019', 'location': 'Moscow, Russia', 'time': '05 April'}\n",
      "{'name': 'DjangoCon Europe 2019', 'location': 'Copenhagen, Denmark', 'time': '10 April – 14 April '}\n",
      "{'name': 'PythonCamp 2019 - Cologne', 'location': 'GFU Cyrus AG, Am Grauen Stein 27,,51105 Köln, Germany', 'time': '13 April – 14 April '}\n",
      "{'name': 'PyCon APAC 2019', 'location': 'iACADEMY, Nexus Campus, Makati City, Philippines', 'time': '23 Feb. – 24 Feb. '}\n",
      "{'name': 'PyCascades 2019', 'location': 'Seattle, Washington, USA', 'time': '23 Feb. – 24 Feb. '}\n"
     ]
    }
   ],
   "source": [
    "process = CrawlerProcess({ 'LOG_LEVEL': 'ERROR'})\n",
    "process.crawl(PythonEventsSpider)\n",
    "spider = next(iter(process.crawlers)).spider\n",
    "process.start()\n",
    "\n",
    "for event in spider.found_events: print(event)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}