Appendix_D_Data_Collection_Scripts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix D: Data Collection Scripts\n",
    "\n",
    "The following scripts are used to collect twitter messages and store them in a format accessible to both python and unicage. While a databasing system has its obvious advantages, this methodology is the paragon of simplicity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Curl URL Builder\n",
    "This script generates a properly signed URL for opening a twitter stream via curl\n",
    "\n",
    "```python\n",
    "\"\"\" twitter_curl_url_builder.py \"\"\"\n",
    "\n",
    "import oauth2 as oauth\n",
    "import time\n",
    "\n",
    "# Set the API endpoint\n",
    "url = 'https://stream.twitter.com/1.1/statuses/sample.json'\n",
    "\n",
    "# Set the base oauth_* parameters along with any other parameters required\n",
    "# for the API call.\n",
    "params = {\n",
    "    'oauth_version': \"1.0\",\n",
    "    'oauth_nonce': oauth.generate_nonce(),\n",
    "    'oauth_timestamp': int(time.time())\n",
    "}\n",
    "\n",
    "# Set up instances of our Token and Consumer.\n",
    "token = oauth.Token(key='*****************************',\n",
    "                    secret='**************************')\n",
    "consumer = oauth.Consumer(key='***********************',\n",
    "                          secret='********************')\n",
    "\n",
    "# Set our token/key parameters\n",
    "params['oauth_token'] = token.key\n",
    "params['oauth_consumer_key'] = consumer.key\n",
    "\n",
    "# Create our request. Change method, etc. accordingly.\n",
    "req = oauth.Request(method=\"GET\", url=url, parameters=params)\n",
    "\n",
    "# Sign the request.\n",
    "signature_method = oauth.SignatureMethod_HMAC_SHA1()\n",
    "req.sign_request(signature_method, consumer, token)\n",
    "\n",
    "print req.to_url()\n",
    "```\n",
    "\n",
    "To use, at the command prompt:\n",
    "```shell\n",
    "$ URL=$(python twitter_curl_kickstarter.py)\n",
    "$ curl -get \"$URL\"\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twitter Stream Opener\n",
    "\n",
    "This script starts a curl process to get posts from twitter and saves them to 100000 post long files. We use the python script `twitter_curl_url_builder.py` to handle the oauth bits, as they can be complicated in bash. \n",
    "\n",
    "```shell\n",
    "#!/bin/bash\n",
    "\n",
    "URL=$(python twitter_curl_url_builder.py)\n",
    "\n",
    "curl --get \"$URL\" | split -l 100000 - ../data/posts_sample_`date \"+%Y%m%d_%H%M%S\"`_\n",
    "\n",
    "echo \"`date` Twitter stream broken with error: ${PIPESTATUS[0]}\" >> tw_collect_log.txt\n",
    "\n",
    "# the curl should go on indefinitely, so if we get to this point, an error has occurred, raise a nonzero flag\n",
    "exit 1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twitter Stream Monitor\n",
    "This script starts the stream, and watches to see if it fails. If so, it restarts the process after some amount of time.\n",
    "\n",
    "For a great description of the watchdog loop, see:\n",
    "http://stackoverflow.com/questions/696839/how-do-i-write-a-bash-script-to-restart-a-process-if-it-dies\n",
    "\n",
    "\n",
    "```shell\n",
    "#!/bin/bash\n",
    "reconnect_delay=600\n",
    "\n",
    "until ./twitter_stream_opener.sh; do\n",
    "    echo \"`date` Twitter curl process interrupted. Attempting reconnect after $reconnect_delay seconds\"\n",
    "\n",
    "    echo \"`date` Twitter curl process interrupted. Attempting reconnect after $reconnect_delay seconds\" >> tw_collect_log.txt\n",
    "\n",
    "    sleep \"$reconnect_delay\"\n",
    "\n",
    "done\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}