Lectures/L-HTTP-REST-API/Ex-02-REST-API-wdata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to run this service\n",
    "\n",
    "* Launch a terminal and enter into the directory, that contains this notebook\n",
    "* then type\n",
    "```\n",
    "preview-nb-api.sh Ex-02-REST-API-wdata.ipynb \n",
    "```\n",
    "* the first line of output will tell you the URL of the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Facilities and Educational Data for Boston Public Schools\n",
    "Data is acquired from here: https://data.boston.gov/dataset/buildbps-facilities-and-educational-data-for-boston-public-schools\n",
    "where the terms are explained in a  pdf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T12:28:34.939036Z",
     "start_time": "2021-03-09T12:28:34.934746Z"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "# Kooplex related variables\n",
    "api_url = os.path.join(\"https://kooplex-edu.elte.hu/\", os.getenv(\"NB_URL\"), \"report\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-06T07:58:25.852661Z",
     "start_time": "2020-12-06T07:58:25.841316Z"
    }
   },
   "outputs": [],
   "source": [
    "fname = 'data/Globalco2emissions-original.csv'\n",
    "df = pd.read_csv(fname, parse_dates=[1])\n",
    "#df['Year'] = df['Year'].dt.strftime('%Y')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-06T06:55:59.475953Z",
     "start_time": "2020-12-06T06:55:59.471921Z"
    }
   },
   "outputs": [],
   "source": [
    "# for debugging\n",
    "REQUEST = json.dumps({\n",
    " 'args' : {}\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /\n",
    "help_txt = (\"Please visit %s/help !\"%api_url)\n",
    "print(help_txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /help\n",
    "help_txt = (\"\"\"\n",
    "<h2>HELP</h2>\n",
    "<h4>GET /info</h4>\n",
    "Description: Retrieve either some informatin about the dataset\n",
    "Requires: No parameters required\n",
    "Example query: /api/info\n",
    "Example result: {\"name\": \"Global CO2 emissions - CDIAC and UN Population\", \"title\": \"Global CO2 emissions .... }\n",
    "\n",
    "<h4>GET /data</h4>\n",
    "Description: Retrieves either the all the data, or the from and until the given dates\n",
    "Requires: \n",
    "<ul>\n",
    "<li> from (int)</li>\n",
    "<li> until (int)</li>\n",
    "<li> avg (Bool) (optional): calculates average</li>\n",
    "</ul>\n",
    "Returns: \n",
    "<ul>\n",
    "<li> year</li>\n",
    "<li> metric tons per capita </li>\n",
    "</ul>\n",
    "Example query: /api/data?from=1965&until=1988\n",
    "\n",
    "<h4>POST /api/data/</h4>\n",
    "Description: Adds an extra data point into the dataset for the given 'year' with the given 'value'\n",
    "Requires: \n",
    "<ul>\n",
    "<li> year (int) </li>\n",
    "<li> value (float)</li>\n",
    "</ul>\n",
    "Returns: \n",
    "<ul>\n",
    "<li> Status 200: upload successful</li>\n",
    "</ul>\n",
    "Example query: /api/data?year=2016&value=3.9152\n",
    "\n",
    "<h4>PUT /data/</h4>\n",
    "Description: Replaces anexisting data point in the dataset for the given 'year' with the given 'value'\n",
    "Requires: \n",
    "<ul>\n",
    "<li> year (int) </li>\n",
    "<li> value (float)</li>\n",
    "</ul>\n",
    "Returns: \n",
    "<ul>\n",
    "<li> Status 200: upload successful</li>\n",
    "</ul>\n",
    "Example query: /api/data?year=2016&value=3.9152\n",
    "\n",
    "<h4>GET /agg</h4>\n",
    "<ul>\n",
    "<li>Description: Retrieves an aggregated sum of the CO2 consumption for the last <years>. </li>\n",
    "<li>Requires: years </li>\n",
    "<li>Returns: a number</li>\n",
    "<li>Example query: /api/agg?interval=6</li>\n",
    "</ul>\n",
    "\"\"\")\n",
    "help_txt=help_txt.replace('\\n',\"\")\n",
    "\n",
    "print(help_txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ResponseInfo GET /help\n",
    "print(json.dumps({\n",
    "    \"headers\" : {\n",
    "        \"Content-Type\" : \"text/html\"\n",
    "    },\n",
    "    \"status\" : 200\n",
    "}))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /api/info\n",
    "req = json.loads(REQUEST)\n",
    "args = req['args']\n",
    "\n",
    "resp = {\n",
    "    \"name\":\"Global CO2 emissions - CDIAC and UN Population\",\n",
    "    \"title\":\"Global CO2 emissions - CDIAC and UN Population\",\n",
    "    \"dataset_id\": 652,\n",
    "    \"description\":\"Per capita CO2 emissions have been calculated based on the combination of global emissions data from CDIAC (described below), and UN Population Prospects data.\\r\\n\\r\\nPopulation data was derived from: The History Database of the Global Environment (HYDE) collected the data by earlier publications. For the 'OurWorldInData'-series we used various sources: The data for the period before 1900 are taken from the History Database of the Global Environment (HYDE). The data for the World Population between 1900 and 1940 is taken from the UN puplication 'The World at Six Billion'. The annual data for the World Population between 1950 and 2015 is taken from the UN's World Population Prospects: The 2015 Revision. It is the series 'Total Population - Both Sexes' online available at: https://esa.un.org/unpd/wpp/.\\r\\n\\r\\nEmissions data have been sourced from the Carbon Dioxide Information Analysis Centre (CDIAC) database. Emissions data have been converted from units of carbon to carbon dioxide (CO2) using a conversion factor of 3.67.\\r\\n\\r\\nCDIAC denote a \\\"statistical difference\\\" component which has been included in this data. This statistical difference represents the difference between estimated global CO2 emissions and the sum of national totals. Estimates of CO2 emissions show that the global total of emissions is not equal to the sum of emissions from all countries. This is introduced in several cases: emissions within international territories, which are included in global totals but not attributed to individual countries; inconsistent national reporting where global import and export data is imbalanced; and differing treatment of non-fuel uses of hydrocarbons.\\r\\n\\r\\nFull methodology on global, regional, national and statistical difference estimations can be found in Le Quere et al. (2016): Le Quéré, C., Andrew, R. M., Canadell, J. G., Sitch, S., Korsbakken, J. I., Peters, G. P., ... & Keeling, R. F. (2016). Global carbon budget 2016. Earth System Science Data, 8(2), 605. Available at: doi:10.5194/essd-8-605-2016.\",\n",
    "    \"sources\": {\n",
    "      \"name\": \"Global CO2 emissions - CDIAC and UN Population\",\n",
    "      \"dataPublishedBy\": \"Carbon Dioxide Information Analysis Center (CDIAC)\",\n",
    "      \"dataPublisherSource\": \"Boden, T. A. and Andres, R. J.: Global, Regional, and National Fossil-Fuel CO2 Emissions. Oak Ridge National Laboratory, US Department of Energy, Oak Ridge, Tenn., USA (2016)\",\n",
    "      \"link\": \"http://cdiac.ornl.gov/CO2_Emission/\",\n",
    "      \"retrievedDate\": \"31/07/2017\"\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "print(json.dumps(resp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ResponseInfo GET /api/info\n",
    "print(json.dumps({\n",
    "    \"headers\" : {\n",
    "        \"Content-Type\" : \"application/json\"\n",
    "    },\n",
    "    \"status\" : 200\n",
    "}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-06T07:59:09.584580Z",
     "start_time": "2020-12-06T07:59:09.579904Z"
    }
   },
   "outputs": [],
   "source": [
    "# # for debugging\n",
    "# REQUEST = json.dumps({\n",
    "#  'args' : {'from':[1850], 'until':[1860], 'avg': True}\n",
    "# })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-06T08:01:50.425394Z",
     "start_time": "2020-12-06T08:01:50.406535Z"
    }
   },
   "outputs": [],
   "source": [
    "# GET /api/data\n",
    "req = json.loads(REQUEST)\n",
    "args = req['args']\n",
    "\n",
    "response = \"\"\n",
    "\n",
    "if not \"from\" in args or not \"until\" in args:\n",
    "    response = {\"Error\": \"Missing argument. Expecting both 'from' and 'until'\"}  \n",
    "else:\n",
    "    date_from = pd.to_datetime(args[\"from\"][0], format=\"%Y\")\n",
    "    date_until = pd.to_datetime(args[\"until\"][0], format=\"%Y\")\n",
    "    if date_from >= date_until:\n",
    "        response = {\"queryParams\" : query_params_dict, \"message\" : \"'from' should be an earlier year than until\"}\n",
    "    elif (date_from > df.Year.max()) or (date_until<df.Year.min()):\n",
    "        query_params_dict = {\"from\" : args[\"from\"][0], \"until\" : args[\"until\"][0]}\n",
    "        response = {\n",
    "            \"queryParams\" : query_params_dict,\n",
    "            \"message\" : \"valid range for 'from' and 'until' is between {} and {} \".format(df['Year'].dt.strftime('%Y').min(), \n",
    "                                                                                          df['Year'].dt.strftime('%Y').max())\n",
    "        }\n",
    "    else:\n",
    "        query_params_dict = {\"from\" : args[\"from\"][0], \"until\" : args[\"until\"][0]}\n",
    "        \n",
    "        rows = df[(df[\"Year\"] >= date_from) &\n",
    "                             (df[\"Year\"] <= date_until)][[\"Year\", \"Per capita CO2 emissions (CDIAC and UN Population)\"]].copy()\n",
    "        if \"avg\" in args and args.get(\"avg\", False):\n",
    "            data_dict = {\"average\" : rows['Per capita CO2 emissions (CDIAC and UN Population)'].mean()}\n",
    "        else:\n",
    "            data_dict = [{\"date\" : date.strftime('%Y'), \"metric tons per capita\" : numbers} \n",
    "                     for date, numbers in rows.values ]\n",
    "        \n",
    "        response = {\n",
    "            \"queryParams\" : query_params_dict, \n",
    "            \"data\" : data_dict\n",
    "        }\n",
    "    \n",
    "print (json.dumps (response))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-06T07:54:52.704474Z",
     "start_time": "2020-12-06T07:54:52.699759Z"
    }
   },
   "outputs": [],
   "source": [
    "# ResponseInfo GET /api/data\n",
    "print(json.dumps({\n",
    "    \"headers\" : {\n",
    "        \"Content-Type\" : \"application/json\"\n",
    "    },\n",
    "    \"status\" : 200\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insert new data with POST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# POST /api/data\n",
    "req = json.loads(REQUEST)\n",
    "args = req['args']\n",
    "\n",
    "if not \"date\" in args or not \"value\" in args:\n",
    "    response = { \"Error\" : \"Missing argument. Expecting both 'date' and 'value'\" }\n",
    "    print (json.dumps (response))\n",
    "else:\n",
    "    date = args[\"date\"][0]\n",
    "    value = float(args[\"value\"][0])\n",
    "    if (df.Year.dt.strftime(\"%Y\")==date).sum():\n",
    "        response = {\n",
    "            \"Error\" : \"'date' should not be an existing value\"\n",
    "            }\n",
    "        print (json.dumps (response))\n",
    "    else:\n",
    "        new_row = {'Year': pd.to_datetime(date),\n",
    "                   'CO2 emissions (CDIAC and UN Population)': value,\n",
    "                   'Per capita CO2 emissions (CDIAC and UN Population)': value,\n",
    "                   'Entity': 'World'}\n",
    "        df = df.append(new_row, ignore_index=True)\n",
    "        df.to_csv(fname, index=False, index_label=None)\n",
    "        response = {\n",
    "            \"Message\" : \"upload successful\"\n",
    "        }\n",
    "        print (json.dumps (response))\n",
    "\n",
    "\n",
    "# curl -X POST \"https://kooplex-edu.elte.hu/notebook/wfct0p-rere-wfct0p/report/api/data?date=1718&value=2.324\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ResponseInfo POST /api/data\n",
    "print(json.dumps({\n",
    "    \"headers\" : {\n",
    "        \"Content-Type\" : \"application/json\"\n",
    "    },\n",
    "    \"status\" : 200\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace existing data with PUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PUT /api/data\n",
    "req = json.loads(REQUEST)\n",
    "args = req['args']\n",
    "\n",
    "response = \"\"\n",
    "\n",
    "if not \"date\" in args or not \"value\" in args:\n",
    "    response = {\n",
    "    \"Error\" : \"Missing argument. Expecting both 'date' and 'value'\"\n",
    "    }\n",
    "    print (json.dumps (response))\n",
    "else:\n",
    "    date = args[\"date\"][0]\n",
    "    value = float(args[\"value\"][0])\n",
    "    if not (df.Year.dt.strftime(\"%Y\")==date).sum():\n",
    "        response = {\n",
    "            \"Error\" : \"'date' should be an existing one\"\n",
    "            }\n",
    "        \n",
    "    else:\n",
    "        df.loc[df.Year.dt.strftime(\"%Y\")==date, 'Per capita CO2 emissions (CDIAC and UN Population)'] = value\n",
    "        df.to_csv(fname, index=False, index_label=None)\n",
    "        response = {\n",
    "            \"Message\" : \"Data has been changed\",\n",
    "            \"Year\": date,\n",
    "            \"New value\": value\n",
    "        }\n",
    "        \n",
    "print (json.dumps (response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /api/agg\n",
    "req = json.loads(REQUEST)\n",
    "args = req['args']\n",
    "\n",
    "response = {\n",
    "    \"Error\" : \"This function is not implemented yet\" \n",
    "}\n",
    "print(json.dumps (response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # ResponseInfo GET /api/image\n",
    "# print(json.dumps({\n",
    "#     \"headers\" : {\n",
    "#         \"Content-Type\" : \"image/jpeg\"\n",
    "#     },\n",
    "#     \"status\" : 200\n",
    "# }))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /:nothingfound\n",
    "\n",
    "print (\"You seem lost, try /help\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "variables": {
     "api_url": "&#39;<a href=\"https://kooplex-fiek.elte.hu/notebook/wfct0p-shareproba-wfct0p/report&#39;\">https://kooplex-fiek.elte.hu/notebook/wfct0p-shareproba-wfct0p/report&#39;</a>"
    }
   },
   "source": [
    "All requests in bash command line with curl:\n",
    "```\n",
    "API_URL={{api_url}}\n",
    "curl -s \"$API_URL/help\"\n",
    "curl -s \"$API_URL/api/info\"\n",
    "curl -s \"$API_URL/api/data?from=1850&until=1854\" | json_pp\n",
    "curl -s \"$API_URL/api/data?from=1850&until=1854&avg=True\" | json_pp\n",
    "curl -s -X POST \"$API_URL/api/data?date=1718&value=2.324\"\n",
    "curl -s -X PUT \"$API_URL/api/data?date=1768&value=0.1564\"\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}