NextGEOSS-Pilot-1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T10:00:06.701976Z",
     "start_time": "2019-03-04T10:00:06.690428Z"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "PATH = \"./img/\"\n",
    "Image(filename = PATH + \"NextGEOSS-Logo.png\", width=512, height=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Introduction to NextGEOSS Energy Pilot (README First !)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Access time series of gridded data from CAMS Radiation\n",
    "\n",
    "**This pilot allows to requests for time-series of solar radiation over a regular grid of points covering area of interest (AOI).**\n",
    "\n",
    "### GEOGRAPHICAL AREA\n",
    "\n",
    "Geographical coverage of the CAMS radiation product is the field-of-view of the Meteosat satellite, roughly speaking Europe, Africa, Atlantic Ocean, Middle East (-66° to 66° in both latitudes and longitudes).\n",
    "\n",
    "### DATA\n",
    "\n",
    "The data you request come from CAMS (Copernicus Atmosphere Monitoring Service) Radiation product from a two years time period including the following parameters (Atmosphere optical properties, Ground reflectance, Cloud optical properties, Atmosphere radiative transfer). **#IMPORTANT NOTE: ONLY THE YEARS 2005 AND 2006 ARE CURRENTLY AVAILABLE FOR THIS PILOT#**\n",
    "\n",
    "### HOW TO REQUEST THE DATA\n",
    "\n",
    "The request to the data is made using an **OGC (Open Geospatial Consortium)** standard Web service called **WPS (Web Processing Service)**. This WPS is located remotely on a cloud server and provided as an asynchronous request to the CAMS Radiation process in order to deliver time series of gridded data on a selected AOI (Area Of Interest).\n",
    "\n",
    "The parameters are first encoded in an url and sent to the Cloud as a WPS request. \n",
    "\n",
    "This notebook will first demonstrate how to build the first url. It will then provide a set of command that fully automate the subsequent \"treasure hunt\" up to the downloading of the data.\n",
    "\n",
    "### RESULT VISUALISATION\n",
    "\n",
    "Once downloaded as an HDF5 encode file, the Notebook provides you with results visulaisation:\n",
    "* An animated display of the time frame layers according to your selection including the global and direct irradiation from both full and clear sky.\n",
    "\n",
    "**Please allow some time for data to be processed and downloaded. When done you'll have a preview of your time series of gridded data at the bottom of the page. The default example retrieve a full day every 15 mn. from 00:15 providing you with 96 layers over an AOI covering France**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Setup the Notebook environment (README)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The current version of this Notebook only runs on the Jupyter Hub environment of Terradue and the needed environment to run the application has been already setup.\n",
    "\n",
    "In order to connect please follow the information as explained in the [« Access instructions »](NextGEOSS-ARMINES-Pilot-1-Access-Instructions.pdf) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.478743Z",
     "start_time": "2019-03-04T09:00:00.988821Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import re, time, os\n",
    "from collections import OrderedDict\n",
    "from datetime import date\n",
    "\n",
    "# some GUI Elements\n",
    "from ipywidgets import FloatProgress\n",
    "from IPython.display import display\n",
    "import ipywidgets as wdg\n",
    "from ipyleaflet import Map, basemaps, basemap_to_tiles, DrawControl, GeoJSON\n",
    "\n",
    "# Usuall data analysis\n",
    "import pandas as pd\n",
    "import h5py\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# To read xml files\n",
    "import xml.etree.ElementTree as xml\n",
    "\n",
    "# To download data\n",
    "from urllib.request import urlopen\n",
    "\n",
    "\n",
    "class value_handler:\n",
    "    def __init__(self, key, long_name, value):\n",
    "        self.key = key\n",
    "        self.long_name = long_name\n",
    "        self.value = value\n",
    "    def make_widget(self):\n",
    "        l = wdg.Label(self.long_name, layout={'width': '150px'})\n",
    "        t = wdg.Text(value=str(self.value), layout={'width': '150px'})\n",
    "        t.observe(self.on_change, names='value')\n",
    "        return wdg.HBox([l, t])\n",
    "    def on_change(self, x):\n",
    "        self.from_str(x['new'])\n",
    "    \n",
    "class value_handler_flt(value_handler):\n",
    "    def __init__(self, *args):\n",
    "        super().__init__(*args)\n",
    "    def from_str(self, value):\n",
    "        self.value = float(value)\n",
    "\n",
    "class value_handler_flt_map(value_handler):\n",
    "    def __init__(self, *args):\n",
    "        super().__init__(*args)\n",
    "    def from_str(self, value):\n",
    "        self.value = float(value)\n",
    "    def on_change(self, x):\n",
    "        self.from_str(x['new'])\n",
    "        update_current_bbox()\n",
    "        \n",
    "class value_handler_int(value_handler):\n",
    "    def __init__(self, *args):\n",
    "        super().__init__(*args)\n",
    "    def from_str(self, value):\n",
    "        self.value = int(value)\n",
    "\n",
    "class value_handler_str(value_handler):\n",
    "    def __init__(self, *args):\n",
    "        super().__init__(*args)\n",
    "    def from_str(self, value):\n",
    "        self.value = str(value)\n",
    "        \n",
    "class value_handler_dropdown(value_handler):\n",
    "    def __init__(self, key, long_name, value, l):\n",
    "        super().__init__(key, long_name, value)\n",
    "        self.l = l\n",
    "    def from_str(self, value):\n",
    "        self.value = str(value)\n",
    "    def make_widget(self):\n",
    "        l = wdg.Label(self.long_name, layout={'width': '150px'})\n",
    "        t = wdg.Dropdown(options=self.l, value=str(self.value), layout={'width': '150px'})\n",
    "        t.observe(self.on_change, names='value')\n",
    "        return wdg.HBox([l, t])\n",
    "    \n",
    "class value_handler_date(value_handler):\n",
    "    def __init__(self, key, long_name, value):\n",
    "        super().__init__(key, long_name, value)\n",
    "    def from_str(self, value):\n",
    "        self.value = value\n",
    "    def make_widget(self):\n",
    "        l = wdg.Label(self.long_name, layout={'width': '150px'})\n",
    "        t = wdg.DatePicker(value=self.value, layout={'width': '150px'})\n",
    "        t.observe(self.on_change, names='value')\n",
    "        return wdg.HBox([l, t])\n",
    "        \n",
    "def make_circle(center, radius, rez):\n",
    "    angles = np.linspace(0.0, 2*np.pi, rez)\n",
    "    x = np.cos(angles)*radius+center[0]\n",
    "    y = np.sin(angles)*radius+center[1]\n",
    "    return np.vstack((y[::-1], x[::-1])).T\n",
    "\n",
    "\n",
    "def cv(d,m,s):\n",
    "    return d+m/60+s/60/60\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.484168Z",
     "start_time": "2019-03-04T09:00:02.480792Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "namespace = {\n",
    "    'wps': 'http://www.opengis.net/wps/1.0.0',\n",
    "    'ows': 'http://www.opengis.net/ows/1.1',\n",
    "    'mlk': 'http://www.metalinker.org/'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Encoding the request in a url (Optional reading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "\n",
    "The first url is the only wat you need to worry about, as it is where you define the particulars of your request (inputs, methods, outputs etc). \n",
    "\n",
    "**Url decomposition**\n",
    "The url can be decomposed in 4 parts:\n",
    "* The 'base string' defines the requested service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.489716Z",
     "start_time": "2019-03-04T09:00:02.486382Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# release server\n",
    "base_str = 'http://gridded-data.armines.nextgeoss.terradue.com/wps/WebProcessingService?service=wps&version=1.0.0&request=Execute&identifier=com.terradue.wps_oozie.process.OozieAbstractAlgorithm&'\n",
    "#base_str = 'http://10.15.22.22:8080/wps/WebProcessingService?service=wps&version=1.0.0&request=Execute&identifier=com.terradue.wps_oozie.process.OozieAbstractAlgorithm&'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* The inputs string sets the parameters of the request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.494993Z",
     "start_time": "2019-03-04T09:00:02.492079Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#input_str = 'dataInputs=bbox=-5.0,46.0,6.0,45.0;width=300;height=300;datex=2453385.510416666666;dt=0.010441666666;count=96;min_tile_width=100;min_tile_height=100;&'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* The output string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.500055Z",
     "start_time": "2019-03-04T09:00:02.496946Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "output_str = 'ResponseDocument=result_distribution@mimeType=application/xml;result_osd@mimeType=application/xml&storeExecuteResponse=true&status=true'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The final url is the aggregation of the 4 parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.505258Z",
     "start_time": "2019-03-04T09:00:02.502469Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#url_request = base_str+input_str+output_str\n",
    "#print(url_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Change the input parameters (README)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "\n",
    "In this Notebook we've tried to ease the user input parameters selection by including graphical elements known as \"widgets\". Two  widgets are availables:\n",
    "* **A free rectangle tool to select an AOI (Area Of Interest)**\n",
    "* **A set of forms to select additional inputs parameters**\n",
    "By default those parameters are already filled.\n",
    "\n",
    "**IMPORTANT**:\n",
    "\n",
    "When you will see this Notebook for the first time those widgets will not be visible because they rely on external components such as OpenStreetMap background map. Consequetly the Notebook need to be run once before allowing you to view and play with the widget's components.  \n",
    "\n",
    "As a shortcut you could first directly launch a request with the defaults parameters to enable the display of the graphical widgets.\n",
    "To do so, go to the JupyterLab top menu and under the \"Kernel\" dropdown menu select \"Restart Kernel and Run All Cells...\" item. Confirm the option by clicking on the \"RESTART\" red button.\n",
    "\n",
    "Allow for some time for the process to remotly execute and to extract, gather, download and display the results in the Notebook.\n",
    "\n",
    "Progress bar are availables for both the process execution and when finnished for the download remaining time.\n",
    "\n",
    "A file named \"result.nc\" should be available in the \"Files\" left menu of the JupyterLab and an animation of the gridded time serie should be available at the bottom of the Notebook.\n",
    "\n",
    "It is also possible to manually modify the url parameters by directly editing the python parameters values below. \n",
    "The input parameters can be specified by changing the value in <font color='green'>green</font>. The default example is set for an AOI (Area Of Interest) covering France with a 300x300 resolution for one day (2005 June 22) starting at 00:15h every 15 minutes resulting in HDF file of 96 layers This includes:\n",
    "\n",
    "* **bot_lat** (Bottom size of a bounding box latitude wise in degree - default is 41.0)\n",
    "* **top_lat** (Top side of a bounding box latitude wise in degree - default is 51.3)\n",
    "* **left_lon** (Left side of a bounding box longitude wise in degree - default is -5.5)\n",
    "* **right_lon** (Right side of a bounding box longiture wise in degree - default is 8.5)\n",
    "* **width** (Size in pixel of the width of the result map - default is 300)\n",
    "* **height** (Size in pixel of the height of the result map - default is 300)\n",
    "* **first_instant** (Begin date in Julian date - default is 2005 Jun 22 at 00:15)\n",
    "* **dt** (Time span for the number of occurence in Julian time - default is 15 mn)\n",
    "* **count** (Number of occurence in the time serie of gridded data - default is 96 (Covers 1 day every 15 mn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.514490Z",
     "start_time": "2019-03-04T09:00:02.508169Z"
    },
    "hidden": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialise inputs to default value if not already created\n",
    "if \"inputs\" not in globals():\n",
    "    inputs = OrderedDict([\n",
    "        (\"bot_lat\",       value_handler_flt_map(\"bot_lat\", \"Latitude at bottom\", 41.0)),\n",
    "        (\"top_lat\",       value_handler_flt_map(\"top_lat\", \"Latitude at top\", 51.3)),\n",
    "        (\"left_lon\",      value_handler_flt_map(\"left_lon\", \"Longitude at left\", -5.5)),\n",
    "        (\"right_lon\",     value_handler_flt_map(\"right_lon\", \"Longitude at right\", 8.5)),\n",
    "        (\"width\",         value_handler_int(\"width\", \"Output width\", 300)),\n",
    "        (\"height\",        value_handler_int(\"height\", \"Output height\", 300)),\n",
    "        (\"first_instant\", value_handler_date(\"first_instant\", \"First instant of the time-series\", date(2005,6,22))),\n",
    "        (\"dt\",            value_handler_int(\"dt\", \"Interval between instants in minute(s)\", 15)),\n",
    "        (\"count\",         value_handler_int(\"count\", \"Number of instant\", 96))\n",
    "    ])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.774945Z",
     "start_time": "2019-03-04T09:00:02.517844Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "\n",
    "map_context = {\n",
    "    'url': 'http://tiles.osm.webservice-energy.org/osm/{z}/{x}/{y}.png',\n",
    "    'attribution': '<a href=\"https://www.openstreetmap.org/copyright\">OpenStreeMap</a>',\n",
    "    'name': 'openstreetmap',\n",
    "    'max_zoom': 9\n",
    "}\n",
    "\n",
    "m = Map(center=(30.0, 0.0), zoom=2, layout={'width': '600px', 'height': '300px'})\n",
    "openstreetmap_layer = basemap_to_tiles(map_context)\n",
    "m.add_layer(openstreetmap_layer)\n",
    "\n",
    "available_area = GeoJSON(data={\n",
    "            'type': 'Feature', \n",
    "        'properties': {\n",
    "            'style': {\n",
    "                'stroke': True, \n",
    "                'color': '#00ff00', \n",
    "                'weight': 1, \n",
    "                'opacity': 0.5,\n",
    "                'fill': True, \n",
    "                'fillColor': None, \n",
    "                'fillOpacity': 0.05, \n",
    "                'showArea': True, \n",
    "                'clickable': False\n",
    "            }\n",
    "        }, \n",
    "        'geometry': {\n",
    "            'type': 'Polygon', \n",
    "            'coordinates':  [ make_circle((0.0, 0.0), 60.0, 100).tolist() ]\n",
    "        }\n",
    "})\n",
    "\n",
    "m.add_layer(available_area)\n",
    "\n",
    "\n",
    "def update_current_bbox():\n",
    "    global current_bbox, m, inputs\n",
    "    \n",
    "    try:\n",
    "        m.remove_layer(current_bbox)\n",
    "        current_bbox = None\n",
    "    except:\n",
    "        pass\n",
    "    \n",
    "    current_bbox = GeoJSON(data={\n",
    "        'type': 'Feature', \n",
    "        'properties': {\n",
    "            'style': {\n",
    "                'stroke': True, \n",
    "                'color': '#ff0000', \n",
    "                'weight': 4, \n",
    "                'opacity': 0.5, \n",
    "                'fill': True, \n",
    "                'fillColor': None, \n",
    "                'fillOpacity': 0.2, \n",
    "                'showArea': True, \n",
    "                'clickable': True\n",
    "            }\n",
    "        }, \n",
    "        'geometry': {\n",
    "            'type': 'Polygon', \n",
    "            'coordinates': [\n",
    "                [[inputs['left_lon'].value,  inputs['bot_lat'].value], \n",
    "                 [inputs['left_lon'].value,  inputs['top_lat'].value], \n",
    "                 [inputs['left_lon'].value,  inputs['top_lat'].value], \n",
    "                 [inputs['right_lon'].value, inputs['top_lat'].value], \n",
    "                 [inputs['right_lon'].value, inputs['bot_lat'].value]]\n",
    "            ]\n",
    "        }\n",
    "    })\n",
    "\n",
    "    m.add_layer(current_bbox)\n",
    "\n",
    "update_current_bbox()\n",
    "\n",
    "wdgs = {}\n",
    "for xp in inputs.values():\n",
    "    wdgs[xp.key] = xp.make_widget()\n",
    "    \n",
    "def xdraw(dc, action, geo_json):\n",
    "    global m, current_bbox\n",
    "    dc.clear()    \n",
    "    # update inputs data\n",
    "    bbox = np.squeeze(np.array(geo_json['geometry']['coordinates']))\n",
    "    \n",
    "    wdgs[\"left_lon\"].children[1].value = str(np.min(bbox[:,0]))\n",
    "    wdgs[\"right_lon\"].children[1].value = str(np.max(bbox[:,0]))\n",
    "    wdgs[\"top_lat\"].children[1].value = str(np.max(bbox[:,1]))\n",
    "    wdgs[\"bot_lat\"].children[1].value = str(np.min(bbox[:,1]))\n",
    "\n",
    "dc = DrawControl(rectangle={'shapeOptions': {'color': '#ff0000'}}, circlemarker={}, polyline={}, polygon={}, edit=False, remove=False)\n",
    "dc.on_draw(xdraw)\n",
    "m.add_control(dc)\n",
    "print(\"Select an area using the rectangle tool\")\n",
    "\n",
    "display(m)\n",
    "\n",
    "layout = [\n",
    "        [\"width\",          \"left_lon\"],\n",
    "        [\"height\",         \"right_lon\"],\n",
    "        [\"first_instant\",  \"top_lat\"],\n",
    "        [\"dt\",             \"bot_lat\"],\n",
    "        [\"count\",          None]]\n",
    "\n",
    "display(wdg.VBox([wdg.HBox([wdgs[x] for x in l if x is not None]) for l in layout]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.808688Z",
     "start_time": "2019-03-04T09:00:02.777066Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Display inputs in human-readble form,\n",
    "readable_input = pd.DataFrame([x.value for x in inputs.values()],columns=[''],dtype='object')\n",
    "readable_input.index = [x.long_name for x in inputs.values()]\n",
    "display(readable_input)\n",
    "\n",
    "# Translate input format WPS-appropriate format,\n",
    "wps_inputs = {x.key: x.value for x in inputs.values()}\n",
    "wps_inputs[\"dt\"] = wps_inputs['dt']\n",
    "wps_inputs[\"first_instant\"] = wps_inputs[\"first_instant\"].strftime(\"%Y-%m-%d\")\n",
    "\n",
    "input_str = 'dataInputs=bbox={left_lon:f},{top_lat:f},{right_lon:f},{bot_lat:f};width={width:d};height={height:d};time_offset={first_instant:s};time_delta={dt:d};time_count={count:d};tiles_factor_count=8;avg_tile_size=10000000&'\n",
    "input_str = input_str.format(**wps_inputs)\n",
    "\n",
    "request_size = (wps_inputs[\"right_lon\"]-wps_inputs[\"left_lon\"])*(wps_inputs[\"top_lat\"]-wps_inputs[\"bot_lat\"])*(wps_inputs[\"count\"]*wps_inputs[\"dt\"])/(24*60)\n",
    "display(HTML(\"<p>Estimated time: %.1f minutes</p>\"%(request_size/14+5,)))\n",
    "if request_size > 600.0:\n",
    "    display(HTML(\"\"\"<span style=\"color: red;\">Request is probably to big</span>\"\"\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.816572Z",
     "start_time": "2019-03-04T09:00:02.812098Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "url_request = base_str+input_str+output_str\n",
    "print(url_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Submit the request to the WPS server (Optional reading)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:00:02.974306Z",
     "start_time": "2019-03-04T09:00:02.819191Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Warning: this line launches a request. It should not be called several time\n",
    "# r = PoolManager().request('GET', url_request)\n",
    "r = urlopen(url_request)\n",
    "tree = xml.fromstring(r.read().decode(\"utf-8\"))\n",
    "if not tree.tag == \"{http://www.opengis.net/wps/1.0.0}ExecuteResponse\":\n",
    "    raise Exception(\"Unexpected response\")\n",
    "if tree.find(\"./{http://www.opengis.net/wps/1.0.0}Status/{http://www.opengis.net/wps/1.0.0}ProcessAccepted\") is None:\n",
    "    raise Exception(\"Process was not accepted, please check your parameters\")\n",
    "status_url = tree.attrib[\"statusLocation\"]\n",
    "print(\"Process accepted with folowing status url:\")\n",
    "print(status_url)\n",
    "print(\"you do not need to run it again.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Wait for the WPS response (Display progress bar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:04:16.317401Z",
     "start_time": "2019-03-04T09:00:02.976603Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "## The server response may be failed or Succeded\n",
    "print(\"please wait for the server\")\n",
    "\n",
    "p = wdg.FloatProgress(min=0, max=100, description='Waiting:')\n",
    "l = wdg.Label()\n",
    "display(wdg.HBox([p, l]))\n",
    "\n",
    "while True:\n",
    "    r = urlopen(status_url)\n",
    "    tree = xml.fromstring(r.read().decode(\"utf-8\"))\n",
    "    if not tree.tag == \"{http://www.opengis.net/wps/1.0.0}ExecuteResponse\":\n",
    "        Exception(\"Unexpected response\")\n",
    "    status = tree.find(\"./{http://www.opengis.net/wps/1.0.0}Status/*\")\n",
    "    if status.tag == \"{http://www.opengis.net/wps/1.0.0}ProcessFailed\":\n",
    "        error = status.find(\".//{http://www.opengis.net/ows/1.1}ExceptionText\")\n",
    "        raise Exception(\"WPS Process fail with error: %s\"%(error.text))\n",
    "    elif status.tag == \"{http://www.opengis.net/wps/1.0.0}ProcessStarted\":\n",
    "        p.value = int(status.attrib[\"percentCompleted\"])\n",
    "        l.value = \"%d%%\"%(p.value,)\n",
    "    elif status.tag == \"{http://www.opengis.net/wps/1.0.0}ProcessAccepted\":\n",
    "        pass\n",
    "    elif status.tag == \"{http://www.opengis.net/wps/1.0.0}ProcessPaused\":\n",
    "        print(\"Process paused\")\n",
    "    elif status.tag == \"{http://www.opengis.net/wps/1.0.0}ProcessSucceeded\":\n",
    "        p.value = 100\n",
    "        l.value = \"100%\"\n",
    "        break\n",
    "    time.sleep(10)\n",
    "print(\"Process succeeded\")\n",
    "# TODO: Get the result link"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Retreive the result from the WPS server (Optional reading)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:04:16.646380Z",
     "start_time": "2019-03-04T09:04:16.319543Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "outputs = tree.findall(\"./wps:ProcessOutputs/wps:Output\", namespace)\n",
    "if outputs is None:\n",
    "    raise Exception(\"No outputs found!\")\n",
    "\n",
    "output_metalink = None\n",
    "for o in outputs:\n",
    "    identifier = o.find(\"./ows:Identifier\", namespace)\n",
    "    if identifier is None:\n",
    "        continue\n",
    "    if identifier.text == \"result_distribution\":\n",
    "        ref = o.find(\"./wps:Data/wps:ComplexData/wps:Reference\", namespace)\n",
    "        if ref is None:\n",
    "            raise Exception(\"No Reference link found!\")\n",
    "        output_metalink = ref.attrib['href']\n",
    "        \n",
    "if output_metalink is None:\n",
    "    raise Exception(\"Output reference link not found!\")\n",
    "\n",
    "r = urlopen(output_metalink)\n",
    "t = xml.fromstring(r.read().decode(\"utf-8\"))\n",
    "if t.tag != '{http://www.metalinker.org/}metalink':\n",
    "    raise Exception(\"Invalid metalink\")\n",
    "\n",
    "link = t.find(\"./mlk:files/mlk:file[@name='result.nc']/mlk:resources/mlk:url\", namespace)\n",
    "if link is None:\n",
    "    raise Exception(\"No result file url found!\")\n",
    "final_url = link.text\n",
    "\n",
    "link = t.find(\"./mlk:files/mlk:file[@name='meta.json']/mlk:resources/mlk:url\", namespace)\n",
    "if link is None:\n",
    "    raise Exception(\"No meta file url found!\")\n",
    "meta_url = link.text\n",
    "\n",
    "print(\"Result URL: \", final_url)\n",
    "print(\"Meta URL: \", meta_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Download final data (Display progress bar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:04:22.990911Z",
     "start_time": "2019-03-04T09:04:16.649248Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "r = urlopen(final_url)\n",
    "\n",
    "if 'content-length' in r.headers:\n",
    "    content_length = int(r.headers['content-length'])\n",
    "\n",
    "    print('Downloading %.1fMo:'%(content_length/1e6))\n",
    "    p = wdg.FloatProgress(min=0, max=content_length, description='Downloading:')\n",
    "    l = wdg.Label()\n",
    "    display(wdg.HBox([p, l]))\n",
    "\n",
    "    with open(\"result.nc\", \"wb\") as f:\n",
    "        while True:\n",
    "            buffer = r.read(4096*64)\n",
    "            if len(buffer) == 0:\n",
    "                break\n",
    "            p.value += len(buffer)\n",
    "            l.value = \"%.1f%%\"%((p.value*100/content_length),)\n",
    "            f.write(buffer)\n",
    "\n",
    "    print(\"Download finished!\")\n",
    "else:\n",
    "    print('Downloading...')\n",
    "    with open(\"result.nc\", \"wb\") as f:\n",
    "        while True:\n",
    "            buffer = r.read(4096*64)\n",
    "            if len(buffer) == 0:\n",
    "                break\n",
    "            f.write(buffer)\n",
    "    print(\"Download finished!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-23T08:46:50.095534Z",
     "start_time": "2018-10-23T08:46:50.091158Z"
    },
    "heading_collapsed": true
   },
   "source": [
    "# Animation of the time serie of gridded data (README)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Code and libraries needed to display the data (Optional reading)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:04:23.435389Z",
     "start_time": "2019-03-04T09:04:22.994166Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.animation as animation\n",
    "import matplotlib.pyplot as plt\n",
    "import h5py\n",
    "import sys\n",
    "\n",
    "# Guess the FFMEG executatble path\n",
    "plt.rc('animation', ffmpeg_path=sys.executable.replace('python', 'ffmpeg'))\n",
    "plt.rc('animation', embed_limit=2**31)\n",
    "\n",
    "# Some possible more options\n",
    "#plt.rc('animation', ffmpeg_path='/workspace/.conda/envs/armines-pilot/bin/ffmpeg')\n",
    "#plt.rc('animation', writer='ffmpeg_file')\n",
    "#plt.rc('animation', codec='vp9')\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from itertools import product\n",
    "\n",
    "import cartopy.crs as ccrs\n",
    "import cartopy.feature as cfeature\n",
    "from cartopy.feature import NaturalEarthFeature, COLORS\n",
    "from cartopy.io import DownloadWarning\n",
    "import warnings\n",
    "\n",
    "# ignore cartopy warning\n",
    "warnings.simplefilter(\"ignore\",  category=DownloadWarning)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:04:23.449143Z",
     "start_time": "2019-03-04T09:04:23.438364Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Define cartopy features:\n",
    "resolution = \"50m\"\n",
    "BORDERS = NaturalEarthFeature('cultural', 'admin_0_boundary_lines_land',\n",
    "                              resolution, edgecolor='black', facecolor='none')\n",
    "STATES = NaturalEarthFeature('cultural', 'admin_1_states_provinces_lakes',\n",
    "                             resolution, edgecolor='black', facecolor='none')\n",
    "COASTLINE = NaturalEarthFeature('physical', 'coastline', resolution,\n",
    "                                edgecolor='black', facecolor='none')\n",
    "LAKES = NaturalEarthFeature('physical', 'lakes', resolution,\n",
    "                            edgecolor='face',\n",
    "                            facecolor=COLORS['water'])\n",
    "LAND = NaturalEarthFeature('physical', 'land', resolution,\n",
    "                           edgecolor='face',\n",
    "                           facecolor=COLORS['land'], zorder=-1)\n",
    "OCEAN = NaturalEarthFeature('physical', 'ocean', resolution,\n",
    "                            edgecolor='face',\n",
    "                            facecolor=COLORS['water'], zorder=-1)\n",
    "RIVERS = NaturalEarthFeature('physical', 'rivers_lake_centerlines', resolution,\n",
    "                             edgecolor=COLORS['water'],\n",
    "                             facecolor='none')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-23T09:06:58.799013Z",
     "start_time": "2018-10-23T09:06:58.796397Z"
    },
    "hidden": true
   },
   "source": [
    "### Animation of the variability in space and in time of the solar resource within the selected period (REAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Click on the > play button below to run the animation.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:41:33.586798Z",
     "start_time": "2019-03-04T09:41:07.537395Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Load downloaded data in notebook:\n",
    "\n",
    "from tempfile import NamedTemporaryFile\n",
    "\n",
    "# Create animation:\n",
    "extent = [inputs[\"left_lon\"].value, inputs[\"right_lon\"].value, inputs[\"bot_lat\"].value, inputs[\"top_lat\"].value]\n",
    "\n",
    "fig, ax = plt.subplots(2, 2, figsize=(11, 7), subplot_kw={'projection': ccrs.PlateCarree()})\n",
    "im = np.zeros((2,2), dtype='object')\n",
    "cb = np.zeros((2,2), dtype='object')\n",
    "ne = np.array([['GHI', 'BHI'], ['GHI_CLS', 'BHI_CLS']])\n",
    "\n",
    "with h5py.File('result.nc', 'r') as hf:\n",
    "\n",
    "    for a, b in product(range(2), range(2)):\n",
    "        name = ne[a,b]\n",
    "        data = hf[name][:,:,:]\n",
    "        xmax = np.max(data[np.isfinite(data)])\n",
    "        im[a,b] = ax[a,b].imshow(hf[name][0,:,:], animated=True, vmin=0, vmax=xmax, origin='lower', transform=ccrs.PlateCarree(), extent=extent)\n",
    "        ax[a,b].set_title(\"%s, frame: %d\"%(name, 0))\n",
    "        ax[a,b].add_feature(BORDERS)\n",
    "        ax[a,b].add_feature(COASTLINE)\n",
    "        cb[a,b] = fig.colorbar(im[a,b], ax=ax[a,b], shrink=0.7)\n",
    "        cb[a,b].set_label(r'$W/m^{2}$', size='large')\n",
    "\n",
    "    def updatefig(frame, *args):\n",
    "        global im, hf, ax\n",
    "        for a, b in product(range(2), range(2)):\n",
    "            name = ne[a,b]\n",
    "            ax[a,b].set_title(\"%s, frame: %d\"%(name, frame))\n",
    "            im[a,b].set_data(hf[name][frame, :, :])\n",
    "        return im[0,0], im[0,1], im[1,0], im[1,1]\n",
    "\n",
    "    ani = animation.FuncAnimation(fig, updatefig, frames=range(inputs[\"count\"].value), interval=200, blit=True)\n",
    "\n",
    "    fvideo = NamedTemporaryFile(prefix=\"video_\", suffix=\".mp4\", dir='.', delete=True)\n",
    "    ani.save(fvideo.name, writer=animation.FFMpegWriter(codec='h264', bitrate=2000000))\n",
    "    plt.close(fig) # Avoid figure to be drawn twice\n",
    "\n",
    "# Show the video\n",
    "HTML(\"\"\"\n",
    "<video controls autoplay>\n",
    "  <source src=\"{fvideo}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(fvideo=\"./\"+os.path.basename(fvideo.name)))\n",
    "\n",
    "# Do not work with this version of matplotlib\n",
    "#HTML(ani.to_jshtml())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T09:43:49.946704Z",
     "start_time": "2019-03-04T09:43:49.943532Z"
    }
   },
   "source": [
    "# Exploit the information to support decision making (README)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this Notebook example we want to provide you with the possibily to go further than the process and download possibilties. This is why we have developped a second Notebook that will illustrate **how to use and exploit the information retreived in order to support decision making**. This Notebook will illustrates the following:\n",
    "\n",
    "* A use case where a time-series of a single is point is compared to an average of gridded data over a given region (NUTS-3 shape)\n",
    "* A use case where a time-series of a single is point is compared to an average of gridded data over a given geometrical AOI (Area Of Interest)\n",
    "* A use case where we compute the daily average irrdiation per department for France\n",
    "\n",
    "Please load the following link to proceed: [here](NextGEOSSFranceAggregation.ipynb)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python [conda env:.conda-armines-pilot]",
   "language": "python",
   "name": "conda-env-.conda-armines-pilot-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}