exiobase_as_aggregated.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows the integration of EXIOBASE into brightway2 as aggregated results. In other words, EXIOBASE will only be available as aggregated LCIA scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all modules can be imported with pip, outside of brightway2\n",
    "import pandas as pd\n",
    "import pymrio\n",
    "import hashlib\n",
    "from brightway2 import *\n",
    "from bw2agg.scores import add_all_unit_score_exchanges_and_cfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select (or create) the brightway2 project you wish to integrated EXIOBASE into"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "projects.set_current('Importing EXIOBASE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Need to setup brightway2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bw2setup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Need to download IMPACT World+ from https://zenodo.org/record/3521034 and load it into brightway:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_IW_package = '/.../Brightway_IW_damage_1_46_and_midpoint_1_28.bw2package'\n",
    "BW2Package.import_file(path_to_IW_package)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load EXIOBASE3 (for EXIOBASE2, use the already existing importer of brightway2) using pymrio and define normalized matrices using .calc_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pymrio.core.mriosystem.IOSystem at 0x2810594ff60>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_exiobase = 'path_to_exiobase_folder'\n",
    "io = pymrio.parse_exiobase3(path_exiobase)\n",
    "io.calc_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EXIOBASE3 includes numbers in some of the sectors, we just swiftly remove them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sectors_without_numbers = []\n",
    "for sector in list(io.get_sectors()):\n",
    "    if any(char.isdigit() for char in sector):\n",
    "        sectors_without_numbers.append(sector[:-5])\n",
    "    else:\n",
    "        sectors_without_numbers.append(sector)\n",
    "index_without_numbers = pd.MultiIndex.from_product([list(io.get_regions()),sectors_without_numbers])\n",
    "\n",
    "io.A.index = index_without_numbers\n",
    "io.A.columns = index_without_numbers\n",
    "io.L.index = index_without_numbers\n",
    "io.L.columns = index_without_numbers\n",
    "io.satellite.S.columns = index_without_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We give hashcodes to EXIOBASE sectors to follow how brightway2 typically deals with unique identifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create hash codes for exiobase sectors\n",
    "give_hash_to_exio = dict.fromkeys(io.A.index)\n",
    "give_hash_to_exio = {k:hashlib.md5(str(k).encode()).hexdigest() for k,v in give_hash_to_exio.items()}\n",
    "# set those hash codes as index\n",
    "io.A.index = [give_hash_to_exio[i] for i in io.A.index]\n",
    "io.A.columns = [give_hash_to_exio[i] for i in io.A.columns]\n",
    "io.L.index = [give_hash_to_exio[i] for i in io.L.index]\n",
    "io.L.columns = [give_hash_to_exio[i] for i in io.L.columns]\n",
    "io.satellite.S.columns = [give_hash_to_exio[i] for i in io.satellite.S.columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pymrio does not include characterization matrices. We fetch it. Use it to aggregate results of EXIOBASE and link charcaterization methods names to the ones used by brightway2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load IW+ characterization factors for exiobase (available here: https://zenodo.org/record/3955079)\n",
    "C = pd.read_csv('path_to_csv_file')\n",
    "\n",
    "# reindex to have a name dataframe\n",
    "C.set_index('Unnamed: 0',drop=True,inplace=True)\n",
    "\n",
    "# aggregate results of exiobase\n",
    "aggregated_results = C.dot(io.satellite.S.dot(io.L).iloc[:-9])/1000000\n",
    "\n",
    "# link IMPACT World+ impact categories typology to the methods abbreviations used in brightway2\n",
    "IW_pylcaio_to_bw2 = dict.fromkeys(C.index)\n",
    "\n",
    "for IW_category in IW_pylcaio_to_bw2:\n",
    "    if ('PDF' in IW_category or 'DALY' in IW_category):\n",
    "        IW_pylcaio_to_bw2[IW_category] = methods.get(('IMPACTWorld+ (Default_Recommended_Damage 1.46)', IW_category.split(' (')[0]))['abbreviation']\n",
    "    else:\n",
    "        IW_pylcaio_to_bw2[IW_category] = methods.get(('IMPACTWorld+ (Default_Recommended_Midpoint 1.28)', IW_category.split(' (')[0]))['abbreviation']\n",
    "\n",
    "# define these abbreviations as index for C\n",
    "aggregated_results.index = list(IW_pylcaio_to_bw2.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To operate with aggregated data, we need to use functionality from bw2agg. Here, dummy exchanges and characterization factors are create to enable the storing of LCIA scores into exchanges directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding unit score biosphere exchanges and characterization factors to all 895 methods in project\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:06:41\n"
     ]
    }
   ],
   "source": [
    "add_all_unit_score_exchanges_and_cfs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We change the dataframe of exchanges into the dictionary format used by brightway2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "bw2_dict = dict.fromkeys(list(zip(['exiobase']*len(aggregated_results.columns),aggregated_results.columns)),{\n",
    "    'name': '',\n",
    "    'unit':'',\n",
    "    'location':'',\n",
    "    'exchanges':[],\n",
    "})\n",
    "\n",
    "for sector_hash in aggregated_results.columns:\n",
    "    bw2_dict[('exiobase',sector_hash)] = {'name' : list({k:v for k,v in give_hash_to_exio.items() if v == sector_hash}.keys())[0][1],\n",
    "                                          'unit' : 'euro',\n",
    "                                          'location' : list({k:v for k,v in give_hash_to_exio.items() if v == sector_hash}.keys())[0][0],\n",
    "                                          'exchanges': aggregated_results.loc[:, sector_hash][aggregated_results.loc[:,sector_hash]!=0].to_dict()}\n",
    "for sector_hash in aggregated_results.columns:\n",
    "    bw2_dict[('exiobase',sector_hash)]['exchanges'] = list({k:{'input':('biosphere3',k), 'amount':v, 'type':'biosphere'} for k,v in bw2_dict[('exiobase',sector_hash)]['exchanges'].items()}.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dictionary created as a database (whose name you can choose) of brightway2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Writing activities to SQLite3 database:\n",
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:06:13\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Writing activities to SQLite3 database:\n",
      "  Started: 08/04/2020 16:18:14\n",
      "  Finished: 08/04/2020 16:24:27\n",
      "  Total time elapsed: 00:06:13\n",
      "  CPU %: 10.50\n",
      "  Memory %: 9.72\n"
     ]
    }
   ],
   "source": [
    "Database('exiobase').write(bw2_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finished, EXIOBASE is integrated into your brightway2 project."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}