pybliometrics_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63e46583-0aed-4834-9134-05e5fd389d00",
   "metadata": {},
   "source": [
    "# Systematic Literature Reviews\n",
    "\n",
    "\n",
    "## Why are they useful?\n",
    "1. Robust\n",
    "2. Transparent\n",
    "3. Repeatable\n",
    "\n",
    "## What are the steps?\n",
    "1. Identify a research question (e.g., How do energy models account for structural uncertainty?)\n",
    "2. ???\n",
    "3. Profit.\n",
    "\n",
    "# ??? - How does one systematically search for papers?\n",
    "\n",
    "The subject of this notebook is finding papers to include in your review. We will review two methods. \n",
    "\n",
    "1. Search engines.\n",
    "2. Database API (Scopus)\n",
    "\n",
    "## Method 1: Search Engines\n",
    "\n",
    "You know this one.\n",
    "\n",
    "## Method 2: Query a database\n",
    "\n",
    "A more sophisticated way to search for papers.\n",
    "\n",
    "[Scopus Database](https://www.scopus.com/search)\n",
    "\n",
    "\n",
    "## Method 3: Database API (Scopus)\n",
    "\n",
    "Scopus is the largest database of articles and a great place to start.\n",
    "\n",
    "### Pre-requisites\n",
    "1. A Scopus account (sign up through the institution).\n",
    "2. An [API key](https://dev.elsevier.com/) (follow the instructions).\n",
    "3. For full access (while at home), you will also need an institutional access token (these are hard to acquire)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3b3f7fdc-46d1-40a2-8e85-5a0a78755548",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "with open('es-api-key.yml', 'r') as f:\n",
    "    data = yaml.load(f, Loader=yaml.SafeLoader)\n",
    "     \n",
    "api_key = data['apiKey']\n",
    "inst_token = data['instToken']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2662c0-293c-4929-93d7-8276fef7903f",
   "metadata": {},
   "source": [
    "## Python Requests\n",
    "\n",
    "Fundamentally, creating an API call is the same as constructing a web address with a specific structure called a **well-known text** (WKT). \n",
    "\n",
    "`requests` is a Python library specifically for this use case.\n",
    "\n",
    "We need: \n",
    "\n",
    "1. A base URL (the parent)\n",
    "2. A set of \"query\" parameters (this changes across APIs -- read the docs for your specific API (e.g., EIA, NOAA, Twitter, etc.)).\n",
    "\n",
    "[This page](https://dev.elsevier.com/documentation/ScopusSearchAPI.wadl#simple) lists the necessary query params."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "78522cde-1cc7-461b-8bb1-7b182a2574ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json # for reading the results.\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "023ed36e-fa67-440a-8cbb-a800532ecfa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "SEARCH = \"http://api.elsevier.com/content/search/scopus\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3de4e081-86fd-423b-bc8b-5bc3b968ee98",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"TITLE-ABS-KEY ( \\\"method of characteristics\\\" ) AND TITLE-ABS-KEY ( \\\"neutron transport\\\" )\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b83a3147-9909-4184-8c17-559e339772dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "index = 0\n",
    "view = 'COMPLETE'\n",
    "par = {'apikey': api_key, 'insttoken':inst_token, \n",
    "       'query': query, 'start': index,\n",
    "       'httpAccept': 'application/json', 'view': view}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "86bedbc4-8b76-491d-abee-de4cb169dab2",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(SEARCH, params=par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d6ba6128-f5c8-482e-bff2-42b7319c2c06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82d92328-5218-4b5c-810b-b655cc6407b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['opensearch:totalResults', 'opensearch:startIndex', 'opensearch:itemsPerPage', 'opensearch:Query', 'link', 'entry'])\n"
     ]
    }
   ],
   "source": [
    "results = response.json()\n",
    "print(results['search-results'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8d4e36a4-d917-49ea-80c5-fd1b5b97c8da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'239'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = results['search-results']\n",
    "results['opensearch:totalResults']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "59bc42e1-68f9-4cfe-a015-a372336dd2f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>@_fa</th>\n",
       "      <th>link</th>\n",
       "      <th>prism:url</th>\n",
       "      <th>dc:identifier</th>\n",
       "      <th>eid</th>\n",
       "      <th>dc:title</th>\n",
       "      <th>dc:creator</th>\n",
       "      <th>prism:publicationName</th>\n",
       "      <th>prism:issn</th>\n",
       "      <th>prism:eIssn</th>\n",
       "      <th>...</th>\n",
       "      <th>source-id</th>\n",
       "      <th>fund-acr</th>\n",
       "      <th>fund-no</th>\n",
       "      <th>fund-sponsor</th>\n",
       "      <th>openaccess</th>\n",
       "      <th>openaccessFlag</th>\n",
       "      <th>prism:issueIdentifier</th>\n",
       "      <th>prism:isbn</th>\n",
       "      <th>freetoread</th>\n",
       "      <th>freetoreadLabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>true</td>\n",
       "      <td>[{'@_fa': 'true', '@ref': 'self', '@href': 'ht...</td>\n",
       "      <td>https://api.elsevier.com/content/abstract/scop...</td>\n",
       "      <td>SCOPUS_ID:85185561082</td>\n",
       "      <td>2-s2.0-85185561082</td>\n",
       "      <td>A memory-efficient neutron noise algorithm for...</td>\n",
       "      <td>Cosgrove P.</td>\n",
       "      <td>Annals of Nuclear Energy</td>\n",
       "      <td>03064549</td>\n",
       "      <td>18732100</td>\n",
       "      <td>...</td>\n",
       "      <td>29363</td>\n",
       "      <td>EPSRC</td>\n",
       "      <td>EP/T022159/1</td>\n",
       "      <td>Engineering and Physical Sciences Research Cou...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>true</td>\n",
       "      <td>[{'@_fa': 'true', '@ref': 'self', '@href': 'ht...</td>\n",
       "      <td>https://api.elsevier.com/content/abstract/scop...</td>\n",
       "      <td>SCOPUS_ID:85184710274</td>\n",
       "      <td>2-s2.0-85184710274</td>\n",
       "      <td>Time-dependent neutron transport method based ...</td>\n",
       "      <td>Song Q.</td>\n",
       "      <td>Annals of Nuclear Energy</td>\n",
       "      <td>03064549</td>\n",
       "      <td>18732100</td>\n",
       "      <td>...</td>\n",
       "      <td>29363</td>\n",
       "      <td>NSFC</td>\n",
       "      <td>12105170</td>\n",
       "      <td>National Natural Science Foundation of China</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   @_fa                                               link  \\\n",
       "0  true  [{'@_fa': 'true', '@ref': 'self', '@href': 'ht...   \n",
       "1  true  [{'@_fa': 'true', '@ref': 'self', '@href': 'ht...   \n",
       "\n",
       "                                           prism:url          dc:identifier  \\\n",
       "0  https://api.elsevier.com/content/abstract/scop...  SCOPUS_ID:85185561082   \n",
       "1  https://api.elsevier.com/content/abstract/scop...  SCOPUS_ID:85184710274   \n",
       "\n",
       "                  eid                                           dc:title  \\\n",
       "0  2-s2.0-85185561082  A memory-efficient neutron noise algorithm for...   \n",
       "1  2-s2.0-85184710274  Time-dependent neutron transport method based ...   \n",
       "\n",
       "    dc:creator     prism:publicationName prism:issn prism:eIssn  ...  \\\n",
       "0  Cosgrove P.  Annals of Nuclear Energy   03064549    18732100  ...   \n",
       "1      Song Q.  Annals of Nuclear Energy   03064549    18732100  ...   \n",
       "\n",
       "  source-id fund-acr       fund-no  \\\n",
       "0     29363    EPSRC  EP/T022159/1   \n",
       "1     29363     NSFC      12105170   \n",
       "\n",
       "                                        fund-sponsor openaccess  \\\n",
       "0  Engineering and Physical Sciences Research Cou...          1   \n",
       "1       National Natural Science Foundation of China          0   \n",
       "\n",
       "  openaccessFlag prism:issueIdentifier prism:isbn freetoread freetoreadLabel  \n",
       "0           True                   NaN        NaN        NaN             NaN  \n",
       "1          False                   NaN        NaN        NaN             NaN  \n",
       "\n",
       "[2 rows x 36 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(results['entry'])\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7d883f76-3740-44c3-a3ad-26c513ca0c3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The neutron noise equation contains a fixed source term where the neutron angular flux is present. Deterministic noise solvers have tended to solve for and store this angular flux term in a preprocessing step. This can be a substantial memory burden as the angular flux is a function of space, energy, and angle. This can limit these solvers to smaller problems and can limit the obtainable angular resolution. This paper proposes solving the neutron noise equation and the standard eigenvalue neutron transport equation simultaneously, reducing the memory footprint of a noise solve while making frequency-domain method of characteristics solvers practical. This approach is demonstrated using the random ray method but is applicable to any sweep-based methods. Using the random ray method also has the advantage of exact geometry representation, no ray effects, and cheaper transport sweeps, all of which have been problematic. These advantages are illustrated on several noise problems.'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['dc:description'].values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30a0c33-ebda-4dd5-a490-c8acbe57315c",
   "metadata": {},
   "source": [
    "## A better way... \n",
    "\n",
    "\n",
    "`pybliometrics` abstracts the API calls \n",
    "\n",
    "\n",
    "```{note}\n",
    "This library requires you to be on campus, even with the \"STANDARD\" view, unless you have an institutional access token.\n",
    "```\n",
    "\n",
    "The first time you run `pybliometrics` from the commandline, it will ask you to type in your access credentials (I have already done this)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0ffed00b-f78c-4c98-ae5f-ea148d7f9640",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pybliometrics.scopus import ScopusSearch\n",
    "\n",
    "x = ScopusSearch(query=query, view='COMPLETE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7261a7a7-b6d6-4655-b3e2-afab38bb0d82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "239"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.get_results_size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "61621978-1215-4470-9e37-9f3b7509e102",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "239"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eid</th>\n",
       "      <th>doi</th>\n",
       "      <th>pii</th>\n",
       "      <th>pubmed_id</th>\n",
       "      <th>title</th>\n",
       "      <th>subtype</th>\n",
       "      <th>subtypeDescription</th>\n",
       "      <th>creator</th>\n",
       "      <th>afid</th>\n",
       "      <th>affilname</th>\n",
       "      <th>...</th>\n",
       "      <th>pageRange</th>\n",
       "      <th>description</th>\n",
       "      <th>authkeywords</th>\n",
       "      <th>citedby_count</th>\n",
       "      <th>openaccess</th>\n",
       "      <th>freetoread</th>\n",
       "      <th>freetoreadLabel</th>\n",
       "      <th>fund_acr</th>\n",
       "      <th>fund_no</th>\n",
       "      <th>fund_sponsor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2-s2.0-85185561082</td>\n",
       "      <td>10.1016/j.anucene.2024.110450</td>\n",
       "      <td>S0306454924001130</td>\n",
       "      <td>None</td>\n",
       "      <td>A memory-efficient neutron noise algorithm for...</td>\n",
       "      <td>ar</td>\n",
       "      <td>Article</td>\n",
       "      <td>Cosgrove P.</td>\n",
       "      <td>60120016</td>\n",
       "      <td>Department of Engineering</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>The neutron noise equation contains a fixed so...</td>\n",
       "      <td>Method of characteristics | Neutron noise | Ne...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>EPSRC</td>\n",
       "      <td>EP/T022159/1</td>\n",
       "      <td>Engineering and Physical Sciences Research Cou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2-s2.0-85184710274</td>\n",
       "      <td>10.1016/j.anucene.2023.110334</td>\n",
       "      <td>S0306454923006539</td>\n",
       "      <td>None</td>\n",
       "      <td>Time-dependent neutron transport method based ...</td>\n",
       "      <td>ar</td>\n",
       "      <td>Article</td>\n",
       "      <td>Song Q.</td>\n",
       "      <td>60025084</td>\n",
       "      <td>Shanghai Jiao Tong University</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>The 3D Method of Characteristics (MOC) has hig...</td>\n",
       "      <td>C5G7-TD | Method of characteristics | Neutron ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NSFC</td>\n",
       "      <td>12105170</td>\n",
       "      <td>National Natural Science Foundation of China</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  eid                            doi                pii  \\\n",
       "0  2-s2.0-85185561082  10.1016/j.anucene.2024.110450  S0306454924001130   \n",
       "1  2-s2.0-85184710274  10.1016/j.anucene.2023.110334  S0306454923006539   \n",
       "\n",
       "  pubmed_id                                              title subtype  \\\n",
       "0      None  A memory-efficient neutron noise algorithm for...      ar   \n",
       "1      None  Time-dependent neutron transport method based ...      ar   \n",
       "\n",
       "  subtypeDescription      creator      afid                      affilname  \\\n",
       "0            Article  Cosgrove P.  60120016      Department of Engineering   \n",
       "1            Article      Song Q.  60025084  Shanghai Jiao Tong University   \n",
       "\n",
       "   ... pageRange                                        description  \\\n",
       "0  ...      None  The neutron noise equation contains a fixed so...   \n",
       "1  ...      None  The 3D Method of Characteristics (MOC) has hig...   \n",
       "\n",
       "                                        authkeywords citedby_count openaccess  \\\n",
       "0  Method of characteristics | Neutron noise | Ne...             0          1   \n",
       "1  C5G7-TD | Method of characteristics | Neutron ...             0          0   \n",
       "\n",
       "  freetoread freetoreadLabel fund_acr       fund_no  \\\n",
       "0       None            None    EPSRC  EP/T022159/1   \n",
       "1       None            None     NSFC      12105170   \n",
       "\n",
       "                                        fund_sponsor  \n",
       "0  Engineering and Physical Sciences Research Cou...  \n",
       "1       National Natural Science Foundation of China  \n",
       "\n",
       "[2 rows x 36 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyb_df = pd.DataFrame([doc._asdict() for doc in x.results])\n",
    "display(len(pyb_df))\n",
    "pyb_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a6f7895-6d93-4b02-8ce4-9e4b839fc3a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42132b09-5a1a-48b9-8d86-e8ee6bcc52af",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}