cord19kg/examples/notebooks/Nexus-hosted co-occurrence analysis tutorail.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites and installation instructions for a local environment\n",
    "\n",
    "In order to run this notebook `graph-tool` must be installed manually (it cannot be installed as a part of `pip install bluegraph`, as it is not an ordinary Python library, but a wrapper around a C++ library). Please, see [graph-tool installation instructions](https://git.skewed.de/count0/graph-tool/-/wikis/installation-instructions#native-installation) (currently, BlueGraph supports :code:`graph-tool<=2.37`.)\n",
    "\n",
    "We recommend using `conda` for installing `graph-tool`. For example:\n",
    "\n",
    "```\n",
    "conda install -c conda-forge graph-tool==2.37\n",
    "```\n",
    "\n",
    "or as a part of a new `conda` environment:\n",
    "\n",
    "```\n",
    "conda create --name <your_environment> -c conda-forge graph-tool==2.37\n",
    "conda activate <your_environment>\n",
    "```\n",
    "\n",
    "\n",
    "BlueGraph and the set of dependecies supporting custom tools for CORD-19 analysis can be installed using:\n",
    "\n",
    " ```\n",
    " pip install bluegraph[cord19kg]\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nexus-hosted co-occurrence network analysis of CORD-19\n",
    "\n",
    "In this notebook we will illustrate how different datasets from the co-occurrence analysis can be pushed and accessed within a Blue Brain Nexus project. If you are not familiar with the co-occurrence analysis pipeline proposed in the `cord19kg` package, we recommend you to have a look at the `Co-occurrence analysis tutorial` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import json\n",
    "import jwt\n",
    "import yaml\n",
    "import zipfile\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import dash_cytoscape as cyto\n",
    "import nexussdk as nexus\n",
    "import pandas as pd\n",
    "\n",
    "from kgforge.core import KnowledgeGraphForge\n",
    "\n",
    "from cord19kg.apps.topic_widgets import (TopicWidget, DataSaverWidget)\n",
    "from cord19kg.utils import (generate_curation_table,\n",
    "                            link_ontology,\n",
    "                            generate_cooccurrence_analysis)\n",
    "from cord19kg.apps.curation_app import curation_app\n",
    "from cord19kg.apps.visualization_app import visualization_app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setting up a project and creating a `kgforge` configuration\n",
    "\n",
    "If you have already set up a Nexus project and generated a 'forge' configuration, simply get your access token (__1.1__) and go directly to the step __2. Set up a topic__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Login to Nexus and get the access token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Nexus web application](https://staging.nexus.ocp.bbp.epfl.ch/) can be used to login and get a token:\n",
    "\n",
    "1. Click on the login button on the right corner and follow the instructions.\n",
    "<img src=\"../figures/nexus_log_in.png\" alt=\"Drawing\" style=\"width: 1000px;\"/>\n",
    "\n",
    "\n",
    "2. Once logged in, click on the `Copy token` button. The token will be copied to the clipboard.\n",
    "<img src=\"../figures/nexus_logged_in.png\" alt=\"Drawing\" style=\"width: 1000px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below and paste the token in the input field generated by the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'getpass' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-de0337b623fa>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mTOKEN\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetpass\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetpass\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'getpass' is not defined"
     ]
    }
   ],
   "source": [
    "TOKEN = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, choose a new project name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # TODO: Bring back the sandbox endpoint\n",
    "# endpoint = \"https://sandbox.bluebrainnexus.io/v1\"\n",
    "# org =\"tutorialnexus\"\n",
    "# project =\"cord19kgExampleProject\"  # TODO: Choose a project name\n",
    "\n",
    "endpoint = \"https://staging.nexus.ocp.bbp.epfl.ch/v1\"\n",
    "org = \"covid19-kg\"\n",
    "project = \"data\"\n",
    "description = \"cord19kg save/load example project\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Create a Nexus project programmatically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below modify the variable `project` to chose a new project name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nexus.config.set_environment(endpoint)\n",
    "nexus.config.set_token(TOKEN)\n",
    "try:\n",
    "    nexus.projects.create(org_label=org,project_label=project, description=description)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Generate a `kgforge` configuration for your project\n",
    "\n",
    "The following cell automatically generates a configuration file for the project set up above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = dict()\n",
    "\n",
    "neuroshapes_path = \"../models/neuroshapes\"\n",
    "if not Path(neuroshapes_path).is_dir():\n",
    "    ! git clone https://github.com/INCF/neuroshapes.git $neuroshapes_path\n",
    "    ! cp -R $neuroshapes_path/shapes/neurosciencegraph/datashapes/core/dataset $neuroshapes_path/shapes/neurosciencegraph/commons/\n",
    "    ! cp -R $neuroshapes_path/shapes/neurosciencegraph/datashapes/core/activity $neuroshapes_path/shapes/neurosciencegraph/commons/\n",
    "    ! cp -R $neuroshapes_path/shapes/neurosciencegraph/datashapes/core/entity $neuroshapes_path/shapes/neurosciencegraph/commons/\n",
    "    ! cp -R $neuroshapes_path/shapes/neurosciencegraph/datashapes/core/ontology $neuroshapes_path/shapes/neurosciencegraph/commons/\n",
    "    ! cp -R $neuroshapes_path/shapes/neurosciencegraph/datashapes/core/person $neuroshapes_path/shapes/neurosciencegraph/commons/\n",
    "\n",
    "config['Model'] = {\n",
    "    \"name\": \"RdfModel\",\n",
    "    \"origin\": \"directory\",\n",
    "    \"source\": f\"{neuroshapes_path}/shapes/neurosciencegraph/commons/\",\n",
    "    \"context\": {\n",
    "        \"iri\": \"../models/neuroshapes_context.json\",\n",
    "    },\n",
    "}\n",
    "\n",
    "config[\"Store\"] = {\n",
    "    \"name\": \"BlueBrainNexus\",\n",
    "    \"endpoint\": endpoint,\n",
    "    \"searchendpoints\":{\n",
    "        \"sparql\":{\n",
    "            \"endpoint\":\"https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex\"\n",
    "        }\n",
    "    },\n",
    "    \"bucket\": f\"{org}/{project}\",\n",
    "    \"versioned_id_template\": \"{x.id}?rev={x._store_metadata._rev}\",\n",
    "    \"file_resource_mapping\": \"../config/file-to-resource-mapping.hjson\"\n",
    "}\n",
    "\n",
    "with open(\"../config/forge-config.yml\", \"w\") as f:\n",
    "    yaml.dump(config, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Set up a topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Blue Brain Nexus provides a prepared schema for storing CORD-19 datasets included as a part of this repository. The main entity of the underlying knowledge graph is a `Topic`: the user can create multiple topics of interest, annotate them and push/pull datasets related to the co-occurrence analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fist, let us create a 'forge' to manage (create, access and deploy) knowledge within the given Blue Brain Nexus Project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "forge_config_file = \"../config/forge-config.yml\"\n",
    "forge = KnowledgeGraphForge(forge_config_file, token=TOKEN, debug=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch the topic selection widget by executing the following cell. Using the widget, create a new topic using the respective widget tab. Go back to the `Select topic` tab, click on `List all your topics` and select the newly created topic in the dropdown menu. This tab also provides means for updating the selected topic meta-data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'TopicWidget' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-86a4cce7c42c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mwidget\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTopicWidget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mforge\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTOKEN\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mwidget\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'TopicWidget' is not defined"
     ]
    }
   ],
   "source": [
    "widget = TopicWidget(forge, TOKEN)\n",
    "widget.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having selected a topic using the selection widget, you retreive the `topic_resource_id` variable corresponding to the unique identifier of the topic in your Nexus project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'id'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-47c97950b84f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mvisualization_configs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;31m# By default, no visualization app configs are provided\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mtopic_resource_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwidget\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_topic_resource_id\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mtopic_resource_id\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/bluegraph/lib/python3.6/site-packages/cord19kg/apps/topic_widgets.py\u001b[0m in \u001b[0;36mget_topic_resource_id\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    408\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    409\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_topic_resource_id\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 410\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_topic_resource\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    411\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    412\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_all\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'id'"
     ]
    }
   ],
   "source": [
    "visualization_configs = None # By default, no visualization app configs are provided\n",
    "\n",
    "topic_resource_id = widget.get_topic_resource_id()\n",
    "topic_resource_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run the curation/analysis pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Load the input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "occurrence_data = pd.read_csv(\"../data/Glucose_risk_20_papers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Transform the input data into aggregated occurrences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_input_table, factor_counts = generate_curation_table(occurrence_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Load the ontology linking data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "print(\"Loading the ontology linking data...\")\n",
    "    \n",
    "print(\"\\tDecompressing the input data file...\")\n",
    "with zipfile.ZipFile(\"../data/NCIT_ontology_linking_3000_papers.csv.zip\", 'r') as zip_ref:\n",
    "    zip_ref.extractall(\"../data/\")\n",
    "\n",
    "print(\"\\tLoading the linking dataframe in memory...\")\n",
    "ontology_linking = pd.read_csv(\"../data/NCIT_ontology_linking_3000_papers.csv\")\n",
    "\n",
    "print(\"\\tLoading ontology type mapping...\")\n",
    "with open(\"../data/NCIT_type_mapping.json\", \"rb\") as f:\n",
    "    type_mapping = json.load(f)\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4. Initialize and run the curation app\n",
    "\n",
    "(If you don't perform ontology linking you may experience some errors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "curation_app.set_table(curation_input_table.copy())\n",
    "default_entities_to_keep = [\"glucose\", \"covid-19\"]\n",
    "curation_app.set_default_terms_to_include(default_entities_to_keep)\n",
    "curation_app.set_ontology_linking_callback(lambda x: link_ontology(ontology_linking, type_mapping, x))\n",
    "curation_app.run(port=8074, mode=\"inline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5. Retrieve curated data table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If this cell produces KeyError, make sure you clicked Linked NCIT ontology in the app\n",
    "curated_occurrence_data = curation_app.get_curated_table()\n",
    "curation_meta_data = {\n",
    "    \"factor_counts\": factor_counts,\n",
    "    \"nodes_to_keep\": curation_app.get_terms_to_include(),\n",
    "    \"n_most_frequent\": curation_app.n_most_frequent if curation_app.n_most_frequent else 100\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4. Generate co-mention networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas.api.types import is_numeric_dtype, is_string_dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_numeric_dtype(curated_occurrence_data[\"paper_frequency\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "type_data = curated_occurrence_data[[\"entity_type\"]].rename(columns={\"entity_type\": \"type\"})\n",
    "\n",
    "graphs, trees = generate_cooccurrence_analysis(\n",
    "    curated_occurrence_data,  factor_counts,\n",
    "    n_most_frequent=curation_app.n_most_frequent if curation_app.n_most_frequent else 100,\n",
    "    type_data=type_data, \n",
    "    factors=[\"paper\", \"paragraph\"],\n",
    "    keep=curation_app.get_terms_to_include(),\n",
    "    cores=10)\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5. Initialize and run the visualization app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app.set_backend(\"graph_tool\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app.add_graph(\n",
    "    \"Paper-based graph\", graphs[\"paper\"],\n",
    "    tree=trees[\"paper\"], default_top_n=100)\n",
    "visualization_app.add_graph(\n",
    "    \"Paragraph-based graph\", graphs[\"paragraph\"],\n",
    "    tree=trees[\"paragraph\"], default_top_n=100)\n",
    "visualization_app.set_current_graph(\"Paragraph-based graph\")\n",
    "\n",
    "paper_data = pd.read_csv(\"../data/Glucose_risk_3000_paper_meta_data.csv\")\n",
    "paper_data = paper_data.set_index(\"id\")\n",
    "paper_data.head(3)\n",
    "\n",
    "def list_papers(paper_data, selected_papers, limit=200):\n",
    "    selected_paper_data = paper_data.loc[[int(p) for p in selected_papers]].head(200)\n",
    "    return selected_paper_data.to_dict(\"records\")\n",
    "\n",
    "visualization_app.set_list_papers_callback(lambda x: list_papers(paper_data, x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_n(data_dict, n, smallest=False):\n",
    "    \"\"\"Return top `n` keys of the input dictionary by their value.\"\"\"\n",
    "    df = pd.DataFrame(dict(data_dict).items(), columns=[\"id\", \"value\"])\n",
    "    if smallest:\n",
    "        df = df.nsmallest(n, columns=[\"value\"])\n",
    "    else:\n",
    "        df = df.nlargest(n, columns=[\"value\"])\n",
    "    return(list(df[\"id\"]))\n",
    "\n",
    "\n",
    "def get_aggregated_entities(entity, n):\n",
    "    if \"aggregated_entities\" in curated_occurrence_data.columns:\n",
    "        aggregated = curated_occurrence_data.loc[entity][\"aggregated_entities\"]\n",
    "    else:\n",
    "        aggregated = [entity]\n",
    "    if curation_input_table is not None:\n",
    "        df = curation_input_table.set_index(\"entity\")\n",
    "        if entity in curated_occurrence_data.index:\n",
    "            freqs = df.loc[aggregated][\"paper_frequency\"].to_dict()\n",
    "        else:\n",
    "            return {}\n",
    "    else:\n",
    "        df = data.copy()\n",
    "        df[\"entity\"] = data[\"entity\"].apply(lambda x: x.lower())\n",
    "        freqs = df[df[\"entity\"].apply(lambda x: x.lower() in aggregated)].groupby(\"entity\").aggregate(\n",
    "            lambda x: len(x))[\"entity_type\"].to_dict()\n",
    "    if len(freqs) == 0:\n",
    "        return {}\n",
    "    return {e: freqs[e] for e in top_n(freqs, n)}\n",
    "\n",
    "visualization_app.set_aggregated_entities_callback(\n",
    "    lambda x: get_aggregated_entities(x, 10))\n",
    "\n",
    "definitions = ontology_linking[[\"concept\", \"definition\"]].groupby(\n",
    "    \"concept\").aggregate(lambda x: list(x)[0]).to_dict()[\"definition\"]\n",
    "visualization_app.set_entity_definitons(definitions)\n",
    "\n",
    "if visualization_configs is not None:\n",
    "    visualization_app._current_graph = visualization_configs[\"current_graph\"]\n",
    "    visualization_app._configure_layout(visualization_configs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below loads additional graph layouts used in the graph visualization app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cyto.load_extra_layouts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app.run(port=8084, mode=\"external\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Save datasets under the selected topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bluegraph.backends.graph_tool import GTGraphProcessor\n",
    "from cord19kg.apps.resources import CORD19_PROP_TYPES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc = GTGraphProcessor.from_graph_object(visualization_app._graphs['Paper-based graph'][\"object\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CORD19_PROP_TYPES[\"nodes\"][\"paper_frequency\"] = \"numeric\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc.get_pgframe(node_prop_types=CORD19_PROP_TYPES[\"nodes\"], edge_prop_types=CORD19_PROP_TYPES[\"edges\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app._graphs.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exported_graphs = visualization_app.export_graphs(\n",
    "    [\"Paper-based graph\", \"Paragraph-based graph\"], \n",
    ")\n",
    "visualization_configs = visualization_app.get_configs()\n",
    "edit_history = visualization_app.get_edit_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can launch the saver widget and, using the provided form, specify the name and the descripton of the dataset. To save the dataset, press `Register Dataset`. If you want to upload a new version of already existing dataset you can click on `Tag Dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "saver_widget = DataSaverWidget(\n",
    "    forge, TOKEN, topic_resource_id,\n",
    "    occurrence_data,\n",
    "    curated_occurrence_data,\n",
    "    curation_meta_data,\n",
    "    exported_graphs,\n",
    "    visualization_configs,\n",
    "    edit_history,\n",
    "    temp_prefix=\"../data\")\n",
    "\n",
    "saver_widget.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Load datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previously seen topic selection widget allows to load the datasets pushed to Nexus. Have selected a topic press the `Show datasets for selected topic` button. On the right you will see all the datasets already available for this topic in your Nexus project. Select a dataset and click `Reuse selected dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "widget = TopicWidget(forge, TOKEN)\n",
    "widget.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By executing the cell below, you can retrieve the following variables (if available, otherwise `None`):\n",
    "\n",
    "- `occurrence_data` table containing raw entity occurrence data;\n",
    "- `curated_occurrence_data` and `curation_meta_data` table containing curated entity occurrence data and some curation meta-data;\n",
    "- `loaded_graphs` graph objects saved under the current topic resource;\n",
    "- `visualization_configs` configurations of the last graph visualization session of the selected topic.\n",
    "\n",
    "If you want to restart the analysis pipeline starting from the raw entity occurrence data (loaded from Nexus), you can proceed directly to the section 3.2.\n",
    "\n",
    "If you want to restart the analysis pipeline starting from the curated table, you can proceed directly to 3.4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    occurrence_data,\n",
    "    curated_occurrence_data,\n",
    "    curation_meta_data,\n",
    "    loaded_graphs,\n",
    "    visualization_configs,\n",
    "    topic_resource_id\n",
    ") = widget.get_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to restart the analysis pipeline from the previously generated graphs, you can run the following cell and proceed directly to 3.5. Note that the last session of the configuration app will be restored by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graphs[\"paper\"] = loaded_graphs[\"Paper-based graph\"][\"graph\"]\n",
    "graphs[\"paragraph\"] = loaded_graphs[\"Paragraph-based graph\"][\"graph\"]\n",
    "\n",
    "trees[\"paper\"] = loaded_graphs[\"Paper-based graph\"][\"tree\"]\n",
    "trees[\"paragraph\"] = loaded_graphs[\"Paragraph-based graph\"][\"tree\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bluegraph",
   "language": "python",
   "name": "bluegraph"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}