walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SWP-Group 1. Scripts – Walkthrough:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal is to use modified and unmodified graphs with some metadata as input to S2Match, after which we can evaluate our method. <br>\n",
    "\n",
    "In order to do this you must already have an (STS|SICK)-dataset:\n",
    "\n",
    ">1. which we then would convert to a tsv-file, that would contain sentences from all the categories and their scores <br>\n",
    ">2. we can then use this tsv-file to create 2 files with AMRs in them (one file for sentences A and one file for sentences B) <br>\n",
    ">3. for the AMR files we create AMR2Texts alignments <br>\n",
    ">4. do some graph modification or extraction of metadata from these alignments (AMRAnalysis.py) <br>\n",
    ">5. feed the results from 4) to S2Match <br>\n",
    ">6. evaluate different approaches on original (STS|SICK)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparing data for processing/analysis:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.0 Import and helper functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import subprocess, shlex\n",
    "from pathlib import Path\n",
    "from collections import defaultdict\n",
    "\n",
    "def run_commands(commands, shell=False):\n",
    "    \"\"\"\n",
    "    This function executes shell commands from a list of commands.\n",
    "    The disadvantage of this function is that the output is only shown after execution.\n",
    "   \n",
    "   \n",
    "    Args:\n",
    "        commands (list): list of commands (str) to be executed\n",
    "        shell (bool, optional): if \"True\" the process is executed through the shell\n",
    "        \n",
    "    \"\"\"\n",
    "    for command in commands:\n",
    "        command = command.strip()\n",
    "        print(f'\"{command}\"', '\\nis being executed...','\\n')\n",
    "        res = subprocess.run(shlex.split(command), stdout=subprocess.PIPE, \n",
    "                             stderr=subprocess.STDOUT, text=True, shell=shell)\n",
    "        for line in res.stdout.split('\\n'):            \n",
    "            if line:\n",
    "                print(line)\n",
    "        print()\n",
    "\n",
    "# venvs on my PC, change according to your needs!\n",
    "# win_env has the latest penman and amrlib\n",
    "# linux_env has all dependencies for the AMR2Text-tool and penman==0.6.2\n",
    "win_env   = 'conda activate dlp38 && python'\n",
    "linux_env = 'wsl python3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Running scripts to convert datasets to AMR and align them with text:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commands to run in a shell:\n",
    "```shell\n",
    "{virtualenv} python3 sts2tsv.py -i datasets/sts/sts2016-english-with-gs-v1.0 -o data/STS2016_full.tsv \n",
    "{virtualenv with penman>=1.0} python3 to_amr.py -i data/STS2016_full.tsv -o data/amr/STS2016_corpus\n",
    "{virtualenv with penman==0.6.2} python3 amr_pipeline.py -t AMR2Text -i data/STS2016_full.tsv -o data/amr/STS2016_corpus\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.1 Commands for STS-dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model C:\\ProgramData\\Anaconda3\\envs\\dlp38\\Lib\\site-packages\\amrlib\\data\\model_stog\\model.pt\n",
      "Stog model loaded sucessfully!\n"
     ]
    }
   ],
   "source": [
    "commands_on_my_pc_sts = [\n",
    "    f'{linux_env} sts2tsv.py -i datasets/sts/sts2016-english-with-gs-v1.0 -o data/STS2016_full.tsv',\n",
    "    f'{win_env} tsv2amr.py -i data/STS2016_full_fix.tsv -o data/amr/STS2016_corpus',\n",
    "    f'{linux_env} amr_pipeline.py -t AMR2Text -o data/amr/STS2016_corpus'\n",
    "]\n",
    "\n",
    "# Run here if necessary, but the output is not live (only after command is completed)!\n",
    "#run_commands(commands_on_my_pc_sick, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.2 Commands for SICK-dataset + subsetting for neutral and positive entailment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"conda activate dlp38 && python sick2tsv.py -i datasets/sick/SICK2014_full.txt -o data/SICK2014.tsv --entailment_exclude contradiction\" \n",
      "is being executed... \n",
      "\n",
      "File \"datasets/sick/SICK2014_full.txt\" successfully converted to \"data/SICK2014.tsv\"!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "commands_on_my_pc_sick = [\n",
    "    f'{win_env} sick2tsv.py -i datasets/sick/SICK2014_full.txt -o data/SICK2014.tsv --entailment_exclude contradiction',\n",
    "    f'{win_env} tsv2amr.py -i data/SICK2014.tsv -o data/amr/SICK2014_corpus',\n",
    "    f'{linux_env} amr_pipeline.py -t AMR2Text -o data/amr/SICK2014_corpus'\n",
    "]\n",
    "\n",
    "# Run here if necessary, but the output is not live (only after command is completed)!\n",
    "run_commands(commands_on_my_pc_sick, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Resetting index in alignments files:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The alignment tool is not perfect and sometimes it can't align certain sentences (in STS, in SICK we didn't have any errors). <br>\n",
    "We could try to rewrite the code, so that these sentences are aligned, but it is pretty unfeasible, so we just throw all \"bad\" sentences out! <br> <br>\n",
    "After we have discarded \"bad\" sentences from the alignments file, we, nonetheless, still want to maintain correct indices of the AMRs, so we have to reset the indexing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 496 in b, others in a\n",
    "sentences_with_errors_sts = [98, 119, 289, 296, 392, 496, 630, 654, 660]\n",
    "\n",
    "def reset_ids_in_aligned():\n",
    "    p = Path('.')/'data'/'amr'\n",
    "    paths = [p/'STS2016_corpus_a_aligned.mrp', p/'STS2016_corpus_b_aligned.mrp']\n",
    "    paths_to_save = [p/'STS2016_corpus_a_aligned_e.mrp', p/'STS2016_corpus_b_aligned_e.mrp']\n",
    "    edited = []\n",
    "\n",
    "    for i, p in enumerate(paths):\n",
    "        content = p.read_text().split('\\n\\n')\n",
    "        with open(paths_to_save[i], 'a') as out:\n",
    "            for a, c in enumerate(content):\n",
    "                if c:\n",
    "                    lines = c.split('\\n')\n",
    "                    id_line_spl = lines[0].split()\n",
    "                    id_line_spl[-1] = str(a)\n",
    "                    lines[0] = ' '.join(id_line_spl)\n",
    "                    c = '\\n'.join(lines) + '\\n\\n'\n",
    "                    out.write(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. AMRAnalysis (transforming graphs or just adding alignments metadata):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp     --output_prefix analysis/sick/SICK2014\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=False, extended_meta=False\n",
      "File: \"analysis\\sick\\SICK2014_corpus_a_reif.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sick\\SICK2014_corpus_b_reif.amr\" was sucessfully saved!\n",
      "\n",
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp     --output_prefix analysis/sick/SICK2014 --extended_meta\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=False, extended_meta=True\n",
      "File: \"analysis\\sick\\SICK2014_corpus_a_reif_ext.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sick\\SICK2014_corpus_b_reif_ext.amr\" was sucessfully saved!\n",
      "\n",
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp     --output_prefix analysis/sick/SICK2014 --concat_rel\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=True, extended_meta=False\n",
      "File: \"analysis\\sick\\SICK2014_corpus_a_concat.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sick\\SICK2014_corpus_b_concat.amr\" was sucessfully saved!\n",
      "\n",
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp     --output_prefix analysis/sts/STS2016\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=False, extended_meta=False\n",
      "File: \"analysis\\sts\\STS2016_corpus_a_reif.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sts\\STS2016_corpus_b_reif.amr\" was sucessfully saved!\n",
      "\n",
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp     --output_prefix analysis/sts/STS2016 --extended_meta\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=False, extended_meta=True\n",
      "File: \"analysis\\sts\\STS2016_corpus_a_reif_ext.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sts\\STS2016_corpus_b_reif_ext.amr\" was sucessfully saved!\n",
      "\n",
      "\"conda activate dlp38 && python AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp     --output_prefix analysis/sts/STS2016 --concat_rel\" \n",
      "is being executed... \n",
      "\n",
      "Input parameters: concat_rel=True, extended_meta=False\n",
      "File: \"analysis\\sts\\STS2016_corpus_a_concat.amr\" was sucessfully saved!\n",
      "File: \"analysis\\sts\\STS2016_corpus_b_concat.amr\" was sucessfully saved!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "commands = [\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sick/SICK2014\n",
    "    \"\"\",\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sick/SICK2014 --extended_meta\n",
    "    \"\"\",\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/SICK2014_corpus_a_aligned.mrp data/amr/SICK2014_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sick/SICK2014 --concat_rel\n",
    "    \"\"\",\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sts/STS2016\n",
    "    \"\"\",\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sts/STS2016 --extended_meta\n",
    "    \"\"\",\n",
    "    f\"\"\"{win_env} AMRAnalysis.py -i data/amr/STS2016_corpus_a_aligned.mrp data/amr/STS2016_corpus_b_aligned.mrp \\\n",
    "    --output_prefix analysis/sts/STS2016 --concat_rel\n",
    "    \"\"\",\n",
    "]\n",
    "\n",
    "run_commands(commands, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Running S2Match scripts on the analysed data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py     -f analysis/sick/SICK2014_corpus_a_reif.amr analysis/sick/SICK2014_corpus_b_reif.amr     -vectors amr_suite/vectors/glove.6B.100d.txt     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_glove_results/SICK2014_orig_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "\n",
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py     -f analysis/sick/SICK2014_corpus_a_reif_ext.amr analysis/sick/SICK2014_corpus_b_reif_ext.amr     -vectors amr_suite/vectors/glove.6B.100d.txt     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_glove_results/SICK2014_concat_ver2_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "C:\\ProgramData\\Anaconda3\\envs\\dlp38\\lib\\site-packages\\scipy\\spatial\\distance.py:714: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  dist = 1.0 - uv / np.sqrt(uu * vv)\n",
      "\n",
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py     -f analysis/sick/SICK2014_corpus_a_concat.amr analysis/sick/SICK2014_corpus_b_concat.amr     -vectors amr_suite/vectors/glove.6B.100d.txt     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_glove_results/SICK2014_concat_ver1_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "\n",
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py     -f analysis/sick/SICK2014_corpus_a_reif.amr analysis/sick/SICK2014_corpus_b_reif.amr     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_sbert_results/SICK2014_orig_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "2021-03-22 17:28:23.186152: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll\n",
      "Precomputing embeddings for file 1 and file 2...\n",
      "Finished! Running S2Match...\n",
      "Embeddings dims = torch.Size([16834, 39, 768])\n",
      "\n",
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py     -f analysis/sick/SICK2014_corpus_a_reif_ext.amr analysis/sick/SICK2014_corpus_b_reif_ext.amr     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_sbert_results/SICK2014_concat_ver2_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "2021-03-22 17:57:10.899753: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll\n",
      "Precomputing embeddings for file 1 and file 2...\n",
      "Finished! Running S2Match...\n",
      "Embeddings dims = torch.Size([16834, 39, 768])\n",
      "\n",
      "conda activate dlp38 && python amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py     -f analysis/sick/SICK2014_corpus_a_concat.amr analysis/sick/SICK2014_corpus_b_concat.amr     -diffsense 0.5 -cutoff 0.5 -v --ms     > analysis/sick/s2match_sbert_results/SICK2014_concat_ver1_results_full.txt\n",
      "     \n",
      "is being executed... \n",
      "\n",
      "2021-03-22 18:26:59.752559: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll\n",
      "Precomputing embeddings for file 1 and file 2...\n",
      "Finished! Running S2Match...\n",
      "Embeddings dims = torch.Size([16834, 39, 768])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# vectors are in \"amr_suite/vectors/glove.6B.100d.txt\"\n",
    "# Run s2match script on both of them:\n",
    "commands_block1 = [\n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_reif.amr analysis/sick/SICK2014_corpus_b_reif.amr \\\n",
    "    -vectors amr_suite/vectors/glove.6B.100d.txt \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_glove_results/SICK2014_orig_results_full.txt\n",
    "    \"\"\",\n",
    "    \n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_reif_ext.amr analysis/sick/SICK2014_corpus_b_reif_ext.amr \\\n",
    "    -vectors amr_suite/vectors/glove.6B.100d.txt \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_glove_results/SICK2014_concat_ver2_results_full.txt\n",
    "    \"\"\",\n",
    "    \n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_glove.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_concat.amr analysis/sick/SICK2014_corpus_b_concat.amr \\\n",
    "    -vectors amr_suite/vectors/glove.6B.100d.txt \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_glove_results/SICK2014_concat_ver1_results_full.txt\n",
    "    \"\"\"\n",
    "]\n",
    "\n",
    "commands_block2 = [\n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_reif.amr analysis/sick/SICK2014_corpus_b_reif.amr \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_sbert_results/SICK2014_orig_results_full.txt\n",
    "    \"\"\",\n",
    "    \n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_reif_ext.amr analysis/sick/SICK2014_corpus_b_reif_ext.amr \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_sbert_results/SICK2014_concat_ver2_results_full.txt\n",
    "    \"\"\",\n",
    "    \n",
    "    f\"\"\"{win_env} amr_suite/py3-Smatch-and-S2match/smatch/s2matchdev_sbert.py \\\n",
    "    -f analysis/sick/SICK2014_corpus_a_concat.amr analysis/sick/SICK2014_corpus_b_concat.amr \\\n",
    "    -diffsense 0.5 -cutoff 0.5 -v --ms \\\n",
    "    > analysis/sick/s2match_sbert_results/SICK2014_concat_ver1_results_full.txt\n",
    "    \"\"\"\n",
    "]\n",
    "\n",
    "run_commands(commands_block1, shell=True)\n",
    "run_commands(commands_block2, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluating on STS/SICK gold-scores and visualizing results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"conda activate dlp38 && python results2png.py --dataset STS --gold data/STS2016_full_fix.tsv     --smatch analysis/sts/s2match_glove_results analysis/sts/s2match_sbert_results     --output analysis/sts/s2match_modification_results.png\" \n",
      "is being executed... \n",
      "\n",
      "2021-03-26 18:53:37.484792: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll\n",
      "\n",
      "\"conda activate dlp38 && python results2png.py --dataset SICK --gold analysis/SICK2014_full_scores.tsv     --output analysis/sick/s2match_modification_results.png\" \n",
      "is being executed... \n",
      "\n",
      "2021-03-26 18:53:50.933120: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll\n",
      "\n"
     ]
    }
   ],
   "source": [
    "commands = [\n",
    "    f\"\"\"{win_env} results2png.py --dataset STS --gold data/STS2016_full_fix.tsv \\\n",
    "    --smatch analysis/sts/s2match_glove_results analysis/sts/s2match_sbert_results \\\n",
    "    --output analysis/sts/s2match_modification_results.png\"\"\",\n",
    "    \n",
    "    f\"\"\"{win_env} results2png.py --dataset SICK --gold analysis/SICK2014_full_scores.tsv \\\n",
    "    --output analysis/sick/s2match_modification_results.png\"\"\"\n",
    "]\n",
    "run_commands(commands, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other experiments (can be ignored):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic version of the concatenation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(MRPNode-0 / possible-01\n",
      "           :ARG1 (MRPNode-1 / wrong-02\n",
      "                            :ARG1 (MRPNode-2 / amr-unknown)\n",
      "                            :ARG2 (MRPNode-3 / air_i_conditioner))) \n",
      "\n",
      "(MRPNode-0 / possible-01\n",
      "           :ARG1 (MRPNode-1 / wrong-02\n",
      "                            :ARG1 (MRPNode-2 / amr-unknown)\n",
      "                            :ARG2 (MRPNode-3 / conditioner\n",
      "                                             :mod (MRPNode-4 / i)\n",
      "                                             :mod (MRPNode-5 / air))))\n"
     ]
    }
   ],
   "source": [
    "def concat_rel(g, rel=':mod'):    \n",
    "    forbidden_nodes_with_instances = {}\n",
    "    triples_filtered = []\n",
    "    for triple in g.triples:\n",
    "        if triple[1] == rel:\n",
    "            invoked = triple[0]\n",
    "            forbidden_node = triple[2]\n",
    "            instance = [concept for node, _, concept in g.instances() if node == forbidden_node]\n",
    "            if instance:\n",
    "                forbidden_nodes_with_instances[forbidden_node] = (instance[0], invoked)\n",
    "            else:\n",
    "                forbidden_nodes_with_instances[forbidden_node] = ('', invoked)\n",
    "        else:\n",
    "            triples_filtered.append(triple)       \n",
    "    for forbidden_node in forbidden_nodes_with_instances:\n",
    "        instance, invoked = forbidden_nodes_with_instances[forbidden_node]\n",
    "        for i in range(len(triples_filtered)):\n",
    "            n, r, c = triples_filtered[i]\n",
    "            if n == invoked and r == ':instance' and c != 'amr-unknown':\n",
    "                triples_filtered[i] = (n, r, f'{instance}_{c}')\n",
    "    triples_filtered = [t for t in triples_filtered if t[0] not in forbidden_nodes_with_instances]\n",
    "    epidata = {(n, r, c):g.epidata[(n, r, c.split('_')[-1])] for n, r, c in triples_filtered}\n",
    "    new_g = Graph(triples=triples_filtered, epidata=epidata)    \n",
    "    return new_g\n",
    "\n",
    "new_g = concat_rel(g, ':mod')\n",
    "print(penman.encode(new_g), '\\n')\n",
    "print(penman.encode(g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Similarity Measures: SBert models vs GloVe 6B.100d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.spatial.distance import cosine\n",
    "from sentence_transformers import SentenceTransformer, util"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cos(a,b):\n",
    "    #cosine similarity\n",
    "    dist = cosine(a,b)\n",
    "    sim = 1 - min(1,dist)\n",
    "    return sim\n",
    "\n",
    "def load_glove(fp):\n",
    "    dic={}\n",
    "    if not fp:\n",
    "        return dic\n",
    "    with open(fp,\"r\") as f:\n",
    "        for line in f:\n",
    "            ls = line.split()\n",
    "            word = ls[0]\n",
    "            vec = np.array([float(x) for x in ls[1:]])\n",
    "            dic[word] = vec\n",
    "    return dic\n",
    "\n",
    "def vecs_of_sents(m, sents):\n",
    "    s_vs = np.asarray([np.sum([m[word] for word in sent.split()], axis=0)/len(sent.split()) for sent in sents])\n",
    "    return s_vs\n",
    "\n",
    "def print_scores(s1, s2, cosine_scores):\n",
    "    max_s1 = max([len(s) for s in s1])\n",
    "    max_s2 = max([len(s) for s in s2])\n",
    "    \n",
    "    for i in range(cosine_scores.shape[0]):\n",
    "        for j in range(cosine_scores.shape[1]):\n",
    "            print(f'{s1[i]:{max_s1}}\\t{s2[j]:{max_s2}}\\tScore: {cosine_scores[i, j]:.4f}')\n",
    "            \n",
    "def sbert_sim(model, s1, s2):\n",
    "    embeddings1 = model.encode(s1, convert_to_tensor=True)\n",
    "    embeddings2 = model.encode(s2, convert_to_tensor=True)\n",
    "    \n",
    "    cosine_scores = util.pytorch_cos_sim(embeddings1, embeddings2)\n",
    "    \n",
    "    print_scores(s1, s2, cosine_scores)\n",
    "            \n",
    "def glove_sim(model, s1, s2):\n",
    "    embeddings1 = vecs_of_sents(model, s1)\n",
    "    embeddings2 = vecs_of_sents(model, s2)\n",
    "    \n",
    "    cosine_scores = util.pytorch_cos_sim(embeddings1, embeddings2)\n",
    "    \n",
    "    print_scores(s1, s2, cosine_scores)\n",
    "\n",
    "\n",
    "sbert1 = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')\n",
    "sbert2 = SentenceTransformer('paraphrase-distilroberta-base-v1')\n",
    "glove  = load_glove('amr_suite/vectors/glove.6B.100d.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"paraphrase-distilroberta-base-v1\":\n",
      "We eat french fries. \tWe eat chips. \tScore: 0.5431\n",
      "We eat french fries. \tWe eat chips .\tScore: 0.3898\n",
      "We eat french fries. \tWe eat chips  \tScore: 0.5067\n",
      "We eat french fries .\tWe eat chips. \tScore: 0.3834\n",
      "We eat french fries .\tWe eat chips .\tScore: 0.4945\n",
      "We eat french fries .\tWe eat chips  \tScore: 0.3968\n",
      "We eat french fries  \tWe eat chips. \tScore: 0.5006\n",
      "We eat french fries  \tWe eat chips .\tScore: 0.3953\n",
      "We eat french fries  \tWe eat chips  \tScore: 0.5043\n",
      "\n",
      "\n",
      "\"distilbert-base-nli-stsb-mean-tokens\":\n",
      "We eat french fries. \tWe eat chips. \tScore: 0.2958\n",
      "We eat french fries. \tWe eat chips .\tScore: 0.2958\n",
      "We eat french fries. \tWe eat chips  \tScore: 0.3143\n",
      "We eat french fries .\tWe eat chips. \tScore: 0.2958\n",
      "We eat french fries .\tWe eat chips .\tScore: 0.2958\n",
      "We eat french fries .\tWe eat chips  \tScore: 0.3143\n",
      "We eat french fries  \tWe eat chips. \tScore: 0.2940\n",
      "We eat french fries  \tWe eat chips .\tScore: 0.2940\n",
      "We eat french fries  \tWe eat chips  \tScore: 0.3407\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\nprint(\\'\\n\\')\\nprint(\\'\"GloVe average\":\\')\\nglove_sim(glove, s1, s2)\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#s1 = ['french fries']\n",
    "#s2 = ['chip', 'chips']\n",
    "s1 = ['We eat french fries.', 'We eat french fries .', 'We eat french fries']\n",
    "s2 = ['We eat chips.', 'We eat chips .', 'We eat chips']\n",
    "\n",
    "#s1_glove = ['how do i pump up water pressure in my shower ?']\n",
    "#s2_glove = ['how can i boost the water pressure in my shower ?']\n",
    "\n",
    "print('\"paraphrase-distilroberta-base-v1\":')\n",
    "sbert_sim(sbert2, s1, s2)\n",
    "print('\\n')\n",
    "print('\"distilbert-base-nli-stsb-mean-tokens\":')\n",
    "sbert_sim(sbert1, s1, s2)\n",
    "\n",
    "\"\"\"\n",
    "print('\\n')\n",
    "print('\"GloVe average\":')\n",
    "glove_sim(glove, s1, s2)\n",
    "\"\"\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}