Facebook Challenge.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/nachikethinge/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n",
      "No model was supplied, defaulted to sshleifer/distilbart-cnn-12-6 (https://huggingface.co/sshleifer/distilbart-cnn-12-6)\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "import nltk\n",
    "import statistics as s\n",
    "from newspaper import Article\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "import numpy as np\n",
    "from transformers import pipeline\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "nltk.download('punkt')\n",
    "\n",
    "\n",
    "model = SentenceTransformer('all-MiniLM-L6-v2')\n",
    "classifier = pipeline(\"sentiment-analysis\")\n",
    "summarizer = pipeline(\"summarization\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"dslim/bert-base-NER\")\n",
    "model_token = AutoModelForTokenClassification.from_pretrained(\"dslim/bert-base-NER\")\n",
    "token = pipeline(\"ner\", model=model_token, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collecting Articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'NEW You can now listen to Fox News articles!\\n\\nKatya Hill, sister of slain U.S. citizen Jimmy Hill, spoke to reporters Saturday about her brother\\'s death in Ukraine.\\n\\nJimmy Hill was reported dead Wednesday morning after a Russian bomb exploded in the city of Chernihiv. Jimmy Hill was in Ukraine at the time to help his partner get critical health care procedures when he was reportedly killed in a breadline. But Katya Hill updated the press on new discoveries in the circumstances of her brother\\'s death.\\n\\n\"Although initial reports indicated that he was killed in a breadline, the State Department informed us that his death was from a Russian bomb. Jimmy was in a civilian area of the city near the Chernihiv hospital. The State Department has not yet contacted the family to let us know the specifics,\" Hill told the press.\\n\\nAMERICAN KILLED BY RUSSIAN FORCES IN UKRAINE A \\'HELPER, A PEACEMAKER\\': SISTER SAYS\\n\\n\"My brother was in Chernihiv with the purpose of organizing medical treatment for his life\\'s partner, Irina Teslenko,\" Hill added. \"She has multiple sclerosis, and Jimmy has been by her side for years as the disease attacked her body. He never gave up hope that he would find treatment that would help stop the progression of MS.\"\\n\\nKatya Hill received word from the U.S. government on the updated circumstances of Jimmy\\'s death moments before addressing the media.\\n\\n\"I did receive – minutes before this news conference started – more specifics about how my brother died, the circumstances, the woman that I indicated my brother had befriended that lived near the hospital. Her name is Katrina,\" Hill explained. \"Katrina and Jimmy had gone out for a chance to locate busses that might be taking people out of town here on a safe corridor. When they got to the line, there were over a thousand people waiting to try to get on a bus to get out.\\n\\n\"So my brother decided to turn back, go back to the hospital to Irene\\'s side. And that\\'s when the bomb hit, Katrina shared on our messenger group.\"\\n\\nThe frontline city of Chernihiv has been under heavy bombardment over the last few days, with at least 53 civilians killed in a 24-hour period, the region\\'s governor said. This figure could not be independently confirmed.\\n\\nHill elaborated on the Russian tactics observed in the bombing, warning that the military will use lulls in action to draw civilians out into the open.\\n\\n\"The strategy that is being used in killing civilians – there\\'ll be intense bombing that will then stop and there\\'ll be no bombing for an hour or several hours,\" she explained \"People then feel safe, and they don\\'t have food, so stores and breadlines will open. They\\'ll go out and stand in line to try to get bread or food or other supplies, and then the bombing resumes, and that bombing targets those lines.\"\\n\\nHill told reporters that she had special thanks for U.S. senators Amy Klobachar and Tina Smith, Democrats from Minnesota., and Bob Casey, D-Pa., for their help connecting the family with the U.S. government for more information.\\n\\n\"The family wishes to thank Sen. Casey of Pennsylvania and Senator Smith and Klobuchar from Minnesota for reaching out to the State Department on the family\\'s behalf,\" she said.\\n\\nThis will be Katya Hill\\'s final public appearance for some time as she intends to spend the coming weeks grieving with her family in private, she said.\\n\\nCLICK HERE TO GET THE FOX NEWS APP\\n\\nIn the weeks before his death, Jimmy Hill described in a series of harrowing Facebook posts the dire conditions in the embattled city. He posted a picture of his partner under blankets in her hospital bed.\\n\\n\"Nobody in Chernihiv is safe. Indiscriminate bombing,\" he wrote March 2. \"Ukrainian forces hold city but are surrounded. It\\'s a siege here. Nobody in. Nobody out.\"\\n\\nAs the days passed, he documented the escalating shelling and artillery fire and a desire to flee the country, but he worried it was too dangerous and that his partner was too weak to make the journey.\\n\\n\"It\\'s a living nightmare, but we are alive,\" Hill wrote March 11.\\n\\nHe is at the least the second U.S. citizen to be killed by Russian forces in Ukraine since the invasion began. Journalist and filmmaker Brent Renaud was shot and killed by Russian troops while reporting on the refugee crisis.\\n\\nFox News\\'s Rebecca Rosenberg contributed to this report.'"
      ]
     },
     "execution_count": 262,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#About Ukraine (1)\n",
    "url1 = 'https://www.foxnews.com/us/sister-us-citizen-killed-ukraine-recounts-final-days' \n",
    "article1 = Article(url1)\n",
    "article1.download()\n",
    "article1.parse()\n",
    "article1.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "#About Ukraine (2)\n",
    "url2 = 'https://www.cbsnews.com/news/james-hill-us-citizen-killed-ukraine/' \n",
    "article2 = Article(url2)\n",
    "article2.download()\n",
    "article2.parse()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "#About March Madness\n",
    "url3 = 'https://longhornswire.usatoday.com/lists/texas-basketball-big-12-ncaa-tournament-power-conferences-first-round-results/'\n",
    "article3 = Article(url3)\n",
    "article3.download()\n",
    "article3.parse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Summary of the articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#article1.nlp()\n",
    "#summary1 = article1.summary\n",
    "#class1 = nlp(summary1)\n",
    "#sentences1 = str(list(class1.sents))\n",
    "#sentences1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary1 = list(summarizer(article1.text, max_length = 130, min_length = 30)[0].values())[0]\n",
    "summary1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \"My brother was a giver, and he just felt everybody in the world should love me\" \"My parents raised us to see the world would let this happen, and that\\'s how he lived\" \"A good person gone too soon\" has not decided whether the public will \"want to hear\" or \"don\\'t feel that the public is going to be going to the end of the year\" \"I\\'m not going to hear that we didn\\'t want the public to hear,\" said the public should \"have no way of hearing\" The public will be able to decide whether or not to hear the public\\'s view of the'"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary2 = list(summarizer(article2.text, max_length = 130, min_length = 30)[0].values())[0]\n",
    "summary2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' It has been just two days of the March Madness tournament and there have been multiple upsets already . The Big 12 did well, as all six teams that made the big dance were able to survive and advance . Here is how the Power 6 conferences fared .'"
      ]
     },
     "execution_count": 238,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary3 = list(summarizer(article3.text, max_length = 130, min_length = 30)[0].values())[0]\n",
    "summary3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(summary3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Similarities of Articles "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 81.91240429878235 %\n"
     ]
    }
   ],
   "source": [
    "#1st Article: Ukraine.. 2nd Article: Ukraine SIMILARITY\n",
    "emb1 = model.encode(summary1)\n",
    "emb2 = model.encode(summary2)\n",
    "cos_sim12 = util.cos_sim(emb1, emb2).numpy()[0][0]\n",
    "print(f\"\",cos_sim12 * 100, \"%\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1.989033818244934 %\n"
     ]
    }
   ],
   "source": [
    "#1st Article: Ukraine.. 3RD Article: MARCH MADNESS SIMILARITY\n",
    "emb1 = model.encode(summary1)\n",
    "emb3 = model.encode(summary3)\n",
    "cos_sim13 = util.cos_sim(emb1, emb3).numpy()[0][0]\n",
    "print(f\"\",cos_sim13 * 100, \"%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 3.5989344120025635 %\n"
     ]
    }
   ],
   "source": [
    "#2nd Article: Ukraine.. 3RD Article: MARCH MADNESS SIMILARITY\n",
    "emb2 = model.encode(summary2)\n",
    "emb3 = model.encode(summary3)\n",
    "cos_sim23 = util.cos_sim(emb2, emb3).numpy()[0][0]\n",
    "print(f\"\",cos_sim23 * 100, \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Positive and Negative score of an article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'NEGATIVE', 'score': 0.9888350367546082}]"
      ]
     },
     "execution_count": 237,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier(summary1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'NEGATIVE', 'score': 0.9913444519042969}]"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier(article2.summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'POSITIVE', 'score': 0.9779484272003174}]"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier(article3.summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Getting everything together\n",
    "def get_article(url):\n",
    "    article = Article(url)\n",
    "    article.download()\n",
    "    article.parse()\n",
    "    return article.text\n",
    "\n",
    "def user_summarize(article):\n",
    "    summary = list(summarizer(article)[0].values())[0]\n",
    "    return summary\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome!\n",
      "\n",
      "Enter the url of the article to evaluate: https://thehill.com/changing-america/enrichment/arts-culture/598832-james-webb-telescope-sends-back-new-image-of-star\n",
      "\n",
      "Getting the information...\n",
      "\n",
      "Here is your information: \n",
      "\n",
      "Summary:   NASA’s new James Webb telescope has reached another alignment milestone and sent back its clearest image of a star to date . The Webb Space Telescope launched from French Guiana on Christmas Day to replace the aging Hubble telescope . NASA stressed that the telescope will not be able to capture images as designed for several more months .\n",
      "\n",
      "The text is labeled as: POSITIVE with a score of 0.6808596253395081\n"
     ]
    }
   ],
   "source": [
    "print(\"Welcome!\\n\")\n",
    "user_url = input(\"Enter the url of the article to evaluate: \")\n",
    "print(\"\\nGetting the information...\\n\")\n",
    "article = get_article(str(user_url))\n",
    "summary = user_summarize(article)\n",
    "result = classifier(summary)\n",
    "print(\"Here is your information: \\n\")\n",
    "print(\"Summary: \", summary)\n",
    "print(\"\")\n",
    "print(f\"The text is labeled as: {list(result[0].values())[0]} with a score of {list(result[0].values())[1]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'James'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(token(summary)[].values())[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'entity': 'B-ORG',\n",
       "  'score': 0.9989575,\n",
       "  'index': 1,\n",
       "  'word': 'NASA',\n",
       "  'start': 1,\n",
       "  'end': 5},\n",
       " {'entity': 'B-PER',\n",
       "  'score': 0.9675067,\n",
       "  'index': 5,\n",
       "  'word': 'James',\n",
       "  'start': 12,\n",
       "  'end': 17},\n",
       " {'entity': 'I-PER',\n",
       "  'score': 0.76386505,\n",
       "  'index': 6,\n",
       "  'word': 'Webb',\n",
       "  'start': 18,\n",
       "  'end': 22},\n",
       " {'entity': 'B-ORG',\n",
       "  'score': 0.98394287,\n",
       "  'index': 27,\n",
       "  'word': 'Webb',\n",
       "  'start': 130,\n",
       "  'end': 134},\n",
       " {'entity': 'I-ORG',\n",
       "  'score': 0.9885953,\n",
       "  'index': 28,\n",
       "  'word': 'Space',\n",
       "  'start': 135,\n",
       "  'end': 140},\n",
       " {'entity': 'I-ORG',\n",
       "  'score': 0.98297256,\n",
       "  'index': 29,\n",
       "  'word': 'Tel',\n",
       "  'start': 141,\n",
       "  'end': 144},\n",
       " {'entity': 'I-ORG',\n",
       "  'score': 0.9603502,\n",
       "  'index': 30,\n",
       "  'word': '##es',\n",
       "  'start': 144,\n",
       "  'end': 146},\n",
       " {'entity': 'I-ORG',\n",
       "  'score': 0.90703136,\n",
       "  'index': 31,\n",
       "  'word': '##cope',\n",
       "  'start': 146,\n",
       "  'end': 150},\n",
       " {'entity': 'B-LOC',\n",
       "  'score': 0.9995146,\n",
       "  'index': 34,\n",
       "  'word': 'French',\n",
       "  'start': 165,\n",
       "  'end': 171},\n",
       " {'entity': 'I-LOC',\n",
       "  'score': 0.99926096,\n",
       "  'index': 35,\n",
       "  'word': 'Guiana',\n",
       "  'start': 172,\n",
       "  'end': 178},\n",
       " {'entity': 'I-MISC',\n",
       "  'score': 0.89985716,\n",
       "  'index': 38,\n",
       "  'word': 'Day',\n",
       "  'start': 192,\n",
       "  'end': 195},\n",
       " {'entity': 'B-ORG',\n",
       "  'score': 0.83526367,\n",
       "  'index': 43,\n",
       "  'word': 'Hu',\n",
       "  'start': 217,\n",
       "  'end': 219},\n",
       " {'entity': 'I-MISC',\n",
       "  'score': 0.51106286,\n",
       "  'index': 44,\n",
       "  'word': '##bble',\n",
       "  'start': 219,\n",
       "  'end': 223},\n",
       " {'entity': 'B-ORG',\n",
       "  'score': 0.99898976,\n",
       "  'index': 47,\n",
       "  'word': 'NASA',\n",
       "  'start': 236,\n",
       "  'end': 240}]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}