Document_Summarizer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.3"
    },
    "colab": {
      "name": "Document Summarizer",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nwn2Kead33hq",
        "colab_type": "text"
      },
      "source": [
        "# Text Summarization in Python\n",
        "\n",
        "## Motivation: \n",
        "The length of textual data is increasing and people have less time. Often the newspaper articles run into a long text of, say 1000 -1200 words. As wearable devices leap to prominence (Google Glass, Apple Watch, to name a few), content must adapt to the limited screen space available on these devices.\n",
        "The task of generating intelligent and accurate summaries for long pieces of text has become a popular research as well as industry problem."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndW1_hIJ33hs",
        "colab_type": "text"
      },
      "source": [
        "## Approach: \n",
        "Extractive text summarization is all about finding the more important sentences from a document as a summary of that document.\n",
        "Our approach is using the TextRank algorithm to find these 'important' sentences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNeDZ8Yq33hw",
        "colab_type": "text"
      },
      "source": [
        "## Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "Elrpx6FC33hz",
        "colab_type": "text"
      },
      "source": [
        "### 1. Importing important libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5PymxzGy33h1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 382
        },
        "outputId": "9368b883-ddfd-46c1-8baa-4dfba5ed6c90"
      },
      "source": [
        "# numpy library helps in working with arrays: array creation and manipulation\n",
        "# this implementation uses array for storing the matrices generated as 2-D arrays\n",
        "# PyPDF2 is a library used for reading the PDF files\n",
        "# docx2txt is the library used for reading Word documents \n",
        "# sys library has been used for printing the size of data structures used in the program\n",
        "!pip install PyPDF2\n",
        "!pip install docx2txt\n",
        "import numpy as np\n",
        "import PyPDF2\n",
        "import docx2txt\n",
        "import sys"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting PyPDF2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b4/01/68fcc0d43daf4c6bdbc6b33cc3f77bda531c86b174cac56ef0ffdb96faab/PyPDF2-1.26.0.tar.gz (77kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 1.4MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: PyPDF2\n",
            "  Building wheel for PyPDF2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for PyPDF2: filename=PyPDF2-1.26.0-cp36-none-any.whl size=61086 sha256=64d1401df220ccb2d178f42b9b69f21ed57f4dd14c5d0a58ae00122d11e36089\n",
            "  Stored in directory: /root/.cache/pip/wheels/53/84/19/35bc977c8bf5f0c23a8a011aa958acd4da4bbd7a229315c1b7\n",
            "Successfully built PyPDF2\n",
            "Installing collected packages: PyPDF2\n",
            "Successfully installed PyPDF2-1.26.0\n",
            "Collecting docx2txt\n",
            "  Downloading https://files.pythonhosted.org/packages/7d/7d/60ee3f2b16d9bfdfa72e8599470a2c1a5b759cb113c6fe1006be28359327/docx2txt-0.8.tar.gz\n",
            "Building wheels for collected packages: docx2txt\n",
            "  Building wheel for docx2txt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for docx2txt: filename=docx2txt-0.8-cp36-none-any.whl size=3963 sha256=ccfe5b594996d55331181340da8b44f0160d8e517dd110f143f1a28e0a1b0b0c\n",
            "  Stored in directory: /root/.cache/pip/wheels/b2/1f/26/a051209bbb77fc6bcfae2bb7e01fa0ff941b82292ab084d596\n",
            "Successfully built docx2txt\n",
            "Installing collected packages: docx2txt\n",
            "Successfully installed docx2txt-0.8\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eKh31zh133iE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# matplotlib is a library that is used to visualize the data by drawing graphs of matrix inputs\n",
        "# we will use it for drawing the matrices generated later in the program \n",
        "# %matplotlib inline is a command used to show the graphs in the jupyter notebook\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nxB-D63V33iN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# networkx library helps in working with graphs ...\n",
        "# and later performing the PageRank algorithm ...\n",
        "# which is the crux of this implementation to find ...\n",
        "# the importance of each sentence using their 'rank' as a metric ...\n",
        "# rank, the output of the method textrank, is a measure of importance of sentences\n",
        "# this library has been used in the cell no. ()\n",
        "\n",
        "import networkx as nx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x50weS8D33iW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# the PunktSentenceTokenizer library is being imported from the file punkt.py contained in package nltk.tokenize \n",
        "# this is used to tokenize the document into sentences\n",
        "\n",
        "# Tokenization: Tokenization is the process of demarcating and possibly classifying.. \n",
        "# sections of a string of input characters. \n",
        "# The resulting tokens are then passed on to some other form of processing. \n",
        "\n",
        "from nltk.tokenize.punkt import PunktSentenceTokenizer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f3TZPUPq33if",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# TfidfTransformer and CountVectorizer libraries are being imported\n",
        "\n",
        "# CountVectorizer: In this implementation, a CountVectorizer object is being created that ..\n",
        "# will be used for creating the document-term matrix\n",
        "\n",
        "# tFidTransformer: In this implementation,TfidfTransformer is used for executing the method fit_transform()... \n",
        "# which provides the output as a document-term matrix normalized (value 0-1) according to the TF-IDF\n",
        "# TF(Term Frequency): the no. of times a term(a word here) appears in the current document(single sentence here)\n",
        "# IDF(Inverse Document Frequency): the no. of times a term(a word here) appears in the entire corpus\n",
        "# Corpus: set of all sentences\n",
        "\n",
        "\n",
        "from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4mR_bmr633io",
        "colab_type": "text"
      },
      "source": [
        "### 2.  Function to read the document from user\n",
        "Supported formats: .txt, .pdf \n",
        "\n",
        "Input: Takes the name of the file as input. \n",
        "\n",
        "Output: Returns a string output containing the contents of the file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YeBsl0Xf33io",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        },
        "outputId": "7851598c-9c44-410a-c331-fdd722f15a6a"
      },
      "source": [
        "# we are going to show an example of how the method is working\n",
        "# first let's take the document as an input\n",
        "def readDoc():\n",
        "    name = input('Please input a file name: ') \n",
        "    print('You have asked for the document {}'.format(name))\n",
        "\n",
        "    # now read the type of document\n",
        "    if name.lower().endswith('.txt'):\n",
        "        choice = 1\n",
        "    elif name.lower().endswith('.pdf'):\n",
        "        choice = 2\n",
        "    else:\n",
        "        choice = 3\n",
        "        # print(name)\n",
        "    print(choice)\n",
        "    # Case 1: if it is a .txt file\n",
        "        \n",
        "    if choice == 1:\n",
        "        f = open(name, 'r')\n",
        "        document = f.read()\n",
        "        f.close()\n",
        "            \n",
        "    # Case 2: if it is a .pdf file\n",
        "    elif choice == 2:\n",
        "        pdfFileObj = open(name, 'rb')\n",
        "        pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
        "        pageObj = pdfReader.getPage(0)\n",
        "        document = pageObj.extractText()\n",
        "        pdfFileObj.close()\n",
        "    \n",
        "    # Case 3: none of the format\n",
        "    else:\n",
        "        print('Failed to load a valid file')\n",
        "        print('Returning an empty string')\n",
        "        document = ''\n",
        "    \n",
        "    print(type(document))\n",
        "    return document\n",
        "readDoc()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Please input a file name: b.txt\n",
            "You have asked for the document b.txt\n",
            "1\n",
            "<class 'str'>\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/317420253\\nA survey on extractive text summarization\\nConference Paper · January 2017\\nDOI: 10.1109/ICCCSP.2017.7944061\\nCITATIONS\\n26\\nREADS\\n3,140\\n2 authors:\\nSome of the authors of this publication are also working on these related projects:\\nhuman action recognition View project\\nHuman Activity Recognition View project\\nN. Moratanch\\nAnna University, Chennai\\n4 PUBLICATIONS 51 CITATIONS\\nSEE PROFILE\\nChitrakala Gopalan\\nAnna University, Chennai\\n115 PUBLICATIONS 211 CITATIONS\\nSEE PROFILE\\nAll content following this page was uploaded by N. Moratanch on 07 November 2017.\\nThe user has requested enhancement of the downloaded file.\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\nA Survey on Extractive Text Summarization\\nN.Moratanch* ,S.Chitrakala t\\n*Research Scholar, t Associate Professor\\n*tDepartment of CSE\\nAnna University,CEG,Chennai\\n* tancyanbil@ gmail.com, t aU.chitras@ gmail.com\\nAbstract-Text Summarization is the process of obtaining\\nsalient information from an authentic text document. In this\\ntechnique, the extracted information is achieved as a summarized\\nreport and conferred as a concise summary to the user. It is very\\ncrucial for humans to understand and to describe the content\\nof the text. Text Summarization techniques are classified into\\nabstractive and extractive summarization. The extractive summarization\\ntechnique focuses on choosing how paragraphs,important\\nsentences, etc produces the original documents in precise form.\\nThe implication of sentences is determined based on linguistic\\nand statistical features. In this work, a comprehensive review\\nof extractive text summarization process methods has been\\nascertained. In this paper, the various techniques, populous\\nbenchmarking datasets and challenges of extractive summarization\\nhave been reviewed. This paper interprets extractive text\\nsummarization methods with a less redundant summary, highly\\nadhesive, coherent and depth information.\\nIndex Terms-Text Summarization, Unsupervised Learning,\\nSupervised Learning, Sentence Fusion, Extraction Scheme, Sentence\\nRevision, Extractive Summary\\nI. INTRODUCTION\\nIn a recent advance, the significance of text summarization\\n[1] accomplishes more attention due to data inundation on\\nthe web. Hence this information overwhelms yields in the\\nbig requirement for more reliable and capable progressive text\\nsummarizers. Text Summarization gains its importance due to\\nits various types of applications just like the summaries of\\nbooks, digest- (summary of stories), the stock market, news,\\nHighlights- (meeting, event, sport), Abstract of scientific papers,\\nnewspaper articles, magazine etc. Due to its tremendous\\ngrowth, many finest universities like Faculty of Informatics\\n- Masaryk University, Czech Republic, Concordia University,\\nMontreal, Canada- Semantic Software Lab, IHR Nexus Lab\\nat Arizona State University, Arizona, USA and finally Lab of\\nTopic Maps-Leipzig University, Germany has been persistently\\nworking on its rapid enhancements.\\nText summarization has grown into a crucial and appropriate\\nengine for supporting and illustrate text content in the latest\\nspeedy emergent information age. It\\'s far very complex for\\nhumans to physically summarize oversized documents of\\ntext. There is a wealth of textual content available on the\\ninternet. But, usually, the internet contribute more data than\\nis desired. Therefore, a twin problem is detected: Seeking for\\nappropriate documents through an awe-inspiring number of\\nreports offered, and fascinating a high volume of important\\ninformation. The objective of automatic text summarization is\\nto condense the origin text into a precise version preserves\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\nits report content and global denotation. The main advantage\\nof a text summarization is reading time of the user can be\\nreduced. A marvelous text summary system should reproduce\\nthe assorted theme of the document even as keeping repetition\\nto a minimum. Text Summarization methods are publicly\\nrestricted into abstractive and extractive summarization.\\nAn extractive summarization technique consists of selecting\\nvital sentences, paragraphs, etc, from the original manuscript\\nand concatenating them into a shorter form. The significance\\nof sentences is strongly based on statistical and linguistic\\nfeatures of sentences. This paper generally summarizes the\\nextensive methodologies fitted, issues launch, exploration and\\nfuture directions in text summarization. This paper [1] is\\norganized as follows. Section 2 depicts about the features for\\nextractive text summarization, Section 3 describes extractive\\ntext summarization methods, Section 4 illustrate inferences\\nmade, Section 5 represent challenges and future research\\ndirections, Section 6 detail about evaluation metrics and the\\nfinal sketch is the conclusion.\\nII. FEATURES FOR EXTRACTIVE TEXT\\nSUMMARIZATION\\nEarlier techniques involve assigning a score to sentences\\nbased on a countenance that are predefined based on the\\nmethodology applied. Both word level and sentence level\\nfeatures are employed in text summarization literature. Certain\\nfeatures discussed are [2] [3] [4]used to exclusive sentences\\nto be included in the summary are:\\n1. WORD LEVEL FEATURES\\n1.1 Content Word feature\\nKeywords are essential in identifying the importance of the\\nsentence. The sentence that consists of main keywords is most\\nlikely included in the final summary. The content (keyword)\\nwords are words that are nouns, verbs, adjectives and adverbs\\nthat are commonly determined based on tf x idf measure.\\n1.2 Title Word feature\\nThe sentences in the original document which consists of\\nwords mentioned in the title have greater chances to contribute\\nto the final summary since they serve as indicators of the theme\\nof the document.\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\n1.3 Cue phrase feature\\nCue phrases are words and phrases that indicate the structure\\nof the document flow and it is used as a feature in\\nsentence selection. The sentence that contains cue phrases\\n(e.g. \"denouement\", \"because\", \"this information\", \"summary\",\\n\"develop\", \"desire\" etc.) are mostly to be included in the final\\nsummary.\\n1.4 Biased word feature\\nThe sentences that consist of biased words are more likely\\nimportant. The biased words are a list of the predefined set\\nof words that may be domain specific. They are relatively\\nimportant words that describe the theme of the document.\\n1.5 Upper case word feature\\nThe words which are in uppercase such as \"UNICEF\" are\\nconsidered to be important words and those sentences that\\nconsist of these words are termed important in the context of\\nsentence selection for the final summary.\\n2. SENTENCE LEVEL FEATURES\\n2.1 Sentence location feature\\nThe sentences that occur in the beginning and the conclusion\\npart of the document are most likely important since most\\ndocuments are hierarchically structured with important information\\nin the beginning and the end of the paragraphs.\\n2.2 Sentence length feature\\nThe sentence length plays an important role in identifying\\nkey sentences. Shorter texts do not convey essential information\\nand very long sentences also need not be included in the\\nsummary. The normalized length of the sentence is calculated\\nas the ratio between a number of words in the sentence to the\\nnumber of words in the longest sentence in the document.\\n2.3 Paragraph location feature\\nSimilar to sentence location, paragraph location also plays\\na crucial role in selecting key sentences. A Higher score is\\nassigned to the paragraph in the peripheral sections (beginning\\nand end paragraphs of the document).\\n2.4 Sentence-to-Sentence Cohesion\\nThe cohesion between sentences for every sentence(s), the\\nsimilarity between s and alternative sentences are calculated\\nwhich are summed up and coarse value of the aspect is\\nobtained for s. The feature values are normalized between\\n[0, 1] where value closer to 1.0 indicates a higher degree of\\ncohesion between sentences.\\nIII. EXTRACTIVE TEXT SUMMARIZATION\\nMETHODS\\nExtractive Text Summarization methods can be broadly\\nclassified as Unsupervised Learning and Supervised learning\\nmethods. Recent works rely on Unsupervised Learning methods\\nfor text summarization.\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\nFig. 1. Overview of Extractive Summarization\\nA. UNSUPERVISED LEARNING METHODS\\nIn this section, unsupervised techniques for sentence extraction\\ntask is discussed. The unsupervised approaches do not\\nneed human summaries (user input) in deciding the important\\nfeatures of the document, it requires the most sophisticated\\nalgorithm to provide compensation for the lack of human\\nknowledge. Unsupervised summaries provide a higher level\\nof automation compared to supervised model and are more\\nsuitable for processing Big Data. Unsupervised learning models\\nhave proved successful in text summarization task.\\nFig. 2. Overview of Unsupervised Learning Methods\\n1. Graph based approach\\nGraph-based models are extensively used in document summarization\\nsince graphs can efficiently represent the document\\nstructure. Extractive text summarization using external knowledge\\nfrom Wikipedia incorporating bipartite graph framework\\n[4 ]has been used. They have proposed an iterative ranking\\nalgorithm (variation of HITS algorithm [5]) which is efficient\\nin selecting important sentences and also ensures coherency\\nin the final summary. The uniqueness of this paper is that\\nit combines both graph based and concept based approach\\ntowards summarization task. Another graph based approach\\nLexRank [6], where the salience of the sentence is determined\\nby the concept of Eigen vector centrality. The sentences in the\\ndocument are represented as a graph and the edges between\\nthe sentences represents weighted cosine similarity values. The\\nsentences are clustered into groups based on their similarity\\nmeasures and then the sentences are ranked based on their\\nLexRank scores similar to PageRank algorithm [7]except that\\nthe similarity graph is undirected in LexRank method. The\\nmethod outperforms earlier versions of lead and centroid based\\napproaches. The performance of the system is evaluated with\\nDUC dataset [8].\\n2. Fuzzy logic based approach\\nThe fuzzy logic approach mainly contains four components:\\ndefuzzifier, fuzzifier, fuzzy knowledge base and inference engine.\\nThe textual characteristics input of Fuzzy logic approach\\nare sentenced length, sentence similarity etc which is later\\ngiven to the fuzzy system [9] [10].\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\nTABLE I\\nSUPERVISED AND UNSUPERVISED LEARNING METHODS FOR TEXT SUMMARIZATION\\nCategories Methodology Concept\\nSUPERVISED Machine Learning ap- Summarization task\\nLEARNING proach Bayes rule modelled as classification\\nAPPROACHES problem\\nTrainable summarization -\\nSUPERVISED neural network is trained,\\nLEARNING Artificial Neural Net- pruned and generalized to\\nAPPROACHES work filter sentences and classify\\nthem as \"summary\" or\\n\"non-summary sentence\"\\nSUPERVISED Statistical modelling ap-\\nLEARNING Conditional Random proach which uses CRF as\\nAPPROACHES Fields (CRF) a sequence labelling prob- lem\\nUNSUPERVISED Graph based Construction of graph to\\nLEARNING Approach capture relationship be-\\nAPPROACHES tween sentences\\nImportance of sentences\\nUNSUPERVISED Concept oriented ap- calculated based on\\nLEARNING the concepts retrieved\\nAPPROACHES proach from external knowledge\\nbase(wikipedia, HowNet)\\nUNSUPERVISED Fuzzy Logic based ap- Summarization based on\\nLEARNING fuzzy rule using various\\nAPPROACHES proach sets of features\\nFig. 3. Example of Sentence concept bipartite graph proposed in [4]\\nLadda Suanmali et al [11] proposed fuzzy logic approach\\nis used for automatic text summarization which is the initial\\nstep , the text document is pre-processed followed by feature\\nextraction(Title features, Sentence length, Sentence position,\\nSentence-sentence similarity, term weight, Proper noun and\\nNumerical data. The summary is generated by ordering the\\nranked sentences in the order they occur in the original\\ndocument to maintain coherency. The proposed method shows\\nimprovement in the quality of summarization but issues such\\nas dangling anaphora are not handled.\\n3. Concept-based approach\\nIn concept-based approach, the concepts are extracted from\\na piece of text from external knowledge base such HowNet\\n[l2]and Wikipedia [4]. In the methodology proposed [12], the\\nimportance of sentences is calculated based on the concepts\\nretrieved from HowNet instead of words. A conceptual vector\\nmodel is built to obtain a rough summarization and similarity\\nmeasures are calculated between the sentences to reduce\\nredundancy in the final summary. A good summarizer focuses\\non higher coverage and lower redundancy. Ramanathan\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\nAdvantages Limitations\\nLarge set of training data im- Human interruption required for proves the sentence selection for generating manual summaries summary\\nThe network can be trained ac- I)Neural Network is slow in\\ncording to the style of human training phase and also in apreader.\\nThe set of features can be plication phase. 2) It is difficult\\naltered to reflect user\\'s need and to determine how the net makes\\nrequirements decision. 3) Requires human interruption\\nfor training data\\nIdentifies correct features and 1) focuses on domain specific\\nprovides better representation of which requires an external do- sentences and groups terms ap- main specific corpus for training\\npropriately into its segments step. 2) Limitation is that linguistic\\nfeatures are not considered\\n1) Captures redundant inforrna- Doesn\\'t focus on issues such as\\ntion 2)Improves coherency dangling anaphora problem\\nincorporation of similarity mea- Dangling anaphora and verb refsures\\nto reduce redundancy erents not considered\\nimproved quality in summary by membership functions and work\\nmaintaining coherency of the fuzzy logic system\\nFig. 4. Overall architecture of text summarization based on fuzzy logic\\napproach proposed in [10]\\nFig. 5. Example of concepts retrieved for sentence from Wikipedia as\\nproposed in [4]\\net al [4] proposed a Wikipedia-based summarization which\\nutilizes graph structure to produce summaries. The method\\nuses Wikipedia to obtain concept for each sentence and\\nbuilds a sentence-concept bipartite graph as already mentioned\\nin Graph-based summarization. The basic steps in concept\\nbased summarization are: i) Retrieve concepts of a text from\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\nexternal knowledge base(HowNet, WordNet, Wikipedia) ii)\\nBuild a conceptual vector or graph model to depict relationship\\nbetween concept and sentences iii) Apply ranking algorithm to\\nscore sentences iv) Generate summaries based on the ranking\\nscores of sentences\\n4. Latent Semantic Analysis Method(LSA)\\nLatent Semantic Analysis(LSA) [13] [14] is a method which\\nexcerpt hidden semantic structures of sentences and words\\nthat are popularly used in text summarization task. It is an\\nunsupervised learning approach that does not demand any sort\\nof external or training knowledge. LSA captures the text of the\\ninput document and excerpt information such as words that\\nfrequently occur together and words that are commonly seen in\\ndifferent sentences. A high number of common words amongst\\nthe sentences illustrate that the sentences are semantically\\nrelated. Singular Value Decomposition(SVD) [13], is a method\\nused to find out the interrelations between words and sentences\\nwhich also has the competence of noise reduction that helps\\nto improve accuracy. SVD, [15] when enforced to document\\nword matrices, can group documents that are semantically\\nassociated to one other despite them sharing no common\\nwords. The set of words that ensue in connected text is\\nalso connected within the same peculiar dimensional space.\\nLSA technique is applied to excerpt the subject-related words\\nand important content conveying sentences from report. The\\nadvantage of adopting LSA vectors for summarization over\\nword vectors is that conceptual relations as represented in the\\nhuman brain are naturally captured in the LSA. Choice of\\nthe representative sentence from every scale of the capacity\\nensures relevancy of sentence to the document and ensures\\nnon-redundancy. LS works with text data and the principal\\nambit due to the collection of topics can be located.\\nConsidering an example to depict LSA representatieach\\notheron, Example 1: Consider following 3 sentences given to\\nLSA based system. dO: \\'The man was walked the dog. dl:\\n\\'The man took the dog to the park in the evening. d2: \\'The\\ndog went to the park in the evening. From Fig6 [13] it is to\\nbe noted in order that dl is associated to d2 than dO and the\\nconversation \\'walked\\' is linked to the talk \\'man\\' but it is not\\nsignificant to the word \\'park\\'. These kind of interpretations\\ncan be built by using input data and LSA, beyond need for\\nany extraneous awareness.\\nB. SUPERVISED LEARNING METHODS\\nSupervised extractive summarizationrelated techniques are\\nbased on a classification approach at sentence level where\\nthe system learns by examples to classify between summary\\nand non-summary sentences. The major drawback with the\\nsupervised approach is that it requires known manually created\\nsummaries by a human to label the sentences in the original\\ntraining document enclosed with \"summary sentence\" or \"nonsummary\\nsentence\" and it also requires more labeled training\\ndata for classification.\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\nFig. 6. Representation of LSA for Example [13]\\nFig. 7. Overview of Supervised Learning Methods\\n1. Machine Learning Approach based on Bayes rule\\nA set of training documents along with its extractive summaries\\nis fed as input to the training stage. The machine\\nlearning approach views classification problem in text summarization.\\nThe sentences are restricted as a non-summary\\nand summary sentence based on the feature possessed by the\\nsentence. The probability of classification are learned from the\\ntraining data by the following Bayes rule [16]: where s represents\\nthe set of sentences in the document and fi represents\\nthe features used in classification stage and S represents the\\nset of sentences in the summary. P (s E< SII1,h,h, .... In)\\nrepresents the probability of the sentences to be included in\\nthe summary based on the given features possessed by the\\nsentence.\\n2. Neural Network based approach\\nFig. 8. Neural network after training (a) and after pruning (b) [17]\\nIn the approach proposed in [18], RankNet algorautomaticallyithm\\nusing neural nets to identify the important sentences\\nin the document. It uses a two-layer neural network with\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\nback propagation trained using RankNet algorithm. The first\\nstep involves labeling the training data using a machinelearning\\napproach and then extract features of the sentences\\nin both test set and train sets which is then inputted to the\\nneural network system to rank the sentences in the document.\\nAnother approach [17]uses a three layered feed-forward neural\\nnetwork which learns in the training stage the characteristics\\nof summary and non-summary sentences. The major phase\\nis the feature fusion phase where the relationship between\\nthe features are identified through two stages 1) eliminating\\ninfrequent features 2) collapsing frequent features after\\nwhich sentence ranking is done to identify the important\\nsummary sentences.Neural Network [17]after feature fusion\\nis depicted in Fig 8. Dharmendra Hingu, Deep Shah and\\nSandeep S.Udmale proposed an extractive approach [19]for\\nsummarizing the Wikipedia articles by identifying the text\\nfeatures and scoring the sentences by incorporating neural\\nnetwork model [5]. This system gets the Wikipedia articles\\nas input followed by tokenisation and stemming. The preprocessed\\npassage is sent to the feature extraction steps, which\\nis based on multiple features of sentences and words. The\\nscores obtained after the feature extraction are fed to the\\nneural network, which produces a single value as output score,\\nsignifying the importance of the sentences. Usage of the words\\nand sentences is not considered while assigning the weights\\nwhich results in less accuracy.\\n3. Conditional Random Fields\\nConditional Random Fields are a statistical modeling approach\\nthat focuses on machine leaning to provide a structured\\nprediction. The proposed system overcomes the issues faced\\nby non-negative matrix Factorization (NMF) methods by incorporating\\nconditional random fields (CRF) to identify and\\nextract correct features to determine the important sentence\\nof the given text. The main advantage of the method is that\\nit is able to identify correct features and provides a better\\nrepresentation of sentences and groups terms appropriately\\ninto its segments. The major drawback of the method is that it\\nfocuses on domain-specific which requires an external domain\\nspecific corpus for training step, thus this method cannot be\\napplied generically to any document without building a domain\\ncorpus which is a time-consuming task. The approach specified\\nin [20]uses CRF as a sequence labelling problem and also\\ncaptures interaction between sentences through the features\\nextracted for each sentence and it also incorporates complex\\nfeatures such as LSA_scores [21] and lilTS_score [22] but\\nthe limitation is that linguistic features are not considered.\\nIV. INFERENCES MADE\\n• Abounding variations of the extractive path [15] have\\nbeen focused in the prior ten years. However, it is difficult\\nto say how analytical improvement (sentence or text\\nlevel) devote to work.\\n• Beyond NLP, the achieved summary might endure a lack\\nof semantics and cohesion. In texts consist of numerous\\ntopics, the provoked summary may not be fair. Conclusive\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\nproper weights for respective features is vital to the\\nquality of concluding summary depends on it.\\n• Feature weights should be given more importance because\\nit plays a major role in choosing key sentences.\\nIn text Summarization, the most challenging task is\\nto summarize the contented from a number of semistructured\\nsources and textual, which includes web pages\\nand databases, in the proper way (size, format, time,\\nlanguage,) for an explicit user.\\n• Text summary software should crop effective summary\\nwithin a fewer amount of redundancy and time. Summarization\\nappraise using extrinsic or intrinsic part.\\n• Intrinsic parts pursuit to measure summary nature adopting\\nhuman evaluation whereas, extrinsic parts measure\\nthe same over a effort-based work measure being the\\ninformation rehabilitation-oriented task.\\nV. EVALUATION METRICS\\nNumerous benchmarking datasets [1] are used for experimental\\nevaluation of extractive summarization. Document Understanding\\nConferences (DUC) is the most common benchmarking\\ndatasets used for text summarization. There are a\\nnumber of datasets like TIPSTER, TREC , TAC , DUC, CNN.\\nIt contains documents along with their summaries that are\\ncreated automatically, manually and submitted summaries[20].\\nFrom papers surveyed within the previous sections et al in\\nliterature, it\\'s been found that agreement between human\\nsummarizers is sort of low, each for evaluating and generating\\nsummaries quite the shape of the outline, it is tough to judge\\nthe outline content.\\ni) Human Evaluation\\nHuman judgment usually has wide variance on what\\'s\\nthought-about a \"good\" outline, which implies that creating\\nthe analysis method automatic is especially tough. Manual\\nanalysis is used, however, this can be each time and laborintensive\\nbecause it needs humans to browse not solely the\\nsummaries however conjointly the supply documents. Other\\nissues are those regarding coherence and coverage.\\nii) Rouge\\nFormally, ROUGE-N is an n-gram recall between a candidate\\nsummary and a set of reference summaries. ROUGE-N\\nis computed as follows:\\nROUGE-N = LSEreference_summaries LN-grams Countmatch{N-gram)\\nL S Ereference_summaries LN-grams Count{N -gram)\\nwhere, n stands for the length of the n-gram Countmatch(Ngram)\\nis the maximum number of n-grams co-occurring in\\na candidate summary and a set of reference summaries.\\nCount(N-gram) is the number of N-grams in the set of\\nreference summaries.\\n... ) R 11 R _ ISref n Scandl\\nm eca - ISrefl\\nwhere Sref n Scand indicates the number of sentences that\\nco-occur in both reference and candidate summaries.\\nIEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\\nt.V ) P rect.s.w n (P) P = ISreIf n ScIa ndl\\nScand\\n)\\n_ 2(Precision)(Recall)\\nv F -measure F - PreCI.S l.O n + Rec all\\nvi) Compression Ratio Cr = Slen . Iien\\nwhere, Slen and Iien are the length of summary and source\\ntext respectively.\\nVI. CHALLENGES AND FUTURE RESEARCH\\nDIRECTIONS\\nEvaluating summaries (either automatically or manually) is\\na difficult task. The main problem in evaluation comes from\\nthe impossibility of building a standard against which the\\nresults of the systems that have to be compared. Further, it\\nis very hard to find out what a correct summary is because\\nthere is a chance of the system to generate a better summary\\nthat is different from any human summary which is used as an\\napproximation to correct output. Content choice [23] is not a\\nsettled problem. People are completely different and subjective\\nauthors would possibly select completely different sentences.\\nTwo distinct sentences expressed in disparate words will\\nspecific a similar can explicit the same meaning also known\\nas paraphrasing. There exists an approach to automatically\\nevaluate summaries using paraphrases (paraEval). Most text\\nsummarization systems perform extractive summarization approach\\n(selecting and photocopying extensive sentences from\\nthe professional documents). Though humans can cut and paste\\nrelevant data from a text, most of the times they rephrase sentences\\nwhenever necessary, or they may join different related\\ndata into one sentence. The low inter-annotator agreement\\nfigures observed during manual evaluations suggest that the\\nfuture of this research area massively depends on the capacity\\nto find efficient ways of automatically evaluating the systems.\\nVII. CONCLUSION\\nThis review has shown assorted mechanism of extractive\\ntext summarization process. Extractive summarization process\\nis highly coherent, less redundant and cohesive (summary and\\ninformation rich). The aim is to give a comprehensive review\\nand comparison of distinctive approaches and techniques of\\nextractive text summarization process. Although research on\\nsummarization started way long back, there is still a long way\\nto go. Over the time, focused has drifted from summarizing\\nscientific articles to advertisements, blogs, electronic mail\\nmessages and news articles. Simple eradication of sentences\\nhas composed satisfactory results in massive applications.\\nSome trends in automatic evaluation of summary system have\\nbeen focused. However, the work has not focused the different\\nchallenges of extractive text summarization process to its full\\nintensity in premises of time and space complication.\\nREFERENCES\\n[1] N. Moratanch and S. Chitrakala, \"A survey on abstractive text summarization,\"\\nin Circuit, Power and Computing Technologies (ICCPCT),\\n2016 International Conference on. IEEE, 2016, pp. 1-7.\\n978-1-5090-3716-2/17/$31.00 ©2017 IEEE\\n[2] F. Kiyomarsi and F. R. Esfahani, \"Optimizing persian text summarization\\nbased on fuzzy logic approach,\" in 2011 International Conference on\\nIntelligent Building and Management, 2011.\\n[3] F. Chen, K. Han, and G. Chen, \"An approach to sentence-selectionbased\\ntext summarization,\" in TENCON\\'02. Proceedings. 2002 IEEE\\nRegion 10 Conference on Computers, Communications, Control and\\nPower Engineering, vol. 1. IEEE, 2002, pp. 489-493.\\n[4] Y. Sankarasubramaniam, K. Ramanathan, and S. Ghosh, \"Text summarization\\nusing wikipedia,\" Information Processing & Management,\\nvol. 50, no. 3, pp. 443-461, 2014.\\n[5] J. M. Kleinberg, \"Authoritative sources in a hyperlinked environment,\"\\nJournal of the ACM (JACM), vol. 46, no. 5, pp. 604--632, 1999.\\n[6] G. Erkan and D. R. Radev, \"Lexrank: Graph-based lexical centrality\\nas salience in text summarization,\" Journal of Artificial Intelligence\\nResearch, pp. 457-479, 2004.\\n[7] S. M. R .. W. T. L., Brin, \"The pagerank citation ranking: Bringing\\norder to the web,\" Technical report, Stanford University, Stanford, CA.,\\nTech. Rep., (1998).\\n[8] (2004) Document understanding conferences dataset. Online. [Online].\\nAvailable: http://duc.nist.gov/data.htrnl.\\n[9] F. Kyoomarsi, H. Khosravi, E. Eslami, P. K. Dehkordy, and A. Tajoddin,\\n\"Optimizing text summarization based on fuzzy logic,\" in Seventh\\nIEEElACIS International Conference on Computer and Information\\nScience. IEEE, 2008, pp. 347-352.\\n[10] L. SUanmali, M. S. Binwahlan, and N. Salim, \"Sentence features fusion\\nfor text summarization using fuzzy logic,\" in Hybrid Intelligent Systems,\\n2009. HIS\\'09. Ninth International Conference on, vol. 1. IEEE, 2009,\\npp. 142-146.\\n[11] L. Suanmali, N. Salim, and M. S. Binwahlan, \"Fuzzy logic based method\\nfor improving text summarization,\" arXiv pre print arXiv:0906.4690,\\n2009.\\n[12] X. W. Meng Wang and C. Xu, \"An approach to concept oriented text\\nsummarization,\" in in Proceedings of ISClTS05, IEEE international\\nconference, China,1290-1293\" 2005.\\n[13] M. G. Ozsoy, F. N. Alpaslan, and 1. Cicekli, \"Text summarization using\\nlatent semantic analysis,\" Journal of Information Science, vol. 37, no. 4,\\npp. 405-417, 2011.\\n[14] 1. Mashechkin, M. Petrovskiy, D. Popov, and D. V. Tsarev, \"Automatic\\ntext summarization using latent semantic analysis,\" Programming and\\nComputer Software, vol. 37, no. 6, pp. 299-305, 2011.\\n[15] V. Gupta and G. S. Lehal, \"A survey of text summarization extractive\\ntechniques,\" Journal of emerging technologies in web intelligence, vol. 2,\\nno. 3, pp. 258-268, 2010.\\n[16] J. L. Neto, A. A. Freitas, and C. A. Kaestner, \"Automatic text summarization\\nusing a machine learning approach,\" in Advances in Artificial\\nIntelligence. Springer, 2002, pp. 205-215.\\n[17] K. Kaikhah, \"Automatic text summarization with neural networks,\"\\n2004.\\n[18] K. M. Svore, L. Vanderwende, and C. J. Burges, \"Enhancing singledocument\\nsummarization by combining ranknet and third-party sources.\"\\nin EMNLP-CoNLL, 2007, pp. 448-457.\\n[19] D. Hingu, D. Shah, and S. S. Udmale, \"Automatic text summarization\\nof wikipedia articles,\" in Communication, Information & Computing\\nTechnology (ICCICT), 2015 International Conference on. IEEE,2015,\\npp.I-4.\\n[20] D. Shen, J.-T. Sun, H. Li, Q. Yang, and Z. Chen, \"Document summarization\\nusing conditional random fields.\" in IJCAI, vol. 7, 2007, pp.\\n2862-2867.\\n[21] Y. Gong and X. Liu, \"Generic text summarization using relevance\\nmeasure and latent semantic analysis,\" in Proceedings of the 24th annual\\ninternational ACM SIGIR conference on Research and development in\\ninformation retrieval. ACM, 2001, pp. 19-25.\\n[22] R. Mihalcea, \"Language independent extractive summarization,\" in\\nProceedings of the ACL 2005 on Interactive poster and demonstration\\nsessions. Association for Computational Linguistics, 2005, pp. 49-52.\\n[23] N. Lalithamani, R. Sukumaran, K. Alagamrnai, K. K. Sowmya, V. Divyalakshmi,\\nand S. Shanmugapriya, \\'\\'A mixed-initiative approach for\\nsummarizing discussions coupled with sentimental analysis,\" in Proceedings\\nof the 2014 International Conference on Interdisciplinary Advances\\nin Applied Computing. ACM, 2014, p. 5.\\nView publication stats'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJ8J5Qop33iw",
        "colab_type": "text"
      },
      "source": [
        "### 3. Function to tokenize the document\n",
        "Input: String of text document\n",
        "\n",
        "Output: A list containing sentences as its elements"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qwrCMDqM33ix",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# the function used for tokenizing the sentences\n",
        "# tokenization of a sentence: '''provided in cell() above'''\n",
        "\n",
        "def tokenize(document):\n",
        "    # We are tokenizing using the PunktSentenceTokenizer\n",
        "    # we call an instance of this class as sentence_tokenizer\n",
        "    doc_tokenizer = PunktSentenceTokenizer()\n",
        "    \n",
        "    # tokenize() method: takes our document as input and returns a list of all the sentences in the document\n",
        "    \n",
        "    # sentences is a list containing each sentence of the document as an element\n",
        "    sentences_list = doc_tokenizer.tokenize(document)\n",
        "    return sentences_list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6Z5ubiU33i3",
        "colab_type": "text"
      },
      "source": [
        "### 4. Read the document"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eE-usdCd33i4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        },
        "outputId": "63b124cf-4610-412d-f67d-7507c5f6cf4f"
      },
      "source": [
        "# reading a file and \n",
        "# printing the size of the file\n",
        "document = readDoc()\n",
        "print('The length of the file is:', end=' ')\n",
        "print(len(document))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Please input a file name: b.txt\n",
            "You have asked for the document b.txt\n",
            "1\n",
            "<class 'str'>\n",
            "The length of the file is: 31576\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWEicoUc33jE",
        "colab_type": "text"
      },
      "source": [
        "### 5. Generate a list of sentences in the document"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBrm9E8d33jG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "5fe10f66-853e-491e-92e8-300571093fda"
      },
      "source": [
        "# we want to tokenize the document for further processing\n",
        "# tokenizing the sentence means that we are creating a list of all the sentences of the document.\n",
        "# Need of tokenizing the document: Initially the document is in just a string format.\n",
        "# if we want to process the document, we need to store it in a data structure.\n",
        "# Tokenization of document into words is also possible, but we will go with the tokenizing with the sentences\n",
        "# Since we want to choose the most relevant sentences, we need to generate tokens of sentences only\n",
        "sentences_list = tokenize(document)\n",
        "\n",
        "# let us print the size of memory used by the list sentences\n",
        "print('The size of the list in Bytes is: {}'.format(sys.getsizeof(sentences_list)))\n",
        "\n",
        "# the size of one of the element of the list\n",
        "print('The size of the item 0 in Bytes is: {}'.format(sys.getsizeof(sentences_list[0])))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The size of the list in Bytes is: 2504\n",
            "The size of the item 0 in Bytes is: 744\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHdiiYbm33jM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6e3fb3a9-c7bd-4d72-be68-667560ca2310"
      },
      "source": [
        "# let us see the data type of sentences_list\n",
        "# It will be list\n",
        "print(type(sentences_list))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'list'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3f6qdbHx33jU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2af19a6a-1ff1-47f1-8991-bddb34916179"
      },
      "source": [
        "# let us analyse the elements of the sentences\n",
        "# len() method applies on the list and provides the number of elements in the list\n",
        "print('The size of the list \"sentences\" is: {}'.format(len(sentences_list)))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The size of the list \"sentences\" is: 266\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b54BAv2b33jd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "37bbd261-6240-49a3-f42f-a4c7d16c5a01"
      },
      "source": [
        "# print the elements of the list\n",
        "# If the input document is long, which on realistically will be wrong, we would not like to print the entire document\n",
        "for i in sentences_list:\n",
        "    print(i)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/317420253\n",
            "A survey on extractive text summarization\n",
            "Conference Paper · January 2017\n",
            "DOI: 10.1109/ICCCSP.2017.7944061\n",
            "CITATIONS\n",
            "26\n",
            "READS\n",
            "3,140\n",
            "2 authors:\n",
            "Some of the authors of this publication are also working on these related projects:\n",
            "human action recognition View project\n",
            "Human Activity Recognition View project\n",
            "N. Moratanch\n",
            "Anna University, Chennai\n",
            "4 PUBLICATIONS 51 CITATIONS\n",
            "SEE PROFILE\n",
            "Chitrakala Gopalan\n",
            "Anna University, Chennai\n",
            "115 PUBLICATIONS 211 CITATIONS\n",
            "SEE PROFILE\n",
            "All content following this page was uploaded by N. Moratanch on 07 November 2017.\n",
            "The user has requested enhancement of the downloaded file.\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "A Survey on Extractive Text Summarization\n",
            "N.Moratanch* ,S.Chitrakala t\n",
            "*Research Scholar, t Associate Professor\n",
            "*tDepartment of CSE\n",
            "Anna University,CEG,Chennai\n",
            "* tancyanbil@ gmail.com, t aU.chitras@ gmail.com\n",
            "Abstract-Text Summarization is the process of obtaining\n",
            "salient information from an authentic text document.\n",
            "In this\n",
            "technique, the extracted information is achieved as a summarized\n",
            "report and conferred as a concise summary to the user.\n",
            "It is very\n",
            "crucial for humans to understand and to describe the content\n",
            "of the text.\n",
            "Text Summarization techniques are classified into\n",
            "abstractive and extractive summarization.\n",
            "The extractive summarization\n",
            "technique focuses on choosing how paragraphs,important\n",
            "sentences, etc produces the original documents in precise form.\n",
            "The implication of sentences is determined based on linguistic\n",
            "and statistical features.\n",
            "In this work, a comprehensive review\n",
            "of extractive text summarization process methods has been\n",
            "ascertained.\n",
            "In this paper, the various techniques, populous\n",
            "benchmarking datasets and challenges of extractive summarization\n",
            "have been reviewed.\n",
            "This paper interprets extractive text\n",
            "summarization methods with a less redundant summary, highly\n",
            "adhesive, coherent and depth information.\n",
            "Index Terms-Text Summarization, Unsupervised Learning,\n",
            "Supervised Learning, Sentence Fusion, Extraction Scheme, Sentence\n",
            "Revision, Extractive Summary\n",
            "I. INTRODUCTION\n",
            "In a recent advance, the significance of text summarization\n",
            "[1] accomplishes more attention due to data inundation on\n",
            "the web.\n",
            "Hence this information overwhelms yields in the\n",
            "big requirement for more reliable and capable progressive text\n",
            "summarizers.\n",
            "Text Summarization gains its importance due to\n",
            "its various types of applications just like the summaries of\n",
            "books, digest- (summary of stories), the stock market, news,\n",
            "Highlights- (meeting, event, sport), Abstract of scientific papers,\n",
            "newspaper articles, magazine etc.\n",
            "Due to its tremendous\n",
            "growth, many finest universities like Faculty of Informatics\n",
            "- Masaryk University, Czech Republic, Concordia University,\n",
            "Montreal, Canada- Semantic Software Lab, IHR Nexus Lab\n",
            "at Arizona State University, Arizona, USA and finally Lab of\n",
            "Topic Maps-Leipzig University, Germany has been persistently\n",
            "working on its rapid enhancements.\n",
            "Text summarization has grown into a crucial and appropriate\n",
            "engine for supporting and illustrate text content in the latest\n",
            "speedy emergent information age.\n",
            "It's far very complex for\n",
            "humans to physically summarize oversized documents of\n",
            "text.\n",
            "There is a wealth of textual content available on the\n",
            "internet.\n",
            "But, usually, the internet contribute more data than\n",
            "is desired.\n",
            "Therefore, a twin problem is detected: Seeking for\n",
            "appropriate documents through an awe-inspiring number of\n",
            "reports offered, and fascinating a high volume of important\n",
            "information.\n",
            "The objective of automatic text summarization is\n",
            "to condense the origin text into a precise version preserves\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "its report content and global denotation.\n",
            "The main advantage\n",
            "of a text summarization is reading time of the user can be\n",
            "reduced.\n",
            "A marvelous text summary system should reproduce\n",
            "the assorted theme of the document even as keeping repetition\n",
            "to a minimum.\n",
            "Text Summarization methods are publicly\n",
            "restricted into abstractive and extractive summarization.\n",
            "An extractive summarization technique consists of selecting\n",
            "vital sentences, paragraphs, etc, from the original manuscript\n",
            "and concatenating them into a shorter form.\n",
            "The significance\n",
            "of sentences is strongly based on statistical and linguistic\n",
            "features of sentences.\n",
            "This paper generally summarizes the\n",
            "extensive methodologies fitted, issues launch, exploration and\n",
            "future directions in text summarization.\n",
            "This paper [1] is\n",
            "organized as follows.\n",
            "Section 2 depicts about the features for\n",
            "extractive text summarization, Section 3 describes extractive\n",
            "text summarization methods, Section 4 illustrate inferences\n",
            "made, Section 5 represent challenges and future research\n",
            "directions, Section 6 detail about evaluation metrics and the\n",
            "final sketch is the conclusion.\n",
            "II.\n",
            "FEATURES FOR EXTRACTIVE TEXT\n",
            "SUMMARIZATION\n",
            "Earlier techniques involve assigning a score to sentences\n",
            "based on a countenance that are predefined based on the\n",
            "methodology applied.\n",
            "Both word level and sentence level\n",
            "features are employed in text summarization literature.\n",
            "Certain\n",
            "features discussed are [2] [3] [4]used to exclusive sentences\n",
            "to be included in the summary are:\n",
            "1.\n",
            "WORD LEVEL FEATURES\n",
            "1.1 Content Word feature\n",
            "Keywords are essential in identifying the importance of the\n",
            "sentence.\n",
            "The sentence that consists of main keywords is most\n",
            "likely included in the final summary.\n",
            "The content (keyword)\n",
            "words are words that are nouns, verbs, adjectives and adverbs\n",
            "that are commonly determined based on tf x idf measure.\n",
            "1.2 Title Word feature\n",
            "The sentences in the original document which consists of\n",
            "words mentioned in the title have greater chances to contribute\n",
            "to the final summary since they serve as indicators of the theme\n",
            "of the document.\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "1.3 Cue phrase feature\n",
            "Cue phrases are words and phrases that indicate the structure\n",
            "of the document flow and it is used as a feature in\n",
            "sentence selection.\n",
            "The sentence that contains cue phrases\n",
            "(e.g.\n",
            "\"denouement\", \"because\", \"this information\", \"summary\",\n",
            "\"develop\", \"desire\" etc.)\n",
            "are mostly to be included in the final\n",
            "summary.\n",
            "1.4 Biased word feature\n",
            "The sentences that consist of biased words are more likely\n",
            "important.\n",
            "The biased words are a list of the predefined set\n",
            "of words that may be domain specific.\n",
            "They are relatively\n",
            "important words that describe the theme of the document.\n",
            "1.5 Upper case word feature\n",
            "The words which are in uppercase such as \"UNICEF\" are\n",
            "considered to be important words and those sentences that\n",
            "consist of these words are termed important in the context of\n",
            "sentence selection for the final summary.\n",
            "2.\n",
            "SENTENCE LEVEL FEATURES\n",
            "2.1 Sentence location feature\n",
            "The sentences that occur in the beginning and the conclusion\n",
            "part of the document are most likely important since most\n",
            "documents are hierarchically structured with important information\n",
            "in the beginning and the end of the paragraphs.\n",
            "2.2 Sentence length feature\n",
            "The sentence length plays an important role in identifying\n",
            "key sentences.\n",
            "Shorter texts do not convey essential information\n",
            "and very long sentences also need not be included in the\n",
            "summary.\n",
            "The normalized length of the sentence is calculated\n",
            "as the ratio between a number of words in the sentence to the\n",
            "number of words in the longest sentence in the document.\n",
            "2.3 Paragraph location feature\n",
            "Similar to sentence location, paragraph location also plays\n",
            "a crucial role in selecting key sentences.\n",
            "A Higher score is\n",
            "assigned to the paragraph in the peripheral sections (beginning\n",
            "and end paragraphs of the document).\n",
            "2.4 Sentence-to-Sentence Cohesion\n",
            "The cohesion between sentences for every sentence(s), the\n",
            "similarity between s and alternative sentences are calculated\n",
            "which are summed up and coarse value of the aspect is\n",
            "obtained for s. The feature values are normalized between\n",
            "[0, 1] where value closer to 1.0 indicates a higher degree of\n",
            "cohesion between sentences.\n",
            "III.\n",
            "EXTRACTIVE TEXT SUMMARIZATION\n",
            "METHODS\n",
            "Extractive Text Summarization methods can be broadly\n",
            "classified as Unsupervised Learning and Supervised learning\n",
            "methods.\n",
            "Recent works rely on Unsupervised Learning methods\n",
            "for text summarization.\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "Fig.\n",
            "1.\n",
            "Overview of Extractive Summarization\n",
            "A. UNSUPERVISED LEARNING METHODS\n",
            "In this section, unsupervised techniques for sentence extraction\n",
            "task is discussed.\n",
            "The unsupervised approaches do not\n",
            "need human summaries (user input) in deciding the important\n",
            "features of the document, it requires the most sophisticated\n",
            "algorithm to provide compensation for the lack of human\n",
            "knowledge.\n",
            "Unsupervised summaries provide a higher level\n",
            "of automation compared to supervised model and are more\n",
            "suitable for processing Big Data.\n",
            "Unsupervised learning models\n",
            "have proved successful in text summarization task.\n",
            "Fig.\n",
            "2.\n",
            "Overview of Unsupervised Learning Methods\n",
            "1.\n",
            "Graph based approach\n",
            "Graph-based models are extensively used in document summarization\n",
            "since graphs can efficiently represent the document\n",
            "structure.\n",
            "Extractive text summarization using external knowledge\n",
            "from Wikipedia incorporating bipartite graph framework\n",
            "[4 ]has been used.\n",
            "They have proposed an iterative ranking\n",
            "algorithm (variation of HITS algorithm [5]) which is efficient\n",
            "in selecting important sentences and also ensures coherency\n",
            "in the final summary.\n",
            "The uniqueness of this paper is that\n",
            "it combines both graph based and concept based approach\n",
            "towards summarization task.\n",
            "Another graph based approach\n",
            "LexRank [6], where the salience of the sentence is determined\n",
            "by the concept of Eigen vector centrality.\n",
            "The sentences in the\n",
            "document are represented as a graph and the edges between\n",
            "the sentences represents weighted cosine similarity values.\n",
            "The\n",
            "sentences are clustered into groups based on their similarity\n",
            "measures and then the sentences are ranked based on their\n",
            "LexRank scores similar to PageRank algorithm [7]except that\n",
            "the similarity graph is undirected in LexRank method.\n",
            "The\n",
            "method outperforms earlier versions of lead and centroid based\n",
            "approaches.\n",
            "The performance of the system is evaluated with\n",
            "DUC dataset [8].\n",
            "2.\n",
            "Fuzzy logic based approach\n",
            "The fuzzy logic approach mainly contains four components:\n",
            "defuzzifier, fuzzifier, fuzzy knowledge base and inference engine.\n",
            "The textual characteristics input of Fuzzy logic approach\n",
            "are sentenced length, sentence similarity etc which is later\n",
            "given to the fuzzy system [9] [10].\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "TABLE I\n",
            "SUPERVISED AND UNSUPERVISED LEARNING METHODS FOR TEXT SUMMARIZATION\n",
            "Categories Methodology Concept\n",
            "SUPERVISED Machine Learning ap- Summarization task\n",
            "LEARNING proach Bayes rule modelled as classification\n",
            "APPROACHES problem\n",
            "Trainable summarization -\n",
            "SUPERVISED neural network is trained,\n",
            "LEARNING Artificial Neural Net- pruned and generalized to\n",
            "APPROACHES work filter sentences and classify\n",
            "them as \"summary\" or\n",
            "\"non-summary sentence\"\n",
            "SUPERVISED Statistical modelling ap-\n",
            "LEARNING Conditional Random proach which uses CRF as\n",
            "APPROACHES Fields (CRF) a sequence labelling prob- lem\n",
            "UNSUPERVISED Graph based Construction of graph to\n",
            "LEARNING Approach capture relationship be-\n",
            "APPROACHES tween sentences\n",
            "Importance of sentences\n",
            "UNSUPERVISED Concept oriented ap- calculated based on\n",
            "LEARNING the concepts retrieved\n",
            "APPROACHES proach from external knowledge\n",
            "base(wikipedia, HowNet)\n",
            "UNSUPERVISED Fuzzy Logic based ap- Summarization based on\n",
            "LEARNING fuzzy rule using various\n",
            "APPROACHES proach sets of features\n",
            "Fig.\n",
            "3.\n",
            "Example of Sentence concept bipartite graph proposed in [4]\n",
            "Ladda Suanmali et al [11] proposed fuzzy logic approach\n",
            "is used for automatic text summarization which is the initial\n",
            "step , the text document is pre-processed followed by feature\n",
            "extraction(Title features, Sentence length, Sentence position,\n",
            "Sentence-sentence similarity, term weight, Proper noun and\n",
            "Numerical data.\n",
            "The summary is generated by ordering the\n",
            "ranked sentences in the order they occur in the original\n",
            "document to maintain coherency.\n",
            "The proposed method shows\n",
            "improvement in the quality of summarization but issues such\n",
            "as dangling anaphora are not handled.\n",
            "3.\n",
            "Concept-based approach\n",
            "In concept-based approach, the concepts are extracted from\n",
            "a piece of text from external knowledge base such HowNet\n",
            "[l2]and Wikipedia [4].\n",
            "In the methodology proposed [12], the\n",
            "importance of sentences is calculated based on the concepts\n",
            "retrieved from HowNet instead of words.\n",
            "A conceptual vector\n",
            "model is built to obtain a rough summarization and similarity\n",
            "measures are calculated between the sentences to reduce\n",
            "redundancy in the final summary.\n",
            "A good summarizer focuses\n",
            "on higher coverage and lower redundancy.\n",
            "Ramanathan\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "Advantages Limitations\n",
            "Large set of training data im- Human interruption required for proves the sentence selection for generating manual summaries summary\n",
            "The network can be trained ac- I)Neural Network is slow in\n",
            "cording to the style of human training phase and also in apreader.\n",
            "The set of features can be plication phase.\n",
            "2) It is difficult\n",
            "altered to reflect user's need and to determine how the net makes\n",
            "requirements decision.\n",
            "3) Requires human interruption\n",
            "for training data\n",
            "Identifies correct features and 1) focuses on domain specific\n",
            "provides better representation of which requires an external do- sentences and groups terms ap- main specific corpus for training\n",
            "propriately into its segments step.\n",
            "2) Limitation is that linguistic\n",
            "features are not considered\n",
            "1) Captures redundant inforrna- Doesn't focus on issues such as\n",
            "tion 2)Improves coherency dangling anaphora problem\n",
            "incorporation of similarity mea- Dangling anaphora and verb refsures\n",
            "to reduce redundancy erents not considered\n",
            "improved quality in summary by membership functions and work\n",
            "maintaining coherency of the fuzzy logic system\n",
            "Fig.\n",
            "4.\n",
            "Overall architecture of text summarization based on fuzzy logic\n",
            "approach proposed in [10]\n",
            "Fig.\n",
            "5.\n",
            "Example of concepts retrieved for sentence from Wikipedia as\n",
            "proposed in [4]\n",
            "et al [4] proposed a Wikipedia-based summarization which\n",
            "utilizes graph structure to produce summaries.\n",
            "The method\n",
            "uses Wikipedia to obtain concept for each sentence and\n",
            "builds a sentence-concept bipartite graph as already mentioned\n",
            "in Graph-based summarization.\n",
            "The basic steps in concept\n",
            "based summarization are: i) Retrieve concepts of a text from\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "external knowledge base(HowNet, WordNet, Wikipedia) ii)\n",
            "Build a conceptual vector or graph model to depict relationship\n",
            "between concept and sentences iii) Apply ranking algorithm to\n",
            "score sentences iv) Generate summaries based on the ranking\n",
            "scores of sentences\n",
            "4.\n",
            "Latent Semantic Analysis Method(LSA)\n",
            "Latent Semantic Analysis(LSA) [13] [14] is a method which\n",
            "excerpt hidden semantic structures of sentences and words\n",
            "that are popularly used in text summarization task.\n",
            "It is an\n",
            "unsupervised learning approach that does not demand any sort\n",
            "of external or training knowledge.\n",
            "LSA captures the text of the\n",
            "input document and excerpt information such as words that\n",
            "frequently occur together and words that are commonly seen in\n",
            "different sentences.\n",
            "A high number of common words amongst\n",
            "the sentences illustrate that the sentences are semantically\n",
            "related.\n",
            "Singular Value Decomposition(SVD) [13], is a method\n",
            "used to find out the interrelations between words and sentences\n",
            "which also has the competence of noise reduction that helps\n",
            "to improve accuracy.\n",
            "SVD, [15] when enforced to document\n",
            "word matrices, can group documents that are semantically\n",
            "associated to one other despite them sharing no common\n",
            "words.\n",
            "The set of words that ensue in connected text is\n",
            "also connected within the same peculiar dimensional space.\n",
            "LSA technique is applied to excerpt the subject-related words\n",
            "and important content conveying sentences from report.\n",
            "The\n",
            "advantage of adopting LSA vectors for summarization over\n",
            "word vectors is that conceptual relations as represented in the\n",
            "human brain are naturally captured in the LSA.\n",
            "Choice of\n",
            "the representative sentence from every scale of the capacity\n",
            "ensures relevancy of sentence to the document and ensures\n",
            "non-redundancy.\n",
            "LS works with text data and the principal\n",
            "ambit due to the collection of topics can be located.\n",
            "Considering an example to depict LSA representatieach\n",
            "otheron, Example 1: Consider following 3 sentences given to\n",
            "LSA based system.\n",
            "dO: 'The man was walked the dog.\n",
            "dl:\n",
            "'The man took the dog to the park in the evening.\n",
            "d2: 'The\n",
            "dog went to the park in the evening.\n",
            "From Fig6 [13] it is to\n",
            "be noted in order that dl is associated to d2 than dO and the\n",
            "conversation 'walked' is linked to the talk 'man' but it is not\n",
            "significant to the word 'park'.\n",
            "These kind of interpretations\n",
            "can be built by using input data and LSA, beyond need for\n",
            "any extraneous awareness.\n",
            "B. SUPERVISED LEARNING METHODS\n",
            "Supervised extractive summarizationrelated techniques are\n",
            "based on a classification approach at sentence level where\n",
            "the system learns by examples to classify between summary\n",
            "and non-summary sentences.\n",
            "The major drawback with the\n",
            "supervised approach is that it requires known manually created\n",
            "summaries by a human to label the sentences in the original\n",
            "training document enclosed with \"summary sentence\" or \"nonsummary\n",
            "sentence\" and it also requires more labeled training\n",
            "data for classification.\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "Fig.\n",
            "6.\n",
            "Representation of LSA for Example [13]\n",
            "Fig.\n",
            "7.\n",
            "Overview of Supervised Learning Methods\n",
            "1.\n",
            "Machine Learning Approach based on Bayes rule\n",
            "A set of training documents along with its extractive summaries\n",
            "is fed as input to the training stage.\n",
            "The machine\n",
            "learning approach views classification problem in text summarization.\n",
            "The sentences are restricted as a non-summary\n",
            "and summary sentence based on the feature possessed by the\n",
            "sentence.\n",
            "The probability of classification are learned from the\n",
            "training data by the following Bayes rule [16]: where s represents\n",
            "the set of sentences in the document and fi represents\n",
            "the features used in classification stage and S represents the\n",
            "set of sentences in the summary.\n",
            "P (s E< SII1,h,h, .... In)\n",
            "represents the probability of the sentences to be included in\n",
            "the summary based on the given features possessed by the\n",
            "sentence.\n",
            "2.\n",
            "Neural Network based approach\n",
            "Fig.\n",
            "8.\n",
            "Neural network after training (a) and after pruning (b) [17]\n",
            "In the approach proposed in [18], RankNet algorautomaticallyithm\n",
            "using neural nets to identify the important sentences\n",
            "in the document.\n",
            "It uses a two-layer neural network with\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "back propagation trained using RankNet algorithm.\n",
            "The first\n",
            "step involves labeling the training data using a machinelearning\n",
            "approach and then extract features of the sentences\n",
            "in both test set and train sets which is then inputted to the\n",
            "neural network system to rank the sentences in the document.\n",
            "Another approach [17]uses a three layered feed-forward neural\n",
            "network which learns in the training stage the characteristics\n",
            "of summary and non-summary sentences.\n",
            "The major phase\n",
            "is the feature fusion phase where the relationship between\n",
            "the features are identified through two stages 1) eliminating\n",
            "infrequent features 2) collapsing frequent features after\n",
            "which sentence ranking is done to identify the important\n",
            "summary sentences.Neural Network [17]after feature fusion\n",
            "is depicted in Fig 8.\n",
            "Dharmendra Hingu, Deep Shah and\n",
            "Sandeep S.Udmale proposed an extractive approach [19]for\n",
            "summarizing the Wikipedia articles by identifying the text\n",
            "features and scoring the sentences by incorporating neural\n",
            "network model [5].\n",
            "This system gets the Wikipedia articles\n",
            "as input followed by tokenisation and stemming.\n",
            "The preprocessed\n",
            "passage is sent to the feature extraction steps, which\n",
            "is based on multiple features of sentences and words.\n",
            "The\n",
            "scores obtained after the feature extraction are fed to the\n",
            "neural network, which produces a single value as output score,\n",
            "signifying the importance of the sentences.\n",
            "Usage of the words\n",
            "and sentences is not considered while assigning the weights\n",
            "which results in less accuracy.\n",
            "3.\n",
            "Conditional Random Fields\n",
            "Conditional Random Fields are a statistical modeling approach\n",
            "that focuses on machine leaning to provide a structured\n",
            "prediction.\n",
            "The proposed system overcomes the issues faced\n",
            "by non-negative matrix Factorization (NMF) methods by incorporating\n",
            "conditional random fields (CRF) to identify and\n",
            "extract correct features to determine the important sentence\n",
            "of the given text.\n",
            "The main advantage of the method is that\n",
            "it is able to identify correct features and provides a better\n",
            "representation of sentences and groups terms appropriately\n",
            "into its segments.\n",
            "The major drawback of the method is that it\n",
            "focuses on domain-specific which requires an external domain\n",
            "specific corpus for training step, thus this method cannot be\n",
            "applied generically to any document without building a domain\n",
            "corpus which is a time-consuming task.\n",
            "The approach specified\n",
            "in [20]uses CRF as a sequence labelling problem and also\n",
            "captures interaction between sentences through the features\n",
            "extracted for each sentence and it also incorporates complex\n",
            "features such as LSA_scores [21] and lilTS_score [22] but\n",
            "the limitation is that linguistic features are not considered.\n",
            "IV.\n",
            "INFERENCES MADE\n",
            "• Abounding variations of the extractive path [15] have\n",
            "been focused in the prior ten years.\n",
            "However, it is difficult\n",
            "to say how analytical improvement (sentence or text\n",
            "level) devote to work.\n",
            "• Beyond NLP, the achieved summary might endure a lack\n",
            "of semantics and cohesion.\n",
            "In texts consist of numerous\n",
            "topics, the provoked summary may not be fair.\n",
            "Conclusive\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "proper weights for respective features is vital to the\n",
            "quality of concluding summary depends on it.\n",
            "• Feature weights should be given more importance because\n",
            "it plays a major role in choosing key sentences.\n",
            "In text Summarization, the most challenging task is\n",
            "to summarize the contented from a number of semistructured\n",
            "sources and textual, which includes web pages\n",
            "and databases, in the proper way (size, format, time,\n",
            "language,) for an explicit user.\n",
            "• Text summary software should crop effective summary\n",
            "within a fewer amount of redundancy and time.\n",
            "Summarization\n",
            "appraise using extrinsic or intrinsic part.\n",
            "• Intrinsic parts pursuit to measure summary nature adopting\n",
            "human evaluation whereas, extrinsic parts measure\n",
            "the same over a effort-based work measure being the\n",
            "information rehabilitation-oriented task.\n",
            "V. EVALUATION METRICS\n",
            "Numerous benchmarking datasets [1] are used for experimental\n",
            "evaluation of extractive summarization.\n",
            "Document Understanding\n",
            "Conferences (DUC) is the most common benchmarking\n",
            "datasets used for text summarization.\n",
            "There are a\n",
            "number of datasets like TIPSTER, TREC , TAC , DUC, CNN.\n",
            "It contains documents along with their summaries that are\n",
            "created automatically, manually and submitted summaries[20].\n",
            "From papers surveyed within the previous sections et al in\n",
            "literature, it's been found that agreement between human\n",
            "summarizers is sort of low, each for evaluating and generating\n",
            "summaries quite the shape of the outline, it is tough to judge\n",
            "the outline content.\n",
            "i) Human Evaluation\n",
            "Human judgment usually has wide variance on what's\n",
            "thought-about a \"good\" outline, which implies that creating\n",
            "the analysis method automatic is especially tough.\n",
            "Manual\n",
            "analysis is used, however, this can be each time and laborintensive\n",
            "because it needs humans to browse not solely the\n",
            "summaries however conjointly the supply documents.\n",
            "Other\n",
            "issues are those regarding coherence and coverage.\n",
            "ii) Rouge\n",
            "Formally, ROUGE-N is an n-gram recall between a candidate\n",
            "summary and a set of reference summaries.\n",
            "ROUGE-N\n",
            "is computed as follows:\n",
            "ROUGE-N = LSEreference_summaries LN-grams Countmatch{N-gram)\n",
            "L S Ereference_summaries LN-grams Count{N -gram)\n",
            "where, n stands for the length of the n-gram Countmatch(Ngram)\n",
            "is the maximum number of n-grams co-occurring in\n",
            "a candidate summary and a set of reference summaries.\n",
            "Count(N-gram) is the number of N-grams in the set of\n",
            "reference summaries.\n",
            "... ) R 11 R _ ISref n Scandl\n",
            "m eca - ISrefl\n",
            "where Sref n Scand indicates the number of sentences that\n",
            "co-occur in both reference and candidate summaries.\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "t.V ) P rect.s.w n (P) P = ISreIf n ScIa ndl\n",
            "Scand\n",
            ")\n",
            "_ 2(Precision)(Recall)\n",
            "v F -measure F - PreCI.S l.O n + Rec all\n",
            "vi) Compression Ratio Cr = Slen .\n",
            "Iien\n",
            "where, Slen and Iien are the length of summary and source\n",
            "text respectively.\n",
            "VI.\n",
            "CHALLENGES AND FUTURE RESEARCH\n",
            "DIRECTIONS\n",
            "Evaluating summaries (either automatically or manually) is\n",
            "a difficult task.\n",
            "The main problem in evaluation comes from\n",
            "the impossibility of building a standard against which the\n",
            "results of the systems that have to be compared.\n",
            "Further, it\n",
            "is very hard to find out what a correct summary is because\n",
            "there is a chance of the system to generate a better summary\n",
            "that is different from any human summary which is used as an\n",
            "approximation to correct output.\n",
            "Content choice [23] is not a\n",
            "settled problem.\n",
            "People are completely different and subjective\n",
            "authors would possibly select completely different sentences.\n",
            "Two distinct sentences expressed in disparate words will\n",
            "specific a similar can explicit the same meaning also known\n",
            "as paraphrasing.\n",
            "There exists an approach to automatically\n",
            "evaluate summaries using paraphrases (paraEval).\n",
            "Most text\n",
            "summarization systems perform extractive summarization approach\n",
            "(selecting and photocopying extensive sentences from\n",
            "the professional documents).\n",
            "Though humans can cut and paste\n",
            "relevant data from a text, most of the times they rephrase sentences\n",
            "whenever necessary, or they may join different related\n",
            "data into one sentence.\n",
            "The low inter-annotator agreement\n",
            "figures observed during manual evaluations suggest that the\n",
            "future of this research area massively depends on the capacity\n",
            "to find efficient ways of automatically evaluating the systems.\n",
            "VII.\n",
            "CONCLUSION\n",
            "This review has shown assorted mechanism of extractive\n",
            "text summarization process.\n",
            "Extractive summarization process\n",
            "is highly coherent, less redundant and cohesive (summary and\n",
            "information rich).\n",
            "The aim is to give a comprehensive review\n",
            "and comparison of distinctive approaches and techniques of\n",
            "extractive text summarization process.\n",
            "Although research on\n",
            "summarization started way long back, there is still a long way\n",
            "to go.\n",
            "Over the time, focused has drifted from summarizing\n",
            "scientific articles to advertisements, blogs, electronic mail\n",
            "messages and news articles.\n",
            "Simple eradication of sentences\n",
            "has composed satisfactory results in massive applications.\n",
            "Some trends in automatic evaluation of summary system have\n",
            "been focused.\n",
            "However, the work has not focused the different\n",
            "challenges of extractive text summarization process to its full\n",
            "intensity in premises of time and space complication.\n",
            "REFERENCES\n",
            "[1] N. Moratanch and S. Chitrakala, \"A survey on abstractive text summarization,\"\n",
            "in Circuit, Power and Computing Technologies (ICCPCT),\n",
            "2016 International Conference on.\n",
            "IEEE, 2016, pp.\n",
            "1-7.\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "[2] F. Kiyomarsi and F. R. Esfahani, \"Optimizing persian text summarization\n",
            "based on fuzzy logic approach,\" in 2011 International Conference on\n",
            "Intelligent Building and Management, 2011.\n",
            "[3] F. Chen, K. Han, and G. Chen, \"An approach to sentence-selectionbased\n",
            "text summarization,\" in TENCON'02.\n",
            "Proceedings.\n",
            "2002 IEEE\n",
            "Region 10 Conference on Computers, Communications, Control and\n",
            "Power Engineering, vol.\n",
            "1.\n",
            "IEEE, 2002, pp.\n",
            "489-493.\n",
            "[4] Y. Sankarasubramaniam, K. Ramanathan, and S. Ghosh, \"Text summarization\n",
            "using wikipedia,\" Information Processing & Management,\n",
            "vol.\n",
            "50, no.\n",
            "3, pp.\n",
            "443-461, 2014.\n",
            "[5] J. M. Kleinberg, \"Authoritative sources in a hyperlinked environment,\"\n",
            "Journal of the ACM (JACM), vol.\n",
            "46, no.\n",
            "5, pp.\n",
            "604--632, 1999.\n",
            "[6] G. Erkan and D. R. Radev, \"Lexrank: Graph-based lexical centrality\n",
            "as salience in text summarization,\" Journal of Artificial Intelligence\n",
            "Research, pp.\n",
            "457-479, 2004.\n",
            "[7] S. M. R .. W. T. L., Brin, \"The pagerank citation ranking: Bringing\n",
            "order to the web,\" Technical report, Stanford University, Stanford, CA.,\n",
            "Tech.\n",
            "Rep., (1998).\n",
            "[8] (2004) Document understanding conferences dataset.\n",
            "Online.\n",
            "[Online].\n",
            "Available: http://duc.nist.gov/data.htrnl.\n",
            "[9] F. Kyoomarsi, H. Khosravi, E. Eslami, P. K. Dehkordy, and A. Tajoddin,\n",
            "\"Optimizing text summarization based on fuzzy logic,\" in Seventh\n",
            "IEEElACIS International Conference on Computer and Information\n",
            "Science.\n",
            "IEEE, 2008, pp.\n",
            "347-352.\n",
            "[10] L. SUanmali, M. S. Binwahlan, and N. Salim, \"Sentence features fusion\n",
            "for text summarization using fuzzy logic,\" in Hybrid Intelligent Systems,\n",
            "2009.\n",
            "HIS'09.\n",
            "Ninth International Conference on, vol.\n",
            "1.\n",
            "IEEE, 2009,\n",
            "pp.\n",
            "142-146.\n",
            "[11] L. Suanmali, N. Salim, and M. S. Binwahlan, \"Fuzzy logic based method\n",
            "for improving text summarization,\" arXiv pre print arXiv:0906.4690,\n",
            "2009.\n",
            "[12] X. W. Meng Wang and C. Xu, \"An approach to concept oriented text\n",
            "summarization,\" in in Proceedings of ISClTS05, IEEE international\n",
            "conference, China,1290-1293\" 2005.\n",
            "[13] M. G. Ozsoy, F. N. Alpaslan, and 1.\n",
            "Cicekli, \"Text summarization using\n",
            "latent semantic analysis,\" Journal of Information Science, vol.\n",
            "37, no.\n",
            "4,\n",
            "pp.\n",
            "405-417, 2011.\n",
            "[14] 1.\n",
            "Mashechkin, M. Petrovskiy, D. Popov, and D. V. Tsarev, \"Automatic\n",
            "text summarization using latent semantic analysis,\" Programming and\n",
            "Computer Software, vol.\n",
            "37, no.\n",
            "6, pp.\n",
            "299-305, 2011.\n",
            "[15] V. Gupta and G. S. Lehal, \"A survey of text summarization extractive\n",
            "techniques,\" Journal of emerging technologies in web intelligence, vol.\n",
            "2,\n",
            "no.\n",
            "3, pp.\n",
            "258-268, 2010.\n",
            "[16] J. L. Neto, A. A. Freitas, and C. A. Kaestner, \"Automatic text summarization\n",
            "using a machine learning approach,\" in Advances in Artificial\n",
            "Intelligence.\n",
            "Springer, 2002, pp.\n",
            "205-215.\n",
            "[17] K. Kaikhah, \"Automatic text summarization with neural networks,\"\n",
            "2004.\n",
            "[18] K. M. Svore, L. Vanderwende, and C. J. Burges, \"Enhancing singledocument\n",
            "summarization by combining ranknet and third-party sources.\"\n",
            "in EMNLP-CoNLL, 2007, pp.\n",
            "448-457.\n",
            "[19] D. Hingu, D. Shah, and S. S. Udmale, \"Automatic text summarization\n",
            "of wikipedia articles,\" in Communication, Information & Computing\n",
            "Technology (ICCICT), 2015 International Conference on.\n",
            "IEEE,2015,\n",
            "pp.I-4.\n",
            "[20] D. Shen, J.-T.\n",
            "Sun, H. Li, Q. Yang, and Z. Chen, \"Document summarization\n",
            "using conditional random fields.\"\n",
            "in IJCAI, vol.\n",
            "7, 2007, pp.\n",
            "2862-2867.\n",
            "[21] Y. Gong and X. Liu, \"Generic text summarization using relevance\n",
            "measure and latent semantic analysis,\" in Proceedings of the 24th annual\n",
            "international ACM SIGIR conference on Research and development in\n",
            "information retrieval.\n",
            "ACM, 2001, pp.\n",
            "19-25.\n",
            "[22] R. Mihalcea, \"Language independent extractive summarization,\" in\n",
            "Proceedings of the ACL 2005 on Interactive poster and demonstration\n",
            "sessions.\n",
            "Association for Computational Linguistics, 2005, pp.\n",
            "49-52.\n",
            "[23] N. Lalithamani, R. Sukumaran, K. Alagamrnai, K. K. Sowmya, V. Divyalakshmi,\n",
            "and S. Shanmugapriya, ''A mixed-initiative approach for\n",
            "summarizing discussions coupled with sentimental analysis,\" in Proceedings\n",
            "of the 2014 International Conference on Interdisciplinary Advances\n",
            "in Applied Computing.\n",
            "ACM, 2014, p.\n",
            "5.\n",
            "View publication stats\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Us-qLC3S33ji",
        "colab_type": "text"
      },
      "source": [
        "### 6. Generate term-document matrix (TD matrix) of the data "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BEz4ASsI33jj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Convert a collection of text documents to a matrix of token counts\n",
        "# fit_transform method of CountVectorizer() class \n",
        "# Learn the vocabulary dictionary and return term-document matrix. \n",
        "# I/p: An iterable which yields either str, unicode or file objects.\n",
        "# O/p: The term-document matrix named cv_matrix\n",
        "cv = CountVectorizer()\n",
        "cv_matrix = cv.fit_transform(sentences_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0KPat0333jq",
        "colab_type": "text"
      },
      "source": [
        "**So what does CountVectorizer.fit_transform() do?**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KKx8ET5T33js",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "976c2235-6959-4dc6-e613-cdedef8f6bc2"
      },
      "source": [
        "# a demo of what CountVectorizer().fit_transform(text) does\n",
        "cv_demo = CountVectorizer() # a demo object of class CountVectorizer\n",
        "\n",
        "# I have repeated the words to make a non-ambiguous array of the document text matrix \n",
        "\n",
        "text_demo = [\"Ashish is good, you are bad\", \"I am not bad\"] \n",
        "res_demo = cv_demo.fit_transform(text_demo)\n",
        "print('Result demo array is {}'.format(res_demo.toarray()))\n",
        "\n",
        "# Result is 2-d matrix containing document text matrix\n",
        "# Notice that in the second row, there is 2.\n",
        "# also, bad is repeated twice in that sentence.\n",
        "# so we can infer that 2 is corresponding to the word 'bad'\n",
        "print('Feature list: {}'.format(cv_demo.get_feature_names()))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Result demo array is [[0 1 1 1 1 1 0 1]\n",
            " [1 0 0 1 0 0 1 0]]\n",
            "Feature list: ['am', 'are', 'ashish', 'bad', 'good', 'is', 'not', 'you']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "swGz33yn33jx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "2abbd047-297c-415b-bb37-663e97822905"
      },
      "source": [
        "# printing the cv_matrix type\n",
        "# and how it is being stored in memory?\n",
        "# it is stored in the compressed row format\n",
        "# compressed row format: \n",
        "print('The data type of bow matrix {}'.format(type(cv_matrix)))\n",
        "print('Shape of the matrix {}'.format(cv_matrix.get_shape))\n",
        "print('Size of the matrix is: {}'.format(sys.getsizeof(cv_matrix)))\n",
        "print(cv.get_feature_names())\n",
        "print(cv_matrix.toarray())"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The data type of bow matrix <class 'scipy.sparse.csr.csr_matrix'>\n",
            "Shape of the matrix <bound method spmatrix.get_shape of <266x1284 sparse matrix of type '<class 'numpy.int64'>'\n",
            "\twith 3878 stored elements in Compressed Sparse Row format>>\n",
            "Size of the matrix is: 56\n",
            "['00', '02', '07', '09', '0906', '10', '11', '1109', '115', '12', '1290', '1293', '13', '14', '140', '142', '146', '15', '16', '17', '18', '19', '1998', '1999', '20', '2001', '2002', '2004', '2005', '2007', '2008', '2009', '2010', '2011', '2014', '2015', '2016', '2017', '205', '21', '211', '215', '22', '23', '24th', '25', '258', '26', '268', '2862', '2867', '299', '305', '31', '317420253', '347', '352', '37', '3716', '405', '417', '443', '448', '457', '46', '461', '4690', '479', '489', '49', '493', '50', '5090', '51', '52', '604', '632', '7944061', '978', 'able', 'abounding', 'about', 'abstract', 'abstractive', 'ac', 'accomplishes', 'accuracy', 'achieved', 'acl', 'acm', 'action', 'activity', 'adhesive', 'adjectives', 'adopting', 'advance', 'advances', 'advantage', 'advantages', 'adverbs', 'advertisements', 'after', 'against', 'age', 'agreement', 'aim', 'al', 'alagamrnai', 'algorautomaticallyithm', 'algorithm', 'all', 'along', 'alpaslan', 'already', 'also', 'altered', 'alternative', 'although', 'ambit', 'amongst', 'amount', 'an', 'analysis', 'analytical', 'anaphora', 'and', 'anna', 'annotator', 'annual', 'another', 'any', 'ap', 'applications', 'applied', 'apply', 'appraise', 'approach', 'approaches', 'appropriate', 'appropriately', 'approximation', 'apreader', 'architecture', 'are', 'area', 'arizona', 'articles', 'artificial', 'arxiv', 'as', 'ascertained', 'aspect', 'assigned', 'assigning', 'associate', 'associated', 'association', 'assorted', 'at', 'attention', 'au', 'authentic', 'author', 'authoritative', 'authors', 'automatic', 'automatically', 'automation', 'available', 'awareness', 'awe', 'back', 'base', 'based', 'basic', 'bayes', 'be', 'because', 'been', 'beginning', 'being', 'benchmarking', 'better', 'between', 'beyond', 'biased', 'big', 'binwahlan', 'bipartite', 'blogs', 'books', 'both', 'brain', 'brin', 'bringing', 'broadly', 'browse', 'build', 'building', 'builds', 'built', 'burges', 'but', 'by', 'ca', 'calculated', 'can', 'canada', 'candidate', 'cannot', 'capable', 'capacity', 'capture', 'captured', 'captures', 'case', 'categories', 'ceg', 'centrality', 'centroid', 'certain', 'challenges', 'challenging', 'chance', 'chances', 'characteristics', 'chen', 'chennai', 'china', 'chitrakala', 'chitras', 'choice', 'choosing', 'cicekli', 'circuit', 'citation', 'citations', 'classification', 'classified', 'classify', 'closer', 'clustered', 'cnn', 'co', 'coarse', 'coherence', 'coherency', 'coherent', 'cohesion', 'cohesive', 'collapsing', 'collection', 'com', 'combines', 'combining', 'comes', 'common', 'commonly', 'communication', 'communications', 'compared', 'comparison', 'compensation', 'competence', 'completely', 'complex', 'complication', 'components', 'composed', 'comprehensive', 'compression', 'computational', 'computed', 'computer', 'computers', 'computing', 'concatenating', 'concept', 'concepts', 'conceptual', 'concise', 'concluding', 'conclusion', 'conclusive', 'concordia', 'condense', 'conditional', 'conference', 'conferences', 'conferred', 'conjointly', 'conll', 'connected', 'consider', 'considered', 'considering', 'consist', 'consists', 'construction', 'consuming', 'contains', 'content', 'contented', 'context', 'contribute', 'control', 'conversation', 'convey', 'conveying', 'cording', 'corpus', 'correct', 'cosine', 'count', 'countenance', 'countmatch', 'coupled', 'coverage', 'cr', 'created', 'creating', 'crf', 'crop', 'crucial', 'cse', 'cue', 'cut', 'czech', 'd2', 'dangling', 'data', 'databases', 'dataset', 'datasets', 'deciding', 'decision', 'decomposition', 'deep', 'defuzzifier', 'degree', 'dehkordy', 'demand', 'demonstration', 'denotation', 'denouement', 'depends', 'depict', 'depicted', 'depicts', 'depth', 'describe', 'describes', 'desire', 'desired', 'despite', 'detail', 'detected', 'determine', 'determined', 'develop', 'development', 'devote', 'dharmendra', 'different', 'difficult', 'digest', 'dimensional', 'directions', 'discussed', 'discussions', 'disparate', 'distinct', 'distinctive', 'divyalakshmi', 'dl', 'do', 'document', 'documents', 'does', 'doesn', 'dog', 'doi', 'domain', 'done', 'downloaded', 'drawback', 'drifted', 'duc', 'due', 'during', 'each', 'earlier', 'eca', 'edges', 'effective', 'efficient', 'efficiently', 'effort', 'eigen', 'either', 'electronic', 'eliminating', 'emergent', 'emerging', 'emnlp', 'employed', 'enclosed', 'end', 'endure', 'enforced', 'engine', 'engineering', 'enhancement', 'enhancements', 'enhancing', 'ensue', 'ensures', 'environment', 'eradication', 'ereference_summaries', 'erents', 'erkan', 'esfahani', 'eslami', 'especially', 'essential', 'et', 'etc', 'evaluate', 'evaluated', 'evaluating', 'evaluation', 'evaluations', 'even', 'evening', 'event', 'every', 'example', 'examples', 'except', 'excerpt', 'exclusive', 'exists', 'experimental', 'explicit', 'exploration', 'expressed', 'extensive', 'extensively', 'external', 'extract', 'extracted', 'extraction', 'extractive', 'extraneous', 'extrinsic', 'faced', 'factorization', 'faculty', 'fair', 'far', 'fascinating', 'feature', 'features', 'fed', 'feed', 'fewer', 'fi', 'fields', 'fig', 'fig6', 'figures', 'file', 'filter', 'final', 'finally', 'find', 'finest', 'first', 'fitted', 'flow', 'focus', 'focused', 'focuses', 'followed', 'following', 'follows', 'for', 'form', 'formally', 'format', 'forward', 'found', 'four', 'framework', 'freitas', 'frequent', 'frequently', 'from', 'full', 'functions', 'further', 'fusion', 'future', 'fuzzifier', 'fuzzy', 'gains', 'generalized', 'generally', 'generate', 'generated', 'generating', 'generic', 'generically', 'germany', 'gets', 'ghosh', 'give', 'given', 'global', 'gmail', 'go', 'gong', 'good', 'gopalan', 'gov', 'gram', 'grams', 'graph', 'graphs', 'greater', 'group', 'groups', 'grown', 'growth', 'gupta', 'han', 'handled', 'hard', 'has', 'have', 'helps', 'hence', 'hidden', 'hierarchically', 'high', 'higher', 'highlights', 'highly', 'hingu', 'his', 'hits', 'how', 'however', 'hownet', 'htrnl', 'http', 'https', 'human', 'humans', 'hybrid', 'hyperlinked', 'icccsp', 'iccict', 'iccpct', 'identified', 'identifies', 'identify', 'identifying', 'idf', 'ieee', 'ieeelacis', 'ihr', 'ii', 'iien', 'iii', 'ijcai', 'illustrate', 'im', 'implication', 'implies', 'importance', 'important', 'impossibility', 'improve', 'improved', 'improvement', 'improves', 'improving', 'in', 'included', 'includes', 'incorporates', 'incorporating', 'incorporation', 'independent', 'index', 'indicate', 'indicates', 'indicators', 'inference', 'inferences', 'informatics', 'information', 'inforrna', 'infrequent', 'initial', 'initiative', 'input', 'inputted', 'inspiring', 'instead', 'intelligence', 'intelligent', 'intensity', 'inter', 'interaction', 'interactive', 'interdisciplinary', 'international', 'internet', 'interpretations', 'interprets', 'interrelations', 'interruption', 'into', 'intrinsic', 'introduction', 'inundation', 'involve', 'involves', 'is', 'isclts05', 'isref', 'isrefl', 'isreif', 'issues', 'it', 'iterative', 'its', 'iv', 'jacm', 'january', 'join', 'journal', 'judge', 'judgment', 'just', 'kaestner', 'kaikhah', 'keeping', 'key', 'keyword', 'keywords', 'khosravi', 'kind', 'kiyomarsi', 'kleinberg', 'knowledge', 'known', 'kyoomarsi', 'l2', 'lab', 'label', 'labeled', 'labeling', 'labelling', 'laborintensive', 'lack', 'ladda', 'lalithamani', 'language', 'large', 'latent', 'later', 'latest', 'launch', 'layer', 'layered', 'lead', 'leaning', 'learned', 'learning', 'learns', 'lehal', 'leipzig', 'lem', 'length', 'less', 'level', 'lexical', 'lexrank', 'li', 'like', 'likely', 'lilts_score', 'limitation', 'limitations', 'linguistic', 'linguistics', 'linked', 'list', 'literature', 'liu', 'ln', 'located', 'location', 'logic', 'long', 'longest', 'low', 'lower', 'ls', 'lsa', 'lsa_scores', 'lsereference_summaries', 'machine', 'machinelearning', 'made', 'magazine', 'mail', 'main', 'mainly', 'maintain', 'maintaining', 'major', 'makes', 'man', 'management', 'manual', 'manually', 'manuscript', 'many', 'maps', 'market', 'marvelous', 'masaryk', 'mashechkin', 'massive', 'massively', 'matrices', 'matrix', 'maximum', 'may', 'mea', 'meaning', 'measure', 'measures', 'mechanism', 'meeting', 'membership', 'meng', 'mentioned', 'messages', 'method', 'methodologies', 'methodology', 'methods', 'metrics', 'might', 'mihalcea', 'minimum', 'mixed', 'model', 'modeling', 'modelled', 'modelling', 'models', 'montreal', 'moratanch', 'more', 'most', 'mostly', 'multiple', 'naturally', 'nature', 'ndl', 'necessary', 'need', 'needs', 'negative', 'net', 'neto', 'nets', 'network', 'networks', 'neural', 'news', 'newspaper', 'nexus', 'ngram', 'ninth', 'nist', 'nlp', 'nmf', 'no', 'noise', 'non', 'nonsummary', 'normalized', 'not', 'noted', 'noun', 'nouns', 'november', 'number', 'numerical', 'numerous', 'objective', 'observed', 'obtain', 'obtained', 'obtaining', 'occur', 'occurring', 'of', 'offered', 'on', 'one', 'online', 'optimizing', 'or', 'order', 'ordering', 'organized', 'oriented', 'origin', 'original', 'other', 'otheron', 'out', 'outline', 'outperforms', 'output', 'over', 'overall', 'overcomes', 'oversized', 'overview', 'overwhelms', 'ozsoy', 'page', 'pagerank', 'pages', 'paper', 'papers', 'paraeval', 'paragraph', 'paragraphs', 'paraphrases', 'paraphrasing', 'park', 'part', 'parts', 'party', 'passage', 'paste', 'path', 'peculiar', 'people', 'perform', 'performance', 'peripheral', 'persian', 'persistently', 'petrovskiy', 'phase', 'photocopying', 'phrase', 'phrases', 'physically', 'piece', 'plays', 'plication', 'popov', 'popularly', 'populous', 'position', 'possessed', 'possibly', 'poster', 'power', 'pp', 'pre', 'preci', 'precise', 'precision', 'predefined', 'prediction', 'premises', 'preprocessed', 'preserves', 'previous', 'principal', 'print', 'prior', 'proach', 'prob', 'probability', 'problem', 'proceedings', 'process', 'processed', 'processing', 'produce', 'produces', 'professional', 'professor', 'profile', 'profiles', 'programming', 'progressive', 'project', 'projects', 'propagation', 'proper', 'proposed', 'propriately', 'proved', 'proves', 'provide', 'provides', 'provoked', 'pruned', 'pruning', 'publication', 'publications', 'publicly', 'pursuit', 'quality', 'quite', 'radev', 'ramanathan', 'random', 'rank', 'ranked', 'ranking', 'ranknet', 'rapid', 'ratio', 'reading', 'reads', 'rec', 'recall', 'recent', 'recognition', 'rect', 'reduce', 'reduced', 'reduction', 'redundancy', 'redundant', 'reference', 'references', 'reflect', 'refsures', 'regarding', 'region', 'rehabilitation', 'related', 'relations', 'relationship', 'relatively', 'relevance', 'relevancy', 'relevant', 'reliable', 'rely', 'rep', 'repetition', 'rephrase', 'report', 'reports', 'represent', 'representatieach', 'representation', 'representative', 'represented', 'represents', 'reproduce', 'republic', 'requested', 'required', 'requirement', 'requirements', 'requires', 'research', 'researchgate', 'respective', 'respectively', 'restricted', 'results', 'retrieval', 'retrieve', 'retrieved', 'review', 'reviewed', 'revision', 'rich', 'role', 'rouge', 'rough', 'rule', 'salience', 'salient', 'salim', 'same', 'sandeep', 'sankarasubramaniam', 'satisfactory', 'say', 'scale', 'scand', 'scandl', 'scheme', 'scholar', 'scia', 'science', 'scientific', 'score', 'scores', 'scoring', 'section', 'sections', 'see', 'seeking', 'seen', 'segments', 'select', 'selecting', 'selection', 'selectionbased', 'semantic', 'semantically', 'semantics', 'semistructured', 'sent', 'sentence', 'sentenced', 'sentences', 'sentimental', 'sequence', 'serve', 'sessions', 'set', 'sets', 'settled', 'seventh', 'shah', 'shanmugapriya', 'shape', 'sharing', 'shen', 'shorter', 'should', 'shown', 'shows', 'sigir', 'signal', 'significance', 'significant', 'signifying', 'sii1', 'similar', 'similarity', 'simple', 'since', 'single', 'singledocument', 'singular', 'size', 'sketch', 'slen', 'slow', 'software', 'solely', 'some', 'sophisticated', 'sort', 'source', 'sources', 'sowmya', 'space', 'specific', 'specified', 'speedy', 'sport', 'springer', 'sref', 'stage', 'stages', 'standard', 'stands', 'stanford', 'started', 'state', 'statistical', 'stats', 'stemming', 'step', 'steps', 'still', 'stock', 'stories', 'strongly', 'structure', 'structured', 'structures', 'style', 'suanmali', 'subject', 'subjective', 'submitted', 'successful', 'such', 'suggest', 'suitable', 'sukumaran', 'summaries', 'summarization', 'summarizationrelated', 'summarize', 'summarized', 'summarizer', 'summarizers', 'summarizes', 'summarizing', 'summary', 'summed', 'sun', 'supervised', 'supply', 'supporting', 'survey', 'surveyed', 'svd', 'svore', 'system', 'systems', 'table', 'tac', 'tajoddin', 'talk', 'tancyanbil', 'task', 'tdepartment', 'tech', 'technical', 'technique', 'techniques', 'technologies', 'technology', 'ten', 'tencon', 'term', 'termed', 'terms', 'test', 'text', 'texts', 'textual', 'tf', 'than', 'that', 'the', 'their', 'them', 'theme', 'then', 'there', 'therefore', 'these', 'they', 'third', 'this', 'those', 'though', 'thought', 'three', 'through', 'thus', 'time', 'times', 'tion', 'tipster', 'title', 'to', 'together', 'tokenisation', 'took', 'topic', 'topics', 'tough', 'towards', 'train', 'trainable', 'trained', 'training', 'trec', 'tremendous', 'trends', 'tsarev', 'tween', 'twin', 'two', 'types', 'udmale', 'understand', 'understanding', 'undirected', 'unicef', 'uniqueness', 'universities', 'university', 'unsupervised', 'up', 'uploaded', 'upper', 'uppercase', 'usa', 'usage', 'used', 'user', 'uses', 'using', 'usually', 'utilizes', 'value', 'values', 'vanderwende', 'variance', 'variation', 'variations', 'various', 'vector', 'vectors', 'verb', 'verbs', 'version', 'versions', 'very', 'vi', 'view', 'views', 'vii', 'vital', 'vol', 'volume', 'walked', 'wang', 'was', 'way', 'ways', 'wealth', 'web', 'weight', 'weighted', 'weights', 'went', 'what', 'when', 'whenever', 'where', 'whereas', 'which', 'while', 'wide', 'wikipedia', 'will', 'with', 'within', 'without', 'word', 'wordnet', 'words', 'work', 'working', 'works', 'would', 'www', 'xu', 'yang', 'years', 'yields']\n",
            "[[0 0 1 ... 0 0 0]\n",
            " [0 0 0 ... 0 0 0]\n",
            " [0 0 0 ... 0 0 0]\n",
            " ...\n",
            " [0 0 0 ... 0 0 0]\n",
            " [0 0 0 ... 0 0 0]\n",
            " [0 0 0 ... 0 0 0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFRdgsLL33j3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 144
        },
        "outputId": "19651ad5-5edc-44f7-80fb-b9e310d45751"
      },
      "source": [
        "# Tnormalized: document-term matrix normalized (value 0-1) according to the TF-IDF\n",
        "# TF(Term Frequency): the no. of times a term(a word here) appears in the current document(single sentence here)\n",
        "# IDF(Inverse Document Frequency): the no. of times a term(a word here) appears in the entire corpus\n",
        "# Corpus: set of all sentences\n",
        "\n",
        "normal_matrix = TfidfTransformer().fit_transform(cv_matrix)\n",
        "print(normal_matrix.toarray())"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.         0.         0.09391521 ... 0.         0.         0.        ]\n",
            " [0.         0.         0.         ... 0.         0.         0.        ]\n",
            " [0.         0.         0.         ... 0.         0.         0.        ]\n",
            " ...\n",
            " [0.         0.         0.         ... 0.         0.         0.        ]\n",
            " [0.         0.         0.         ... 0.         0.         0.        ]\n",
            " [0.         0.         0.         ... 0.         0.         0.        ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l9Qsme_Z33j9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "377de677-7559-4236-d8bb-12d1f2428213"
      },
      "source": [
        "print(normal_matrix.T.toarray)\n",
        "res_graph = normal_matrix * normal_matrix.T\n",
        "# plt.spy(res_graph)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method _cs_matrix.toarray of <1284x266 sparse matrix of type '<class 'numpy.float64'>'\n",
            "\twith 3878 stored elements in Compressed Sparse Column format>>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YJASW-oD33kC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "b4b49846-e6c6-4b85-eed2-aca029a5e1bc"
      },
      "source": [
        "# drawing a graph to proceed for the textrank algorithm\n",
        "# nx_graph is a graph developed using the networkx library\n",
        "# each node represents a sentence\n",
        "# an edge represents that they have words in common\n",
        "# the edge weight is the number of words that are common in both of the sentences(nodes)\n",
        "# nx.draw() method is used to draw the graph created\n",
        "\n",
        "nx_graph = nx.from_scipy_sparse_matrix(res_graph)\n",
        "nx.draw_circular(nx_graph)\n",
        "print('Number of edges {}'.format(nx_graph.number_of_edges()))\n",
        "print('Number of vertices {}'.format(nx_graph.number_of_nodes()))\n",
        "plt.show()\n",
        "print('The memory used by the graph in Bytes is: {}'.format(sys.getsizeof(nx_graph)))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of edges 16882\n",
            "Number of vertices 266\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAEuCAYAAADx63eqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3da5BV5Z3v8d9au3t301eaa5NgdBBQ\n4IhUHB3IqVG8RCHmZA6KkzcenUyS0TjGZKo0mjljpk6SOQejVckkZopMMuccLd+kvFWsQdSgAacq\nEJPMERxEESkSSEBufW96d++91nmx2bjT7m666edZ61lrfT9VeSGFz94g+sv///z/a3lhGIYCACAj\n/Li/AAAAUSL4AACZQvABADKF4AMAZArBBwDIFIIPAJApBB8AIFMIPgBAphB8AIBMIfgAAJlSF/cX\nAJLkeH9BT/36kN460qveoaLaGut03owmeZJ+e3JQvUNFNdT5OjVc0rS8r0IxrPlzzvXvq/VzLu5s\n0y2XzdfMloZ4f3OAhPB4VieyaLIBdqxvWL/rHtTR3oJyvqfhkjv/2jTU+SoGoea0NujD06dpfsc0\nAhMYB8GHTKgE3fb9J7T79z06OTAsT5JD+RW5WoFJGCILCD4k3ujqbaxKTcp20E1Ene8pCEPNaM5r\n2YfatHz+9A+0aAlHJB3Bh0SpDrlDXaecbT+m2ehwXLVgFkGIRCH44DRalO7LeZI8j5YpEoPgQ2xo\nUaYXLVO4jOBDJGhRYjRapogLwQerdh7s1ve37tO2vcckSYViEPM3gqvqPCmX87X6otm666qFuvS8\n6XF/JaQUwQcrjvcX9LfP7tLLe47SpsSk5Tzp2iVz9b/WXUIFCOMIPhgxegjlRP+w+IOFqfIkdbY3\nMjQDowg+TNpY93USQyiwi3tBmEDw4axYKYCrTm9SEISYFIIPY6oeTCkFoYoBf1TgNgZkMBEEHz6A\nwRSkAQMyGAvBBwZTkGoMyGA0gi+Dat3ZhaEIO2QCAzIg+DKEOzs7Kv8KeZ73Bz82+q9r/Zxz/fuq\n/xpTw71g9hB8GcCd3cRMJGSCkYI8P6dS/0kVe46W/9d9WKE81U/vlN/YrLBYkFfXqGCkIL8+r2Bo\n4AM/p9aPTeTvq/45dS0dyrXPUa5lhhSU5Nc3TOrXgg/iXjAbCL4Uq1R4r7x1lOquhrBUlDxPpcEe\nDR95V8OH9541iIaPHdDAG1sUnOqN++uf4U9rU/Ml1yk/+4IJhWp1YHphKK+uPu5fgnNynnTd0rlU\ngClF8KUQFd7EKjUXQyxKlcBs/MglyndeqFxTu8IwlJ+rO/NzalWOWUIFmE4EXwpkfSozKI7I87wz\noVYaOJmISs0141WODfMWjRmOWTGrhWGYtCD4EiyrwyqjW5RDv32DUItIdTjWtc/JZMuUYZjkI/gS\nKM2tTFqUyZPllmmd7+ne6xfrzqsWxv1VMAkEX4KkbVglDEMpDFTqO0GLMkUm1jKdLnleaoJwybxW\nbVi3nOovIQi+BEhbhVcJvMG9O3Tyxe8TahnkT2vTjBv+Wk2L/kTyc6kIQF/lSVAGYdxH8DksbRVe\nUByRFOrUu79U7/YnNXxkX9xfCTHLdy5U26pbNO3Cy6Uw/INdxKRiFcJ9BJ+D0lLhMYSCiUrj0Ayr\nEO4i+BxxvL+g7778jn6654iO9BQSt44weqWAIRRMVa2hGYWSl8vF/dUmjVUItxB8MUtqO7N8Txeq\nNNhNNYdIfDAIkzcgwyqEGwi+mCS1nclgClyR9AEZViHiQ/BFLKkVHoMpcFXSB2RYhYgewRehjdv2\n6ZGX9iYi8MKgJEkMpiAxkjwgwypEtAi+COw82K0Hnt2lPYf74v4qY+LODmmUtHtBViGiQfBZVLnH\n27LnqFwt8rizQ5Yk5V6Q+z+7CD4LknCPF4ahwqCkU/t+wZ0dMqdyL9i00O0A5P7PDoLPsCd2HNA/\nPP+WhkZKTu7iUeEB70tCBdiQ8/XgJ5fo1pUXxP1VUoPgM+iJHQf0zU1vaqjo3m8pU5nA2KonQz15\nzg3D0Po0i+Az5KlfHdRXntnlzF0ewyrA5FUPwzTMXyq/ocmpKnDlghn66poltD6niOAzYOO2fdrw\nwttxfw1JtDIBk1ovX6fpq2+T59c5E4BUf1NH8E2BS2sKDKsAdrg6CMPgy7kj+M6BS2sKVHhANFwc\nhMl50n03XET1N0kE3yS4tKZAhQfEw8UKkOpvcgi+CXLlcWNUeIAbXKsAqf4mjuA7C1fu8ajwADe5\ntgpB9Xd2BN84nthxQN/YtEeFYhDL559ZSeg7roG929X78x9T4QGOcmkVgqX38RF8Y4hzGZ12JpB8\nca9CNNb7+rtPEH61EHw1xLmMHoahht97VydfeJR2JpBwcQ/C+J70rZuXa/1l50X6ua4j+EaJaxk9\nDAMpKKlr6+Pq++WzkX8+AHvODMIsXil5fqQB6Em6fw1DL9UIvtPiGmJhaAXIjnznQs1Y80Xl5y6I\nvPpj6OV9BJ/KVd7DL76tUoS/E+Xf9lCDb2/nHg/ImLju/1h5KMt08MW5qhAGJZ3Y9B0N7P5Z5J8N\nIH75zoWavvozajx/OdVfxDIbfHGuKgQjBXW9/CP1v7458s8G4JbWy9dp+lW3yY94/y/LKw+ZDL64\nVhXCMFRYGlbXFkIPwPtaVqxVx3Wfl5erj7T6y+rKQ+aCL65VBdYUAIwnrsGXLK48ZCr4nthxQA/+\nZLei/AWzpgBgMsqDL7fLi3DvL2srD5kJvg2b9+if/21/ZJUeawoAzlVc1V9Whl4yEXwbNu/Rxlf3\nR/JZPG4MgClxVH9ZGHpJffBF2d7kHg+AaXFUf9Pqff33FA+9pDr4ohpkqbxFoetn/4d7PABWtF6+\nTh1Xf0byvEgCsKHO15N3rNLy+elre/pxfwFbnthxQPc9HdH0ZhjoxKZvE3oArOn75bM6sek7UhjN\n7nGhGOjep3ZG8llRS2XFF+UgC8voAKLUsmKtOq75rPx8YySft2hOsx5ZvyJVAy+pC76oBllYRgcQ\nl6gX3tM28JKq4ItqkIUhFgBxi3roJU0DL6kJvigGWcIwPL2M/hj3eQCcEOXKQ1oGXlIRfE/sOKCv\nPbfbeuhR5QFwUZTV39J5rXr+niutfoZtiQ++8gOn92jI0lsWWFUAkBRRrTx8dc1FuiPBjzdLdPDt\nPNitP//BdhVK9sZ7eW8egCRpXnaNZt74JXl+zurnPLI+uQ+2TvQe3/3P7LIeeidf2kjoAUiMgd2v\n6ORLGxUMD1n9nK88vUtP7Dhg9TNsSWzw/f1zb+itI/benB6GoXp2PM2qAoDE6X99s7pe+RcFxWHZ\nauoFofTgT3Zrw+Y9Vs63KZGtTtu7emFQUs+Op9Xz6uPWPgMAbMt3LtSMT3xZ+dnnW73z+8KVC3T/\n2iXWzjctccFnc1ev8maFky9tpNIDkBqdn/me8nMusBZ+vid9/VPLErPjl6hW51O/OqivPWdxQT0M\ndGLTdwg9AKlycvN3FZaGrZ0fhNLX//VN7TrUbe0zTEpM8D2x44DVBXUGWQCk1fCRd9S15UdWB16G\nS6EeeGaXtfNNSkTwVXb1rIUegywAUi6KgZc3D/fpB9vcf8CH83d8tnf1GGQBkCW2B15ynvTsXf/Z\n6ceaOV/x2drVC8PwTHuT0AOQFcNH9unI/75bw0cPWKn8SqGcb3k6HXwbt+2zt6sXhgyyAMgsmwMv\nrrc8nQ2+nQe79fCLb1s5Ozz97E0GWQBk1ZmBl5GClfO/9eLbzk55Oht89z+zSyULt4/ltyzs54HT\nADKv//XN6nr5RwqDkvG2p8stTyeDz2aLMyyN6OQL37NyNgAkTf/rm3Vi0z9KoflZCldbns4Fn9UW\nZ6mkri0/5H16AFCl8mDrMCgZP9vFlqdzwWezxdnzC3b1AKCW/tc3q2fHM5loeToVfLZanGFQUs/2\nJ1lbAIBx9Lz6mHp/9Zzx8HvzcJ/+/if/YfTMqXAm+HYe7NZDL5htcbKrBwCT0/3yDzX83n7j4ffY\njt/oIUdeYeRM8N3/zC7zD59mVw8AJu3kC49KFu77fvBv+514ea0Twbdt71HjLU529QDg3AwfeUdd\n2x43XvUFofTNTXtiH3ZxIvjufXKn0fPY1QOAqel77RkrLc+hYqB/2hrvZH3swbdx2z4d6zf72Jww\nKLGrBwBTZKvluWXPUZ3ot/PEmImINfhs7OyFYajurY+xqwcAU2Sr5VkMQn3vlXeMnjkZsQaf6Z09\nWpwAYJatluez/+93Rs+bjNiCz8bOHi1OADDPRsuzZ6gY2+PMYgm+nQe79chLe42eSYsTAOyw1fJ8\n+KW9sUx4xhJ839+6T8XA3G9gGIYq9Z2gxQkAlthoeRaDMJYJz8iD73h/QS/vec/4uSc2fdv4mQCA\n95184VGFpRGjZ8Yx4Rl58H3vlXesDLQM/cbsLiAA4A+VX177Q+NVX9QTnpEH30tvGq72goCBFgCI\nSP/rmzX0m11Gwy/qCc9Ig+94f0GHe4aMnReGobq2/l8GWgAgQsef+5bRKc+oJzwjDT6T5Sw7ewAQ\nj2CwR4P7XlNo8K3tUU54Rhp8JstZdvYAID6925+UAnPBF+WEZ2TBt3HbPvUOFY2cxc4eAMTLxm5f\nVBOekQSf6YX1YKifFicAxKzvtWcUDJl7AldUE56RBJ/JhfUwDDXwH68YOQsAMDUDu7carfqMT/7X\nYD34jC+sh6F6fv5jc+cBAM5Zz89/bHTC83DPkPV2p/XgM72wXuw7ruBUr7kDAQDnzPSEZyizGwC1\nWA8+k2VrGIY6tXe7sfMAAFNnesLTdrvTavCZXlhXUKLNCQCOMT3habvdaTX4TC+sD+77BW1OAHCQ\nyQlP2+1Oq8FntM0ZFMvlNADASSYnPG22O60Fn8k2Z3lh/XEW1gHAYT0//7GUgHanteAzWaaysA4A\n7gsGe1TqP27kLJvtTmvBZ6pMZWEdAJJjcO8O59udVoLveH9B7/UamuZkYR0AEiMJ7U4rwffUrw+Z\n+nWzsA4ACZKEdqeV4HvrSK9M5B4L6wCQPK63Oy21OofNHESbEwASx/V2p5Xg+33XoJFzgsIAbU4A\nSBjX251Wgu+YoYqvNBDNa+gBAGa53O40HnzH+wvqK5h503qp96iRcwAA0TLZ7jzSa7bdaTz4TJak\npYEeY2cBAKJjst2pUHrq3w+ZOUsWgs/Y4npQ0vCxA0bOAgBEz1S7M5D01mEzD8CWDAef0cV1SQNv\nbDF2FgAgWibbncddbXWaXVw/wUQnACRYMNijYNjMlP/vuk8ZOUcyHHwsrgMAqpX6Txo551ifuW6i\n4VYni+sAgPcVe48ZOae/UDI22Wk0+LoGzHwpFtcBIB2CQTPT+Z5nbrLTaPANDgdGzmFxHQDSYeTY\nAYVBacrnBKG5yU6jwdc7NGLkHBbXASAd+t94WZJn5CxTGWMs+I73F9Q1aOpRZSyuA0AaBIM9Kg2Y\nGXBpqDMTWcaC76lfHzKS6UGpyOI6AKRIsfuokUX2UyNTb5lKBoPvrSO9KhnYZfA8j8V1AEiRUv8J\ned7US6Np9TkD38Zoq9NQm3Owl4lOAEgRv6HZyDl9Q2ZegGAs+IytMgyZex4bACB+flObkXNMPb3F\nWPCFhqZ2whGzb9oFAMTM0LMsTT29xVjwtTbUGTknKJh5rhsAwA0lQ9dXpp7eYiz4+guG9isamoyc\nAwBwg7Gnt8jM01uca3UCANLF2NNbZObpLbQ6AQBWmXx6i4n38tHqBABYZfK9fCaeEEarEwBgnan3\n8pkYEKXVCQCwrtjfZeSclsapZw2tTgCAdXUtM4yc02/g6S20OgEA1uVaOoycY+CRn7Q6AQB2+U3t\n8vPTjJzV0ZSf8hm0OgEAVrVccq2xs2a1NEz5DFqdAACr6mdfIM+f+iuFfE+6eF7r1M+Z8gmn0eoE\nANSSa5pu5JxQ0vqPzp/yOc61Ok1dgAIA3GDqtUStDXWamcZWZ67ZzMgrAMARhl5LNKt56qEnGQy+\nmc1Tn7SRysMt/jQz/+8AABC/oGDmBbKz21wLvhYzwSeFar7kOkNnAQDiZuoKa6Aw9eV1yWDwXdzZ\nZqTZ6fk55WdfYOAkAIALcs2GhlvMdEzNBd/6y+bL0HeS39xu6CQAQJz8pnZj+9kzTF2pGTlF5aXC\n1oap72lIUl3bbCPnAADi5dryumQw+CRpVkujkXOY7ASAdDC1vO7JzPK6ZDj45neYeRYbk50AkA6m\nltflmVlelwwHn7HJTs9T28c+beYsAEBscoaurkwtr0uGg8/YZKfnqXnxKgMnAQDiZGqi09TyumQ4\n+NZfNl+mnlWda51FuxMAEqw80dls5KwPG7pKk4wPtzSo09BmPYvsAJBs7R/7tJk3x8rcRKdkOPgk\n6eNLOo2cwyI7ACRb06KV8gwEn8mJTslC8N1z7SJjZ7HIDgDJ5De1K9c6y8hZnsGJTslC8LHIDgAw\n2ebsbGs0NtEpWQg+iUV2AMg6U21OSbp+2Vwj51RYCT4W2QEgu0y2OSXpi1ebu0KTLAUfi+wAkF0m\n25wfajfb5pQsBR+L7ACQXS63OSVLwcciOwBkk+ttTsnacIvBRXbanQCQGK63OSVLwScZXGT3PLUs\nu9rIWQAAu1xvc0oWg++eaxeZ6nbKn9aq1svXGToNAGBDEtqcksXgm9XSoM52MyWq53nqWH2b8p0L\njZwHADAvCW1OyWLwSebanZIkP6e2VbeYOw8AYFTzsqudb3NKloPPZLvT83xNW/gnTHgCgINar7hJ\nfmOLsfNstTkly8Fnst0pld/YwIQnALgl37lI06/6b8aqPZttTsly8Elm250stAOAe9pW3SLPrzN2\nns02pxRB8N1z7SLV+aYaniy0A4BL/KZ2NS28wli1J9ltc0oRBN+slgZdc/Eccwey0A4Azmj/2Kcl\n38yr6CT7bU4pguCTpL9evdBY1cdCOwC4w+Qkp2S/zSlFFHyXnjdd916/2Nh5LLQDQPxMT3LmPPtt\nTimi4JOkO69aqLZGM5ef5YX221loB4CYmJ7klKSPL51rvc0pRRh8kvRfV3zY3GF+TjPW3G3uPADA\nhJme5KzzPd21OppiJtLgM7vQ7ik/90JangAQMRuTnPdev1jL5083dt54Ig0+4wvtPMMTACJnepKz\nvbFOd14V3X/HIw0+yfDzOyWe4QkAETM9ybnuowavwSYg8uAzvdDOMzwBIDpJneSsFnnwGV9oF8/w\nBIAolCc5b0vkJGe1yINPMrvQLrHUDgBRmLH2bnkG7/ainOSsFkvwmV5ol1hqBwCbWq+4Sfk5CxI7\nyVktluCTygvtS+a1GjuPpXYAsMNGizPqSc5qsQWfJG1Yt1w5c7+PLLUDgAWmW5xS9JOc1WINvkvP\nm677brjI2HmVpfbp13ze2JkAkGXTr/0r4y3OOCY5q8UafJLZZ3hK5fBru/xTar/yNmNnAkAWtV95\nu9r++L8YDT0pnknOarEHn2T4GZ4qh1/7ypvVsmKt0XMBICtaVqxV+8qbjIdeXJOc1ZwIPtNL7VJ5\nt2/G9XeqmTUHAJiUfOcidVz3eeP3elJ8k5zVnAg+G0vtkiTP18wbv0zlBwCTMGPt3fJy9cbPXTqv\nNbZJzmpOBJ9kfqldKrc8PT+njms/R/gBwATY2NeTygMtG25abvTMc+VM8NlYaq/w6xvUcd3n2PED\ngHHY2NeruO+Gi2JvcVY4E3yS+aX2al4uz44fAIxj5ie+ZOVez5UWZ4VTwSdZWGo/jR0/AKgt37lI\nnX/5qOpnn5/qFmeFc8Fneqm9Gjt+APCHWlas1dxbNyhvIfQkt1qcFc4Fn2S55cmOHwBIKodex7Wf\nlV/XYCX0XGtxVjgZfFK55Zm30fMUO34AUNnV8+sb7Zyf85xrcVY4G3yXnjddX/vkUhnecHgfO34A\nMszWrp4k+Z70tU8uda7FWeFs8EnSrSsv0F/96QIrZ7PjByCrbO3qVdxx5QLduvICK2eb4HTwSdID\na5foC1ctkK3Cjx0/AFnSvOwadaz+C2uhd/uq83X/miVWzjbF+eCTpPvXLNE3/myZtbanl8trxtp7\n7BwOAI5oWbFWM2/8kmQp9JbOa9X/+NR/snK2SYkIPqnc9nz4ZjsXpZ7nKT/nj9T5l48q3xnfO6IA\nwJb2K2/XjOvvlOfnrFR7Lu7rjSUxwSdJN192nh5YY2/HLz/7fM299SHu/ACkSvuVt6t91XorT2Wp\ncHFfbyyJCj7J/o6fX5dXxzWfJfwApIKt9+pVc3VfbyyJCz7J7o6fJPn5RgZeACRe87JrzrQ3bWnI\n+YlpcVYkMvis7/iJgRcAydZ6xU2aeeOXJc/ef+an1ft68JNLEtPirPDCMAzj/hLnasPmPdr46n5r\n54dhqOFjv9HJ5/9Rw0fesfY5AGBKvnORZqy5W/m59vb0JKmhzteDNy5xel9vLIkOPkl66IU92rht\nv2z9IsIwVFgaUdeWH6r/9c2WPgUApq5lxVp1XPc5ebm81dBbPLdFj6y/NHGVXkXig0+SnthxQF97\nbrcCi7+SYHhIXa/8C+EHwEnlB05/Tn59g9XPWTynWS/9zWqrn2FbIu/4RrO541fh5xt5sDUAJ1WG\nWGyHXkPO1yO3rLD6GVFIRfBJdnf8zvB8zbzxb9R6+Tq7nwMAExTFEIskNSZ0kKWWVLQ6q6397qva\nc7jP6meEYajh9/br5AuPMvQCIBZRDbFI5bctfP1TyxI5yFJL6oJv58Fu/fk/b1ehGFj9nDAMFQYl\ndW99TH2/fNbqZwFAtdYrblLHVbdJlh4/Vs33pIfXL9fNHz3P6udEKXXBJ5WHXf7h+T06NWI3/CSq\nPwDRibLKkyRP0jf+LD2VXkUqg08qh983/nWPCqVowo+VBwA2RbWqUO0LVy1w/hVD5yK1wSdJuw51\n694nX9feowORfB4rDwBsiGpVocL3pDv+dIHuX5u+0JNSHnwVH//2Nr1ztD+SzwqDkk5s+o4Gdv8s\nks8DkG7Ny67RzBu/ZPV5m9XS2t6slongi2rgRSq3PaVQg2/v0MkXH1Vwqtf6ZwJIp9YrblLH6r+Q\nPC+S9mYaB1lqyUTwSdEOvEinAzAoaXDfa+rd/iSDLwAmLOohFklqrPP0dzcuTXWlV5GZ4JOiHXip\nKK89FNW99XHWHgCcVZSrChVJfuD0uchU8EnlgZcHntmlNy0vuY/G2gOA8cRR5Unll8huuGl5Kp7I\nMlGZC76Kjdv26aEX3rb2VodaWHoHMJrf1K4ZN9ylpkUrJc+PNPS+uuYi3ZGgN6ebktngk6Snf31Q\n9z29y+pbHWqh+gOQ71yktlW3qGnhFZG2NaXsDLGMJdPBJ5Xv/b75/B4NRTT0UhGGoRQGGtz7C6Y/\ngYx5/x7Pl2f54dKjZWmIZSyZDz4pnqGXCqY/geyI6x6vImtDLGMh+E6La+ilgulPIL3ivMeryOIQ\ny1gIvlE2btunh198W6WYfle4/wPSI857vIqcJ33lhmwOsYyF4KvBheqP+z8g2eK8x6ugyquN4BtH\nHCsP1bj/A5In7nu8iqyuKkwEwXcWca08VOP+D3CfC/d4EqsKE0HwTUBcKw+jcf8HuMeFe7wKVhUm\nhuCboPJDrt/SqZFSrN+D+z/ADa5UeBWsKkwcwTcJuw5165+27tNP33wvtqnPCu7/gHi4VOFVMMQy\nOQTfOTjRX9DfPvuGfvrme4q3+VmpAEsa3PsaFSBgkWsVnsSqwrki+KYg7rWHauUADFUa7Nbwe/tV\n+O0u9e/aQhACU+A3tav9Y59W0+KVyrXMiuyFsBNBlXfuCD4D4l56ryUsjihUqFPv/opWKDBJLrYz\nK+p8T/ddv5gqbwoIPkNcqv6q0QoFJs7FdmZFne/puiVzdNfqhVR5U0TwGbZx2z498tJeFeNc/KuB\nYRhgbC5XeL4nfXzJXP3PdZdoZktD3F8nFQg+C1ya/hyNChB4n8sVnsQ9ni0En0UuTX+OxjAMsshv\nalfLJdeq4SOXKD/3QuWapjs1sFLBPZ5dBF8EXL3/q8YwDNLsTCvzwj9WKE9+XX3cX6km7vGiQfBF\nyNX7v2q0QpEmrrcyK7jHixbBFzGX7/+q0QpF0lTamPWzL1Bd2xzVTZ+jXMtMpwNP4h4vDgRfTFy+\n/6slLJUkTyoN9hCEcMIH7+vapTCUl6uL+6tNCPd48SH4YpaE+79auBNEXJJyXzcW7vHiR/A5Ign3\nf7VwJ4ioJOW+biw5T7qOezwnEHwOScr9Xy2V1yWV+k6o2HNUxd6jGjl2gHYozllSVg/OhgrPPQSf\ngyr3f1v2HFUpwf94glJRnudxL4izqjmY0jxDYRgmrpVZQYXnLoLPYUmuAGthQAYVSR9MGQ8VnvsI\nvgRISwU4WlgcUeh5CvpP0h5NuZpBJ8nzczF/M3Oo8JKD4EuQSgX4s7ePqVQKVEzhPznao8mXxrbl\neKjwkofgS6AT/QU99e+HtP3dE9r9+x4d6x+O+ytZQ1XovjS3LcfiSZrX3qjrl87VF69ZRIWXMARf\nCqS1FTqW0VXh8OG9UhiqrmOe/IZmBYUBwtGQ6urNb2hWWByWV9+ocGRIueaO1Fdzo9HOTAeCL0XS\nNgwzUZU/wtVj7rRMJ2/MFmVQkl///n/kwzBM3ErBVNHOTBeCL4WyVgFOxOiWaWmg60zl4tXlU18l\njq7cgsKAit1HpDBU/kOLM9OinCwqvHQi+FIsC8MwUzG6cglGCpKfGzccK2FR3Vat9WOmQnS8VuNE\nvtN4oVarUka5usv5nq6+aDYVXkoRfBmQpWEY00aHY62wmEirtdY95OjAGvP+7Cytxol+J4zNkzSr\nJa9lH2rXqgtnav1H51PhpRjBl0G0QqM30XDK4v1ZnGhlZhPBl2G0QpFFjXW+QolWZoYRfPhAK/TE\nwLDCUOIPBpIud7p4ntvWqA9Pn6b5HU26eF4rrcyMI/jwAdwJIsm4r8PZEHw4K+4EkQTc12GiCD5M\nWPWdoCdpqBjE/ZWQcawe4FwQfJi0Siv0rcN9OtQ1qN91n9J7fQUpDDP1xBhEL+eV755nNtPKxLkj\n+GAEAzIwjcEU2ELwwYpaQeh7nooBf9xQmyfJ86jmYB/Bh0jQHsVotC0RF4IPsaEqzA7alnAJwQdn\nUBWmB9UcXEbwwWlUhe5ryKpxcPUAAADXSURBVJX/eVDNISkIPiRKdVXYOzSitsZ6fWTGNEnSzkM9\nhKNho1uUs1sbdGqkpGn1ORWKgdoa6wk5JA7Bh9QZq2Va53sqsHQ/LlqUyAKCD5kwulJsqPPPVC7H\n+gqZCMfRoXbp/HZJ0m9PnjpTPVO9IQsIPuC08cKx0tartFWrw8Jmq3X0/VmtVuN434lQAz6I4AMM\nG+8ecrxwqhW0hBVgHsEHAMgUP+4vAABAlAg+AECmEHwAgEwh+AAAmULwAQAyheADAGQKwQcAyBSC\nDwCQKQQfACBT/j91zT52DauSEQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The memory used by the graph in Bytes is: 56\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBXi5xcm33kJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#  note that the graph above is dense and therefor it resembles a circle\n",
        "# if a shorter document is taken, a beautiful circular graph can be seen "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T5cuEYc033kP",
        "colab_type": "text"
      },
      "source": [
        "### 8. Getting the rank of every sentence using textrank"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EtlXHoZS33kP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "039d25b6-5fc3-4410-8125-be2cf4bc24ad"
      },
      "source": [
        "# ranks is a dictionary with key=node(sentences) and value=textrank (the rank of each of the sentences)\n",
        "ranks = nx.pagerank(nx_graph)\n",
        "\n",
        "# analyse the data type of ranks\n",
        "print(type(ranks))\n",
        "print('The size used by the dictionary in Bytes is: {}'.format(sys.getsizeof(ranks)))\n",
        "\n",
        "# print the dictionary\n",
        "for i in ranks:\n",
        "    print(i, ranks[i])\n"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'dict'>\n",
            "The size used by the dictionary in Bytes is: 9320\n",
            "0 0.0035299969616808284\n",
            "1 0.0033717558603198836\n",
            "2 0.004655642837456238\n",
            "3 0.004920943518324018\n",
            "4 0.005906130798249928\n",
            "5 0.004423774622355444\n",
            "6 0.0044689056181346914\n",
            "7 0.0052308125794850585\n",
            "8 0.004138189569552129\n",
            "9 0.004394162767172467\n",
            "10 0.0036068354793937956\n",
            "11 0.005409612050655677\n",
            "12 0.0033213017340794977\n",
            "13 0.00391684191737175\n",
            "14 0.002149772294369349\n",
            "15 0.004224965908712696\n",
            "16 0.002975868184396672\n",
            "17 0.003711313683407367\n",
            "18 0.002599564266003766\n",
            "19 0.0030546976135088205\n",
            "20 0.005129699470647373\n",
            "21 0.005346867557068253\n",
            "22 0.004088511317481875\n",
            "23 0.004266500449287019\n",
            "24 0.003877958836369414\n",
            "25 0.005724457744379733\n",
            "26 0.0034240709550260374\n",
            "27 0.0023925471658529183\n",
            "28 0.0039893738038417215\n",
            "29 0.0018524313083226602\n",
            "30 0.005292873135080768\n",
            "31 0.0040573761429255435\n",
            "32 0.005100045500163795\n",
            "33 0.004816777107578484\n",
            "34 0.005353073447217089\n",
            "35 0.003935541659104468\n",
            "36 0.006639593760492873\n",
            "37 0.0058652789688904555\n",
            "38 0.0031246989607352635\n",
            "39 0.002056751859546224\n",
            "40 0.004713173453429848\n",
            "41 0.003937734262574678\n",
            "42 0.004746714410150624\n",
            "43 0.004827182045105384\n",
            "44 0.006505350693726728\n",
            "45 0.0005963029218843173\n",
            "46 0.007223113628007046\n",
            "47 0.003790297240560326\n",
            "48 0.003916274539992649\n",
            "49 0.007911754919798864\n",
            "50 0.0024473657811740763\n",
            "51 0.00515672940209403\n",
            "52 0.005469750019921675\n",
            "53 0.002476177311095907\n",
            "54 0.004609109170684494\n",
            "55 0.0036331184923462565\n",
            "56 0.002835064403381015\n",
            "57 0.0005963029218843173\n",
            "58 0.004848417704494141\n",
            "59 0.006127905905679062\n",
            "60 0.003692815968927817\n",
            "61 0.003260930283262805\n",
            "62 0.0018452003011959594\n",
            "63 0.0005963029218843173\n",
            "64 0.0031327196117034504\n",
            "65 0.004101174138452965\n",
            "66 0.003415498075711529\n",
            "67 0.004499430902005435\n",
            "68 0.0052120951070874786\n",
            "69 0.005358687265054034\n",
            "70 0.005724999660499136\n",
            "71 0.005247870868029912\n",
            "72 0.0032384776865104843\n",
            "73 0.004254850894738354\n",
            "74 0.0005963029218843173\n",
            "75 0.002894857949640477\n",
            "76 0.004862884828242823\n",
            "77 0.005900713088031841\n",
            "78 0.0005963029218843173\n",
            "79 0.005757806020514685\n",
            "80 0.005736774097299159\n",
            "81 0.004483064002707279\n",
            "82 0.0005963029218843173\n",
            "83 0.004729461710293424\n",
            "84 0.006025576037197665\n",
            "85 0.005343424061706735\n",
            "86 0.0021917242256513578\n",
            "87 0.005668451599425489\n",
            "88 0.003693132817477676\n",
            "89 0.0032925602312217995\n",
            "90 0.0037022643086191873\n",
            "91 0.004333015409460195\n",
            "92 0.0005963029218843173\n",
            "93 0.0048168324381279715\n",
            "94 0.0005963029218843173\n",
            "95 0.004250087291500136\n",
            "96 0.004635985427893599\n",
            "97 0.006891312367526255\n",
            "98 0.004474595496311073\n",
            "99 0.003703465726784949\n",
            "100 0.005830105675444339\n",
            "101 0.004642268497936911\n",
            "102 0.004546296446638604\n",
            "103 0.0033910289855974642\n",
            "104 0.004085614759676287\n",
            "105 0.004194035371476511\n",
            "106 0.004879813647861621\n",
            "107 0.005075953931917514\n",
            "108 0.004307581730666764\n",
            "109 0.002733636054944165\n",
            "110 0.0030160154618964776\n",
            "111 0.0049005952613938464\n",
            "112 0.004506846745402283\n",
            "113 0.005443069024911237\n",
            "114 0.003063220423717455\n",
            "115 0.005342260782788124\n",
            "116 0.006103264272865159\n",
            "117 0.002835064403381015\n",
            "118 0.0005963029218843173\n",
            "119 0.0030156609676756105\n",
            "120 0.0005963029218843173\n",
            "121 0.002980030610469879\n",
            "122 0.004929151963167811\n",
            "123 0.0045430602806722006\n",
            "124 0.006851877751274347\n",
            "125 0.007129919278014332\n",
            "126 0.007600993429744846\n",
            "127 0.0005963029218843173\n",
            "128 0.003093073181901295\n",
            "129 0.0005963029218843173\n",
            "130 0.00543385372848868\n",
            "131 0.003425316582810672\n",
            "132 0.0066910960630705415\n",
            "133 0.004743122665035255\n",
            "134 0.005510210920125782\n",
            "135 0.005246985757170013\n",
            "136 0.003228870688029603\n",
            "137 0.005853515581785317\n",
            "138 0.005453139921831883\n",
            "139 0.004900173796257148\n",
            "140 0.0005963029218843173\n",
            "141 0.0025364174766494196\n",
            "142 0.005224808341822369\n",
            "143 0.0054597202921985374\n",
            "144 0.004257369652145\n",
            "145 0.005558586627431149\n",
            "146 0.002476177311095907\n",
            "147 0.0033602882156845465\n",
            "148 0.003283334234481211\n",
            "149 0.002917175038051204\n",
            "150 0.0034247350635705637\n",
            "151 0.004600449014393239\n",
            "152 0.0028036925323871066\n",
            "153 0.005517820139204584\n",
            "154 0.0030111933374663245\n",
            "155 0.0020186619371596773\n",
            "156 0.002917307625708671\n",
            "157 0.003193621869584556\n",
            "158 0.004289165565358159\n",
            "159 0.0022807069164697056\n",
            "160 0.0031690552763140984\n",
            "161 0.005131585384414078\n",
            "162 0.0029227139993108853\n",
            "163 0.004040497356245457\n",
            "164 0.0021135659291359726\n",
            "165 0.0034068110139734045\n",
            "166 0.004014557698096177\n",
            "167 0.005246812782949882\n",
            "168 0.0034725302598578164\n",
            "169 0.002809575698102568\n",
            "170 0.003918942754843001\n",
            "171 0.002322603249317632\n",
            "172 0.0025886299935204736\n",
            "173 0.005475671617624084\n",
            "174 0.004979023808604784\n",
            "175 0.0020834552620697896\n",
            "176 0.0019954315511082404\n",
            "177 0.002861522321651767\n",
            "178 0.002464779296224248\n",
            "179 0.004539003568626861\n",
            "180 0.0035303250294575913\n",
            "181 0.004131509553178491\n",
            "182 0.003975352812562114\n",
            "183 0.003268738320674704\n",
            "184 0.0036880899144269667\n",
            "185 0.00520820514227591\n",
            "186 0.002376899861998483\n",
            "187 0.002648645829112554\n",
            "188 0.002438920848793066\n",
            "189 0.0031176852081783494\n",
            "190 0.005359782581739572\n",
            "191 0.003776029899679076\n",
            "192 0.003614399916752277\n",
            "193 0.0005963029218843173\n",
            "194 0.005197448749422049\n",
            "195 0.003616846859316757\n",
            "196 0.001622944244899785\n",
            "197 0.002681444805031302\n",
            "198 0.0005963029218843173\n",
            "199 0.003878733348313567\n",
            "200 0.003975352812562114\n",
            "201 0.0032912464275434016\n",
            "202 0.0031805612526318826\n",
            "203 0.004489761506996552\n",
            "204 0.002256349232413582\n",
            "205 0.003118597253502066\n",
            "206 0.0031805612526318826\n",
            "207 0.004489761506996552\n",
            "208 0.003975352812562114\n",
            "209 0.004476383008039152\n",
            "210 0.002775939914780692\n",
            "211 0.002440623333200445\n",
            "212 0.003975352812562114\n",
            "213 0.00205481297141211\n",
            "214 0.0039753528125621145\n",
            "215 0.0039753528125621145\n",
            "216 0.0015951466311979678\n",
            "217 0.004073193881250137\n",
            "218 0.003463194420759863\n",
            "219 0.003975352812562114\n",
            "220 0.003959164659310291\n",
            "221 0.003975352812562114\n",
            "222 0.002470235197969012\n",
            "223 0.0005963029218843173\n",
            "224 0.0037327694572027257\n",
            "225 0.003975352812562114\n",
            "226 0.002780453906607067\n",
            "227 0.004441732598910898\n",
            "228 0.0018991540512945798\n",
            "229 0.0035037955485447275\n",
            "230 0.0037349174395513007\n",
            "231 0.0044897615069965525\n",
            "232 0.0024701375938452974\n",
            "233 0.0023195800569304065\n",
            "234 0.0032634987021973798\n",
            "235 0.003734917439551301\n",
            "236 0.004489761506996552\n",
            "237 0.002470137593845297\n",
            "238 0.004054007662264586\n",
            "239 0.003848304451709631\n",
            "240 0.0044897615069965525\n",
            "241 0.003975352812562114\n",
            "242 0.003802764370633314\n",
            "243 0.002861253411400545\n",
            "244 0.003975352812562114\n",
            "245 0.003069604933499586\n",
            "246 0.002270148265013172\n",
            "247 0.0030956670057615417\n",
            "248 0.0028304012335268096\n",
            "249 0.004257976775868553\n",
            "250 0.003617556499298102\n",
            "251 0.0020417603286681392\n",
            "252 0.002616843774736992\n",
            "253 0.0022873110197895023\n",
            "254 0.003241017627583549\n",
            "255 0.003975352812562114\n",
            "256 0.005075771220753901\n",
            "257 0.003116096421242289\n",
            "258 0.001985161256478637\n",
            "259 0.0037916926114320787\n",
            "260 0.0025850290158498687\n",
            "261 0.003975352812562114\n",
            "262 0.003983883092837713\n",
            "263 0.0023056432561534956\n",
            "264 0.0005963029218843173\n",
            "265 0.0020445927002342296\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iTJg-n6r33kV",
        "colab_type": "text"
      },
      "source": [
        "### 9. Finding important sentences and generating summary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oIkElh4g33kW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# enumerate method: returns an enumerate object\n",
        "# Use of list Comprehensions\n",
        "# O/p: sentence_array is the sorted(descending order w.r.t. score value) 2-d array of ranks[sentence] and sentence \n",
        "# For example, if there are two sentences: S1 (with a score of S1 = s1) and S2 with score s2, with s2>s1\n",
        "# then sentence_array is [[s2, S2], [s1, S1]]\n",
        "sentence_array = sorted(((ranks[i], s) for i, s in enumerate(sentences_list)), reverse=True)\n",
        "sentence_array = np.asarray(sentence_array)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B3AuIeIQ33kc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# as sentence_array is in descending order wrt score value\n",
        "# fmax is the largest score value(the score of first element)\n",
        "# fmin is the smallest score value(the score of last element)\n",
        "\n",
        "rank_max = float(sentence_array[0][0])\n",
        "rank_min = float(sentence_array[len(sentence_array) - 1][0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HwsxIsHC33ki",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "4504f8fd-4ee1-4fad-b576-8ad85aaf0faf"
      },
      "source": [
        "# print the largest and smallest value of scores of the sentence\n",
        "print(rank_max)\n",
        "print(rank_min)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.007911754919798864\n",
            "0.0005963029218843173\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yK0TH2Ui33ks",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "48e1a6ca-e9a8-438f-b548-f11489ff8046"
      },
      "source": [
        "# Normalization of the scores\n",
        "# so that it comes out in the range 0-1\n",
        "# fmax becomes 1\n",
        "# fmin becomes 0\n",
        "# store the normalized values in the list temp_array\n",
        "\n",
        "temp_array = []\n",
        "\n",
        "# if all sentences have equal ranks, means they are all the same\n",
        "# taking any sentence will give the summary, say the first sentence\n",
        "flag = 0\n",
        "if rank_max - rank_min == 0:\n",
        "    temp_array.append(0)\n",
        "    flag = 1\n",
        "\n",
        "# If the sentence has different ranks\n",
        "if flag != 1:\n",
        "    for i in range(0, len(sentence_array)):\n",
        "        temp_array.append((float(sentence_array[i][0]) - rank_min) / (rank_max - rank_min))\n",
        "\n",
        "print(len(temp_array))\n"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "266\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P7mLTKs333kx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculation of threshold:\n",
        "# We take the mean value of normalized scores\n",
        "# any sentence with the normalized score 0.2 more than the mean value is considered to be \n",
        "threshold = (sum(temp_array) / len(temp_array)) + 0.2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2UFgra1T33k2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Separate out the sentences that satiasfy the criteria of having a score above the threshold\n",
        "sentence_list = []\n",
        "if len(temp_array) > 1:\n",
        "    for i in range(0, len(temp_array)):\n",
        "        if temp_array[i] > threshold:\n",
        "                sentence_list.append(sentence_array[i][1])\n",
        "else:\n",
        "    sentence_list.append(sentence_array[0][1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zsa6bFzW33k6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = sentence_list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "erQ-e_fl33k_",
        "colab_type": "text"
      },
      "source": [
        "### 10. Writing the summary to a new file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "BihmgJHI33lB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ec75c45d-b59c-4369-a59c-b1109d0fdc05"
      },
      "source": [
        "# print(sentence_list)\n",
        "summary = \" \".join(str(x) for x in sentence_list)\n",
        "print(summary)\n",
        "# save the data in another file, names sum.txt\n",
        "f = open('final3.txt', 'a+')\n",
        "#print(type(f))\n",
        "f.write('\\n')\n",
        "f.write(summary)\n",
        "f.close\n"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The normalized length of the sentence is calculated\n",
            "as the ratio between a number of words in the sentence to the\n",
            "number of words in the longest sentence in the document. P (s E< SII1,h,h, .... In)\n",
            "represents the probability of the sentences to be included in\n",
            "the summary based on the given features possessed by the\n",
            "sentence. SENTENCE LEVEL FEATURES\n",
            "2.1 Sentence location feature\n",
            "The sentences that occur in the beginning and the conclusion\n",
            "part of the document are most likely important since most\n",
            "documents are hierarchically structured with important information\n",
            "in the beginning and the end of the paragraphs. The probability of classification are learned from the\n",
            "training data by the following Bayes rule [16]: where s represents\n",
            "the set of sentences in the document and fi represents\n",
            "the features used in classification stage and S represents the\n",
            "set of sentences in the summary. The basic steps in concept\n",
            "based summarization are: i) Retrieve concepts of a text from\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "external knowledge base(HowNet, WordNet, Wikipedia) ii)\n",
            "Build a conceptual vector or graph model to depict relationship\n",
            "between concept and sentences iii) Apply ranking algorithm to\n",
            "score sentences iv) Generate summaries based on the ranking\n",
            "scores of sentences\n",
            "4. The sentences are restricted as a non-summary\n",
            "and summary sentence based on the feature possessed by the\n",
            "sentence. The first\n",
            "step involves labeling the training data using a machinelearning\n",
            "approach and then extract features of the sentences\n",
            "in both test set and train sets which is then inputted to the\n",
            "neural network system to rank the sentences in the document. 1.2 Title Word feature\n",
            "The sentences in the original document which consists of\n",
            "words mentioned in the title have greater chances to contribute\n",
            "to the final summary since they serve as indicators of the theme\n",
            "of the document. 1.5 Upper case word feature\n",
            "The words which are in uppercase such as \"UNICEF\" are\n",
            "considered to be important words and those sentences that\n",
            "consist of these words are termed important in the context of\n",
            "sentence selection for the final summary. The unsupervised approaches do not\n",
            "need human summaries (user input) in deciding the important\n",
            "features of the document, it requires the most sophisticated\n",
            "algorithm to provide compensation for the lack of human\n",
            "knowledge. The major drawback with the\n",
            "supervised approach is that it requires known manually created\n",
            "summaries by a human to label the sentences in the original\n",
            "training document enclosed with \"summary sentence\" or \"nonsummary\n",
            "sentence\" and it also requires more labeled training\n",
            "data for classification. In the methodology proposed [12], the\n",
            "importance of sentences is calculated based on the concepts\n",
            "retrieved from HowNet instead of words. It is very\n",
            "crucial for humans to understand and to describe the content\n",
            "of the text. IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "TABLE I\n",
            "SUPERVISED AND UNSUPERVISED LEARNING METHODS FOR TEXT SUMMARIZATION\n",
            "Categories Methodology Concept\n",
            "SUPERVISED Machine Learning ap- Summarization task\n",
            "LEARNING proach Bayes rule modelled as classification\n",
            "APPROACHES problem\n",
            "Trainable summarization -\n",
            "SUPERVISED neural network is trained,\n",
            "LEARNING Artificial Neural Net- pruned and generalized to\n",
            "APPROACHES work filter sentences and classify\n",
            "them as \"summary\" or\n",
            "\"non-summary sentence\"\n",
            "SUPERVISED Statistical modelling ap-\n",
            "LEARNING Conditional Random proach which uses CRF as\n",
            "APPROACHES Fields (CRF) a sequence labelling prob- lem\n",
            "UNSUPERVISED Graph based Construction of graph to\n",
            "LEARNING Approach capture relationship be-\n",
            "APPROACHES tween sentences\n",
            "Importance of sentences\n",
            "UNSUPERVISED Concept oriented ap- calculated based on\n",
            "LEARNING the concepts retrieved\n",
            "APPROACHES proach from external knowledge\n",
            "base(wikipedia, HowNet)\n",
            "UNSUPERVISED Fuzzy Logic based ap- Summarization based on\n",
            "LEARNING fuzzy rule using various\n",
            "APPROACHES proach sets of features\n",
            "Fig. IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "1.3 Cue phrase feature\n",
            "Cue phrases are words and phrases that indicate the structure\n",
            "of the document flow and it is used as a feature in\n",
            "sentence selection. The preprocessed\n",
            "passage is sent to the feature extraction steps, which\n",
            "is based on multiple features of sentences and words. LSA captures the text of the\n",
            "input document and excerpt information such as words that\n",
            "frequently occur together and words that are commonly seen in\n",
            "different sentences. Example of Sentence concept bipartite graph proposed in [4]\n",
            "Ladda Suanmali et al [11] proposed fuzzy logic approach\n",
            "is used for automatic text summarization which is the initial\n",
            "step , the text document is pre-processed followed by feature\n",
            "extraction(Title features, Sentence length, Sentence position,\n",
            "Sentence-sentence similarity, term weight, Proper noun and\n",
            "Numerical data. The summary is generated by ordering the\n",
            "ranked sentences in the order they occur in the original\n",
            "document to maintain coherency. The sentences in the\n",
            "document are represented as a graph and the edges between\n",
            "the sentences represents weighted cosine similarity values. The significance\n",
            "of sentences is strongly based on statistical and linguistic\n",
            "features of sentences. Ramanathan\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "Advantages Limitations\n",
            "Large set of training data im- Human interruption required for proves the sentence selection for generating manual summaries summary\n",
            "The network can be trained ac- I)Neural Network is slow in\n",
            "cording to the style of human training phase and also in apreader. The approach specified\n",
            "in [20]uses CRF as a sequence labelling problem and also\n",
            "captures interaction between sentences through the features\n",
            "extracted for each sentence and it also incorporates complex\n",
            "features such as LSA_scores [21] and lilTS_score [22] but\n",
            "the limitation is that linguistic features are not considered. In text Summarization, the most challenging task is\n",
            "to summarize the contented from a number of semistructured\n",
            "sources and textual, which includes web pages\n",
            "and databases, in the proper way (size, format, time,\n",
            "language,) for an explicit user. The major phase\n",
            "is the feature fusion phase where the relationship between\n",
            "the features are identified through two stages 1) eliminating\n",
            "infrequent features 2) collapsing frequent features after\n",
            "which sentence ranking is done to identify the important\n",
            "summary sentences.Neural Network [17]after feature fusion\n",
            "is depicted in Fig 8. The main problem in evaluation comes from\n",
            "the impossibility of building a standard against which the\n",
            "results of the systems that have to be compared. 2.4 Sentence-to-Sentence Cohesion\n",
            "The cohesion between sentences for every sentence(s), the\n",
            "similarity between s and alternative sentences are calculated\n",
            "which are summed up and coarse value of the aspect is\n",
            "obtained for s. The feature values are normalized between\n",
            "[0, 1] where value closer to 1.0 indicates a higher degree of\n",
            "cohesion between sentences. The main advantage of the method is that\n",
            "it is able to identify correct features and provides a better\n",
            "representation of sentences and groups terms appropriately\n",
            "into its segments. The\n",
            "scores obtained after the feature extraction are fed to the\n",
            "neural network, which produces a single value as output score,\n",
            "signifying the importance of the sentences. From Fig6 [13] it is to\n",
            "be noted in order that dl is associated to d2 than dO and the\n",
            "conversation 'walked' is linked to the talk 'man' but it is not\n",
            "significant to the word 'park'. Neural network after training (a) and after pruning (b) [17]\n",
            "In the approach proposed in [18], RankNet algorautomaticallyithm\n",
            "using neural nets to identify the important sentences\n",
            "in the document. Index Terms-Text Summarization, Unsupervised Learning,\n",
            "Supervised Learning, Sentence Fusion, Extraction Scheme, Sentence\n",
            "Revision, Extractive Summary\n",
            "I. INTRODUCTION\n",
            "In a recent advance, the significance of text summarization\n",
            "[1] accomplishes more attention due to data inundation on\n",
            "the web. However, the work has not focused the different\n",
            "challenges of extractive text summarization process to its full\n",
            "intensity in premises of time and space complication. Another graph based approach\n",
            "LexRank [6], where the salience of the sentence is determined\n",
            "by the concept of Eigen vector centrality. The sentence that consists of main keywords is most\n",
            "likely included in the final summary. The main advantage\n",
            "of a text summarization is reading time of the user can be\n",
            "reduced. A conceptual vector\n",
            "model is built to obtain a rough summarization and similarity\n",
            "measures are calculated between the sentences to reduce\n",
            "redundancy in the final summary. B. SUPERVISED LEARNING METHODS\n",
            "Supervised extractive summarizationrelated techniques are\n",
            "based on a classification approach at sentence level where\n",
            "the system learns by examples to classify between summary\n",
            "and non-summary sentences. FEATURES FOR EXTRACTIVE TEXT\n",
            "SUMMARIZATION\n",
            "Earlier techniques involve assigning a score to sentences\n",
            "based on a countenance that are predefined based on the\n",
            "methodology applied. The\n",
            "sentences are clustered into groups based on their similarity\n",
            "measures and then the sentences are ranked based on their\n",
            "LexRank scores similar to PageRank algorithm [7]except that\n",
            "the similarity graph is undirected in LexRank method. Dharmendra Hingu, Deep Shah and\n",
            "Sandeep S.Udmale proposed an extractive approach [19]for\n",
            "summarizing the Wikipedia articles by identifying the text\n",
            "features and scoring the sentences by incorporating neural\n",
            "network model [5]. Count(N-gram) is the number of N-grams in the set of\n",
            "reference summaries. The implication of sentences is determined based on linguistic\n",
            "and statistical features. The proposed system overcomes the issues faced\n",
            "by non-negative matrix Factorization (NMF) methods by incorporating\n",
            "conditional random fields (CRF) to identify and\n",
            "extract correct features to determine the important sentence\n",
            "of the given text.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<function TextIOWrapper.close>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "pibEeZG233lF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d93fa625-d621-43e7-9e6c-62c2040bd82a"
      },
      "source": [
        "for lines in sentence_list:\n",
        "    print(lines)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The normalized length of the sentence is calculated\n",
            "as the ratio between a number of words in the sentence to the\n",
            "number of words in the longest sentence in the document.\n",
            "P (s E< SII1,h,h, .... In)\n",
            "represents the probability of the sentences to be included in\n",
            "the summary based on the given features possessed by the\n",
            "sentence.\n",
            "SENTENCE LEVEL FEATURES\n",
            "2.1 Sentence location feature\n",
            "The sentences that occur in the beginning and the conclusion\n",
            "part of the document are most likely important since most\n",
            "documents are hierarchically structured with important information\n",
            "in the beginning and the end of the paragraphs.\n",
            "The probability of classification are learned from the\n",
            "training data by the following Bayes rule [16]: where s represents\n",
            "the set of sentences in the document and fi represents\n",
            "the features used in classification stage and S represents the\n",
            "set of sentences in the summary.\n",
            "The basic steps in concept\n",
            "based summarization are: i) Retrieve concepts of a text from\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "external knowledge base(HowNet, WordNet, Wikipedia) ii)\n",
            "Build a conceptual vector or graph model to depict relationship\n",
            "between concept and sentences iii) Apply ranking algorithm to\n",
            "score sentences iv) Generate summaries based on the ranking\n",
            "scores of sentences\n",
            "4.\n",
            "The sentences are restricted as a non-summary\n",
            "and summary sentence based on the feature possessed by the\n",
            "sentence.\n",
            "The first\n",
            "step involves labeling the training data using a machinelearning\n",
            "approach and then extract features of the sentences\n",
            "in both test set and train sets which is then inputted to the\n",
            "neural network system to rank the sentences in the document.\n",
            "1.2 Title Word feature\n",
            "The sentences in the original document which consists of\n",
            "words mentioned in the title have greater chances to contribute\n",
            "to the final summary since they serve as indicators of the theme\n",
            "of the document.\n",
            "1.5 Upper case word feature\n",
            "The words which are in uppercase such as \"UNICEF\" are\n",
            "considered to be important words and those sentences that\n",
            "consist of these words are termed important in the context of\n",
            "sentence selection for the final summary.\n",
            "The unsupervised approaches do not\n",
            "need human summaries (user input) in deciding the important\n",
            "features of the document, it requires the most sophisticated\n",
            "algorithm to provide compensation for the lack of human\n",
            "knowledge.\n",
            "The major drawback with the\n",
            "supervised approach is that it requires known manually created\n",
            "summaries by a human to label the sentences in the original\n",
            "training document enclosed with \"summary sentence\" or \"nonsummary\n",
            "sentence\" and it also requires more labeled training\n",
            "data for classification.\n",
            "In the methodology proposed [12], the\n",
            "importance of sentences is calculated based on the concepts\n",
            "retrieved from HowNet instead of words.\n",
            "It is very\n",
            "crucial for humans to understand and to describe the content\n",
            "of the text.\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "TABLE I\n",
            "SUPERVISED AND UNSUPERVISED LEARNING METHODS FOR TEXT SUMMARIZATION\n",
            "Categories Methodology Concept\n",
            "SUPERVISED Machine Learning ap- Summarization task\n",
            "LEARNING proach Bayes rule modelled as classification\n",
            "APPROACHES problem\n",
            "Trainable summarization -\n",
            "SUPERVISED neural network is trained,\n",
            "LEARNING Artificial Neural Net- pruned and generalized to\n",
            "APPROACHES work filter sentences and classify\n",
            "them as \"summary\" or\n",
            "\"non-summary sentence\"\n",
            "SUPERVISED Statistical modelling ap-\n",
            "LEARNING Conditional Random proach which uses CRF as\n",
            "APPROACHES Fields (CRF) a sequence labelling prob- lem\n",
            "UNSUPERVISED Graph based Construction of graph to\n",
            "LEARNING Approach capture relationship be-\n",
            "APPROACHES tween sentences\n",
            "Importance of sentences\n",
            "UNSUPERVISED Concept oriented ap- calculated based on\n",
            "LEARNING the concepts retrieved\n",
            "APPROACHES proach from external knowledge\n",
            "base(wikipedia, HowNet)\n",
            "UNSUPERVISED Fuzzy Logic based ap- Summarization based on\n",
            "LEARNING fuzzy rule using various\n",
            "APPROACHES proach sets of features\n",
            "Fig.\n",
            "IEEE International Conference on Computer, Communication, and Signal Processing (ICCCSP-2017)\n",
            "1.3 Cue phrase feature\n",
            "Cue phrases are words and phrases that indicate the structure\n",
            "of the document flow and it is used as a feature in\n",
            "sentence selection.\n",
            "The preprocessed\n",
            "passage is sent to the feature extraction steps, which\n",
            "is based on multiple features of sentences and words.\n",
            "LSA captures the text of the\n",
            "input document and excerpt information such as words that\n",
            "frequently occur together and words that are commonly seen in\n",
            "different sentences.\n",
            "Example of Sentence concept bipartite graph proposed in [4]\n",
            "Ladda Suanmali et al [11] proposed fuzzy logic approach\n",
            "is used for automatic text summarization which is the initial\n",
            "step , the text document is pre-processed followed by feature\n",
            "extraction(Title features, Sentence length, Sentence position,\n",
            "Sentence-sentence similarity, term weight, Proper noun and\n",
            "Numerical data.\n",
            "The summary is generated by ordering the\n",
            "ranked sentences in the order they occur in the original\n",
            "document to maintain coherency.\n",
            "The sentences in the\n",
            "document are represented as a graph and the edges between\n",
            "the sentences represents weighted cosine similarity values.\n",
            "The significance\n",
            "of sentences is strongly based on statistical and linguistic\n",
            "features of sentences.\n",
            "Ramanathan\n",
            "978-1-5090-3716-2/17/$31.00 ©2017 IEEE\n",
            "Advantages Limitations\n",
            "Large set of training data im- Human interruption required for proves the sentence selection for generating manual summaries summary\n",
            "The network can be trained ac- I)Neural Network is slow in\n",
            "cording to the style of human training phase and also in apreader.\n",
            "The approach specified\n",
            "in [20]uses CRF as a sequence labelling problem and also\n",
            "captures interaction between sentences through the features\n",
            "extracted for each sentence and it also incorporates complex\n",
            "features such as LSA_scores [21] and lilTS_score [22] but\n",
            "the limitation is that linguistic features are not considered.\n",
            "In text Summarization, the most challenging task is\n",
            "to summarize the contented from a number of semistructured\n",
            "sources and textual, which includes web pages\n",
            "and databases, in the proper way (size, format, time,\n",
            "language,) for an explicit user.\n",
            "The major phase\n",
            "is the feature fusion phase where the relationship between\n",
            "the features are identified through two stages 1) eliminating\n",
            "infrequent features 2) collapsing frequent features after\n",
            "which sentence ranking is done to identify the important\n",
            "summary sentences.Neural Network [17]after feature fusion\n",
            "is depicted in Fig 8.\n",
            "The main problem in evaluation comes from\n",
            "the impossibility of building a standard against which the\n",
            "results of the systems that have to be compared.\n",
            "2.4 Sentence-to-Sentence Cohesion\n",
            "The cohesion between sentences for every sentence(s), the\n",
            "similarity between s and alternative sentences are calculated\n",
            "which are summed up and coarse value of the aspect is\n",
            "obtained for s. The feature values are normalized between\n",
            "[0, 1] where value closer to 1.0 indicates a higher degree of\n",
            "cohesion between sentences.\n",
            "The main advantage of the method is that\n",
            "it is able to identify correct features and provides a better\n",
            "representation of sentences and groups terms appropriately\n",
            "into its segments.\n",
            "The\n",
            "scores obtained after the feature extraction are fed to the\n",
            "neural network, which produces a single value as output score,\n",
            "signifying the importance of the sentences.\n",
            "From Fig6 [13] it is to\n",
            "be noted in order that dl is associated to d2 than dO and the\n",
            "conversation 'walked' is linked to the talk 'man' but it is not\n",
            "significant to the word 'park'.\n",
            "Neural network after training (a) and after pruning (b) [17]\n",
            "In the approach proposed in [18], RankNet algorautomaticallyithm\n",
            "using neural nets to identify the important sentences\n",
            "in the document.\n",
            "Index Terms-Text Summarization, Unsupervised Learning,\n",
            "Supervised Learning, Sentence Fusion, Extraction Scheme, Sentence\n",
            "Revision, Extractive Summary\n",
            "I. INTRODUCTION\n",
            "In a recent advance, the significance of text summarization\n",
            "[1] accomplishes more attention due to data inundation on\n",
            "the web.\n",
            "However, the work has not focused the different\n",
            "challenges of extractive text summarization process to its full\n",
            "intensity in premises of time and space complication.\n",
            "Another graph based approach\n",
            "LexRank [6], where the salience of the sentence is determined\n",
            "by the concept of Eigen vector centrality.\n",
            "The sentence that consists of main keywords is most\n",
            "likely included in the final summary.\n",
            "The main advantage\n",
            "of a text summarization is reading time of the user can be\n",
            "reduced.\n",
            "A conceptual vector\n",
            "model is built to obtain a rough summarization and similarity\n",
            "measures are calculated between the sentences to reduce\n",
            "redundancy in the final summary.\n",
            "B. SUPERVISED LEARNING METHODS\n",
            "Supervised extractive summarizationrelated techniques are\n",
            "based on a classification approach at sentence level where\n",
            "the system learns by examples to classify between summary\n",
            "and non-summary sentences.\n",
            "FEATURES FOR EXTRACTIVE TEXT\n",
            "SUMMARIZATION\n",
            "Earlier techniques involve assigning a score to sentences\n",
            "based on a countenance that are predefined based on the\n",
            "methodology applied.\n",
            "The\n",
            "sentences are clustered into groups based on their similarity\n",
            "measures and then the sentences are ranked based on their\n",
            "LexRank scores similar to PageRank algorithm [7]except that\n",
            "the similarity graph is undirected in LexRank method.\n",
            "Dharmendra Hingu, Deep Shah and\n",
            "Sandeep S.Udmale proposed an extractive approach [19]for\n",
            "summarizing the Wikipedia articles by identifying the text\n",
            "features and scoring the sentences by incorporating neural\n",
            "network model [5].\n",
            "Count(N-gram) is the number of N-grams in the set of\n",
            "reference summaries.\n",
            "The implication of sentences is determined based on linguistic\n",
            "and statistical features.\n",
            "The proposed system overcomes the issues faced\n",
            "by non-negative matrix Factorization (NMF) methods by incorporating\n",
            "conditional random fields (CRF) to identify and\n",
            "extract correct features to determine the important sentence\n",
            "of the given text.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O2vG1XWR33lJ",
        "colab_type": "text"
      },
      "source": [
        "# End of the notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "emFE-KU733lJ",
        "colab_type": "text"
      },
      "source": [
        "Please feel free to contribue for any improvements."
      ]
    }
  ]
}