nlp regexp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "자연어 처리 툴킷을 불러온다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "s는 스트링이고 t는 리스트이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'man', 'in', 'the', 'house']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s='a man in the house'\n",
    "t=['a', 'man', 'in', 'the', 'house']\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "텍스트를 자연어처리로 받았다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Text: a man in the house...>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a=nltk.Text(t)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "스플릿 함수를 사용하였다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Text: a man in the house...>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.Text(s.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "그렇게 나누어진 텍스트의 타입과 결과물을 찾아보았다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'nltk.text.Text'>\n",
      "<Text: a man in the house...>\n"
     ]
    }
   ],
   "source": [
    "b=nltk.Text(s.split())\n",
    "print(type(b))\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 내 pc의 python이 작업하고 있는 경로를 탐색하였다. 거기서 06_01이라는 텍스트 파일을 불러왔고, 인코더는 국민인코더 utf8을 사용하였다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "raw=open(os.getcwd()+r'/06_01.txt', encoding='utf8').read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "클래스는 string으로 나온다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'str'>\n",
      "1154507\n",
      "The Project Gutenberg EBook of Crime and Punishment, by Fyodor Dostoevsky\n",
      "\n",
      "This eBook is for the use of anyone anywhere at no cost and with\n",
      "almost no restrictions whatsoever.  You may copy it, give it\n"
     ]
    }
   ],
   "source": [
    "print(type(raw))\n",
    "print(len(raw))\n",
    "print(raw[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tokenize하기 위한 패키지 punkt을 다운받았다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\admin\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "그렇게 tokenize 된 raw의 타입을 알아보았다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "257726\n",
      "['The', 'Project', 'Gutenberg', 'EBook', 'of', 'Crime', 'and', 'Punishment', ',', 'by']\n"
     ]
    }
   ],
   "source": [
    "tokens=nltk.word_tokenize(raw)\n",
    "print(type(tokens))\n",
    "print(len(tokens))\n",
    "print(tokens[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ";를 첨가하여 보는것과 :10 까지 나열해보았다. text[:10]은 리스트로 나온다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'nltk.text.Text'>\n",
      "['The', 'Project', 'Gutenberg', 'EBook', 'of', 'Crime', 'and', 'Punishment', ',', 'by']\n",
      "Katerina Ivanovna; Pyotr Petrovitch; Pulcheria Alexandrovna; Avdotya\n",
      "Romanovna; Rodion Romanovitch; Marfa Petrovna; Sofya Semyonovna; old\n",
      "woman; Project Gutenberg-tm; Porfiry Petrovitch; Amalia Ivanovna;\n",
      "great deal; young man; Nikodim Fomitch; Ilya Petrovitch; Project\n",
      "Gutenberg; Andrey Semyonovitch; Hay Market; Dmitri Prokofitch; Good\n",
      "heavens\n"
     ]
    }
   ],
   "source": [
    "text=nltk.Text(tokens)\n",
    "print(type(text))\n",
    "print(text[:10])\n",
    "text.collocations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Text: The Project Gutenberg EBook of Crime and Punishment...>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.Text(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\admin\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping corpora\\stopwords.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Katerina Ivanovna; Pyotr Petrovitch; Pulcheria Alexandrovna; Avdotya\n",
      "Romanovna; Rodion Romanovitch; Marfa Petrovna; Sofya Semyonovna; old\n",
      "woman; Project Gutenberg-tm; Porfiry Petrovitch; Amalia Ivanovna;\n",
      "great deal; young man; Nikodim Fomitch; Ilya Petrovitch; Project\n",
      "Gutenberg; Andrey Semyonovitch; Hay Market; Dmitri Prokofitch; Good\n",
      "heavens\n"
     ]
    }
   ],
   "source": [
    "text.collocations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 25 of 162 matches:\n",
      "and were more frequent in periods of great strain . In 1859 he was allowed to r\n",
      "ndency and of late she had read with great interest a book she got through Mr. \n",
      " the bosom of her family ... . And a great deal more ... . Quite excusable , si\n",
      "that you had heard that Dounia had a great deal to put up with in the Svidrigra\n",
      "g will she has . Dounia can endure a great deal and even in the most difficult \n",
      " that letter she reproached him with great heat and indignation for the basenes\n",
      "putation ; they had seen and known a great deal more than Mr. Svidrigailov had \n",
      "n other people ’ s . In my opinion a great deal , a very great deal of all this\n",
      " In my opinion a great deal , a very great deal of all this was unnecessary ; b\n",
      " . He is a very busy man and is in a great hurry to get to Petersburg , so that\n",
      " me that , though he is not a man of great education , he is clever and seems t\n",
      " very well . Of course , there is no great love either on his side , or on hers\n",
      "tted the matter has been arranged in great haste . Besides he is a man of great\n",
      "great haste . Besides he is a man of great prudence and he will see , to be sur\n",
      "d that she is ready to put up with a great deal , if only their future relation\n",
      " off for Petersburg , where he has a great deal of business , and he wants to o\n",
      "a or I breathed a word to him of the great hopes we have of his helping us to p\n",
      "ites that ‘ Dounia can put up with a great deal. ’ I know that very well . I kn\n",
      "at , that ‘ Dounia can put up with a great deal. ’ If she could put up with Mr.\n",
      "it , she certainly can put up with a great deal . And now mother and she have t\n",
      "e young , and she was walking in the great heat bareheaded and with no parasol \n",
      "f the skirt , close to the waist : a great piece was rent and hanging loose . A\n",
      "ts or conversations . He worked with great intensity without sparing himself , \n",
      " uproarious and was reputed to be of great physical strength . One night , when\n",
      ". His legs felt suddenly heavy and a great drowsiness came upon him . He turned\n"
     ]
    }
   ],
   "source": [
    "text.concordance('great',79,25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package gutenberg to\n",
      "[nltk_data]     C:\\Users\\admin\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping corpora\\gutenberg.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('gutenberg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "소문자e,t,a,o,n이 각각 멜빌의 모비딕에서 몇 번 사용됐는지를 표시했다. 가장 빈번하게 사용된 알파벳이 무엇인지를 먼저 찾고 for loop가 5회를 돌게 된다. 그 리고 몇번을 나왔는지 보여준다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('e', 117092), ('t', 87996), ('a', 77916), ('o', 69326), ('n', 65617)]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['e', 't', 'a', 'o', 'n']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw=nltk.corpus.gutenberg.raw('melville-moby_dick.txt')\n",
    "#print(raw[:200])\n",
    "#fdist=nltk.FreqDist(raw)\n",
    "#print(fdist.most_common(50))\n",
    "\n",
    "[ch.lower()for ch in raw if ch.isalpha()]\n",
    "\n",
    "fdist=nltk.FreqDist(ch.lower()for ch in raw if ch.isalpha())\n",
    "print(fdist.most_common(5))\n",
    "[char for (char, count) in fdist.most_common(5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'aa', 'aal', 'aalii', 'aam', 'aardvark', 'aardwolf', 'aba', 'abac', 'abaca']\n"
     ]
    }
   ],
   "source": [
    "wordlist=[w for w in nltk.corpus.words.words('en') if w.islower()]\n",
    "print(wordlist[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package words to\n",
      "[nltk_data]     C:\\Users\\admin\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping corpora\\words.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "result=[w for w in wordlist if re.search('ed$',w)][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abaissed',\n",
       " 'abandoned',\n",
       " 'abased',\n",
       " 'abashed',\n",
       " 'abatised',\n",
       " 'abed',\n",
       " 'aborted',\n",
       " 'abridged',\n",
       " 'abscessed',\n",
       " 'absconded']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "jang 950325-1030210\n",
      "jeon 980226-2030211\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data='''\n",
    "jang 950325-1030210\n",
    "jeon 980226-2030211\n",
    "'''\n",
    "\n",
    "result=[]\n",
    "for line in data.split('\\n'):\n",
    "    word_result=[]\n",
    "    for word in line.split(' '):\n",
    "        if len(word) == 14 and word[:6].isdigit() and word[:7].isdigit():\n",
    "            word=word[:6] + '-' + '*******'\n",
    "        word_result.append(word)\n",
    "    result.append(' '.join(word_result))\n",
    "print('\\n'.join(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}