Preprocessing-text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing text\n",
    "\n",
    "In preparing text for analysis, we can make a distinction between general preprocessing of text and specific transformations we might do to it later.\n",
    "This preprocessing typically consists of two kinds of **tokenization**, word tokenization and sentence tokenization.\n",
    "Tokenization is the process of deciding where, in a sequence of characters, *text of interest* starts and stops, e.g. where a word starts and stops or sentence starts and stops.\n",
    "Equivalently, we can think of tokenization as identifying particular *boundaries* in the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn\n",
    "\n",
    "You will learn about tokenization at the word and sentence levels and how this can help you prepare text for later analysis.\n",
    "  \n",
    "We will cover:\n",
    "\n",
    "- Raw, word-tokenized, and sentence-tokenized text\n",
    "- Techniques for word and sentence tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use tokenization\n",
    "\n",
    "Tokenization is a common first step in text analysis.\n",
    "Often it is practical to do tokenization up front and use the resulting data in future analyses, rather than repeating tokenization every time.\n",
    "Use tokenization unless you are working with software libraries that automatically do tokenization when they build a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Three views of text\n",
    "\n",
    "Let's begin by looking at the end product of tokenization to better understand the goal.\n",
    "We'll use NLTK, a well-known software library for teaching NLP that is paired with a [free book](https://www.nltk.org/book/).\n",
    "If you haven't already downloaded the `book` data, execute the following code and select `book` from the download menu.\n",
    "Once you are done, simply close the window to continue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import nltk\n",
    "sys.argv=['']\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we proceed, let's define some terms.\n",
    "A **corpus** is a collection of text. Corpora is the plural of corpus.\n",
    "Typically corpora are created for a particular purpose. For example, if I'm interested in Twitter, I might create a corpus of tweets. \n",
    "Therefore there isn't a single all-purpose corpus, unless you consider the Internet to be a corpus.\n",
    "\n",
    "NLTK has preprocessed many common corpora, and their structure serves as a nice model of what preprocessing should look like.\n",
    "We will take a look before moving on to preparing our own corpora.\n",
    "\n",
    "Here is the general structure for NLTK corpora:\n",
    "\n",
    "- Each corpus is a collection of files which have `fileids`\n",
    "- Each corpus can be read `raw`, and `fileids` can be read raw. Raw means you get the original text.\n",
    "- Each corpus can be read as `words`, and `fileids` can be read as `words`. That means it will give you each word in order.\n",
    "- Each corpus can be read as `sents`, and `fileids` can be read as `sents`. That means it will give you each sentence in order.\n",
    "\n",
    "Raw, words, and sentences are three views of text useful from a preprocessing perspective.\n",
    "\n",
    "Let's import the `gutenberg` corpus. This is a collection of books out of copyright. \n",
    "\n",
    "<!-- Using the same downloading technique we used before, click on the \"Corpora\" tab, select \"gutenberg\", and press \"download.\" **Make sure to close the window when it finishes.** -->\n",
    "\n",
    "- from `nltk.corpus` import `gutenberg`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import gutenberg\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"importFrom\" id=\"XD9YVa/m9vX;ax-@^}(K\" x=\"16\" y=\"64\"><field name=\"libraryName\">nltk.corpus</field><field name=\"libraryAlias\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by getting a list of fileids:\n",
    "\n",
    "- with `gutenberg` do `fileids`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gutenberg.fileids()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\" x=\"26\" y=\"476\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">fileids</field><data>gutenberg:fileids</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each one of these is a file containing the text of a book, e.g. [Jane Austin's Emma](https://en.wikipedia.org/wiki/Emma_(novel)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw\n",
    "Let's take `'austen-emma.txt'` and look at the raw form of it:\n",
    "\n",
    "- set `emma` to with `gutenberg` do `raw` using `'austen-emma.txt'` \n",
    "- in text `emma` get substring from `first letter` to letter # `477`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"[Emma by Jane Austen 1816]\\n\\nVOLUME I\\n\\nCHAPTER I\\n\\n\\nEmma Woodhouse, handsome, clever, and rich, with a comfortable home\\nand happy disposition, seemed to unite some of the best blessings\\nof existence; and had lived nearly twenty-one years in the world\\nwith very little to distress or vex her.\\n\\nShe was the youngest of the two daughters of a most affectionate,\\nindulgent father; and had, in consequence of her sister's marriage,\\nbeen mistress of his house from a very early period.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emma = gutenberg.raw('austen-emma.txt')\n",
    "\n",
    "emma[ : 477]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"~{Lr[.?oEVR{TA827|}Z\">emma</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"8U[%/(,=7*1d(PKaMDtN\" x=\"-7\" y=\"402\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"t(78BFv`F9G)$Ewv`Y6]\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">raw</field><data>gutenberg:raw</data><value name=\"INPUT\"><block type=\"text\" id=\"}/De;X%+0y3nz/Ii$B`w\"><field name=\"TEXT\">austen-emma.txt</field></block></value></block></value></block><block type=\"text_getSubstring\" id=\"T6I]|jqY2V[AT5dEzF9X\" x=\"-12\" y=\"458\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"STRING\"><block type=\"variables_get\" id=\"_5j@],RYC?A)PFN)buGZ\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"|=MueZ1H5an*lY}XX1lq\"><field name=\"NUM\">477</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare this to the [plain text available at Project Gutenberg](https://www.gutenberg.org/cache/epub/158/pg158.txt):\n",
    "\n",
    "...\n",
    "> VOLUME I\n",
    ">\n",
    ">\n",
    ">\n",
    ">\n",
    "> CHAPTER I\n",
    ">\n",
    ">\n",
    "> Emma Woodhouse, handsome, clever, and rich, with a comfortable home and\n",
    "> happy disposition, seemed to unite some of the best blessings of\n",
    "> existence; and had lived nearly twenty-one years in the world with very\n",
    "> little to distress or vex her.\n",
    ">\n",
    "> She was the youngest of the two daughters of a most affectionate,\n",
    "> indulgent father; and had, in consequence of her sister’s marriage,\n",
    "> been mistress of his house from a very early period. \n",
    "\n",
    "We can see that this is a mostly faithful representation of the source text, except for some differences in newlines (`\\n`).\n",
    "Most importantly, we see that the result is a continuous sequence of characters, or *string*, with no explicit boundaries between words or sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Words\n",
    "\n",
    "Let's look at the same text, but this time in words:\n",
    "\n",
    "- set `emmaWords` to with `gutenberg` do `words` using `'austen-emma.txt'` \n",
    "- in list `emmaWords` get sublist from `first` to # `99`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', 'VOLUME', 'I', 'CHAPTER', 'I', 'Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.', 'She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'\", 's', 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emmaWords = gutenberg.words('austen-emma.txt')\n",
    "\n",
    "emmaWords[ : 99]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Xqgp6t-PV4#yzg[[zMob\">emmaWords</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"3OifKIg*V}`wTrVr-U+I\" x=\"38\" y=\"246\"><field name=\"VAR\" id=\"Xqgp6t-PV4#yzg[[zMob\">emmaWords</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"t(78BFv`F9G)$Ewv`Y6]\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">words</field><data>gutenberg:words</data><value name=\"INPUT\"><block type=\"text\" id=\"}/De;X%+0y3nz/Ii$B`w\"><field name=\"TEXT\">austen-emma.txt</field></block></value></block></value></block><block type=\"lists_getSublist\" id=\"hcUX-ml_?56X#302Hu+~\" x=\"160\" y=\"316\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"K0%9m4`F(HvgI0YB@1l%\"><field name=\"VAR\" id=\"Xqgp6t-PV4#yzg[[zMob\">emmaWords</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"Q9K)qPl.,-KFM}ZRFJ3f\"><field name=\"NUM\">99</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result this time is a list of tokens, which includes both words and *punctuation*.\n",
    "The newline characters are completely gone.\n",
    "Notice in particular how compound words like `twenty-one` have been handled, as have possessives like `sister's` - both have been decomposed into constituent elements.\n",
    "These are examples of some of the subtle issues that arise in word tokenization.\n",
    "\n",
    "Let's consider the possessive handling a bit more closely - why break it up into constituent elements?\n",
    "Well, if we are counting occurrences of `sister`, breaking `sister's` into `sister`, `'`, and `s` will prevent us from overlooking possessive forms of `sister`.\n",
    "Similarly, if we are looking at possessives, breaking things apart could make it easier to analyze them, because possessive marking (i.e. the `'` and `s`) has been unbound from the base word.\n",
    "\n",
    "The same idea applies to treating punctuation as it's own token - we don't want to treat `her.` differently from `her` just because it is at the end of the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentences\n",
    "\n",
    "Let's look again at this text, but as sentences:\n",
    "\n",
    "- set `emmaSents` to with `gutenberg` do `sents` using `'austen-emma.txt'` \n",
    "- in list `emmaSents` get sublist from `first` to # `5`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'], ['VOLUME', 'I'], ['CHAPTER', 'I'], ['Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.'], ['She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'\", 's', 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.']]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emmaSents = gutenberg.sents('austen-emma.txt')\n",
    "\n",
    "emmaSents[ : 5]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"]kZqQLAcFQ}sq5~yofrj\" x=\"-12\" y=\"391\"><field name=\"VAR\" id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"t(78BFv`F9G)$Ewv`Y6]\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">sents</field><data>gutenberg:sents</data><value name=\"INPUT\"><block type=\"text\" id=\"}/De;X%+0y3nz/Ii$B`w\"><field name=\"TEXT\">austen-emma.txt</field></block></value></block></value></block><block type=\"lists_getSublist\" id=\"h;d!0,b!=].To}2fLmGI\" x=\"-10\" y=\"462\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"?UJ[[J=fEuk}dk2z`1G9\"><field name=\"VAR\" id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"g25x1,DzoN6uz)KnIK-`\"><field name=\"NUM\">6</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we get a **list of lists** of tokens, rather than a list of tokens.\n",
    "Each inner list corresponds to the tokens in a sentence.\n",
    "So really this is no different than what we got with word tokenization, except that now we have additionally grouped tokens by sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizing manually\n",
    "\n",
    "As nice as NLTK's preprocessed corpora are (check out the [full list here](https://www.nltk.org/nltk_data/)]), chances are the text you want to study is not already in NLTK.\n",
    "Therefore, you're going to need to do the preprocessing yourself.\n",
    "We're going to keep working with NLTK, but you can do tokenization with just about any NLP library; a current popular alternative is [spaCy](https://spacy.io/usage/spacy-101).\n",
    "\n",
    "In the examples that follow, we'll use the same text as above, but this time load it manually:\n",
    "\n",
    "- import `nltk` as `nltk`\n",
    "- set `emma` to\n",
    "    - read text from file (see I/O) connected to\n",
    "    - with `gutenberg` do `abspath` using `austen-emma.txt`\n",
    "    \n",
    "You could read any arbitrary file then same way - `gutenberg.abspath('austen-emma.txt')` is just the file path to `austen-emma.txt` in the NLTK storage folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk as nltk\n",
    "\n",
    "emma = open((gutenberg.abspath('austen-emma.txt')),encoding='utf-8').read()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"~{Lr[.?oEVR{TA827|}Z\">emma</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"importAs\" id=\"$o8RA~*8X9Q|cbW?zr;E\" x=\"71\" y=\"81\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field></block><block type=\"variables_set\" id=\"BjKkWY`m(S98Nd):#J=V\" x=\"59\" y=\"155\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field><value name=\"VALUE\"><block type=\"textFromFile\" id=\"GI())F?=o,*MK.I%*`ks\"><value name=\"FILENAME\"><shadow type=\"text\" id=\"~*D.|b9xA7[NmQgY-H^1\"><field name=\"TEXT\">name of file</field></shadow><block type=\"varDoMethod\" id=\"jfx_*M4zaU~HuQbXM|oC\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">abspath</field><data>emmaSents:</data><value name=\"INPUT\"><block type=\"text\" id=\"0II}$K#u5Zv@_=/(*:5[\"><field name=\"TEXT\">austen-emma.txt</field></block></value></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing words\n",
    "\n",
    "NLTK has a handy function called `word_tokenize`. Let's use it and compare to the original we got directly from NLTK:\n",
    "\n",
    "- set `emmaWords2` to with `nltk` do `word_tokenize` using `emma`\n",
    "- print in list `emmaWords` get sublist from `first` to # `99`\n",
    "- print in list `emmaWords2` get sublist from `first` to # `99`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', 'VOLUME', 'I', 'CHAPTER', 'I', 'Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.', 'She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'\", 's', 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.']\n",
      "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', 'VOLUME', 'I', 'CHAPTER', 'I', 'Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty-one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.', 'She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'s\", 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.', 'Her', 'mother', 'had']\n"
     ]
    }
   ],
   "source": [
    "emmaWords2 = nltk.word_tokenize(emma)\n",
    "print(emmaWords[ : 99])\n",
    "print(emmaWords2[ : 99])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"K,0]PBDLVIX*%3xW%U}7\">emmaWords2</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"~{Lr[.?oEVR{TA827|}Z\">emma</variable><variable id=\"Xqgp6t-PV4#yzg[[zMob\">emmaWords</variable></variables><block type=\"variables_set\" id=\"aU`_G5Z/zt~N{AO*g::M\" x=\"86\" y=\"164\"><field name=\"VAR\" id=\"K,0]PBDLVIX*%3xW%U}7\">emmaWords2</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"4w-PTl0Y*;o5k:Cqzu6Y\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">word_tokenize</field><data>nltk:word_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"__(y#:Q,Y4P)$8|u*+%Y\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field></block></value></block></value><next><block type=\"text_print\" id=\"Gj/{Q?NFnH1ICV_[Qu7R\"><value name=\"TEXT\"><shadow type=\"text\" id=\"T%?w2b@h1c)y%Jw}3f5X\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KUp-K(cJm1okZ?L=_5OC\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"rq~%1IJ~yvXiI]`)0Vu[\"><field name=\"VAR\" id=\"Xqgp6t-PV4#yzg[[zMob\">emmaWords</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"H=6./b%q+OqNC)CH;~,N\"><field name=\"NUM\">99</field></block></value></block></value><next><block type=\"text_print\" id=\"4wt`aU/QlG+j?5~BjZ9^\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KV(L4I}c|510m]g(ukTf\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"}DiVK@bY%ZxK(KzOA(sh\"><field name=\"VAR\" id=\"K,0]PBDLVIX*%3xW%U}7\">emmaWords2</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\".?_iG5|BNQ|Z|E_[n-1n\"><field name=\"NUM\">99</field></block></value></block></value></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice we have some differences.** In the tokenization we just did, `twenty-one` has not been decomposed, and `'s` has been treated as a single unit (this is PennTreebank style).\n",
    "Again, there are subtle distinctions in tokenization styles, and different styles may be preferred depending on your analysis. \n",
    "Or, you may just prefer to get very familiar with a particular style so you aren't surprised by the decisions it makes.\n",
    "A list of styles for available in NLTK can be found [here](https://www.nltk.org/api/nltk.tokenize.html#submodules)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing sentences\n",
    "\n",
    "Let's manually tokenize sentences using NLTK's `sent_tokenize`:\n",
    "\n",
    "- set `emmaSents2` to with `nltk` do `sent_tokenize` using `emma`\n",
    "- print in list `emmaSents` get sublist from `first` to # `5`\n",
    "- print in list `emmaSents2` get sublist from `first` to # `5`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'], ['VOLUME', 'I'], ['CHAPTER', 'I'], ['Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.'], ['She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'\", 's', 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.']]\n",
      "['[Emma by Jane Austen 1816]\\n\\nVOLUME I\\n\\nCHAPTER I\\n\\n\\nEmma Woodhouse, handsome, clever, and rich, with a comfortable home\\nand happy disposition, seemed to unite some of the best blessings\\nof existence; and had lived nearly twenty-one years in the world\\nwith very little to distress or vex her.', \"She was the youngest of the two daughters of a most affectionate,\\nindulgent father; and had, in consequence of her sister's marriage,\\nbeen mistress of his house from a very early period.\", 'Her mother\\nhad died too long ago for her to have more than an indistinct\\nremembrance of her caresses; and her place had been supplied\\nby an excellent woman as governess, who had fallen little short\\nof a mother in affection.', \"Sixteen years had Miss Taylor been in Mr. Woodhouse's family,\\nless as a governess than a friend, very fond of both daughters,\\nbut particularly of Emma.\", 'Between _them_ it was more the intimacy\\nof sisters.']\n"
     ]
    }
   ],
   "source": [
    "emmaSents2 = nltk.sent_tokenize(emma)\n",
    "print(emmaSents[ : 5])\n",
    "print(emmaSents2[ : 5])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"V:qE@zR0KPo88fkuJ}ht\">emmaSents2</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"~{Lr[.?oEVR{TA827|}Z\">emma</variable><variable id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</variable></variables><block type=\"variables_set\" id=\"aU`_G5Z/zt~N{AO*g::M\" x=\"86\" y=\"164\"><field name=\"VAR\" id=\"V:qE@zR0KPo88fkuJ}ht\">emmaSents2</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"4w-PTl0Y*;o5k:Cqzu6Y\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">sent_tokenize</field><data>nltk:sent_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"__(y#:Q,Y4P)$8|u*+%Y\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field></block></value></block></value><next><block type=\"text_print\" id=\"Gj/{Q?NFnH1ICV_[Qu7R\"><value name=\"TEXT\"><shadow type=\"text\" id=\"T%?w2b@h1c)y%Jw}3f5X\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KUp-K(cJm1okZ?L=_5OC\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"rq~%1IJ~yvXiI]`)0Vu[\"><field name=\"VAR\" id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"H=6./b%q+OqNC)CH;~,N\"><field name=\"NUM\">5</field></block></value></block></value><next><block type=\"text_print\" id=\"4wt`aU/QlG+j?5~BjZ9^\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KV(L4I}c|510m]g(ukTf\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"}DiVK@bY%ZxK(KzOA(sh\"><field name=\"VAR\" id=\"V:qE@zR0KPo88fkuJ}ht\">emmaSents2</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\".?_iG5|BNQ|Z|E_[n-1n\"><field name=\"NUM\">5</field></block></value></block></value></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the differences are more severe.\n",
    "What happened?\n",
    "If you look closely, you'll see that the manual sentence tokenization returned each sentence as a sequence of characters, or *string*, rather than as a list of word tokens.\n",
    "Also notice that the title and headings were not separated from the first sentence.\n",
    "\n",
    "If we want both words and sentences tokenized together, we can combine the two operations.\n",
    "Let's look at how to do this using **comprehensions**.\n",
    "A comprehension is basically a list generator.\n",
    "In other words, instead of having a `for` loop that adds elements to a list, we can define a list by specifying how to generate each element in the list.\n",
    "Comprehensions are pretty useful for NLP.\n",
    "Let's try it:\n",
    "\n",
    "- Create `emmaSents3` and set to\n",
    "    - a list with one element containing\n",
    "        - for each item `i` in list with `nltk` do `sent_tokenize` using `emma` (see LOOPS)\n",
    "            - yield with `nltk` do `word_tokenize` using `i`\n",
    "- print in list `emmaSents` get sublist from `first` to # `5`\n",
    "- print in list `emmaSents3` get sublist from `first` to # `5`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'], ['VOLUME', 'I'], ['CHAPTER', 'I'], ['Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.'], ['She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'\", 's', 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.']]\n",
      "[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', 'VOLUME', 'I', 'CHAPTER', 'I', 'Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty-one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.'], ['She', 'was', 'the', 'youngest', 'of', 'the', 'two', 'daughters', 'of', 'a', 'most', 'affectionate', ',', 'indulgent', 'father', ';', 'and', 'had', ',', 'in', 'consequence', 'of', 'her', 'sister', \"'s\", 'marriage', ',', 'been', 'mistress', 'of', 'his', 'house', 'from', 'a', 'very', 'early', 'period', '.'], ['Her', 'mother', 'had', 'died', 'too', 'long', 'ago', 'for', 'her', 'to', 'have', 'more', 'than', 'an', 'indistinct', 'remembrance', 'of', 'her', 'caresses', ';', 'and', 'her', 'place', 'had', 'been', 'supplied', 'by', 'an', 'excellent', 'woman', 'as', 'governess', ',', 'who', 'had', 'fallen', 'little', 'short', 'of', 'a', 'mother', 'in', 'affection', '.'], ['Sixteen', 'years', 'had', 'Miss', 'Taylor', 'been', 'in', 'Mr.', 'Woodhouse', \"'s\", 'family', ',', 'less', 'as', 'a', 'governess', 'than', 'a', 'friend', ',', 'very', 'fond', 'of', 'both', 'daughters', ',', 'but', 'particularly', 'of', 'Emma', '.'], ['Between', '_them_', 'it', 'was', 'more', 'the', 'intimacy', 'of', 'sisters', '.']]\n"
     ]
    }
   ],
   "source": [
    "emmaSents3 = [(nltk.word_tokenize(i)) for i in (nltk.sent_tokenize(emma))]\n",
    "print(emmaSents[ : 5])\n",
    "print(emmaSents3[ : 5])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"SS`8(eF^8i7x|~c%g!go\">emmaSents3</variable><variable id=\"vm#^l?f3f{nfc06H7ta7\">i</variable><variable id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"~{Lr[.?oEVR{TA827|}Z\">emma</variable></variables><block type=\"variables_set\" id=\"aU`_G5Z/zt~N{AO*g::M\" x=\"29\" y=\"204\"><field name=\"VAR\" id=\"SS`8(eF^8i7x|~c%g!go\">emmaSents3</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"he+/o3bmtQnCkPO)M4_T\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"o;W2uLos,hSk20@(p7y1\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"4w-PTl0Y*;o5k:Cqzu6Y\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">sent_tokenize</field><data>nltk:sent_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"__(y#:Q,Y4P)$8|u*+%Y\"><field name=\"VAR\" id=\"~{Lr[.?oEVR{TA827|}Z\">emma</field></block></value></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"#s*EXxByy$-41LlvlrHk\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">word_tokenize</field><data>nltk:word_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"{8R3$6Qk=2$li.9UOoI2\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field></block></value></block></value></block></value></block></value><next><block type=\"text_print\" id=\"Gj/{Q?NFnH1ICV_[Qu7R\"><value name=\"TEXT\"><shadow type=\"text\" id=\"T%?w2b@h1c)y%Jw}3f5X\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KUp-K(cJm1okZ?L=_5OC\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"rq~%1IJ~yvXiI]`)0Vu[\"><field name=\"VAR\" id=\"dAM-/@sFPVu5R?$h[.lk\">emmaSents</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"H=6./b%q+OqNC)CH;~,N\"><field name=\"NUM\">5</field></block></value></block></value><next><block type=\"text_print\" id=\"4wt`aU/QlG+j?5~BjZ9^\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"KV(L4I}c|510m]g(ukTf\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"}DiVK@bY%ZxK(KzOA(sh\"><field name=\"VAR\" id=\"SS`8(eF^8i7x|~c%g!go\">emmaSents3</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\".?_iG5|BNQ|Z|E_[n-1n\"><field name=\"NUM\">5</field></block></value></block></value></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we got what we expected, with the aforementioned differences in the handling of the title/headings and tokenization of words like `sister's`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}