Single-word-transformations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Single word transformations\n",
    "\n",
    "In [Text as data](Text-as-data.ipynb), we discussed how we can think about words as reflecting latent variables.\n",
    "From this perspective, the words themselves are not what we are interested in - rather they are indicators of some latent variable of interest.\n",
    "In this notebook, we expand on this idea by considering transformations on words, token for token, that take us closer, in some sense, to the latent variable we care about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn\n",
    "\n",
    "You will learn about transformations of single words that can be used as features in predictive models.\n",
    "  \n",
    "We will cover:\n",
    "\n",
    "- Parts of speech\n",
    "- Stemming and lemmatization\n",
    "- Dictionary and regular expression tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use single word transformations\n",
    "\n",
    "Single word transformations can be thought of as a kind of preprocessing that converts words into features.\n",
    "The transformations we discuss take a nontrivial amount of a data to learn, so they can also be viewed as a kind of data augmentation (or reduction) that is useful when you are working with a small amount of data: by redescribing your text data using a smaller set of tokens, you are making it easier for your model to learn the structure of the data.\n",
    "While some transformations we'll cover are fairly theory-neutral, we will also describe transformations that allow you to impose your own theory regarding the latent variables behind the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parts of speech\n",
    "\n",
    "We can infer parts of speech through a process called part of speech tagging.\n",
    "Every word has one and only one part of speech (i.e. a single feature), but there are many part of speech categories.\n",
    "Below is a list of Penn Treebank tags, which are widely used, to give you a sense of the part of speech tagging problem.\n",
    "Note that some tagging systems have more tags and some have less.\n",
    "\n",
    "|Tag|Gloss|\n",
    "|:---|:---|\n",
    "|CC|Coordinating conjunction|\n",
    "|CD|Cardinal number|\n",
    "|DT|Determiner|\n",
    "|EX|Existential there|\n",
    "|FW|Foreign word|\n",
    "|IN|Preposition or subordinating conjunction|\n",
    "|JJ|Adjective|\n",
    "|JJR|Adjective, comparative|\n",
    "|JJS|Adjective, superlative|\n",
    "|LS|List item marker|\n",
    "|MD|Modal|\n",
    "|NN|Noun, singular or mass|\n",
    "|NNS|Noun, plural|\n",
    "|NNP|Proper noun, singular|\n",
    "|NNPS|Proper noun, plural|\n",
    "|PDT|Predeterminer|\n",
    "|POS|Possessive ending|\n",
    "|PRP|Personal pronoun|\n",
    "|PRP\\$|Possessive pronoun|\n",
    "|RB|Adverb|\n",
    "|RBR|Adverb, comparative|\n",
    "|RBS|Adverb, superlative|\n",
    "|RP|Particle|\n",
    "|SYM|Symbol|\n",
    "|TO|to|\n",
    "|UH|Interjection|\n",
    "|VB|Verb, base form|\n",
    "|VBD|Verb, past tense|\n",
    "|VBG|Verb, gerund or present participle|\n",
    "|VBN|Verb, past participle|\n",
    "|VBP|Verb, non-3rd person singular present|\n",
    "|VBZ|Verb, 3rd person singular present|\n",
    "|WDT|Wh-determiner|\n",
    "|WP|Wh-pronoun|\n",
    "|WP\\$|Possessive wh-pronoun|\n",
    "|WRB|Wh-adverb|\n",
    "|.|period|\n",
    "|,|comma|\n",
    "|:|colon|\n",
    "|(|left separator|\n",
    "|)|right separator|\n",
    "|$|dollar sign|\n",
    "|\\`\\`|open double quotes|\n",
    "|''|close double quotes|\n",
    "\n",
    "\n",
    "Let's start by importing `nltk` and storing the words that we will use in our running example:\n",
    "\n",
    "- `import nltk as nltk`\n",
    "- Set `words` to make list from text `\"I like to fly more than a fly in the ointment . My tooth is aching .\"` with delimiter `\" \"` (a single space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk as nltk\n",
    "words = 'I like to fly more than a fly in the ointment . My tooth is aching .'.split(' ')\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</variable><variable id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</variable></variables><block type=\"importAs\" id=\"|Lgb)._R/(OoNvhctV=J\" x=\"139\" y=\"155\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><next><block type=\"variables_set\" id=\"Y`HIOjJh?SQ$=OMt4Pr5\"><field name=\"VAR\" id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</field><value name=\"VALUE\"><block type=\"lists_split\" id=\"LW@]%KZ*Y}C:2NUEomQ@\"><mutation mode=\"SPLIT\"></mutation><field name=\"MODE\">SPLIT</field><value name=\"INPUT\"><block type=\"text\" id=\"45|G8`H+}-r%H.,1lqpB\"><field name=\"TEXT\">I like to fly more than a fly in the ointment . My tooth is aching .</field></block></value><value name=\"DELIM\"><shadow type=\"text\" id=\"BkEBlQE=V]{t?pMU|`%c\"><field name=\"TEXT\"> </field></shadow></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To tag the words, do:\n",
    "    \n",
    "- Set `pos` to with `nltk` do `pos_tag` using `words`\n",
    "- Display `pos`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('I', 'PRP'), ('like', 'VBP'), ('to', 'TO'), ('fly', 'VB'), ('more', 'JJR'), ('than', 'IN'), ('a', 'DT'), ('fly', 'NN'), ('in', 'IN'), ('the', 'DT'), ('ointment', 'NN'), ('.', '.'), ('My', 'PRP$'), ('tooth', 'NN'), ('is', 'VBZ'), ('aching', 'VBG'), ('.', '.')]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos = nltk.pos_tag(words)\n",
    "\n",
    "pos\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"U`QjFH|a^qhm])]1I$;@\">pos</variable><variable id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</variable><variable id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</variable></variables><block type=\"variables_set\" id=\"~_HKg/pHwNNe1hYFjUZk\" x=\"58\" y=\"176\"><field name=\"VAR\" id=\"U`QjFH|a^qhm])]1I$;@\">pos</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"BMN2~lB}esR80hrHf_#T\"><field name=\"VAR\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><field name=\"MEMBER\">pos_tag</field><data>nltk:pos_tag</data><value name=\"INPUT\"><block type=\"variables_get\" id=\",t9APXKJ5Yhm/t2ZaL-J\"><field name=\"VAR\" id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</field></block></value></block></value></block><block type=\"variables_get\" id=\"xh)]qdv`jesXCXAU%[iP\" x=\"57\" y=\"237\"><field name=\"VAR\" id=\"U`QjFH|a^qhm])]1I$;@\">pos</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we have two senses of `fly` in this text.\n",
    "One is `to fly` (a verb) and the other is `insect fly` (a noun).\n",
    "This is one example use of part of speech, which is to disambiguate.\n",
    "For disambiguation, we could transform our words into `word/pos`:\n",
    "\n",
    "- Create a list with one element containing\n",
    "    - for each item `i` in list `pos`\n",
    "    - yield a freestyle `i[0] + \"/\" + i[1]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I/PRP', 'like/VBP', 'to/TO', 'fly/VB', 'more/JJR', 'than/IN', 'a/DT', 'fly/NN', 'in/IN', 'the/DT', 'ointment/NN', './.', 'My/PRP$', 'tooth/NN', 'is/VBZ', 'aching/VBG', './.']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i[0] + \"/\" + i[1] for i in pos]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"K-[j~N(`e*]acWp:SRf{\">i</variable><variable id=\"U`QjFH|a^qhm])]1I$;@\">pos</variable></variables><block type=\"lists_create_with\" id=\"..T%=|_^VRJTL`kjaSn(\" x=\"8\" y=\"60\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"eI@m:2o_oaXDe_A6h(If\"><field name=\"VAR\" id=\"K-[j~N(`e*]acWp:SRf{\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"C^.`U:$]*?]bnNC7nVWN\"><field name=\"VAR\" id=\"U`QjFH|a^qhm])]1I$;@\">pos</field></block></value><value name=\"YIELD\"><block type=\"dummyOutputCodeBlock\" id=\"_g*G1XZ6S*Y7nl^ek~Yr\"><field name=\"CODE\">i[0] + \"/\" + i[1]</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use these transformed tokens in the place of words in later analysis, e.g. we could vectorize them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming and lemmatization\n",
    "\n",
    "Stemming and lemmatization both address the problem of morphology, but in different ways.\n",
    "Stemming applies fairly simple rules to strip affixes (see [here](https://tartarus.org/martin/PorterStemmer/) for a description of the popular Porter stemmer).\n",
    "Lemmatization also seeks to remove affixes, and otherwise reduce words to their base form (e.g. `is` to `be`), but without creating non-words in the process, which stemming often does.\n",
    "In other words, if you want a fast approach that can create nonwords, stemming is fine, otherwise you want lemmatization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are [many stemmers](https://www.nltk.org/api/nltk.stem.html?highlight=stemming) available in NLTK; here we will demonstrate the famous Porter stemmer.\n",
    "\n",
    "- Create variable `stemmer` and set to `with nltk create PorterStemmer`\n",
    "\n",
    "NLTK doesn't have a similar diversity of lemmatizers - there's just the `WordNetLemmatizer`.\n",
    "This is possibly because a good lemmatizer needs access to a lexical database (like WordNet); otherwise, how could it know things like `be` is the base form of `is`?\n",
    "\n",
    "- Create variable `lemmatizer` and set to `with nltk create WordNetLemmatizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = nltk.PorterStemmer()\n",
    "lemmatizer = nltk.WordNetLemmatizer()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"gc[OXAJ:}?^trO)TCN#S\">stemmer</variable><variable id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</variable><variable id=\"aZCw6y4p!sn)Cw$X1ojp\">lemmatizer</variable></variables><block type=\"variables_set\" id=\"[sj83m4H-6.6Jzxp[$:d\" x=\"15\" y=\"249\"><field name=\"VAR\" id=\"gc[OXAJ:}?^trO)TCN#S\">stemmer</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"/F8MZ{fsYHeL6}c4r%)p\"><field name=\"VAR\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><field name=\"MEMBER\">PorterStemmer</field><data>nltk:PorterStemmer</data></block></value><next><block type=\"variables_set\" id=\"=j$`NjMR@F]pC/vWJjFB\"><field name=\"VAR\" id=\"aZCw6y4p!sn)Cw$X1ojp\">lemmatizer</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"5L`9Lg_!?klsQH=nGlb1\"><field name=\"VAR\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><field name=\"MEMBER\">WordNetLemmatizer</field><data>nltk:WordNetLemmatizer</data></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lemmatizer needs part of speech to work correctly, otherwise it assumes everything is a noun.\n",
    "NLTK doesn't handle this well, so execute the code below to do the mapping for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def map_tag(tag):\n",
    "    if tag.startswith('J'):\n",
    "        return 'a'\n",
    "    elif tag.startswith('V'):\n",
    "        return 'v'\n",
    "    elif tag.startswith('R'):\n",
    "        return 'r'\n",
    "    else:\n",
    "        return 'n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare their outputs:\n",
    "\n",
    "- print a list with one element containing\n",
    "    - for each item `i` in list `words`\n",
    "    - yield with `stemmer` to `stem` using `i`\n",
    "- print a list with one element containing\n",
    "    - for each item `i` in list `pos`\n",
    "    - yield with `lemmatizer` to `lemmatize` using a list containing\n",
    "        - freestyle `i[0]` (the word)\n",
    "        - freestyle `map_tag(i[1])` (the pos, mapped using the function above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'like', 'to', 'fli', 'more', 'than', 'a', 'fli', 'in', 'the', 'ointment', '.', 'My', 'tooth', 'is', 'ach', '.']\n",
      "['I', 'like', 'to', 'fly', 'more', 'than', 'a', 'fly', 'in', 'the', 'ointment', '.', 'My', 'tooth', 'be', 'ache', '.']\n",
      "['n', 'v', 'n', 'v', 'a', 'n', 'n', 'n', 'n', 'n', 'n', 'n', 'n', 'n', 'v', 'v', 'n']\n"
     ]
    }
   ],
   "source": [
    "print([(stemmer.stem(i)) for i in words])\n",
    "print([(lemmatizer.lemmatize(i[0], map_tag(i[1]))) for i in pos])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"K-[j~N(`e*]acWp:SRf{\">i</variable><variable id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</variable><variable id=\"gc[OXAJ:}?^trO)TCN#S\">stemmer</variable><variable id=\"U`QjFH|a^qhm])]1I$;@\">pos</variable><variable id=\"aZCw6y4p!sn)Cw$X1ojp\">lemmatizer</variable></variables><block type=\"text_print\" id=\"VT`s0$yo./*;G`3Utl;[\" x=\"67\" y=\"134\"><value name=\"TEXT\"><shadow type=\"text\" id=\"D|ngef97g0q4Yp?G~3+(\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_create_with\" id=\"}[B$VVkV,eLCd,;ybax}\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"g/M{1rT*bV3r7?-0dkmR\"><field name=\"VAR\" id=\"K-[j~N(`e*]acWp:SRf{\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"E/y?aR*D.LCQ*IJASFdA\"><field name=\"VAR\" id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"]!Ow/xb#c|t[-C4tT2;~\"><field name=\"VAR\" id=\"gc[OXAJ:}?^trO)TCN#S\">stemmer</field><field name=\"MEMBER\">stem</field><data>i:</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"USK=d/dWP/OpD7hGS(bA\"><field name=\"VAR\" id=\"K-[j~N(`e*]acWp:SRf{\">i</field></block></value></block></value></block></value></block></value><next><block type=\"text_print\" id=\"ZF#vpBn`Yo=184o-|m-j\"><value name=\"TEXT\"><shadow type=\"text\" id=\"{d5gr^GWdYLq.+2O~D1!\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_create_with\" id=\"*Ru~d3Ng[kEjZ~rN^@vf\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"YUUS}RShyry+ioEtv6q4\"><field name=\"VAR\" id=\"K-[j~N(`e*]acWp:SRf{\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"fQ8c6s3lO09lbBM?s2zm\"><field name=\"VAR\" id=\"U`QjFH|a^qhm])]1I$;@\">pos</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"OL/x56hpr7m}u:$XhDR*\"><field name=\"VAR\" id=\"aZCw6y4p!sn)Cw$X1ojp\">lemmatizer</field><field name=\"MEMBER\">lemmatize</field><data>i:</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\":TvIV]_46$E8Ap#;%]*R\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"dummyOutputCodeBlock\" id=\"=]1kVhsPh30IaTigFV%~\"><field name=\"CODE\">i[0]</field></block></value><value name=\"ADD1\"><block type=\"dummyOutputCodeBlock\" id=\"jXZ(Gvks.QEs2QV_H3=6\"><field name=\"CODE\">map_tag(i[1])</field></block></value></block></value></block></value></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 3 differences in how words are handled:\n",
    "\n",
    "- `fli` vs `fly`\n",
    "- `is` vs `be`\n",
    "- `ach` vs `ache`\n",
    "\n",
    "These are consistent with our expectations: **stemmers can return nonwords and miss base forms for irregular forms.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionary and regular expression tagging\n",
    "\n",
    "So far the word transformations we've looked at are relatively theory neutral (if you'll excuse the particular tagset used in part of speech tagging).\n",
    "However, sometimes we want to do tagging according to our own theory.\n",
    "NLTK has some utility classes that allow us to define our own taggers fairly simply, using either dictionaries or regular expressions.\n",
    "\n",
    "We've glancingly looked at dictionaries before: a **dictionary** is a variable that holds other variables, like a **list**.\n",
    "The difference between a dictionary and a list is that you can get things by **position in a list** but you get things by **name in a dictionary**.\n",
    "The name is typically called a **key** and the thing you stored is called a **value**.\n",
    "We can define a dictionary in many different ways, but the most convenient for us is to start with a list of tuples representing (key, value) pairs and then run the list through `dict` to make a dictionary.\n",
    "\n",
    "Let's make a simple dictionary-based tagger that can work on our running example.\n",
    "\n",
    "| Word | Tag    |\n",
    "|:--------|:-------|\n",
    "| like      | Affect   |\n",
    "| aching     | Affect   |\n",
    "| I     | Person   |\n",
    "| my     | Person   |\n",
    "\n",
    "It's important to note that there's no limit on the number of words or classes we could use in the dictionary.\n",
    "\n",
    "Create the dictionary and give it to the tagger to use:\n",
    "\n",
    "- Set `dictionary` to dict of a list of a list containing\n",
    "    - A tuple for each entry in the table, where the first element (Word) is the key and the second element (Tag) is the value (both as strings)\n",
    "- Set `tagger` to with `nltk` create `UnigramTagger` using freestyle `model=dictionary`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = dict([('like','Affect'), ('aching','Affect'), ('I','Person'), ('my','Person')])\n",
    "\n",
    "tagger = nltk.UnigramTagger(model=dictionary)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"4OkA/upbH!NKP#DwLQ}E\">dictionary</variable><variable id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</variable><variable id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</variable></variables><block type=\"variables_set\" id=\"tD^8gW0/~Y^(+;8=bg(_\" x=\"3\" y=\"279\"><field name=\"VAR\" id=\"4OkA/upbH!NKP#DwLQ}E\">dictionary</field><value name=\"VALUE\"><block type=\"dictBlock\" id=\"^[zw5Ls)E6hDn7R(_ozN\"><value name=\"x\"><block type=\"lists_create_with\" id=\"QT(9-/+45@YdXbG@Y+,T\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"lists_create_with\" id=\"#-x[a7w92yQ,!{FJ!@CB\"><mutation items=\"4\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"},c(jbT(Ig-):JcYJp.8\"><value name=\"FIRST\"><block type=\"text\" id=\"aN}f]yCy6Eqrj8{*[JS3\"><field name=\"TEXT\">like</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"-qFHPe8Xx;V9gr!zl,;1\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"52qn-~6K-x*LYt]}zc]m\"><value name=\"FIRST\"><block type=\"text\" id=\"[24JJpm7GAb$R@xde|ta\"><field name=\"TEXT\">aching</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"VyD|N4:w$3l*#;].w[j/\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD2\"><block type=\"tupleBlock\" id=\"K/+d_/@];GR|2};Bzkm.\"><value name=\"FIRST\"><block type=\"text\" id=\"V^aF-50?5E}ZJEy@[iGv\"><field name=\"TEXT\">I</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"E*Bzdp@4_.rVAjST%ORO\"><field name=\"TEXT\">Person</field></block></value></block></value><value name=\"ADD3\"><block type=\"tupleBlock\" id=\"mlIYIFMZ!8NRVirF6iz$\"><value name=\"FIRST\"><block type=\"text\" id=\":^z1a|kk5RxaN9SUDu:s\"><field name=\"TEXT\">my</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"5/5|fI1h7B`IBxbU!e`J\"><field name=\"TEXT\">Person</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_set\" id=\"a{z?s[rNDGmZyw8Z*hj~\" x=\"32\" y=\"466\"><field name=\"VAR\" id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"2zq,D/1d1OCD6b)%vD,4\"><field name=\"VAR\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><field name=\"MEMBER\">UnigramTagger</field><data>nltk:UnigramTagger</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"|CGNfhm5=]nr8kB!X*ED\"><field name=\"CODE\">model=dictionary</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now tag our text:\n",
    "\n",
    "- with `tagger` do `tag` using `words`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('I', 'Person'), ('like', 'Affect'), ('to', None), ('fly', None), ('more', None), ('than', None), ('a', None), ('fly', None), ('in', None), ('the', None), ('ointment', None), ('.', None), ('My', None), ('tooth', None), ('is', None), ('aching', 'Affect'), ('.', None)]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger.tag(words)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</variable><variable id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</variable></variables><block type=\"varDoMethod\" id=\"6GlJD++{XCVAkG_`xtUK\" x=\"8\" y=\"188\"><field name=\"VAR\" id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</field><field name=\"MEMBER\">tag</field><data>tagger:tag</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"C?S#j}!jz#@zPn]p*(id\"><field name=\"VAR\" id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we were only interested in words in our dictionary, we could filter out all words matching `None`, which is the default tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we like the idea of dictionary tagging, but we want to capture multiple word forms that resolve to the same base word.\n",
    "One way to do this would be to use stemming/lemmatization first, followed by a dictionary tagger defined on the stems/lemmas.\n",
    "Another way is to combine the effect of stemming/lemmatization with the dictionary using **regular expressions**.\n",
    "\n",
    "Regular expressions are a notation for searching text for matching text.\n",
    "You've seen them before as wildcards, e.g. `*` or `.*`, however they can be much more complicated than that, e.g. `^H.+` matches an `H` at the start of a line, followed by at least one other character.\n",
    "There's more to regular expressions than makes sense for us to try to cover right now. [Here is a cheat sheet that covers the essentials](http://web.mit.edu/hackl/www/lab/turkshop/slides/regex-cheatsheet.pdf).\n",
    "\n",
    "A well-known program called [LIWC](http://liwc.wpengine.com/) defines [categories of words based on regular expressions](https://repositories.lib.utexas.edu/bitstream/handle/2152/31333/LIWC2015_LanguageManual.pdf)), and these are commonly used in the social sciences to transform words into more meaningful categories.\n",
    "We're going to use regular expressions to partially replicate the LIWC category of `Affect`, i.e. we will tag words that represent affect/emotion.\n",
    "\n",
    "The process for making a tagger based on regular expressions is nearly identical to making one with a dictionary; the only difference is that `dict` is not used, and regular expressions are used instead of words.\n",
    "\n",
    "Make the tagger:\n",
    "\n",
    "- Set `patterns` to a list of tuples where the first element is a regular expression (see below) and the second element is `\"Affect\"`.\n",
    "    - Regular expressions: `\"abandon.*\"`, `\"abuse.*\"`, `\"ache.*\"`, `\"aching\"`, `\"active.*\"`, `\"abusive\"`\n",
    "- Set tagger to with `nltk` create `RegexpTagger` using patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns = [('abandon.*','Affect'), ('abuse.*','Affect'), ('ache.*','Affect'), ('aching','Affect'), ('active.*','Affect'), ('abusive','Affect')]\n",
    "\n",
    "tagger = nltk.RegexpTagger(patterns)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</variable><variable id=\"M^Y}Gmy;f?HQ$[nPsp+j\">patterns</variable><variable id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</variable></variables><block type=\"variables_set\" id=\"pdC#N(zVL|0sG6AC7l8e\" x=\"33\" y=\"242\"><field name=\"VAR\" id=\"M^Y}Gmy;f?HQ$[nPsp+j\">patterns</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"wg#F}P#qId-L/guUEp0^\"><mutation items=\"6\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"OyL93(-/sW$j3L~Inmm^\"><value name=\"FIRST\"><block type=\"text\" id=\"JFvZDY/LnafV2UUiY*Tm\"><field name=\"TEXT\">abandon.*</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"|6Gh[or6uHrv9%.xl*5Q\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"Q/vqLzK/*IitkMbb{xEC\"><value name=\"FIRST\"><block type=\"text\" id=\"PuxeBwe(0s4qh_20{l/9\"><field name=\"TEXT\">abuse.*</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"KXnCDfJ}Mo#4}nBxRpK*\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD2\"><block type=\"tupleBlock\" id=\"G9U%JaEabqM96?9hM#?q\"><value name=\"FIRST\"><block type=\"text\" id=\"K0C3Wc7C?O{*c9SfM2}#\"><field name=\"TEXT\">ache.*</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"{@o7nd6IP]S3KF@J$^lQ\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD3\"><block type=\"tupleBlock\" id=\"L^Oo$tYEnwon]gG_Gf%Z\"><value name=\"FIRST\"><block type=\"text\" id=\"B[XQ}(p3d|sUg#ExW{7P\"><field name=\"TEXT\">aching</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"7!7Grc6qL.S3Q~.]3*q]\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD4\"><block type=\"tupleBlock\" id=\"a|B]ra~za,/=:;T)rOSg\"><value name=\"FIRST\"><block type=\"text\" id=\"t0c(b{85rw!br;Uphp.~\"><field name=\"TEXT\">active.*</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"QU?|cn~Ji~hK=BON6#51\"><field name=\"TEXT\">Affect</field></block></value></block></value><value name=\"ADD5\"><block type=\"tupleBlock\" id=\"/a,s]-:tc1A2b@ils5zO\"><value name=\"FIRST\"><block type=\"text\" id=\"yp|+wHafRYv4NYRKA-y8\"><field name=\"TEXT\">abusive</field></block></value><value name=\"SECOND\"><block type=\"text\" id=\"TP,b,9CAIEh/77[M)1E:\"><field name=\"TEXT\">Affect</field></block></value></block></value></block></value></block><block type=\"variables_set\" id=\"9cJ2Pr@IscZEuDldNW:|\" x=\"21\" y=\"523\"><field name=\"VAR\" id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\".e+S!DKKN=zd@95a[MLp\"><field name=\"VAR\" id=\"sHgTZgsI9vJQ3M#lu+,B\">nltk</field><field name=\"MEMBER\">RegexpTagger</field><data>nltk:RegexpTagger</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"(|^:5?A~QUcLCcB1r+}`\"><field name=\"VAR\" id=\"M^Y}Gmy;f?HQ$[nPsp+j\">patterns</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now tag our text:\n",
    "\n",
    "- with `tagger` do `tag` using `words`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('I', None), ('like', None), ('to', None), ('fly', None), ('more', None), ('than', None), ('a', None), ('fly', None), ('in', None), ('the', None), ('ointment', None), ('.', None), ('My', None), ('tooth', None), ('is', None), ('aching', 'Affect'), ('.', None)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger.tag(words)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</variable><variable id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</variable></variables><block type=\"varDoMethod\" id=\"6GlJD++{XCVAkG_`xtUK\" x=\"8\" y=\"188\"><field name=\"VAR\" id=\"{3t%G%Q5rLV)rz:2c,Yg\">tagger</field><field name=\"MEMBER\">tag</field><data>tagger:tag</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"C?S#j}!jz#@zPn]p*(id\"><field name=\"VAR\" id=\"cc)LJ@o:#+Q/bjV+nw7E\">words</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, `None` is the default tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summing up\n",
    "\n",
    "We've covered various transformations that operate on single words.\n",
    "Once you've made these transformations, you could do length- or distribution-based descriptive statistics on the transformations (e.g., on the resulting tags), or you could use them as input for vectorization, either instead of the original words or in addition to the original words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}