Vectorization-weighting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorization and weighting\n",
    "\n",
    "Transforming text into a distribution is useful for descriptive statistics, but it is also very useful for getting *features* from text that you can use in predictive models.\n",
    "Because these features come from a distribution, they are ultimately based on **counts**.\n",
    "As we have seen, frequent words with high counts are not necessarily good features, so it is common to further transform counts using **weights**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn\n",
    "\n",
    "You will learn about how distributions of text can be used to generate features for predictive models.\n",
    "  \n",
    "We will cover:\n",
    "\n",
    "- Vectorization\n",
    "- tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use vectorization and weighting\n",
    "\n",
    "Vectorization followed by weighting produces useful simple features. \n",
    "Creating a model with vectorized text features can be a good first model, or baseline, as you consider additional more complex features.\n",
    "You can build remarkably performant models for many applications using only vectorized text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorization\n",
    "\n",
    "Let's look at `tabulate` again, but this time with artificially small texts so the output fits in Jupyter.\n",
    "\n",
    "Start by importing `nltk`:\n",
    "\n",
    "- `import nltk as nltk`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk as nltk\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Wer,Q4C`j@I;1xVaLJBR\">nltk</variable></variables><block type=\"importAs\" id=\"BTn3GreO!LDoBjfD3-7R\" x=\"16\" y=\"10\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"Wer,Q4C`j@I;1xVaLJBR\">nltk</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next create a `ConditionalFreqDist` with a list of tuples, where 1/2/3 are the names of our samples: \n",
    "\n",
    "- Set `tinyCDF` to with `nltk` create `ConditionalFreqDist` using a freestyle block containing `[[(1,'dogs'),(1,'chase'),(1,'cats'),(2,'cats'),(2,'chase'),(2,'mice'),(3,'mice'),(3,'eat'),(3,'cheese')]]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tinyCDF = nltk.ConditionalFreqDist([(1,'dogs'),(1,'chase'),(1,'cats'),(2,'cats'),(2,'chase'),(2,'mice'),(3,'mice'),(3,'eat'),(3,'cheese')])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</variable><variable id=\"Wer,Q4C`j@I;1xVaLJBR\">nltk</variable></variables><block type=\"variables_set\" id=\"ZLnW_SEDhRyQVR,sN0JV\" x=\"-34\" y=\"225\"><field name=\"VAR\" id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"pf?[15c:jmz{U?v7-=0O\"><field name=\"VAR\" id=\"Wer,Q4C`j@I;1xVaLJBR\">nltk</field><field name=\"MEMBER\">ConditionalFreqDist</field><data>nltk:ConditionalFreqDist</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"%gO}d*#h.6j$DbYK#Mq`\"><field name=\"CODE\">[[(1,'dogs'),(1,'chase'),(1,'cats'),(2,'cats'),(2,'chase'),(2,'mice'),(3,'mice'),(3,'eat'),(3,'cheese')]]</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now do `tabulate`, which includes all samples and words by default:\n",
    "\n",
    "- with `tinyCDF` do `tabulate`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    cats  chase cheese   dogs    eat   mice \n",
      "1      1      1      0      1      0      0 \n",
      "2      1      1      0      0      0      1 \n",
      "3      0      0      1      0      1      1 \n"
     ]
    }
   ],
   "source": [
    "tinyCDF.tabulate()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</variable></variables><block type=\"varDoMethod\" id=\"*]]?sXKT(qy:/NP1`qWf\" x=\"-82\" y=\"188\"><field name=\"VAR\" id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</field><field name=\"MEMBER\">tabulate</field><data>tinyCDF:tabulate</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this output, which contains all words, we can easily see which texts contain which words and compare texts to each other (based on the words they contain) and compare words to each other (based on the texts that contain them).\n",
    "\n",
    "In other words, we can consider column vectors (e.g. both `cats` and `chase` are 1 1 0) as well as row vectors (`1` is 1 1 0 1 0 0 and `2` is      1      1      0      0      0      1 ).\n",
    "When we have vectors, we can use metrics for similarity to answer questions like \"how similar are `cats` and `chase`\"?\n",
    "\n",
    "A popular metric is **cosine**, which is based on the dot product.\n",
    "The dot product of `cats` and `chase` is the sum of the product of their first elements (1 and 1), their second elements (1 and 1), and their third elements (0 and 0), which sum to 2.\n",
    "As you can see, the maximum value of the dot product is unbounded - longer vectors will have larger values.\n",
    "To deal with this, cosine divides the dot product by the the lengths of the vectors involved:\n",
    "\n",
    "\\begin{equation*}\n",
    "{\\displaystyle {\\text{cosine similarity}}=\\cos(\\theta )={\\mathbf {A} \\cdot \\mathbf {B}  \\over \\|\\mathbf {A} \\|\\|\\mathbf {B} \\|}={\\frac {\\sum \\limits _{i=1}^{n}{A_{i}B_{i}}}{{\\sqrt {\\sum \\limits _{i=1}^{n}{A_{i}^{2}}}}{\\sqrt {\\sum \\limits _{i=1}^{n}{B_{i}^{2}}}}}}}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This kind of transformation on text is commonly called **vectorization**, by which we project words into a **vector space** based on counts across documents.\n",
    "The matrix of word counts across documents is called a **term-document matrix** (where term is a more generic sense of word).\n",
    "\n",
    "`ConditionalFreqDist` implicitly performs vectorization in `tabulate`, but `tabulate` only prints the vectorization, which is not useful for using the vectorization in further code.\n",
    "\n",
    "We can implement our own vectorization based on `ConditionalFreqDist` by noting two key things.\n",
    "First, we need to generate an ordered list of all words across all documents (these define our columns).\n",
    "Second, we need to loop over each sample (the rows) and use the list of all words to look up the count of a given word in that sample (this corresponds to an entry in the table).\n",
    "Let's do these two steps:\n",
    "\n",
    "- Set `allWords` to as sorted set a list with one element containing for each item `word` in list\n",
    "    - `tinyCDF`[`condition`]\n",
    "    - yield for each item `condition` in list\n",
    "        - with `tinyCDF` do `conditions`\n",
    "        - yield `word`\n",
    "- Display `allWords`\n",
    "\n",
    "*Remember you can split a nested comprehension into two steps if you need to, just store the first one in a variable and use that variable in the second one.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cats', 'chase', 'cheese', 'dogs', 'eat', 'mice']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allWords = sorted(set(word for condition in (tinyCDF.conditions()) for word in tinyCDF[condition]))\n",
    "\n",
    "allWords\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"`N`}VB)._T*Ri]X+{|B,\">allWords</variable><variable id=\"c|WH1pZH%^rWv9G-!to~\">word</variable><variable id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</variable><variable id=\"38/jAP)H-9MNFc-cEO/V\">condition</variable></variables><block type=\"variables_set\" id=\"yp(?$RM`dZ=_{uEbL0IE\" x=\"11\" y=\"230\"><field name=\"VAR\" id=\"`N`}VB)._T*Ri]X+{|B,\">allWords</field><value name=\"VALUE\"><block type=\"sortedBlock\" id=\"%wTG~4h5g5)4[#H3nq=f\"><value name=\"x\"><block type=\"setBlock\" id=\"[SrI?fXgcPAl+!w4vc6G\"><value name=\"x\"><block type=\"lists_create_with\" id=\"bqaxiZLk2-O3MQ-ru=Dt\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"jF9#))eep(BQ]tDW)%($\"><field name=\"VAR\" id=\"c|WH1pZH%^rWv9G-!to~\">word</field><value name=\"LIST\"><block type=\"indexer\" id=\"2Pd1bZ{Na!.*O+XErL-J\"><field name=\"VAR\" id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</field><value name=\"INDEX\"><block type=\"variables_get\" id=\"mF#Hb=maNUj}OcH[/xI:\"><field name=\"VAR\" id=\"38/jAP)H-9MNFc-cEO/V\">condition</field></block></value></block></value><value name=\"YIELD\"><block type=\"comprehensionForEach\" id=\"r]/g3M37%r7SL`8qJHu*\"><field name=\"VAR\" id=\"38/jAP)H-9MNFc-cEO/V\">condition</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"#4;f);LaiWPHhM6w_kR%\"><field name=\"VAR\" id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</field><field name=\"MEMBER\">conditions</field><data>tinyCDF:conditions</data></block></value><value name=\"YIELD\"><block type=\"variables_get\" id=\":IS7]{j!]ib1AVJd%q#s\"><field name=\"VAR\" id=\"c|WH1pZH%^rWv9G-!to~\">word</field></block></value></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"RV!$EMdy8OXAq9AA%{c#\" x=\"24\" y=\"394\"><field name=\"VAR\" id=\"`N`}VB)._T*Ri]X+{|B,\">allWords</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that every word in the text is represented exactly once.\n",
    "\n",
    "Now for each condition, dump out the counts for `allWords`.\n",
    "Remember that `tinyCDF` contains `FreqDist`s which contain words.\n",
    "So `tinyCDF[3]['cheese']` is equal to 1, the count of \"cheese\" in the third sample:\n",
    "\n",
    "- Set `conditionVectors` to a list with one element containing\n",
    "    - for each item `condition` in list with `tinyCDF` do `conditions`\n",
    "    - yield a list with one element containing\n",
    "        - for each item `word` in list `allWords`\n",
    "        - yield a freestyle containing `tinyCDF[condition][word]`\n",
    "- Display `conditionVectors`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 1, 0, 1, 0, 0], [1, 1, 0, 0, 0, 1], [0, 0, 1, 0, 1, 1]]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conditionVectors = [[tinyCDF[condition][word] for word in allWords] for condition in (tinyCDF.conditions())]\n",
    "\n",
    "conditionVectors\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"S3pa*LE;]w2dV[UZT-?C\">conditionVectors</variable><variable id=\"38/jAP)H-9MNFc-cEO/V\">condition</variable><variable id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</variable><variable id=\"c|WH1pZH%^rWv9G-!to~\">word</variable><variable id=\"`N`}VB)._T*Ri]X+{|B,\">allWords</variable></variables><block type=\"variables_set\" id=\"yp(?$RM`dZ=_{uEbL0IE\" x=\"11\" y=\"230\"><field name=\"VAR\" id=\"S3pa*LE;]w2dV[UZT-?C\">conditionVectors</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"bqaxiZLk2-O3MQ-ru=Dt\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"r]/g3M37%r7SL`8qJHu*\"><field name=\"VAR\" id=\"38/jAP)H-9MNFc-cEO/V\">condition</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"#4;f);LaiWPHhM6w_kR%\"><field name=\"VAR\" id=\"c[J=TW=ny|GCtDLK0iv3\">tinyCDF</field><field name=\"MEMBER\">conditions</field><data>tinyCDF:conditions</data></block></value><value name=\"YIELD\"><block type=\"lists_create_with\" id=\"2c6V}cpIYg=A`{JCd}fv\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"[Ht)y1ThJ5=$evU|Lf(T\"><field name=\"VAR\" id=\"c|WH1pZH%^rWv9G-!to~\">word</field><value name=\"LIST\"><block type=\"variables_get\" id=\"[s0ZYq*W~%Z%SVsjmaA]\"><field name=\"VAR\" id=\"`N`}VB)._T*Ri]X+{|B,\">allWords</field></block></value><value name=\"YIELD\"><block type=\"dummyOutputCodeBlock\" id=\"$q(0-Y!G3mkw7`El83;)\"><field name=\"CODE\">tinyCDF[condition][word]</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"RV!$EMdy8OXAq9AA%{c#\" x=\"24\" y=\"394\"><field name=\"VAR\" id=\"S3pa*LE;]w2dV[UZT-?C\">conditionVectors</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result should match the `tabulate` output from before, and now you can use the vectors in an analysis, e.g. bring them into `pandas`.\n",
    "\n",
    "Vectorization is a common enough operation that many libraries have a streamlined API for it.\n",
    "Let's look at `scikit-learn`'s API:\n",
    "\n",
    "- import `sklearn.feature_extraction.text` as `text`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.feature_extraction.text as text\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"y*%FH]Xz:N5?J=p7So4;\">text</variable></variables><block type=\"importAs\" id=\"oTX6-0d~y$]Xv#M)+_z!\" x=\"-33\" y=\"-322\"><field name=\"libraryName\">sklearn.feature_extraction.text</field><field name=\"libraryAlias\" id=\"y*%FH]Xz:N5?J=p7So4;\">text</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now create a `CountVectorizer`:\n",
    " \n",
    " - Set `vectorizer` to `with text create CountVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = text.CountVectorizer()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Xu=hAdiWJ.`f(n4Tn*5t\">vectorizer</variable><variable id=\"y*%FH]Xz:N5?J=p7So4;\">text</variable></variables><block type=\"variables_set\" id=\"`#yMT01y=4DItX`0~kI.\" x=\"10\" y=\"-259\"><field name=\"VAR\" id=\"Xu=hAdiWJ.`f(n4Tn*5t\">vectorizer</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"e+2wjiwOjX%pfxd5nKAj\"><field name=\"VAR\" id=\"y*%FH]Xz:N5?J=p7So4;\">text</field><field name=\"MEMBER\">CountVectorizer</field><data>text:CountVectorizer</data></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to apply `vectorizer` to a list of texts and store the result:\n",
    "\n",
    "- Set `texts` to a list containing `\"dogs chase cats\"`, `\"cats chase mice\"`, `\"mice eat cheese\"`\n",
    "- Set `matrix` to `with vectorizer do fit_transform` using `texts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = ['dogs chase cats', 'cats chase mice', 'mice eat cheese']\n",
    "\n",
    "matrix = vectorizer.fit_transform(texts)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable><variable id=\"g#vU}+I%b#efZeGj-i8*\">matrix</variable><variable id=\"Xu=hAdiWJ.`f(n4Tn*5t\">vectorizer</variable></variables><block type=\"variables_set\" id=\"!vG9N/84G5VO1Doa%O2:\" x=\"-17\" y=\"-268\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"NP?[gln!A4vn(*7OSuu=\"><mutation items=\"3\"></mutation><value name=\"ADD0\"><block type=\"text\" id=\"h+]`Selu9A_[n/z||T-F\"><field name=\"TEXT\">dogs chase cats</field></block></value><value name=\"ADD1\"><block type=\"text\" id=\"Lh_U+o=L0ZA;B)X7eHuG\"><field name=\"TEXT\">cats chase mice</field></block></value><value name=\"ADD2\"><block type=\"text\" id=\"|U1sSWnAauA7EXm.erIz\"><field name=\"TEXT\">mice eat cheese</field></block></value></block></value></block><block type=\"variables_set\" id=\"scYNV_rn?aY)~0.5mcLO\" x=\"-12\" y=\"-180\"><field name=\"VAR\" id=\"g#vU}+I%b#efZeGj-i8*\">matrix</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"]#ME2Y;blQoh|J?@,oYL\"><field name=\"VAR\" id=\"Xu=hAdiWJ.`f(n4Tn*5t\">vectorizer</field><field name=\"MEMBER\">fit_transform</field><data>vectorizer:fit_transform</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"`-S9z$LuTar;c3hNHrVR\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The work has been done, but to get Jupyter to display it, we have to use a special function:\n",
    "\n",
    "- with `matrix` do `todense` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[1, 1, 0, 1, 0, 0],\n",
       "        [1, 1, 0, 0, 0, 1],\n",
       "        [0, 0, 1, 0, 1, 1]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix.todense()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"g#vU}+I%b#efZeGj-i8*\">matrix</variable></variables><block type=\"varDoMethod\" id=\"#mfu5h62K1:KTU[5oTL(\" x=\"-25\" y=\"-134\"><field name=\"VAR\" id=\"g#vU}+I%b#efZeGj-i8*\">matrix</field><field name=\"MEMBER\">todense</field><data>matrix:todense</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `CountVectorizer` did all the tokenizing for us.\n",
    "All we had to do is break up the texts.\n",
    "\n",
    "You may be asking yourself why bother with NLTK at all for vectorizing - why not just use `sklearn`?\n",
    "The answer is that NLTK gives you more tools/more control over tokenizing and other operations you might want to perform.\n",
    "\n",
    "If you ever find yourself limited by [`sklearn`'s approach](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html), you can use the approach we described above with NLTK.\n",
    "If you are looking for a quick approach and don't really care about how words are tokenized, the `CountVectorizer` is a good way to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf-idf\n",
    "\n",
    "We've seen that most words are rare, and that the most frequent words tend to be unimportant (e.g. functions words like \"a\" or \"of\").\n",
    "As a result, when we vectorize documents, their counts don't correspond to importance as well as we'd like.\n",
    "To address this, it is common to **weight** the counts to bring them closer to our intuition of what importance is.\n",
    "\n",
    "[**Term frequency - inverse document frequency** or **tf-idf**](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)\n",
    "is perhaps the most famous and most popular way to weight a term-document matrix.\n",
    "\n",
    "There are two key intuitions behind tf-idf.\n",
    "First, if some documents are longer than others, they will have higher counts just because they have more words.\n",
    "If we divide the count of a word in the document by the total number of words in the document, we get a proportion such that all the term frequencies sum to 1.\n",
    "This is *term frequency*.\n",
    "Second, we are particularly interested in words that are distinctive.\n",
    "So if a word occurs in many documents, e.g. \"the\", we don't want to give it much weight, but if a word occurs in a few documents, we want to give it more weight, i.e., we want to weight a word *inversely* to the number of documents it appears in.\n",
    "This is *inverse document frequency*.\n",
    "Tf-idf weights each count in the term document matrix by replacing the count by the product of tf and idf for that term.\n",
    "\n",
    "Note that there are [many variations](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) of tf-idf, for example we might take the log of a word count or consider presence/absence instead of term frequency; there are comparable variations for document frequency.\n",
    "<!-- The upshot here is that tf-idf will make our potential features less noisy before we even send them to the classifier. -->\n",
    "\n",
    "To implement tf-idf, we'll use an `sklearn` pipeline where tf-idf follows vectorization.\n",
    "Pipelines are easier/simpler to code and, as you know, they allow us to calculate features from our training folds separately from our testing fold (if we are doing crossvalidation).\n",
    "\n",
    "Let's start by doing imports for the pipeline:\n",
    "\n",
    "- `import sklearn.pipeline as pipeline`\n",
    "<!-- - `import sklearn.feature_extraction.text as text` -->\n",
    "<!-- - `import sklearn.naive_bayes as naive_bayes` -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.pipeline as pipeline\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable></variables><block type=\"importAs\" id=\"NVlK(81jaZyoh0(2^Y$,\" x=\"-17\" y=\"-312\"><field name=\"libraryName\">sklearn.pipeline</field><field name=\"libraryAlias\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the pipeline:\n",
    "\n",
    "- Set `myPipeline` to `with pipeline create Pipeline using` a list with a list of tuples inside it:\n",
    "    - `\"vect\"` and `with text create CountVectorizer using` nothing\n",
    "    - `\"tfidf\"` and `with text create TfidfTransformer using` nothing\n",
    "<!-- - `\"clf\"` and `with naive_bayes create MultinomialNB using` nothing -->\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "myPipeline = pipeline.Pipeline([('vect',(text.CountVectorizer())), ('tfidf',(text.TfidfTransformer()))])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"^ui0{[8zsM/[=}wzc`z6\">myPipeline</variable><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable><variable id=\"y*%FH]Xz:N5?J=p7So4;\">text</variable></variables><block type=\"variables_set\" id=\"Q10/rZu8e=AvH:wt~C}j\" x=\"-112\" y=\"-35\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">myPipeline</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"BzYWJLXq~9j,;R*._61a\"><field name=\"VAR\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field><field name=\"MEMBER\">Pipeline</field><data>pipeline:Pipeline</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"`1ELRLa]NxLgeIwsaDFk\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"lists_create_with\" id=\"Mm:2ZbCM3#-T*f)I5z36\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"$R.~F/tJt1N@A%_O77vJ\"><value name=\"FIRST\"><block type=\"text\" id=\".7_?X@cUZ^Rse9gf#fO1\"><field name=\"TEXT\">vect</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"[ikM.(mL15^j{iu4w1[K\"><field name=\"VAR\" id=\"y*%FH]Xz:N5?J=p7So4;\">text</field><field name=\"MEMBER\">CountVectorizer</field><data>text:CountVectorizer</data></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"ol+s}#MgI+[/XblBOn5K\"><value name=\"FIRST\"><block type=\"text\" id=\",~_OzX34K[?bBb5t;bDE\"><field name=\"TEXT\">tfidf</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"FSrb=o-]zbmGtA:7V~6n\"><field name=\"VAR\" id=\"y*%FH]Xz:N5?J=p7So4;\">text</field><field name=\"MEMBER\">TfidfTransformer</field><data>text:TfidfTransformer</data></block></value></block></value></block></value></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply `myPipeline` to our list of texts and store the result:\n",
    "\n",
    "- Set `textsTfidf` to `with myPipeline do fit_transform` using `texts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "textsTfidf = myPipeline.fit_transform(texts)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Y7`4K#9-J~%uK701Csn|\">textsTfidf</variable><variable id=\"^ui0{[8zsM/[=}wzc`z6\">myPipeline</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable></variables><block type=\"variables_set\" id=\"K]ktAAQ+*#+ZUPFvm_T6\" x=\"-132\" y=\"-219\"><field name=\"VAR\" id=\"Y7`4K#9-J~%uK701Csn|\">textsTfidf</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"kiQae81EsGbn4josKloB\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">myPipeline</field><field name=\"MEMBER\">fit_transform</field><data>myPipeline:fit_transform</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"(`9BqByr_~|En(]S]a7n\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The work has been done, but to get Jupyter to display it, we have to use a special function:\n",
    "\n",
    "- with `textsTfidf` do `todense` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.51785612, 0.51785612, 0.        , 0.68091856, 0.        ,\n",
       "         0.        ],\n",
       "        [0.57735027, 0.57735027, 0.        , 0.        , 0.        ,\n",
       "         0.57735027],\n",
       "        [0.        , 0.        , 0.62276601, 0.        , 0.62276601,\n",
       "         0.4736296 ]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textsTfidf.todense()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Y7`4K#9-J~%uK701Csn|\">textsTfidf</variable></variables><block type=\"varDoMethod\" id=\"HN5eW[0`B(K_0U#B+j$}\" x=\"-132\" y=\"-134\"><field name=\"VAR\" id=\"Y7`4K#9-J~%uK701Csn|\">textsTfidf</field><field name=\"MEMBER\">todense</field><data>textsTfidf:todense</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our original term document matrix was\n",
    "```\n",
    "matrix([[1, 1, 0, 1, 0, 0],\n",
    "        [1, 1, 0, 0, 0, 1],\n",
    "        [0, 0, 1, 0, 1, 1]])\n",
    "```\n",
    "so as you can see, the tf-idf transform has downweighted the 1's but left the zero counts alone.\n",
    "\n",
    "If you want more control over the tf-idf weighting schemes than what `sklearn` affords, you can try the [`gensim` API](https://radimrehurek.com/gensim_3.8.3/sklearn_api/tfidf.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}