Latent-variable-vectorization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Latent variable vectorization\n",
    "\n",
    "We previously discussed vectorization, by which words are converted into numeric features based on their distribution in a text.\n",
    "We've seen that we can perform vectorization on words, transformations on words, or transformations on multi-word units.\n",
    "However, in all of these cases, the vectorization is ultimately based on counts in documents (where a document can be a sentence, paragraph, or something longer).\n",
    "As a result, the **length of the vectors** or equivalently **the dimension of the vector space**.\n",
    "\n",
    "What if we could represent the text vectors in a lower dimensional space, especially one where the dimensions correspond to latent variables of interest rather than document boundaries?\n",
    "There are two reasons why this could be a good idea.\n",
    "First, our vector spaces become extremely large as our text collections and vocabulary get bigger, which makes computation more costly.\n",
    "Second, if we can re-represent our vector space according to latent variables of interest, the resulting vectors should be even better features in our predictive models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn\n",
    "\n",
    "You will learn various methods for representing a vector space as a lower dimensional space of latent variables.\n",
    "  \n",
    "We will cover:\n",
    "\n",
    "- Latent semantic analysis\n",
    "- Latent Dirichlet allocation\n",
    "- Word2vec/Doc2vec\n",
    "- Contextual embeddings with BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use latent variable vectorization\n",
    "\n",
    "Vectorization based on latent variables generally produces better features than vectorization based on counts alone.\n",
    "However, it can be computationally costly to construct vectorizations based on latent variables, and if you use a precomputed model, it's possible words in your text will have no associated representation in the model.\n",
    "A good strategy is to try a precomputed model and compare to count-based vectorization; if the precomputed model is worse, you can consider if the problem is a mismatch with your text and whether it is worth correcting it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Latent semantic analysis\n",
    "\n",
    "[Latent semantic analysis (LSA; also known as latent semantic indexing)](https://en.wikipedia.org/wiki/Latent_semantic_analysis) is an old but still popular approach to latent variable vectorization.\n",
    "We start with it because it is based on the term-document matrix - LSA simply applies truncated [singular value decomposition (SVD)](https://en.wikipedia.org/wiki/Principal_component_analysis) to the term-document matrix.\n",
    "If you are unfamiliar with SVD, the basic idea is that it finds a low-dimensional approximation of the original matrix (in the least squares sense) ranked by the dimensions of the approximation (i.e. the first dimension captures most of the signal, then the second, and so on).\n",
    "\n",
    "Here and throughout this notebook, we'll make extensive use of a new library, [gensim](https://radimrehurek.com/gensim/), which provides support for a wide variety of vector space methods.\n",
    "Importantly, gensim provides wrappers for sklearn, so you can call gensim transformations as part of sklearn pipelines (i.e. the same way you've already called `CountVectorizer`).\n",
    "This is extremely useful, because it allows us to use gensim without deeply going into its native API, which is slightly unusual.\n",
    "Note these wrappers were [deprecated in v4 of gensim](https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4), so they won't work with the latest version.\n",
    "\n",
    "Let's use our previous example from [Vectorization](Vectorization-weighting.ipynb) so we can compare results:\n",
    "\n",
    "- import `gensim.sklearn_api` as `sklearn_api`\n",
    "- import `sklearn.pipeline` as `pipeline`\n",
    "- Set `texts` to a list containing `\"dogs chase cats\"`, `\"cats chase mice\"`, `\"mice eat cheese\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.sklearn_api as sklearn_api\n",
    "import sklearn.pipeline as pipeline\n",
    "texts = ['dogs chase cats', 'cats chase mice', 'mice eat cheese']\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</variable><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable></variables><block type=\"importAs\" id=\"ol,.vQWu()Zr?H$[$e~,\" x=\"-155\" y=\"-53\"><field name=\"libraryName\">gensim.sklearn_api</field><field name=\"libraryAlias\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><next><block type=\"importAs\" id=\"eW9]EDTCnI!FSOiJt(Zc\"><field name=\"libraryName\">sklearn.pipeline</field><field name=\"libraryAlias\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field><next><block type=\"variables_set\" id=\"!vG9N/84G5VO1Doa%O2:\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"NP?[gln!A4vn(*7OSuu=\"><mutation items=\"3\"></mutation><value name=\"ADD0\"><block type=\"text\" id=\"h+]`Selu9A_[n/z||T-F\"><field name=\"TEXT\">dogs chase cats</field></block></value><value name=\"ADD1\"><block type=\"text\" id=\"Lh_U+o=L0ZA;B)X7eHuG\"><field name=\"TEXT\">cats chase mice</field></block></value><value name=\"ADD2\"><block type=\"text\" id=\"|U1sSWnAauA7EXm.erIz\"><field name=\"TEXT\">mice eat cheese</field></block></value></block></value></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim assumes the input texts are in its [native bag of words format (BOW)](https://radimrehurek.com/gensim_3.8.3/sklearn_api/text2bow.html) so we need a pipeline that starts with the BOW transformation followed by the LSA transformation.\n",
    "We can specify the number of dimensions/latent variables we want using `num_topics`.\n",
    "\n",
    "- Set `pipe` to with `pipeline` create `Pipeline` using a list with a list of tuples inside it:\n",
    "    - `\"bow\"` and with `sklearn_api` create `Text2BowTransformer`\n",
    "    - `\"lsi\"` and with `sklearn_api` create `LsiTransformer` using `num_topics=5`\n",
    "\n",
    "*Note: 5 is a small number so we can display the results in Jupyter and compare to standard vectorization.\n",
    "In practice, you want 50 to 1000 dimensions.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = pipeline.Pipeline([('bow',(sklearn_api.Text2BowTransformer())), ('lsi',(sklearn_api.LsiTransformer(num_topics=5)))])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</variable><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable><variable id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</variable></variables><block type=\"variables_set\" id=\"Q10/rZu8e=AvH:wt~C}j\" x=\"-112\" y=\"-35\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"BzYWJLXq~9j,;R*._61a\"><field name=\"VAR\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field><field name=\"MEMBER\">Pipeline</field><data>pipeline:Pipeline</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"`1ELRLa]NxLgeIwsaDFk\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"lists_create_with\" id=\"Mm:2ZbCM3#-T*f)I5z36\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"$R.~F/tJt1N@A%_O77vJ\"><value name=\"FIRST\"><block type=\"text\" id=\".7_?X@cUZ^Rse9gf#fO1\"><field name=\"TEXT\">bow</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"[ikM.(mL15^j{iu4w1[K\"><field name=\"VAR\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><field name=\"MEMBER\">Text2BowTransformer</field><data>sklearn_api:Text2BowTransformer</data></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"ol+s}#MgI+[/XblBOn5K\"><value name=\"FIRST\"><block type=\"text\" id=\",~_OzX34K[?bBb5t;bDE\"><field name=\"TEXT\">lsi</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"FSrb=o-]zbmGtA:7V~6n\"><field name=\"VAR\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><field name=\"MEMBER\">LsiTransformer</field><data>sklearn_api:LsiTransformer</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"8@1/F)hRELZ~c^Hm:i/U\"><field name=\"CODE\">num_topics=5</field></block></value></block></value></block></value></block></value></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply `pipe` to our list of texts and store the result:\n",
    "\n",
    "- Set `matrix` to with `pipe` do `fit_transform` using `texts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = pipe.fit_transform(texts)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</variable><variable id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable></variables><block type=\"variables_set\" id=\"Soedj}vxx];/oJ=F.;0B\" x=\"5\" y=\"-113\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"fb9bsZObm*gRL5/,}7`M\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><field name=\"MEMBER\">fit_transform</field><data>pipe:fit_transform</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"/GWGl`-g^F6Jc.1/@6(E\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The work has been done, but to get Jupyter to display it, we have to use a special function:\n",
    "\n",
    "- with `matrix` do `tolist` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.4472136497497559, -0.7745966911315918, 0.5527864098548889, 0.0, 0.0],\n",
       " [1.6180340051651, 0.0, -0.6180340051651001, 0.0, 0.0],\n",
       " [0.7236068248748779, 1.5491933822631836, 0.27639320492744446, 0.0, 0.0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix.tolist()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</variable></variables><block type=\"varDoMethod\" id=\"65K2,gM_O0r}-G5@hHhv\" x=\"8\" y=\"-124\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><field name=\"MEMBER\">tolist</field><data>matrix:tolist</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our original term document matrix was\n",
    "```\n",
    "matrix([[1, 1, 0, 1, 0, 0],\n",
    "        [1, 1, 0, 0, 0, 1],\n",
    "        [0, 0, 1, 0, 1, 1]])\n",
    "```\n",
    "Interestingly our new matrix, though it has 5 dimensions, has zeros in the last two dimensions.\n",
    "This suggests that is is properly a 3 dimensional space.\n",
    "If these dimensions are latent variables, what do they mean?\n",
    "In general, we can't expect to recover the meaning of latent dimensions directly or exactly.\n",
    "If you are interested in exploring what latent dimensions mean more, you should check out gensims `print_topics` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Latent Dirichlet allocation\n",
    "\n",
    "[Latent Dirichlet allocation (LDA)](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) is similar to LSA in spirit but has its basis in generative Bayesian modeling rather than linear algebra.\n",
    "LDA also looks at word counts in documents but supposes that each document is a mixture of latent topics and that each word in a document is generated from one of these topic distributions.\n",
    "Like LSA, there is a free parameter for the number of topics, which correspond to the latent variables/dimensions of the space.\n",
    "\n",
    "We can use the [same approach for LDA](https://radimrehurek.com/gensim_3.8.3/sklearn_api/ldamodel.html) as LSA:\n",
    "\n",
    "- Set `pipe` to with `pipeline` create `Pipeline` using a list with a list of tuples inside it:\n",
    "    - `\"bow\"` and with `sklearn_api` create `Text2BowTransformer`\n",
    "    - `\"lda\"` and with `sklearn_api` create `LdaTransformer` using `num_topics=5`\n",
    "- Set `matrix` to with `pipe` do `fit_transform` using `texts`\n",
    "- with `matrix` do `tolist`\n",
    "\n",
    "*Note: You may wish to copy the blocks from above into the cell below and change as appropriate.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0.7987903952598572,\n",
       "  0.05001029744744301,\n",
       "  0.050049956887960434,\n",
       "  0.05109935253858566,\n",
       "  0.05004999786615372],\n",
       " [0.05107072368264198,\n",
       "  0.05050072446465492,\n",
       "  0.050051089376211166,\n",
       "  0.7983267307281494,\n",
       "  0.050050731748342514],\n",
       " [0.05001014098525047,\n",
       "  0.7993845343589783,\n",
       "  0.05004967376589775,\n",
       "  0.050505951046943665,\n",
       "  0.05004971846938133]]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = pipeline.Pipeline([('bow',(sklearn_api.Text2BowTransformer())), ('lda',(sklearn_api.LdaTransformer(num_topics=5)))])\n",
    "matrix = pipe.fit_transform(texts)\n",
    "\n",
    "matrix.tolist()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</variable><variable id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</variable><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable><variable id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</variable></variables><block type=\"variables_set\" id=\"lEf:wYereJ`%x9cBnyUH\" x=\"-112\" y=\"-35\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"~__@Q?rrcL=fN{LS#2QO\"><field name=\"VAR\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field><field name=\"MEMBER\">Pipeline</field><data>pipeline:Pipeline</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"5KCj?nw*.J7B32-kAA;5\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"lists_create_with\" id=\"4*mK7Q*4HeN+r[bzpw||\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"o+}KY@XCLPp6[S-}un%s\"><value name=\"FIRST\"><block type=\"text\" id=\"57q*]bSD+qAXEN{Ig}g3\"><field name=\"TEXT\">bow</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"e~+AGc+|/Z8!ji]}Ag0y\"><field name=\"VAR\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><field name=\"MEMBER\">Text2BowTransformer</field><data>sklearn_api:Text2BowTransformer</data></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"20TLY`Q[[fy-q,?)%*^%\"><value name=\"FIRST\"><block type=\"text\" id=\"6#GWtL1~wZk~r|Z@ar7X\"><field name=\"TEXT\">lda</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"!F{!#]k~{2Ng9LtAV{B.\"><field name=\"VAR\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><field name=\"MEMBER\">LdaTransformer</field><data>sklearn_api:LdaTransformer</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"o%*8,({(+E4}pIM4{F@%\"><field name=\"CODE\">num_topics=5</field></block></value></block></value></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"sN{4|UIN/TUx8:5/q+2?\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"~K2R0foQ#Ft^nRAf!=}i\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><field name=\"MEMBER\">fit_transform</field><data>pipe:fit_transform</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"(~K-t(4?@N@yN-,D9PdS\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></next></block><block type=\"varDoMethod\" id=\"+/)y+{slS+=9/(oGbYc%\" x=\"-109\" y=\"126\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><field name=\"MEMBER\">tolist</field><data>matrix:tolist</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result this time is both interesting and less impressive.\n",
    "It seems that LDA has created a dummy code with one large value in each document in a different position.\n",
    "This is perhaps a bit of an unfair conclusion, because methods like this need a lot of text in practice to work well.\n",
    "However, it does serve to illustrate the difference between LSA and LDA, as well as the importance of choosing an appropriate number of dimensions when doing LDA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2vec/Doc2vec\n",
    "\n",
    "[Word2vec (w2v)](https://en.wikipedia.org/wiki/Word2vec) is different from LSA/LDA in that it does not start with term-document matrix and then find latent dimensions but rather tries to predict words in context using a neural network.\n",
    "Therefore while LSA/LDA can be described as latent vectorization based on counts, w2v can be described as latent vectorization based on predicting words in context.\n",
    "w2v and other prediction-based methods have both better scaling properties than count-based methods as well as better predictive qualities when trained with large amounts of text.\n",
    "Doc2vec (d2v) is based on a similar idea as w2v, but predicts the words in the document using a vector.\n",
    "D2v is more appropriate for our use here, because it gives us one vector per text rather than one vector per word.\n",
    "\n",
    "We can again use the [same approach for d2v](https://radimrehurek.com/gensim_3.8.3/models/doc2vec.html#gensim.models.doc2vec.Doc2Vec) in gensim, but with a few tweaks.\n",
    "\n",
    "First, the d2v implementation in `sklearn_api` [isn't entirely compatible with sklearn](https://github.com/RaRe-Technologies/gensim/issues/2403), so we need a special preprocessor to go in our pipeline in front of it (kind of like `Text2BowTransformer`).\n",
    "This code was adapted from [here](https://github.com/alex2awesome/gensim-sklearn-tutorial/blob/master/notebooks/gensim-in-sklearn-pipelines.ipynb); if you are interested in doing more complex things with `gensim`/`sklearn` pipelines, it has some good examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, MetaEstimatorMixin\n",
    "import nltk\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "class MyTokenizer(BaseEstimator, MetaEstimatorMixin):\n",
    "    \"\"\"Tokenize input strings using NLTK.\"\"\"\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        #This is where you'd put any custom NLTK processing\n",
    "        #The final output must be a list of word lists\n",
    "        return [nltk.word_tokenize(t) for t in X]\n",
    "    \n",
    "    def fit_transform(self, X, y=None):\n",
    "        return self.fit(X, y).transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you execute the above code, do this:\n",
    "\n",
    "- Set `pipe` to with `pipeline` create `Pipeline` using a list with a list of tuples inside it:\n",
    "    - `\"mytokens\"` and freestyle `MyTokenizer()`\n",
    "    - `\"d2v\"` and with `sklearn_api` create `D2VTransformer` using freestyle `min_count=1, size=5`\n",
    "- Set `matrix` to with `pipe` do `fit_transform` using `texts`\n",
    "- with `matrix` do `tolist`\n",
    "\n",
    "*Notes:*\n",
    "\n",
    "- *`min_count=1` is needed for our small example and prevents low frequency word filtering*\n",
    "- *You may wish to copy the blocks from above into the cell below and change as appropriate.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0.07412126660346985,\n",
       "  -0.0947391465306282,\n",
       "  -0.04089774191379547,\n",
       "  -0.0947369858622551,\n",
       "  0.03851011022925377],\n",
       " [-0.050562601536512375,\n",
       "  -0.06402575224637985,\n",
       "  0.09507104754447937,\n",
       "  0.08805355429649353,\n",
       "  0.056405507028102875],\n",
       " [0.016094354912638664,\n",
       "  0.0790107399225235,\n",
       "  -0.09235618263483047,\n",
       "  0.029193880036473274,\n",
       "  -0.07684177905321121]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = pipeline.Pipeline([('mytokens',MyTokenizer()), ('d2v',(sklearn_api.D2VTransformer(min_count=1, size=5)))])\n",
    "matrix = pipe.fit_transform(texts)\n",
    "\n",
    "matrix.tolist()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</variable><variable id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</variable><variable id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable><variable id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</variable></variables><block type=\"variables_set\" id=\"lEf:wYereJ`%x9cBnyUH\" x=\"-112\" y=\"-35\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"~__@Q?rrcL=fN{LS#2QO\"><field name=\"VAR\" id=\"c:,wNTq#akK[c:VY8h$A\">pipeline</field><field name=\"MEMBER\">Pipeline</field><data>pipeline:Pipeline</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"5KCj?nw*.J7B32-kAA;5\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"lists_create_with\" id=\"4*mK7Q*4HeN+r[bzpw||\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"tupleBlock\" id=\"u3Qo)%9SI%,.?qmt(!Q!\"><value name=\"FIRST\"><block type=\"text\" id=\"ts34moP=[t?7B9.f]q6*\"><field name=\"TEXT\">mytokens</field></block></value><value name=\"SECOND\"><block type=\"dummyOutputCodeBlock\" id=\";TiG`q;b|kf`*p0o4@=N\"><field name=\"CODE\">MyTokenizer()</field></block></value></block></value><value name=\"ADD1\"><block type=\"tupleBlock\" id=\"20TLY`Q[[fy-q,?)%*^%\"><value name=\"FIRST\"><block type=\"text\" id=\"6#GWtL1~wZk~r|Z@ar7X\"><field name=\"TEXT\">d2v</field></block></value><value name=\"SECOND\"><block type=\"varCreateObject\" id=\"!F{!#]k~{2Ng9LtAV{B.\"><field name=\"VAR\" id=\"+YmMk1^)_8M1pm|fI/@(\">sklearn_api</field><field name=\"MEMBER\">D2VTransformer</field><data>sklearn_api:D2VTransformer</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"o%*8,({(+E4}pIM4{F@%\"><field name=\"CODE\">min_count=1, size=5</field></block></value></block></value></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"sN{4|UIN/TUx8:5/q+2?\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"~K2R0foQ#Ft^nRAf!=}i\"><field name=\"VAR\" id=\"^ui0{[8zsM/[=}wzc`z6\">pipe</field><field name=\"MEMBER\">fit_transform</field><data>pipe:fit_transform</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"(~K-t(4?@N@yN-,D9PdS\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></next></block><block type=\"varDoMethod\" id=\"+/)y+{slS+=9/(oGbYc%\" x=\"-109\" y=\"126\"><field name=\"VAR\" id=\"S|H:+?;2rlq.$U$9H|(V\">matrix</field><field name=\"MEMBER\">tolist</field><data>matrix:tolist</data></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output seems even less structured than with LSA/LDA, but this is likely due to the small size of our corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretrained models\n",
    "\n",
    "Gensim has pretrained models for w2v and related word-level methods that you can download in order vectorize your text data.\n",
    "If you text data is small, you will very likely get a better result using these models than if you trained your own.\n",
    "However, to use these in a pipeline, you will need to combine the word vectors to get a single vector representing the document.\n",
    "`W2VTransformerDocLevel` [as shown here](https://github.com/alex2awesome/gensim-sklearn-tutorial/blob/master/notebooks/gensim-in-sklearn-pipelines.ipynb) gives a good start for how to do this.\n",
    "Another option would be to download the models but then [load them into spaCy](https://stackoverflow.com/questions/50466643/in-spacy-how-to-use-your-own-word2vec-model-created-in-gensim); you'll no longer be using them in an sklearn pipeline, but that's not a problem per se (see next section)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300', 'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', 'glove-wiki-gigaword-100', 'glove-wiki-gigaword-200', 'glove-wiki-gigaword-300', 'glove-twitter-25', 'glove-twitter-50', 'glove-twitter-100', 'glove-twitter-200', '__testing_word2vec-matrix-synopsis'])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gensim.downloader as api\n",
    "info = api.info() \n",
    "info['models'].keys() #lists available models; use 'corpora' to show corpora\n",
    "# model = api.load(\"word2vec-google-news-300\") #uncomment to run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contextual embeddings with BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above latent variable vectorization techniques are not context sensitive, meaning that they create a single global representation for each text.\n",
    "This is easiest to understand at the word level.\n",
    "Consider what happens when we have one vector for a word with multiple senses, e.g. `fly` - the vector carries some of the meaning of all of those senses at once.\n",
    "Contextual embeddings create vectors for words *in context*, so only the sense in a given context is represented (ideally speaking).\n",
    "\n",
    "We can work with contextual embeddings using an extension to spaCy called `spacy-transformers`.\n",
    "This software library integrates several transformer-based models (including BERT) into the spaCy pipeline.\n",
    "We will download a pre-trained model (~500 MB) and use it.\n",
    "\n",
    "Run the following in your **terminal**:\n",
    "\n",
    "- `python -m spacy download en_core_web_trf`\n",
    "\n",
    "*Note: You can [download a model with use glove vectors](https://spacy.io/models/en#en_core_web_lg), which are like w2v; to use w2v with spaCy, see the previous section.*\n",
    "\n",
    "Next import spacy:\n",
    "\n",
    "- import `spacy` as `spacy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy as spacy\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"wo4UApWDaefy4v}x]:aB\">spacy</variable></variables><block type=\"importAs\" id=\"_@8DH2rDVjgaG~;xITM8\" x=\"16\" y=\"10\"><field name=\"libraryName\">spacy</field><field name=\"libraryAlias\" id=\"wo4UApWDaefy4v}x]:aB\">spacy</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember spaCy creates an NLP pipleine which we typically call `nlp`.\n",
    "\n",
    "- Set `nlp` to with `spacy` do `load` using `\"en_core_web_trf\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_trf')\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\",E~J@5;3-)HPm.VY_#{n\">nlp</variable><variable id=\"wo4UApWDaefy4v}x]:aB\">spacy</variable></variables><block type=\"variables_set\" id=\"Cg^6n$`lG^`#lq}+~w70\" x=\"66\" y=\"128\"><field name=\"VAR\" id=\",E~J@5;3-)HPm.VY_#{n\">nlp</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"zFJKxKMRouc6,b.zH%bg\"><field name=\"VAR\" id=\"wo4UApWDaefy4v}x]:aB\">spacy</field><field name=\"MEMBER\">load</field><data>spacy:load</data><value name=\"INPUT\"><block type=\"text\" id=\"lU/b5URW5wf-ceQ1(+@h\"><field name=\"TEXT\">en_core_web_trf</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's process **all** our `texts`; to process a list of strings, use `pipe` instead of `__call__`.\n",
    "Since this gives use a list of docs, we use a comprehension to get the tensors out:\n",
    "\n",
    "- Set `docs` to list with `nlp` do `pip` using `texts`\n",
    "- Set `tensors` to list for each item `i` in `docs` yield freestyle `i._.trf_data.tensors[0][0]`\n",
    "<!-- freestyle `i.tensor` -->\n",
    "- freestyle `tensors[0].shape` (this is the tensor for `dogs chase cats`)\n",
    "\n",
    "*Note: The standard API to get a tensor would be `i.tensor` here, but that appears to be broken, so we are using a more low-level approach.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 768)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = list(nlp.pipe(texts))\n",
    "tensors = list(i._.trf_data.tensors[0][0] for i in docs)\n",
    "\n",
    "tensors[0].shape\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"(1d9)^aniU$`5DS[O+uR\">docs</variable><variable id=\"x6K1x,OPRAST^)f^a%Vj\">tensors</variable><variable id=\",E~J@5;3-)HPm.VY_#{n\">nlp</variable><variable id=\"E3]e9N:_cfl3`6*DrUyF\">texts</variable><variable id=\"pVbrAvyAR[G;F*6dkZSY\">i</variable></variables><block type=\"variables_set\" id=\"u95WS8!kR,R,R[p,F=Rr\" x=\"-208\" y=\"209\"><field name=\"VAR\" id=\"(1d9)^aniU$`5DS[O+uR\">docs</field><value name=\"VALUE\"><block type=\"listBlock\" id=\"}$cA{hd:Nw21b,|0_dr:\"><value name=\"x\"><block type=\"varDoMethod\" id=\"/BZ=-}Gx@Q2z9yeL(1/J\"><field name=\"VAR\" id=\",E~J@5;3-)HPm.VY_#{n\">nlp</field><field name=\"MEMBER\">pipe</field><data>nlp:pipe</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"SlM6-1y!se`Fu(bUP$cw\"><field name=\"VAR\" id=\"E3]e9N:_cfl3`6*DrUyF\">texts</field></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"2,Uip@po_tYY9..pRn$V\"><field name=\"VAR\" id=\"x6K1x,OPRAST^)f^a%Vj\">tensors</field><value name=\"VALUE\"><block type=\"listBlock\" id=\"_%)?4jR%~Q,t)`/ZuFjK\"><value name=\"x\"><block type=\"comprehensionForEach\" id=\"_6S/?i,DcBCPPFDC5j0L\"><field name=\"VAR\" id=\"pVbrAvyAR[G;F*6dkZSY\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"AIA;=K5r!F1O!6,^Fq]w\"><field name=\"VAR\" id=\"(1d9)^aniU$`5DS[O+uR\">docs</field></block></value><value name=\"YIELD\"><block type=\"dummyOutputCodeBlock\" id=\",fQpZ/OelblSWs/|=oJ;\"><field name=\"CODE\">i._.trf_data.tensors[0][0]</field></block></value></block></value></block></value></block></next></block><block type=\"dummyOutputCodeBlock\" id=\"KX:waQjl)oXb~$wEJrV9\" x=\"-209\" y=\"398\"><field name=\"CODE\">tensors[0].shape</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 3 words in the document, but we have 6 vectors in the tensor!\n",
    "The extra vectors are for hidden start/end tokens and *padding* (you would also see extra vectors for punctuation).\n",
    "\n",
    "So for each of the underlying tokens, we have a 768 long vector.\n",
    "However, we want a single vector to represent the entire text.\n",
    "\n",
    "To get a single vector for the sentence, we can sum them:\n",
    "\n",
    "- Set `docVectors` to list for each `i` in list `tensors` yield freestyle `i.sum(axis=0)`\n",
    "- Display length of `docVectors[0]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "768\n"
     ]
    }
   ],
   "source": [
    "docVectors = list(i.sum(axis=0) for i in tensors)\n",
    "print(len(docVectors[0]))\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"A(R^DIZnN1rq%retD=M}\">docVectors</variable><variable id=\"pVbrAvyAR[G;F*6dkZSY\">i</variable><variable id=\"x6K1x,OPRAST^)f^a%Vj\">tensors</variable></variables><block type=\"variables_set\" id=\";R3n-P@Dqot}l+^hp#P$\" x=\"-228\" y=\"83\"><field name=\"VAR\" id=\"A(R^DIZnN1rq%retD=M}\">docVectors</field><value name=\"VALUE\"><block type=\"listBlock\" id=\"[zCC[,S2fJV:|?IFn.=)\"><value name=\"x\"><block type=\"comprehensionForEach\" id=\"m9}iRBGi(s:6//.a0$)}\"><field name=\"VAR\" id=\"pVbrAvyAR[G;F*6dkZSY\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"rAQZ)8#UFR.uyW;IIfOG\"><field name=\"VAR\" id=\"x6K1x,OPRAST^)f^a%Vj\">tensors</field></block></value><value name=\"YIELD\"><block type=\"dummyOutputCodeBlock\" id=\"/JFA#Y^GO0kO6]bnc;z)\"><field name=\"CODE\">i.sum(axis=0)</field></block></value></block></value></block></value><next><block type=\"text_print\" id=\"2nv,M]vMI]$Aj^Cblv;w\"><value name=\"TEXT\"><shadow type=\"text\" id=\"L8Z]Ma}sD*%#tH]}MWZT\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_length\" id=\"d3I1*iP+~y]#(v|GWuZ)\"><value name=\"VALUE\"><block type=\"indexer\" id=\"?q-:QXg04X]qdSY?5_xc\"><field name=\"VAR\" id=\"A(R^DIZnN1rq%retD=M}\">docVectors</field><value name=\"INDEX\"><block type=\"math_number\" id=\"]u(NjCwP{7$?k7KX8Rkz\"><field name=\"NUM\">0</field></block></value></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why the `sum(axis=0)`? \n",
    "Because tensors can be summed in different directions, we had to tell it which direction to sum in. \n",
    "Otherwise we would have gotten a vector that was 6 long instead of 768.\n",
    "\n",
    "Now we have created contextual embeddings for our texts, just as in the other methods above. \n",
    "The major difference in usage is that we do not have this set up in an sklearn pipeline.\n",
    "However it shouldn't be that hard to use the sample code above to put it into a pipeline, or leave it as is and possibly copy the results back to a dataframe before running a model with the other variables in said dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- # docs = nlp(\"Apple shares rose on the news.\")\n",
    "# print(docs.tensor.shape)\n",
    "# print(docs.vector.shape)\n",
    "# data = docs._.trf_data\n",
    "# apple = docs[0]\n",
    "# import torch\n",
    "# torch.cuda.is_available()\n",
    "# spacy.prefer_gpu()\n",
    "# spacy.require_gpu()\n",
    "# import cupy\n",
    "# cupy.zeros((1,1))\n",
    "# data.tensors[0].shape #(1, 9, 768)\n",
    "# docs._.trf_data.tensors[0][0].shape #(9, 768)\n",
    "# data.tensors[0][0].sum(axis=0) #(768,) -->"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}