Preprocessing-text-PS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing text: Problem solving\n",
    "\n",
    "In this session, we'll work with Twitter data from NLTK.\n",
    "Execute the code below to download `twitter_samples` from `corpora`.\n",
    "Close the window when the download completes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import nltk\n",
    "sys.argv=['']\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "\n",
    "Start by importing `twitter_samples` from `nltk.corpus` and `nltk`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import twitter_samples\n",
    "import nltk as nltk\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable></variables><block type=\"importFrom\" id=\"Jzjqgf~f}NetD1+v]eM^\" x=\"16\" y=\"64\"><field name=\"libraryName\">nltk.corpus</field><field name=\"libraryAlias\" id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</field><next><block type=\"importAs\" id=\"Ub-RL@h_kQXn-V1vNr.e\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now get the raw text of `tweets.20150430-223406.json` and store it in variable `tweetsJson`.\n",
    "\n",
    "Show the substring of `tweetsJson` from its start to `3000`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"in_reply_to_status_id\": null, \"contributors\": null, \"filter_level\": \"low\", \"coordinates\": null, \"id_str\": \"593891099434983425\", \"favorited\": false, \"created_at\": \"Thu Apr 30 21:34:06 +0000 2015\", \"favorite_count\": 0, \"in_reply_to_status_id_str\": null, \"retweeted_status\": {\"in_reply_to_status_id\": null, \"contributors\": null, \"filter_level\": \"low\", \"coordinates\": null, \"id_str\": \"593885295323521025\", \"favorited\": false, \"created_at\": \"Thu Apr 30 21:11:02 +0000 2015\", \"favorite_count\": 3, \"in_reply_to_status_id_str\": null, \"in_reply_to_user_id\": null, \"in_reply_to_screen_name\": null, \"place\": null, \"id\": 593885295323521025, \"retweeted\": false, \"truncated\": false, \"text\": \"Indirect cost of the UK being in the EU is estimated to be costing Britain \\\\u00a3170 billion per year! #BetterOffOut #UKIP\", \"in_reply_to_user_id_str\": null, \"lang\": \"en\", \"retweet_count\": 11, \"user\": {\"statuses_count\": 4589, \"id_str\": \"929903647\", \"followers_count\": 2973, \"listed_count\": 39, \"geo_enabled\": false, \"created_at\": \"Tue Nov 06 14:56:03 +0000 2012\", \"favourites_count\": 2137, \"profile_sidebar_fill_color\": \"DDEEF6\", \"is_translator\": false, \"protected\": false, \"profile_text_color\": \"333333\", \"profile_background_tile\": true, \"following\": null, \"follow_request_sent\": null, \"contributors_enabled\": false, \"profile_image_url_https\": \"https://pbs.twimg.com/profile_images/592490695581159424/RHicaeez_normal.jpg\", \"profile_link_color\": \"6E0C6E\", \"screen_name\": \"KirkKus\", \"id\": 929903647, \"default_profile_image\": false, \"profile_background_image_url\": \"http://pbs.twimg.com/profile_background_images/464858470500417536/oVGrCW1A.jpeg\", \"utc_offset\": 3600, \"profile_use_background_image\": true, \"profile_banner_url\": \"https://pbs.twimg.com/profile_banners/929903647/1430095851\", \"verified\": false, \"name\": \"Kirk Kus\", \"profile_background_image_url_https\": \"https://pbs.twimg.com/profile_background_images/464858470500417536/oVGrCW1A.jpeg\", \"location\": \"Sheffield + Derby\", \"notifications\": null, \"friends_count\": 2678, \"lang\": \"en\", \"description\": \"22. UKIP YI Chairman for Derbyshire. IT Technical. Studying Network & Security at @DerbyUni. @UKIP member & activist. Darley Candidate. Member of @tfa4freedom\", \"time_zone\": \"London\", \"profile_image_url\": \"http://pbs.twimg.com/profile_images/592490695581159424/RHicaeez_normal.jpg\", \"profile_sidebar_border_color\": \"FFFFFF\", \"profile_background_color\": \"FFFFFF\", \"url\": \"http://www.mmp4uk.org\", \"default_profile\": false}, \"geo\": null, \"possibly_sensitive\": false, \"entities\": {\"symbols\": [], \"trends\": [], \"user_mentions\": [], \"urls\": [], \"hashtags\": [{\"indices\": [98, 111], \"text\": \"BetterOffOut\"}, {\"indices\": [112, 117], \"text\": \"UKIP\"}]}, \"source\": \"<a href=\\\\\"http://twitter.com\\\\\" rel=\\\\\"nofollow\\\\\">Twitter Web Client</a>\"}, \"in_reply_to_user_id\": null, \"in_reply_to_screen_name\": null, \"place\": null, \"id\": 593891099434983425, \"retweeted\": false, \"truncated\": false, \"text\": \"RT @KirkKus: Indirect cost of the UK being in the EU is estimated to be costing Br'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweetsJson = twitter_samples.raw('tweets.20150430-223406.json')\n",
    "\n",
    "tweetsJson[ : 3000]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Ws0}bS$/?#R5]:h%]1Q_\">tweetsJson</variable><variable id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</variable></variables><block type=\"variables_set\" id=\"Q=bB?13U{5~]-mP$F$RJ\" x=\"41\" y=\"200\"><field name=\"VAR\" id=\"Ws0}bS$/?#R5]:h%]1Q_\">tweetsJson</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"5L0KrAA7.ekjugpH0_+(\"><field name=\"VAR\" id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</field><field name=\"MEMBER\">raw</field><data>twitter_samples:raw</data><value name=\"INPUT\"><block type=\"text\" id=\"@8%*,}Voj8$m/8XlDHQz\"><field name=\"TEXT\">tweets.20150430-223406.json</field></block></value></block></value></block><block type=\"text_getSubstring\" id=\"EiYC?M!TV+R:gf_1a|%9\" x=\"8\" y=\"422\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"STRING\"><block type=\"variables_get\" id=\"+JP*z!I697F|rOb:sQ|4\"><field name=\"VAR\" id=\"Ws0}bS$/?#R5]:h%]1Q_\">tweetsJson</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"s)?j`{oE+jEk3MU2h-]]\"><field name=\"NUM\">300</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tweet is in a file format called [JSON](https://en.wikipedia.org/wiki/JSON), which stands for JavaScript Object Notation.\n",
    "JSON is a textual representation of an **object**, and objects can have complex nested structure.\n",
    "A lot of it is metadata that we don't care about (we only care about the `text` field), so to simplify things, let's use the `twitter_samples.strings` instead of `raw`.\n",
    "\n",
    "Create a variable `tweets` and set it to `twitter_samples.strings` using `tweets.20150430-223406.json`\n",
    "\n",
    "Then show the first `10` elements of `tweets`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['RT @KirkKus: Indirect cost of the UK being in the EU is estimated to be costing Britain £170 billion per year! #BetterOffOut #UKIP', 'VIDEO: Sturgeon on post-election deals http://t.co/BTJwrpbmOY', 'RT @LabourEoin: The economy was growing 3 times faster on the day David Cameron became Prime Minister than it is today.. #BBCqt http://t.co…', 'RT @GregLauder: the UKIP east lothian candidate looks about 16 and still has an msn addy http://t.co/7eIU0c5Fm1', \"RT @thesundaypeople: UKIP's housing spokesman rakes in £800k in housing benefit from migrants.  http://t.co/GVwb9Rcb4w http://t.co/c1AZxcLh…\", 'RT @Nigel_Farage: Make sure you tune in to #AskNigelFarage tonight on BBC 1 at 22:50! #UKIP http://t.co/ogHSc2Rsr2', 'RT @joannetallis: Ed Milliband is an embarrassment. Would you want him representing the UK?!  #bbcqt vote @Conservatives', \"RT @abstex: The FT is backing the Tories. On an unrelated note, here's a photo of FT leader writer Jonathan Ford (next to Boris) http://t.c…\", \"RT @NivenJ1: “@George_Osborne: Ed Miliband proved tonight why he's not up to the job” Tbf you've spent 5 years doing that you salivating do…\", \"LOLZ to Trickle Down Wealth. It's never trickling past their own wallets. Greed always wins $$$ for the greedy.  https://t.co/X7deoPbS97\"]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = twitter_samples.strings('tweets.20150430-223406.json')\n",
    "\n",
    "tweets[ : 10]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</variable><variable id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</variable></variables><block type=\"variables_set\" id=\"Q=bB?13U{5~]-mP$F$RJ\" x=\"41\" y=\"200\"><field name=\"VAR\" id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"5L0KrAA7.ekjugpH0_+(\"><field name=\"VAR\" id=\"B@`]CGD1A2?+*muAJugJ\">twitter_samples</field><field name=\"MEMBER\">strings</field><data>twitter_samples:strings</data><value name=\"INPUT\"><block type=\"text\" id=\"@8%*,}Voj8$m/8XlDHQz\"><field name=\"TEXT\">tweets.20150430-223406.json</field></block></value></block></value></block><block type=\"lists_getSublist\" id=\"F1K:OnZ`Jb-a{,KN*$N-\" x=\"39\" y=\"320\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"(hu+(?ZE;XU3$kDrGoHU\"><field name=\"VAR\" id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"aPw(:XHX?/z91(}T0feF\"><field name=\"NUM\">10</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, each tweet is a little document, i.e. a relatively short string of characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Words\n",
    "\n",
    "Notice that this is a list of tweets, not a single continuous text like `emma`.\n",
    "\n",
    "Tokenize each tweet into words and store in `tweetTokens1`.\n",
    "\n",
    "Once you have `tweetTokens1`, show its first two elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['RT', '@', 'KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', 'the', 'EU', 'is', 'estimated', 'to', 'be', 'costing', 'Britain', '£170', 'billion', 'per', 'year', '!', '#', 'BetterOffOut', '#', 'UKIP'], ['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http', ':', '//t.co/BTJwrpbmOY']]\n"
     ]
    }
   ],
   "source": [
    "tweetTokens1 = [(nltk.word_tokenize(i)) for i in tweets]\n",
    "print(tweetTokens1[ : 2])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Q52-g3_[/R0qdo!:lsik\">tweetTokens1</variable><variable id=\"vm#^l?f3f{nfc06H7ta7\">i</variable><variable id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable></variables><block type=\"variables_set\" id=\"Zf$0_!.iE^S~):S2$:Y)\" x=\"29\" y=\"200\"><field name=\"VAR\" id=\"Q52-g3_[/R0qdo!:lsik\">tweetTokens1</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"UYo/uYP;mBU`1EuTNCkI\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"q$bdPlP3#D)qGF]yOK]b\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"R,a!C~+.Y7ZWe!MBjt#)\"><field name=\"VAR\" id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"~s.Dw!@c(8npsi6MpcW^\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">word_tokenize</field><data>nltk:word_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"zvm+ZUd[Eb^)P,Mi4ho7\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field></block></value></block></value></block></value></block></value><next><block type=\"text_print\" id=\"0|gP)^e):dhC1@z?X$8-\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"])T^~l.j`_k~cW%AbUcU\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"mR7/Rgk95Z.N6t(OitdJ\"><field name=\"VAR\" id=\"Q52-g3_[/R0qdo!:lsik\">tweetTokens1</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"Mof?1STzpe4u~KdsI-D=\"><field name=\"NUM\">2</field></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "Which of these tokens violate what we know about Twitter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*@ is part of a handle and should not be separated, otherwise we don't know what words are handles.\n",
    "The same applies to # for hashtags.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to use a Twitter-aware tokenizer for these data.\n",
    "Fortunately, NLTK has one built in.\n",
    "\n",
    "Create a `TweetTokenizer` using `nltk` and store it in variable `tweetTokenizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweetTokenizer = nltk.TweetTokenizer()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable></variables><block type=\"variables_set\" id=\"M6T0am_1LQ~%mS$Po-v^\" x=\"-4\" y=\"179\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"A40ZnNV*DXCia`@$Xj`b\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">TweetTokenizer</field><data>nltk:TweetTokenizer</data></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now tokenize each tweet using `tweetTokenizer` and store in `tweetTokens2`.\n",
    "\n",
    "Once you have `tweetTokens1`, show its first two elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['RT', '@KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', 'the', 'EU', 'is', 'estimated', 'to', 'be', 'costing', 'Britain', '£', '170', 'billion', 'per', 'year', '!', '#BetterOffOut', '#UKIP'], ['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http://t.co/BTJwrpbmOY']]\n"
     ]
    }
   ],
   "source": [
    "tweetTokens2 = [(tweetTokenizer.tokenize(i)) for i in tweets]\n",
    "print(tweetTokens2[ : 2])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</variable><variable id=\"vm#^l?f3f{nfc06H7ta7\">i</variable><variable id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</variable><variable id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</variable></variables><block type=\"variables_set\" id=\"Zf$0_!.iE^S~):S2$:Y)\" x=\"29\" y=\"200\"><field name=\"VAR\" id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"UYo/uYP;mBU`1EuTNCkI\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"q$bdPlP3#D)qGF]yOK]b\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"R,a!C~+.Y7ZWe!MBjt#)\"><field name=\"VAR\" id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"~s.Dw!@c(8npsi6MpcW^\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><field name=\"MEMBER\">tokenize</field><data>nltk:word_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"zvm+ZUd[Eb^)P,Mi4ho7\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field></block></value></block></value></block></value></block></value><next><block type=\"text_print\" id=\"0|gP)^e):dhC1@z?X$8-\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"])T^~l.j`_k~cW%AbUcU\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"mR7/Rgk95Z.N6t(OitdJ\"><field name=\"VAR\" id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"Mof?1STzpe4u~KdsI-D=\"><field name=\"NUM\">2</field></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "In addition to fixing handles and hashtags, what other changes did the new tokenizer introduce?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*The tokenizer is keeping URLs together, which is good. \n",
    "Another change is that the currency symbol for pounds sterling is now separated from 170*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentences and words\n",
    "\n",
    "Now process the tweets one more time, this time using the sentence tokenizer first, followed by `tweetTokenizer`, and print the first two elements.\n",
    "\n",
    "*Hint*: This is a bit of a mind bender, but you can do this with nested comprehensions.\n",
    "Keep your list comprehension block exactly the same as before, *except* change the `yield` part by:\n",
    "\n",
    "- Adding another list comprehension (don't forget to put the comprehension in a list like we've been doing)\n",
    "- In the new comprehension, iterate over `j` and \n",
    "    - Use `nltk.sent_tokenize(i)` as input\n",
    "    - Use `tweetTokenizer.tokenize(j)` for the yield\n",
    "    \n",
    "You should be able to figure out the corresponding blocks from what we've covered; if you do it correctly your code will match the above.\n",
    "\n",
    "*Hint*: If you ever have problems wrapping your head around a nested comprehension, just store the first comprehension in a variable and then use that variable in the second comprehension. Divide and conquer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[['RT', '@KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', 'the', 'EU', 'is', 'estimated', 'to', 'be', 'costing', 'Britain', '£', '170', 'billion', 'per', 'year', '!'], ['#BetterOffOut', '#UKIP']], [['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http://t.co/BTJwrpbmOY']]]\n"
     ]
    }
   ],
   "source": [
    "tweetTokens2 = [[(tweetTokenizer.tokenize(j)) for j in (nltk.sent_tokenize(i))] for i in tweets]\n",
    "print(tweetTokens2[ : 2])\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</variable><variable id=\"vm#^l?f3f{nfc06H7ta7\">i</variable><variable id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</variable><variable id=\"HEsPK%?1|o7!C+UG!1P(\">j</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</variable></variables><block type=\"variables_set\" id=\"Zf$0_!.iE^S~):S2$:Y)\" x=\"29\" y=\"200\"><field name=\"VAR\" id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"rdRUoL2(PI03TK*V3Y7y\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"q$bdPlP3#D)qGF]yOK]b\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"R,a!C~+.Y7ZWe!MBjt#)\"><field name=\"VAR\" id=\"c/CodMn|Ryb@7N-i~LN6\">tweets</field></block></value><value name=\"YIELD\"><block type=\"lists_create_with\" id=\"uQ6):(y|m7w_B{C-?]nk\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\",Z=/F!k-Z{GH[K7ZXOpP\"><field name=\"VAR\" id=\"HEsPK%?1|o7!C+UG!1P(\">j</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"~s.Dw!@c(8npsi6MpcW^\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">sent_tokenize</field><data>nltk:sent_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"zvm+ZUd[Eb^)P,Mi4ho7\"><field name=\"VAR\" id=\"vm#^l?f3f{nfc06H7ta7\">i</field></block></value></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"}AsNv:.WQ;)SyfZ01D3{\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><field name=\"MEMBER\">tokenize</field><data>nltk:word_tokenize</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"%q)y5nk]v0v48h6-$(7j\"><field name=\"VAR\" id=\"HEsPK%?1|o7!C+UG!1P(\">j</field></block></value></block></value></block></value></block></value></block></value></block></value><next><block type=\"text_print\" id=\"0|gP)^e):dhC1@z?X$8-\"><value name=\"TEXT\"><shadow type=\"text\"><field name=\"TEXT\">abc</field></shadow><block type=\"lists_getSublist\" id=\"])T^~l.j`_k~cW%AbUcU\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"mR7/Rgk95Z.N6t(OitdJ\"><field name=\"VAR\" id=\"uocLxC)3{l9!x]6gHkBp\">tweetTokens2</field></block></value><value name=\"AT2\"><block type=\"math_number\" id=\"Mof?1STzpe4u~KdsI-D=\"><field name=\"NUM\">2</field></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we now have a list of tweets, where each tweet is now a list of sentences, and each sentence is a list of word tokens (Twitter-style)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "Do the sentences in the first two tweets make sense?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Somewhat. It is a bit odd that the second 'sentence' of the first tweet is just some hashtags and not a 'real' sentence.\n",
    "If we wanted to do additional analysis that assumed well-formed sentences, we'd need to make some changes.*\n",
    "\n",
    "----------------------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}