Text-as-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text as data\n",
    "\n",
    "This notebook introduces foundational concepts in using text as data.\n",
    "As a result, this notebook is just reading without practical exercises, unlike later notebooks.\n",
    "\n",
    "We have organized this notebook around some **big ideas** in linguistics and natural language processing (NLP). \n",
    "You may wish to refer to this notebook throughout the course when these ideas come up. It's OK if you don't completely understand them today. Some of these ideas are quite subtle and take time to master."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text is (almost always) unstructured, nonexperimental, and dirty\n",
    "\n",
    "When researchers collect data, they do it with analysis in mind - in other words they plan how they will collect and store the data.\n",
    "Typically this is done in a tabular format like spreadsheets, where each row represents an observation, or datapoint.\n",
    "Each column of the spreadsheet likewise refers to a single variable of interest.\n",
    "This structured format is required by the majority of analysis techniques. \n",
    "\n",
    "Obviously, books and other forms of text are not structured in spreadsheets.\n",
    "Rather they are structured according to their natural purpose, which is communication.\n",
    "Similarly text is rarely the result of an experiment, but rather observed in its naturally occurring form.\n",
    "As a result, when we see differences in text, it can be difficult to explain *why* the differences exist, i.e. what *caused* the differences. \n",
    "To answer these kinds of questions, we'd need to run an experiment.\n",
    "\n",
    "So if text is unstructured and nonexperimental, how is it *dirty*? Let us count the ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Words are complicated\n",
    "\n",
    "In English, we can generally identify words in text by looking at white space, e.g. spaces, between alphanumeric characters. \n",
    "But words can be complicated by inflection, polysemy, synonomy/coreference, and idiomatic usage.\n",
    "\n",
    "To make this clear, let's introduce a running example.\n",
    "Suppose we are analyzing some text and are particularly interested in the word `fly`.\n",
    "In order to create structured data for analysis, we are going to count how often `fly` occurs in different texts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inflection\n",
    "\n",
    "The first thing we notice is that in addition to `fly`, we also have `flies`.\n",
    "This is a phenomenon called **inflectional morphology** whereby base words like `fly` are modified by affixes that precede the word (prefixes) or follow the word (suffixes) to change the meaning of the word.\n",
    "In this case, `fly` has been made plural by the addition of `s`, and a spelling transformation on the result produces `flies`.\n",
    "\n",
    "Consider how inflectional morphology impacts our simple task of counting `fly`. \n",
    "Suddenly we have to accept that we will **undercount** because we are ignoring plural forms, or we will need to change our approach to account for inflectional morphology, counting both `fly` and its inflected forms like `flies`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Polysemy\n",
    "\n",
    "As we continue, we notice that there are some meanings of `fly` that are not consistent with our original target, which was the *insect* `fly`.\n",
    "In addition to the insect sense of the word, a noun, we are also finding `fly` used as a verb (*I saw the bird fly*) and an adjective (*Your girlfriend is fly*).\n",
    "When a word has multiple meanings like this, it is said to have **polysemy**.\n",
    "Polysemy is very common in English; in fact the shorter a word is, the more likely it is to have polysemy.\n",
    "\n",
    "Now we face the problem of **overcount**: if we simply count `fly` without paying attention to the specific meaning, we will be including many words that don't refer to the insect at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Synonomy/coreference\n",
    "\n",
    "So we shift gears to consider the meaning of words while we are counting.\n",
    "And in doing so, we notice that other words are being used to refer to the **concept** fly besides the word `fly`, like `housefly`.\n",
    "Again, we are faced with **undercounting** if we don't include **synonyms** like `housefly` in our count.\n",
    "\n",
    "Similarly, we notice that `it` is sometimes being used to refer to `fly`.\n",
    "This is called **coreference** because `it` and `fly` refer to the same thing (if you have heard of anaphoric/cataphoric reference, coreference is an umbrella term).\n",
    "Again, we are faced with **undercounting** if we fail to include the word `it` in our count - but of course only when it refers to `fly` and not something else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Idiomatic usage\n",
    "\n",
    "Just when we think that we've covered all the different ways counting `fly` can go wrong, we encounter the phrase:\n",
    "\n",
    "> fly in the ointment\n",
    "\n",
    "Here `fly` does refer to the insect `fly`, but when taken in context, the expression does not refer to a fly at all but rather means \"a problem\".\n",
    "`Fly in the ointment` is an **idiomatic expression** where the meaning of the expression as a whole is not compositionally expressed by the words in the expression.\n",
    "\n",
    "So here we face a problem - the phrase clearly refers to `fly` in a literal sense within the expression, but **the expression as a whole** has a figurative sense that has nothing to do with `fly`."
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAACpCAYAAAB597WBAAAABHNCSVQICAgIfAhkiAAAIABJREFUeF7tnWvW1MQWhtuz/I3iAFAcgOIEVCbgB/yXixNAHYCIAxAYAcgA5DIAUCaAMAGUCYD4nz71lGf3qa+6cq+kK8m718rqTqUue7+VvNl1zTtbJxuJEBACQkAI9EbgP71TKqEQEAJCQAh4BGZLpClHOhWmehYCQkAIjI3AbIn0nXfe2cOGMJHpHiwKEAJCYGQEZkekTUSZItiRMVT2QkAIrByB2RGpiHLld6zMFwIFIvBugTp1UunZs2ebO3fubN5///3NH3/84X8/+uijzbVr1zrlo8hCQAgIgb4IvDPn6U+///775ssvv9z8+eefmw8//NBj8ODBg83Nmzc3jx8/7ouJ0gkBISAEOiEwa4/0t99+25w5c2ZHolh+dHTkiVUiBISAEJgKgdn1kYbA0ISnOX/r1q1jeJ07d86fNw1MTQWyyhECQmDZCMy6af/333/7pj1kCqn++OOPm0uXLi27xmSdEBACxSFQvEda51W+9957G5r3V69e9c35y5cvb86ePbv566+/igNaCgkBIbBcBGbtkYbVwsATg0z379/fQLCQKb8SISAEhMDYCBTvkdYBwLQnky+++GJz7969zS+//LKhyR9eq8tD14SAEBACQxGYNZHSnI+b8RcvXvQj+a9fvx6KjdILASEgBFohMGsiZZUT/aKhQKzWX9oKAUUSAkJACAxEYNbzSLGdlUyfffaZH73nP817mvU2QX8gPkouBISAEGhEYDGDTY2WKoIQEAJCYCQEZt20b4tJ3RSqtnkonhAQAkKgCoFZEWlMiOE5I/UcKdGOUSlUFCYEhEAuBGZFpBgdkylhLBFlZRPH9evX90byc4GlfISAEBACKQRmRaR4lqF3+fDhw83p06c333777ebTTz/1055YJgqhXrlypZJQU2ScAkdhQkAICIE2CMyKSM0gVjGxFNQ2J2GUnqWibJ3HunumRBEGoRKH+KGoqd/m1lAcISAEWiPAfqRzETc/dOtIks9Hb91Up63zPitVt7jEI76bHrV1ZFsZXxeEgBAQAn0RoM+xKHn79u2ePm6VkidNCJHDNeW3bmBpL14qwNIaoTovdXv79u1UVIUJASEgBHohUNQ8UmfBsT5Q3GoGkuj3ZMknzXQ2Jukz2d7W35OelU80++lbpRtAm5u0bsAoohAQAikEetHvgEQpjzOVndvFaYv3iAfqNiTJ0iy3sl3/6S5v6yLAc+0ibe3okqfiCgEhME8EJh9sahroqRpIYnenoWJls/nzixcv/ABVPNL/8uXLVsU02dEqE0USAkJgGQiUwv91A0lV3l9VeFebGIRy3Qa7PlgGtNzo/y6bXOV01UvxhYAQmAcCBx9sokntJtEfG0jq2sxuA3UbMgzJ3L0mNdLfBljFEQJCYHvQwaZcA0lD2wbuPjg2yMVWfMxDZWCKQS66FZjgr+9BDUVa6YXAQhE4xMskHEgqZX5nymONp0652QKaOnWIG0ZlCoHCEZh0sInBHVuRhBeI18dqpBwDSUPfc6nBI6ZFXbt2bfPq1SuvK3HwTE+ePOnX9FdtkjJUF6UXAkJgXghM0rSnqcxcUMiIzZeZvwlBzVH4JhR28FIwW5iLeurUqb05sHO0TzoLASHQA4GcHnPcPE6tSBpjICmnDU15mY2M9NMt4SD3ByP9DFZJhIAQWB8Co43au4GarWsae5JhatFSSYZpUrb+32zVmv71PUiyeN0IZCfStgNJsfc6t2qI9Y83SeFFIhECQmAdCGTrI2Xg5ejoyG9Zxzp2+kSZLuRgXFXfoa3p52N87JEqEQJCYPkIZCNSBpSuXr3qv+hZN5BkxLo2gl3+rSQLhcB6EchGpCkIRZYpVBQmBITA0hAYbR4pJLo0WaJNS6sj2SMEDoHAaESKMalJ7ocwMleZS7MnFy7KRwisHYF3xwLASGdJzXsm47MptAkDSuGqLPYOYG0+wibUTNq3cybvs2UfUsJKrp0R+iMEhMBgBFr3kRohMirtpjj5A2GVT7jDPNfZ7AMSgWhYxWTXrWk8Z8/uu+++8/Zhd2oTEwbbuG5kyTlCGAIe7IX67NmzXjv9+0wkQkAIlIWAI7fOwhxKRyS7yfapDJikvkRhvqirQT8JPxauub0EjgWz+onDhHm2pNc80xg9nQuB+SLQq48Uj5K5ojRX8Uzx0mLh+pLEVbE3h+9FsbYeu+NNS9xH9fwUMIubsv+rr77ywebRp+IoTAgIgXkh0ItIzURIg34/mq30H4ZSRybzguhfbcPuCIiUvs+YDFmMwKKEsH+Y1CEW9imTpb1o5lin0lkI5EJgEJFCGPSFQgqQi1t3vtMr7gddErHS/2nfejKDeZEwwBSKYYDn+uTJk82DBw98Hylp+ZUIASGwDAR6E6kRIwNJeGY089lrlBVOKYmJNRVnTmEQISP4eKEIg0+8TKoEvGzbvadPn2r5aBVQChcCM0SgkUirPMmQGFlTDpHQ3MUri/sOZ4hLo8qM2NNfit28PPgfzl4IM4BA8WLtaMxcEYSAEJgVAo1EWudJhiRL3yAbldC8r/PMZoWOU7bqRYId7JZvnmhqwG1utkpfISAE+iHQSKR12caDKmxWAonS1KfvdAlS9yKxfk48ce30tITalg1CoB8CWVY2hWTDCH446NRPrbJS4ZWmCJWmPC8OFh7EArky6d6wYKCJeFXN/zi9zoWAEJgPAq1XNoUmGUFAEvSJxkse6SOFYH799dckAc0HnmZNsbUNOVaRcXMJiiEEhEDpCPQi0tKNkn5CQAgIgSkRGNRHGitaNzATx13Sudkd2l81DWxJdssWISAE/kUgK5Gm+hHXALTZzS/zStmohEUK169fX4P5slEIrB6BrEQao7kmD5W+UqZAMaBE3zGLE5gOBqnSpxzLmrCJbde5EFgaAtmI9PTp03vr7ZfmoVaRH3NJ8UCZscDEfKZ+PXr0aEegLAmNvdOlYbO0B0P2CIFOCDhyyCKOTLZ84z2U+JPFWQoqKBO2zWOLPAe4P5wHunXTnvY0JJzrjlD3MNqLrAAhIARmh4BG7Vu+dlzN7qZy2ebVNN0RmvN4pXijVULznilhNPtJV/el1ao8FC4EhECZCIhIO9YLg0kQIhuWMH+Uz4ukdsqvypYmPkRKc58doz755BMfNSTqqrQKFwJCoEwEsvWRQgqpQZUyzW6vFQSHMJ3p/PnzfhAJEmV5KGFdSJR88ETxSsk37DtVn2n7OlFMIVAaAtmIlAEWCGZpAsHhdUJ67CHAmnpsvXHjRqsVTSk8yAMyhVTxTqtG9lNpFSYEhECBCOTo1Z3ToFIXXR1h+gEiV21btxXeKN9ZYoDOymBQqk666F6Xj64JASGQFwGamJIIAUbeXT/objTe7Sewdc34UXGCRCHs1Mi+CHRU6JW5EBiMQLamfYHOdi+V6OtlTqjNDaUZf+/evc2pU6d65dc2kfWdEj+ed6r+07YoKp4QOAwC2YiUVT0MxsxB3OvHq2m//GegjIEkRuTZAo++S74///nnn09mEn2nfIaEstV3OhnsKkgIDEYg2/Qn26QDz21OHlSfOaGDUU9kAKmHuGneaQIkBQmBQhHIRqSF2lerln3V0+aE0qznkyklSTjv9Pbt277ZLxECQqAsBLI17csyq14bmxPKptThnNDSSBQrwr5TpknFa/brLdVVISAEJkFg8HDV/zJgqpAblMmV3Wj5MDrOVCYHbnKEfLSCM2RcN7KfIXtlIQSEQE8EsnmkTDAveWUTo+94dAziIOzUVNr35V0d1n61NB7Zp2tCkg8B8JcIgT4IrKKPlJVJ9sVPmvNMbWrznaU+gE6VBpvYLIW1+nMa3GuDD4SW26Yx8mxji+KsA4F312Hmvzs00b845XSmttj2ecivXr3aNvvZxctNogAwRp6zA1YKj4bAIj3SPsQ0GsIdMx5L97Hy7Wje5NHb2h3Hi88nV7ywAueKR6x3fJ4L5mx9pCVNyJ+z9zGW7mPlm+tGHCuftnbH8eLzsfTrmy+EwDGVlI5HFQ6x3vF5Vbqu4YM9UmN4m5Bft7lxV+XGiD/WG2kMXXPkGdprD95YN1OTvnXYszAi7Leui9tUTpfrU5XTRSfFnR8Cg4l0fibPQ2NmQLBtH8IAGctHQ2HEnpkSCJP03ZQuv72fiYWxb8AhX24hUUGWzDxgc2wE3RgwY0kuv1988cVO/5x/KJcyDAe+L8b5q1evdsWAd4hxqQSLwwJWzH+mvsfCLMYffGze9ZTlxnp0PY/rNUyfs45HGWzKqWBX4MIJ6+hx5cqV3QNky0Etz1I/94HePNQQDQdTtVj3D1masHiAeDxYtpAAcuBmJz5xIVoeOmYssH/qIcS8X7BHF3TiCEmLOsKWsYR6vnDhwu4+QI+QRMGQOPbiOuT924QBLwPuian3/qW+KHvqcpvwaLrO3hlMc0xJzpZZNiLFy+BB5qHOqWAKgLowSAMy4a1JpYfeGE1HrtuNWOoDY/ihL54ltrChCr9h8/fkyZMbDhO7Rho8FQ6menEcikhNN+oEAgX/2Ltmbm/TA/ry5Uv/MkGavDDq9c2bN7tPwTBVDC/U8OHhMuKGRNEN4R5GN5sPTRrEymUe8okTJ3xYSkJPu20rgPLJ3+oc2yifZ6mqvL4vnfh+j7tTUjYRZh596rm2PM12XuBx/Vbla+HYy4uNe7kubZO+oX3EZWYL9xy6hS8f/retnybd7Xq2wSbzNNoWPFY8HhY8HOTx48d7xeB18DATL3Vj7CUoIAAv20ioTp2UPZAED6Xd6HXpx7rGTc1LADHSCsvipjZypO6wg30PeLBs0QEPCQSIN8k1SI54ZhdlcA5WeJ+8OHhA8c6R8P7kpcSBcJ1rYISO5MOLh3LY2yAs1zwbyjTd0MN0JD3puL8YfEUfK4ey0Juwu3fv+t+ffvrJ64DNpKNcBF14uVQRptUz8Q2vsCWGTd98843Pw7Akr3/++WeHETu1EY/wKnvAAjupM8pCr1hMF7OdOkIncCEteZgQDi78cs1eWOQLThCf4YZ+4Eo8XoS8VKgnpEpf04U46APe9p8XtemI3ZRHfPI2O/lPuehgQjz0BV90qBQHZFYpZRNi99bxn0qOhTBXcXFwsedmg200zTJRE3djbDnCc1fRx8IsnbuRDmajewj9klyOJmH5LjojjpD8cl6rL7CwPLCb/+4h93Hdw+HP3YvSn9uvpQlxCnWxfByR+3QIGBPHsI7z4N5CN4RfzhH3gPt07qHz55bOEYA/J57FDW0xfKyu0YmwKon1AS9wohzqOcSF/1xDqjCqssfiG3ZxuaF+ZjsbkyOGq2FoNroX3O6a1R3xrU4ok4P64zq6mV0hjiH+7kUcqrL7H9ZzqBN4sJwd3dzLceu+u+aXiyOUSzpwdETs/3P/cfC/ql6yeaSuEC/hW8HCxv79tx6Ol8Kb0VWm70M04b+rmFmuauItSpMdL8PetFW48ubmrU0aMOANm7spU1V2Kjzs2w09lDguHgjeCXWE0GoIvdk4PufW12m7YuE94UW4hz4V3Yel7tFUWCoD8DTvBi/F9MW7svswvh+JY+nMNlpG5mGhMxjZ/co1wrhu3jO/oacU6kaelEE66pl7xDxO4nENSWHEfVJlD/mF6cIy4/9ms9W1tTAsD+oDvT7++GPv3YK36UVenJMWr5SDLkKa+eiG3tzLnKfwx/s1LGO9UufoAr70i5M3zxPlUJ/kz3V+aZEg6E35hFeVk51IU4qPHZZ6CGheGQhWPoBY+Ng65c4fUqFyudmwwZpF8UNr5RJOU4Q0P//8c251OuUXklrVjUiG8TXqFTvi8FTh1h/MNWyu28mrCrNUvnFcu9eoBx5EHja6kOr69sjX0qEbQn3ap7j5b/clDyx59+l6MpKGHOjagGBCiTGCTExS9piuFifGoimc61Z39iJHL/SL8+LciN7y5XnFIULABaIzHGnyh/in8rR87NfSxuVwHVInP/SjPsOvA/NS4hrhVWMN2YiUt2VYMTFQsVFjn3MjUgm8EfFqOHjrNN3wY+vVNf8QRyNTKh1y4jd8iYQ3Ctd5cA7piZqtMVHEGOAV8WJI3eDErQrnmtlMeu4/e9mQZxtpuk9DfMnPyIpyzLtmIKzO045tMN0YELN0EClkBmEYeXCvPnr0yD/AdQ+xeXbUOfmRnpdo/DKJMcKGlD203MjHrhkZxlgYvhYeXje7jBOePHniPTt0q6vPsM7AiRcVmCCmL+WAh5Vh+oZpU/+r6trsDDcBAisL51NDJubA7OXvMs8ijqR8v0JJgj7OYN/XQd9GVf9GSTrHuriHIw7ydmCXI6jGPtK9xAcKcA/77oOC9G/RF+UeDP/rHq5dP6h7yHZ92/znmgnn2O0e7F0/m+FDXu6h9fmQvyMRn8w9vD4N+RAH4ZwDIb4jMN8X5x5Yf07+XKc88uI655aeeJw7ovP5232FLYSjB4JuYTry4pz8uEbaUDhP1fexSEG+lGv6W152z1MGfbWmO3FjjExPswd7Q3vsPkOnEIdYb/Qjb2wDR6tTzh3xePUtL8PU4pLOEaJPy2HCWAvlWFmkc4NB/nKVvrvE//tj+JMHuFj9xBhbfYOV8zx9mZQPttiDXthv+cTlcD7ahHyXd7Ivyik1qdC3hCfK275qPtmkCnUsDP1Tsw9oAfCmdjfibsSbN7i7SY6FdSxu9Oh4EeZhUBgeS9xKYPQULwuPwJpYeB3UI0KzjvvLPDHSky+eE3WMx4KHhyeMmAcYTi0inHuCtDa1ivLMgycN2JpHRVkWn7TWr4Z3ZZ5f6AWTl+kbpsN2dA3TeSWdEE6ZsSdp1+0Xz4l6prVl93a4iQ16WJl4UOiO7WAIRuZhhhil7KE8W/iBvuTDwX/rAzWd7N4jnHuTMtDR4mEbuoAvR0pH8gqxIj75hDYar1TpG2KFx8/9YHla/XB/mB6h/mBPvVHndu+AmXsJ+DyoF64je/yWYtclhdlbiDdKqZKa6cDb0FWs94qwwVXonvp4FMRBeKuax4P3xblkHgjgrTmy8J7WXIX7EH6Jvb252tNV72weKWzP26rpbRq+Mab6T6c7+o39SeWp7LFyrI/I3p5Tl6/y8iCAh41nzKAGnqV7iItozbW1Dn3pA8UD5Ui1oNrmNdd47+ZSHBe8VCJFtyWSzRJtynU/zikfmpSIdXFUDeqUaFNI+mYHTfK4u6ZE3XPqlM0jzamU8qpHYG4eS701urpUBNZ0n/5nqZVodlGZc5NY5/h8Th7L3LCXvvkQWNN9OhqRxg9/vurpltMcKzPW2c5jTOn3rZzX1g0mxRYCgxCI781Bmc0wcTYiZSoFneUmMRnMEJuiVbaNLpiKUrVkdO03d9EVGCkX1xXncVjJ9qSe9znpPxTbbINNVeuAhyqYIz2rPFhRMfcPxtnNCokyt41VH3il2IaEy9o4T93cOfBUHvkRiOsqPs9fYp4cIcsqXavC85RcVi7ZiNQmMtcBeyjTIR2bSHsoHZrKbYubkSg2QZxGolVk2lSurguBIQisiSzrcMpGpFZIicDOYSpGG9xiEjXMw30zCYs907obQNeEgBAYjkA2Ig0n5Lf1roarv54cqkgUBMAbDxUylme6nntClpaDQDYiDSfkt/GupoSAwRgGZebgmaZwMRLFjosXL+5FMbxt/8QmMtWLbg9CBQiBQQhkI9KSB3JYcRFuhjAIsYkTh57o119/XVs6BNmGTEt70dUapYtCYAYIZCPSkm01cilZx5RuzISw0fk2/Z5tPVN5pCm0FSYE+iOgJaL9sRslpZEczXia6Gy71tfbx5tlU1r6r+farTEKyMpUCGRGIBuR4j2xbyAPrjyeYbVkJAqR2oi8eZtdsGV3KHbjoV7o3hCZDqsXpRYCVQhkI1L7yFwJn7aIjWVzZPpIw08GxHEOcQ4pImGfZUyiQ/UyMuXb6fJMh6Kp9EIgjUA2Ik1nX0Yo69EhUvYj7ePZTWVFbhI1veWZTlWDKmetCKyCSEuq3Kqm+VgkKjItqfaly1IRyLZpCc3G8Ct8SwWsq13WfLd0qalHRqKMzI81w8C+QErXC/2meOmxbl1tU3whIAT+RSAbkTIhn8OkpIcUojoUyaeIM7z5Qk+UqU5jCmTKC499ByDT58+fHyvO6qykuhsTD+UtBHIhsIqmPTMK6CN1n1rNhVvrfOKmfHg+dnM+VtLKbtNnGusd56VzISAE/o/AKoi01Apnez+WrlY158ciM/L9559//KdyGc23z+SWipP0EgKlIyAiLb2GRtQPz5QJ/+H3zUcsTlkLgcUikI1IeRiZS1raXM3Sam4sL7M0O6WPEFgTAtmINJyQXxpZMLBCH+n9+/fXVLeyVQgIgYkQyEakE+nbqxibkD/lqiteLCzNtJeK7T51yJcMZb9588a/UNANneJ1/NbcNz25TjyTQ+rfq/KVSAhMgMAqiDTGcSoyYMrVuXPn/FQjDhPzjJnudKj17z/++OPm+vXrG35TsxlYVvvo0aNjy1enwi2uL50LgeIRcA9HFnGbYmwdQWTJa0mZuBtg68hq+/bt22NmOWLdOk9v6+beHsRc6gvdOByh7+mAzhIhIATaITDahPyS3iB3797dm5Dv4BlFxap844n5NuWJQboxpUofyrTReubZhospxtRHeQuBJSKQjUjpS0s1EUsA7enTp3tEERNbXz1jomqbL6uLaPazvZ0N1PXVoS5dnT70fbLSCRtoytM/KhECQqA7AtmItHvR06W4cePGaCRfR1RNFtonohn4OZTQR8tc0tevX/t+XJHpoWpC5c4ZgVUQaVhBsQc558prq3uTzWyWwqATzfuxuxra6qx4QmBOCGT7ZlO4Qz4A8PAO8dZygghBnDx5csP0p7F1Cu1uIjDzRFkmOqa0sZluGXBiJoF5ymPqpLyFwJIQyEaksSfT5uGdCkjbtGSKVVeh3XUY0IRmGlRqeWbOl1CXvBgAg9zxTum/lQgBIdAOgWxEOuVk93am/T8WfYDhpPKu6fvEryMwSBQCxROFtMaUKjJPhdvqL/pN6TOVCAEh0A6BVfSRQgxTEH3YlIeobDTevpfECDlb5zGoA2k9fvx4wx6hsaRILo4z5JyVXjaTAZ1CAac4bEhZSisE1oBAtpVNPHx4MUdHR0XhZuQ2NjnFRtsS0TicczzRFIGm4h4qDP2nePkcyj6VKwRyIpCtac9ARYlECoHSR8oASryuPCeQcV6Q0JyIKO6KmJPuMfY6FwJTI5CNSKckqT4gqc+vHrWpPfZ6bXRVCMwLgWxN+zmZHXtfc9JdugoBIVAeAqsYbIphl/cVI6JzISAEhiCQjUjphzx//vwQXUZLyyj1y5cvR8tfGQsBIbBuBLIRKfMima9p0rSqZ0rY0e3QfbjMasAThtRLE6ZknT59ujS1pI8QmA0Co/SRltYHGS4RPVTNMAkfPfhyZ2nCVCdWWR36ZVMaLtJHCLRFYBQibVv4GuKV9lJZA+ayUQhMjUC2pj37avJpDclxBDSwpTtCCCwfgWxESt9fqbusMxB269YtvyMVcqj+W76RVOp+n+hmcih8lv+4ycKlIpCFSHnwSt4hn8pjQr55h1N6iUZKEChr60tcGEAfKZun2EDYlPgs9cGSXetCQH2k66pvWSsEhMAICGTxSEfQaxVZqgm9imqWkStAIBuRlj4hf8wPzPW9T0prQovY+9ak0q0dgWxNeyOqU6dOjf45j66VZvt/MlfyUEIfKXuQMiDH/qglCXXH7ljsjs+uT5qyVVLtSJc5IJBt96eSt11jxH7qHfLjymf/UaaIlUai6EndMdhkGJXmKcdY6lwIlIZANo+0NMOkjxAQAkJgKgSy9ZGylvyQTeepAFM5QkAICIEYgR2RVg00VIXHGfENoOfPnx9ksnuTjjYhP9Y513lT+VZOOOk9V9nk07b8ujLrdMuRf13ZuiYE5o5Ar6Z9ajAiFVYCOOj1/fff++/a//DDDzuVptbXBnRKHmwqUbcS7iHpIASaEOhMpFUEVBXepICuCwEhIATmjsCxPtI2TThGdFPx5jjSm7IjZ4WOnX+Trocuv0k/XRcCS0HgGJG2JcO6eDy8h3yA25RtcersyFHBTfm30XWIHmH5VlaXMrvEHaKn0gqBuSPQqWnPg9VEDiUBUpq+pelTUl1JFyEwZwQ6EWnphoqoSq8h6ScElonA3vSnN2/e+L07mQ7DwYqXpnXqXCfNZ599NjlKbP3G9KbLly9Xessszzx79uyk30uizDt37vgPAjLHdmqx8qkTVlSlpK7pDq5XrlzZfdCQ/Pi2E/kdwp6U/goTAqUgsFsiak32EydO+L1FeWBevHjRav9MlhiyVrtqY+cxPUWWXKJ7HdmzPNPW24fAkyZe2hrq2kdv8mS/AcqERCGjQ3wLifLPnDnj66SqO6YqHIzAlb1Tbf/UML9Sbl7pIQRKQaBy1J511108zHAte+zp2AMbh+cCgYe8Sa5du3aMNPGg2aQjlpBc6ogmTsc5XhsevKUzvdrmkxMf8oJIh0ic3vYJyKnnEP2UVgiUgoAnUnsw2jzwFrfuYarKh26DQwkkZ+XTbIVY+c3ZTMXzxAMkzzoPuQqDKtyq4teF58wrLmfMvOOydC4E5oCAJ9K2DwZ9kRcuXPB9kh988IHvd4zF+tZo6vOtdAiFg2/L47U+fPjQN3nxdu16nEffcz6+R96sYjLdIFDb/YllrAh9hoRzzuc/cgh2YydNYfIPvV3CwAyb0Q0MTOiHRmd+uYZHm1voq6WO7QhfHtQButH/Sfn2uZHcOig/IbBoBJxnmRTXp7jlMHEbkmxdU8+fvn37duvIYeuIcesefB/myIMvy/lfhOvhuV0nHHFEc+y6D+wp6OCI2+eJoCtlm8S6EB7q1rPYvWToEWJm5RhGnIOh84Z9WkdoW/ede/8fcYTn9UL/XEJ+N2/e9NlRB5QPVkiME7qHuvI/1I80Y+DmlZEIgRkj0Hr3J7waBmwQPBsSLOx+AAAC+ElEQVQGpc6dO7dxD6kPs3D34Pn/Vf2WFh4P8lgefX7RBw/YNiWO9x6lTPPGwvxdvfUpLpmmLi/DjYToZi2AGzdu+P82QwIvFo8UzzCnWF8ndUP+lINXyrkNEOKRhl60ld+2tZJTX+UlBOaGgB+1hwSaHhiaqzTPwwcM8rJRXcvnEACEJFZlB3FisquK28eGPnmBHSRLf20uMRur9AlfMvyHUCFXSJ36TEnV/VEVnspDYUJgyQi09kjxalIeS+htGVCphzkmsSpQ28arSt8UPnb+TeWH1yEyPP1YeGn11RMCrSLRsBxIEwJl/u29e/eOkWhcdlV+VeGxPToXAktH4Nhgk02sN6PD6S8049m4mUEaE4iVBzEWe8BCj6XtQ9c2Xlxm1XkVKUBi6I89fUbYU+VZnlyrGrQJ7QNTdGCuqQn/GQDKjYPlTx1SLt0geKPmoYLBq1evfAujy+wK7AzviRQuChMCi0fAEc1ObECIwQkGHmxgiAj8d6TpByuIRxzO7ZrzbvxABL/EtYEM99BuXT+cD6+67h5iP4DVRxwRbSmDvCmTssOyuI6uYdmUQxrnlW0vXbrUp9hkGuykHLADoxADbOQ6ZXLwH3HdJT6NHQw45ZSjo6NjdYbdVq+GE/qiB9ccsfpBL/SzgSk3u8GnSeGILTZ41rcOc9qrvITAIRDwa+2d+BcGXpBNjbFBI66F3hEeCF4LYnH8iZMwbpyu63XLc+m/4MSXBcAU73CMj+NZnaXyxxPFK6YubfpWFx3Im26CqsHFpdef7BMCINBq05IUKc4dvrFtGjv/Q+G/VLsOhafKXQYCe4NNPCiI/fJ/rP46KycsaypYzaYcZafyGBOzqTBKldPGrhQeqbwUJgSWgkCjR9rXA+mbbiiwhyoXvQ9Z9lDc6tIv1a46m3VNCHRBYI9I9dB0gW8/rvDbx0QhQmDpCOw17ds03UJQhjTjhqQttWK64leaHVYnfeqmT5rS7Jc+QqAPAnseaZ9MlEYICAEhsGYE/gsG6jp7QYj7yAAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequences of words are complicated\n",
    "\n",
    "Of course text isn't just about words but rather sequences of words like sentences, paragraphs, etc.\n",
    "These sequences have their own properties that add complications on top of words.\n",
    "Consider these two sentences:\n",
    "\n",
    "> Do you want to drive me?\n",
    "\n",
    "> Do you want me to drive?\n",
    "\n",
    "These two sentences use the same words, and the words refer to the same things.\n",
    "However, the order of the words changes who the driver is, `me` or `you`.\n",
    "We can also put the same words together in ways that are considered ungrammatical, e.g.:\n",
    "\n",
    "> Me to drive do you want?\n",
    "\n",
    "These examples show that different orderings of the same words can mean different things or be ungrammatical, even when the meanings of the words themselves don't change.\n",
    "\n",
    "In linguistics, whether a sequence of words is grammatical is a question of **syntax**.\n",
    "Syntactic rules are typically defined in terms of parts of speech like nouns, verbs, etc., and the application of these rules gives a *parse tree* for a sentence like this:\n",
    "\n",
    "[![image.png](attachment:image.png)](https://commons.wikimedia.org/wiki/File:Parse_tree_1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, syntax defined strictly in terms of parts of speech can be *underdetermined*. \n",
    "Consider these two sentences:\n",
    "\n",
    "> John saw the man with the shoes.\n",
    "\n",
    "> John saw the man with the binoculars.\n",
    "\n",
    "In the first case, we understand that the man has the shoes.\n",
    "However in the second case, we're not sure who has the binoculars.\n",
    "It could be John, who is using them to do the seeing, or is could be the man who is carrying them.\n",
    "This second case shows how the individual words can be important for building the right parse tree, i.e. the parse tree that matches the meaning of the sentence.\n",
    "\n",
    "In linguistics, what a sequence of words means is a question of **semantics** and **pragmatics**.\n",
    "Semantics tends to follow the structure of the syntax, as in the examples above, but also captures invariances like active/passive voice:\n",
    "\n",
    "> John saw the man\n",
    "\n",
    "> The man was seen by John\n",
    "\n",
    "These two sentences mean the same thing even though the subjects and objects have been reversed.\n",
    "Pragmatics goes even further from surface meaning by considering what the speaker *intended* rather than what was *said*.\n",
    "A classic example is\n",
    "\n",
    "> Can you pass the salt?\n",
    "\n",
    "The correct response to this is not a yes/no answer but rather the physical act of passing the salt, because this is a *request*, not an information-seeking question.\n",
    "\n",
    "Larger sequences of text beyond the sentence likewise introduce additional complications and structure.\n",
    "For example, essays typically have introductions and conclusions, and we expect *narrative* and *expository* text to have their own distinct conventions related to their purposes.\n",
    "However, in this course we will primarily focus on words and to some extent, sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Words/sentences as latent variables\n",
    "\n",
    "Why are words so complicated?\n",
    "Let's step back for a moment and consider non-complicated data and how it is different.\n",
    "\n",
    "One example of non-complicated data is height.\n",
    "When we want to measure height, we get a device like a measuring stick and read height directly off the stick.\n",
    "Another example is counting cars passing by us on the street.\n",
    "How are words different from measuring height or counting cars?\n",
    "\n",
    "One way of understanding the difference is in terms of **observable variables** and **latent variables.**\n",
    "Height and cars passing by are things we can directly observe and measure - we can use our perception and instruments (like a measuring stick) to quantify them.\n",
    "Latent variables, on the other hand, cannot be directly observed; instead we must **infer** their values using observed variables that are *associated* with the latent variable.\n",
    "Importantly, latent variables can have *many* associated observed variables. \n",
    "\n",
    "When you think about words, it makes sense to think of the **meaning as a latent variable and words as an observed variable.**\n",
    "At any given moment, a speaker has a meaning they want to convey and chooses some words to express that meaning, but in a different moment they may use different words to express the same meaning.\n",
    "There is clearly an *association* between the words used and the meaning, but it is not an exact one-to-one relationship. \n",
    "Rather one meaning can be expressed by many words (e.g. `fly` and `housefly`), and one word can express many different meanings (e.g.`fly` as insect, motion verb, or adjective meaning 'cool').\n",
    "\n",
    "When we think about words this way, we can intuitively see that even though multiple words can be used to express the same meaning, not all words are equally likely.\n",
    "Consider the table of probabilities below for words that a child might use to refer to their father:\n",
    "\n",
    "| Word   | Probability |\n",
    "|--------|-------------|\n",
    "| dad    | .82         |\n",
    "| father | .15         |\n",
    "| Steve  | .03         |\n",
    "\n",
    "We call this a **distribution of words** over the latent variable, with `dad` being the most likely, and the proper name `Steve` being the least likely, all summing to 1 as a probability must.\n",
    "\n",
    "Just as with words, we can also think of a latent variable representing the meaning of the sentence and a the many *sequences* of words as the observed variable.\n",
    "For example, we might consider:\n",
    "\n",
    "> John saw the man\n",
    "\n",
    "> The man was seen by John\n",
    "\n",
    "as observed sequences of words generated by the same latent variable.\n",
    "While in theory it makes sense to think of sentences this way, in practice we can do more if we assume *composition* of multiple latent variables in a sentence.\n",
    "We assume that compositionality of latent variables explains the systematicity in language, e.g. the recurring patterns and \"rules\" that seem to structure language.\n",
    "\n",
    "If all of this seems a little be abstract and not very useful to you, consider that some of the language phenomenon we've already talked about are obviously latent variables (you can't directly observe them).\n",
    "For example, we all agree that **parts of speech** like noun, verb, etc, exist, but notice you can never directly observe \"noun\" but only a word that acts like a noun in a given sentence.\n",
    "Likewise, we all agree that there is some kind of structure in sentences, like parse trees, even though we don't directly observe them.\n",
    "For both parts of speech and parse trees, we must infer their existence and properties from the patterns of words (which we do directly observe)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moving forward with text as data\n",
    "\n",
    "So where does this leave us?\n",
    "Can we do anything with text as data?\n",
    "The answer is most assuredly YES, but we need to think carefully about how to *transform* text into structured data that we can use to answer our research questions.\n",
    "We also need to be aware of the *limitations* of transforming the data - have we distorted the thing we are trying to study through our transformation?\n",
    "\n",
    "If we make a mistake, there are different consequences depending on what we do with the data.\n",
    "If we are trying to show that two different collections of text are different from each other, a bad transformation may cause us to find a difference that doesn't exist (what statisticians call a type 1 error) or fail to find a difference that does exist (what statisticians call a type 2 error).\n",
    "On the other hand, if our only goal is to predict something using text as data, then we may create a variable (also called a feature or predictor in this context) that doesn't work as well as it could.\n",
    "\n",
    "The key to success when using text as data is to familiarize yourself with common transformations of text into structured data, understand which transformations are more appropriate for specific questions, and pick up enough basic knowledge in linguistics that you can modify these transformations to new situations that you encounter.\n",
    "This is the focus of the rest of this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}