Descriptive-statistics-length-PS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Descriptive statistics length-based metrics: Problem solving\n",
    "\n",
    "In this session, we'll work with medication pamphlet data from the [Patient Information Leaflet (PIL) Corpus](http://mcs.open.ac.uk/nlg/old_projects/pills/corpus/PIL/)\n",
    "\n",
    "Our question of interest is whether medication instructions could have an effect on medication overdoses (suppose a colleague has the overdose data).\n",
    "We will calculate [Flesch Kincaid Grade Level (FKGL)](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests) as a metric of text difficulty, under the hypothesis that if the medication instructions are too difficult, patients will be less likely to understand them and overdose.\n",
    "\n",
    "In theory, this corpus is available from NLTK, but [it has been broken for some time](https://github.com/nltk/nltk/issues/1851).\n",
    "Therefore we have included the first 50 files of the corpus in the `datasets/pil` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "\n",
    "Start by importing `nltk` and importing `reader` from `nltk.corpus`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk as nltk\n",
    "from nltk.corpus import reader\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"c$0G$;i#PH-Rh{N@jV}`\">reader</variable></variables><block type=\"importAs\" id=\"aN-,nCP!gVNy`aijN.:*\" x=\"3\" y=\"126\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><next><block type=\"importFrom\" id=\"`p5.6(Uv@:/Rqfl0Ndp!\"><field name=\"libraryName\">nltk.corpus</field><field name=\"libraryAlias\" id=\"c$0G$;i#PH-Rh{N@jV}`\">reader</field></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `reader` create `PlaintextCorpusReader` using a list containing `\"datasets/pil\"` and `\".*\"`, then store the result in a new variable `pil`.\n",
    "\n",
    "*Note: We haven't loaded our own corpus before, but it's pretty easy with NLTK - just give it the name of the folder with the text files and a regular expression pattern (in our case, a wildcard, `.*`). \n",
    "This will use the default word/sentence tokenizers, but you can override these, use a different `CorpusReader`, or implement your own.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pil = reader.PlaintextCorpusReader('datasets/pil', '.*')\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"jS$TX:+[N5`f|}~n(My(\">pil</variable><variable id=\"c$0G$;i#PH-Rh{N@jV}`\">reader</variable></variables><block type=\"variables_set\" id=\"EeAkbuz6YwB^hQdQhcUw\" x=\"54\" y=\"256\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"B@hsB@Upeucpk|sNF0^n\"><field name=\"VAR\" id=\"c$0G$;i#PH-Rh{N@jV}`\">reader</field><field name=\"MEMBER\">PlaintextCorpusReader</field><data>reader:PlaintextCorpusReader</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"IPJ8Dvf)InkJrOSj!;xc\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"text\" id=\"J8H~r:b,elkI,?{v(WZy\"><field name=\"TEXT\">datasets/pil</field></block></value><value name=\"ADD1\"><block type=\"text\" id=\"7}`avk29E@uM$xO18M?P\"><field name=\"TEXT\">.*</field></block></value></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now use `pil` like `gutenberg` in the worked example notebook for this module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate length-based metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word lengths\n",
    "\n",
    "Get the word lengths for each text in the corpus and store in `wordLengths`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1326, 1, 10633, 2155, 1700, 1088, 639, 1166, 601, 2086, 1533, 1404, 1400, 1831, 922, 1257, 1053, 1313, 2147, 1164, 1222, 1766, 1566, 1206, 1003, 1003, 1996, 2725, 3399, 1180, 2043, 1125, 1784, 864, 1607, 801, 709, 1576, 1327, 1019, 1305, 1125, 1298, 447, 8940, 1070, 1021, 2781, 1430, 8108]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wordLengths = [(len(pil.words(i))) for i in (pil.fileids())]\n",
    "\n",
    "wordLengths\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</variable><variable id=\"LZ#}.J~9XYczA[nu4?|Q\">i</variable><variable id=\"jS$TX:+[N5`f|}~n(My(\">pil</variable></variables><block type=\"variables_set\" id=\"SKZpRLP{wcl/g*{^W3WV\" x=\"4\" y=\"319\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"8eO0[B%0~mtEmLX:+=dw\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"qHXxT3|WBM:kMw~muDTN\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><field name=\"MEMBER\">fileids</field><data>pil:fileids</data></block></value><value name=\"YIELD\"><block type=\"lists_length\" id=\"b5(0SiwR87=9]SU8IBvy\"><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"#IalxaHkKH5=q1de@8Ar\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><field name=\"MEMBER\">words</field><data>pil:words</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"u[Lug07Y.B({Q]G-6du|\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"aJWMl/VaGmo=-c|`$nxp\" x=\"8\" y=\"436\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence lengths\n",
    "\n",
    "Get the sentence lengths for each text in the corpus and store in `sentenceLengths`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[120, 1, 146, 129, 135, 86, 45, 68, 40, 44, 100, 107, 104, 94, 83, 106, 89, 100, 52, 88, 88, 58, 84, 59, 46, 84, 66, 88, 91, 59, 61, 80, 64, 73, 34, 35, 59, 105, 96, 77, 97, 44, 66, 43, 133, 91, 84, 134, 83, 225]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentenceLengths = [(len(pil.sents(i))) for i in (pil.fileids())]\n",
    "\n",
    "sentenceLengths\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</variable><variable id=\"LZ#}.J~9XYczA[nu4?|Q\">i</variable><variable id=\"jS$TX:+[N5`f|}~n(My(\">pil</variable></variables><block type=\"variables_set\" id=\"SKZpRLP{wcl/g*{^W3WV\" x=\"4\" y=\"319\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"8eO0[B%0~mtEmLX:+=dw\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"qHXxT3|WBM:kMw~muDTN\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><field name=\"MEMBER\">fileids</field><data>pil:fileids</data></block></value><value name=\"YIELD\"><block type=\"lists_length\" id=\"b5(0SiwR87=9]SU8IBvy\"><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"#IalxaHkKH5=q1de@8Ar\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><field name=\"MEMBER\">sents</field><data>pil:sents</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"u[Lug07Y.B({Q]G-6du|\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"aJWMl/VaGmo=-c|`$nxp\" x=\"8\" y=\"436\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect in dataframe\n",
    "\n",
    "Make a `dataframe` with `fileids`, `wordLengths`,  and `sentenceLengths`\n",
    "Start by importing `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"_V`RIwppcpbRKT:m6^qH\">pd</variable></variables><block type=\"importAs\" id=\"Gy5)p-`[BHUUWE}k1DeL\" x=\"16\" y=\"10\"><field name=\"libraryName\">pandas</field><field name=\"libraryAlias\" id=\"_V`RIwppcpbRKT:m6^qH\">pd</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now make the `dataframe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>words</th>\n",
       "      <th>sentences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Acepril_Tablets.txt</td>\n",
       "      <td>1326</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Action_Asthma.txt</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Actrapid_Pen.txt</td>\n",
       "      <td>10633</td>\n",
       "      <td>146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adalat_LA_30.txt</td>\n",
       "      <td>2155</td>\n",
       "      <td>129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adcortyl.txt</td>\n",
       "      <td>1700</td>\n",
       "      <td>135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Adenocor.txt</td>\n",
       "      <td>1088</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Adenoscan.txt</td>\n",
       "      <td>639</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Adipine_MR_10.txt</td>\n",
       "      <td>1166</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Adipine_TM_10.txt</td>\n",
       "      <td>601</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Aerocrom_Inhaler.txt</td>\n",
       "      <td>2086</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Aerocrom_Syncroner.txt</td>\n",
       "      <td>1533</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Algitec_Chewtab_Tablets.txt</td>\n",
       "      <td>1404</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Algitec_Suspension.txt</td>\n",
       "      <td>1400</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Allegron.txt</td>\n",
       "      <td>1831</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Amoxil_Capsules.txt</td>\n",
       "      <td>922</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Amoxil_Paediatric_Susp.txt</td>\n",
       "      <td>1257</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Amoxil_Syrup.txt</td>\n",
       "      <td>1053</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ampliclox_Neonatal.txt</td>\n",
       "      <td>1313</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Amsidine_Injection.txt</td>\n",
       "      <td>2147</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Anafranil.txt</td>\n",
       "      <td>1164</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Anafranil_Syrup.txt</td>\n",
       "      <td>1222</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Androcur.txt</td>\n",
       "      <td>1766</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Angitil_SR.txt</td>\n",
       "      <td>1566</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Antepsin_Suspension.txt</td>\n",
       "      <td>1206</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Antepsin_Tablets.txt</td>\n",
       "      <td>1003</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Anturan.txt</td>\n",
       "      <td>1003</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Anugesic_HC_Cream_Supps.txt</td>\n",
       "      <td>1996</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Anusol_Cream.txt</td>\n",
       "      <td>2725</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Anusol_Suppositories.txt</td>\n",
       "      <td>3399</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Arimidex.txt</td>\n",
       "      <td>1180</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Artane.txt</td>\n",
       "      <td>2043</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Asacol_Tablets.txt</td>\n",
       "      <td>1125</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Ascabiol_Emulsion.txt</td>\n",
       "      <td>1784</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Atrovent_Aerocaps.txt</td>\n",
       "      <td>864</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Atrovent_Autohaler.txt</td>\n",
       "      <td>1607</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Atrovent_Forte_Inhaler.txt</td>\n",
       "      <td>801</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Atrovent_UDVs.txt</td>\n",
       "      <td>709</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Augmentin_125_Susp.txt</td>\n",
       "      <td>1576</td>\n",
       "      <td>105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Augmentin_375mg.txt</td>\n",
       "      <td>1327</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Augmentin_Intravenous.txt</td>\n",
       "      <td>1019</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Augmentin_disp_375mg.txt</td>\n",
       "      <td>1305</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Avloclor_Tablets.txt</td>\n",
       "      <td>1125</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Axid_Capsules.txt</td>\n",
       "      <td>1298</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Axsain_Cream.txt</td>\n",
       "      <td>447</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>BDP_Spacehaler.txt</td>\n",
       "      <td>8940</td>\n",
       "      <td>133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Bactroban_Nasal_Ointment.txt</td>\n",
       "      <td>1070</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Bactroban_Ointment.txt</td>\n",
       "      <td>1021</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Becotide_50_Inhaler.txt</td>\n",
       "      <td>2781</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Berotec_100_Inhaler.txt</td>\n",
       "      <td>1430</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Betaferon.txt</td>\n",
       "      <td>8108</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/latex": [
       "None"
      ],
      "text/plain": [
       "                          corpus  words  sentences\n",
       "0            Acepril_Tablets.txt   1326        120\n",
       "1              Action_Asthma.txt      1          1\n",
       "2               Actrapid_Pen.txt  10633        146\n",
       "3               Adalat_LA_30.txt   2155        129\n",
       "4                   Adcortyl.txt   1700        135\n",
       "5                   Adenocor.txt   1088         86\n",
       "6                  Adenoscan.txt    639         45\n",
       "7              Adipine_MR_10.txt   1166         68\n",
       "8              Adipine_TM_10.txt    601         40\n",
       "9           Aerocrom_Inhaler.txt   2086         44\n",
       "10        Aerocrom_Syncroner.txt   1533        100\n",
       "11   Algitec_Chewtab_Tablets.txt   1404        107\n",
       "12        Algitec_Suspension.txt   1400        104\n",
       "13                  Allegron.txt   1831         94\n",
       "14           Amoxil_Capsules.txt    922         83\n",
       "15    Amoxil_Paediatric_Susp.txt   1257        106\n",
       "16              Amoxil_Syrup.txt   1053         89\n",
       "17        Ampliclox_Neonatal.txt   1313        100\n",
       "18        Amsidine_Injection.txt   2147         52\n",
       "19                 Anafranil.txt   1164         88\n",
       "20           Anafranil_Syrup.txt   1222         88\n",
       "21                  Androcur.txt   1766         58\n",
       "22                Angitil_SR.txt   1566         84\n",
       "23       Antepsin_Suspension.txt   1206         59\n",
       "24          Antepsin_Tablets.txt   1003         46\n",
       "25                   Anturan.txt   1003         84\n",
       "26   Anugesic_HC_Cream_Supps.txt   1996         66\n",
       "27              Anusol_Cream.txt   2725         88\n",
       "28      Anusol_Suppositories.txt   3399         91\n",
       "29                  Arimidex.txt   1180         59\n",
       "30                    Artane.txt   2043         61\n",
       "31            Asacol_Tablets.txt   1125         80\n",
       "32         Ascabiol_Emulsion.txt   1784         64\n",
       "33         Atrovent_Aerocaps.txt    864         73\n",
       "34        Atrovent_Autohaler.txt   1607         34\n",
       "35    Atrovent_Forte_Inhaler.txt    801         35\n",
       "36             Atrovent_UDVs.txt    709         59\n",
       "37        Augmentin_125_Susp.txt   1576        105\n",
       "38           Augmentin_375mg.txt   1327         96\n",
       "39     Augmentin_Intravenous.txt   1019         77\n",
       "40      Augmentin_disp_375mg.txt   1305         97\n",
       "41          Avloclor_Tablets.txt   1125         44\n",
       "42             Axid_Capsules.txt   1298         66\n",
       "43              Axsain_Cream.txt    447         43\n",
       "44            BDP_Spacehaler.txt   8940        133\n",
       "45  Bactroban_Nasal_Ointment.txt   1070         91\n",
       "46        Bactroban_Ointment.txt   1021         84\n",
       "47       Becotide_50_Inhaler.txt   2781        134\n",
       "48       Berotec_100_Inhaler.txt   1430         83\n",
       "49                 Betaferon.txt   8108        225"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = pd.DataFrame(zip(pil.fileids(), wordLengths, sentenceLengths), columns=['corpus','words','sentences'])\n",
    "\n",
    "dataframe\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</variable><variable id=\"_V`RIwppcpbRKT:m6^qH\">pd</variable><variable id=\"jS$TX:+[N5`f|}~n(My(\">pil</variable><variable id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</variable><variable id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</variable></variables><block type=\"variables_set\" id=\"Nnj3jwtJa6+~cV1=RDn_\" x=\"-149\" y=\"237\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"*C0V~q-fS]kDUMRq`O-N\"><field name=\"VAR\" id=\"_V`RIwppcpbRKT:m6^qH\">pd</field><field name=\"MEMBER\">DataFrame</field><data>pd:DataFrame</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"DoM~-@qI)6TgbDc;vBMb\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"zipBlock\" id=\"nv7/65]-+;=,M.B)yU%U\"><value name=\"x\"><block type=\"lists_create_with\" id=\"@PN$;KCRy[Jv;QJ+d#($\"><mutation items=\"3\"></mutation><value name=\"ADD0\"><block type=\"varDoMethod\" id=\"FJ#DraHg!(/g)_[-^Dn]\"><field name=\"VAR\" id=\"jS$TX:+[N5`f|}~n(My(\">pil</field><field name=\"MEMBER\">fileids</field><data>pil:fileids</data></block></value><value name=\"ADD1\"><block type=\"variables_get\" id=\"1UbO_xg6I)qqO#p=]Trh\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field></block></value><value name=\"ADD2\"><block type=\"variables_get\" id=\"%~eQRKbkLX{by+raZ$LQ\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field></block></value></block></value></block></value><value name=\"ADD1\"><block type=\"dummyOutputCodeBlock\" id=\"o35uqta54?^UVk|[,.|(\"><field name=\"CODE\">columns=['corpus','words','sentences']</field></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"(gZ^x=Q!@}~:|xjc+ZXy\" x=\"-133\" y=\"397\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "Do you see any problems?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Action_Asthma.txt has one word and one sentence. It should be disregarded in the analysis.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Readability\n",
    "\n",
    "Calculate FKGL for each text using this formula and add a new FKGL column to the `dataframe`:\n",
    "\n",
    "\\begin{equation*}\n",
    "0.39 \\left( \\frac{\\mbox{total words}}{\\mbox{total sentences}} \\right) +11.8 \\left( \\frac{\\mbox{total syllables}}{\\mbox{total words}} \\right) - 15.59\n",
    "\\end{equation*}\n",
    "\n",
    "*Note assume 1.5 syllables per word.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>words</th>\n",
       "      <th>sentences</th>\n",
       "      <th>fkgl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Acepril_Tablets.txt</td>\n",
       "      <td>1326</td>\n",
       "      <td>120</td>\n",
       "      <td>6.419500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Action_Asthma.txt</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Actrapid_Pen.txt</td>\n",
       "      <td>10633</td>\n",
       "      <td>146</td>\n",
       "      <td>30.513219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adalat_LA_30.txt</td>\n",
       "      <td>2155</td>\n",
       "      <td>129</td>\n",
       "      <td>8.625116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adcortyl.txt</td>\n",
       "      <td>1700</td>\n",
       "      <td>135</td>\n",
       "      <td>7.021111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Adenocor.txt</td>\n",
       "      <td>1088</td>\n",
       "      <td>86</td>\n",
       "      <td>7.043953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Adenoscan.txt</td>\n",
       "      <td>639</td>\n",
       "      <td>45</td>\n",
       "      <td>7.648000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Adipine_MR_10.txt</td>\n",
       "      <td>1166</td>\n",
       "      <td>68</td>\n",
       "      <td>8.797353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Adipine_TM_10.txt</td>\n",
       "      <td>601</td>\n",
       "      <td>40</td>\n",
       "      <td>7.969750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Aerocrom_Inhaler.txt</td>\n",
       "      <td>2086</td>\n",
       "      <td>44</td>\n",
       "      <td>20.599545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Aerocrom_Syncroner.txt</td>\n",
       "      <td>1533</td>\n",
       "      <td>100</td>\n",
       "      <td>8.088700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Algitec_Chewtab_Tablets.txt</td>\n",
       "      <td>1404</td>\n",
       "      <td>107</td>\n",
       "      <td>7.227383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Algitec_Suspension.txt</td>\n",
       "      <td>1400</td>\n",
       "      <td>104</td>\n",
       "      <td>7.360000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Allegron.txt</td>\n",
       "      <td>1831</td>\n",
       "      <td>94</td>\n",
       "      <td>9.706702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Amoxil_Capsules.txt</td>\n",
       "      <td>922</td>\n",
       "      <td>83</td>\n",
       "      <td>6.442289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Amoxil_Paediatric_Susp.txt</td>\n",
       "      <td>1257</td>\n",
       "      <td>106</td>\n",
       "      <td>6.734811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Amoxil_Syrup.txt</td>\n",
       "      <td>1053</td>\n",
       "      <td>89</td>\n",
       "      <td>6.724270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ampliclox_Neonatal.txt</td>\n",
       "      <td>1313</td>\n",
       "      <td>100</td>\n",
       "      <td>7.230700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Amsidine_Injection.txt</td>\n",
       "      <td>2147</td>\n",
       "      <td>52</td>\n",
       "      <td>18.212500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Anafranil.txt</td>\n",
       "      <td>1164</td>\n",
       "      <td>88</td>\n",
       "      <td>7.268636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Anafranil_Syrup.txt</td>\n",
       "      <td>1222</td>\n",
       "      <td>88</td>\n",
       "      <td>7.525682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Androcur.txt</td>\n",
       "      <td>1766</td>\n",
       "      <td>58</td>\n",
       "      <td>13.984828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Angitil_SR.txt</td>\n",
       "      <td>1566</td>\n",
       "      <td>84</td>\n",
       "      <td>9.380714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Antepsin_Suspension.txt</td>\n",
       "      <td>1206</td>\n",
       "      <td>59</td>\n",
       "      <td>10.081864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Antepsin_Tablets.txt</td>\n",
       "      <td>1003</td>\n",
       "      <td>46</td>\n",
       "      <td>10.613696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Anturan.txt</td>\n",
       "      <td>1003</td>\n",
       "      <td>84</td>\n",
       "      <td>6.766786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Anugesic_HC_Cream_Supps.txt</td>\n",
       "      <td>1996</td>\n",
       "      <td>66</td>\n",
       "      <td>13.904545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Anusol_Cream.txt</td>\n",
       "      <td>2725</td>\n",
       "      <td>88</td>\n",
       "      <td>14.186705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Anusol_Suppositories.txt</td>\n",
       "      <td>3399</td>\n",
       "      <td>91</td>\n",
       "      <td>16.677143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Arimidex.txt</td>\n",
       "      <td>1180</td>\n",
       "      <td>59</td>\n",
       "      <td>9.910000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Artane.txt</td>\n",
       "      <td>2043</td>\n",
       "      <td>61</td>\n",
       "      <td>15.171803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Asacol_Tablets.txt</td>\n",
       "      <td>1125</td>\n",
       "      <td>80</td>\n",
       "      <td>7.594375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Ascabiol_Emulsion.txt</td>\n",
       "      <td>1784</td>\n",
       "      <td>64</td>\n",
       "      <td>12.981250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Atrovent_Aerocaps.txt</td>\n",
       "      <td>864</td>\n",
       "      <td>73</td>\n",
       "      <td>6.725890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Atrovent_Autohaler.txt</td>\n",
       "      <td>1607</td>\n",
       "      <td>34</td>\n",
       "      <td>20.543235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Atrovent_Forte_Inhaler.txt</td>\n",
       "      <td>801</td>\n",
       "      <td>35</td>\n",
       "      <td>11.035429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Atrovent_UDVs.txt</td>\n",
       "      <td>709</td>\n",
       "      <td>59</td>\n",
       "      <td>6.796610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Augmentin_125_Susp.txt</td>\n",
       "      <td>1576</td>\n",
       "      <td>105</td>\n",
       "      <td>7.963714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Augmentin_375mg.txt</td>\n",
       "      <td>1327</td>\n",
       "      <td>96</td>\n",
       "      <td>7.500938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Augmentin_Intravenous.txt</td>\n",
       "      <td>1019</td>\n",
       "      <td>77</td>\n",
       "      <td>7.271169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Augmentin_disp_375mg.txt</td>\n",
       "      <td>1305</td>\n",
       "      <td>97</td>\n",
       "      <td>7.356907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Avloclor_Tablets.txt</td>\n",
       "      <td>1125</td>\n",
       "      <td>44</td>\n",
       "      <td>12.081591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Axid_Capsules.txt</td>\n",
       "      <td>1298</td>\n",
       "      <td>66</td>\n",
       "      <td>9.780000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Axsain_Cream.txt</td>\n",
       "      <td>447</td>\n",
       "      <td>43</td>\n",
       "      <td>6.164186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>BDP_Spacehaler.txt</td>\n",
       "      <td>8940</td>\n",
       "      <td>133</td>\n",
       "      <td>28.325038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Bactroban_Nasal_Ointment.txt</td>\n",
       "      <td>1070</td>\n",
       "      <td>91</td>\n",
       "      <td>6.695714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Bactroban_Ointment.txt</td>\n",
       "      <td>1021</td>\n",
       "      <td>84</td>\n",
       "      <td>6.850357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Becotide_50_Inhaler.txt</td>\n",
       "      <td>2781</td>\n",
       "      <td>134</td>\n",
       "      <td>10.203955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Berotec_100_Inhaler.txt</td>\n",
       "      <td>1430</td>\n",
       "      <td>83</td>\n",
       "      <td>8.829277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Betaferon.txt</td>\n",
       "      <td>8108</td>\n",
       "      <td>225</td>\n",
       "      <td>16.163867</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/latex": [
       "None"
      ],
      "text/plain": [
       "                          corpus  words  sentences       fkgl\n",
       "0            Acepril_Tablets.txt   1326        120   6.419500\n",
       "1              Action_Asthma.txt      1          1   2.500000\n",
       "2               Actrapid_Pen.txt  10633        146  30.513219\n",
       "3               Adalat_LA_30.txt   2155        129   8.625116\n",
       "4                   Adcortyl.txt   1700        135   7.021111\n",
       "5                   Adenocor.txt   1088         86   7.043953\n",
       "6                  Adenoscan.txt    639         45   7.648000\n",
       "7              Adipine_MR_10.txt   1166         68   8.797353\n",
       "8              Adipine_TM_10.txt    601         40   7.969750\n",
       "9           Aerocrom_Inhaler.txt   2086         44  20.599545\n",
       "10        Aerocrom_Syncroner.txt   1533        100   8.088700\n",
       "11   Algitec_Chewtab_Tablets.txt   1404        107   7.227383\n",
       "12        Algitec_Suspension.txt   1400        104   7.360000\n",
       "13                  Allegron.txt   1831         94   9.706702\n",
       "14           Amoxil_Capsules.txt    922         83   6.442289\n",
       "15    Amoxil_Paediatric_Susp.txt   1257        106   6.734811\n",
       "16              Amoxil_Syrup.txt   1053         89   6.724270\n",
       "17        Ampliclox_Neonatal.txt   1313        100   7.230700\n",
       "18        Amsidine_Injection.txt   2147         52  18.212500\n",
       "19                 Anafranil.txt   1164         88   7.268636\n",
       "20           Anafranil_Syrup.txt   1222         88   7.525682\n",
       "21                  Androcur.txt   1766         58  13.984828\n",
       "22                Angitil_SR.txt   1566         84   9.380714\n",
       "23       Antepsin_Suspension.txt   1206         59  10.081864\n",
       "24          Antepsin_Tablets.txt   1003         46  10.613696\n",
       "25                   Anturan.txt   1003         84   6.766786\n",
       "26   Anugesic_HC_Cream_Supps.txt   1996         66  13.904545\n",
       "27              Anusol_Cream.txt   2725         88  14.186705\n",
       "28      Anusol_Suppositories.txt   3399         91  16.677143\n",
       "29                  Arimidex.txt   1180         59   9.910000\n",
       "30                    Artane.txt   2043         61  15.171803\n",
       "31            Asacol_Tablets.txt   1125         80   7.594375\n",
       "32         Ascabiol_Emulsion.txt   1784         64  12.981250\n",
       "33         Atrovent_Aerocaps.txt    864         73   6.725890\n",
       "34        Atrovent_Autohaler.txt   1607         34  20.543235\n",
       "35    Atrovent_Forte_Inhaler.txt    801         35  11.035429\n",
       "36             Atrovent_UDVs.txt    709         59   6.796610\n",
       "37        Augmentin_125_Susp.txt   1576        105   7.963714\n",
       "38           Augmentin_375mg.txt   1327         96   7.500938\n",
       "39     Augmentin_Intravenous.txt   1019         77   7.271169\n",
       "40      Augmentin_disp_375mg.txt   1305         97   7.356907\n",
       "41          Avloclor_Tablets.txt   1125         44  12.081591\n",
       "42             Axid_Capsules.txt   1298         66   9.780000\n",
       "43              Axsain_Cream.txt    447         43   6.164186\n",
       "44            BDP_Spacehaler.txt   8940        133  28.325038\n",
       "45  Bactroban_Nasal_Ointment.txt   1070         91   6.695714\n",
       "46        Bactroban_Ointment.txt   1021         84   6.850357\n",
       "47       Becotide_50_Inhaler.txt   2781        134  10.203955\n",
       "48       Berotec_100_Inhaler.txt   1430         83   8.829277\n",
       "49                 Betaferon.txt   8108        225  16.163867"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = dataframe.assign(fkgl= ((0.39 * (dataframe['words'] / dataframe['sentences']) + 11.8 * 1.5) - 15.59))\n",
    "\n",
    "dataframe\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</variable></variables><block type=\"variables_set\" id=\"d3ELqhcW@UA^R%PLy3cV\" x=\"-83\" y=\"308\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"!?g`gMP):imN8F)T,@V|\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><field name=\"MEMBER\">assign</field><data>dataframe:assign</data><value name=\"INPUT\"><block type=\"valueOutputCodeBlock\" id=\"k^_3ol/yS5*#Dha3rPVq\"><field name=\"CODE\">fkgl=</field><value name=\"INPUT\"><block type=\"math_arithmetic\" id=\"Md5u*{OHNtri0AjDj],6\"><field name=\"OP\">MINUS</field><value name=\"A\"><shadow type=\"math_number\" id=\"oOOHvHpa1xT0^Jls*1X+\"><field name=\"NUM\">0.39</field></shadow><block type=\"math_arithmetic\" id=\"QGJQ@m3mx5!l=,g9-R+d\"><field name=\"OP\">ADD</field><value name=\"A\"><shadow type=\"math_number\" id=\"X]w:+`6NY)?+=r#3v?Aj\"><field name=\"NUM\">0.39</field></shadow><block type=\"math_arithmetic\" id=\",gsasO{}sRYc]60t@#ud\"><field name=\"OP\">MULTIPLY</field><value name=\"A\"><shadow type=\"math_number\" id=\"^KWa~qEgh3Q5R?==H)+9\"><field name=\"NUM\">0.39</field></shadow></value><value name=\"B\"><shadow type=\"math_number\" id=\"?4DQu[d7x;oy^4iY|Y3(\"><field name=\"NUM\">1</field></shadow><block type=\"math_arithmetic\" id=\"b1}M?uM#pV~iqEz`%Dh~\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"/t1A;EYsDfw+BYK-f{P8\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"/`cSF)tsx%3]HefEnKlV\"><field name=\"TEXT\">words</field></block></value></block></value><value name=\"B\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"9_HN7=?;UGbK`ZFpBD)n\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"SDk,ItV,c;N.~+Z%vOH]\"><field name=\"TEXT\">sentences</field></block></value></block></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"osy!WzPZ^Ac.Li?,M]Sq\"><field name=\"NUM\">1</field></shadow><block type=\"math_arithmetic\" id=\"pns2jEFkEFV2potV{wzJ\"><field name=\"OP\">MULTIPLY</field><value name=\"A\"><shadow type=\"math_number\" id=\"~z95el]}Uct`QV~*3@LC\"><field name=\"NUM\">11.8</field></shadow></value><value name=\"B\"><shadow type=\"math_number\" id=\"L,wO%x|x/NAv$%-5KtZc\"><field name=\"NUM\">1.5</field></shadow></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"N6x=21,Sww{nHR%@JF|@\"><field name=\"NUM\">15.59</field></shadow></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"F^RD,W:_|bYpb/e8Wq3b\" x=\"-90\" y=\"410\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "Google the three highest FKGL entries. Do you see a common theme? What might we do next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Many of the higher FKGL are related to asthma. We might consider marking all asthma medications and comparing their FKGL to the rest of the dataset to see if there is a systematic difference.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}