21_Machine_Translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #21\n",
    "# Machine Translation\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Tutorial #20 showed how to use a Recurrent Neural Network (RNN) to do so-called sentiment analysis on texts of movie reviews. This tutorial will extend that idea to do Machine Translation of human languages by combining two RNN's.\n",
    "\n",
    "You should be familiar with TensorFlow, Keras and the basics of Natural Language Processing, see Tutorials #01, #03-C and #20."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart\n",
    "\n",
    "The following flowchart shows roughly how the neural network is constructed. It is split into two parts: An encoder which maps the source-text to a \"thought vector\" that summarizes the text's contents, which is then input to the second part of the neural network that decodes the \"thought vector\" to the destination-text.\n",
    "\n",
    "The neural network cannot work directly on text so first we need to convert each word to an integer-token using a tokenizer. But the neural network cannot work on integers either, so we use a so-called Embedding Layer to convert each integer-token to a vector of floating-point values. The embedding is trained alongside the rest of the neural network to map words with similar semantic meaning to similar vectors of floating-point values.\n",
    "\n",
    "For example, consider the Danish text \"der var engang\" which is the beginning of any fairytale and literally means \"there was once\" but is commonly translated into English as \"once upon a time\". We first convert the entire data-set to integer-tokens so the text \"der var engang\" becomes [12, 54, 1097]. Each of these integer-tokens is then mapped to an embedding-vector with e.g. 128 elements, so the integer-token 12 could for example become [0.12, -0.56, ..., 1.19] and the integer-token 54 could for example become [0.39, 0.09, ..., -0.12]. These embedding-vectors can then be input to the Recurrent Neural Network, which has 3 GRU-layers. See Tutorial #20 for a more detailed explanation.\n",
    "\n",
    "The last GRU-layer outputs a single vector - the \"thought vector\" that summarizes the contents of the source-text - which is then used as the initial state of the GRU-units in the decoder-part.\n",
    "\n",
    "The destination-text \"once upon a time\" is padded with special markers \"ssss\" and \"eeee\" to indicate its beginning and end, so the sequence of integer-tokens becomes [2, 337, 640, 9, 79, 3]. During training, the decoder will be given this entire sequence as input and  the desired output sequence is [337, 640, 9, 79, 3] which is the same sequence but time-shifted one step. We are trying to teach the decoder to map the \"thought vector\" and the start-token \"ssss\" (integer 2) to the next word \"once\" (integer 337), and then map the word \"once\" to the word \"upon\" (integer 640), and so forth.\n",
    "\n",
    "This flow-chart depicts the main idea but does not show all the necessary details e.g. regarding the loss function which is also somewhat complicated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Flowchart](images/21_machine_translation_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Model  # This does not work!\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Dense, GRU, Embedding\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and package versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.2.4-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "We will use the Europarl data-set which has sentence-pairs in most European languages. The data was created by the European Union which translates a lot of their communications to the languages of the member-countries of the European Union.\n",
    "\n",
    "http://www.statmt.org/europarl/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import europarl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial I have used the English-Danish data-set which contains about 2 million sentence-pairs. You can use another language by changing this language-code, see `europarl.py` for a list of available language-codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "language_code='da'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order for the decoder to know when to begin and end a sentence, we need to mark the start and end of each sentence with words that most likely don't occur in the data-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mark_start = 'ssss '\n",
    "mark_end = ' eeee'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the directory for the data-files if you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_dir = \"data/europarl/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will automatically download and extract the data-files if you don't have them already.\n",
    "\n",
    "**WARNING: The file for the English-Danish data-set is about 587 MB!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has apparently already been downloaded and unpacked.\n"
     ]
    }
   ],
   "source": [
    "europarl.maybe_download_and_extract(language_code=language_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the texts for the source-language, here we use Danish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_src = europarl.load_data(english=False,\n",
    "                              language_code=language_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the texts for the destination-language, here we use English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dest = europarl.load_data(english=True,\n",
    "                               language_code=language_code,\n",
    "                               start=mark_start,\n",
    "                               end=mark_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will build a model to translate from the source language (Danish) to the destination language (English). If you want to make the inverse translation you can merely exchange the source and destination data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example Data\n",
    "\n",
    "The data is just a list of texts that is ordered so the source and destination texts match. I can confirm that this example is an accurate translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Som De kan se, indfandt det store \"år 2000-problem\" sig ikke. Til gengæld har borgerne i en del af medlemslandene været ramt af meget forfærdelige naturkatastrofer.'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"ssss Although, as you will have seen, the dreaded 'millennium bug' failed to materialise, still the people in a number of countries suffered a series of natural disasters that truly were dreadful. eeee\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error in Data\n",
    "\n",
    "The data-set contains about 2 million sentence-pairs. Some of the data is incorrect. This example appears to be French (or some other weird language I don't understand), although the Danish text is also included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 8002"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"Car il savait ce que cette foule en joie ignorait, et qu\\'on peut lire dans les livres, que le bacille de la peste ne meurt ni ne disparaît jamais, qu\\'il peut rester pendant des dizaines d\\'années endormi dans les meubles et le linge, qu\\'il attend patiemment dans les chambres, les caves, les malles, les mouchoirs et les paperasses, et que, peut-être, le jour viendrait où, pour le malheur et l\\'enseignement des hommes, la peste réveillerait ses rats et les enverrait mourir dans une cité heureuse.\" (Thi han vidste det, som denne glade forsamling ikke vidste, og som man kan læse i bøger, at pestens bacille aldrig dør og aldrig forsvinder, at den kan sove i mange år i møbler og linned, at den venter tålmodigt i kamre, kældre, kufferter, lommetørklæder og papirer, og at den dag måske kommer, hvor pesten til menneskenes skade og oplysning vågner sine rotter og sender dem ud for at dø i en lykkelig by.)'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss \"He knew what those jubilant crowds did not know but could have learned from books: that the plague bacillus never dies or disappears for good; that it can lie dormant for years and years in furniture and linen-chests; that it bides its time in bedrooms, cellars, trunks, and bookshelves; and that perhaps the day would come when, for the bane and the enlightening of men, it would rouse up its rats again and send them forth to die in a happy city.\" eeee'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer\n",
    "\n",
    "Neural Networks cannot work directly on text-data. We use a two-step process to convert text into numbers that can be used in a neural network. The first step is to convert text-words into so-called integer-tokens. The second step is to convert integer-tokens into vectors of floating-point numbers using a so-called embedding-layer. See Tutorial #20 for a more detailed explanation.\n",
    "\n",
    "Set the maximum number of words in our vocabulary. This means that we will only use e.g. the 10000 most frequent words in the data-set. We use the same number for both the source and destination languages, but these could be different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a few more functions than provided by Keras' Tokenizer-class so we wrap it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TokenizerWrap(Tokenizer):\n",
    "    \"\"\"Wrap the Tokenizer-class from Keras with more functionality.\"\"\"\n",
    "    \n",
    "    def __init__(self, texts, padding,\n",
    "                 reverse=False, num_words=None):\n",
    "        \"\"\"\n",
    "        :param texts: List of strings. This is the data-set.\n",
    "        :param padding: Either 'post' or 'pre' padding.\n",
    "        :param reverse: Boolean whether to reverse token-lists.\n",
    "        :param num_words: Max number of words to use.\n",
    "        \"\"\"\n",
    "\n",
    "        Tokenizer.__init__(self, num_words=num_words)\n",
    "\n",
    "        # Create the vocabulary from the texts.\n",
    "        self.fit_on_texts(texts)\n",
    "\n",
    "        # Create inverse lookup from integer-tokens to words.\n",
    "        self.index_to_word = dict(zip(self.word_index.values(),\n",
    "                                      self.word_index.keys()))\n",
    "\n",
    "        # Convert all texts to lists of integer-tokens.\n",
    "        # Note that the sequences may have different lengths.\n",
    "        self.tokens = self.texts_to_sequences(texts)\n",
    "\n",
    "        if reverse:\n",
    "            # Reverse the token-sequences.\n",
    "            self.tokens = [list(reversed(x)) for x in self.tokens]\n",
    "        \n",
    "            # Sequences that are too long should now be truncated\n",
    "            # at the beginning, which corresponds to the end of\n",
    "            # the original sequences.\n",
    "            truncating = 'pre'\n",
    "        else:\n",
    "            # Sequences that are too long should be truncated\n",
    "            # at the end.\n",
    "            truncating = 'post'\n",
    "\n",
    "        # The number of integer-tokens in each sequence.\n",
    "        self.num_tokens = [len(x) for x in self.tokens]\n",
    "\n",
    "        # Max number of tokens to use in all sequences.\n",
    "        # We will pad / truncate all sequences to this length.\n",
    "        # This is a compromise so we save a lot of memory and\n",
    "        # only have to truncate maybe 5% of all the sequences.\n",
    "        self.max_tokens = np.mean(self.num_tokens) \\\n",
    "                          + 2 * np.std(self.num_tokens)\n",
    "        self.max_tokens = int(self.max_tokens)\n",
    "\n",
    "        # Pad / truncate all token-sequences to the given length.\n",
    "        # This creates a 2-dim numpy matrix that is easier to use.\n",
    "        self.tokens_padded = pad_sequences(self.tokens,\n",
    "                                           maxlen=self.max_tokens,\n",
    "                                           padding=padding,\n",
    "                                           truncating=truncating)\n",
    "\n",
    "    def token_to_word(self, token):\n",
    "        \"\"\"Lookup a single word from an integer-token.\"\"\"\n",
    "\n",
    "        word = \" \" if token == 0 else self.index_to_word[token]\n",
    "        return word \n",
    "\n",
    "    def tokens_to_string(self, tokens):\n",
    "        \"\"\"Convert a list of integer-tokens to a string.\"\"\"\n",
    "\n",
    "        # Create a list of the individual words.\n",
    "        words = [self.index_to_word[token]\n",
    "                 for token in tokens\n",
    "                 if token != 0]\n",
    "        \n",
    "        # Concatenate the words to a single string\n",
    "        # with space between all the words.\n",
    "        text = \" \".join(words)\n",
    "\n",
    "        return text\n",
    "    \n",
    "    def text_to_tokens(self, text, reverse=False, padding=False):\n",
    "        \"\"\"\n",
    "        Convert a single text-string to tokens with optional\n",
    "        reversal and padding.\n",
    "        \"\"\"\n",
    "\n",
    "        # Convert to tokens. Note that we assume there is only\n",
    "        # a single text-string so we wrap it in a list.\n",
    "        tokens = self.texts_to_sequences([text])\n",
    "        tokens = np.array(tokens)\n",
    "\n",
    "        if reverse:\n",
    "            # Reverse the tokens.\n",
    "            tokens = np.flip(tokens, axis=1)\n",
    "\n",
    "            # Sequences that are too long should now be truncated\n",
    "            # at the beginning, which corresponds to the end of\n",
    "            # the original sequences.\n",
    "            truncating = 'pre'\n",
    "        else:\n",
    "            # Sequences that are too long should be truncated\n",
    "            # at the end.\n",
    "            truncating = 'post'\n",
    "\n",
    "        if padding:\n",
    "            # Pad and truncate sequences to the given length.\n",
    "            tokens = pad_sequences(tokens,\n",
    "                                   maxlen=self.max_tokens,\n",
    "                                   padding='pre',\n",
    "                                   truncating=truncating)\n",
    "\n",
    "        return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a tokenizer for the source-language. Note that we pad zeros at the beginning ('pre') of the sequences. We also reverse the sequences of tokens because the research literature suggests that this might improve performance, because the last words seen by the encoder match the first words produced by the decoder, so short-term dependencies are supposedly modelled more accurately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 18s, sys: 940 ms, total: 2min 19s\n",
      "Wall time: 2min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer_src = TokenizerWrap(texts=data_src,\n",
    "                              padding='pre',\n",
    "                              reverse=True,\n",
    "                              num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create the tokenizer for the destination language. We need a tokenizer for both the source- and destination-languages because their vocabularies are different. Note that this tokenizer does not reverse the sequences and it pads zeros at the end ('post') of the arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 49s, sys: 752 ms, total: 1min 50s\n",
      "Wall time: 1min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer_dest = TokenizerWrap(texts=data_dest,\n",
    "                               padding='post',\n",
    "                               reverse=False,\n",
    "                               num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define convenience variables for the padded token sequences. These are just 2-dimensional numpy arrays of integer-tokens.\n",
    "\n",
    "Note that the sequence-lengths are different for the source and destination languages. This is because texts with the same meaning may have different numbers of words in the two languages. \n",
    "\n",
    "Furthermore, we have made a compromise when tokenizing the original texts in order to save a lot of memory. This means we only truncate about 5% of the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1968800, 47)\n",
      "(1968800, 55)\n"
     ]
    }
   ],
   "source": [
    "tokens_src = tokenizer_src.tokens_padded\n",
    "tokens_dest = tokenizer_dest.tokens_padded\n",
    "print(tokens_src.shape)\n",
    "print(tokens_dest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the integer-token used to mark the beginning of a text in the destination-language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_start = tokenizer_dest.word_index[mark_start.strip()]\n",
    "token_start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the integer-token used to mark the end of a text in the destination-language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_end = tokenizer_dest.word_index[mark_end.strip()]\n",
    "token_end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example of Token Sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the output of the tokenizer. Note how it is padded with zeros at the beginning (pre-padding)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0, 3069,\n",
       "       3374,   43,    7, 1386,  108, 1995,    7,  178,    9,    3,  302,\n",
       "         19, 2076,    8,   20,   39,  285,  499,   69,  136,    5,  166,\n",
       "         24,   10,   13], dtype=int32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_src[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can reconstruct the original text by converting each integer-token back to its corresponding word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'naturkatastrofer forfærdelige meget af ramt været medlemslandene af del en i borgerne har gengæld til ikke sig problem 2000 år store det se kan de som'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_src.tokens_to_string(tokens_src[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This text is actually reversed, as can be seen when compared to the original text from the data-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Som De kan se, indfandt det store \"år 2000-problem\" sig ikke. Til gengæld har borgerne i en del af medlemslandene været ramt af meget forfærdelige naturkatastrofer.'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the sequence of integer-tokens for the corresponding text in the destination-language. Note how it is padded with zeros at the end (post-padding)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,  404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,\n",
       "        174,    1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,\n",
       "          4,  892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can reconstruct the original text by converting each integer-token back to its corresponding word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss although as you will have seen the failed to materialise still the people in a number of countries suffered a series of natural disasters that truly were dreadful eeee'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(tokens_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare this to the original text from the data-set, which is almost identical except for punctuation marks and a few words such as \"dreaded millennium bug\". This is because we only use a vocabulary of the 10000 most frequent words in the data-set and those 3 words were apparently not used frequently enough to be included in the vocabulary, so they are merely skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"ssss Although, as you will have seen, the dreaded 'millennium bug' failed to materialise, still the people in a number of countries suffered a series of natural disasters that truly were dreadful. eeee\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Data\n",
    "\n",
    "Now that the data-set has been converted to sequences of integer-tokens that are padded and truncated and saved in numpy arrays, we can easily prepare the data for use in training the neural network.\n",
    "\n",
    "The input to the encoder is merely the numpy array for the padded and truncated sequences of integer-tokens produced by the tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input_data = tokens_src"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input and output data for the decoder is identical, except shifted one time-step. We can use the same numpy array to save memory by slicing it, which merely creates different 'views' of the same data in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1968800, 54)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_input_data = tokens_dest[:, :-1]\n",
    "decoder_input_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1968800, 54)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_output_data = tokens_dest[:, 1:]\n",
    "decoder_output_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, these token-sequences are identical except they are shifted one time-step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,  404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,\n",
       "        174,    1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,\n",
       "          4,  892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_input_data[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,  174,\n",
       "          1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,    4,\n",
       "        892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_output_data[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we use the tokenizer to convert these sequences back into text, we see that they are identical except for the first word which is 'ssss' that marks the beginning of a text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss although as you will have seen the failed to materialise still the people in a number of countries suffered a series of natural disasters that truly were dreadful eeee'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(decoder_input_data[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'although as you will have seen the failed to materialise still the people in a number of countries suffered a series of natural disasters that truly were dreadful eeee'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(decoder_output_data[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Neural Network\n",
    "\n",
    "### Create the Encoder\n",
    "\n",
    "First we create the encoder-part of the neural network which maps a sequence of integer-tokens to a \"thought vector\". We will use the so-called functional API of Keras for this, where we first create the objects for all the layers of the neural network and then we connect them later, this allows for more flexibility than the so-called sequential API in Keras, which is useful when experimenting with more complicated architectures and ways of connecting the encoder and decoder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the input for the encoder which takes batches of integer-token sequences. The `None` indicates that the sequences can have arbitrary length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input = Input(shape=(None, ), name='encoder_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the length of the vectors output by the embedding-layer, which maps integer-tokens to vectors of values roughly between -1 and 1, so that words that have similar semantic meanings are mapped to vectors that are similar. See Tutorial #20 for a more detailed explanation of this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the embedding-layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_embedding = Embedding(input_dim=num_words,\n",
    "                              output_dim=embedding_size,\n",
    "                              name='encoder_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the size of the internal states of the Gated Recurrent Units (GRU). The same size is used in both the encoder and decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "state_size = 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates the 3 GRU layers that will map from a sequence of embedding-vectors to a single \"thought vector\" which summarizes the contents of the input-text. Note that the last GRU-layer does not return a sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_gru1 = GRU(state_size, name='encoder_gru1',\n",
    "                   return_sequences=True)\n",
    "encoder_gru2 = GRU(state_size, name='encoder_gru2',\n",
    "                   return_sequences=True)\n",
    "encoder_gru3 = GRU(state_size, name='encoder_gru3',\n",
    "                   return_sequences=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This helper-function connects all the layers of the encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def connect_encoder():\n",
    "    # Start the neural network with its input-layer.\n",
    "    net = encoder_input\n",
    "    \n",
    "    # Connect the embedding-layer.\n",
    "    net = encoder_embedding(net)\n",
    "\n",
    "    # Connect all the GRU-layers.\n",
    "    net = encoder_gru1(net)\n",
    "    net = encoder_gru2(net)\n",
    "    net = encoder_gru3(net)\n",
    "\n",
    "    # This is the output of the encoder.\n",
    "    encoder_output = net\n",
    "    \n",
    "    return encoder_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the encoder uses the normal output from its last GRU-layer as the \"thought vector\". Research papers often use the internal state of the encoder's last recurrent layer as the \"thought vector\". But this makes the implementation more complicated and is not necessary when using the GRU. But if you were using the LSTM instead then it is necessary to use the LSTM's internal states as the \"thought vector\" because it actually has two internal vectors, which we would need to initialize the two internal states of the decoder's LSTM units.\n",
    "\n",
    "We can now use this function to connect all the layers in the encoder so it can be connected to the decoder further below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_output = connect_encoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Decoder\n",
    "\n",
    "Create the decoder-part which maps the \"thought vector\" to a sequence of integer-tokens.\n",
    "\n",
    "The decoder takes two inputs. First it needs the \"thought vector\" produced by the encoder which summarizes the contents of the input-text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_initial_state = Input(shape=(state_size,),\n",
    "                              name='decoder_initial_state')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decoder also needs a sequence of integer-tokens as inputs. During training we will supply this with a full sequence of integer-tokens e.g. corresponding to the text \"ssss once upon a time eeee\". \n",
    "\n",
    "During inference when we are translating new input-texts, we will start by feeding a sequence with just one integer-token for \"ssss\" which marks the beginning of a text, and combined with the \"thought vector\" from the encoder, the decoder will hopefully be able to produce the correct next word e.g. \"once\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_input = Input(shape=(None, ), name='decoder_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the embedding-layer which converts integer-tokens to vectors of real-valued numbers roughly between -1 and 1. Note that we have different embedding-layers for the encoder and decoder because we have two different vocabularies and two different tokenizers for the source and destination languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_embedding = Embedding(input_dim=num_words,\n",
    "                              output_dim=embedding_size,\n",
    "                              name='decoder_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates the 3 GRU layers of the decoder. Note that they all return sequences because we ultimately want to output a sequence of integer-tokens that can be converted into a text-sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_gru1 = GRU(state_size, name='decoder_gru1',\n",
    "                   return_sequences=True)\n",
    "decoder_gru2 = GRU(state_size, name='decoder_gru2',\n",
    "                   return_sequences=True)\n",
    "decoder_gru3 = GRU(state_size, name='decoder_gru3',\n",
    "                   return_sequences=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The GRU layers output a tensor with shape `[batch_size, sequence_length, state_size]`, where each \"word\" is encoded as a vector of length `state_size`. We need to convert this into sequences of integer-tokens that can be interpreted as words from our vocabulary.\n",
    "\n",
    "One way of doing this is to convert the GRU output to a one-hot encoded array. It works but it is extremely wasteful, because for a vocabulary of e.g. 10000 words we need a vector with 10000 elements, so we can select the index of the highest element to be the integer-token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_dense = Dense(num_words,\n",
    "                      activation='softmax',\n",
    "                      name='decoder_output')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decoder is built using the functional API of Keras, which allows more flexibility in connecting the layers e.g. to route different inputs to the decoder. This is useful because we have to connect the decoder directly to the encoder, but we will also connect the decoder to another input so we can run it separately.\n",
    "\n",
    "This function connects all the layers of the decoder to some input of the initial-state values for the GRU layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def connect_decoder(initial_state):\n",
    "    # Start the decoder-network with its input-layer.\n",
    "    net = decoder_input\n",
    "\n",
    "    # Connect the embedding-layer.\n",
    "    net = decoder_embedding(net)\n",
    "    \n",
    "    # Connect all the GRU-layers.\n",
    "    net = decoder_gru1(net, initial_state=initial_state)\n",
    "    net = decoder_gru2(net, initial_state=initial_state)\n",
    "    net = decoder_gru3(net, initial_state=initial_state)\n",
    "\n",
    "    # Connect the final dense layer that converts to\n",
    "    # one-hot encoded arrays.\n",
    "    decoder_output = decoder_dense(net)\n",
    "    \n",
    "    return decoder_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect and Create the Models\n",
    "\n",
    "We can now connect the encoder and decoder in different ways.\n",
    "\n",
    "First we connect the encoder directly to the decoder so it is one whole model that can be trained end-to-end. This means the initial-state of the decoder's GRU units are set to the output of the encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_output = connect_decoder(initial_state=encoder_output)\n",
    "\n",
    "model_train = Model(inputs=[encoder_input, decoder_input],\n",
    "                    outputs=[decoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a model for just the encoder alone. This is useful for mapping a sequence of integer-tokens to a \"thought-vector\" summarizing its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_encoder = Model(inputs=[encoder_input],\n",
    "                      outputs=[encoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a model for just the decoder alone. This allows us to directly input the initial state for the decoder's GRU units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_output = connect_decoder(initial_state=decoder_initial_state)\n",
    "\n",
    "model_decoder = Model(inputs=[decoder_input, decoder_initial_state],\n",
    "                      outputs=[decoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that all these models use the same weights and variables of the encoder and decoder. We are merely changing how they are connected. So once the entire model has been trained, we can run the encoder and decoder models separately with the trained weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile the Model\n",
    "\n",
    "The output of the decoder is a sequence of one-hot encoded arrays. In order to train the decoder we need to supply the one-hot encoded arrays that we desire to see on the decoder's output, and then use a loss-function like cross-entropy to train the decoder to produce this desired output.\n",
    "\n",
    "However, our data-set contains integer-tokens instead of one-hot encoded arrays. Each one-hot encoded array has 10000 elements so it would be extremely wasteful to convert the entire data-set to one-hot encoded arrays.\n",
    "\n",
    "A better way is to use a so-called sparse cross-entropy loss-function, which does the conversion internally from integers to one-hot encoded arrays.\n",
    "\n",
    "We have used the Adam optimizer in many of the previous tutorials, but it seems to diverge in some of these experiments with Recurrent Neural Networks. RMSprop seems to work much better for these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_train.compile(optimizer=RMSprop(lr=1e-3),\n",
    "                    loss='sparse_categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback Functions\n",
    "\n",
    "During training we want to save checkpoints and log the progress to TensorBoard so we create the appropriate callbacks for Keras.\n",
    "\n",
    "This is the callback for writing checkpoints during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_checkpoint = '21_checkpoint.keras'\n",
    "callback_checkpoint = ModelCheckpoint(filepath=path_checkpoint,\n",
    "                                      monitor='val_loss',\n",
    "                                      verbose=1,\n",
    "                                      save_weights_only=True,\n",
    "                                      save_best_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the callback for stopping the optimization when performance worsens on the validation-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "callback_early_stopping = EarlyStopping(monitor='val_loss',\n",
    "                                        patience=3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the callback for writing the TensorBoard log during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "callback_tensorboard = TensorBoard(log_dir='./21_logs/',\n",
    "                                   histogram_freq=0,\n",
    "                                   write_graph=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [callback_early_stopping,\n",
    "             callback_checkpoint,\n",
    "             callback_tensorboard]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Checkpoint\n",
    "\n",
    "You can reload the last saved checkpoint so you don't have to train the model every time you want to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    model_train.load_weights(path_checkpoint)\n",
    "except Exception as error:\n",
    "    print(\"Error trying to load checkpoint.\")\n",
    "    print(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the Model\n",
    "\n",
    "We wrap the data in named dicts so we are sure the data is assigned correctly to the inputs and outputs of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_data = \\\n",
    "{\n",
    "    'encoder_input': encoder_input_data,\n",
    "    'decoder_input': decoder_input_data\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_data = \\\n",
    "{\n",
    "    'decoder_output': decoder_output_data\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want a validation-set of 10000 sequences but Keras needs this number as a fraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0050792360828931325"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_split = 10000 / len(encoder_input_data)\n",
    "validation_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the model. One epoch of training took about 1 hour on a GTX 1070 GPU. You probably need to run 10 epochs or more during training. After 10 epochs the loss was about 1.10 on the training-set and about 1.15 on the validation-set.\n",
    "\n",
    "The batch-size was chosen to keep the GPU running at nearly 100% while being within the memory limits of 8GB for this GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_train.fit(x=x_data,\n",
    "                y=y_data,\n",
    "                batch_size=384,\n",
    "                epochs=10,\n",
    "                validation_split=validation_split,\n",
    "                callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate Texts\n",
    "\n",
    "This function translates a text from the source-language to the destination-language and optionally prints a true translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate(input_text, true_output_text=None):\n",
    "    \"\"\"Translate a single text-string.\"\"\"\n",
    "\n",
    "    # Convert the input-text to integer-tokens.\n",
    "    # Note the sequence of tokens has to be reversed.\n",
    "    # Padding is probably not necessary.\n",
    "    input_tokens = tokenizer_src.text_to_tokens(text=input_text,\n",
    "                                                reverse=True,\n",
    "                                                padding=True)\n",
    "    \n",
    "    # Get the output of the encoder's GRU which will be\n",
    "    # used as the initial state in the decoder's GRU.\n",
    "    # This could also have been the encoder's final state\n",
    "    # but that is really only necessary if the encoder\n",
    "    # and decoder use the LSTM instead of GRU because\n",
    "    # the LSTM has two internal states.\n",
    "    initial_state = model_encoder.predict(input_tokens)\n",
    "\n",
    "    # Max number of tokens / words in the output sequence.\n",
    "    max_tokens = tokenizer_dest.max_tokens\n",
    "\n",
    "    # Pre-allocate the 2-dim array used as input to the decoder.\n",
    "    # This holds just a single sequence of integer-tokens,\n",
    "    # but the decoder-model expects a batch of sequences.\n",
    "    shape = (1, max_tokens)\n",
    "    decoder_input_data = np.zeros(shape=shape, dtype=np.int)\n",
    "\n",
    "    # The first input-token is the special start-token for 'ssss '.\n",
    "    token_int = token_start\n",
    "\n",
    "    # Initialize an empty output-text.\n",
    "    output_text = ''\n",
    "\n",
    "    # Initialize the number of tokens we have processed.\n",
    "    count_tokens = 0\n",
    "\n",
    "    # While we haven't sampled the special end-token for ' eeee'\n",
    "    # and we haven't processed the max number of tokens.\n",
    "    while token_int != token_end and count_tokens < max_tokens:\n",
    "        # Update the input-sequence to the decoder\n",
    "        # with the last token that was sampled.\n",
    "        # In the first iteration this will set the\n",
    "        # first element to the start-token.\n",
    "        decoder_input_data[0, count_tokens] = token_int\n",
    "\n",
    "        # Wrap the input-data in a dict for clarity and safety,\n",
    "        # so we are sure we input the data in the right order.\n",
    "        x_data = \\\n",
    "        {\n",
    "            'decoder_initial_state': initial_state,\n",
    "            'decoder_input': decoder_input_data\n",
    "        }\n",
    "\n",
    "        # Note that we input the entire sequence of tokens\n",
    "        # to the decoder. This wastes a lot of computation\n",
    "        # because we are only interested in the last input\n",
    "        # and output. We could modify the code to return\n",
    "        # the GRU-states when calling predict() and then\n",
    "        # feeding these GRU-states as well the next time\n",
    "        # we call predict(), but it would make the code\n",
    "        # much more complicated.\n",
    "\n",
    "        # Input this data to the decoder and get the predicted output.\n",
    "        decoder_output = model_decoder.predict(x_data)\n",
    "\n",
    "        # Get the last predicted token as a one-hot encoded array.\n",
    "        token_onehot = decoder_output[0, count_tokens, :]\n",
    "        \n",
    "        # Convert to an integer-token.\n",
    "        token_int = np.argmax(token_onehot)\n",
    "\n",
    "        # Lookup the word corresponding to this integer-token.\n",
    "        sampled_word = tokenizer_dest.token_to_word(token_int)\n",
    "\n",
    "        # Append the word to the output-text.\n",
    "        output_text += \" \" + sampled_word\n",
    "\n",
    "        # Increment the token-counter.\n",
    "        count_tokens += 1\n",
    "\n",
    "    # Sequence of tokens output by the decoder.\n",
    "    output_tokens = decoder_input_data[0]\n",
    "    \n",
    "    # Print the input-text.\n",
    "    print(\"Input text:\")\n",
    "    print(input_text)\n",
    "    print()\n",
    "\n",
    "    # Print the translated output-text.\n",
    "    print(\"Translated text:\")\n",
    "    print(output_text)\n",
    "    print()\n",
    "\n",
    "    # Optionally print the true translated text.\n",
    "    if true_output_text is not None:\n",
    "        print(\"True output text:\")\n",
    "        print(true_output_text)\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples\n",
    "\n",
    "Translate a text from the training-data. This translation is quite good. Note how it is not identical to the translation from the training-data, but the actual meaning is similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.\n",
      "\n",
      "Translated text:\n",
      " you have expressed a desire to speak on this subject during the debate eeee\n",
      "\n",
      "True output text:\n",
      "ssss You have requested a debate on this subject in the course of the next few days, during this part-session. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx],\n",
    "          true_output_text=data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is another example which is also a reasonable translation, although it has incorrectly translated the natural disasters. Note \"countries of the European Union\" has instead been translated as \"member states\" which are synonyms in this context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.\n",
      "\n",
      "Translated text:\n",
      " in the meantime i have a part of the house who have also asked for a member of the victims of the terrible victims of the tragedy of which we were affected eeee\n",
      "\n",
      "True output text:\n",
      "ssss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 4\n",
    "translate(input_text=data_src[idx],\n",
    "          true_output_text=data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we join two texts from the training-set. The model first sends this combined text through the encoder, which produces a \"thought-vector\" that seems to summarize both texts reasonably well so the decoder can produce a reasonable translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.\n",
      "\n",
      "Translated text:\n",
      " you have a part of this debate in which i have had a request to speak during the debate in the portuguese presidency as a member of the victims of the tragedy of which we have been victims of this morning eeee\n",
      "\n",
      "True output text:\n",
      "ssss You have requested a debate on this subject in the course of the next few days, during this part-session. eeeessss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx] + data_src[idx+1],\n",
    "          true_output_text=data_dest[idx] + data_dest[idx+1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we reverse the order of these two texts then the meaning is not quite so clear for the latter text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.\n",
      "\n",
      "Translated text:\n",
      " in the meantime i have also asked a member of the members who have asked for a debate on the victims of this type of attack and that you have expressed a part of this debate as part of the spanish presidency eeee\n",
      "\n",
      "True output text:\n",
      "ssss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeeessss You have requested a debate on this subject in the course of the next few days, during this part-session. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx+1] + data_src[idx],\n",
    "          true_output_text=data_dest[idx+1] + data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example I made up. It is a quite broken translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "der var engang et land der hed Danmark\n",
      "\n",
      "Translated text:\n",
      " there were a member of the european commission eeee\n",
      "\n",
      "True output text:\n",
      "Once there was a country named Denmark\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"der var engang et land der hed Danmark\",\n",
    "          true_output_text='Once there was a country named Denmark')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is another example I made up. This is a better translation even though it is perhaps a more complicated text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Idag kan man læse i avisen at Danmark er blevet fornuftigt\n",
      "\n",
      "Translated text:\n",
      " read you read that the netherlands has been sensible eeee\n",
      "\n",
      "True output text:\n",
      "Today you can read in the newspaper that Denmark has become sensible.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"Idag kan man læse i avisen at Danmark er blevet fornuftigt\",\n",
    "          true_output_text=\"Today you can read in the newspaper that Denmark has become sensible.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a text from a Danish song. It doesn't even make much sense in Danish. However the translation is probably so broken because several of the words are not in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Hvem spæner ud af en butik og tygger de stærkeste bolcher?\n",
      "\n",
      "Translated text:\n",
      " who is a of the and the eeee\n",
      "\n",
      "True output text:\n",
      "Who runs out of a shop and chews the strongest bon-bons?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"Hvem spæner ud af en butik og tygger de stærkeste bolcher?\",\n",
    "          true_output_text=\"Who runs out of a shop and chews the strongest bon-bons?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This tutorial showed the basic idea of using two Recurrent Neural Networks in a so-called encoder/decoder model to do Machine Translation of human languages. It was demonstrated on the very large Europarl data-set from the European Union.\n",
    "\n",
    "The model could produce reasonable translations for some texts but not for others. It is possible that a better architecture for the neural network and more training epochs could improve performance. There are also more advanced models that are known to improve quality of the translations.\n",
    "\n",
    "However, it is important to note that these models do not really understand human language. The models have no knowledge of the actual meaning of the words. The models are merely very advanced function approximators that can map between sequences of integer-tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.\n",
    "\n",
    "You may want to backup this Notebook before making any changes.\n",
    "\n",
    "* Train for more than 10 epochs. Does it improve the translations?\n",
    "* Increase the size of the vocabulary. Does it improve the translations? Would it make sense to have different sizes for the vocabularies of the source and destination languages?\n",
    "* Find another data-set and use it together with Europarl.\n",
    "* Change the architectures of the neural network, for example change the state-size for the GRU layers, the number of GRU layers, the embedding-size, etc. Does it improve the translations?\n",
    "* Use hyper-parameter optimization from Tutorial #19 to automatically find the best hyper-parameters.\n",
    "* When translating texts, instead of using `np.argmax()` to sample the next integer-token, could you sample the decoder's output as if it was a probability distribution instead? Note that the decoder's output is not softmax-limited so you have to do that first to turn it into a probability-distribution.\n",
    "* Can you generate multiple sequences by doing this sampling? Can you find a way to select the best of these different sequences?\n",
    "* Disable the reversal of words for the source-language. Does it improve the translations?\n",
    "* What is a Bi-Directional GRU and can you use it here?\n",
    "* We use the **output** of the encoder's GRU as the initial state of the decoder's GRU. The research literature often uses an LSTM instead of the GRU, so they used the encoder's **state** instead of its output as the initial state of the decoder. Can you rewrite this code to use the encoder's state as the decoder's initial state? Is there a reason to do this, or is the encoder's output sufficient to use as the decoder's initial state?\n",
    "* Is it possible to connect multiple encoders and decoders in a single neural network, so that you can train it on different languages and allow for direct translation e.g. from Danish to Polish, German and French?\n",
    "* Explain to a friend how the program works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}