ARASAAC_fine_tuned_PictoBERT.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZWF35G5JVRhg"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github//jayralencar/pictoBERT/blob/main/ARASAAC_fine_tuned_PictoBERT.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/jayralencar/pictoBERT/blob/main/ARASAAC_fine_tuned_PictoBERT.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PaE6kkKYgAl"
      },
      "source": [
        "# PictoBERT: Transformers for Pictogram Prediction (ARASAAC fine-tuning)\n",
        "\n",
        "This notebook presents the procedures for fine-tuning PictoBERT to make predictions based on the vocabulary from [ARASAAC](https://arasaac.org/) portal. This notebook refers to section 5.2.2 of the paper."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qyJDrqTKneOs"
      },
      "source": [
        "## Verify if you are using a GPU\n",
        "\n",
        "For fine-tuning, we suggest using a GPU, which can allow you to train fast."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EMI_8Z03ne1I",
        "outputId": "d8274fcf-d144-4d22-f626-01d3eda4eae2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Thu Mar 24 20:36:26 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   37C    P8    28W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "gpu_info = !nvidia-smi\n",
        "gpu_info = '\\n'.join(gpu_info)\n",
        "if gpu_info.find('failed') >= 0:\n",
        "  print('Select the Runtime > \"Change runtime type\" menu to enable a GPU accelerator, ')\n",
        "  print('and then re-execute this cell.')\n",
        "else:\n",
        "  print(gpu_info)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hkLg9oRZC6z"
      },
      "source": [
        "## Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FzS-yDjgfcMe",
        "outputId": "729c7eb2-2f6b-48d0-b177-c9e298a16d5d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "  Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 5.2 MB/s \n",
            "\u001b[?25hCollecting tokenizers\n",
            "  Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.5 MB 26.8 MB/s \n",
            "\u001b[?25hCollecting pytorch_lightning==1.2.10\n",
            "  Downloading pytorch_lightning-1.2.10-py3-none-any.whl (841 kB)\n",
            "\u001b[K     |████████████████████████████████| 841 kB 24.6 MB/s \n",
            "\u001b[?25hCollecting future>=0.17.1\n",
            "  Downloading future-0.18.2.tar.gz (829 kB)\n",
            "\u001b[K     |████████████████████████████████| 829 kB 44.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (1.21.5)\n",
            "Requirement already satisfied: tqdm>=4.41.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (4.63.0)\n",
            "Collecting torchmetrics==0.2.0\n",
            "  Downloading torchmetrics-0.2.0-py3-none-any.whl (176 kB)\n",
            "\u001b[K     |████████████████████████████████| 176 kB 43.9 MB/s \n",
            "\u001b[?25hCollecting PyYAML!=5.4.*,>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 39.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (21.3)\n",
            "Collecting fsspec[http]>=0.8.1\n",
            "  Downloading fsspec-2022.2.0-py3-none-any.whl (134 kB)\n",
            "\u001b[K     |████████████████████████████████| 134 kB 43.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch>=1.4 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (1.10.0+cu111)\n",
            "Requirement already satisfied: tensorboard!=2.5.0,>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (2.8.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.23.0)\n",
            "Collecting aiohttp\n",
            "  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 30.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.4.6)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.17.3)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.37.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.3.6)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.35.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.6.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.0.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.8.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (57.4.0)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.44.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.0.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.15.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.8)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.2.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.11.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.7.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.10.0.2)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.4.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (3.0.4)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.2.0)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
            "\u001b[K     |████████████████████████████████| 67 kB 2.6 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 40.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->pytorch_lightning==1.2.10) (3.0.7)\n",
            "Collecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)\n",
            "Collecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)\n",
            "\u001b[K     |████████████████████████████████| 144 kB 24.1 MB/s \n",
            "\u001b[?25hCollecting asynctest==0.13.0\n",
            "  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
            "Collecting yarl<2.0,>=1.0\n",
            "  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
            "\u001b[K     |████████████████████████████████| 271 kB 42.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (21.4.0)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.0.12)\n",
            "Collecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Collecting multidict<7.0,>=4.5\n",
            "  Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)\n",
            "\u001b[K     |████████████████████████████████| 94 kB 2.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Building wheels for collected packages: future\n",
            "  Building wheel for future (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=fee5465ccb5cffa49e9b0ff1c91df1e698336ece416528b4f8cd1054365a634a\n",
            "  Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0\n",
            "Successfully built future\n",
            "Installing collected packages: multidict, frozenlist, yarl, asynctest, async-timeout, aiosignal, PyYAML, fsspec, aiohttp, torchmetrics, tokenizers, sacremoses, huggingface-hub, future, transformers, pytorch-lightning\n",
            "  Attempting uninstall: PyYAML\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Attempting uninstall: future\n",
            "    Found existing installation: future 0.16.0\n",
            "    Uninstalling future-0.16.0:\n",
            "      Successfully uninstalled future-0.16.0\n",
            "Successfully installed PyYAML-6.0 aiohttp-3.8.1 aiosignal-1.2.0 async-timeout-4.0.2 asynctest-0.13.0 frozenlist-1.3.0 fsspec-2022.2.0 future-0.18.2 huggingface-hub-0.4.0 multidict-6.0.2 pytorch-lightning-1.2.10 sacremoses-0.0.49 tokenizers-0.11.6 torchmetrics-0.2.0 transformers-4.17.0 yarl-1.7.2\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers tokenizers pytorch_lightning==1.2.10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9QTVE2oRZA2f"
      },
      "source": [
        "## Download data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QerGMEqGZGac"
      },
      "source": [
        "### Download ARASAAC all pictograms\n",
        "\n",
        "You can download ARASAAC all pictograms by using the API provided by the portal: https://arasaac.org/developers/api\n",
        "\n",
        "Or a dump from fev 2, 2022: http://jayr.clubedosgeeks.com.br/pictobert/ARASAAC_All_pictograms.json "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ys0Fa9VmYSQA",
        "outputId": "955d63b7-8462-48b0-d83c-d2cabf43c3b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:36:59--  http://jayr.clubedosgeeks.com.br/pictobert/ARASAAC_All_pictograms.json\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 9402637 (9.0M) [application/json]\n",
            "Saving to: ‘ARASAAC_All_pictograms.json’\n",
            "\n",
            "ARASAAC_All_pictogr 100%[===================>]   8.97M  25.9MB/s    in 0.3s    \n",
            "\n",
            "2022-03-24 20:36:59 (25.9 MB/s) - ‘ARASAAC_All_pictograms.json’ saved [9402637/9402637]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/ARASAAC_All_pictograms.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppYidHVgdfuw"
      },
      "source": [
        "### Download pictogram to sense mappings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "40ceDrutdijq",
        "outputId": "7ba5069d-8d5b-44d1-82ec-488aaf567219"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:36:59--  http://jayr.clubedosgeeks.com.br/pictobert/arasaac_mapping.csv\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 489601 (478K) [text/csv]\n",
            "Saving to: ‘arasaac_mapping.csv’\n",
            "\n",
            "arasaac_mapping.csv 100%[===================>] 478.13K  2.54MB/s    in 0.2s    \n",
            "\n",
            "2022-03-24 20:37:00 (2.54 MB/s) - ‘arasaac_mapping.csv’ saved [489601/489601]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/arasaac_mapping.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYnW3T5SeESX"
      },
      "source": [
        "### Download SemCHILDES"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wyI6kcDteJDL",
        "outputId": "1777d303-a9ce-44fd-a1d9-86fd17d54645"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:37:00--  http://jayr.clubedosgeeks.com.br/pictobert/all_mt_2.txt\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 52489765 (50M) [text/plain]\n",
            "Saving to: ‘all_mt_2.txt’\n",
            "\n",
            "all_mt_2.txt        100%[===================>]  50.06M  39.8MB/s    in 1.3s    \n",
            "\n",
            "2022-03-24 20:37:01 (39.8 MB/s) - ‘all_mt_2.txt’ saved [52489765/52489765]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/all_mt_2.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-zAWdvKyhxHx"
      },
      "source": [
        "### Download corpus already adapted for ARASAAC"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wo8Sgi9Ch6CO",
        "outputId": "6365e23d-6b81-41a3-ca6e-d00411ab4f23"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:37:01--  http://jayr.clubedosgeeks.com.br/pictobert/corpus_arasaac.txt\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2834625 (2.7M) [text/plain]\n",
            "Saving to: ‘corpus_arasaac.txt’\n",
            "\n",
            "corpus_arasaac.txt  100%[===================>]   2.70M  10.3MB/s    in 0.3s    \n",
            "\n",
            "2022-03-24 20:37:02 (10.3 MB/s) - ‘corpus_arasaac.txt’ saved [2834625/2834625]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/corpus_arasaac.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwn5hP2HlZyV"
      },
      "source": [
        "### Download already trained tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "znGQHJLJldHF",
        "outputId": "48ccb6d7-ed36-41bc-e8cf-4fe141056bd9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:37:02--  http://jayr.clubedosgeeks.com.br/pictobert/tokenizer_arasaac.txt\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 404 Not Found\n",
            "2022-03-24 20:37:03 ERROR 404: Not Found.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/tokenizer_arasaac.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SdCFQZeFl_aG"
      },
      "source": [
        "### Download PictoBERT versions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vdvr3NR8mC1j",
        "outputId": "bddc0c14-9dff-4a67-8c30-254cd7f72eca"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:37:03--  http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-contextual.zip\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1180295214 (1.1G) [application/zip]\n",
            "Saving to: ‘pictobert-large-contextual.zip’\n",
            "\n",
            "pictobert-large-con 100%[===================>]   1.10G  73.0MB/s    in 16s     \n",
            "\n",
            "2022-03-24 20:37:19 (71.9 MB/s) - ‘pictobert-large-contextual.zip’ saved [1180295214/1180295214]\n",
            "\n",
            "--2022-03-24 20:37:19--  http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-gloss.zip\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1180231318 (1.1G) [application/zip]\n",
            "Saving to: ‘pictobert-large-gloss.zip’\n",
            "\n",
            "pictobert-large-glo 100%[===================>]   1.10G  75.8MB/s    in 15s     \n",
            "\n",
            "2022-03-24 20:37:34 (75.9 MB/s) - ‘pictobert-large-gloss.zip’ saved [1180231318/1180231318]\n",
            "\n",
            "Archive:  pictobert-large-contextual.zip\n",
            "   creating: pictobert/\n",
            "  inflating: pictobert/config.json   \n",
            "  inflating: pictobert/pytorch_model.bin  \n",
            "Archive:  pictobert-large-gloss.zip\n",
            "   creating: pictobert-gloss/\n",
            "  inflating: pictobert-gloss/config.json  \n",
            "  inflating: pictobert-gloss/pytorch_model.bin  \n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-contextual.zip\n",
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-gloss.zip\n",
        "\n",
        "!unzip pictobert-large-contextual.zip\n",
        "!unzip pictobert-large-gloss.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BA8Yc79xmVzl"
      },
      "source": [
        "### Download PictoBERT tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FNz1P-6NmZFT",
        "outputId": "1c860fd7-7040-41be-d1ec-3b20f13b78e9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:38:02--  http://jayr.clubedosgeeks.com.br/pictobert/childes_all_new.json\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 332233 (324K) [application/json]\n",
            "Saving to: ‘childes_all_new.json’\n",
            "\n",
            "childes_all_new.jso 100%[===================>] 324.45K  --.-KB/s    in 0.1s    \n",
            "\n",
            "2022-03-24 20:38:02 (2.47 MB/s) - ‘childes_all_new.json’ saved [332233/332233]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/childes_all_new.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mLcMxzbgm-yn"
      },
      "source": [
        "### Download ARES embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Iaki7JZPnAkq",
        "outputId": "ba3151cb-a47f-42b9-a7bb-6ce615c293df"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 20:38:02--  http://jayr.clubedosgeeks.com.br/pictobert/ares_1024_gloss.bin\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 852260167 (813M) [application/octet-stream]\n",
            "Saving to: ‘ares_1024_gloss.bin’\n",
            "\n",
            "ares_1024_gloss.bin 100%[===================>] 812.78M  83.4MB/s    in 9.9s    \n",
            "\n",
            "2022-03-24 20:38:12 (82.0 MB/s) - ‘ares_1024_gloss.bin’ saved [852260167/852260167]\n",
            "\n",
            "--2022-03-24 20:38:12--  http://jayr.clubedosgeeks.com.br/pictobert/ares_1024.bin\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 852260167 (813M) [application/octet-stream]\n",
            "Saving to: ‘ares_1024.bin’\n",
            "\n",
            "ares_1024.bin       100%[===================>] 812.78M  82.9MB/s    in 11s     \n",
            "\n",
            "2022-03-24 20:38:23 (76.7 MB/s) - ‘ares_1024.bin’ saved [852260167/852260167]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/ares_1024_gloss.bin\n",
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/ares_1024.bin"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b8HOSPybaJFf"
      },
      "source": [
        "## Map ARASAAC to WordNET 3.0 word-senses\n",
        "\n",
        "This process can take several minutes. You can download the already processed file from http://jayr.clubedosgeeks.com.br/pictobert/arasaac_mappings.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GaRrHeWxbBOH",
        "outputId": "32b82990-2a08-45c7-b700-43c630e3612b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download(\"wordnet\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-hiSp0zvaGX2",
        "outputId": "9e6ed203-041c-4dcb-8ce2-f7fdd7a84836"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 21%|██▏       | 2443/11406 [04:04<10:55, 13.67it/s]"
          ]
        }
      ],
      "source": [
        "import requests, json\n",
        "from tqdm import tqdm\n",
        "from nltk.corpus import wordnet as wn\n",
        "import pandas as pd\n",
        "\n",
        "def wordnet_map(wn_id):\n",
        "    try:\n",
        "        url = \"http://wordnet-rdf.princeton.edu/json/id/\"+wn_id\n",
        "        r = requests.get(url)\n",
        "        response_json = r.json()\n",
        "        if len(response_json[0]['old_keys']) > 0:\n",
        "            return response_json[0]['old_keys']['pwn30'][0]\n",
        "        else:\n",
        "            return None\n",
        "    except:\n",
        "        return None\n",
        "\n",
        "pictograms = json.load(open('ARASAAC_All_pictograms.json'))\n",
        "\n",
        "pictograms_dic = []\n",
        "\n",
        "for i,pictogram in enumerate(tqdm(pictograms)):\n",
        "    pictogram_id = pictogram['_id']\n",
        "    if \"personal pronoun\" in pictogram['categories']:\n",
        "        # download_pictogram(pictogram_id)\n",
        "        for keyword in pictogram['keywords']:\n",
        "            pictograms_dic.append({\n",
        "                \"word\": keyword['keyword'],\n",
        "                \"pictogram_id\": pictogram['_id'],\n",
        "                \"synset\": keyword['keyword']\n",
        "            })\n",
        "    for s in pictogram['synsets']:\n",
        "        synset = wordnet_map(s)\n",
        "        if synset is not None:\n",
        "          wn_ss = wn.of2ss(synset)\n",
        "          lemma = wn_ss.lemmas()[0].key()\n",
        "        else:\n",
        "          lemma = None\n",
        "        for keyword in pictogram['keywords']:\n",
        "            if lemma is None:\n",
        "              lemma = keyword['keyword']\n",
        "            pictograms_dic.append({\n",
        "                \"word\": keyword['keyword'],\n",
        "                \"pictogram_id\": pictogram['_id'],\n",
        "                \"synset\": synset,\n",
        "                \"word_senses\": lemma\n",
        "            })\n",
        "df = pd.DataFrame(pictograms_dic)\n",
        "df.to_csv(\"arasaac_mappings.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gnmS6eaOeXBH"
      },
      "source": [
        "## Adapt SemCHILDES to fit ARASAAC vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "8suuTTicebiP",
        "outputId": "a9e07573-82c1-4b75-eb86-94394eeab2c4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-c01b7eca-a713-403b-b2b6-051a1e2bfb67\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>Unnamed: 0.1</th>\n",
              "      <th>word</th>\n",
              "      <th>pictogram_id</th>\n",
              "      <th>synset</th>\n",
              "      <th>word_senses</th>\n",
              "      <th>Unnamed: 6</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>pavement</td>\n",
              "      <td>2247.0</td>\n",
              "      <td>04215402-n</td>\n",
              "      <td>pavement%1:06:01::</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>sidewalk</td>\n",
              "      <td>2247.0</td>\n",
              "      <td>04215402-n</td>\n",
              "      <td>sidewalk%1:06:00::</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>carpet</td>\n",
              "      <td>2249.0</td>\n",
              "      <td>04118021-n</td>\n",
              "      <td>carpet%1:06:00::</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>rug</td>\n",
              "      <td>2249.0</td>\n",
              "      <td>04118021-n</td>\n",
              "      <td>rug%1:06:00::</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>pillow</td>\n",
              "      <td>2250.0</td>\n",
              "      <td>03938244-n</td>\n",
              "      <td>pillow%1:06:00::</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c01b7eca-a713-403b-b2b6-051a1e2bfb67')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c01b7eca-a713-403b-b2b6-051a1e2bfb67 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c01b7eca-a713-403b-b2b6-051a1e2bfb67');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   Unnamed: 0  Unnamed: 0.1      word  pictogram_id      synset  \\\n",
              "0           0             0  pavement        2247.0  04215402-n   \n",
              "1           1             1  sidewalk        2247.0  04215402-n   \n",
              "2           4             4    carpet        2249.0  04118021-n   \n",
              "3           5             5       rug        2249.0  04118021-n   \n",
              "4           6             6    pillow        2250.0  03938244-n   \n",
              "\n",
              "          word_senses Unnamed: 6  \n",
              "0  pavement%1:06:01::        NaN  \n",
              "1  sidewalk%1:06:00::        NaN  \n",
              "2    carpet%1:06:00::        NaN  \n",
              "3       rug%1:06:00::        NaN  \n",
              "4    pillow%1:06:00::        NaN  "
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "pic_map = pd.read_csv(\"./arasaac_mapping.csv\")\n",
        "vocab = ['_'.join(w.split(\" \")) for w in list(pic_map['word_senses'])]\n",
        "pic_map.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 417
        },
        "id": "aQeRhU-UekHj",
        "outputId": "c8b07843-939e-499b-cabc-3660592e4ea7"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  6%|▌         | 59262/955489 [00:44<11:17, 1322.26it/s]\n"
          ]
        },
        {
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-daa3103ac221>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mf2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"./newcorpus.txt\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'w'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mnew_sentences\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtqdm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadlines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m   \u001b[0mwrite\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m   \u001b[0;32mfor\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0;32min\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36m__iter__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1203\u001b[0m                     \u001b[0mdt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcur_t\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlast_print_t\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1204\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0mdt\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mmininterval\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mcur_t\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mmin_start_t\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1205\u001b[0;31m                         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlast_print_n\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1206\u001b[0m                         \u001b[0mlast_print_n\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlast_print_n\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1207\u001b[0m                         \u001b[0mlast_print_t\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlast_print_t\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self, n)\u001b[0m\n\u001b[1;32m   1254\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ema_dn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1255\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ema_dt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1256\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrefresh\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlock_args\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlock_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1257\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdynamic_miniters\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1258\u001b[0m                     \u001b[0;31m# If no `miniters` was specified, adjust automatically to the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36mrefresh\u001b[0;34m(self, nolock, lock_args)\u001b[0m\n\u001b[1;32m   1359\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1360\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1361\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1362\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mnolock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1363\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelease\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36mdisplay\u001b[0;34m(self, msg, pos)\u001b[0m\n\u001b[1;32m   1507\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mpos\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1508\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmoveto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpos\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1509\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__str__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmsg\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1510\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mpos\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1511\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmoveto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mpos\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36mprint_status\u001b[0;34m(s)\u001b[0m\n\u001b[1;32m    348\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mprint_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m             \u001b[0mlen_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdisp_len\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 350\u001b[0;31m             \u001b[0mfp_write\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\r'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0ms\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m' '\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlast_len\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlen_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    351\u001b[0m             \u001b[0mlast_len\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen_s\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    352\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/std.py\u001b[0m in \u001b[0;36mfp_write\u001b[0;34m(s)\u001b[0m\n\u001b[1;32m    342\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mfp_write\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    343\u001b[0m             \u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_unicode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 344\u001b[0;31m             \u001b[0mfp_flush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    345\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    346\u001b[0m         \u001b[0mlast_len\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tqdm/utils.py\u001b[0m in \u001b[0;36minner\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    143\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0minner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 145\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    146\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    147\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merrno\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m5\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ipykernel/iostream.py\u001b[0m in \u001b[0;36mflush\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    339\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpub_thread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mthread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_alive\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    340\u001b[0m             \u001b[0;31m# request flush on the background thread\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 341\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpub_thread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschedule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_flush\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    342\u001b[0m             \u001b[0;31m# wait for flush to actually get through, if we can.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    343\u001b[0m             \u001b[0;31m# waiting across threads during import can cause deadlocks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ipykernel/iostream.py\u001b[0m in \u001b[0;36mschedule\u001b[0;34m(self, f)\u001b[0m\n\u001b[1;32m    201\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_events\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;31m# wake event thread (message content is ignored)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 203\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_event_pipe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mb''\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    204\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m             \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/zmq/sugar/socket.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, data, flags, copy, track, routing_id, group)\u001b[0m\n\u001b[1;32m    545\u001b[0m                 )\n\u001b[1;32m    546\u001b[0m             \u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroup\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 547\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mSocket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mflags\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrack\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrack\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    548\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    549\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0msend_multipart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg_parts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrack\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket.Socket.send\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket.Socket.send\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket._send_copy\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/zmq/backend/cython/checkrc.pxd\u001b[0m in \u001b[0;36mzmq.backend.cython.checkrc._check_rc\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "from tqdm import tqdm\n",
        "f = open(\"./all_mt_2.txt\",'r')\n",
        "f2 = open(\"./newcorpus.txt\",'w')\n",
        "new_sentences = []\n",
        "for l in tqdm(f.readlines()):\n",
        "  write = True\n",
        "  for token in l.rstrip().split(\" \"):\n",
        "    if token not in vocab and token not in ['.',',',';','?']:\n",
        "      write = False\n",
        "  if write and len(l.rstrip().split(\" \")) > 3:\n",
        "    new_sentences.append(l)\n",
        "    f2.write(l)\n",
        "  \n",
        "len(new_sentences)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NPdtRSLZi_PO",
        "outputId": "b9442f60-46cf-403c-e74e-7df0c10647b3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "8259"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(vocab)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ymMb7IqfYKK",
        "outputId": "71b3f937-4890-4f06-a3f3-66432c3c23a0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "3880"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sentences = [s.rstrip() for s in new_sentences]\n",
        "len(sentences)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcBXwgbHffqE"
      },
      "source": [
        "## Create Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ywKWo28HfnBk"
      },
      "outputs": [],
      "source": [
        "from tokenizers import Tokenizer\n",
        "from tokenizers.models import WordLevel\n",
        "from tokenizers.pre_tokenizers import WhitespaceSplit\n",
        "from tokenizers.processors import BertProcessing\n",
        "\n",
        "sense_tokenizer = Tokenizer(WordLevel(unk_token=\"[UNK]\"\n",
        "  ))\n",
        "sense_tokenizer.add_special_tokens([\"[SEP]\", \"[CLS]\", \"[PAD]\", \"[MASK]\",\"[UNK]\"])\n",
        "sense_tokenizer.pre_tokenizer = WhitespaceSplit()\n",
        "\n",
        "sep_token = \"[SEP]\"\n",
        "cls_token = \"[CLS]\"\n",
        "pad_token = \"[PAD]\"\n",
        "unk_token = \"[UNK]\"\n",
        "sep_token_id = sense_tokenizer.token_to_id(str(sep_token))\n",
        "cls_token_id = sense_tokenizer.token_to_id(str(cls_token))\n",
        "pad_token_id = sense_tokenizer.token_to_id(str(pad_token))\n",
        "unk_token_id = sense_tokenizer.token_to_id(str(unk_token))\n",
        "\n",
        "\n",
        "sense_tokenizer.post_processor = BertProcessing(\n",
        "                (str(sep_token), sep_token_id), (str(cls_token), cls_token_id)\n",
        "            )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rUxh818Tfo7k",
        "outputId": "8110c50b-f238-418a-d2a1-5d42d8676e98"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vocab size:  8266\n"
          ]
        }
      ],
      "source": [
        "from tokenizers.trainers import WordLevelTrainer\n",
        "g = WordLevelTrainer(special_tokens=[\"[UNK]\"])\n",
        "sense_tokenizer.train_from_iterator(sentences+vocab, trainer=g)\n",
        "print(\"Vocab size: \", sense_tokenizer.get_vocab_size())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6Tg84Qc0gENC",
        "outputId": "a1b09eae-64c9-461a-f826-4a9b1ee09921"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer_vocab = [w for w,i in sense_tokenizer.get_vocab().items()]\n",
        "difference = list(set(vocab).difference(set(tokenizer_vocab)))\n",
        "difference"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yH28jhObgtXF"
      },
      "outputs": [],
      "source": [
        "sense_tokenizer.save(\"./tokenizer_arasaac.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ILpwUsjjg3NV"
      },
      "source": [
        "## Dataset preparation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b-LjKEOKg40f",
        "outputId": "0e64bb3a-c9ac-4fca-ad97-6cf1fb3ef3d0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(3104, 388, 388)"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "TEST_SIZE = 0.2\n",
        "from sklearn.model_selection import train_test_split\n",
        "train_idx, val_idx = train_test_split(list(range(len(sentences))), test_size=TEST_SIZE, random_state=8)\n",
        "test_idx, val_idx = train_test_split(val_idx, test_size=0.5, random_state=8)\n",
        "\n",
        "import numpy as np\n",
        "train_examples = np.array(sentences).take(train_idx)\n",
        "val_examples = np.array(sentences).take(val_idx)\n",
        "test_examples = np.array(sentences).take(test_idx)\n",
        "len(train_examples),len(val_examples), len(test_examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ctIurBAOlENJ"
      },
      "outputs": [],
      "source": [
        "from transformers import PreTrainedTokenizerFast\n",
        "TOKENIZER_PATH = \"./tokenizer_arasaac.json\"\n",
        "loaded_tokenizer = PreTrainedTokenizerFast(tokenizer_file=TOKENIZER_PATH)\n",
        "loaded_tokenizer.pad_token = \"[PAD]\"\n",
        "loaded_tokenizer.sep_token = \"[SEP]\"\n",
        "loaded_tokenizer.mask_token = \"[MASK]\"\n",
        "loaded_tokenizer.cls_token = \"[CLS]\"\n",
        "loaded_tokenizer.unk_token = \"[UNK]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lFKB7NgOljEY"
      },
      "outputs": [],
      "source": [
        "max_len = 16\n",
        "\n",
        "def tokenize_function(tokenizer,examples):\n",
        "    # Remove empty lines\n",
        "    examples = [line for line in examples if len(line) > 0 and not line.isspace()]\n",
        "    bert = tokenizer(\n",
        "        examples,\n",
        "        padding=\"max_length\",\n",
        "        max_length=max_len,\n",
        "        return_special_tokens_mask=True,\n",
        "        truncation=True\n",
        "    )\n",
        "    return bert"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_bCyrpVylkyu"
      },
      "outputs": [],
      "source": [
        "train_tokenized_examples = tokenize_function(loaded_tokenizer,train_examples)\n",
        "val_tokenized_examples = tokenize_function(loaded_tokenizer,val_examples)\n",
        "test_tokenized_examples = tokenize_function(loaded_tokenizer,test_examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yaSxcUDwlnaG"
      },
      "outputs": [],
      "source": [
        "from torch import tensor\n",
        "def make_dict(examples):\n",
        "  return {\n",
        "      \"input_ids\": examples.input_ids,\n",
        "      \"attention_mask\":examples.attention_mask,\n",
        "      \"special_tokens_mask\":examples.special_tokens_mask\n",
        "  }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Km5zFrCBlos1"
      },
      "outputs": [],
      "source": [
        "!mkdir data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T6uJvWR6lqS8"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "TRAIN_DATA_PATH = \"./data/train_data.pt\"\n",
        "TEST_DATA_PATH = \"./data/test_data.pt\"\n",
        "VAL_DATA_PATH = \"./data/val_data.pt\"\n",
        "\n",
        "pickle.dump(make_dict(train_tokenized_examples),open(TRAIN_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(val_tokenized_examples),open(TEST_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(test_tokenized_examples),open(VAL_DATA_PATH ,'wb'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yuSDOnMFltMs"
      },
      "source": [
        "## PictoBERT adaptation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_lP5L6xnmLi_"
      },
      "source": [
        "### Load PictoBERT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PFkT3zIRl1xl"
      },
      "outputs": [],
      "source": [
        "from transformers import BertForMaskedLM\n",
        "pictobert_version = \"contextual\" #@param [\"contextual\",\"gloss\"]\n",
        "\n",
        "if pictobert_version == \"contextual\":\n",
        "  pictobert = BertForMaskedLM.from_pretrained(\"./pictobert\")\n",
        "else:\n",
        "  pictobert = BertForMaskedLM.from_pretrained(\"./pictobert-gloss\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWe-Efu2mhIa"
      },
      "source": [
        "### Load BERT\n",
        "\n",
        "At this point, we need to also load BERT as its embeddings matrix may be used to calculate the input embeddings of tokens that are not on PictoBERT or ARES vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217,
          "referenced_widgets": [
            "9d6e38e136194930b68ab02cd081b629",
            "adf49eb3cf2f4826a23a7f7b8e2417bf",
            "206140f900724537adea491e3db6f3d9",
            "c7323798b6504908980276d683669f77",
            "5175b1a54bc843418e0cdeebdf948235",
            "ec74c7bd0ccd477aa26ab3bd40a5efe1",
            "54a414ed8d884bf19b7f569cce7ff806",
            "98345cb4fbfa4ec08d2426978077153f",
            "e1d89b1f4f6e484ca080781202b7fa89",
            "786b8ee4ac90446c8937e536997f42c7",
            "7662cbd8bb5f428c929fd56be4db1782",
            "689a171db62e4ac7b60c15f6b776ff60",
            "3edfc2462e9d46a3ac027fea6ef43611",
            "36d51ed8ea514cdfa8f9ffad3fa43dd2",
            "ea41c677305b4ffd97e51e0794fe4c3c",
            "e6bc6682eb7e4bdbae5efc7327e27669",
            "ac63a25adf6d427b90fc1bb719b9325b",
            "b5ccd4fb32e84121a829f6caac41f557",
            "8c7e1d7324e146438bbe212fa917311b",
            "3b745ebad34d4295a323fdf73245dbc3",
            "d6eeb1c85a944e04ab47d98442d459e3",
            "323b87cdb89240bcacffdb5f714d94f7",
            "43110be75d9f47c9b5f8bf6359267a8c",
            "4f5dc08ca94947918d78809ffb3e7cc1",
            "7fc0494b8daf49089d82737a795135b2",
            "5962c91a33ab4e77b701f73fd9e61bc9",
            "1dc39865e4694315b9fe843bd9462a31",
            "94dfe3b11d4049b7ac76e0893bd883a3",
            "b6916faa864b492bad298c88ac740ff8",
            "d75c14850615436687009374642f3bd9",
            "8bec4635bb184abfba49d901a7194949",
            "2d206a60143e4e589b4cfd549112e1a5",
            "4495429827734035b739638c983767e2",
            "b07a67450bae4ce8b0f71f3b540ef4b7",
            "cbe95676de4a4dc39f1d60fc98d5a323",
            "5d74f401e67e4c70bbf5fbf7b80aeadc",
            "e91dd32b825e406aaa8f6fa6cdfc3555",
            "77df5bfcf70c4544951bc87752057745",
            "88bceac7864e43638727495cf5116998",
            "5bb942bdbeee447297e5318b60b17d17",
            "eb6e58c5cd4f4cf6bd42ad689558033e",
            "6d33ea78b690455c8d30672d5c27d6e9",
            "4c6aabfa8f1f48fa8201d5a615551aa3",
            "5ac89fd149b14a0db83b562e5b432757"
          ]
        },
        "id": "dgfVqcX7mihk",
        "outputId": "c86d6eed-c7e4-4a43-c2dc-b25b249f8c30"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9d6e38e136194930b68ab02cd081b629",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "689a171db62e4ac7b60c15f6b776ff60",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "43110be75d9f47c9b5f8bf6359267a8c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/571 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b07a67450bae4ce8b0f71f3b540ef4b7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.25G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-large-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "from transformers import BertTokenizer\n",
        "\n",
        "pretrained_w = 'bert-large-uncased'\n",
        "tokenizer_bert = BertTokenizer.from_pretrained(pretrained_w)\n",
        "bert = BertForMaskedLM.from_pretrained(pretrained_w)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9DSB9BGjmpN2"
      },
      "source": [
        "### Load PictoBERT tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tl_4CvD6mpvP"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./childes_all_new.json\" # you can change this path to use your custom tokenizer\n",
        "\n",
        "from transformers import PreTrainedTokenizerFast\n",
        "\n",
        "pictobert_tokenizer = PreTrainedTokenizerFast(tokenizer_file=TOKENIZER_PATH)\n",
        "pictobert_tokenizer.pad_token = \"[PAD]\"\n",
        "pictobert_tokenizer.sep_token = \"[SEP]\"\n",
        "pictobert_tokenizer.mask_token = \"[MASK]\"\n",
        "pictobert_tokenizer.cls_token = \"[CLS]\"\n",
        "pictobert_tokenizer.unk_token = \"[UNK]\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU2-jsOWmsgH"
      },
      "source": [
        "### Update embeddings layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ea3a8sHVmtu7"
      },
      "outputs": [],
      "source": [
        "new_vocab = loaded_tokenizer.get_vocab()\n",
        "pictobert_vocab = pictobert_tokenizer.get_vocab()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aul1STvCm0XS",
        "outputId": "f04407f1-5d6e-46ce-db46-3752d2cd8f78"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "New vocab size: 8265\n",
            "PictoBERT vocab size: 13583\n",
            "Commom: 3513\n",
            "New tokens: 4752\n"
          ]
        }
      ],
      "source": [
        "in_pictobert = []\n",
        "not_in = []\n",
        "for w,idx_new in new_vocab.items():\n",
        "  idx_old = pictobert_vocab.get(w, -1)\n",
        "  if idx_old >= 0:\n",
        "    in_pictobert.append(w)\n",
        "  else:\n",
        "    not_in.append(w)\n",
        "  \n",
        "print(\"New vocab size:\",len(new_vocab))\n",
        "print(\"PictoBERT vocab size:\", len(pictobert_vocab))\n",
        "print(\"Commom:\",len(in_pictobert))\n",
        "print(\"New tokens:\",len(not_in))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vr7tNNtTm4XI"
      },
      "outputs": [],
      "source": [
        "bert_embeddings = bert.get_input_embeddings()\n",
        "pictobert_embeddings = pictobert.get_input_embeddings()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YINX7f_Tm46Z"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from gensim.models import KeyedVectors\n",
        "\n",
        "if pictobert_version == \"contextual\":\n",
        "  ares = KeyedVectors.load_word2vec_format(\"/content/ares_1024.bin\", binary=True)\n",
        "else:\n",
        "  ares = KeyedVectors.load_word2vec_format(\"/content/ares_1024_gloss.bin\", binary=True)\n",
        "ares_contextual_mean = torch.tensor(ares.vectors).mean(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U401pr55nFfq"
      },
      "outputs": [],
      "source": [
        "bert_vocab = tokenizer_bert.get_vocab()\n",
        "# new_vocab = cs_tokenizer.get_vocab()\n",
        "\n",
        "bert_wgts = bert.get_input_embeddings().weight.clone().detach()\n",
        "pictobert_wgts = pictobert.get_input_embeddings().weight.clone().detach()\n",
        "\n",
        "new_vocab_size = len(loaded_tokenizer.vocab)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q2rdTYFqnISE"
      },
      "outputs": [],
      "source": [
        "new_wgts = pictobert_wgts.new_zeros(new_vocab_size,pictobert_wgts.size(1))\n",
        "\n",
        "same_tokens_list = list()\n",
        "different_tokens_list = list()\n",
        "received_mean = list()\n",
        "\n",
        "for w,idx_new in new_vocab.items():\n",
        "  idx_pictobert = pictobert_vocab.get(w,-1)\n",
        "  if idx_pictobert >= 0:\n",
        "    new_wgts[idx_new] = pictobert_wgts[idx_pictobert]\n",
        "  elif w in ares:\n",
        "    new_wgts[idx_new] = torch.tensor(ares[w])\n",
        "  else:\n",
        "    w_ = \" \".join(w.split(\"_\"))\n",
        "    tokenized = tokenizer_bert(w_,add_special_tokens=False,return_tensors='pt')\n",
        "    v_ = bert_embeddings(tokenized.input_ids[0]).mean(0)\n",
        "    new_wgts[idx_new] = v_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zCx02JFAnJta"
      },
      "outputs": [],
      "source": [
        "from torch import nn\n",
        "\n",
        "new_wte = nn.Embedding(new_vocab_size,pictobert_wgts.size(1)) # new embeddings\n",
        "new_wte.weight.data.normal_(mean=0,std=pictobert.config.initializer_range) \n",
        "new_wte.weight.data = new_wgts\n",
        "\n",
        "pictobert.resize_token_embeddings(len(loaded_tokenizer))\n",
        "pictobert.set_input_embeddings(new_wte)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UARVDUW2nL0J"
      },
      "outputs": [],
      "source": [
        "MODEL_NAME = \"./pictobert-ARASAAC-{0}\".format(pictobert_version)\n",
        "pictobert.save_pretrained(MODEL_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11n1uTCqnPqE"
      },
      "source": [
        "## Train Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcnW6QEPnSb3"
      },
      "source": [
        "### Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FJAXBRWQnpvt"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./cs_tokenizer.json\"\n",
        "LOGS_PATH = \"./logs\"\n",
        "CHECKPOINTS_PATH = \"./checkpoints\"\n",
        "\n",
        "TRAIN_DATASET_PATH = \"./data/train_data.pt\"\n",
        "VAL_DATASET_PATH = \"./data/val_data.pt\"\n",
        "TEST_DATASET_PATH = \"./data/test_data.pt\"\n",
        "\n",
        "MAX_EPOCHS = 10\n",
        "WARMUP_STEPS = int(MAX_EPOCHS * 0.15)\n",
        "BATCH_SIZE = 32\n",
        "NUM_WORKERS = 4\n",
        "GPUS = 1\n",
        "LEARNING_RATE = 1e-06\n",
        "ACCUMULATE_GRAD_BATCHES = 4\n",
        "LOGGER_VERSION = '1e06'\n",
        "LOGGER_INFO = \"first_run\"\n",
        "FREEZE_TO = None\n",
        "MLM_PROBABILITY= 0.15"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0SYNwQMnzVy"
      },
      "source": [
        "### Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4qQ2Gzlln0yV"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import Dataset, Subset\n",
        "from torch import tensor\n",
        "from sklearn.model_selection import train_test_split\n",
        "import pickle\n",
        "\n",
        "class MyDataset(Dataset):\n",
        "  def __init__(self, examples):\n",
        "    \n",
        "    self.input_ids = examples['input_ids']\n",
        "    self.attention_mask = examples['attention_mask']\n",
        "    self.special_tokens_mask = examples['special_tokens_mask']\n",
        "    self.labels = None\n",
        "    if 'labels' in examples:\n",
        "      self.labels = examples['labels']\n",
        "  \n",
        "  def __len__(self):\n",
        "    return len(self.input_ids)\n",
        "  \n",
        "  def __getitem__(self, idx):\n",
        "    input_ids = tensor(self.input_ids[idx])\n",
        "    attention_mask = tensor(self.attention_mask[idx])\n",
        "    special_tokens_mask = tensor(self.special_tokens_mask[idx])\n",
        "\n",
        "    out_dict = {\n",
        "      \"input_ids\":input_ids, \n",
        "      \"attention_mask\":attention_mask, \n",
        "      \"special_tokens_mask\":special_tokens_mask\n",
        "    }\n",
        "\n",
        "    if self.labels is not None:\n",
        "      out_dict['labels'] = self.labels[idx]\n",
        "\n",
        "    return out_dict\n",
        "\n",
        "\n",
        "tds = pickle.load(open(TRAIN_DATASET_PATH,'rb'))\n",
        "train_dataset = MyDataset(tds)\n",
        "\n",
        "vds = pickle.load(open(VAL_DATASET_PATH,'rb'))\n",
        "val_dataset = MyDataset(vds)\n",
        "\n",
        "tsds = pickle.load(open(TEST_DATASET_PATH,'rb'))\n",
        "test_dataset = MyDataset(tsds)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PStuQ5JToA6K"
      },
      "outputs": [],
      "source": [
        "from transformers import DataCollatorForLanguageModeling\n",
        "data_collator = DataCollatorForLanguageModeling(tokenizer=loaded_tokenizer, mlm_probability=MLM_PROBABILITY)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fECpGxf0oCn6"
      },
      "outputs": [],
      "source": [
        "def top_n_data_collator(examples):\n",
        "  \n",
        "  batch = {\n",
        "      \"input_ids\" : torch.stack([example['input_ids'] for example in examples]),\n",
        "      \"attention_mask\": torch.stack([example['attention_mask'] for example in examples]),\n",
        "  }\n",
        "  special_tokens_mask = torch.stack([example['special_tokens_mask'] for example in examples])\n",
        "  masked_indices = special_tokens_mask.bool()\n",
        "\n",
        "  mask_labels = special_tokens_mask.clone()\n",
        "  probability_matrix = mask_labels\n",
        "  probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)\n",
        "  masked_indices = probability_matrix.bool()\n",
        "\n",
        "  labels = batch['input_ids'].clone()\n",
        "  \n",
        "  for i, row in enumerate(special_tokens_mask.bool()):\n",
        "    mask_id = torch.where(row==False)[0][-2]\n",
        "    masked_indices[i][mask_id] = True\n",
        "    batch['input_ids'][i][mask_id] = loaded_tokenizer.mask_token_id\n",
        "\n",
        "\n",
        "  labels[~masked_indices] =  -100\n",
        "  \n",
        "  batch['labels'] = labels\n",
        "\n",
        "  return batch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HSxP4BDZoERD",
        "outputId": "01ca1287-e76d-4f15-e4e2-1bf6c74cdf42"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
            "  cpuset_checked))\n"
          ]
        }
      ],
      "source": [
        "from torch.utils.data import DataLoader\n",
        "from torch.utils.data.distributed import DistributedSampler\n",
        "\n",
        "train_dataloader = DataLoader(\n",
        "    train_dataset,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    num_workers=NUM_WORKERS,\n",
        "    pin_memory=True,\n",
        "    collate_fn=data_collator,\n",
        "    drop_last = True,\n",
        "    shuffle=True,\n",
        "    \n",
        ")\n",
        "\n",
        "val_dataloader = DataLoader(\n",
        "    val_dataset,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    num_workers=NUM_WORKERS,\n",
        "    pin_memory=True,\n",
        "    collate_fn=data_collator,\n",
        "    drop_last = True\n",
        ")\n",
        "\n",
        "test_dataloader = DataLoader(\n",
        "    test_dataset,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    num_workers=NUM_WORKERS,\n",
        "    collate_fn=top_n_data_collator,\n",
        "    pin_memory=True,\n",
        "    drop_last = True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xPBkIFYjoLGV"
      },
      "source": [
        "### Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xGY6KucxoM4N"
      },
      "outputs": [],
      "source": [
        "from argparse import ArgumentParser\n",
        "import math\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from transformers import get_polynomial_decay_schedule_with_warmup\n",
        "from transformers import AdamW\n",
        "import pytorch_lightning as pl\n",
        "from sklearn.metrics import accuracy_score\n",
        "from pytorch_lightning.callbacks import ModelCheckpoint\n",
        "from scipy import stats\n",
        "import numpy as np\n",
        "import torch.nn.functional as F\n",
        "\n",
        "percentiles = [\n",
        "  {\n",
        "    \"percentil\":'1%',\n",
        "    \"z\": stats.norm.ppf(1-0.01),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'5%',\n",
        "    \"z\": stats.norm.ppf(1-0.05),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'10%',\n",
        "    \"z\": stats.norm.ppf(1-0.1),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'15%',\n",
        "    \"z\": stats.norm.ppf(1-0.15),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'20%',\n",
        "    \"z\": stats.norm.ppf(1-0.2),\n",
        "  },\n",
        "]\n",
        "\n",
        "firsts = [1,9,18,\t25,\t36]\n",
        "\n",
        "from transformers import BertLMHeadModel\n",
        "from transformers import BertForMaskedLM\n",
        "\n",
        "class LitBertClassifier(pl.LightningModule):\n",
        "    def __init__(self, pretrained_model_name='bert-large-uncased'):\n",
        "        super().__init__()\n",
        "        self.save_hyperparameters()\n",
        "        self.batch_size = 128\n",
        "        self.lr = LEARNING_RATE\n",
        "        self.train_dataset = train_dataset\n",
        "        self.bert = BertForMaskedLM.from_pretrained(pretrained_model_name)\n",
        "      \n",
        "    \n",
        "    def freeze_to(self, layers):\n",
        "      for param in self.bert.bert.encoder.layer[:layers].parameters():\n",
        "        param.requires_grad = False\n",
        "\n",
        "\n",
        "    def forward(self, input_ids, attention_mask, labels=None):\n",
        "        if labels == None:\n",
        "            return self.bert(\n",
        "                input_ids=input_ids,\n",
        "                attention_mask=attention_mask,\n",
        "            )    \n",
        "        return self.bert(\n",
        "            input_ids=input_ids,\n",
        "            attention_mask=attention_mask,\n",
        "            labels = labels\n",
        "        )\n",
        "\n",
        "    def training_step(self, batch, batch_idx):\n",
        "        outputs = self._shared_step(batch, batch_idx)\n",
        "        loss = outputs[0]\n",
        "\n",
        "        self.log(\"train_loss\", loss, on_epoch=True, prog_bar=True,)\n",
        "\n",
        "        return loss\n",
        "\n",
        "\n",
        "    def train_dataloader(self):\n",
        "      # return DataLoader(self.train_dataset,batch_size=self.batch_size,num_workers=NUM_WORKERS,pin_memory=True,collate_fn=data_collator)\n",
        "      train_dataloader = DataLoader(\n",
        "          train_dataset,\n",
        "          batch_size=self.batch_size,\n",
        "          num_workers=NUM_WORKERS,\n",
        "          pin_memory=True,\n",
        "          collate_fn=data_collator,\n",
        "          drop_last = True,\n",
        "          shuffle=True\n",
        "      )\n",
        "      return train_dataloader\n",
        "      # return DataLoader(self.train_dataset, batch_size=self.batch_size)\n",
        "\n",
        "    def validation_step(self, batch, batch_idx):\n",
        "        with torch.no_grad():\n",
        "          result = self._shared_step(batch, batch_idx)\n",
        "          loss = result[0].detach()\n",
        "          \n",
        "          return {\n",
        "              \"val_loss\":loss\n",
        "          }\n",
        "    \n",
        "    def validation_epoch_end(self, outputs):\n",
        "        val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()\n",
        "        self.log(\"val_loss\", val_loss, on_epoch=True, prog_bar=True,)\n",
        "    \n",
        "    \n",
        "    def test_step(self, batch, batch_idx):\n",
        "        with torch.no_grad():\n",
        "          result = self._shared_step(batch, batch_idx)\n",
        "          loss = result[0].detach()\n",
        "          perplexity = torch.exp(loss)\n",
        "          self.log(\"test_ppl\", perplexity, on_epoch=True, prog_bar=True,)\n",
        "          self.log(\"test_loss\", loss, on_epoch=True, prog_bar=True,)\n",
        "\n",
        "          predictions = F.softmax(result[1], dim=-1)\n",
        "          labels = batch['labels']\n",
        "          masked = batch['input_ids']\n",
        "          n = masked.detach().cpu().numpy()\n",
        "          predicted = predictions.detach().cpu().numpy()[n == loaded_tokenizer.mask_token_id]\n",
        "\n",
        "          ids = np.argsort(-1*predicted,axis=1)\n",
        "          for first in firsts:\n",
        "            count = 0\n",
        "            for i, data in enumerate(ids):\n",
        "              if labels[masked == loaded_tokenizer.mask_token_id][i].item() in data[:first]:\n",
        "                count += 1\n",
        "            isin = count/predicted.shape[0]\n",
        "\n",
        "            self.log(\"test_\"+str(first), isin, on_epoch=True, prog_bar=True,)\n",
        "          \n",
        "          return {\n",
        "              \"test_ppl\":perplexity,\n",
        "              \"test_loss\":loss,\n",
        "          }\n",
        "\n",
        "\n",
        "    def _shared_step(self, batch, batch_idx):\n",
        "        input_ids = batch[\"input_ids\"]\n",
        "        attention_mask = batch[\"attention_mask\"]\n",
        "        labels = batch[\"labels\"]\n",
        "\n",
        "        outputs = self.forward(\n",
        "            input_ids=input_ids,\n",
        "            attention_mask=attention_mask,\n",
        "            labels=labels\n",
        "        )\n",
        "\n",
        "        return outputs\n",
        "\n",
        "    def configure_optimizers(self):\n",
        "      optimizer = AdamW(self.parameters(), lr=self.lr,betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01)\n",
        "      scheduler = {\n",
        "          'scheduler': get_polynomial_decay_schedule_with_warmup(optimizer, num_warmup_steps=WARMUP_STEPS,num_training_steps=MAX_EPOCHS,lr_end=1e-09),\n",
        "          'name': 'lr'\n",
        "      }\n",
        "      return [optimizer],[scheduler]\n",
        "    \n",
        "    def backward(self, loss, optimizer, idx):\n",
        "        loss.backward()\n",
        "    \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AJsGoLT7oLOZ"
      },
      "source": [
        "### Logger and Checkpointing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_t2NjeMpopmJ"
      },
      "outputs": [],
      "source": [
        "from pytorch_lightning import loggers as pl_loggers\n",
        "from pytorch_lightning.callbacks import LearningRateMonitor\n",
        "\n",
        "tb_logger = pl_loggers.TensorBoardLogger(LOGS_PATH,name=LOGGER_INFO, version=LOGGER_VERSION)\n",
        "lr_monitor = LearningRateMonitor(logging_interval='epoch')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_hhGymY4os7-"
      },
      "outputs": [],
      "source": [
        "checkpoint_callback = ModelCheckpoint(\n",
        "    dirpath=CHECKPOINTS_PATH,\n",
        "    filename='bert-large-{epoch:02d}-{val_loss:.2f}',\n",
        "    mode='min',\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZV4qvkIoxLO"
      },
      "source": [
        "### Trainer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QEmRhQkCo1Sn",
        "outputId": "e701e6a9-20f8-472d-bcbd-2f88a2463b2a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "GPU available: True, used: True\n",
            "TPU available: False, using: 0 TPU cores\n",
            "Using native 16bit precision.\n"
          ]
        }
      ],
      "source": [
        "trainer = pl.Trainer(\n",
        "    accelerator='ddp',\n",
        "    max_epochs=MAX_EPOCHS,\n",
        "    logger=tb_logger,\n",
        "    gpus=GPUS,\n",
        "    callbacks=[checkpoint_callback, lr_monitor],\n",
        "    precision=16,\n",
        "    auto_scale_batch_size=\"binsearch\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k5sVs8kvo7PC"
      },
      "outputs": [],
      "source": [
        "to_train = LitBertClassifier(MODEL_NAME)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LkK2VMTJo80y",
        "outputId": "86b6463d-2b56-465a-9a5f-50e1242fc83f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:68: UserWarning: you defined a validation_step but have no val_dataloader. Skipping validation loop\n",
            "  warnings.warn(*args, **kwargs)\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n",
            "initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1\n",
            "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
            "  cpuset_checked))\n",
            "/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
            "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n",
            "Batch size 2 succeeded, trying batch size 4\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 4 succeeded, trying batch size 8\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 8 succeeded, trying batch size 16\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 16 succeeded, trying batch size 32\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 32 succeeded, trying batch size 64\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 64 succeeded, trying batch size 128\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 128 succeeded, trying batch size 256\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 256 failed, trying batch size 192\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 192 succeeded, trying batch size 224\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 224 failed, trying batch size 208\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 208 failed, trying batch size 200\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 200 failed, trying batch size 196\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 196 failed, trying batch size 194\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 194 failed, trying batch size 193\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "Batch size 193 failed, trying batch size 192\n",
            "Finished batch size finder, will continue with full run using batch size 192\n",
            "Restored states from the checkpoint file at /content/scale_batch_size_temp_model.ckpt\n"
          ]
        }
      ],
      "source": [
        "trainer.tune(to_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 344,
          "referenced_widgets": [
            "96467dc42d27424e93f5854c610ef93a",
            "4cdc7f248bc84a9b9b7ce31479f9ecc6",
            "f5802c59788f42848b8426ee332d1c60",
            "244a9ea2a5aa450abe8f457839035fec",
            "cd0d3bab38664e989009c8dbfc68e07f",
            "adea2d46de85410aad6450eb09cdef60",
            "b49e39e229714df6aa81f1f08c46ca06",
            "74057ba12b5c47b4ad1dde01103e24f9",
            "dfa2caac84dd4c1e9eef81d6156446dd",
            "c88f33e308d042bd8b70080bc23b37e2",
            "91aa7a1238fd42e1b59953abdd10cceb",
            "6b72aae676ee467c9008ac372e2e5775",
            "93f4bfe2727e4493abfb02fc8b01595b",
            "b1c5195163ba4455b444fa75c27e0ba6",
            "12d0e27a12b04aa49a9dd60ae5f92ba2",
            "7de9547ad6e64af6ac6c98c6aa2afa43",
            "a4cca6e0331a47628471d0257f288f2c",
            "4d544fb8047648ed934e9ba6fa98a033",
            "5b8a81347a91469b98377f2e83338395",
            "ec58e3cb1f4e48b68288e6d6674cf2e2",
            "c445082bf7a64db99fe7849001a1790b",
            "5dc0a3ab6a3c4a87a0432a1c0c6cc45c"
          ]
        },
        "id": "Cz7gcRbJpFNU",
        "outputId": "709d4c06-cfd2-4971-e3fc-92cf256852be"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n",
            "\n",
            "  | Name | Type            | Params\n",
            "-----------------------------------------\n",
            "0 | bert | BertForMaskedLM | 312 M \n",
            "-----------------------------------------\n",
            "312 M     Trainable params\n",
            "0         Non-trainable params\n",
            "312 M     Total params\n",
            "1,249.444 Total estimated model params size (MB)\n",
            "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
            "  cpuset_checked))\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "96467dc42d27424e93f5854c610ef93a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Validation sanity check: 0it [00:00, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6b72aae676ee467c9008ac372e2e5775",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Training: 0it [00:00, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "trainer.fit(to_train, train_dataloader,val_dataloader)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "ARASAAC fine-tuned PictoBERT.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "12d0e27a12b04aa49a9dd60ae5f92ba2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c445082bf7a64db99fe7849001a1790b",
            "placeholder": "​",
            "style": "IPY_MODEL_5dc0a3ab6a3c4a87a0432a1c0c6cc45c",
            "value": " 40/109 [01:16&lt;02:11,  1.91s/it, loss=17.9, v_num=1e06, train_loss_step=14.30, train_loss_epoch=18.00, val_loss=17.20]"
          }
        },
        "1dc39865e4694315b9fe843bd9462a31": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "206140f900724537adea491e3db6f3d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_98345cb4fbfa4ec08d2426978077153f",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e1d89b1f4f6e484ca080781202b7fa89",
            "value": 231508
          }
        },
        "244a9ea2a5aa450abe8f457839035fec": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c88f33e308d042bd8b70080bc23b37e2",
            "placeholder": "​",
            "style": "IPY_MODEL_91aa7a1238fd42e1b59953abdd10cceb",
            "value": " 0/2 [00:01&lt;?, ?it/s]"
          }
        },
        "2d206a60143e4e589b4cfd549112e1a5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "323b87cdb89240bcacffdb5f714d94f7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "36d51ed8ea514cdfa8f9ffad3fa43dd2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8c7e1d7324e146438bbe212fa917311b",
            "max": 28,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3b745ebad34d4295a323fdf73245dbc3",
            "value": 28
          }
        },
        "3b745ebad34d4295a323fdf73245dbc3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3edfc2462e9d46a3ac027fea6ef43611": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ac63a25adf6d427b90fc1bb719b9325b",
            "placeholder": "​",
            "style": "IPY_MODEL_b5ccd4fb32e84121a829f6caac41f557",
            "value": "Downloading: 100%"
          }
        },
        "43110be75d9f47c9b5f8bf6359267a8c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4f5dc08ca94947918d78809ffb3e7cc1",
              "IPY_MODEL_7fc0494b8daf49089d82737a795135b2",
              "IPY_MODEL_5962c91a33ab4e77b701f73fd9e61bc9"
            ],
            "layout": "IPY_MODEL_1dc39865e4694315b9fe843bd9462a31"
          }
        },
        "4495429827734035b739638c983767e2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4c6aabfa8f1f48fa8201d5a615551aa3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4cdc7f248bc84a9b9b7ce31479f9ecc6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_adea2d46de85410aad6450eb09cdef60",
            "placeholder": "​",
            "style": "IPY_MODEL_b49e39e229714df6aa81f1f08c46ca06",
            "value": "Validation sanity check:   0%"
          }
        },
        "4d544fb8047648ed934e9ba6fa98a033": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4f5dc08ca94947918d78809ffb3e7cc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_94dfe3b11d4049b7ac76e0893bd883a3",
            "placeholder": "​",
            "style": "IPY_MODEL_b6916faa864b492bad298c88ac740ff8",
            "value": "Downloading: 100%"
          }
        },
        "5175b1a54bc843418e0cdeebdf948235": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "54a414ed8d884bf19b7f569cce7ff806": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5962c91a33ab4e77b701f73fd9e61bc9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2d206a60143e4e589b4cfd549112e1a5",
            "placeholder": "​",
            "style": "IPY_MODEL_4495429827734035b739638c983767e2",
            "value": " 571/571 [00:00&lt;00:00, 12.7kB/s]"
          }
        },
        "5ac89fd149b14a0db83b562e5b432757": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5b8a81347a91469b98377f2e83338395": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5bb942bdbeee447297e5318b60b17d17": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5d74f401e67e4c70bbf5fbf7b80aeadc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eb6e58c5cd4f4cf6bd42ad689558033e",
            "max": 1344997306,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6d33ea78b690455c8d30672d5c27d6e9",
            "value": 1344997306
          }
        },
        "5dc0a3ab6a3c4a87a0432a1c0c6cc45c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "689a171db62e4ac7b60c15f6b776ff60": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3edfc2462e9d46a3ac027fea6ef43611",
              "IPY_MODEL_36d51ed8ea514cdfa8f9ffad3fa43dd2",
              "IPY_MODEL_ea41c677305b4ffd97e51e0794fe4c3c"
            ],
            "layout": "IPY_MODEL_e6bc6682eb7e4bdbae5efc7327e27669"
          }
        },
        "6b72aae676ee467c9008ac372e2e5775": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_93f4bfe2727e4493abfb02fc8b01595b",
              "IPY_MODEL_b1c5195163ba4455b444fa75c27e0ba6",
              "IPY_MODEL_12d0e27a12b04aa49a9dd60ae5f92ba2"
            ],
            "layout": "IPY_MODEL_7de9547ad6e64af6ac6c98c6aa2afa43"
          }
        },
        "6d33ea78b690455c8d30672d5c27d6e9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "74057ba12b5c47b4ad1dde01103e24f9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7662cbd8bb5f428c929fd56be4db1782": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "77df5bfcf70c4544951bc87752057745": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "786b8ee4ac90446c8937e536997f42c7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7de9547ad6e64af6ac6c98c6aa2afa43": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "7fc0494b8daf49089d82737a795135b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d75c14850615436687009374642f3bd9",
            "max": 571,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8bec4635bb184abfba49d901a7194949",
            "value": 571
          }
        },
        "88bceac7864e43638727495cf5116998": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8bec4635bb184abfba49d901a7194949": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8c7e1d7324e146438bbe212fa917311b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "91aa7a1238fd42e1b59953abdd10cceb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "93f4bfe2727e4493abfb02fc8b01595b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a4cca6e0331a47628471d0257f288f2c",
            "placeholder": "​",
            "style": "IPY_MODEL_4d544fb8047648ed934e9ba6fa98a033",
            "value": "Epoch 0:  37%"
          }
        },
        "94dfe3b11d4049b7ac76e0893bd883a3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96467dc42d27424e93f5854c610ef93a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4cdc7f248bc84a9b9b7ce31479f9ecc6",
              "IPY_MODEL_f5802c59788f42848b8426ee332d1c60",
              "IPY_MODEL_244a9ea2a5aa450abe8f457839035fec"
            ],
            "layout": "IPY_MODEL_cd0d3bab38664e989009c8dbfc68e07f"
          }
        },
        "98345cb4fbfa4ec08d2426978077153f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9d6e38e136194930b68ab02cd081b629": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_adf49eb3cf2f4826a23a7f7b8e2417bf",
              "IPY_MODEL_206140f900724537adea491e3db6f3d9",
              "IPY_MODEL_c7323798b6504908980276d683669f77"
            ],
            "layout": "IPY_MODEL_5175b1a54bc843418e0cdeebdf948235"
          }
        },
        "a4cca6e0331a47628471d0257f288f2c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac63a25adf6d427b90fc1bb719b9325b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "adea2d46de85410aad6450eb09cdef60": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "adf49eb3cf2f4826a23a7f7b8e2417bf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ec74c7bd0ccd477aa26ab3bd40a5efe1",
            "placeholder": "​",
            "style": "IPY_MODEL_54a414ed8d884bf19b7f569cce7ff806",
            "value": "Downloading: 100%"
          }
        },
        "b07a67450bae4ce8b0f71f3b540ef4b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cbe95676de4a4dc39f1d60fc98d5a323",
              "IPY_MODEL_5d74f401e67e4c70bbf5fbf7b80aeadc",
              "IPY_MODEL_e91dd32b825e406aaa8f6fa6cdfc3555"
            ],
            "layout": "IPY_MODEL_77df5bfcf70c4544951bc87752057745"
          }
        },
        "b1c5195163ba4455b444fa75c27e0ba6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5b8a81347a91469b98377f2e83338395",
            "max": 109,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ec58e3cb1f4e48b68288e6d6674cf2e2",
            "value": 40
          }
        },
        "b49e39e229714df6aa81f1f08c46ca06": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b5ccd4fb32e84121a829f6caac41f557": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b6916faa864b492bad298c88ac740ff8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c445082bf7a64db99fe7849001a1790b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c7323798b6504908980276d683669f77": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_786b8ee4ac90446c8937e536997f42c7",
            "placeholder": "​",
            "style": "IPY_MODEL_7662cbd8bb5f428c929fd56be4db1782",
            "value": " 226k/226k [00:00&lt;00:00, 1.00MB/s]"
          }
        },
        "c88f33e308d042bd8b70080bc23b37e2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cbe95676de4a4dc39f1d60fc98d5a323": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_88bceac7864e43638727495cf5116998",
            "placeholder": "​",
            "style": "IPY_MODEL_5bb942bdbeee447297e5318b60b17d17",
            "value": "Downloading: 100%"
          }
        },
        "cd0d3bab38664e989009c8dbfc68e07f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "d6eeb1c85a944e04ab47d98442d459e3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d75c14850615436687009374642f3bd9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dfa2caac84dd4c1e9eef81d6156446dd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e1d89b1f4f6e484ca080781202b7fa89": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e6bc6682eb7e4bdbae5efc7327e27669": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e91dd32b825e406aaa8f6fa6cdfc3555": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4c6aabfa8f1f48fa8201d5a615551aa3",
            "placeholder": "​",
            "style": "IPY_MODEL_5ac89fd149b14a0db83b562e5b432757",
            "value": " 1.25G/1.25G [00:41&lt;00:00, 36.5MB/s]"
          }
        },
        "ea41c677305b4ffd97e51e0794fe4c3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d6eeb1c85a944e04ab47d98442d459e3",
            "placeholder": "​",
            "style": "IPY_MODEL_323b87cdb89240bcacffdb5f714d94f7",
            "value": " 28.0/28.0 [00:00&lt;00:00, 685B/s]"
          }
        },
        "eb6e58c5cd4f4cf6bd42ad689558033e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec58e3cb1f4e48b68288e6d6674cf2e2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ec74c7bd0ccd477aa26ab3bd40a5efe1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f5802c59788f42848b8426ee332d1c60": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "danger",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_74057ba12b5c47b4ad1dde01103e24f9",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_dfa2caac84dd4c1e9eef81d6156446dd",
            "value": 0
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}