BERT_Multi_Class_Classification_Liar_Liar_NoDistillv1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT_Multi_Class_Classification_Liar-Liar_NoDistillv1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/aarsanjani/AlternusVera/blob/master/BERT_Multi_Class_Classification_Liar_Liar_NoDistillv1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKOTlwcmxmej"
      },
      "source": [
        "# BERT Multi-Class Classification with Liar-Liar dataset\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lj_eBt3G3bo6"
      },
      "source": [
        "# Part I - Dataset & Tokenization\n",
        "---------------------------------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX_ZDhicpHkV"
      },
      "source": [
        "## S1. Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSU7yERLP_66"
      },
      "source": [
        "### 1.1. Connecting GPU\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv"
      },
      "source": [
        "In order for torch to use the GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we will load data onto the device. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYsV4H8fCpZ-",
        "outputId": "130feb1b-1740-42a3-d3c4-74dfe3af128d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import torch\n",
        "\n",
        "# If there's a GPU available...\n",
        "if torch.cuda.is_available():    \n",
        "\n",
        "    # Tell PyTorch to use the GPU.    \n",
        "    device = torch.device(\"cuda\")\n",
        "\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n",
        "\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\n",
        "\n",
        "# If not...\n",
        "else:\n",
        "    print('No GPU available, using the CPU instead.')\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla P100-PCIE-16GB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ElsnSNUridI"
      },
      "source": [
        "### 1.2. Installing `transformers` from huggingface\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_N2UDLevYWn"
      },
      "source": [
        "\n",
        "Next, let's install the [transformers](https://github.com/huggingface/transformers) package from Hugging Face which will give us a pytorch interface for working with BERT. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NmMdkZO8R6q",
        "outputId": "f904912a-a9c5-4f84-8aa1-2e2b42ba4458",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.6/dist-packages (3.4.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from transformers) (3.12.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers) (0.0.43)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.6/dist-packages (from transformers) (0.1.94)\n",
            "Requirement already satisfied: tokenizers==0.9.2 in /usr/local/lib/python3.6/dist-packages (from transformers) (0.9.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf->transformers) (50.3.2)\n",
            "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from protobuf->transformers) (1.15.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.6.20)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.17.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bt3HTo7LLMtj"
      },
      "source": [
        "## S2. Retrieve & Inspect Dataset\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5w1hKwT-Vyli",
        "outputId": "d6ed8716-ee2d-49b9-a987-28bba2c9b536",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U3LxGahoXiMs"
      },
      "source": [
        "import pandas as pd\n",
        "liar_train_df = pd.read_csv('/content/drive/My Drive/NLPData/train.tsv', sep='\\t', header=None)\n",
        "liar_test_df = pd.read_csv('/content/drive/My Drive/NLPData/test.tsv', sep='\\t', header=None) \n",
        "liar_valid_df = pd.read_csv('/content/drive/My Drive/NLPData/valid.tsv', sep='\\t', header=None) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IeGdojBhuPoQ",
        "outputId": "2bf50900-9950-4a4d-c591-b11ff6029102",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "liar_train_df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2635.json</td>\n",
              "      <td>false</td>\n",
              "      <td>Says the Annies List political group supports ...</td>\n",
              "      <td>abortion</td>\n",
              "      <td>dwayne-bohac</td>\n",
              "      <td>State representative</td>\n",
              "      <td>Texas</td>\n",
              "      <td>republican</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>a mailer</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>10540.json</td>\n",
              "      <td>half-true</td>\n",
              "      <td>When did the decline of coal start? It started...</td>\n",
              "      <td>energy,history,job-accomplishments</td>\n",
              "      <td>scott-surovell</td>\n",
              "      <td>State delegate</td>\n",
              "      <td>Virginia</td>\n",
              "      <td>democrat</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>a floor speech.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>324.json</td>\n",
              "      <td>mostly-true</td>\n",
              "      <td>Hillary Clinton agrees with John McCain \"by vo...</td>\n",
              "      <td>foreign-policy</td>\n",
              "      <td>barack-obama</td>\n",
              "      <td>President</td>\n",
              "      <td>Illinois</td>\n",
              "      <td>democrat</td>\n",
              "      <td>70.0</td>\n",
              "      <td>71.0</td>\n",
              "      <td>160.0</td>\n",
              "      <td>163.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>Denver</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1123.json</td>\n",
              "      <td>false</td>\n",
              "      <td>Health care reform legislation is likely to ma...</td>\n",
              "      <td>health-care</td>\n",
              "      <td>blog-posting</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>none</td>\n",
              "      <td>7.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>44.0</td>\n",
              "      <td>a news release</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9028.json</td>\n",
              "      <td>half-true</td>\n",
              "      <td>The economic turnaround started at the end of ...</td>\n",
              "      <td>economy,jobs</td>\n",
              "      <td>charlie-crist</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Florida</td>\n",
              "      <td>democrat</td>\n",
              "      <td>15.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>20.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>an interview on CNN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           0            1   ...    12                   13\n",
              "0   2635.json        false  ...   0.0             a mailer\n",
              "1  10540.json    half-true  ...   0.0      a floor speech.\n",
              "2    324.json  mostly-true  ...   9.0               Denver\n",
              "3   1123.json        false  ...  44.0       a news release\n",
              "4   9028.json    half-true  ...   2.0  an interview on CNN\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dz06ZOovuWr9",
        "outputId": "305b250d-3f4b-4efd-dc3e-cbfae2c05160",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 340
        }
      },
      "source": [
        "liar_test_df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>11972.json</td>\n",
              "      <td>true</td>\n",
              "      <td>Building a wall on the U.S.-Mexico border will...</td>\n",
              "      <td>immigration</td>\n",
              "      <td>rick-perry</td>\n",
              "      <td>Governor</td>\n",
              "      <td>Texas</td>\n",
              "      <td>republican</td>\n",
              "      <td>30</td>\n",
              "      <td>30</td>\n",
              "      <td>42</td>\n",
              "      <td>23</td>\n",
              "      <td>18</td>\n",
              "      <td>Radio interview</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>11685.json</td>\n",
              "      <td>false</td>\n",
              "      <td>Wisconsin is on pace to double the number of l...</td>\n",
              "      <td>jobs</td>\n",
              "      <td>katrina-shankland</td>\n",
              "      <td>State representative</td>\n",
              "      <td>Wisconsin</td>\n",
              "      <td>democrat</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>a news conference</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>11096.json</td>\n",
              "      <td>false</td>\n",
              "      <td>Says John McCain has done nothing to help the ...</td>\n",
              "      <td>military,veterans,voting-record</td>\n",
              "      <td>donald-trump</td>\n",
              "      <td>President-Elect</td>\n",
              "      <td>New York</td>\n",
              "      <td>republican</td>\n",
              "      <td>63</td>\n",
              "      <td>114</td>\n",
              "      <td>51</td>\n",
              "      <td>37</td>\n",
              "      <td>61</td>\n",
              "      <td>comments on ABC's This Week.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5209.json</td>\n",
              "      <td>half-true</td>\n",
              "      <td>Suzanne Bonamici supports a plan that will cut...</td>\n",
              "      <td>medicare,message-machine-2012,campaign-adverti...</td>\n",
              "      <td>rob-cornilles</td>\n",
              "      <td>consultant</td>\n",
              "      <td>Oregon</td>\n",
              "      <td>republican</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>a radio show</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9524.json</td>\n",
              "      <td>pants-fire</td>\n",
              "      <td>When asked by a reporter whether hes at the ce...</td>\n",
              "      <td>campaign-finance,legal-issues,campaign-adverti...</td>\n",
              "      <td>state-democratic-party-wisconsin</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Wisconsin</td>\n",
              "      <td>democrat</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>a web video</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           0           1   ...  12                            13\n",
              "0  11972.json        true  ...  18               Radio interview\n",
              "1  11685.json       false  ...   0             a news conference\n",
              "2  11096.json       false  ...  61  comments on ABC's This Week.\n",
              "3   5209.json   half-true  ...   1                  a radio show\n",
              "4   9524.json  pants-fire  ...   7                   a web video\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4CRokvtqucfe",
        "outputId": "8a8b0c54-8db6-4318-81dc-ac503bec5289",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "liar_valid_df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>12134.json</td>\n",
              "      <td>barely-true</td>\n",
              "      <td>We have less Americans working now than in the...</td>\n",
              "      <td>economy,jobs</td>\n",
              "      <td>vicky-hartzler</td>\n",
              "      <td>U.S. Representative</td>\n",
              "      <td>Missouri</td>\n",
              "      <td>republican</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>an interview with ABC17 News</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>238.json</td>\n",
              "      <td>pants-fire</td>\n",
              "      <td>When Obama was sworn into office, he DID NOT u...</td>\n",
              "      <td>obama-birth-certificate,religion</td>\n",
              "      <td>chain-email</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>none</td>\n",
              "      <td>11</td>\n",
              "      <td>43</td>\n",
              "      <td>8</td>\n",
              "      <td>5</td>\n",
              "      <td>105</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7891.json</td>\n",
              "      <td>false</td>\n",
              "      <td>Says Having organizations parading as being so...</td>\n",
              "      <td>campaign-finance,congress,taxes</td>\n",
              "      <td>earl-blumenauer</td>\n",
              "      <td>U.S. representative</td>\n",
              "      <td>Oregon</td>\n",
              "      <td>democrat</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>a U.S. Ways and Means hearing</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8169.json</td>\n",
              "      <td>half-true</td>\n",
              "      <td>Says nearly half of Oregons children are poor.</td>\n",
              "      <td>poverty</td>\n",
              "      <td>jim-francesconi</td>\n",
              "      <td>Member of the State Board of Higher Education</td>\n",
              "      <td>Oregon</td>\n",
              "      <td>none</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>an opinion article</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>929.json</td>\n",
              "      <td>half-true</td>\n",
              "      <td>On attacks by Republicans that various program...</td>\n",
              "      <td>economy,stimulus</td>\n",
              "      <td>barack-obama</td>\n",
              "      <td>President</td>\n",
              "      <td>Illinois</td>\n",
              "      <td>democrat</td>\n",
              "      <td>70</td>\n",
              "      <td>71</td>\n",
              "      <td>160</td>\n",
              "      <td>163</td>\n",
              "      <td>9</td>\n",
              "      <td>interview with CBS News</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           0            1   ...   12                             13\n",
              "0  12134.json  barely-true  ...    0   an interview with ABC17 News\n",
              "1    238.json   pants-fire  ...  105                            NaN\n",
              "2   7891.json        false  ...    0  a U.S. Ways and Means hearing\n",
              "3   8169.json    half-true  ...    0             an opinion article\n",
              "4    929.json    half-true  ...    9        interview with CBS News\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUADcjuHmATr"
      },
      "source": [
        "Renaming the columns"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JGPtqpeiNiBZ"
      },
      "source": [
        "liar_test_df = liar_test_df.rename(columns={0: \"ID\", 1: \"target\", 2: \"text\", 3: \"subjects\", 4: \"speaker\", 5: \"job\", 6: \"state\", 7: \"party\", 8: \"barely_true_count\", 9: \"false_count\", 10: \"half_true_count\", 11: \"mostly_true_count\", 12: \"pants_on_fire_count\", 13: \"target_names\"})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ebpYF97ZP6Y-"
      },
      "source": [
        "liar_train_df = liar_train_df.rename(columns={0: \"ID\", 1: \"target\", 2: \"text\", 3: \"subjects\", 4: \"speaker\", 5: \"job\", 6: \"state\", 7: \"party\", 8: \"barely_true_count\", 9: \"false_count\", 10: \"half_true_count\", 11: \"mostly_true_count\", 12: \"pants_on_fire_count\", 13: \"target_names\"})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r_j0siXAM74G"
      },
      "source": [
        "#Combining the text and speaker columns into one statement\n",
        "liar_test_new = liar_test_df.copy()\n",
        "liar_test_new['data']=liar_test_df['text']+'. '+liar_test_df['speaker']\n",
        "liar_test_new = liar_test_new.drop(columns=['ID', 'text','subjects','speaker','job','state','party','barely_true_count','false_count','half_true_count','mostly_true_count','pants_on_fire_count'])\n",
        "\n",
        "liar_train_new = liar_train_df.copy()\n",
        "liar_train_new['data']=liar_train_df['text']+'. '+liar_train_df['speaker']\n",
        "liar_train_new = liar_train_new.drop(columns=['ID', 'text','subjects','speaker','job','state','party','barely_true_count','false_count','half_true_count','mostly_true_count','pants_on_fire_count'])\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "43CPSCZLuIIW"
      },
      "source": [
        "def liar_truthdist_subj(subj=None, ax=None, saveplot=True, longtitle=True):\n",
        "    \"\"\"\n",
        "    Plots the truth distribution of the LIAR dataset, optionally by subject\n",
        "    Return the dataframe representing the plot, and the seaborn plot obj\n",
        "    \"\"\"   \n",
        "    if not ax:\n",
        "        fig, ax = plt.subplots()\n",
        "\n",
        "    if subj == None:\n",
        "        subj_filter = liar_train_new['subjects'].apply(lambda x: True)\n",
        "        plt_title = 'distribution of truth values'\n",
        "        filename = './image/liar_truthvals_dist.png'\n",
        "    else:\n",
        "        subj_filter = liar_train_new['subject_list'].apply(lambda entry: subj in entry)\n",
        "        if longtitle:\n",
        "            plt_title = '\"' + subj + '\": distribution of truth values'\n",
        "        else:\n",
        "            plt_title = subj\n",
        "        filename = './image/liar_truthvals_dist_' + subj + '.png'\n",
        "    \n",
        "    subj_truthdist = pd.DataFrame({'label': liar_label, 'count': label_count} \n",
        "                                  for liar_label, label_count \n",
        "                                  in zip(liar_label_order, liar_train[subj_filter]['label'].value_counts()))\n",
        "    \n",
        "    sns.barplot(data=subj_truthdist, y='label', x='count', palette='coolwarm', ax=ax).set(title=plt_title, xlabel='count')\n",
        "    plt.tight_layout()\n",
        "\n",
        "    if saveplot:\n",
        "        plt.savefig(filename)\n",
        "\n",
        "    return subj_truthdist, ax"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i2pVQrOVTijr",
        "outputId": "ffe8476a-9a01-4af4-9c7f-4c74c32c399e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "for i in range(len(liar_test_new)):\n",
        "    if liar_test_new.loc[i, 'target'] == \"true\": #REAL => 0\n",
        "        liar_test_new.loc[i, 'target'] = 0\n",
        "    elif liar_test_new.loc[i, 'target'] == \"mostly-true\": #mostly-true => 1\n",
        "        liar_test_new.loc[i, 'target'] = 1\n",
        "    elif liar_test_new.loc[i, 'target'] == \"barely-true\": #barely-true => 2\n",
        "        liar_test_new.loc[i, 'target'] = 2\n",
        "    elif liar_test_new.loc[i, 'target'] == \"half-true\": #half-true => 3\n",
        "        liar_test_new.loc[i, 'target'] = 3\n",
        "    elif liar_test_new.loc[i, 'target'] == \"false\": #false => 4\n",
        "        liar_test_new.loc[i, 'target'] = 4\n",
        "    elif liar_test_new.loc[i, 'target'] == \"pants-fire\": #pants-fire => 5\n",
        "        liar_test_new.loc[i, 'target'] = 5            \n",
        "    if liar_test_new.loc[i, 'data'] == \"\":\n",
        "        liar_test_new = liar_test_new.drop([i])\n",
        "liar_test_new.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>target_names</th>\n",
              "      <th>data</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>Radio interview</td>\n",
              "      <td>Building a wall on the U.S.-Mexico border will...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>a news conference</td>\n",
              "      <td>Wisconsin is on pace to double the number of l...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4</td>\n",
              "      <td>comments on ABC's This Week.</td>\n",
              "      <td>Says John McCain has done nothing to help the ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>a radio show</td>\n",
              "      <td>Suzanne Bonamici supports a plan that will cut...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>a web video</td>\n",
              "      <td>When asked by a reporter whether hes at the ce...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  target  ...                                               data\n",
              "0      0  ...  Building a wall on the U.S.-Mexico border will...\n",
              "1      4  ...  Wisconsin is on pace to double the number of l...\n",
              "2      4  ...  Says John McCain has done nothing to help the ...\n",
              "3      3  ...  Suzanne Bonamici supports a plan that will cut...\n",
              "4      5  ...  When asked by a reporter whether hes at the ce...\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t74yVEF5U7jv",
        "outputId": "32cb446e-5944-4ac7-e074-1c4b4b23875a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "for i in range(len(liar_train_new)):\n",
        "    if liar_train_new.loc[i, 'target'] == \"true\": #true => 0\n",
        "        liar_train_new.loc[i, 'target'] = 0\n",
        "    elif liar_train_new.loc[i, 'target'] == \"mostly-true\": #mostly true => 1\n",
        "        liar_train_new.loc[i, 'target'] = 1\n",
        "    elif liar_train_new.loc[i, 'target'] == \"barely-true\": #barely-true => 2\n",
        "        liar_train_new.loc[i, 'target'] = 2\n",
        "    elif liar_train_new.loc[i, 'target'] == \"half-true\": #half-true => 3\n",
        "        liar_train_new.loc[i, 'target'] = 3\n",
        "    elif liar_train_new.loc[i, 'target'] == \"false\": #false => 4\n",
        "        liar_train_new.loc[i, 'target'] = 4\n",
        "    elif liar_train_new.loc[i, 'target'] == \"pants-fire\": #pants-fire => 5\n",
        "        liar_train_new.loc[i, 'target'] = 5            \n",
        "    if liar_train_new.loc[i, 'data'] == \"\":\n",
        "        liar_train_new = liar_train_new.drop([i])\n",
        "liar_train_new.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>target_names</th>\n",
              "      <th>data</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4</td>\n",
              "      <td>a mailer</td>\n",
              "      <td>Says the Annies List political group supports ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>a floor speech.</td>\n",
              "      <td>When did the decline of coal start? It started...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>Denver</td>\n",
              "      <td>Hillary Clinton agrees with John McCain \"by vo...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>a news release</td>\n",
              "      <td>Health care reform legislation is likely to ma...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3</td>\n",
              "      <td>an interview on CNN</td>\n",
              "      <td>The economic turnaround started at the end of ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  target  ...                                               data\n",
              "0      4  ...  Says the Annies List political group supports ...\n",
              "1      3  ...  When did the decline of coal start? It started...\n",
              "2      1  ...  Hillary Clinton agrees with John McCain \"by vo...\n",
              "3      4  ...  Health care reform legislation is likely to ma...\n",
              "4      3  ...  The economic turnaround started at the end of ...\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FA4KaXKm90V"
      },
      "source": [
        "Dropping null value rows from the dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GI5Yxnrpp1z-"
      },
      "source": [
        "liar_train_new = liar_train_new.dropna()\n",
        "liar_test_new = liar_test_new.dropna()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B_bQTSFxLC0U"
      },
      "source": [
        "import sklearn.utils\n",
        "from sklearn.utils import Bunch\n",
        "tlabels = liar_test_new['target'].to_numpy()\n",
        "#labels = labels.to_numpy() \n",
        "type(tlabels)\n",
        "tdata = liar_test_new['data'].tolist()\n",
        "type(tdata)\n",
        "tdata_names = liar_test_new['target_names'].tolist()\n",
        "test = Bunch(target=tlabels, data=tdata, target_names=tdata_names)\n",
        "test.target = test.target.astype(int)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sTdFKVvU_siC"
      },
      "source": [
        "import sklearn.utils\n",
        "from sklearn.utils import Bunch\n",
        "labels = liar_train_new['target'].to_numpy()\n",
        "#labels = labels.to_numpy() \n",
        "type(labels)\n",
        "data = liar_train_new['data'].tolist()\n",
        "type(data)\n",
        "data_names = liar_train_new['target_names'].tolist()\n",
        "train = Bunch(target=labels, data=data, target_names=data_names)\n",
        "train.target = train.target.astype(int)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Njp6-5b5NwiE",
        "outputId": "4a854014-6c9b-4602-bb25-11725bbb50f7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import textwrap\n",
        "import random\n",
        "\n",
        "# Wrap text to 80 characters.\n",
        "wrapper = textwrap.TextWrapper(width=80) \n",
        "\n",
        "# Randomly choose some examples.\n",
        "for i in range(10):\n",
        "    \n",
        "    # Choose a random sample by index.\n",
        "    j = random.choice(range(len(train.data)))\n",
        "    print(j)\n",
        "    # Get the text as 'x' and the label integer as 'y'.\n",
        "    x = train.data[j]\n",
        "    y = train.target[j]\n",
        "\n",
        "    # Print out the name of the category and the text.\n",
        "    print('')\n",
        "    print('========', train.target_names[y], '========')\n",
        "    print(wrapper.fill(x))\n",
        "    print('')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "8727\n",
            "\n",
            "======== a news release ========\n",
            "Jimmy Langevin is having a fundraiser . . . to retire the debt from his campaign\n",
            "for secretary of state, which was 12 years ago.. john-matson\n",
            "\n",
            "4161\n",
            "\n",
            "======== a mailer ========\n",
            "The property appraisers budget is $1 million less today than when I took office\n",
            "16 years ago.. rob-turner\n",
            "\n",
            "8669\n",
            "\n",
            "======== Denver ========\n",
            "Under Scott Walker, Wisconsin moved from 16th to third in percentage of people\n",
            "with student loan debt.. scot-ross\n",
            "\n",
            "8873\n",
            "\n",
            "======== a mailer ========\n",
            "Says McCain once said that on \"the most important issues of our day, I've been\n",
            "totally in agreement and support of President Bush.\". joe-biden\n",
            "\n",
            "4531\n",
            "\n",
            "======== a mailer ========\n",
            "America is No. 1one in oil and gas.. barack-obama\n",
            "\n",
            "6172\n",
            "\n",
            "======== Denver ========\n",
            "Georgia lawmakers are now considering passing the most extreme gun bill in\n",
            "America.. americans-responsible-solutions\n",
            "\n",
            "7676\n",
            "\n",
            "======== an interview on CNN ========\n",
            "The Clintons now charge the Secret Service $10,000 monthly rent for the use of\n",
            "(a) Secret Service residence and that rent is just about equal to their mortgage\n",
            "payment.. chain-email\n",
            "\n",
            "2463\n",
            "\n",
            "======== a news release ========\n",
            "Says Scott Walker cut funding to local governments in Wisconsin, contributing to\n",
            "the second-largest increase in violent crime in the Midwest.. mary-burke\n",
            "\n",
            "6748\n",
            "\n",
            "======== a mailer ========\n",
            "Says President Obama has fired more cruise missiles than all other Peace Prize\n",
            "winners combined.. yard-sign\n",
            "\n",
            "8821\n",
            "\n",
            "======== a news release ========\n",
            "We buy 35 percent of all Chinese exports.. sherrod-brown\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zU5aPNzzLjeq"
      },
      "source": [
        "Let's look at the classes to see how balanced they are."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fcBinIxWLjer",
        "outputId": "91f00a3b-b09d-4dbf-85d3-d659114d2bd5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 443
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (10,5)\n",
        "\n",
        "\n",
        "# Plot the number of tokens of each length.\n",
        "sns.countplot(train.target)\n",
        "plt.title('Class Distribution')\n",
        "plt.xlabel('Category')\n",
        "plt.ylabel('# of Training Samples')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
            "  FutureWarning\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0, 0.5, '# of Training Samples')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXiNd+L//9dJZJGILJoQqb0kaayxlDAdxCBKpUUtFWstrbZDBw3VuT7T6iiNDhVLrSWDplSkqnbGd2qtpVKklNoVxxoJSZDz+8MvZ3qaxQk5OSTPx3W5mvO+3+c+r3Nfmr56rwaTyWQSAAAASjQHewcAAACA/VEKAQAAQCkEAAAApRAAAACiFAIAAECUQgAAAIhSCOAxFBUVpdatW9s7RqE7e/asAgMDNW3aNLt+pj1y2PNzAVinlL0DACgZbt++rfj4eK1fv17Hjh1TWlqaPD09FRISooiICL344osqVerJ+ZUUGBho/tlgMKh06dLy8fFRUFCQWrdurRdeeEGurq6F9nkrVqxQSkqK+vXrV2jrtIWzZ88qISFBbdq0UXBwsL3jACiAJ+c3MIAn1qlTpzR48GCdPHlSYWFhGjx4sLy9vXXlyhXt2LFDY8aM0bFjxzR69Gh7Ry2Q4OBg9e/fX5KUnp6u8+fPa9u2bRo7dqxmzZqladOmKSgoyDw/ICBASUlJcnR0LPBnJSQk6Ny5cwUuhY/ymQ/j3Llzio2NVUBAQI5SWNRZABQMpRCATaWnp2vIkCE6e/aspk2bprZt21osHzx4sJKSkvTTTz/ZKeHDK1++vDp37mwxNmLECK1Zs0ajRo3Sa6+9ptWrV8vT01PS/T2KLi4uRZItNTVVZcqUKdLPfJDHKQuAnDinEIBNLVu2TCdOnFD//v1zFMJsdevW1auvvprvepKSkhQdHa127dqpXr16atCggXr06KENGzbkmPvbb79pzJgxatWqlWrXrq1mzZqpR48eSkhIMM/JysrSF198oU6dOqlBgwYKDQ1Vu3btNHbsWN25c+eRvnNERIQGDhwoo9GoxYsXm8fzOqdu5cqV6tq1qxo1aqT69esrPDxcf/vb33T16lVJUuvWrbV7926dO3dOgYGB5j+7du2S9L9zMM+cOaO3335bTZo0UcOGDfP9zGzffvutOnXqpDp16qhly5aaNm2a7t69azEnr3M8/7juFStWqE+fPpKkMWPGmHNGRUXlm+Xu3buaPXu2OnTooDp16ui5557TsGHDdOTIkTw/b8uWLerSpYvq1KmjFi1aaOLEiTlyAygY9hQCsKl169ZJkrp37/5I69mwYYN+/fVXtW/fXgEBAbp+/boSEhL05ptvKiYmRp06dZJ0v2D0799fFy9eVK9evVS1alWlpqbqyJEj2rNnj1566SVJ0syZM/XZZ5+pVatW6tGjhxwdHXX27Flt3rxZmZmZcnJyeqS83bp106xZs7R161a98cYbec5buXKl3n33XTVq1Ehvv/22XF1d9dtvv2nr1q26cuWKfHx8NHbsWE2ePFnXrl3TmDFjzO+tUaOG+ee0tDT17t1boaGhGj58uLlQ5mfz5s06c+aMXn31VT311FPavHmzYmNjdf78eU2YMKHA37lx48YaOnSoZs2ape7du5uL6VNPPZXv+0aOHKk1a9aoefPm6tmzpy5fvqzFixerR48eWrx4sZ599lmL+Vu3btWSJUvUo0cPdenSRZs2bdL8+fPl6empoUOHFjg3gPsohQBs6pdfflGZMmVUqVKlR1rP66+/rr/97W8WY1FRUYqMjNTMmTPNpfDYsWM6ceKERo4cqUGDBuW5vo0bN6pGjRqaNWuWxfjIkSMfKWe2p59+Wu7u7jp58mS+8zZu3Ch3d3ctXLjQ4kKbv/71r+af27Rpo4ULFyojIyPH4eps169f19ChQzVixAirM/78889avny5QkJCJEm9e/fWm2++qRUrVqh79+6qX7++1euSpEqVKiksLEyzZs1S/fr188z6e9u2bdOaNWsUERGhf/3rXzIYDJLu7219+eWXNX78eC1ZssTiPceOHdO3336rp59+WpLUs2dPderUSf/+978phcAj4PAxAJtKTU2Vu7v7I6/Hzc3N/PPt27d17do13b59W02bNtXx48eVmpoqSfLw8JAk7dq1S1euXMlzfWXKlNHFixe1Z8+eR86W32dk58qLh4eH0tPT9Z///Ecmk+mRPm/gwIEFmh8WFmYuhNL9c/5ee+01Scr1sLwtZH/O0KFDzYVQkoKCgtSqVSvt3bs3x17P8PBwcyGU7ud+7rnnZDQalZaWViS5geKIPYUAbKpMmTKF8h/qK1euaMqUKdq0aVOuZS8lJUVlypRRQECAhg4dqtmzZ6tFixYKDg5W06ZN1b59e9WtW9c8/5133tGwYcP06quvys/PT02aNFHLli3Vrl07OTs7P3Je6X8Xe+RnyJAh+uGHHzRs2DB5eXmpSZMmev755xUREfHA9/6ej4+PypYtW6B8vz/8nO2ZZ56RJJ05c6ZA63pYZ8+elYODQ55ZNm7cqLNnz8rHx8c8ntteZy8vL0n395gWxv+EACURewoB2FTNmjWVmpr6SCXDZDJpwIABSkhIUGRkpP71r39p7ty5WrBggTp27Cjp/oUj2UaMGKH169dr7NixqlSpkpYvX65u3brpk08+Mc9p0KCBNmzYoM8++0x/+ctf9PPPP2vkyJGKjIzU9evXH/4L///Onj2rtLQ0VatWLd95VatW1XfffafZs2frpZde0rlz5zRu3DhFRETo9OnTVn9e6dKlHzVygd27d6/IP1NSvre0edS9rUBJRikEYFPZVxwvW7bsoddx5MgR/fzzzxo8eLBGjx6tDh066E9/+pPCwsIsyuDvVapUSVFRUZo6dar++9//qnHjxpo7d67FXkZ3d3e1a9dOf//737V69Wr9/e9/1/Hjx7V8+fKHzpot+/v++c9/fuBcZ2dn/fnPf1Z0dLRWrFih2bNn69KlS1qwYMEj58jP8ePHc4wdO3ZMkuXeOC8vr1yLcm5F//eHgK1RqVIlZWVl5Zole+z3h4oB2A6lEIBNdevWTdWqVdP8+fO1cePGXOccPHjQ4tYtf+TgcP9X1R/3Ah09ejTHuW83b97McUsZFxcXVa9eXZJ048YNScr16tzs8+uy5zysNWvWaN68efLz83vgrXZyy5F9te3vc7i7u+vGjRuFuids+/btOnTokPm1yWTS3LlzJd2/uCVb1apVlZaWpqSkJPNY9i19/ij73E9rt2H258yePdviux09elSbN29Ww4YNLQ4dA7AdzikEYFOlS5fW559/rsGDB2vYsGFq0aKFwsLC5OXlpatXr2rXrl36/vvvzRc45KZGjRqqWbOm5s6dq/T0dFWrVk0nTpxQfHy8atWqZVFsdu3apffff19t27ZVtWrV5O7uroMHD2r58uWqV6+euRx26NBB9evXV926deXn5yej0aivvvpKTk5OeuGFF6z6bhcvXlRiYqIkKSMjw/xEk6SkJFWpUkXTpk174Hl+AwcOlIeHhxo1aiR/f3+lpKQoISFBBoPB4urdevXqacuWLfrggw/UoEEDOTo6qmnTpipXrpxVWXMTFBSkvn376tVXX5Wvr682bdqk7du3q3PnzmrQoIF53iuvvKIFCxZo2LBh6tOnj5ycnLRu3bpcDx8/88wzcnd315IlS+Tq6qqyZcvKx8dHzZo1yzVD8+bNFRERodWrV+vGjRtq1aqVjEajlixZIhcXF40bN+6hvx+AgqEUArC5KlWqaOXKlYqPj9e6des0a9Ys3bp1S56enqpdu7Y+/vhj8y1lcuPo6KjPP/9cEydOVEJCgm7fvq2aNWtq4sSJ+vnnny1KYWBgoP7yl79o9+7dWrVqlbKysuTv768hQ4ZowIAB5nkDBgzQ1q1bFRcXp5s3b6pcuXKqV6+ehgwZYvFouvwkJyebH83n5uYmb29vBQUF6aOPPlLHjh2tevZxz549tWbNGsXHx+vGjRvy8vJScHCwxo0bp6ZNm5rn9evXT2fOnNG6dev05ZdfKisrS4sWLXqkUti6dWtVq1ZNn3/+uU6cOKFy5crpjTfeyHFfxUqVKmn69On69NNPNXXqVHl5ealz587q0qWLIiIiLOa6urrqX//6l6ZMmaJ//vOfyszMVJMmTfIshZIUExOjZ599VgkJCfr444/l5uamxo0b669//avFM6YB2JbBxFm5AAAAJR7nFAIAAIBSCAAAAEohAAAARCkEAACAKIUAAAAQpRAAAADiPoWF5tq1NGVlcXcfAADw+HJwMMjb2z3XZZTCQpKVZaIUAgCAJxaHjwEAAEApBAAAAKUQAAAAohQCAABAlEIAAACIUggAAABRCgEAACBKIQAAAEQpBAAAgHiiCQA80bw8nOXk6mLvGI+9O+kZun4z094xgMcapRAAnmBOri76rk9/e8d47HVYtECiFAL54vAxAAAAKIUAAACgFAIAAECUQgAAAIhSCAAAANmxFCYlJekf//iHOnTooPr166tly5YaMWKETp06lWPuvn371LNnT9WrV0/NmzfX+PHjdfv27RzzMjMz9cknn6hFixaqW7euXnnlFe3YsSPXz7d2nQAAACWB3Urh3LlztWHDBoWFhem9997TK6+8ot27dysyMlLHjx83z0tOTla/fv2UkZGh6Ohode3aVfHx8RoxYkSOdUZHR2vhwoV68cUX9d5778nBwUGDBg3S/v37LeYVZJ0AAAAlgd3uU9ivXz/FxMTI2dnZPNahQwd16tRJc+bM0ccffyxJ+vTTT+Xl5aW4uDi5u7tLkp5++mmNGzdOO3bsULNmzSTd3/O4evVqjRkzRv369ZMkRUZGqmPHjoqJidHixYvNn2PtOgEAAEoKu+0pDA0NtSiEklS1alXVrFnTvKcwNTVV27dvV2RkpLm8SVLnzp3l5uamNWvWmMfWrl0rJycndevWzTzm4uKirl27au/evbp06VKB1wkAAFBSPFYXmphMJl2+fFne3t6SpCNHjuju3buqXbu2xTxnZ2cFBwcrOTnZPJacnKxq1apZFD1Jqlu3rkwmk3luQdYJAABQUjxWj7n75ptvdPHiRfO5fUajUZLk6+ubY66vr69+/PFH82uj0ajy5cvnOk+SeU9hQdZZEOXKlXmo9wEAioavr4e9IwCPtcemFB4/flwffPCBGjZsqM6dO0uS0tPTJSnHYWbp/qHh7OXZc52cnHKdJ0kZGRkFXmdBXLmSqqws00O9FwAeFkXHekbjTXtHAOzOwcGQ546sx+LwsdFo1JAhQ+Tp6ampU6fKweF+LFdXV0n3bzXzRxkZGebl2XPv3LmT6zzpf+WwIOsEAAAoKey+p/DmzZsaNGiQbt68qaVLl1oc1s3+OfuQ7+8ZjUb5+flZzM0+RPzHeZLMcwuyTgAAgJLCrqUwIyNDQ4cO1cmTJ/XFF1+oevXqFstr1aqlUqVK6eDBg2rbtq15PDMzU8nJyerUqZN5LCgoSHFxcUpLS7O42OTAgQPm5QVdJwAAf+RZtrScXey+T+Wxl5lxVzdSeCjEk8Ruf6vv3bun4cOH68cff9SMGTNUv379HHM8PDzUrFkzJSYmasiQIeayl5iYqFu3bql9+/bmue3bt9f8+fO1bNky830KMzMztWLFCoWGhpovQinIOgEA+CNnl1L653vL7R3jsTf2o672joACslsp/Pjjj7V582a1atVK169fV2JionmZu7u72rRpI0kaMWKEevTooaioKHXr1k0XLlzQggUL9PzzzyssLMz8nnr16ql9+/aKiYmR0WhU5cqVlZCQoPPnz2vChAkWn23tOgEAAEoKu5XCn3/+WZK0ZcsWbdmyxWJZQECAuRSGhIRowYIFiomJ0YQJE1SmTBm98soreuedd3Ksc9KkSZoyZYoSExN148YNBQYGavbs2WrYsKHFvIKsE08ub09nlXJ2sXeMx97dzAxdu5HzwisAQMliMJlM3EelEHBLmsePr6+H9k56zd4xHnsNR8/lVh1PMF9fD33Xp7+9Yzz2OixaUGh/z319PTh8bIWxH3Xld8tj6LG/JQ0AAADsi1IIAAAASiEAAAAohQAAABClEAAAACqEUnjw4EFt27bN/IxhAAAAPHmsvk/hvHnz9MMPP2jWrFnmsb/97W/67rvvJEmVKlXSkiVL9NRTTxV+SgAAANiU1XsKV69eLX9/f/PrHTt2aPXq1erQoYNGjBgho9GouXPn2iQkAAAAbMvqPYXnzp3Tyy+/bH69adMm+fr6KiYmRgaDQdeuXdPmzZsVHR1tk6AAAACwHav3FN6+fVsuLv97ZNjOnTsVFhYmg8EgSapRo4YuXrxY+AkBAABgc1aXwvLly+vo0aOS7u81PHbsmBo3bmxenpKSImdn58JPCAAAAJuz+vBxq1attGTJEt27d08HDhyQs7OzWrZsaV7+yy+/KCAgwBYZAQAAYGNWl8Jhw4bpyJEjWrJkiZydnTV27Fjzlcbp6enasGGDunbtarOgAAAAsB2rS6Gnp6cWLlyo1NRUubi4yMnJyWL5v//9b1WoUKHQAwIAAMD2rC6F2cqUKZNjzNXVVUFBQYUSCAAAAEWvQE80SU1NVWxsrHr27Km2bdtq//79kqSrV68qNjZWx48ft0lIAAAA2JbVewqvXr2qnj176uzZs6pcubLOnDmj9PR0SZKPj49WrlypmzdvasyYMTYLCwAAANuwuhROmTJFly9f1ldffSV/f3+FhYVZLA8PD9eOHTsKPSAAAABsz+rDx1u2bFGvXr0UEhJivmH171WqVEkXLlwo1HAAAAAoGlaXwmvXrqly5cp5LjcYDMrIyCiUUAAAAChaVpdCX19fnTlzJs/lycnJ8vf3L5RQAAAAKFpWl8Lnn39ey5cv16VLl3IsO3DggFauXKnw8PBCDQcAAICiYfWFJm+++aY2b96sl156Sa1bt5bBYNDKlSu1bNkyrV+/Xn5+fho0aJAtswIAAMBGCnT4+KuvvlLdunX19ddfy2QyKTExUWvWrFGLFi20ZMkSeXl52TIrAAAAbKRATzTx9/fXzJkzlZqaql9//VWSVLlyZcogAADAE67Aj7mT7j/qrm7duoWdBQAAAHZSoMfcAQAAoHjKc09hUFBQrjepzo/BYNDhw4cfORQAAACKVp6lMDIyssClEAAAAE+mPEvhxx9/XJQ5AAAAYEecUwgAAICCX3188eJFbdmyxfzIu0qVKqlVq1YqX758oYcDAABA0ShQKZw+fbpmzpype/fuyWQymcfHjx+voUOH6s033yz0gAAAALA9q0vhv//9b02bNk116tRRv379VKNGDUnSsWPH9MUXX2j69Ony8vJS7969bRYWAAAAtmF1KYyLi1PdunW1ZMkSlSr1v7cFBQWpXbt26tmzp+Li4iiFefAo6ypXFyd7x3jspWfc0c2UdHvHAACgxLG6FP7222/q1auXRSHM5uTkpE6dOmny5MmFGq44cXVxUq/Ri+0d47G3ZNKruilKIQAARc3qUujv76+0tLQ8l6elpcnf379QQgF4MpX1dJGLs7O9Yzz2MjIzlXIjw94xAMCC1aWwd+/emjt3rrp27So/Pz+LZRcvXtSXX36pwYMHF3pAAE8OF2dn9VvwV3vHeOx90X+qJEohgMeL1aXQw8ND5cqVU0REhF588UVVr15dknT8+HGtWrVKVatWVZkyZbRy5UqL90VGRhZuYgAAABQ6q0thdHS0+eelS5fmWH7o0CGLOdL9ZyFTCgEAAB5/VpfCRYsW2TIHAAAA7MjqUtikSRNb5gAAAIAd8exjAAAAFOwxd7du3dK3336rkydP6vr16xaPupPun0P4z3/+s1ADAgAAwPasLoX79u3T66+/rhs3buQ5h1IIAADwZLK6FI4fP14ODg6aMWOGGjVqpLJly9oyFwAAAIqQ1aXw2LFjevvtt9W6dWtb5gEAAIAdWH2hia+vb67PPQYAAMCTz+pS2K1bN3377be6d+9eoX34pUuXFBMTo6ioKDVo0ECBgYHatWtXjnmtW7dWYGBgjj8xMTE55qakpOj9999X06ZNVb9+ffXp00fJycm5fv6mTZv00ksvqU6dOmrZsqViY2N19+7dQvt+AAAATwqrd/0NGTJEly5dUvfu3dWzZ08FBATI0dExx7zGjRtb/eEnTpzQnDlzVKVKFQUGBmr//v15zg0JCVHfvn0txmrVqmXxOisrS4MHD9bRo0c1YMAAeXt7a8mSJYqKitKKFStUuXJl89ytW7dq2LBhatq0qd5//30dPXpU06dP17Vr1/T+++9b/R0AAACKA6tLYXp6uq5fv65Dhw5p3LhxOZabTCYZDIY898rlJiQkRDt37pS3t7c2btyoYcOG5Tm3QoUK6ty5c77rW7t2rfbv36/p06erTZs2kqSIiAi1a9dOsbGxmjRpknnupEmT9Oyzz2revHnmcuvu7q7Zs2crKipKVatWtfp7AAAAPOmsLoUffPCB1qxZozZt2qhhw4by9PR85A8vU6ZMgeZnZmbq3r17Kl26dK7L161bJz8/P4WHh5vHfHx8FBERoW+//VZ37tyRk5OTjh07pmPHjumDDz6w2NvZq1cvzZo1S+vXr9fgwYMf7ksBAAA8gawuhZs2bVKXLl00fvx4W+bJ07Zt21S/fn3du3dPlSpV0qBBg9S9e3eLOcnJyQoJCZHBYLAYr1OnjuLj43X69GnVqFFDhw8fliTVrl3bYl758uVVoUIF83IAAICSwupSaDKZVKdOHVtmyVOtWrXUqFEjVa1aVdeuXdNXX32lv//977px44bFHj2j0aimTZvmeL+fn5+k+xe21KhRQ0ajUdL9K6r/yNfXV5cuXSpwxnLlCrbXE3nz9fWwd4QSh21e9NjmRY9tXvTY5k8Wq0thkyZNdODAgRx754rCrFmzLF6//PLL6tWrl2bMmKGePXvKw+P+X7r09HQ5OzvneH/2WHp6usU/c5vr4uKi27dvFzjjlSupysoy5bmcfzGsZzTeLJT1sM2txzYvemzzosc2L3qFtc1ReBwcDHnuyLL6ljRjx47V7t27tWDBAmVmZhZauIfh6Oiovn376vbt2xZXLLu6uuaaLXvM1dXV4p+5zc3IyDAvBwAAKCms3lPYp08f3b59W5MmTdLkyZPl6+srBwfLTmkwGLRx48ZCD5mbChUqSJLFs5jzOvSbPZZ9GDn7sLHRaDSPZTMajWrQoIFNMgMAADyurC6FFStWtGWOAjtz5oyk+1cXZwsKCtL+/fvNt8fJlpSUJDc3N/N9CoODgyVJBw8eVEhIiHnexYsXdeHCBfNyAACAksLqUhgXF2fLHHm6fv26ypYta7FXMiMjQ/PmzZO7u7vq169vHm/fvr3WrVunTZs2me9TePXqVa1du1bh4eFycnKSJNWsWVPVq1dXfHy8unbtar4tzdKlS+Xg4KC2bdsW4TcEAACwP7s/zHjGjBmSpOPHj0uSEhMTtXfvXpUtW1a9e/fW5s2bNWvWLLVr104BAQG6fv26EhISdPLkSf3f//2f3N3dzetq166d6tevr9GjR5ufaLJ06VJlZWXprbfesvjc0aNH6/XXX9fAgQPVoUMHHT16VIsXL1b37t1VrVq1otsAAAAAjwG7l8KpU6davP76668lSQEBAerdu7dq1aql6tWrKzExUVevXpWzs7NCQkIUHR2tVq1aWbzX0dFRs2fP1qRJkxQXF6eMjAzVqVNHEydOVJUqVSzmtmrVSrGxsYqNjdWHH34oHx8fvf7663rjjTds+4UBAAAeQwUqhXv37tXs2bN14MABpaSkyGSyvAWLwWAo8I2fjxw5ku/y2rVr57glTX48PT310Ucf6aOPPnrg3DZt2pgPMwMAAJRkVt+S5ocfflDfvn114MAB1atXT1lZWXruuedUp04dmUwm1axZ84HPJgYAAMDjyepSOGvWLPn6+uq7777ThAkTJElDhgzRV199pblz5+rs2bPq2rWrzYICAADAdqwuhUlJSeratat8fHzMVwJnHz5u0aKFOnfunOP8QAAAADwZrC6FmZmZKl++vKT/PR4uLS3NvDw4OFiHDh0q5HgAAAAoClaXQl9fX124cEGS5ObmprJly+ro0aPm5RcuXFCpUna/mBkAAAAPweoWV6dOHYvnDDdv3lwLFy5UQECAsrKytHjxYtWtW9cmIQEAAGBbVu8p7Nq1q7y8vJSeni5Jeuedd+Ti4qLo6GiNHTtWTk5OGjVqlM2CAgAAwHas3lPYvHlzNW/e3Py6UqVKWrdunXbs2CFHR0c1bNhQHh4eNgkJAAAA23qkkwDd3NwUHh5eWFkAAABgJw9dClNSUrR161ZdvHhRzzzzjFq2bFmIsQAAAFCU8i2FGzZs0IoVKzR+/HiVK1fOPH7o0CENHTpUly9flslkksFgUNOmTTV79mw5OTnZPDQAAAAKV74XmqxZs0YXLlywKISSNGbMGBmNRr3wwgsaN26cmjVrpp07d2rJkiU2DQsAAADbyLcUHjp0SM8991yOsaNHj6p169aKiYlR7969NW/ePD377LNas2aNTcMCAADANvIthZcvX1blypUtxvbs2SODwaDOnTubxwwGg9q1a6dff/3VNikBAABgU/mWwuxnG//eTz/9JElq2LChxfhTTz2lW7duFWI0AAAAFJV8S2FAQICSk5Mtxvbu3St/f3899dRTFuM3b96Ul5dX4ScEAACAzeVbClu0aKFVq1Zpy5Ytun37tr744gv99ttvat26dY65hw8flr+/v82CAgAAwHbyvSXNwIEDlZiYqDfeeEPS/cPJHh4eGjBggMW8jIwMbdmyRV26dLFdUgAAANhMvqXwqaee0vLlyzVv3jydOnVKlStXVv/+/VWxYkWLeQcOHFBoaKgiIiJsGhYAAAC28cAnmlSsWFHvv/9+vnOaNGmiJk2aFFooAAAAFK18zykEAABAyUApBAAAAKUQAAAAlEIAAACIUggAAABRCgEAACBKIQAAAGTFfQqz9enTJ9/lBoNBrq6u8vf3V4sWLRQeHi6DwfDIAQEAAGB7VpfCs2fPKj09XVevXpUklS1bVpKUkpIiSfLx8VFWVpa2bt2q+Ph4hUSlTAkAACAASURBVIaGas6cOXJzc7NBbAAAABQmqw8fL1q0SK6urho4cKC2b9+u3bt3a/fu3dq+fbsGDBig0qVL6+uvv9bOnTvVr18/7d27V9OnT7dldgAAABQSq0vhhAkTFBoaqlGjRsnHx8c87uPjo9GjR6t+/fqaMGGCvLy89O6776ply5Zav369TUIDAACgcFldCnfu3KlGjRrlubxRo0bauXOn+XWzZs104cKFR0sHAACAIlGgq49//fXXfJeZTKb/rdjBQa6urg+fDAAAAEXG6lIYFhampUuXavXq1TmWffvtt/ryyy/VvHlz89jhw4cVEBBQOCkBAABgU1ZffRwdHa2kpCSNHDlSEydOVJUqVSRJp06dktFolK+vr959911JUkZGhs6dO6fIyEjbpAYAAEChsroUBgQEKDExUbNnz9Z//vMfHThwwDzesWNHDRo0SN7e3pIkFxcXLVq0yDaJAQAAUOisLoWS5OXlpdGjR2v06NG2ygMAAAA74DF3AAAAKNieQpPJpO3bt+vkyZO6fv26xdXG0v1H3Q0bNqxQAwIAAMD2rC6FJ0+e1LBhw3Lceub3KIUAAABPJqtL4YcffqjTp09r5MiRatq0qby8vGyZCwAAAEXI6lK4d+9e9e3bVwMHDrRlHgAAANiB1ReaODs76+mnn7ZlFgAAANiJ1aWwRYsW2rdvny2zAAAAwE6sLoXR0dH68ccfNX/+fGVmZtoyEwAAAIqY1ecU9uzZU7dv39Ynn3yiyZMny8/PTw4Olp3SYDBo48aNhR4SAAAAtmV1KaxYsaItcwAAAMCOrC6FcXFxtswBAAAAO7LrY+4uXbqkmJgYRUVFqUGDBgoMDNSuXbtynbtp0ya99NJLqlOnjlq2bKnY2FjdvXs3x7yUlBS9//77atq0qerXr68+ffooOTn5kdYJAABQ3Nm1FJ44cUJz5szRxYsXFRgYmOe8rVu3atiwYfL09NT777+vNm3aaPr06ZowYYLFvKysLA0ePFirV69W7969NWrUKF25ckVRUVE6ffr0Q60TAACgJMjz8HHr1q3l4OCgNWvWyMnJSeHh4Q9cWUEvNAkJCdHOnTvl7e2tjRs35vmIvEmTJunZZ5/VvHnz5OjoKElyd3fX7NmzFRUVpapVq0qS1q5dq/3792v69Olq06aNJCkiIkLt2rVTbGysJk2aVOB1AgAAlAR57ikMCAiQv7+/DAaDpPsXmjzoj7+/f4E+vEyZMvL29s53zrFjx3Ts2DF1797dXN4kqVevXsrKytL69evNY+vWrZOfn59FgfXx8VFERIQ2btyoO3fuFHidAAAAJUGeewr/eGGJvS40OXz4sCSpdu3aFuPly5dXhQoVzMslKTk5WSEhIeYim61OnTqKj4/X6dOnVaNGjQKtEwAAoCSw+upjezEajZIkX1/fHMt8fX116dIli7lNmzbNMc/Pz0/S/QtbatSoUaB1WqtcuTIFfg9y5+vrYe8IJQ7bvOixzYse27zosc2fLI99KUxPT5d0/9nLf+Ti4qLbt29bzM1tXvZY9roKsk5rXbmSqqwsU57L+RfDekbjzUJZD9vcemzzosc2L3ps86JXWNschcfBwZDnjqwClcLVq1crLi5Op06d0vXr13MsNxgMhX7o1dXVVZJyfbReRkaGeXn23NzmZY9lzy3IOgEAAEoCq0vh3LlzNXnyZHl5ealevXoPvECksGQf4jUajebDwNmMRqMaNGhgMTe3Q7/ZY9nvL8g6AQAASgKrS+GSJUtUr149ffHFF0W6Jy04OFiSdPDgQYWEhJjHL168qAsXLpiXS1JQUJD2798vk8lkcbFJUlKS3NzcVLly5QKvEwAAoCSw+ubVRqNRnTp1KvJDqzVr1lT16tUVHx+ve/fumceXLl0qBwcHtW3b1jzWvn17Xbp0SZs2bTKPXb16VWvXrlV4eLicnJwKvE4AAICSwOo9hVWqVNHNm4V/wuiMGTMkScePH5ckJSYmau/evSpbtqx69+4tSRo9erRef/11DRw4UB06dNDRo0e1ePFide/eXdWqVTOvq127dqpfv75Gjx6tAQMGyNvbW0uXLlVWVpbeeusti8+1dp0AAAAlgdWlsH///po5c6aioqLk7u5eaAGmTp1q8frrr7+WdP/m2dmlsFWrVoqNjVVsbKw+/PBD+fj46PXXX9cbb7xh8V5HR0fNnj1bkyZNUlxcnDIyMlSnTh1NnDhRVapUsZhr7ToBAABKAqtLoaOjo8qVK6eIiAh16dJFTz/9tMXTQLJFRkYWKMCRI0esmtemTRvzo+vy4+npqY8++kgfffRRoa0TAACguLO6FEZHR5t/njlzZq5zDAZDgUshAAAA7M/qUrho0SJb5gAAAIAdWV0KmzRpYsscAAAAsCOrb0kDAACA4ivPPYUrV66UJHXu3FkGg8H8+kE4pxAAAODJk2cpjI6OlsFgUIcOHeTs7Gx+bTKZ8lwZF5oAAAA8mfIshdkXljg7O1u8BgAAQPGTZyn844UlXGgCAABQfHGhCQAAAKy/JU22y5cv6+DBg7px40au5xdyTiEAAMCTx+pSmJWVpX/84x9avny5srKy8pxHKQQAAHjyWF0K582bp/j4eL344otq3ry53n33XY0cOVLu7u5auHChPDw89M4779gyKwAAAGzE6lK4cuVK/elPf9KkSZN07do1SVJISIiaNWumzp0768UXX9ShQ4fUrFkzm4UFAAAlj2dZZzm7uNg7xmMvMyNDN1IyH/r9VpfCM2fOqHv37pIkB4f716fcvXtXkuTm5qaXX35Zy5Yt02uvvfbQYQAAAP7I2cVFn44ZYu8Yj713Jnwu6eFLodVXH7u6uqpUqfsd0s3NTQaDQVeuXDEv9/X11YULFx46CAAAAOzH6lJYsWJFnTlzRpLk5OSkypUr67///a95+fbt21WuXLnCTwgAAACbs/rwcdOmTbVhwwa9++67ku4/E/mzzz7TpUuXJEl79uzRgAEDbJMSAAAANmV1KRwwYICaN2+uzMxMOTs7a8iQIbp69aq++eYbOTg46JVXXtHbb79ty6wAAACwEatLoZ+fn/z8/MyvHR0dNW7cOI0bN84mwQAAAFB0rDqnMC0tTX369NGyZctsnQcAAAB2YFUpdHd3108//WTrLAAAALATq68+Dg4O1q+//mrLLAAAALATq0vhW2+9pa+++ko7d+60ZR4AAADYgdUXmnzzzTeqWLGi+vfvr6CgIFWtWlWurq4WcwwGg/75z38WekgAAADYVr6lMDg4WJ988ok6duyohIQE83hycrKSk5NzzKcUAgAAPJnyLYUmk0kmk0mS9PPPPxdJIAAAABQ9q88pBAAAQPFFKQQAAMCDLzT59ddf9cMPP1i9wsaNGz9SIAAAABS9B5bCWbNmadasWVavMLcLUAAAAPB4e2ApbNOmjQIDA4siCwAAAOzkgaWwbdu26tSpU1FkAQAAgJ1woQkAAAAohQAAAKAUAgAAQA84p3DTpk3y8fEpqiwAAACwk3xLYUBAQFHlAAAAgB1x+BgAAACUQgAAAFAKAQAAoHxKYWxsrI4ePWp+ff78eaWnpxdJKAAAABStfEvhkSNHzK/Dw8O1YcOGIgkFAACAopVnKSxbtqxSUlLMr00mU5EEAgAAQNHL85Y0wcHBmjdvnu7evStPT09J0p49e3Tv3r18VxgZGVm4CQEAAGBzeZbCMWPG6M0339SECRMkSQaDQfHx8YqPj89zZQaDgVIIAADwBMqzFAYFBWndunU6c+aMjEajoqKiNHToUIWFhRVlPgAAABSBfJ9o4ujoqKpVq6pq1apq3LixnnvuOTVp0qSosgEAAKCI5FsKfy8uLs6WOfK1a9cu9enTJ9dl3333nWrUqGF+vW/fPn3yySc6fPiwypQpo4iICP3tb39T6dKlLd6XmZmpqVOnKjExUSkpKQoKCtKIESPUrFkzm34XAACAx5HVpVCSsrKylJCQoA0bNujs2bOSpKefflpt27ZVZGSkHBxsey/svn37KiQkxGKsfPny5p+Tk5PVr18/PfPMM4qOjtaFCxc0f/58nT17VrNmzbJ4X3R0tNavX68+ffqoSpUqSkhI0KBBgxQXF6cGDRrY9HsAAAA8bqwuhenp6Ro0aJD27Nkjg8EgX19fSdL/+3//T1u3btXKlSs1Z84cubi42CxskyZN1KZNmzyXf/rpp/Ly8lJcXJzc3d0l3S+t48aN044dO8x7AZOSkrR69WqNGTNG/fr1k3T/qumOHTsqJiZGixcvttl3AAAAeBxZvWtv5syZ+uGHH9S/f3/t2LFDW7du1datW7Vz504NGDBAu3fv1syZM22ZVZKUmpqqu3fv5jq+fft2RUZGmguhJHXu3Flubm5as2aNeWzt2rVycnJSt27dzGMuLi7q2rWr9u7dq0uXLtn2SwAAADxmrC6F3333nSIiIjR69GjzfQul+ze5HjVqlCIiIrR69WqbhMw2atQoNWzYUPXq1dOAAQMsnrhy5MgR3b17V7Vr17Z4j7Ozs4KDg5WcnGweS05OVrVq1SzKoyTVrVtXJpPJYi4AAEBJYHUpvHDhQr5XHjdu3FgXLlwolFB/5OTkpHbt2um9997TjBkzNGzYMCUlJalXr146ceKEJMloNEqS+bD27/n6+lrs/TMajfLz88t1niT2FAIAgBLH6nMKy5Ytq9OnT+e5/PTp0ypbtmyhhPqj0NBQhYaGml+Hh4erdevW6tKli2JjYzV58mSlp6dLur9n8I9cXFzMy6X750c6OTnlOk+SMjIyCpyxXLkyBX4Pcufr62HvCCUO27zosc2LHtu86LHNi96jbHOrS2FYWJgWL16ssLAw/elPf7JY9v3332vp0qVq3779QwcpqKCgIDVr1kw7d+6UJLm6ukq6f6uZP8rIyDAvz557586dXOdJeqiLZa5cSVVWVt7Ph+ZfDOsZjTcLZT1sc+uxzYse27zosc2LHtu86D1omzs4GPLckWV1KRw+fLi+//57DR48WMHBwapZs6Yk6ZdfflFycrK8vb319ttvFyD2o/P39zeXwuxDv9mHkX/vj4eL/3g4+ffzJOV6aBkAAKA4s/qcwoCAAH399dfq0KGDTp48qcTERCUmJurUqVN64YUXtHz5cgUEBNgyaw5nzpyRt7e3JKlWrVoqVaqUDh48aDEnMzNTycnJCg4ONo8FBQXpxIkTSktLs5h74MAB83IAAICSpEB3m65YsaImT56svXv3atu2bdq2bZv27NmjmJgYVaxY0VYZdfXq1Rxje/bs0a5du9SiRQtJkoeHh5o1a6bExESLspeYmKhbt25ZHNpu37697ty5o2XLlpnHMjMztWLFCoWGhlrcEBsAAKAkKNATTbIZDAaVK1eusLPkafjw4SpdurQaNGggb29v/fLLL4qPj5e3t7feeust87wRI0aoR48eioqKUrdu3XThwgUtWLBAzz//vMLCwszz6tWrp/bt2ysmJkZGo1GVK1dWQkKCzp8/rwkTJhTZ9wIAAHhcPFQpLGpt2rTRqlWrtGDBAqWmpsrHx0cdO3bUW2+9ZbGHMiQkRAsWLFBMTIwmTJigMmXK6JVXXtE777yTY52TJk3SlClTlJiYqBs3bigwMFCzZ89Ww4YNi/KrAQAAPBaeiFLYp08f9enTx6q5jRo10pdffvnAeS4uLnr33Xf17rvvPmo8AACAJ16BzikEAABA8UQpBAAAAKUQAAAAlEIAAACoAKUwNTVVffr00eHDh22ZBwAAAHZgdSm8c+eOdu/erRs3bkiSbt26pTFjxuj48eM2CwcAAICikW8pfPvtt/XFF1/owIEDyszMtFiWkZGhlStX5voMYQAAADxZ8r1P4e3btzV9+nTdvHlTpUqVksFg0Jo1a+Tm5qann35aJpOpqHICAADAhvIthXPmzJHJZNKRI0e0bds2ffLJJ1q1apW++uorubm5yWAw6D//+Y88PT0VHBwsg8FQVLkBAABQiB54TqHBYFBQUJBefvllSdKMGTOUmJioQYMGyWQyafHixerSpYuaNGmiIUOG2DwwAAAACl++ewoHDhyohg0bqmHDhqpUqZKk+yUxMDBQvr6+mjp1qj7//HOVLVtWP/zwg/bs2VMkoQEAAFC48i2Fzs7OiouL02effSZHR0cZDAYlJCRIkqpXry5JcnR0VJ06dVSnTh0NGDDA9okBAABQ6PIthTNnzpQknTx5Utu2bdOHH36oLVu2KDExUS4uLjIYDFq/fr1cXV1Vu3ZtlSqV7+oAAADwmLLqPoVVq1ZVhw4dJElTp07VmjVrNGzYMJlMJiUkJKhHjx5q3Lix+vXrZ8usAAAAsJGHesxdtWrV1K1bN0n3LzxZvXq1Ro0aJR8fn0INBwAAgKJh9fFeFxcXvfTSS/Lz88uxrEaNGqpRo4Z69epVqOEAAABQNKwuhW5ubpowYYL5dX4lEQAAAE+Wh74y5I8lEQAAAE+uhzqnEAAAAMULpRAAAACUQgAAAFAKAQAAIEohAAAARCkEAACAKIUAAAAQpRAAAACiFAIAAECUQgAAAIhSCAAAAFEKAQAAIEohAAAARCkEAACAKIUAAAAQpRAAAACiFAIAAECUQgAAAIhSCAAAAFEKAQAAIEohAAAARCkEAACAKIUAAAAQpRAAAACiFAIAAECUQgAAAIhSCAAAAFEKAQAAIEohAAAARCkEAACASngpzMzM1CeffKIWLVqobt26euWVV7Rjxw57xwIAAChyJboURkdHa+HChXrxxRf13nvvycHBQYMGDdL+/fvtHQ0AAKBIldhSmJSUpNWrV2vkyJEaPXq0unfvroULF8rf318xMTH2jgcAAFCkSmwpXLt2rZycnNStWzfzmIuLi7p27aq9e/fq0qVLdkwHAABQtErZO4C9JCcnq1q1anJ3d7cYr1u3rkwmk5KTk+Xn52f1+hwcDA+c85S3+wPnwLptaS3nsuUKbV3FWWFu86fK+BTauoqzwtzmpZ/i77k1CnObe3q5Fdq6irPC3OZlvfh7bo0HbfP8lhtMJpOpsAM9CTp27Kjy5ctr3rx5FuPHjh3TCy+8oPHjx1vsRQQAACjOSuzh4/T0dDk5OeUYd3FxkSRlZGQUdSQAAAC7KbGl0NXVVXfu3Mkxnl0Gs8shAABASVBiS6Gvr2+uF5MYjUZJKtD5hAAAAE+6ElsKg4KCdOLECaWlpVmMHzhwwLwcAACgpCixpbB9+/a6c+eOli1bZh7LzMzUihUrFBoaqvLly9sxHQAAQNEqsbekqVevntq3b6+YmBgZjUZVrlxZCQkJOn/+vCZMmGDveAAAAEWqxN6SRrp/UcmUKVO0atUq3bhxQ4GBgXrnnXcUFhZm72gAAABFqkSXQgAAANxXYs8pBAAAwP9QCgEAAFByLzQpqTIzMzV16lQlJiYqJSVFQUFBGjFihJo1a2bvaMXWpUuXtGjRIh04cEAHDx7UrVu3tGjRIj333HP2jlYsJSUlKSEhQbt27dL58+fl5eWlBg0aaPjw4apSpYq94xVLP/30k2bNmqXDhw/rypUr8vDwUFBQkIYNG6bQ0FB7xysx5syZo5iYGAUFBSkxMdHecYqdXbt2qU+fPrku++6771SjRo0iTlT4KIUlTHR0tNavX68+ffqoSpUqSkhI0KBBgxQXF6cGDRrYO16xdOLECc2ZM0dVqlRRYGCg9u/fb+9IxdrcuXO1b98+tW/fXoGBgTIajVq8eLEiIyO1fPnyYvGL+3Fz5swZ3bt3T926dZOvr69u3rypVatWqXfv3pozZ46aN29u74jFntFo1MyZM+Xm5mbvKMVe3759FRISYjFWXG5jx4UmJUhSUpK6deumMWPGqF+/fpLuX4HdsWNH+fn5afHixfYNWEylpqbqzp078vb21saNGzVs2DD2FNrQvn37VLt2bTk7O5vHTp48qU6dOumFF17Qxx9/bMd0Jcft27fVpk0b1a5dW59//rm94xR70dHROn/+vEwmk1JSUthTaAPZewqnT5+uNm3a2DuOTXBOYQmydu1aOTk5qVu3buYxFxcXde3aVXv37s31sX94dGXKlJG3t7e9Y5QYoaGhFoVQkqpWraqaNWvq+PHjdkpV8pQuXVo+Pj5KSUmxd5RiLykpSd98843GjBlj7yglRmpqqu7evWvvGIWOUliCJCcnq1q1anJ3d7cYr1u3rkwmk5KTk+2UDLAtk8mky5cvU85tLDU1VVevXtWvv/6qTz/9VEePHuV8ZRszmUz68MMPFRkZqeDgYHvHKRFGjRqlhg0bql69ehowYICOHDli70iFhnMKSxCj0ZjreQ++vr6SxJ5CFFvffPONLl68qBEjRtg7SrE2duxYrVu3TpLk5OSkHj16aOjQoXZOVbytXLlSx44d0/Tp0+0dpdhzcnJSu3bt9Pzzz8vb21tHjhzR/Pnz1atXLy1fvlzVqlWzd8RHRiksQdLT0+Xk5JRj3MXFRdL98wuB4ub48eP64IMP1LBhQ3Xu3NnecYq1YcOGqXv37rpw4YISExOVmZmpO3fu5Dicj8KRmpqqyZMna/DgwfLz87N3nGIvNDTU4mr68PBwtW7dWl26dFFsbKwmT55sx3SFg8PHJYirq6vu3LmTYzy7DGaXQ6C4MBqNGjJkiDw9PTV16lQ5OPArz5YCAwPVvHlzdenSRfPmzdOhQ4c4z82GZs6cKScnJ/Xv39/eUUqsoKAgNWvWTDt37rR3lELBb8gSxNfXN9dDxEajUZL4P00UKzdv3tSgQYN08+ZNzZ0713yaBIqGk5OTwsPDtX79eqWnp9s7TrFz6dIlLVy4UL169dLly5d19uxZnT17VhkZGbpz547Onj2rGzdu2DtmieDv719stjWlsAQJCgrSiRMnlJaWZjF+4MAB83KgOMjIyNDQoUN18uRJff7556pevbq9I5VI6enpMplMOX7n4NFduXJFd+7cUUxMjMLDw81/Dhw4oOPHjys8PFxz5syxd8wS4cyZM8XmIjbOKSxB2rdvr/nz52vZsmXm+xRmZmZqxYoVCg0NLTY330TJdu/ePQ0fPlw//vijZsyYofr169s7UrF39epV+fj4WIylpqZq3bp18vf3V7ly5eyUrPh6+umnc724ZMqUKbp165bGjh2rqlWrFn2wYiy3v+d79uzRrl27FBkZaadUhYtSWILUq1dP7du3V0xMjIxGoypXrqyEhASdP39eEyZMsHe8Ym3GjBmSZL5PXmJiovbu3auyZcuqd+/e9oxW7Hz88cfavHmzWrVqpevXr1vcxNfd3b3Y3nTWnoYPHy4XFxc1aNBAvr6++u2337RixQpduHBBn376qb3jFUseHh65/l1euHChHB0d+XtuA8OHD1fp0qXVoEEDeXt765dfflF8fLy8vb311ltv2TteoeCJJiVMRkaGpkyZolWrVunGjRsKDAzUO++8o7CwMHtHK9YCAwNzHQ8ICNDmzZuLOE3xFhUVpd27d+e6jO1tG8uXL1diYqKOHTumlJQUeXh4qH79+howYICaNGli73glSlRUFE80sZFFixZp1apVOn36tFJTU+Xj46MWLVrorbfeUsWKFe0dr1BQCgEAAMCFJgAAAKAUAgAAQJRCAAAAiFIIAAAAUQoBAAAgSiEAAABEKQQAAIAohQAAABCPuQMA3b59W/Hx8Vq/fr2OHTumtLQ0eXp6KiQkRBEREXrxxRdVqlTBf12uWLFCKSkp5meNA8DjjCeaACjRTp06pcGDB+vkyZMKCwtT8+bN5e3trStXrmjHjh3avn27Bg4cqNGjRxd43VFRUTp37hyP1gPwRGBPIYASKz09XUOGDNHZs2c1bdo0tW3b1mL54MGDlZSUpJ9++slOCYueyWTSrVu35O7ubu8oAIoY5xQCKLGWLVumEydOqH///jkKYba6devq1VdfNb/+/vvvNXz4cIWHh6tu3bpq1KiRBgwYoN27d1u8r3Xr1tq9e7fOnTunwMBA859du3aZ55w8eVKjRo1SixYtVLt2bbVu3VoTJ07UrVu3cuTYvXu3unfvrrp166p58+YaP368fvnlFwUGBmratGkWc2/duqXJkyerTZs2ql27tpo3b67Ro0fr3LlzFvN27dqlwMBArVixQosXL1aHDh1Up04dzZ8/X6+//rrq1aun1NTUHFmSkpIUGBio2NjYB29kAE8M9hQCKLHWrVsnSerevbvV70lISNCNGzcUGRmpChUq6OLFi1q2bJn69eunRYsWqVGjRpKksWPHavLkybp27ZrGjBljfn+NGjUkSQcPHlTfvn1VtmxZde/eXeXLl9fPP/+suLg47d+/X3FxcXJycpIk7dmzRwMGDJCnp6cGDx4sDw8PrVmzRvv27cuR786dOxo4cKD27dundu3aqX///jp16pSWLl2qbdu26euvv1aFChUs3rNw4UJdv35d3bp1k6+vrypUqKDatWtr8+bN+vbbb9WjRw+L+cuXL5eDg4O6du1q9XYD8AQwAUAJ1aRJE1NoaGiB3pOWlpZjzGg0mpo0aWJ67bXXLMZ79+5tatWqVa7r6dSpk6ldu3ammzdvWoyvX7/eVKtWLdPXX39tHuvSpYupdu3aptOnT5vHMjMzTd27dzfVqlXL9Nlnn5nH4+PjTbVq1TJNnDjRYr1btmwx1apVyzRy5Ejz2M6dO021atUyNW7c2HT58mWL+Xfv3jX9+c9/NnXp0sVi/NatW6bQ0NAc3xXAk4/DxwBKrNTU1AKfO+fm5mb+OS0tTdeuXZODg4Pq1aunpKQkq9Zx5MgRHTlyRB07dlRmZqauXr1q/tOwYUO5ublp27ZtkqTLly/rp59+Unh4uCpVqmReh5OTk/r06ZNj3Rs2bJCDg4OGDBliMd6yZUsFBwdr06ZNysrKsljWuXNnlStXzmLM0dFRXbp0+f/au7+QJtswjuPfVhBjpoXOEola2IIYRCYVpmAiQcWC0kYlXMv7HAAABMpJREFUKIzsJLCQooM6MkgqyKCDsaICoX8kjIKEkZHRAyo7KqWlaJE6oRVpklrNP++BNBrz7VVeRqW/z+H973p2Mi7u+7qfh7a2Njo6OqLtfr+fL1++aJdQZA7S8bGIzFtJSUkMDw/Pak5PTw+1tbUYhsHQ0FBM34IFC2a0Rnd3NwBXrlyJqwf84ePHjwD09fUBYLPZ4sasWbMmrq2vr4/09HRSUlLi+rKysggGgwwMDMQkgatXr572GUpKSvB4PNTX13P69Glg6ug4NTWVwsLCX/xCEfkbKSkUkXlr7dq1BAIBent7Y3bh/s3w8DClpaWMjo5SXl6O3W7HYrFgMpnwer20tLTMKr7b7SY/P3/avuTk5Fmt9X+YzeZp2zMyMsjPz+fhw4ecPHmS/v5+AoEAbrc7Wu8oInOHkkIRmbd27NhBIBDg/v37VFVV/ef45uZmwuEw586do7i4OKbv8uXLM467atUqAEwmE7m5ub8cm5mZCcDbt2/j+t68eRPXtnLlSp4/f87Q0FBcYtnd3U1SUhLLli2b8bO6XC6amppobGwkGAwC6OhYZI5STaGIzFv79+/HZrNx48YNGhsbpx3T3t7OrVu3gKk6O5h6l9/PDMPgxYsXcXMtFgufP3+OG79+/Xrsdjt3796lt7c3bt7Y2BiDg4MAWK1WHA4HT548iRkbiUSoq6uLm1tUVMTExARXr16NaX/27BmvXr2isLAQk2nmf/0FBQWkp6dz7949fD4f2dnZ0RvUIjK3aKdQROYts9mM1+vlyJEjHD16lLy8PHJzc1m6dCmfPn2itbUVwzA4fPgwAJs2bcJqtXL+/HlCoRArVqwgGAzy4MED7HY7nZ2dMetv2LCBp0+fUl1dzcaNG1m4cCFbt24lNTWVCxcuUF5ezp49eyguLiYrK4uvX7/y7t07Hj9+TFVVFfv27QPg1KlTuN1uDhw4wMGDB6OvpIlEIkBsLePevXvx+Xxcu3aNUChETk4OPT093L59m7S0tBntiP7sx4UTj8cDMOv5IvL30GfuRGTe+/HtY7/fT1dXFyMjI6SkpOBwONi1axdOpzO6S/j69WsuXrzIy5cvGRsbw+FwcOzYMerr6/H5fDE3dUdHRzl79ixNTU0MDAwwMTFBXV0dW7ZsASAUCuH1ejEMg3A4jMViITMzk23btnHo0CEyMjKiazU3N1NbW0swGCQ5OZmdO3fidDpxuVycOHGCioqK6NiRkRE8Hg8NDQ28f/+eJUuWkJeXx/Hjx6PH0TD18uqysjJqamqiCeh0QqEQRUVFmM1mDMOIuYEtInOHkkIRkb+U3++nsrKSS5cusXv37oTFCYfDFBQUUFJSQnV1dcLiiMjvpZpCEZE/3OTkJN++fYtpi0Qi3Lx5k0WLFrF58+aExr9z5w7j4+O4XK6ExhGR30s1hSIif7jv37+zfft2nE4nNpuNwcFBGhoa6OjooKKiAqvVmpC4jx49or+/n+vXr0e/zywic5eOj0VE/nDj4+OcOXOGQCDAhw8fmJycxGaz4XK5KC0tTVjcdevWsXjxYnJycqipqWH58uUJiyUiv5+SQhERERFRTaGIiIiIKCkUEREREZQUioiIiAhKCkVEREQEJYUiIiIigpJCEREREQH+Ab0+tBgbsAvEAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kd1aKF3RQzL1"
      },
      "source": [
        "They look pretty even, except for maybe class 5, which corresponds to 'a an online opinion-piece'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WM2D8cKlQ24c",
        "outputId": "2fe1c317-a610-4168-d289-20bc5df29492",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "train.target_names[5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'a an online opinion-piece'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_HzeZqLFWeh"
      },
      "source": [
        "## S3. Tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k_qjYE4eFciL"
      },
      "source": [
        "BERT has a maximum input length of 512 tokens, so we'll need to truncate any comments which are longer than this.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OMHzhnmuGrpb"
      },
      "source": [
        "### 3.1. Load BertTokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vT0b0-5NGq0y"
      },
      "source": [
        "preprocess the data for BERT to be able to analyze it.\n",
        "\n",
        "In order to see the distribution of comment lengths *in terms of BERT tokens*, we'll need to first apply the BertTokenizer to the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z474sSC6oe7A",
        "outputId": "3f4dcf75-3e8f-41d7-a6e6-e02b86caa4bc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from transformers import BertTokenizer\n",
        "\n",
        "# Load the BERT tokenizer.\n",
        "print('Loading BERT tokenizer...')\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading BERT tokenizer...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8kEDRvShcU5"
      },
      "source": [
        "### 3.2. Sequence Length Distribution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OGkq8hQJfqZY"
      },
      "source": [
        "To decide on a truncation strategy for this dataset, let's first look at the distribution of sequence lengths.\n",
        "\n",
        "To do this, our first step is to tokenize all of the samples in the training set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1M296yz577fV"
      },
      "source": [
        "**Tokenize All Samples**\n",
        "\n",
        "The `tokenizer.encode` function combines multiple steps for us:\n",
        "1. Split the sentence into tokens.\n",
        "2. Add the special `[CLS]` and `[SEP]` tokens.\n",
        "3. Map the tokens to their IDs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bBdb3pt8LuQ",
        "outputId": "9651ae6c-4c71-41d4-b455-e7417251c45b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "\n",
        "# Record the length of each sequence (in terms of BERT tokens).\n",
        "lengths = []\n",
        "\n",
        "print('Tokenizing comments...')\n",
        "\n",
        "# For every sentence...\n",
        "for sen in train.data:\n",
        "    \n",
        "    # Report progress.\n",
        "    if ((len(input_ids) % 20000) == 0):\n",
        "        print('  Read {:,} comments.'.format(len(input_ids)))\n",
        "    \n",
        "    # `encode` will:\n",
        "    #   (1) Tokenize the sentence.\n",
        "    #   (2) Prepend the `[CLS]` token to the start.\n",
        "    #   (3) Append the `[SEP]` token to the end.\n",
        "    #   (4) Map tokens to their IDs.\n",
        "    encoded_sent = tokenizer.encode(\n",
        "                        sen,                      # Sentence to encode.\n",
        "                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                        #max_length = 512,          # Truncate all sentences.                        \n",
        "                        #return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                   )\n",
        "    \n",
        "    # Add the encoded sentence to the list.\n",
        "    input_ids.append(encoded_sent)\n",
        "\n",
        "    # Record the non-truncated length.\n",
        "    lengths.append(len(encoded_sent))\n",
        "\n",
        "print('DONE.')\n",
        "print('{:>10,} comments'.format(len(input_ids)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tokenizing comments...\n",
            "  Read 0 comments.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Token indices sequence length is longer than the specified maximum sequence length for this model (719 > 512). Running this sequence through the model will result in indexing errors\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "DONE.\n",
            "    10,138 comments\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X_erWZU7hDTB"
      },
      "source": [
        "Print the min, max and median sentence lengths"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qmrQVSsjg6By",
        "outputId": "e88b0565-3e07-4aec-d0b3-144f5386960d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print('   Min length: {:,} tokens'.format(min(lengths)))\n",
        "print('   Max length: {:,} tokens'.format(max(lengths)))\n",
        "print('Median length: {:,} tokens'.format(np.median(lengths)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "   Min length: 8 tokens\n",
            "   Max length: 719 tokens\n",
            "Median length: 28.0 tokens\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIkyE9qjSjk3"
      },
      "source": [
        "What percentage are over the 512 limit?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DsAV58bZSmEt",
        "outputId": "26575316-90b6-4b9a-fd8e-86dba4f49fb6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "num_over = 0\n",
        "\n",
        "# For all of the length values...\n",
        "for length in lengths:\n",
        "    # Tally if it's over 512.\n",
        "    if length > 512:\n",
        "        num_over += 1\n",
        "\n",
        "print('{:,} of {:,} comments will be truncated ({:.2%})'.format(num_over, len(lengths), float(num_over) / float(len(lengths))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 of 10,138 comments will be truncated (0.01%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xytsw1oIfnX0"
      },
      "source": [
        "To further analyze it, let's plot the distribution. To keep the scale of the x-axis reasonable, *we'll clip the lengths to 512.*\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iLhi9A49zmsy",
        "outputId": "46598c87-c5ba-416e-ad4b-31ae0271ad29",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 443
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (10,5)\n",
        "\n",
        "# Truncate any comment lengths greater than 512.\n",
        "trunc_lengths = [min(l, 512) for l in lengths]\n",
        "\n",
        "# Plot the distribution of comment lengths.\n",
        "sns.distplot(trunc_lengths, kde=False, rug=False)\n",
        "\n",
        "# Alternatively, you might try using a log scale on the x-axis, but this is \n",
        "# tricky. See here for one approach:\n",
        "# https://stackoverflow.com/questions/47850202/plotting-a-histogram-on-a-log-scale-with-matplotlib?rq=1\n",
        "#plt.xscale('log')\n",
        "\n",
        "plt.title('Comment Lengths')\n",
        "plt.xlabel('Comment Length')\n",
        "plt.ylabel('# of Comments')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/seaborn/distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).\n",
            "  warnings.warn(msg, FutureWarning)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0, 0.5, '# of Comments')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVzU9dr/8fcgiyJqasAxzTVZUhZxB+ykqCFlUgpu4SFN8+TvVNopte2UdWcleVdim2apkbdZKHprLpjZorkmZqGmaekxdZQSEBlAvr8/vJnTNKiDAjPC6/l4+Hg0n881n+/FXE5efFeTYRiGAAAAUKu5OTsBAAAAOB9NIQAAAGgKAQAAQFMIAAAA0RQCAABANIUAAAAQTSEAwMUcPXpUgYGBmjVrlrNTAWoVmkIAVebcuXN6//33NWLECHXr1k0dOnRQZGSkxo4dq/T0dJWUlDg7RZeVnZ2tWbNm6ejRow6/Z9asWQoMDNR3331XhZlVjtzcXM2aNUtbtmxxdioA/g9NIYAq8fPPPys+Pl7Tp0+Xl5eXxo0bp2nTpik5OVklJSWaOnWqZs6c6ew0XVZ2drZSU1P173//29mpVInc3FylpqZq69atzk4FwP9xd3YCAGqewsJC3X///Tp69KhmzZql/v3728yPGzdOu3fvvib2aAFAbcGeQgCVbsmSJTp06JDuvfdeu4awTGhoqEaOHGkzlpmZqWHDhik8PFydOnXSsGHDlJmZaffePn36KCkpSXv37lVycrI6deqknj176sUXX1RJSYksFoteeukl9erVSyEhIRo5cqQOHjxos0Z6eroCAwO1efNmpaamqnfv3goNDVVCQoJ27dolSdq6dauGDx+u8PBwRUdHa/bs2eX+LN99950mTJig7t27q2PHjrrtttv05ptv2h0eT0pKUp8+fXTixAlNmjRJXbt2VVhYmMaMGaNDhw5Z42bNmqWpU6dKkkaNGqXAwEAFBgZqypQpl/nkHbdp0yaNHj1aXbp0UUhIiAYOHKhFixbZxZV91gcPHtS4cePUqVMnde7cWQ8++KDMZrNd/N69ezV69GiFh4ere/fumjx5snJycmzy37Jli2JiYiRJqamp1p+vT58+dutt2LBBgwcPVkhIiKKjo/XSSy/Zfa4//vijHnzwQfXq1UsdO3ZUVFSUkpKS9Pnnn1fCJwXUHnWeeeaZZ5ydBICaZcaMGTp27JhefvllNWrUyKH3pKWlafLkyfL29taoUaPUtWtX7dq1S2lpafLz81PHjh2tsfPnz1dBQYGWLFmi7t27a+DAgSopKdEnn3yioqIipaWl6ddff1VCQoJuvvlmrV27Vhs3btTIkSNlMpkkXTg8u379eh04cEA//fSTEhMT1aVLF23cuFHp6elq166d/vnPf6pfv36KjY3VqVOnlJ6erpYtWyooKMiay+eff64xY8ZIkoYNG6b+/fvLZDJpwYIFOnDggAYMGGCNXbp0qU6cOKGVK1eqWbNmuvvuu9W6dWutWrVKX331lYYPHy43Nzc1atRIhmHo+++/1/jx45WYmKh+/fopKipKzZo1u+hnuHXrVm3dulWJiYny9/e/aNzixYs1ceJENW3aVImJierdu7dyc3M1b948FRQUKDo62uazPnfunD766COFhYVp0KBBatKkiZYvX659+/Zp0KBB1tjDhw9r6NCh+vXXX60N8N69e7V48WKZzWYFBwerb9++qlu3rvz9/fXVV1+pX79+Gj9+vPr166eYmBi1bdtWubm5WrBggc6dO6dly5bp9ttv12233abc3FxlZGTIy8tLXbp0kST99ttvSkhI0LFjxzR06FANHDhQQUFBysvLU1FRkXr06OHQ3z8AkgwAqGTdunUzIiIiHI7//fffjfDwcKNv375GXl6edTwvL8+IiYkxwsPDjTNnzljHe/fubQQEBBirVq2yWeeuu+4yAgMDjfHjxxulpaXW8fnz5xsBAQHGF198YR375JNPjICAACM+Pt6wWCzW8czMTCMgIMC4+eabjd27d1vHLRaLERUVZSQmJlrHCgsLjcjISGPEiBFGcXGxTS7vvfeeERAQYHzzzTfWsXvuuccICAgw3nnnHZvYOXPmXDS/P77/cl5//XUjICDAJu8/O3HihNGxY0dj0qRJdnPPPfecERQUZPzyyy/WsbLPeuXKlTaxzzzzjBEQEGAcPHjQOvbggw8aAQEBxvbt221iH3roISMgIMCYPHmydezIkSNGQECA8frrr9vlUTYXFhZmHDlyxDpeWlpq3H777UZUVJR1rKxef84PQMVx+BhApcvPz1f9+vUdjv/6669VUFCgpKQk+fj4WMd9fHyUlJSkgoICbdq0yeY9/v7+NnvhJCkiIkKGYSgpKcm6R1CSda/Szz//bLft4cOHy9PT0y42NDRUISEh1nFPT0+FhITo8OHDNnmfOnVKd999t3Jzc5WTk2P9c8stt1hj/sjNzU2jRo2yGSvbm1VefpVtzZo1Kioq0pAhQ2zyzcnJUZ8+fVRaWmr3Wfv5+SkuLu6SOZ8/f15ffPGFQkND1blzZ5vY0aNHX1GuMTExatGihfW1yWRS9+7dZTabdfbsWUlSgwYNJElffvml8vPzr2g7AC7gQhMAlc7Hx8f6j7Yjym670r59e7u5srEjR47YjP+xWShTdqj6z3MNGzaUJP3+++9277nxxhsdWqNs7o9rlJ2n+Pjjj9vFljl16pTNaz8/P3l5edmMXXfddRfNr7KV5ZycnHzRmD/n/OfPSLLPOScnRwUFBWrTpo1dbHljjrjcduvXr69u3bopPj5e6enpWrFihTp27KjIyEjFxcXppptuuqLtArUVTSGASte+fXtt27ZNR44cKfcf9spQp06di865uZV/EMQwDIdjL7X+n9d77LHHFBwcXG6Mn5+fw+uWl19lK9vGSy+9ZJdbmT/XzFk5O7rdl156SWPGjNEXX3yh7du367333tNbb72lxx9/XPfcc0+V5QfUNDSFACpd//79tW3bNi1ZskSTJk26bHxZE/Ljjz+qZ8+eNnMHDhywiXElrVu3liTVq1dPkZGRlbr2Hw9/V6aynBs3blypOTdp0kTe3t42V1GXKW+ssn++gIAABQQE6L777lNubq4SEhL0yiuv2FxcBODSOKcQQKVLSEhQmzZtNG/evHJvKSNJe/bsUVpamiQpKipK3t7e+uCDD2zOC8vPz9cHH3wgb29vRUVFVUvuFREdHa2mTZtqzpw55R76LSwsvOLz3Ly9vSVJZ86cuaoc/2zAgAHy9PTUrFmzVFhYaDdfdtVuRdWpU0e9evXS7t27tWPHDpu5efPm2cVX1s/3+++/q7S01GasYcOGatGihc6dOyeLxXJV6wO1CXsKAVS6evXq6e2339a4ceM0YcIERUdHKzIyUtddd51ycnK0ZcsWffXVV7rvvvskXfhH/J///KemTZumxMRE3XXXXZIu3MLl559/1rRp06wXFLgSb29vvfTSS5owYYJiY2M1ePBgtWrVSrm5ufrpp5+0bt06paamqnv37hVeOyQkRG5ubnrrrbd05swZeXt7q0WLFgoLC7vsez/55BN9+eWXduMdOnTQX//6Vz3zzDN68sknFRcXpzvvvFPNmzdXTk6O9u/fr8zMTK1cubLccyov5+GHH7bW9Z577tFf/vIXff7558rJyZFku3ewcePGatWqlVauXKkbb7xR119/verVq1fuvQovZdmyZZo/f7769u2rVq1ayd3dXdu2bdNXX32lAQMGqG7duhX+OYDaiqYQQJVo1aqVli1bpsWLF2vNmjV66623VFBQoEaNGqljx4568cUXNXDgQGv8yJEj5efnp3fffdd6k+igoCDNnj1bffv2ddaPcVm9evXSxx9/rHfeeUfLly/Xb7/9poYNG6ply5ZKTk5WYGDgFa17ww036IUXXtCcOXP07LPPqri4WHfddZdDTWF5N6GWpKFDh+qvf/2rBg8erNatW2vevHlavHix8vLydN1116lNmzZ66KGH5Ovre0U5t23bVmlpaXrppZe0YMECeXl56dZbb9XTTz+tvn372l1gk5KSohdeeEH//d//rXPnzql58+YVbgq7d++u7Oxsff755zKbzXJzc1OLFi00efJkzicEKshkVMeZzQCAWmvPnj0aPHiwHnnkEY0bN87Z6QC4CM4pBABUmj+fp2gYhubOnStJlX4xDoDKxeFjAEClGTRokHr06KGAgACdO3dOGzZs0Pbt2xUXF2fzqEIArofDxwCASvPyyy9rw4YNOn78uEpKStSiRQsNHDhQY8eOlYeHh7PTA3AJNIUAAADgnEIAAADQFAIAAEBcaFJpfvvtrEpLr/xIfNOmPjp9+sqefICqQU1cC/VwPdTE9VAT1+KK9XBzM6lx4/rlztEUVpLSUuOqmsKyNeBaqIlroR6uh5q4HmriWq6lenD4GAAAADSFAAAAoCkEAACAaAoBAAAgmkIAAACIphAAAACiKQQAAIBoCgEAACCaQgAAAIgnmtQ4JaWSpbjksnFeHu5y51cCAADwf2gKaxhLcYm2ZZ+4bFzXYH+5e1F+AABwAfuKAAAAQFMIAAAAmkIAAACIphAAAACiKQQAAIBoCgEAACCaQgAAAIimEAAAAKIpBAAAgGgKAQAAIJpCAAAAiKYQAAAAoikEAACAaAoBAAAgmkIAAACIphAAAACiKQQAAIBoCgEAACCaQgAAAIimEAAAAKIpBAAAgGgKAQAAIBdrCufMmaPAwEANGjTIbm7nzp0aPny4wsLCFBUVpeeff17nzp2ziysqKtKMGTMUHR2t0NBQJSYmavPmzeVuz9E1AQAAajqXaQrNZrPefPNNeXt7281lZ2crOTlZFotFU6ZM0ZAhQ7R48WJNnDjRLnbKlCmaP3++7rzzTj3xxBNyc3PT2LFj9e23317xmgAAADWdu7MTKPPKK6+oY8eOMgxDubm5NnMzZ87Uddddp4ULF6p+/fqSpBYtWujJJ5/U5s2b1bNnT0nS7t27tXLlSk2dOlXJycmSpPj4eN1xxx1KSUlRWlpahdcEAACoDVxiT+Hu3bu1fPlyTZ061W4uPz9fmzZtUnx8vLV5k6RBgwbJ29tbn376qXVs9erV8vDwUEJCgnXMy8tLQ4YM0Y4dO3Ty5MkKrwkAAFAbOL0pNAxDzz33nOLj4xUcHGw3v2/fPpWUlKhjx442456engoODlZ2drZ1LDs7W23atLFp9CQpNDRUhmFYYyuyJgAAQG3g9MPHy5Yt04EDBzR79uxy581msyTJ19fXbs7X11e7du2yifX39y83TpJ1T2FF1nRU06Y+FX6P/bYbXPUaRk6BGvjUvWyct7eXfJvYn78JW5VRE1Qe6uF6qInroSau5Vqqh1Obwvz8fL3yyisaN26c/Pz8yo0pLCyUdGEv3p95eXlZ58tiPTw8yo2TJIvFUuE1HXX6dL5KS40Kv6+Mr28Dmc15V/z+MgWWEuXlXz7/ggKLzOfPX/X2arLKqgkqB/VwPdTE9VAT1+KK9XBzM110R5ZTDx+/+eab8vDw0L333nvRmLp1L+z1KioqspuzWCzW+bLY4uLicuOk/zSHFVkTAACgNnDansKTJ09q/vz5euihh3Tq1CnruMViUXFxsY4ePaoGDRpYD/GWHfL9I7PZbLOH0dfX13qI+M9xkqyxFVkTAACgNnDansLTp0+ruLhYKSkpiomJsf7JysrSwYMHFRMTozlz5iggIEDu7u7as2ePzfuLioqUnZ1tc3FKUFCQDh06pLNnz9rEZmVlWeclVWhNAACA2sBpewpbtGhR7sUlr776qgoKCvT444+rdevWatCggXr27KmMjAzdf//91iuLMzIyVFBQoNjYWOt7Y2NjNW/ePC1ZssR6n8KioiKlp6crIiLCehFKRdYEAACoDZzWFDZo0EB9+/a1G58/f77q1KljMzdx4kQNGzZMSUlJSkhI0PHjx/Xee+/plltuUWRkpDUuLCxMsbGxSklJkdlsVsuWLbV06VIdO3ZM06dPt9mOo2sCAADUBk6/T6EjOnTooPfee0+enp6aPn26lixZosTERL322mt2sS+//LKSkpKUkZGh559/XiUlJXrnnXfUuXPnK14TAACgpjMZhnHl91GBlavckuaspUTbsk9cNq5rsL/qezn9NpUuzRVvJVCbUQ/XQ01cDzVxLa5YD5e9JQ0AAABcA00hAAAAaAoBAABAUwgAAADRFAIAAEA0hQAAABBNIQAAAERTCAAAANEUAgAAQDSFAAAAEE0hAAAARFMIAAAA0RQCAABANIUAAAAQTSEAAABEUwgAAADRFAIAAECV0BTu2bNHX3/9tSwWS2XkAwAAACdwdzTw3Xff1bZt2/TWW29Zxx555BGtWrVKknTjjTfqww8/1PXXX1/5WQIAAKBKObyncOXKlWrWrJn19ebNm7Vy5UrFxcVp4sSJMpvNmjt3bpUkCQAAgKrl8J7Cf//737r77rutr9evXy9fX1+lpKTIZDLpt99+02effaYpU6ZUSaIAAACoOg7vKTx37py8vLysr7/55htFRkbKZDJJktq1a6cTJ05UfoYAAACocg43hf7+/tq/f7+kC3sNDxw4oK5du1rnc3Nz5enpWfkZAgAAoMo5fPi4d+/e+vDDD3X+/HllZWXJ09NTt956q3X+xx9/VPPmzasiRwAAAFQxh5vCCRMmaN++ffrwww/l6empxx9/3HqlcWFhodatW6chQ4ZUWaIAAACoOg43hY0aNdL8+fOVn58vLy8veXh42Mx/8MEHNlcnAwAA4Nrh8DmFqamp2r9/v3x8fOwawrp166pOnTpauHBhpScIAACAqlehpnDfvn0Xnf/xxx81e/bsSkkKAAAA1avSnn1ssVhUp06dyloOAAAA1eiS5xTm5+crNzfX+vr333/XsWPH7OLOnDmjFStWcE4hAADANeqSTeH7779vPSRsMpn0wgsv6IUXXig31jAMPfroo5WfIQAAAKrcJZvCbt26SbrQ8M2ePVv9+vVTYGCgXVz9+vUVFhamiIiIqskSAAAAVeqyTWFZY3js2DENGzZMYWFh1ZIYAAAAqo/D9ymcPn16VeYBAAAAJ3K4KSxz+PBh/fzzz/rtt9/KnY+Pj7/qpAAAAFC9HG4KT506pcmTJ2vTpk2SLpxn+Gcmk4mmEAAA4BrkcFM4bdo0bdq0ScOHD1ePHj103XXXVWVeAAAAqEYON4WbNm3SsGHD9PTTT1dlPgAAAHACh59oUlpaqqCgoKrMBQAAAE7icFPYpUsX7d27typzAQAAgJM43BROmTJF69at05o1a6oyHwAAADiBw+cUPvPMM6pfv74efvhh+fn56cYbb5Sbm21PaTKZNH/+/EpPEgAAAFXL4abw6NGjkqRmzZpJuvCEEwAAANQMDjeFn332WVXmAQAAACdy+JzCyvbdd99pwoQJ6t27t0JDQxUVFaUxY8Zo586ddrE7d+7U8OHDFRYWpqioKD3//PM6d+6cXVxRUZFmzJih6OhohYaGKjExUZs3by53+46uCQAAUBtU+DF3R48e1ebNm3Xq1CkNHDhQLVq0UFFRkU6dOqXrr79enp6eDq1z5MgRnT9/XgkJCfL19VVeXp5WrFihe+65R3PmzFFUVJQkKTs7W8nJybrppps0ZcoUHT9+XPPmzdPRo0f11ltv2aw5ZcoUrV27VqNGjVKrVq20dOlSjR07VgsXLlSnTp2scRVZs6YyuZl01lLiUKyXh7vcnfbrAwAAqA4VagpnzJih999/X+fPn5fJZFJ4eLi1Kbz99tv10EMPKTk52aG14uLiFBcXZzM2fPhw9e3bVwsWLLA2hTNnztR1112nhQsXqn79+pKkFi1a6Mknn9TmzZvVs2dPSdLu3bu1cuVKTZ061ZpDfHy87rjjDqWkpCgtLc26HUfXrMksxeeVtd/sUGzXYH+5e1X49wcAAHANcXj/z//8z//o3Xff1YgRIzRv3jybZx/7+PioT58+2rBhw1UlU69ePTVp0kS5ubmSpPz8fG3atEnx8fHW5k2SBg0aJG9vb3366afWsdWrV8vDw0MJCQnWMS8vLw0ZMkQ7duzQyZMnK7wmAABAbeFwU/jhhx+qX79+euKJJxQcHGw3HxgYqEOHDlU4gfz8fOXk5Oinn37SzJkztX//fuueun379qmkpEQdO3a0eY+np6eCg4OVnZ1tHcvOzlabNm1sGj1JCg0NlWEY1tiKrAkAAFBbOHxM8PDhwxo+fPhF5xs3bqzffvutwgk8/vjj1htie3h4aNiwYRo/frwkyWy+cHjT19fX7n2+vr7atWuX9bXZbJa/v3+5cZKseworsmZFNG3qc0Xvs91+g6tew8gpUAOfupeN8/BwdyhOkry9veTbxPtqU7smVUZNUHmoh+uhJq6HmriWa6keDjeFXl5el7w699ixY2rYsGGFE5gwYYKGDh2q48ePKyMjQ0VFRSouLpanp6cKCwslqdyLV7y8vKzzklRYWCgPD49y4yTJYrFY4xxdsyJOn85Xaalx+cCL8PVtILM574rfX6bAUqK8/Mv/DMXFjsVJUkGBRebz5682tWtOZdUElYN6uB5q4nqoiWtxxXq4uZkuuiPL4cPHoaGhWrduXblzFotFGRkZioiIqHBygYGBioqK0uDBg/Xuu+/q+++/19SpUyVJdete2JNVVFRU7jbL5stii4uLy42T/tMcVmRNAACA2sLhpnDMmDHatWuXHn30Ue3bt0+SdOrUKX355ZdKSkrSiRMnNHr06KtKxsPDQzExMVq7dq0KCwuth3jLDvn+kdlslp+fn/W1r6+v9RDxn+MkWWMrsiYAAEBt4XBTGBkZqWeeeUZr1qzRvffeK0l67LHHNG7cOO3du1fPPfeczb0Ar1RhYaEMw9DZs2cVEBAgd3d37dmzxyamqKhI2dnZNhe8BAUF6dChQzp79qxNbFZWlnVeUoXWBAAAqC0qdEvioUOHav369Xr88cc1fPhwDR06VJMnT9a6det09913V2jDOTk5dmP5+flas2aNmjVrpqZNm6pBgwbq2bOnMjIybJq9jIwMFRQUKDY21joWGxur4uJiLVmyxDpWVFSk9PR0RUREWC9CqciaAAAAtUWF70js6+urpKSkq97www8/LC8vL3Xq1Em+vr769ddflZ6eruPHj2vmzJnWuIkTJ2rYsGFKSkpSQkKCjh8/rvfee0+33HKLIiMjrXFhYWGKjY1VSkqKzGazWrZsqaVLl+rYsWOaPn26zbYdXRMAAKC2MBl/vAt1Nfr444+VkZGhAwcOKDc3Vw0aNFB4eLhGjx6tbt262cRu375dKSkp+uGHH+Tj46O4uDhNmjRJ3t62t0mxWCx69dVXtWLFCp05c0aBgYGaNGlSuY2eo2s6ylWuPj5rKdG27BOXjQsL8K3QE03q18InmrjiVWO1GfVwPdTE9VAT1+KK9bjU1ccVagp37typtLQ0/fzzz/r999/157eaTCZlZmZeXbbXKJrCmscVv8y1GfVwPdTE9VAT1+KK9bhUU+jwv/QfffSR/vWvf8nDw0Nt2rRRs2bNKi1BAAAAOJfDTeFbb72l4OBgzZ07V02aNKnKnAAAAFDNHL76+PTp0xo8eDANIQAAQA3kcFPYrl075ebmVmUuAAAAcBKHm8Lx48frww8/1IkTl7+IAQAAANcWh88p7N+/v86dO6fbb79dMTExat68udzcbHtKk8mkCRMmVHqSAAAAqFoON4WHDh3S66+/rvz8fGVkZJQbQ1MIAABwbXK4KXz22WeVk5OjJ554Ql26dFHDhg2rMi8AAABUI4ebwl27dmnMmDGV8og7AAAAuBaHLzTx8fHhdjQAAAA1lMNN4YABA7R27dqqzAUAAABO4nBTOGzYMJ09e1YPPPCANm/erCNHjujYsWN2fwAAAHDtcficwttvv10mk0l79uzRhg0bLhqXnZ1dKYkBAACg+jjcFE6YMEEmk6kqcwEAAICTONwU/uMf/6jKPAAAAOBEDp9TCAAAgJrL4T2FZQ4fPqyff/5Zv/32W7nz8fHxV50UAAAAqpfDTeHJkyc1ZcoUbd68WZJkGIZdjMlkoikEAAC4BjncFD799NPasmWL/va3v/GYOwAAgBrG4abwm2++0ahRozR58uSqzAcAAABO4PCFJt7e3mrZsmVV5gIAAAAncbgpvPXWW63nEwIAAKBmcbgpnDJlio4ePaoXXnhBR44cKfdCEwAAAFybHD6nsGHDhoqPj9f06dO1cOHCcmNMJpN++OGHSksOAAAA1cPhpnDOnDmaOXOmmjZtqtDQUDVq1Kgq8wIAAEA1crgp/OCDD9StWzfNnTtXHh4eVZkTAAAAqpnD5xSeOXNGAwYMoCEEAACogRxuCoOCgvTrr79WZS4AAABwEoebwocffliLFy/Wd999V5X5AAAAwAkcPqcwIyND/v7+Gjp0qMLDw3XjjTfKzc22pzSZTHrhhRcqPUkAAABULYebwqVLl1r/e+fOndq5c6ddDE0hAADAtcnhpnDv3r1VmQcAAACcyOFzCgEAAFBzObynsIxhGPrhhx905MgRSdKNN96om2++WSaTqdKTAwAAQPWoUFP4xRdf6Nlnn9WxY8dsxps3b65//etf6tWrV6UmBwAAgOrhcFO4Y8cOPfDAA6pXr55GjRqlm266SZJ04MABLV26VH//+9+1YMECRUREVFmyAAAAqBoON4VvvPGGrr/+en300Ufy8/OzmRszZowSExM1e/Zsvfvuu5WeJAAAAKqWwxeaZGVlKTEx0a4hlCQ/Pz8lJCQoKyurUpMDAABA9XC4KSwuLlb9+vUvOu/j46Pi4uJKSQoAAADVy+GmsF27dlq1apVKSkrs5kpKSvTpp5+qXbt2lZocAAAAqofDTeHw4cOVlZWl5ORkff755zpy5IiOHDmiDRs2KDk5WVlZWRo+fHhV5goAAIAq4vCFJgkJCTp8+LDmzZunHTt22M2PGTNGCQkJlZocAAAAqkeF7lP46KOPasiQIVq/fr2OHj0q6cLNq/v06aM2bdpUSYIAAACoehV+okmbNm103333XfWGd+/eraVLl2rLli06duyYrrvuOnXq1EkPP/ywWpy20LQAACAASURBVLVqZRO7c+dOzZgxQz/88IN8fHw0YMAAPfLII6pXr55NXFFRkV577TVlZGQoNzdXQUFBmjhxonr27Gm3fUfXBAAAqA0ue07hokWLtGrVqkvGrFq1SosXL67QhufOnat169YpMjJSTzzxhBITE7V161bFx8fr4MGD1rjs7GwlJyfLYrFoypQpGjJkiBYvXqyJEyfarTllyhTNnz9fd955p5544gm5ublp7Nix+vbbb23iKrImAABAbXDJPYXr1q3TtGnTNHfu3Esu0rBhQz3yyCPy9/fXrbfe6tCGk5OTlZKSIk9PT+tYXFycBg4cqDlz5ujFF1+UJM2cOVPXXXedFi5caL0lTosWLfTkk09q8+bN1r2Au3fv1sqVKzV16lQlJydLkuLj43XHHXcoJSVFaWlp1u04uiYAAEBtcck9hStWrFBYWJiioqIuuUh0dLQiIiK0dOlShzccERFh0xBKUuvWrdW+fXvrnsL8/Hxt2rRJ8fHxNvdIHDRokLy9vfXpp59ax1avXi0PDw+bi128vLw0ZMgQ7dixQydPnqzwmgAAALXFJZvCrKws/fWvf3VooV69el31E00Mw9CpU6fUuHFjSdK+fftUUlKijh072sR5enoqODhY2dnZ1rHs7Gy1adPG7gbboaGhMgzDGluRNQEAAGqLSx4+Pn36tPz9/R1ayM/PT6dPn76qZJYvX64TJ05Yz+0zm82SJF9fX7tYX19f7dq1y/rabDaXm2vZe8v2FFZkzYpo2tTnit5nu/0GV72GkVOgBj51Lxvn4eHuUJwkeXt7ybeJ99Wmdk2qjJqg8lAP10NNXA81cS3XUj0u2RTWq1dP+fn5Di2Un5+vunUdazLKc/DgQU2bNk2dO3fWoEGDJEmFhYWSZHeYWbpwaLhsvizWw8Oj3DhJslgsFV6zIk6fzldpqXFF75Uu/KUxm/Ou+P1lCiwlysu//M9QXOxYnCQVFFhkPn/+alO75lRWTVA5qIfroSauh5q4Flesh5ub6aI7si55+LhVq1batm2bQxvZvn273a1kHGU2m3X//ferUaNGeu211+TmdiGtsiazqKjI7j0Wi8WmCa1bt265z14uawbLmsOKrAkAAFBbXLIpvPXWW/XZZ5/Z3dLlz3bt2qXMzEz17t27wgnk5eVp7NixysvL09y5c20O65b9d9kh3z8ym83y8/OziS07RPznOEnW2IqsCQAAUFtcsikcNWqUGjdurHHjxumjjz6y27tWVFSkJUuWaNy4cWratKmSkpIqtHGLxaLx48fr8OHDevvtt9W2bVub+YCAALm7u2vPnj12283OzlZwcLB1LCgoSIcOHdLZs2dtYssufgkKCqrwmgAAALXFJZvChg0b6o033pCHh4f+9a9/qUuXLho0aJBGjhyp+Ph4denSRU8//bTc3d31xhtvqGHDhg5v+Pz583r44Ye1a9cuvfbaawoPD7eLadCggXr27KmMjAybZi8jI0MFBQWKjY21jsXGxqq4uFhLliyxjhUVFSk9PV0RERHWi1AqsiYAAEBtcdnH3IWGhmr58uWaO3eu1q5dq3379lnnbrjhBvXv31/33Xefrr/++gpt+MUXX9Rnn32m3r176/fff1dGRoZ1rn79+urbt68kaeLEiRo2bJiSkpKUkJCg48eP67333tMtt9yiyMhI63vCwsIUGxurlJQUmc1mtWzZUkuXLtWxY8c0ffp0m207uiYAAEBtYTIMo0KXzJ49e1b5+fny8fGxuydgRSQlJWnr1q3lzjVv3lyfffaZ9fX27duVkpJifU5xXFycJk2aJG9v29ukWCwWvfrqq1qxYoXOnDmjwMBATZo0qdxGz9E1HeUqVx+ftZRoW/aJy8aFBfgqa7/9eZXl6Rrsr/peFX5M9jXPFa8aq82oh+uhJq6HmrgWV6zHpa4+rnBTiPLRFNY8rvhlrs2oh+uhJq6HmrgWV6zHFd+SBgAAALUDTSEAAABoCgEAAEBTCAAAAF2iKUxNTdX+/futr48dO3bFzwUGAACAa7tkU/jHexLGxMRo3bp11ZIUAAAAqtdFm8KGDRsqNzfX+po71wAAANRcF735XHBwsN59912VlJSoUaNGki7c8Pn8+fOXXDA+Pr5yMwQAAECVu2hTOHXqVP2///f/rI+IM5lMWrx4sRYvXnzRxUwmE00hAADANeiiTWFQUJDWrFmjI0eOyGw2KykpSePHj+fZwAAAADXQJZ9dVqdOHbVu3VqtW7dW165d1b17d3Xr1q26cgMAAEA1cfiBtgsXLqzKPAAAAOBEDjeFklRaWqqlS5dq3bp1Onr0qCSpRYsW6t+/v+Lj4+Xmxr2wAQAArkUON4WFhYUaO3astm/fLpPJJF9fX0nSF198oY0bN2rZsmWaM2eOvLy8qixZAAAAVA2Hd+29+eab2rZtm+69915t3rxZGzdu1MaNG/XNN99o9OjR2rp1q958882qzBUAAABVxOGmcNWqVRowYIAee+wx630LpQs3uX700Uc1YMAArVy5skqSBAAAQNVyuCk8fvz4Ja887tq1q44fP14pSQEAAKB6OdwUNmzYUL/88stF53/55Rc1bNiwUpICAABA9XK4KYyMjFRaWpq+/PJLu7mvvvpKixYtUnR0dKUmBwAAgOrh8NXHDz/8sL766iuNGzdOwcHBat++vSTpxx9/VHZ2tho3bqwHH3ywyhIFAABA1XG4KWzevLk++eQTvfLKK9qwYYN++OEHSVL9+vV1++23a9KkSbrhhhuqLFEAAABUnQrdvPqGG27QK6+8IsMwlJOTI0lq0qSJTCZTlSQHAACA6lGhprCMyWRS06ZNKzsXAAAAOAnPpQMAAABNIQAAAGgKAQAAIJpCAAAAiKYQAAAAoikEAACAKtAU5ufna9SoUdabVgMAAKDmcLgpLC4u1tatW3XmzBlJUkFBgaZOnaqDBw9WWXIAAACoHpdsCh988EG9//77ysrKUlFRkc2cxWLRsmXLdPLkySpNEAAAAFXvkk80OXfunGbPnq28vDy5u7vLZDLp008/lbe3t1q0aCHDMKorTwAAAFShSzaFc+bMkWEY2rdvn77++mvNmDFDK1as0EcffSRvb2+ZTCZ9/vnnatSokYKDg3kGMgAAwDXqsucUmkwmBQUF6e6775YkvfHGG8rIyNDYsWNlGIbS0tI0ePBgdevWTffff3+VJwwAAIDKd8k9hWPGjFHnzp3VuXNn3XjjjZIuNImBgYHy9fXVa6+9prffflsNGzbUtm3btH379mpJGgAAAJXrkk2hp6enFi5cqNdff1116tSRyWTS0qVLJUlt27aVJNWpU0chISEKCQnR6NGjqz5jAAAAVLpLNoVvvvmmJOnw4cP6+uuv9dxzz2nDhg3KyMiQl5eXTCaT1q5dq7p166pjx45yd7/kcgAAAHBRDt2nsHXr1oqLi5Mkvfbaa/r00081YcIEGYahpUuXatiwYeratauSk5OrMlc4icnNpLOWksv+KSl1dqYAAOBKXdGuvTZt2ighIUEzZ87UG2+8IT8/P23ZsoVzCmsoS/F5Ze03Xzaua7C/3L3YWwwAwLXI4X/Bvby8dNddd8nPz89url27dmrXrp1GjBhRqckBAACgejjcFHp7e2v69OnW15dqEgEAAHBtueJjfX9uEgEAAHDtcuhCEwAAANRsTm0KT548qZSUFCUlJalTp04KDAzUli1byo1dv3697rrrLoWEhOjWW29VamqqSkpK7OJyc3P11FNPqUePHgoPD9eoUaOUnZ19VWsCAADUdE5tCg8dOqQ5c+boxIkTCgwMvGjcxo0bNWHCBDVq1EhPPfWU+vbtq9mzZ9sdvi4tLdW4ceO0cuVK3XPPPXr00Ud1+vRpJSUl6ZdffrmiNQEAAGoDp94/pEOHDvrmm2/UuHFjZWZmasKECeXGvfzyy7r55pv17rvvqk6dOpKk+vXr65133lFSUpJat24tSVq9erW+/fZbzZ49W3379pUkDRgwQLfddptSU1P18ssvV3hNAACA2sCpewp9fHzUuHHjS8YcOHBABw4c0NChQ63NmySNGDFCpaWlWrt2rXVszZo18vPzU0xMjHWsSZMmGjBggDIzM1VcXFzhNQEAAGoDl7/Q5IcffpAkdezY0Wbc399ff/nLX6zzkpSdna0OHTrIZDLZxIaEhOjs2bPWQ8gVWRMAAKA2cPnHT5jNF56k4evrazfn6+urkydP2sT26NHDLq7sXoonT55Uu3btKrSmo5o29anwe+y33eCq1zByCtTAp+5l4zw83B2Kq0ist7eXfJt4O7TmtaIyaoLKQz1cDzVxPdTEtVxL9XD5prCwsFCS5OnpaTfn5eWlc+fO2cSWF1c2VrZWRdZ01OnT+SotNSr8vjK+vg1kNudddL6kVLIUX/7K6FJDyssvvGxccXGJQ3EViS0osMh8/rxDa14LLlcTVC/q4XqoieuhJq7FFevh5ma66I4sl28K69a9sIeqqKjIbs5isVjny2LLiysbK4utyJquwlJcom3ZJy4bFxZgv/cTAADgclz+nMKyQ7xlh3z/yGw22zxm72KHfsvGymIrsiYAAEBt4PJNYXBwsCRpz549NuMnTpzQ8ePHrfOSFBQUpO+//16GYXsYd/fu3fL29lbLli0rvCYAAEBt4PJNYfv27dW2bVstXrxY5/9wvtqiRYvk5uam/v37W8diY2N18uRJrV+/3jqWk5Oj1atXKyYmRh4eHhVeEwAAoDZw+jmFb7zxhiTp4MGDkqSMjAzt2LFDDRs21D333CNJeuyxx/T3v/9dY8aMUVxcnPbv36+0tDQNHTpUbdq0sa512223KTw8XI899phGjx6txo0ba9GiRSotLdU//vEPm+06uiYAAEBtYDL+fKy1ml3s8XbNmzfXZ599Zn2dmZmp1NRUHTx4UE2aNNHgwYP1wAMPyN3dtq89c+aMXn75ZWVmZspisSgkJERTpkxRhw4d7Lbh6JqOqOqrj89aHL/QJGu//bmSVxpXkdiuwf6q7+X03zMqjSteNVabUQ/XQ01cDzVxLa5Yj0tdfez0prCmoCmkKUTVoh6uh5q4HmriWlyxHpdqCl3+nEIAAABUPZpCAAAA0BQCAACAphAAAACiKQQAAIBoCgEAACCaQgAAAIimEAAAAKIpBAAAgGgKAQAAIJpCAAAAiKYQAAAAoikEAACAaAoBAAAgmkIAAACIphAAAACiKQQAAIBoCgEAACCaQgAAAIimEAAAAKIpBAAAgGgKAQAAIJpCAAAAiKYQAAAAoikEAACAaAoBAAAgmkIAAACIphAAAACiKQQAAIBoCgEAACDJ3dkJoOYwuZl01lJy2TgvD3e58+sIAAAuhaYQlcZSfF5Z+82Xjesa7C93L/7qAQDgSthfAwAAAJpCAAAA0BQCAABANIUAAAAQTSEAAABEUwgAAADRFAIAAEA0hQAAABBNIQAAAERTCAAAANEUAgAAQLW8KSwqKtKMGTMUHR2t0NBQJSYmavPmzc5OCwAAoNrV6qZwypQpmj9/vu6880498cQTcnNz09ixY/Xtt986OzUAAIBq5e7sBJxl9+7dWrlypaZOnark5GRJUnx8vO644w6lpKQoLS3NuQnWYCY3k85aSi4b5+XhLvda/WsLAADVp9Y2hatXr5aHh4cSEhKsY15eXhoyZIj++7//WydPnpSfn58TM6y5LMXnlbXffNm4rsH+cveqtX9FAQCoVrX2X9zs7Gy1adNG9evXtxkPDQ2VYRjKzs6uUFPo5ma66pwutYZ7HTd51/W47BqVHefMbbvXcauUz/VqOHv7sEU9XA81cT3UxLW4Wj0u2WtUYx4uxWw2y9/f327c19dXknTy5MkKrde4cf3LB11G06Y+l5xv0ayRQ+u0bdG4UuOqYs2KbNuZLlcTVC/q4XqoieuhJq7lWqpHrT1jq7CwUB4e9nurvLy8JEkWi6W6UwIAAHCaWtsU1q1bV8XFxXbjZc1gWXMIAABQG9TaptDX17fcQ8Rm84ULILjIBAAA1Ca1tikMCgrSoUOHdPbsWZvxrKws6zwAAEBtUWubwtjYWBUXF2vJkiXWsaKiIqWnpysiIqLci1AAAABqqlp79XFYWJhiY2OVkpIis9msli1baunSpTp27JimT5/u7PQAAACqlckwDMPZSTiLxWLRq6++qhUrVujMmTMKDAzUpEmTFBkZ6ezUAAAAqlWtbgoBAABwQa09pxAAAAD/QVMIAAAAmkJnKioq0owZMxQdHa3Q0FAlJiZq8+bNzk6rxjl58qRSUlKUlJSkTp06KTAwUFu2bCk3dv369brrrrsUEhKiW2+9VampqSopKbGLy83N1VNPPaUePXooPDxco0aNUnZ2dlX/KDXC7t279eyzzyouLk7h4eG69dZbNXHiRP388892sTt37tTw4cMVFhamqKgoPf/88zp37pxdHN+lK/fdd99pwoQJ6t27t0JDQxUVFaUxY8Zo586ddrHUw3nmzJmjwMBADRo0yG6OulS9LVu2KDAwsNw/Bw8etIm9luvBOYVONGnSJK1du1ajRo1Sq1attHTpUu3Zs0cLFy5Up06dnJ1ejbFlyxbrZ9ykSRN9++23WrBggbp3724Tt3HjRt1///3q0aOH4uLitH//fqWlpWnEiBF66qmnrHGlpaUaMWKE9u/fr9GjR6tx48b68MMPdeLECaWnp6tly5bV/SNeUx588EHt3LlTsbGxCgwMlNlsVlpamgoKCvTxxx+rXbt2kqTs7GwNHTpUN910kxISEnT8+HHNmzdPUVFReuutt2zW5Lt05VatWqXly5crNDRUvr6+ysvL04oVK7Rv3z7NmTNHUVFRkqiHM5nNZt12220yDEMtW7ZURkaGdY66VI+yf0f+9re/qUOHDjZzMTEx8vG58Hzja74eBpwiKyvLCAgIMN577z3rWGFhodG3b19jxIgRzkusBsrLyzNycnIMwzCMdevWGQEBAcY333xjFxcXF2fcddddRklJiXVs5syZRlBQkHHo0CHr2MqVK42AgABj3bp11rHTp08bXbp0MR599NGq+0FqiB07dhgWi8Vm7NChQ0bHjh2NyZMnW8fuu+8+o1evXkZ+fr517KOPPjICAgKMTZs2Wcf4LlW+goICIzIy0hg3bpx1jHo4z+TJk42kpCTjnnvuMe68806bOepSPb755hu7/++X51qvB4ePnWT16tXy8PBQQkKCdczLy0tDhgzRjh07yn0EH66Mj4+PGjdufMmYAwcO6MCBAxo6dKjq1KljHR8xYoRKS0u1du1a69iaNWvk5+enmJgY61iTJk00YMAAZWZmlvtMbfxHRESEPD09bcZat26t9u3bWw/D5Ofna9OmTYqPj1f9+vWtcYMGDZK3t7c+/fRT6xjfpcpXr149NWnSRLm5uZKohzPt3r1by5cv19SpU+3mqItz5Ofnl3taUU2oB02hk2RnZ6tNmzY2f3EkKTQ0VIZhcH5aNfvhhx8kSR07drQZ9/f311/+8hfrvHShdh06dJDJZLKJDQkJ0dmzZ/XLL79UfcI1jGEYOnXqlLV537dvn0pKSuzq4enpqeDgYJvvB9+lypGfn6+cnBz99NNPmjlzpvbv36+ePXtKoh7OYhiGnnvuOcXHxys4ONhunrpUv0cffVSdO3dWWFiYRo8erX379lnnakI9au0TTZzNbDaX+yg9X19fSeK3tmpmNpsl/efz/yNfX1+bepjNZvXo0cMuzs/PT9KF2pWdFwfHLF++XCdOnNDEiRMlXb4eu3btsr7mu1Q5Hn/8ca1Zs0aS5OHhoWHDhmn8+PGSqIezLFu2TAcOHNDs2bPLnacu1cfDw0O33XabbrnlFjVu3Fj79u3TvHnzNGLECH388cdq06ZNjagHTaGTFBYWysPDw27cy8tL0oWnraD6FBYWSpLdYU3pQk3+eOVYYWFhuXFlY2VrwTEHDx7UtGnT1LlzZ+uVlZerxx8/Y75LlWPChAkaOnSojh8/royMDBUVFam4uFienp7Uwwny8/P1yiuvaNy4cdZfOP+MulSfiIgIRUREWF/HxMSoT58+Gjx4sFJTU/XKK6/UiHpw+NhJ6tatW+65Z2V/Ecr+YqB61K1bV9KFWwT8mcVisc6XxZYXVzb2x1hcmtls1v33369GjRrptddek5vbhf8lVbQefJeuXmBgoKKiojR48GC9++67+v77763nsVGP6vfmm2/Kw8ND995770VjqItzBQUFqWfPnvrmm28k1Yx60BQ6yZ8PSZYp2/18sd8MUTXKdtmXff5/ZDabbepxsdqVjVE7x+Tl5Wns2LHKy8vT3LlzbQ65VEY9+C5dOQ8PD8XExGjt2rUqLCykHtXs5MmTmj9/vkaMGKFTp07p6NGjOnr0qCwWi4qLi3X06FGdOXOGuriAZs2a6cyZM5Jqxv+3aAqdJCgoSIcOHdLZs2dtxrOysqzzqD5lJ3Hv2bPHZvzEiRM6fvy4zUneQUFB+v7772X86Rafu3fvlre3N/cpdIDFYtH48eN1+PBhvf3222rbtq3NfEBAgNzd3e3qUVRUpOzsbLt68F2qfIWFhTIMQ2fPnqUe1ez06dMqLi5WSkqKYmJirH+ysrJ08OBBxcTEaM6cOdTFBRw5csR6gVxNqAdNoZPExsaquLhYS5YssY4VFRUpPT1dERER5Z6AiqrTvn17tW3bVosXL9b58+et44sWLZKbm5v69+9vHYuNjdXJkye1fv1661hOTo5Wr16tmJiYcs8TwX+cP39eDz/8sHbt2qXXXntN4eHhdjENGjRQz549lZGRYfM/zYyMDBUUFCg2NtY6xnfp6uTk5NiN5efna82aNWrWrJmaNm1KPapZixYtNHv2bLs/7du3V/PmzTV79mzFx8dTl2pU3vdk+/bt2rJli6KjoyXVjP9v8UQTJ3rooYe0fv16/e1vf1PLli2tdzOfP3++Onfu7Oz0apQ33nhD0oWLGv73f/9XgwcPVosWLdSwYUPdc889kqQNGzbo73//u90TTYYOHapnnnnGutb58+c1YsQI/fjjj9YnmixatEi//vqr0tPT1apVK2f8iNeM//qv/9KCBQvUu3dvDRgwwGaufv366tu3ryTp+++/17Bhw9S+fXvrkwHee+89de/eXXPmzLF5H9+lKzdq1Ch5eXmpU6dO8vX1tf49Pn78uGbOnKm4uDhJ1MMVJCUlKTc31+aJJtSleowaNUr16tVTp06d1LhxY/34449avHixGjRooI8//lg33HCDpGu/HjSFTmSxWPTqq69qxYoVOnPmjAIDAzVp0iRFRkY6O7UaJzAwsNzx5s2b67PPPrO+zszMVGpqqg4ePKgmTZpo8ODBeuCBB+Tubnuh/pkzZ/Tyyy8rMzNTFotFISEhmjJlit3jj2AvKSlJW7duLXfuz/XYvn27UlJS9MMPP8jHx0dxcXGaNGmSvL29bd7Hd+nKffzxx8rIyNCBAweUm5urBg0aKDw8XKNHj1a3bt1sYqmHc5XXFErUpTosWLBAK1as0C+//KL8/Hw1adJE0dHR+sc//mFtCMtcy/WgKQQAAADnFAIAAICmEAAAAKIpBAAAgGgKAQAAIJpCAAAAiKYQAAAAoikEAACAaAoBANeAo0ePKjAwULNmzXJ2KkCNRVMIoFqdO3dO77//vkaMGKFu3bqpQ4cOioyM1NixY5Wenq6SkhJnp+iysrOzNWvWLB09etTh98yaNUuBgYH67rvvqjCzypGbm6tZs2Zpy5Ytzk4FqJVoCgFUm59//lnx8fGaPn26vLy8NG7cOE2bNk3JyckqKSnR1KlTNXPmTGen6bKys7OVmpqqf//7385OpUrk5uYqNTX1oo9BBFC13C8fAgBXr7CwUPfff7+OHj2qWbNmqX///jbz48aN0+7du6+JPVoAUBOxpxBAtViyZIkOHTqke++9164hLBMaGqqRI0fajGVmZmrYsGEKDw9Xp06dNGzYMGVmZtq9t0+fPkpKStLevXuVnJysTp06qWfPnnrxxRdVUlIii8Wil156Sb169VJISIhGjhypgwcP2qyRnp6uwMBAbd68Wampqerdu7dCQ0OVkJCgXbt2SZK2bt2q4cOHKzw8XNHR0Zo9e3a5P8t3332nCRMmqHv37urYsaNuu+02vfnmm3aHx5OSktSnTx+dOHFCkyZNUteuXRUWFqYxY8bo0KFD1rhZs2Zp6tSpkqRRo0YpMDBQgYGBmjJlymU+ecdt2rRJo0ePVpcuXRQSEqKBAwdq0aJFdnFln/XBgwc1btw4derUSZ07d9aDDz4os9lsF793716NHj1a4eHh6t69uyZPnqycnByb/Lds2aKYmBhJUmpqqvXn69Onj916GzZs0ODBgxUSEqLo6Gi99NJLnHYAVII6zzzzzDPOTgJAzTdjxgwdO3ZML7/8sho1auTQe9LS0jR58mR5e3tr1KhR6tq1q3bt2qW0tDT5+fmpY8eO1tj58+eroKBAS5YsUffu3TVw4ECVlJTok08+UVFRkdLS0vTrr78qISFBN998s9auXauNGzdq5MiRMplMki4cnl2/fr0OHDign376SYmJierSpYs2btyo9PR0tWvXTv/85z/Vr18/xcbG6tSpU0pPT1fLli0VFBRkzeXzzz/XmDFjJEnDhg1T//79ZTKZtGDBAh04cEADBgywxi5dulQnTpzQypUr1axZM919991q3bq1Vq1apa+++krDhw+Xm5ubGjVqJMMw9P3332v8+PFKTExUv379FBUVpWbNml30M9y6dau2bt2qxMRE+fv7XzRu8eLFmjhxopo2barExET17t1bubm5mjdvngoKChQdHW3zWZ87d04fffSRwsLCNGjQIDVp0kTLly/Xvn37NGjQIGvs4cOHNXToUP3666/WBnjv3r1avHixzGazgoOD1bdvX9WtW1f+/v766quv1K9fP40fP179+vVTCWAy4AAABs1JREFUTEyM2rZtq9zcXC1YsEDnzp3TsmXLdPvtt+u2225Tbm6uMjIy5OXlpS5dujj09wrARRgAUA26detmREREOBz/+++/G+Hh4Ubfvn2NvLw863heXp4RExNjhIeHG2fOnLGO9+7d2wgICDBWrVpls85dd91lBAYGGuPHjzdKS0ut4/PnzzcCAgKML774wjr2ySefGAEBAUZ8fLxhsVis45mZmUbA/2/v/kKaev84gL9Vmuy0bPZnRc7mAk9WTqEutE0zSqIWFrYyKhcS5Y2QYVEQEWYgLYLACszVgtALqSCCXUiB0T8K7Ua86h/ZborklLrm0vL5XvTd8LiZ06/OHz/eL/Bin+dznn3O2cWenec5j7IsVq9eLbq6usLxnz9/CpvNJsrKysKxYDAorFar2L9/vxgeHlbVcuvWLSHLsnj58mU4Vl5eLmRZFk1NTapct9s9bn2jj59IQ0ODkGVZVfdYX758EdnZ2aKmpiai7fz58yIrK0t8+vQpHAtda6/Xq8qtra0VsiyL9+/fh2NHjx4VsiyLzs5OVW51dbWQZVmcOnUqHPP5fEKWZdHQ0BBRR6gtNzdX+Hy+cHxkZERs375d2Gy2v1wFIooFp4+JKC78fj/mzp0bc/7z588RCATgdDqh0+nCcZ1OB6fTiUAggBcvXqiOWbJkieouHACsXbsWQgg4nc7wHUEA4btKPT09Ee+9b98+aDSaiNycnBxYLJZwXKPRwGKx4OPHj6q6e3t7sWvXLvT390NRlPDfhg0bwjmjJSYm4uDBg6pYfn7+uPVNt7a2NgwNDWH37t2qehVFwaZNmzAyMhJxrQ0GA+x2+19r/v37N548eYKcnBysW7dOlXvo0KEp1bp582YYjcbw64SEBOTl5eHr16/48ePHlPokoj/4oAkRxYVOp5vUl3Zo25XMzMyItlDM5/Op4qMHCyGhqeqxbSkpKQCA79+/RxyTnp4eUx+httF9hNYpnj59OiI3pLe3V/XaYDAgOTlZFdPr9ePWN91CNVdUVIybM7bmsdcIiKxZURQEAgGYzeaI3GixWEz0vpP54UFEahwUElFcZGZmoqOjAz6fL+oX+3RISkoaty0xMfrEiBAi5ty/9T+2v5MnT2LVqlVRcwwGQ8z9RqtvuoXew+VyRdQWMvYzm62aZ/taEf0/46CQiOJiy5Yt6OjowJ07d1BTUzNhfmgQ8vbtW6xfv17V9u7dO1XO/5KMjAwAgFarhdVqnda+R09/T6dQzampqdNa84IFCyBJkuop6pBosZk6PyKKDdcUElFc7NmzB2azGR6PJ+qWMgDQ3d2NlpYWAIDNZoMkSWhubobf7w/n+P1+NDc3Q5Ik2Gy2uNQ+GQUFBVi4cCHcbnfUqd9gMKg6n8mQJAkA0NfX959qHGvbtm3QaDS4cuUKgsFgRPvAwACGhoYm3W9SUhIKCwvR1dWF169fq9o8Hk9E/kydHxHFhncKiSgutFotrl+/jsrKSlRVVaGgoABWqxV6vR6KouDVq1d49uwZDh8+DODPmr8TJ06grq4OZWVlKC0tBfBnC5eenh7U1dVh3rx5s3lKUUmSBJfLhaqqKmzduhUOhwMmkwn9/f348OEDHj58iKtXryIvL2/SfVssFiQmJqKxsRF9fX2QJAlGoxG5ubkTHnvv3j08ffo0Ir5mzRoUFRWhtrYWZ86cgd1ux44dO5CWlgZFUfDmzRs8evQIXq836prKiRw7diz8uZaXl2Pp0qV4/PgxFEUBoL47mJqaCpPJBK/Xi/T0dCxatAharTbqXoVENP04KCSiuDGZTLh//z5aW1vR1taGxsZGBAIBzJ8/H9nZ2bhw4QJKSkrC+QcOHIDBYMDNmzfDm0RnZWXh2rVrKC4unq3TmFBhYSHu3r2LpqYmPHjwAN++fUNKSgqWL1+OiooKrFy5ckr9Llu2DPX19XC73Th37hyGh4dRWloa06Aw2ibUALB3714UFRXB4XAgIyMDHo8Hra2tGBgYgF6vh9lsRnV1NRYvXjylmlesWIGWlha4XC7cvn0bycnJ2LhxI86ePYvi4uKIB2wuXbqE+vp6XL58GYODg0hLS+OgkChOEgRX5hIRUZx1d3fD4XDg+PHjqKysnO1yiAhcU0hERDNs7DpFIQRu3LgBANP+MA4RTR2nj4mIaEbt3LkT+fn5kGUZg4ODaG9vR2dnJ+x2u+pfFRLR7OL0MRERzaiLFy+ivb0dnz9/xq9fv2A0GlFSUoIjR45gzpw5s10eEf2Lg0IiIiIi4ppCIiIiIuKgkIiIiIjAQSERERERgYNCIiIiIgIHhUREREQEDgqJiIiICMA/TqqpFgj+ruUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sPN9UmdHCv1w"
      },
      "source": [
        "max_len = 300"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7oQ69-4wU7v-"
      },
      "source": [
        "### 3.3. Perform Tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tIWAoWL2RK1p"
      },
      "source": [
        "Now we're ready to perform the real tokenization.\n",
        "\n",
        "The `tokenizer.encode_plus` function combines multiple steps for us:\n",
        "\n",
        "1. Split the sentence into tokens.\n",
        "2. Add the special `[CLS]` and `[SEP]` tokens.\n",
        "3. Map the tokens to their IDs.\n",
        "4. Pad or truncate all sentences to the same length.\n",
        "5. Create the attention masks which explicitly differentiate real tokens from `[PAD]` tokens.\n",
        "\n",
        "The first four features are in `tokenizer.encode`, but I'm using `tokenizer.encode_plus` to get the fifth item (attention masks). \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ijEWKSYUpHh",
        "outputId": "727f6fcc-da91-4863-98d8-3e24a0080a8b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Tokenize a dataset of comments.\n",
        "\n",
        "# Parameters:\n",
        "#  `comments` - List of comments, represented as strings.\n",
        "#  `labels` - List of integer labels for the corresponding comments.\n",
        "#  `max_len` - Truncate all of the comments down to this length.\n",
        "    \n",
        "# Returns:\n",
        "#  `input_ids` - All of the comments represented as lists of token IDs,\n",
        "#               padded out to `max_len`, and cast as a PyTorch tensor.\n",
        "#  `labels` - The labels for the corresponding comments, formatted as \n",
        "#              a PyTorch tensor.\n",
        "#  `attention_masks` - PyTorch tensor with the same dimensions as\n",
        "#                     `input_ids`. For each token, simply indicates whether\n",
        "#                      it is padding or not.\n",
        "\n",
        "comments = train.data\n",
        "labels   = train.target\n",
        "\n",
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "attention_masks = []\n",
        "\n",
        "print('Tokenizing {:,} comments...'.format(len(comments)))\n",
        "\n",
        "    # For every comment (\"sentence\")...\n",
        "for sent in comments:\n",
        "\n",
        "        # Report progress.\n",
        "        if ((len(input_ids) % 500) == 0):\n",
        "            print('  Tokenized {:,} comments.'.format(len(input_ids)))\n",
        "\n",
        "        # `encode_plus` will:\n",
        "        #   (1) Tokenize the sentence.\n",
        "        #   (2) Prepend the `[CLS]` token to the start.\n",
        "        #   (3) Append the `[SEP]` token to the end.\n",
        "        #   (4) Map tokens to their IDs.\n",
        "        #   (5) Pad or truncate the sentence to `max_length`\n",
        "        #   (6) Create attention masks for [PAD] tokens.\n",
        "        encoded_dict = tokenizer.encode_plus(\n",
        "                            sent,                      # Sentence to encode.\n",
        "                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                            max_length = max_len,      # Pad & truncate all sentences.\n",
        "                            pad_to_max_length = True,\n",
        "                            return_attention_mask = True,   # Construct attn. masks.\n",
        "                            return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                    )\n",
        "        \n",
        "        # Add the encoded sentence to the list.    \n",
        "        input_ids.append(encoded_dict['input_ids'])\n",
        "        \n",
        "        # And its attention mask (simply differentiates padding from non-padding).\n",
        "        attention_masks.append(encoded_dict['attention_mask'])\n",
        "\n",
        "# Convert the lists into tensors.\n",
        "input_ids = torch.cat(input_ids, dim=0)\n",
        "attention_masks = torch.cat(attention_masks, dim=0)\n",
        "\n",
        "# Convert the labels to a tensor.\n",
        "labels = torch.tensor(torch.from_numpy(labels))\n",
        "print(labels)\n",
        "\n",
        "(train_input_ids, \n",
        " train_labels, \n",
        " train_attention_masks) = (input_ids, labels, attention_masks)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Tokenizing 10,138 comments...\n",
            "  Tokenized 0 comments.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:1944: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "  Tokenized 500 comments.\n",
            "  Tokenized 1,000 comments.\n",
            "  Tokenized 1,500 comments.\n",
            "  Tokenized 2,000 comments.\n",
            "  Tokenized 2,500 comments.\n",
            "  Tokenized 3,000 comments.\n",
            "  Tokenized 3,500 comments.\n",
            "  Tokenized 4,000 comments.\n",
            "  Tokenized 4,500 comments.\n",
            "  Tokenized 5,000 comments.\n",
            "  Tokenized 5,500 comments.\n",
            "  Tokenized 6,000 comments.\n",
            "  Tokenized 6,500 comments.\n",
            "  Tokenized 7,000 comments.\n",
            "  Tokenized 7,500 comments.\n",
            "  Tokenized 8,000 comments.\n",
            "  Tokenized 8,500 comments.\n",
            "  Tokenized 9,000 comments.\n",
            "  Tokenized 9,500 comments.\n",
            "  Tokenized 10,000 comments.\n",
            "tensor([4, 3, 1,  ..., 3, 4, 5])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:60: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TT4m3oytdVMK"
      },
      "source": [
        "Now we'll use the above function to actually perform the tokenization of the training comments."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y1CRcdfqdfGi",
        "outputId": "0b7b4909-4621-46f5-d27d-66f1be0bf690",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Print sentence 0, now as a list of IDs.\n",
        "print('Original: ', train.data[0])\n",
        "print('Token IDs:', train_input_ids[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original:  Says the Annies List political group supports third-trimester abortions on demand.. dwayne-bohac\n",
            "Token IDs: tensor([  101,  2758,  1996,  8194,  2015,  2862,  2576,  2177,  6753,  2353,\n",
            "         1011, 12241, 20367, 11324,  2015,  2006,  5157,  1012,  1012,  1040,\n",
            "         4576,  2638,  1011,  8945,  3270,  2278,   102,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G7npHp8DaNq-"
      },
      "source": [
        "Let's also tokenize the test dataset "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6txnuEfuQENJ",
        "outputId": "8774bcdf-ee87-4b52-e20d-f0e83de0495b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Tokenize a dataset of comments.\n",
        "\n",
        "# Parameters:\n",
        "#  `comments` - List of comments, represented as strings.\n",
        "#  `labels` - List of integer labels for the corresponding comments.\n",
        "#  `max_len` - Truncate all of the comments down to this length.\n",
        "    \n",
        "# Returns:\n",
        "#  `input_ids` - All of the comments represented as lists of token IDs,\n",
        "#               padded out to `max_len`, and cast as a PyTorch tensor.\n",
        "#  `labels` - The labels for the corresponding comments, formatted as \n",
        "#              a PyTorch tensor.\n",
        "#  `attention_masks` - PyTorch tensor with the same dimensions as\n",
        "#                     `input_ids`. For each token, simply indicates whether\n",
        "#                      it is padding or not.\n",
        "\n",
        "comments = test.data\n",
        "labels   = test.target\n",
        "\n",
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "attention_masks = []\n",
        "\n",
        "print('Tokenizing {:,} comments...'.format(len(comments)))\n",
        "\n",
        "    # For every comment (\"sentence\")...\n",
        "for sent in comments:\n",
        "\n",
        "        # Report progress.\n",
        "        if ((len(input_ids) % 500) == 0):\n",
        "            print('  Tokenized {:,} comments.'.format(len(input_ids)))\n",
        "\n",
        "        # `encode_plus` will:\n",
        "        #   (1) Tokenize the sentence.\n",
        "        #   (2) Prepend the `[CLS]` token to the start.\n",
        "        #   (3) Append the `[SEP]` token to the end.\n",
        "        #   (4) Map tokens to their IDs.\n",
        "        #   (5) Pad or truncate the sentence to `max_length`\n",
        "        #   (6) Create attention masks for [PAD] tokens.\n",
        "        encoded_dict = tokenizer.encode_plus(\n",
        "                            sent,                      # Sentence to encode.\n",
        "                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                            max_length = max_len,      # Pad & truncate all sentences.\n",
        "                            pad_to_max_length = True,\n",
        "                            return_attention_mask = True,   # Construct attn. masks.\n",
        "                            return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                    )\n",
        "        \n",
        "        # Add the encoded sentence to the list.    \n",
        "        input_ids.append(encoded_dict['input_ids'])\n",
        "        \n",
        "        # And its attention mask (simply differentiates padding from non-padding).\n",
        "        attention_masks.append(encoded_dict['attention_mask'])\n",
        "\n",
        "# Convert the lists into tensors.\n",
        "input_ids = torch.cat(input_ids, dim=0)\n",
        "attention_masks = torch.cat(attention_masks, dim=0)\n",
        "\n",
        "# Convert the labels to a tensor.\n",
        "labels = torch.tensor(torch.from_numpy(labels))\n",
        "print(labels)\n",
        "\n",
        "(test_input_ids, \n",
        " test_labels, \n",
        " test_attention_masks) = (input_ids, labels, attention_masks)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tokenizing 1,250 comments...\n",
            "  Tokenized 0 comments.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:1944: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "  Tokenized 500 comments.\n",
            "  Tokenized 1,000 comments.\n",
            "tensor([0, 4, 4,  ..., 2, 2, 4])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:60: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wl12-BD637Ty"
      },
      "source": [
        "# Part II - BERT Fine-Tuning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bwa6Rts-02-"
      },
      "source": [
        "## S4. Train Our Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xYQ3iLO08SX"
      },
      "source": [
        "Now that our input data is properly formatted, it's time to fine tune the BERT model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cKetqAKJiwpj"
      },
      "source": [
        "### `check_gpu_mem`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DDZZgG6ei1wF"
      },
      "source": [
        "The following function will return a table showing the GPU's current memory usage versus its total memory. It uses the `nvidia-smi` command line tool to retrieve this.\n",
        "\n",
        "Note that the memory values are in \"MiB\". 1 MiB = 2^20 bytes = 1,048,576 bytes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kRJBBgfZiwpl"
      },
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "import csv\n",
        "\n",
        "def check_gpu_mem():\n",
        "    '''\n",
        "    Uses Nvidia's SMI tool to check the current GPU memory usage.\n",
        "    Reported values are in \"MiB\". 1 MiB = 2^20 bytes = 1,048,576 bytes.\n",
        "    '''\n",
        "    \n",
        "    # Run the command line tool and get the results.\n",
        "    buf = os.popen('nvidia-smi --query-gpu=memory.total,memory.used --format=csv')\n",
        "\n",
        "    # Use csv module to read and parse the result.\n",
        "    reader = csv.reader(buf, delimiter=',')\n",
        "\n",
        "    # Use a pandas table just for nice formatting.\n",
        "    df = pd.DataFrame(reader)\n",
        "\n",
        "    # Use the first row as the column headers.\n",
        "    new_header = df.iloc[0] #grab the first row for the header\n",
        "    df = df[1:] #take the data less the header row\n",
        "    df.columns = new_header #set the header row as the df header\n",
        "\n",
        "    # Display the formatted table.\n",
        "    #display(df)\n",
        "\n",
        "    return df\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6TKgyUzPIQc"
      },
      "source": [
        "### 4.1. Load Pre-Trained Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXYitPoE-cjH"
      },
      "source": [
        "For text classification, we'll be using [BertForSequenceClassification](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification). \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7xFPk4RDHcnH"
      },
      "source": [
        "\n",
        "**Model Variant**\n",
        "\n",
        "Using \"bert-base-uncased\" for this example, the version that has only lowercase letters (\"uncased\") and is the smaller version of the two (\"base\" vs \"large\").\n",
        "\n",
        "During fine-tuning, the entire pre-trained BERT model and the additional untrained classification layer is trained. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFsCTp_mporB",
        "outputId": "771cef4a-e3b4-4163-9e96-9ca5755ef44c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from transformers import BertForSequenceClassification, AdamW, BertConfig\n",
        "\n",
        "# Load BertForSequenceClassification, the pretrained BERT model with a single \n",
        "# linear classification layer on top. \n",
        "model = BertForSequenceClassification.from_pretrained(\n",
        "    \"bert-base-uncased\", # Use the 12-layer BERT model, with an uncased vocab.\n",
        "    num_labels = 6, # For our 6 labels!\n",
        "    output_attentions = False, # Whether the model returns attentions weights.\n",
        "    output_hidden_states = False, # Whether the model returns all hidden-states.\n",
        ")\n",
        "\n",
        "# Tell pytorch to run this model on the GPU.\n",
        "desc = model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lbmX98ovPy2d",
        "outputId": "db79f3c7-b1cd-4437-b6de-b2eb6793da15",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "check_gpu_mem()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>memory.total [MiB]</th>\n",
              "      <th>memory.used [MiB]</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>16280 MiB</td>\n",
              "      <td>1395 MiB</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "0 memory.total [MiB]  memory.used [MiB]\n",
              "1          16280 MiB           1395 MiB"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRp4O7D295d_"
      },
      "source": [
        "### 4.2. Splitting off a Validation Set\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qu0ao7p8rb06"
      },
      "source": [
        "This dataset already has a train / test split, but we're going to further divide up our training set to use 90% for training and 10% for *validation*. This validation set will help us detect over-fitting during the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEgLpFVlo1Z-",
        "outputId": "8228f596-95ff-4b16-8989-a9a46ecbc655",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from torch.utils.data import TensorDataset, random_split\n",
        "\n",
        "# Combine the training inputs into a TensorDataset.\n",
        "dataset = TensorDataset(train_input_ids, train_attention_masks, train_labels)\n",
        "\n",
        "# Create a 90-10 train-validation split.\n",
        "\n",
        "# Calculate the number of samples to include in each set.\n",
        "train_size = int(0.9 * len(dataset))\n",
        "val_size = len(dataset) - train_size\n",
        "\n",
        "# Divide the dataset by randomly selecting samples.\n",
        "train_dataset, val_dataset = random_split(dataset, [train_size, val_size])\n",
        "\n",
        "print('{:>5,} training samples'.format(train_size))\n",
        "print('{:>5,} validation samples'.format(val_size))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "9,124 training samples\n",
            "1,014 validation samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ejHQThm_uVnB"
      },
      "source": [
        "### 4.3. Batch Size & DataLoaders"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dD9i6Z2pG-sN"
      },
      "source": [
        "We'll also create an iterator for our dataset using the torch DataLoader class. This helps save on memory during training because, unlike a for loop, with an iterator the entire dataset does not need to be loaded into memory.\n",
        "\n",
        "This is also the point at which we must specify our **batch size**, since this is integral to the data loaders."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XGUqOCtgqGhP"
      },
      "source": [
        "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
        "\n",
        "# The DataLoader needs to know our batch size for training, so we specify it \n",
        "# here. For fine-tuning BERT on a specific task, the authors recommend a batch \n",
        "# size of 16 or 32.\n",
        "batch_size = 16\n",
        "\n",
        "# Create the DataLoaders for our training and validation sets.\n",
        "# We'll take training samples in random order. \n",
        "train_dataloader = DataLoader(\n",
        "            train_dataset,  # The training samples.\n",
        "            sampler = RandomSampler(train_dataset), # Select batches randomly\n",
        "            batch_size = batch_size # Trains with this batch size.\n",
        "        )\n",
        "\n",
        "# For validation the order doesn't matter, so we'll just read them sequentially.\n",
        "validation_dataloader = DataLoader(\n",
        "            val_dataset, # The validation samples.\n",
        "            sampler = SequentialSampler(val_dataset), # Pull out batches sequentially.\n",
        "            batch_size = batch_size # Evaluate with this batch size.\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qRWT-D4U_Pvx"
      },
      "source": [
        "### 4.4. Optimizer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk"
      },
      "source": [
        "Our `model` object handles the execution of a forward pass, and the calculation of gradients during training. \n",
        "\n",
        "The actual updates to the model's weights, however, are performed by an Optimizer object. Here, we create that object and give it a reference to our model's parameters, as well as set some of our training hyperparameters.\n",
        "\n",
        "For the purposes of fine-tuning, the authors recommend choosing from the following values:\n",
        "- Batch size: 16, 32 \n",
        "- Learning rate (Adam): 5e-5, 3e-5, 2e-5  (We'll use 2e-5).\n",
        "- Number of epochs: 2, 3, 4  (We'll use 4). This is a parameter to the \"learning rate scheduler\" in the section.\n",
        "\n",
        "The epsilon parameter `eps = 1e-8` is \"a very small number to prevent any division by zero in the implementation\" "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GLs72DuMODJO"
      },
      "source": [
        "# Note: AdamW is a class from the huggingface library (as opposed to pytorch) \n",
        "# I believe the 'W' stands for 'Weight Decay fix\"\n",
        "optimizer = AdamW(model.parameters(),\n",
        "                  lr = 2e-5, # args.learning_rate - default is 5e-5\n",
        "                  eps = 1e-8 # args.adam_epsilon  - default is 1e-8.\n",
        "                )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_iaG0A5quuqz"
      },
      "source": [
        "### 4.5. Epochs & Learning Rate Scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGeSUdv_wDgF"
      },
      "source": [
        "The learning rate scheduler is responsible for updating the learning rate over the course of the training. Generally speaking, you want the learning rate to gradually get smaller and smaller so that training makes gradually finer adjustments to the weights. \n",
        "\n",
        "This decay needs to happen *across all of the training epochs*, so this is where we need to specify the number of epochs we want to train for. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-p0upAhhRiIx"
      },
      "source": [
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "# Number of training epochs (authors recommend between 2 and 4)\n",
        "epochs = 4\n",
        "\n",
        "# Total number of training steps is number of batches * number of epochs.\n",
        "total_steps = len(train_dataloader) * epochs\n",
        "\n",
        "# Create the learning rate scheduler.\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps = 0, # Default value in run_glue.py\n",
        "                                            num_training_steps = total_steps)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5"
      },
      "source": [
        "**Helper Functions**\n",
        "\n",
        "In each pass, we will train the model on our full training set, and then measure it's accuracy on our 10% holdout validation set. We'll define a helper function for calculating accuracy on the *validation* set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cQNvaZ9bnyy"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNhRtWPXH9C3"
      },
      "source": [
        "Helper function for formatting elapsed times.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gpt6tR83keZD"
      },
      "source": [
        "import time\n",
        "import datetime\n",
        "\n",
        "def format_time(elapsed):\n",
        "    '''\n",
        "    Takes a time in seconds and returns a string hh:mm:ss\n",
        "    '''\n",
        "    # Round to the nearest second.\n",
        "    elapsed_rounded = int(round((elapsed)))\n",
        "    \n",
        "    # Format as hh:mm:ss\n",
        "    return str(datetime.timedelta(seconds=elapsed_rounded))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqfmWwUR_Sox"
      },
      "source": [
        "### 4.5. Training Loop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfNIhN19te3N"
      },
      "source": [
        "\n",
        "\n",
        "For addressing our **GPU memory issue**, Added some calls to `check_gpu_mem` during the training process. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WiDKq4cLQG6H",
        "outputId": "e0712043-7884-4ca4-d57f-218848cb60cd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# This training code is based on the `run_glue.py` script here:\n",
        "# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128\n",
        "\n",
        "# Set the seed value all over the place to make this reproducible.\n",
        "seed_val = 42\n",
        "\n",
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "\n",
        "# We'll store a number of quantities such as training and validation loss, \n",
        "# validation accuracy, and timings.\n",
        "training_stats = []\n",
        "\n",
        "# For each epoch...\n",
        "for epoch_i in range(0, epochs):\n",
        "    \n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    \n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    print(\"\")\n",
        "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
        "    print('Training...')\n",
        "\n",
        "    # Measure how long the training epoch takes.\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_train_loss = 0\n",
        "\n",
        "    # Put the model into training mode. Don't be mislead--the call to \n",
        "    # `train` just changes the *mode*, it doesn't *perform* the training.\n",
        "    # `dropout` and `batchnorm` layers behave differently during training\n",
        "    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)\n",
        "    model.train()\n",
        "\n",
        "    # For each batch of training data...\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "\n",
        "        # Progress update every 50 batches.\n",
        "        if step % 50 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed = format_time(time.time() - t0)\n",
        "            \n",
        "            # Report progress.\n",
        "            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n",
        "\n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using the \n",
        "        # `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "\n",
        "        # Check GPU memory for the first couple steps.\n",
        "        if step < 2:\n",
        "            print('\\n  Step {:} GPU Memory Use:'.format(step))\n",
        "            df = check_gpu_mem()    \n",
        "            print('    Before forward-pass: {:}'.format(df.iloc[0, 1]))\n",
        "\n",
        "        # Always clear any previously calculated gradients before performing a\n",
        "        # backward pass. PyTorch doesn't do this automatically because \n",
        "        # accumulating the gradients is \"convenient while training RNNs\". \n",
        "        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)\n",
        "        model.zero_grad()        \n",
        "\n",
        "        # Perform a forward pass (evaluate the model on this training batch).\n",
        "        # The documentation for this `model` function is here: \n",
        "        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "        # It returns different numbers of parameters depending on what arguments\n",
        "        # arge given and what flags are set. For our useage here, it returns\n",
        "        # the loss (because we provided labels) and the \"logits\"--the model\n",
        "        # outputs prior to activation.\n",
        "        loss, logits = model(b_input_ids, \n",
        "                             token_type_ids=None, \n",
        "                             attention_mask=b_input_mask, \n",
        "                             labels=b_labels)\n",
        "\n",
        "        # Report GPU memory use for the first couple steps.\n",
        "        if step < 2:\n",
        "            df = check_gpu_mem()    \n",
        "            print('     After forward-pass: {:}'.format(df.iloc[0, 1]))\n",
        "\n",
        "        # Accumulate the training loss over all of the batches so that we can\n",
        "        # calculate the average loss at the end. `loss` is a Tensor containing a\n",
        "        # single value; the `.item()` function just returns the Python value \n",
        "        # from the tensor.\n",
        "        total_train_loss += loss.item()\n",
        "\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "\n",
        "        # Report GPU memory use for the first couple steps.\n",
        "        if step < 2:\n",
        "            df = check_gpu_mem()    \n",
        "            print('    After gradient calculation: {:}'.format(df.iloc[0, 1]))\n",
        "\n",
        "        # Clip the norm of the gradients to 1.0.\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "\n",
        "        # Update parameters and take a step using the computed gradient.\n",
        "        # The optimizer dictates the \"update rule\"--how the parameters are\n",
        "        # modified based on their gradients, the learning rate, etc.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_train_loss = total_train_loss / len(train_dataloader)            \n",
        "    \n",
        "    # Measure how long this epoch took.\n",
        "    training_time = format_time(time.time() - t0)\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n",
        "    print(\"  Training epcoh took: {:}\".format(training_time))\n",
        "        \n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"Running Validation...\")\n",
        "\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Put the model in evaluation mode--the dropout layers behave differently\n",
        "    # during evaluation.\n",
        "    model.eval()\n",
        "\n",
        "    # Tracking variables \n",
        "    total_eval_accuracy = 0\n",
        "    total_eval_loss = 0\n",
        "    nb_eval_steps = 0\n",
        "\n",
        "    # Evaluate data for one epoch\n",
        "    for batch in validation_dataloader:\n",
        "        \n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using \n",
        "        # the `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "        \n",
        "        # Tell pytorch not to bother with constructing the compute graph during\n",
        "        # the forward pass, since this is only needed for backprop (training).\n",
        "        with torch.no_grad():        \n",
        "\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # token_type_ids is the same as the \"segment ids\", which \n",
        "            # differentiates sentence 1 and 2 in 2-sentence tasks.\n",
        "            # The documentation for this `model` function is here: \n",
        "            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "            # Get the \"logits\" output by the model. The \"logits\" are the output\n",
        "            # values prior to applying an activation function like the softmax.\n",
        "            (loss, logits) = model(b_input_ids, \n",
        "                                   token_type_ids=None, \n",
        "                                   attention_mask=b_input_mask,\n",
        "                                   labels=b_labels)\n",
        "            \n",
        "        # Accumulate the validation loss.\n",
        "        total_eval_loss += loss.item()\n",
        "\n",
        "        # Move logits and labels to CPU\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "        # Calculate the accuracy for this batch of test sentences, and\n",
        "        # accumulate it over all batches.\n",
        "        total_eval_accuracy += flat_accuracy(logits, label_ids)\n",
        "        \n",
        "\n",
        "    # Report the final accuracy for this validation run.\n",
        "    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)\n",
        "    print(\"  Accuracy: {0:.2f}\".format(avg_val_accuracy))\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_val_loss = total_eval_loss / len(validation_dataloader)\n",
        "    \n",
        "    validation_time = format_time(time.time() - t0)\n",
        "    print(\"  Validation Loss: {0:.2f}\".format(avg_val_loss))\n",
        "    print(\"  Validation took: {:}\".format(validation_time))\n",
        "\n",
        "    # Record all statistics from this epoch.\n",
        "    training_stats.append(\n",
        "        {\n",
        "            'epoch': epoch_i + 1,\n",
        "            'Training Loss': avg_train_loss,\n",
        "            'Valid. Loss': avg_val_loss,\n",
        "            'Valid. Accur.': avg_val_accuracy,\n",
        "            'Training Time': training_time,\n",
        "            'Validation Time': validation_time\n",
        "        }\n",
        "    )\n",
        "\n",
        "print(\"\")\n",
        "print(\"Training complete!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "======== Epoch 1 / 4 ========\n",
            "Training...\n",
            "\n",
            "  Step 0 GPU Memory Use:\n",
            "    Before forward-pass:  1395 MiB\n",
            "     After forward-pass:  7045 MiB\n",
            "    After gradient calculation:  7373 MiB\n",
            "\n",
            "  Step 1 GPU Memory Use:\n",
            "    Before forward-pass:  7643 MiB\n",
            "     After forward-pass:  8433 MiB\n",
            "    After gradient calculation:  8653 MiB\n",
            "  Batch    50  of    571.    Elapsed: 0:00:26.\n",
            "  Batch   100  of    571.    Elapsed: 0:00:52.\n",
            "  Batch   150  of    571.    Elapsed: 0:01:17.\n",
            "  Batch   200  of    571.    Elapsed: 0:01:43.\n",
            "  Batch   250  of    571.    Elapsed: 0:02:09.\n",
            "  Batch   300  of    571.    Elapsed: 0:02:34.\n",
            "  Batch   350  of    571.    Elapsed: 0:03:00.\n",
            "  Batch   400  of    571.    Elapsed: 0:03:25.\n",
            "  Batch   450  of    571.    Elapsed: 0:03:51.\n",
            "  Batch   500  of    571.    Elapsed: 0:04:17.\n",
            "  Batch   550  of    571.    Elapsed: 0:04:42.\n",
            "\n",
            "  Average training loss: 1.73\n",
            "  Training epcoh took: 0:04:53\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.25\n",
            "  Validation Loss: 1.68\n",
            "  Validation took: 0:00:11\n",
            "\n",
            "======== Epoch 2 / 4 ========\n",
            "Training...\n",
            "\n",
            "  Step 0 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "\n",
            "  Step 1 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "  Batch    50  of    571.    Elapsed: 0:00:26.\n",
            "  Batch   100  of    571.    Elapsed: 0:00:52.\n",
            "  Batch   150  of    571.    Elapsed: 0:01:17.\n",
            "  Batch   200  of    571.    Elapsed: 0:01:43.\n",
            "  Batch   250  of    571.    Elapsed: 0:02:09.\n",
            "  Batch   300  of    571.    Elapsed: 0:02:34.\n",
            "  Batch   350  of    571.    Elapsed: 0:03:00.\n",
            "  Batch   400  of    571.    Elapsed: 0:03:26.\n",
            "  Batch   450  of    571.    Elapsed: 0:03:51.\n",
            "  Batch   500  of    571.    Elapsed: 0:04:17.\n",
            "  Batch   550  of    571.    Elapsed: 0:04:43.\n",
            "\n",
            "  Average training loss: 1.61\n",
            "  Training epcoh took: 0:04:53\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.27\n",
            "  Validation Loss: 1.70\n",
            "  Validation took: 0:00:11\n",
            "\n",
            "======== Epoch 3 / 4 ========\n",
            "Training...\n",
            "\n",
            "  Step 0 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "\n",
            "  Step 1 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "  Batch    50  of    571.    Elapsed: 0:00:26.\n",
            "  Batch   100  of    571.    Elapsed: 0:00:52.\n",
            "  Batch   150  of    571.    Elapsed: 0:01:17.\n",
            "  Batch   200  of    571.    Elapsed: 0:01:43.\n",
            "  Batch   250  of    571.    Elapsed: 0:02:09.\n",
            "  Batch   300  of    571.    Elapsed: 0:02:34.\n",
            "  Batch   350  of    571.    Elapsed: 0:03:00.\n",
            "  Batch   400  of    571.    Elapsed: 0:03:26.\n",
            "  Batch   450  of    571.    Elapsed: 0:03:51.\n",
            "  Batch   500  of    571.    Elapsed: 0:04:17.\n",
            "  Batch   550  of    571.    Elapsed: 0:04:42.\n",
            "\n",
            "  Average training loss: 1.44\n",
            "  Training epcoh took: 0:04:53\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.27\n",
            "  Validation Loss: 1.74\n",
            "  Validation took: 0:00:11\n",
            "\n",
            "======== Epoch 4 / 4 ========\n",
            "Training...\n",
            "\n",
            "  Step 0 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "\n",
            "  Step 1 GPU Memory Use:\n",
            "    Before forward-pass:  8677 MiB\n",
            "     After forward-pass:  8677 MiB\n",
            "    After gradient calculation:  8677 MiB\n",
            "  Batch    50  of    571.    Elapsed: 0:00:26.\n",
            "  Batch   100  of    571.    Elapsed: 0:00:52.\n",
            "  Batch   150  of    571.    Elapsed: 0:01:18.\n",
            "  Batch   200  of    571.    Elapsed: 0:01:43.\n",
            "  Batch   250  of    571.    Elapsed: 0:02:09.\n",
            "  Batch   300  of    571.    Elapsed: 0:02:34.\n",
            "  Batch   350  of    571.    Elapsed: 0:03:00.\n",
            "  Batch   400  of    571.    Elapsed: 0:03:26.\n",
            "  Batch   450  of    571.    Elapsed: 0:03:51.\n",
            "  Batch   500  of    571.    Elapsed: 0:04:17.\n",
            "  Batch   550  of    571.    Elapsed: 0:04:42.\n",
            "\n",
            "  Average training loss: 1.26\n",
            "  Training epcoh took: 0:04:53\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.27\n",
            "  Validation Loss: 1.82\n",
            "  Validation took: 0:00:11\n",
            "\n",
            "Training complete!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQTvJ1vRP7u4"
      },
      "source": [
        "Training accuracy should always go up, or at least remain constant, with each additional training epoch. \n",
        "\n",
        "We see that the validation accuracy has *increased* and *remained constant*, hence our model is not over-fitting. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6O_NbXFGMukX",
        "outputId": "df665274-adc7-4e3f-97be-4076b80c34e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Create a DataFrame from our training statistics.\n",
        "df_stats = pd.DataFrame(data=training_stats)\n",
        "\n",
        "# Use the 'epoch' as the row index.\n",
        "df_stats = df_stats.set_index('epoch')\n",
        "\n",
        "# A hack to force the column headers to wrap.\n",
        "#df = df.style.set_table_styles([dict(selector=\"th\",props=[('max-width', '50px')])])\n",
        "\n",
        "# Display floats with two decimal places.\n",
        "pd.set_option('precision', 2)\n",
        "\n",
        "# Display the table.\n",
        "df_stats"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Valid. Loss</th>\n",
              "      <th>Valid. Accur.</th>\n",
              "      <th>Training Time</th>\n",
              "      <th>Validation Time</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>epoch</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.73</td>\n",
              "      <td>1.68</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0:04:53</td>\n",
              "      <td>0:00:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.61</td>\n",
              "      <td>1.70</td>\n",
              "      <td>0.27</td>\n",
              "      <td>0:04:53</td>\n",
              "      <td>0:00:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.44</td>\n",
              "      <td>1.74</td>\n",
              "      <td>0.27</td>\n",
              "      <td>0:04:53</td>\n",
              "      <td>0:00:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.26</td>\n",
              "      <td>1.82</td>\n",
              "      <td>0.27</td>\n",
              "      <td>0:04:53</td>\n",
              "      <td>0:00:11</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       Training Loss  Valid. Loss  Valid. Accur. Training Time Validation Time\n",
              "epoch                                                                         \n",
              "1               1.73         1.68           0.25       0:04:53         0:00:11\n",
              "2               1.61         1.70           0.27       0:04:53         0:00:11\n",
              "3               1.44         1.74           0.27       0:04:53         0:00:11\n",
              "4               1.26         1.82           0.27       0:04:53         0:00:11"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wC64axhF0_aQ"
      },
      "source": [
        "Let's take a look at our training loss over all epochs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOx9lsfm0_aR",
        "outputId": "a3539836-4f0e-455c-eca3-120577b63b6c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "# Use plot styling from seaborn.\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(df_stats['Training Loss'], 'b-o', label=\"Training\")\n",
        "plt.plot(df_stats['Valid. Loss'], 'g-o', label=\"Validation\")\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Training & Validation loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "plt.xticks([1, 2, 3, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAGaCAYAAACopj13AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXiU5b0+8Hv2LDPJLNlnCSSQAFnYEyTsoCyiyKLoURFFqz306OlPj2JrF7XtqVRrq5ZjSylKBRHZBGRRCCAoJGyyg5hA9o3Jvs9k3t8fMxkYkugkJJlJcn+uy0vyzjvvfDPkSW6ePO/3EQmCIICIiIiIiDxG7OkCiIiIiIj6OoZyIiIiIiIPYygnIiIiIvIwhnIiIiIiIg9jKCciIiIi8jCGciIiIiIiD2MoJ6JeLzc3F7GxsXj33Xc7fI1ly5YhNja2E6vqvdp6v2NjY7Fs2TK3rvHuu+8iNjYWubm5nV7f5s2bERsbi7S0tE6/trs642uSiHoXqacLIKK+pz3hdt++fTAYDF1YTc9TW1uL999/Hzt37kRxcTG0Wi1GjhyJ//zP/0R0dLRb13j22WexZ88ebN26FYMHD271HEEQMHXqVFRWVuLw4cPw8fHpzE+jS6WlpSE9PR2PPfYYAgICPF0OEdGPYignom63fPlyl49PnDiBTz75BAsXLsTIkSNdHtNqtbf9enq9HmfOnIFEIunwNV5//XW8+uqrt11LZ3jllVfw+eefY/bs2UhKSkJJSQlSU1Nx+vRpt0P5ggULsGfPHmzatAmvvPJKq+ccPXoUeXl5WLhwYacE8jNnzkAs7p5f0Kanp+O9997D3LlzW4TyOXPm4O6774ZMJuuWWoiI3MFQTkTdbs6cOS4fNzU14ZNPPsGwYcNaPHar6upqKJXKdr2eSCSCQqFod50385YAV1dXh927d2PcuHF46623nMd/9rOfobGx0e3rjBs3DuHh4di+fTtefPFFyOXyFuds3rwZgD3Ad4bb/TvoLBKJ5Lb+gUZE1BW4ppyIvNaUKVPw6KOP4sKFC1iyZAlGjhyJe++9F4A9nL/99tu4//77kZycjPj4eNx555148803UVdX53Kd1tbv3nxs//79mD9/PhISEjBu3Di88cYbsFqtLtdobU1587Gqqir85je/wR133IGEhAQ8+OCDOH36dIvPp6ysDC+//DKSk5MxfPhwLFq0CBcuXMCjjz6KKVOmuPWeiEQiiESiVv+R0FqwbotYLMbcuXNRXl6O1NTUFo9XV1fjiy++QExMDBITE9v1freltTXlNpsNf//73zFlyhQkJCRg9uzZ2LZtW6vPz8jIwG9/+1vcfffdGD58OIYOHYp58+bh008/dTlv2bJleO+99wAAU6dORWxsrMvff1tryktLS/Hqq69i4sSJiI+Px8SJE/Hqq6+irKzM5bzm5x85cgSrVq3CtGnTEB8fj+nTp2PLli1uvRdtsVqt+Mc//oFZs2YhISEBycnJWLp0KS5fvtzi3K1bt2LBggUYNWoUhg0bhqlTp+L5559HaWmp85wrV67g2Wefxfjx4xEfH4+UlBQ8+uijOHDgwG3VSUSdjzPlROTV8vPz8dhjj2HGjBm46667UFtbCwAoKirCxo0bcdddd2H27NmQSqVIT0/HP//5T1y8eBGrVq1y6/oHDx7EunXr8OCDD2L+/PnYt28f/vWvfyEwMBDPPPOMW9dYsmQJtFotli5divLycqxevRo/+clPsG/fPuesfmNjIx5//HFcvHgR8+bNQ0JCAi5fvozHH38cgYGBbr8fPj4+uO+++7Bp0ybs2LEDs2fPdvu5t5o3bx7+7//+D5s3b8aMGTNcHvv8889RX1+P+fPnA+i89/tW//u//4s1a9Zg9OjRWLx4McxmM1577TUYjcYW56anp+P48eOYNGkSDAaD87cGr7zyCkpLS/H0008DABYuXIjq6mp8+eWXePnll6HRaAD88L0MVVVVeOihh5CVlYX58+djyJAhuHjxIj7++GMcPXoUn376aYvf0Lz99tuor6/HwoULIZfL8fHHH2PZsmUwmUwtlmG564UXXsCuXbuQkpKChx56CNevX8fatWvx4IMPYu3atRgyZAgAeyB/6aWXMGrUKDz77LPw8fFBQUEBDh48CLPZDK1Wi7KyMjz22GMAgAcffBAREREoKyvDuXPncPr0aUyaNKlDNRJRFxGIiDxs06ZNQkxMjLBp0yaX45MnTxZiYmKEDRs2tHhOQ0OD0NjY2OL422+/LcTExAinT592HsvJyRFiYmKEd955p8WxoUOHCjk5Oc7jNptNuPvuu4WUlBSX67700ktCTExMq8d+85vfuBzfuXOnEBMTI3z88cfOYx999JEQExMjrFixwuXc5uOTJ09u8bm0pqqqSnjqqaeE+Ph4YciQIcLnn3/u1vPasmjRImHw4MFCUVGRy/EHHnhAiIuLE8xmsyAIt/9+C4IgxMTECC+99JLz44yMDCE2NlZYtGiRYLVancfPnTsnxMbGCjExMS5/NzU1NS1ev6mpSXjkkUeEESNGuNT3zjvvtHh+s+avt6NHjzqP/fnPfxZiYmKEjz76yOXc5r+ft99+u8Xz58yZIzQ0NDiPFxYWCnFxccLPf/7zFq95q9beo8OHDwsxMTHCc889J9hsNufxixcvCoMHDxYeeugh57GlS5cKw4cPFywWS5uvsXfvXiEmJua2v0aIqHtw+QoReTW1Wo158+a1OC6Xy51LOKxWKyoqKlBaWoqxY8cCQKvLR1ozdepUl+4uIpEIycnJKCkpQU1NjVvXWLx4scvHY8aMAQBkZWU5j+3fvx8SiQSLFi1yOff++++HSqVy63VsNhuee+45XLp0Cbt27cKECRPwwgsvYPv27S7n/epXv0JcXJxba8wXLFiApqYmbN261XksIyMD3377LaZMmeK80baz3u+b7du3D4Ig4PHHH3dZ4x0XF4eUlJQW5/v5+Tn/3NDQgLKyMpSXlyMlJQXV1dXIzMxsdw3NvvzyS2i1WixcuNDl+MKFC6HVarF3794Wz/mP//gPlyVDoaGh6N+/P65du9bhGgDgmWeegUgkch4fNGgQJk+ejBMnTjiXpqhUKtTX1+PAgQMQBKHV6zV/XR06dAjV1dUdqomIug+XrxCRVzMajW3elLd27VqsX78e33//PWw2m8tjFRUVbl//Vmq1GgBQXl4Of3//dl+jeblEeXm581hubi5CQkJaXE8ul8NgMKCysvJHX2ffvn04fPgw/vSnP8FgMOCvf/0rfvazn+HFF1+E1WrF3LlzAQCXL19GQkKCW2vM77rrLgQEBGDz5s34yU9+AgDYtGkTADiXrjTrjPf7Zjk5OQCAqKioFo9FR0fj8OHDLsdqamrw3nvvYdeuXSgoKGjxHHfew7bk5uYiPj4eUqnrj0WpVIp+/frhwoULLZ7T1tdOXl5eh2sQi8WtdtAZMGAA9u7di9zcXGi1Wjz99NM4duwYli5dCrVajaSkJEyYMAEzZ850LrNJSkrCfffdh82bN2P79u2Ij4/H2LFjMWvWLAwYMKBDNRJR12EoJyKv5uvr2+rx1atX449//CPGjRuHRYsWISQkBDKZDEVFRVi2bFmbs4e3+qEuHLd7DXef767mGxNHjx4NwB7o33vvPfz0pz/Fyy+/DKvVikGDBuH06dP4/e9/79Y1FQoFZs+ejXXr1uHkyZMYOnQotm3bhrCwMIwfP955Xme937fj+eefx4EDB/DAAw9g9OjRUKvVkEgkOHjwID744IMW/1Doat3V3rE1/fr1w86dO3HkyBEcOXIE6enpeOWVV/DOO+9g7dq1MJlMAIA33ngDS5YswVdffYXjx49j9erVeP/99/GLX/wCjzzyiMfqJ6KWGMqJqEf67LPPoNfrsXLlSpdw9NVXX3mwqrbp9XocOXIENTU1LrPlFosFubm5bm1w0/x55uXlITw8HIA9mK9YsQLPPPMMfvWrX0Gv1yMmJgb33Xef27UtWLAA69atw+bNm1FRUYGSkhI888wzLu9rV7zfzTPNmZmZzhDZLCMjw+XjyspKHDhwAHPmzMFrr73m8tg333zT4to3L/9wt5arV6/CarW6zJZbrVZcu3at1VnxzmY0GmGz2ZCRkYFBgwa5PNb8fty81Eoul2PixImYOHEiAPtNyz/5yU+wevVq/OY3v3GeFxMTg5iYGDz55JOorKzE/fffj7feegsPP/xwu98nIuo6XFNORD2SWCyGSCRymaG1Wq1YuXKlB6tq25QpU9DU1IQ1a9a4HN+wYQOqqqrcukZz+Hr77bdd1osrFAr8+c9/RkBAAHJzczF9+vQWyzB+SFxcHAYPHoydO3di7dq1EIlELXqTd8X7PWXKFIhEIqxevRpNTU3O4+fPn28RtJv/IXDrjHxxcXGLlojAjfXn7i6rmTZtGkpLS1tca8OGDSgtLcW0adPcus7taH6Nf/zjHy6f53fffYfU1FSMHDnSucb/5raHzZo7szR/zuXl5S1+exAQEODsXNPQ0NAlnwcRdQxnyomoR5oxYwbeeustPPXUU7jzzjtRXV2NHTt2tCuMdqf7778f69evx1/+8hdkZ2c7WyLu3r0bkZGRLfqityYlJQULFizAxo0bcffdd2POnDkICwtDTk4OPvvsMwD2gP23v/0N0dHRmDlzptv1LViwAK+//joOHTqEpKSkFjPDXfF+R0dH4+GHH8ZHH32Exx57DHfddRfMZjPWrl2LQYMGuazjViqVSElJwbZt2+Dj44OEhATk5eXhk08+gcFgcFm/DwBDhw4FALz55pu45557oFAoMHDgQMTExLRay5NPPondu3fjtddew4ULFzB48GBcvHgRGzduRP/+/fHkk092+PN0V0pKCmbOnInPP/8cFRUVmDx5MkpKSrBu3TooFAqXnVeXLFkClUqFUaNGITw8HJWVldiyZQtEIpFzA66tW7fiww8/xLRp0xAZGQmpVIpjx47h8OHDmDlzZqfs0kpEncc7f3oREf2IJUuWQBAEbNy4Eb///e8RHByMmTNnYv78+Zg1a5any2tBLpfjww8/xPLly7Fv3z7s2rULiYmJ+OCDD/DLX/4S9fX1bl3n97//PZKSkrB+/XqsWrUKFosFer0eM2bMwBNPPAG5XI6FCxfif/7nf6BSqTBu3Di3rnvPPfdg+fLlaGhoaHGDJ9B17/cvf/lLBAUFYcOGDVi+fDn69euHX//618jKympxc+Wf/vQnvPXWW0hNTcWWLVvQr18//PznP4dUKsXLL7/scu7IkSPxwgsvYP369fjVr34Fq9WKn/3sZ22GcpVKhY8//hjvvPMOUlNTsXnzZuh0Ojz44IP4r//6r3bvIttRb775JoYMGYItW7bgj3/8I/z8/DB69Gg899xzLn3WH3roIezatQuffPIJKioqoFarMXjwYLzyyivO7j/Jycm4ePEiDhw4gJKSEojFYhgMBrz00ktcT07khURCd9ydQ0RErWpqasKYMWOQmJjY4Q14iIio5+OaciKibtLabPj69etRWVnZal9uIiLqO7h8hYiom7zyyitobGzE8OHDIZfLcerUKezYsQORkZF44IEHPF0eERF5EJevEBF1k61bt2Lt2rW4du0aamtrodPpMHHiRDz33HMICgrydHlERORBDOVERERERB7GNeVERERERB7GUE5ERERE5GG80dOhrKwGNlv3ruTR6ZQwm6u79TWJeiKOFSL3cKwQuccTY0UsFkGj8W/zcYZyB5tN6PZQ3vy6RPTjOFaI3MOxQuQebxsrXL5CRERERORhDOVERERERB7GUE5ERERE5GEM5UREREREHsZQTkRERETkYQzlREREREQexlBORERERORhDOVERERERB7GUE5ERERE5GHc0ZOIiIiI+oT0wpPYlrEb5Q3lUCvUuDd6BpLCRni6LAAM5URERETUB6QXnsS6S5tgsVkAAGUN5Vh3aRMAeEUw5/IVIiIiIuq1BEGAua4Um65sdwbyZhabBdsydnuoMlecKSciIiKiXsEm2FBcW4KcqnzkVOUhpzofuVV5qLXWtfmcsobybqywbQzlRERERNTjWGxWFNQUIrc5gFflI686H42O2XCpWAq9fziGhyTCqNJj59UvUdlY1eI6GoW6u0tvFUM5EREREXm1emsD8qoLkFOd5wzhBTVFaBKaAAA+EgX0ygikRCTDoIqAUaVHmF8IJGKJ8xoKidxlTTkAyMQy3Bs9o9s/n9Z4NJQXFxdjzZo1OH36NM6dO4fa2lqsWbMGycnJbj1/586dWL16NTIzMyGTyRATE4NnnnkGY8eO7eLKiYiIiKgr1FhqHTPfecittgfw4trrECAAAJQyfxhVegzRxcKgtAfwIF8txKIfvlWy+WZOdl9pxdWrV7Fy5UpERkYiNjYWp06dcvu5a9euxWuvvYZJkyZh3rx5aGhowKZNm/DEE09g1apVSElJ6cLKiYiIiOh2CIKAisZKe/i+aQ14aX2Z8xyNQg2jSo+RocNgUulhUEZArQiESCTq0GsmhY1AUtgIBAerUFLScimLJ3k0lMfFxeHo0aPQaDTYu3cvli5d6vZzP/roIyQkJOD99993/sXcd999GDduHLZt28ZQTkREROQlBEHA9bpS5FTn3ZgFr8pHlaUaACCCCMF+OvQPMGGC/g4YHQFcKff3cOXdx6OhXKlUdvi51dXVMJlMLv9SCggIgEKhgEKh6IzyiIiIiKidmmxNKKotccx8N8+C56O+qR4AIBaJEe4fijjdIHv4VkXAoAyHj9THw5V7Vo+90TMpKQm7du3Cv//9b0yePBkNDQ1YvXo1BEHAww8/7OnyiIiIiHo9S5MF+TWFzqUnOVV5yK8ugMVmBWC/kdKgDMfosOEwqiJgVOoRrgyDTNxjI2iX6bHvyC9+8QuYzWb87ne/w+9+9zsAQFBQENasWYPY2FgPV0dERETUu9RZ6+0dUKpuLEEprC2GTbABAHylPjAoIzDesfzEqNIjxDfIpQMKta3HhnJfX19ERUUhPDwcEydORE1NDT744AP89Kc/xbp162A0Gtt1PZ2u40tpbkdwsMojr0vU03CsELmHY4U6Q2V9Fa6W5+Bqmf2/a2U5KKgudj4e6BOAKI0RYyKHoZ/aiCiNCcH+ug7fgOkJ3jZWemwof/bZZ6FQKPC3v/3NeWzq1KmYPn06/vKXv+Ctt95q1/XM5mrYbEJnl/mDvPHOXyJvxLFC5B6OFWovQRBQ3lCB7Ko85DrWgOdU5aO8ocJ5js5HY++AEuxYgqLSI1AR4HqhOuB6XXU3V99xnhgrYrHoByeBe2Qoz8nJwaFDh/CHP/zB5bharcaIESPa1VqRiIiIqC+wCTaU1JlvaUGYhxpLLQB7B5RQv2AMVEfBoIpwtiD0k/l5uPK+oUeG8uvXrwMAbDZbi8esViusVmt3l0RERETkNZpsTSioKXLefJnr2IinoakRACARSRChDMPQoDhHBxQ99MpwKCRyD1fed/WIUJ6dnQ0AMJlMAIDIyEiIxWLs3LkT999/v/O8wsJCHD9+3O0dQYmIiIh6usamRuRVFzp2wMxzdEAphNWxBb1cIodBGYEx4aNgVNoDeLh/CKTsgOJVPP63sWLFCgBARkYGAOCzzz7DiRMnEBAQgEceeQQAsHjxYgBAamoqAECr1WL+/Pn49NNP8dhjj+Guu+5CdXU11q1bh8bGRjz11FPd/4kQERERdbFaS51z6/mcqnzkVOehqKbYuQW9v9QPBlUEJhnHwejYgj7YL+hHt6AnzxMJgtC9dzfeoq32hXq93hnCp0yZAuBGKAfsy1TWr1+PjRs3IisrCwCQmJiIpUuXIikpqd118EZPIu/FsULkHo6V3qWysepG+HYsQbleX+p8XK0IhMERvJtvwNQo1D2qA4qneOONnh4P5d6CoZzIe3GsELmHY6VnEgQBpfVlzg14ch09wCsab/xdBvnq7OHbGcL1UMk90865N/DGUO7x5StEREREfYVNsKG4tsQ5+90cwmutdQDsW9CH+YVgkDYGBlUEjMoIGFQR8JX6erhy6moM5URERERdwGKzoqCm8Eb7wap85FXno9FmAQBIxVLo/cMxPCTRuQQlwj8cconMw5WTJzCUExEREd2mhqZG5FXnOzbhsYfwgpoiNDk6oPhIFDCoIpASkexoQRiBML8QbkFPTgzlRERERO1QY6l1zHznOTqh5KO4tsTZAUUp84dRpccQXazzRswgXy07oNAPYignIiIiaoUgCKhorLxlB8x8lNaXOc/RKNT2LehDhzp3wFQrAtkBhdqNoZyIiIj6PEEQcL2uFDmOzXeaQ3iVpRqAfQv6YD8d+geYMEF/h30JijICSrm/hyun3oKhnIiIiPqUJlsTimpLHDPfzQE8H/VN9QDsHVDC/UMRFzTIsQNmBAzKcPhIfTxcOfVmDOVERETUa1maLMivKXQuPbFvQV8Ai80KAJCJZTAowzE6bLh9Ax6lHuHKMMi4BT11M37FERERUa9QZ61HXnWB8ybMnKo8FNYWwybYAAC+Uh8YlXqMdyw/Mar0CPENYgcU8goM5URERNTjVDVW25ed3LQGvLjuuvNxlVwJo0qPxKAhMDgCuM5HwxswyWsxlBMREZHXEgQB5Q0Vjv7feY4Qno/yhgrnOTofDYwqPZLCRtqXoKj0CFQEeLBqovZjKCciIiKvYBNsKKkz28O3swVhHmostQDsHVBC/YIxUB0FgyrC2YLQT+bn4cqJbh9DOREREXW7JlsTCmqKkFOd7wjh9o14GpoaAQBSkQThyjAMDYqHURUBg0oPvTIcConcw5UTdQ2GciIiIupSjU2NyKsudARvewDPrymC1dEBRS6Rw6CMwJjwUY4WhHqE+4dAyg4o1Ifwq52IiIg6Ta2lDrmO2e/sqnzkVuehsKbYuQW9v9QPBlUEJhlSYHRsQR/sF8Qt6KnPYygnIiKiDqlsrHK0Hsx3dEDJw/X6UufjakUgDMoIDAtOcN6AqVGo2QGFqBUM5URERPSDBEFAaX2ZcwOe5jXgFY1VznOCfXUwBhgwNiLJ2QNcJVd6sGqinoWhnIiIiJxsgg3FtSU3dT+xh/Baax0A+xb0YX4hGKSNgcGxA6ZBFQ5fqa+HKyfq2RjKiYiI+iirzWrvgHLTEpS86nw02iwAAKlYCr1/OIaHJDpmvyMQ4R8OuUTm4cqJeh+GciIioh4uvfAktmXsRnlDOdQKNe6NnoGksBEu5zQ0NSKvOt+xCY89gBfUFKFJaAIA+EgUMKgikBKRDKNKD4MqAmF+IdyCnqibMJQTERH1YOmFJ7Hu0iZYHLPbZQ3lWHdpEwqqi+Av93POghfXljg7oChl/jCq9Biii4XB0QElyFfLDihEHsRQTkRE1ANZbFaU1Zdh85UdzkB+4zELvsjeDwDQKNQwqvQYFTrUPgOujIBaEcgOKERehqGciIjIC1maLChtKEdpXRlK68tgri+Dub7U/ue6MlQ0Vv7oNd4Y/xsoZf7dUC0R3S6GciIiIg+4OXSb60thri9zBu7S+lKXdoOAveuJVqGG1leLIbpY6Hw00PposOX7z1FlqW5xfY1CzUBO1IMwlBMREXWB2wvdg5yhW+erhc5Hg0BFQKtrvkUikcuacgCQiWW4N3pGl3+ORNR5GMqJiIg6oLHJgjLnspLmwF3qXGpSeUvologk0PioofPRIE43CFo3Q/ePae6y8mPdV4jIuzGUExERteJ2Qne8bhC0PlrofB3B+zZCtzuSwkYgKWwEgoNVKCmp+vEnEJHXYSgnIqI+qbHJ4gzYpfWljmUlN26orGp0XactEUmg9VFD56NFvG6wY5a7e0I3EfV+DOVERNQr3U7oTtANuSlw22e8A+Qqhm4i6jIM5R5w5HwhNh/MQGllA7QBCsybGI074sI8XRYRUY/S2NR4I2Q7A7fjhsq6shYdSaQiiXMdN0M3EXkbhvJuduR8IT7cdQmNVhsAwFzZgA93XQIABnMiopvcTuhODB5iX9N90xIThm4i8mYM5d1s88EMZyBv1mi14ZPU75E8OBRiMXdYI6K+oaE5dN9082Rz4DbXl6LaUuNyvlQkgdZXA62CoZuIeh+G8m5mrmxo9XhlTSP++93DSIjSIiFah/j+Oih9Zd1cHRFR52l36BZLnWu6Dap4e+D20UDraBmokisZuomo12Io72a6AEWrwVzpK0VClBZnM0tx5HwRRCIgOiIQCdE6JEbpYApVQiTiLDoReY96awNK62/ZAr7uRvtAhm4iIvcxlHezeROjXdaUA4BcKsZD02JwR1wYbDYBVwsrcTbDjDMZZmz5KhNbvspEoFKOhCh7QB/STws/H/7VEVHX6kjobt6F0qTSQ+ejhdbXEbx9tFDJ/Rm6iYjaIBIEQfB0Ed7AbK6GzdY9b0V7uq9U1DTiXKY9oJ+7Woq6BiskYhEGGm7MokcE+XMWnXo1bojSNZpD9803Tza3DyytL28RumVi6Y1dKB1dSxi6vQvHCpF7PDFWxGIRdDplm48zlDt0Zyhv1t4viCabDRl5lTjjmEXPLbF3HtAFKJAQHYTEKB0GR2qgkEu6qmQij2DQ6Jh6az1K68tbDd3m+jLUWGpdzreHbq1jSYnmxvISR8tAlYzL6LwdxwqRe7wxlHMNRA8iEYsRY1QjxqjGgknRKK2sx1nHLPqRc4U4cCoPUokIsSYNEqN0SIzWIVTr5+myiaiL1Fvrb9r+3bG85KYt4X8odJsCjAzdRERehDPlDj1hpvyHWKw2XMktx5kMM85mmlFgtv8wDtH4OgN6rEkNmZSz6NTz9NXZvzpr/Y013beG7roy1FhvDd2ym2a5tc713c3bwTN09359dawQtZc3zpR7NJQXFxdjzZo1OH36NM6dO4fa2lqsWbMGycnJP/rc2NjYNh8bO3YsVq9e3a5aenoov1VxeR3OOgL6xawyWKw2yGViDDZpkBitQ0K0DkGBvl3y2kSdrbcGjebQba4rvTHj7fh/e0K3zvGxUsb7S/q63jpWiDqbN4Zyjy5fuXr1KlauXInIyEjExsbi1KlTbj93+fLlLY6dO3cOa9asQUpKSmeW2SOFqH0xdaQBU0ca0GhpwqXscpzNMC5ZZy8AACAASURBVON0xnWczjADACKC/JEYZQ/oAw2BkEp4gxZRZ2ozdDs+rrXWuZwvF8uc7QH7B5hcZrkZuomIejePhvK4uDgcPXoUGo0Ge/fuxdKlS91+7pw5c1ocS09Ph0gkwuzZszuzzB5PLpMgMdq+hOU/hIEoLK21t1zMNOPL4znYnZ4NH7kEcf3sGxclROmgUSk8XTaR16uz1jmWldzYAr60vty90B0Y6QjcN2a8GbqJiPouj4ZypbLtKfz2amxsxBdffIHRo0cjLKz19oIEiEQihOv8Ea7zx11JJtQ1WHEpqwxnHDeMnviuBABgClHaWy5G6xAVEQCJmLPo1Pe0GrrrbuxMWddK6NY5Q3c/+0Y5DN1EROSGXtN95eDBg6isrMS9997r6VJ6FF+FFMNjgjE8JhiCICCvpMYZ0HcdzcbnR7Lg7yNFXH8tEqN1iI/SIcBP7umyiTpFraXOpUVgaV2Zy3bwLUK3RO7sWBIV2A86X9ee3f4yP4ZuIiLqkF4Tyrdv3w65XI7p06d7upQeSyQSwRCihCFEiVljIlFbb8H5a2U4k3EdZzNLkX6xGCIA/cIDnMthIsNUEDOEUBdJLzyJbRm7Ud5QDrVCjXujZyApbITbz28tdDt3pqwvQ5213uV8uUSOIB8ttD5qhm4iIupWvSKUV1dX48CBA5g4cSICAgI6dI0fuhu2KwUHqzzyuu6KNGoxa3w0bDYBmXkVOH6pCMcvFmHb11fx2eGrUCsVGDEoBKMGhWJ4bDCUnEWnTnIoKx0fX96MxqZGAEBZQzk+vrwZAQG+GB+ZBEEQUGOpRUlNKUpqzCiuMeN6jRnFtfaPS2rMqLW4znQrpAqE+OsQGhCEhLBYBPvrEOyvRYi/DsH+OijlXF5CPZ+3/1wh8hbeNlZ6RSjfs2cPGhoacM8993T4Gr2tJWJXCPSRYOqwCEwdFoGq2kacu1qKsxlmpJ0rQOrxHIhFIkTr7bPoCVE6GEPYE5ncIwgCLDYrLDYLLDYLGpss+PDkRmcgb9bY1Ij30/+Nzed2w1xXhvom15luhURu3/rdR4N+oSbHLPeNHSr9pW3MdDcB9ZUC6lHdlZ8mUZfraT9XiDyFLRG7yPbt26FSqTB58mRPl9JnqPzkuCMuDHfEhdln0Qsq7RsXZZix6WAmNh3MhEalQEKUFglRQRjSTwNfRa/4cusTbILNHpKbHCHZEZQtNovLseY/N5/baLM4g/WNj5sfa4SlydrKc+3nuMtis0Lro8YAddSNHSkdLQP9pL78hyAREfVIPT4lFRcXIy0tDXPnzoVczqUTniAWizBAH4gB+kDMmxCF8uoGnM20B/Rjl4rx1ekCSMQixBjVzrXoYVquzW2PJltTizBr/7Pr7LJrILa2GqJvDcSNN4Xv5vOtQlOHa5WJpZCJZfb/JDLInX+Wwk/mC7k4ADKJ7KZzpDedc+P8Td/vQI2lpsX1NQo1nkl8/HbeTiIiIq/TI0J5dnY2AMBkMrV4bOfOnbDZbLe1dIU6l1qpwPjECIxPjIC1yYaMvAqccfRF/yT1e3yS+j2CAn2cAT3WpIFCJvF02W4TBAFWock12LY1Y9xKgLbPGNvPbTFj3EaYtgm2DtUqgsgekh0hWO4IvvZALIVSrnSGYLlLUHYN0y2f23y+1OV8qVgKsahz2meKRCKsu7QJFpvFeUwmluHe6Bmdcn0iIiJv4vFQvmLFCgBARkYGAOCzzz7DiRMnEBAQgEceeQQAsHjxYgBAampqi+dv27YNISEhSE5O7p6CqV2kEjFiTRrEmjS4f/IAmCvqcdbRcvHw2QKknsyDTCpGrEmNxCh7SA/R+LXrNVpbj9yhGeObZp1dznUJ1PbHBHTs/gOxSOwykyyXuIZdP6nfTYG4rTAtg/ymx24O1LcGa6lI0mN/I9HcZeV2uq8QERH1FB4P5X/9619dPt60aRMAQK/XO0N5WzIzM3H+/Hk8/vjjEHNzG6/R2nrkm8NvkNGC8XoRRln8kXu9CtklZcgtzcKli/XYcNkGP18RtIEyBCgl8PURwSq0tg7ZisamxnavR76VVCS5ZSmFI/CKZVBIFC4zyTLHzPDNAVneSjBubUlGc7CWiHvObwS8QVLYCCSFjeDNa0RE1OuJBEHo3pYjXqo7u6/cbu/l9mpej2wPsjffXHdjVtnd2eXmGWn7+Y2dvh5ZDAlgk6DJKoJgk0AkSOArk0Pp4wO1ny/85YobM8kuM8c3QnPL5Rdtzy531lIL6loM5UTu4Vghcg+7rxDSC0+6rJMtayjH2ksbYa4rxUBN9I+uR25t5vnmrhhduh755qUUYhl85KqbAm7bN/fdHJpbX5ds/08qljhDcoOlCZeyyuy7i35vRk5lPXIA6IP97WvRo3SI1gdCKmGoJiIiop6PM+UO3TVT/srXf0BZQ3mHnisWiX909veH1iO3tpzi5tllb12PLAgCCsy19paLmWZ8l1OOJpsAX4UUcf21SIzSISFKi0ClwtOlUhfh7B+RezhWiNzDmXL6wUD+X8Oe+sEWcX11PbJIJEJEkD8igvwxI9mEugYrLlwrw9nM6ziTYcbxS8UAgMgwlfNm0f7hARCLPf8PCiIiIiJ3MJR3M41C3Wow1yjUGKQd6IGKeh5fhRQjY4MxMjYYgiAgp7ja2dFlx5Fr2P7NNSh9ZYiPss+ix0fpoPSVebpsIiIiojYxlHeze6NnsPdyJxKJRDCFqmAKVeHuO/qhus6CC9dKnUtdjp4vgghAlD7AMYseBGOoEmIvWJZDRERE1Ixryh16c/eVvsomCMgqrLJvXJRhxrWCSggAAv3lSHAscxnSTws/H/7b1NtxnSyRezhWiNzjjWvKGcodujOUN+M3z+5VWdOIc1ftAf1cZilqG6wQi0QYaAhEYrQOCdE66IP8veLmVnLFsULkHo4VIvcwlHsxhvK+pclmQ2Z+pXMWPae4GgCgDVDYu7lE6zA4UgMfOWfRvQHHCpF7OFaI3OONoZyJg/okiViMgQY1BhrUmD8xGmVVDTibacbZDDOOXijCgW/zIZWIEGtUIyE6CInROoRqfDmLTkRERF2CM+UOnCmnZtYmG67kVuBMhr3lYoG5FgAQovZFQrR9LXqsUQ25rG+2qPQEjhUi93CsELnHG2fKGcodGMqpLSXldc6Wi5eyytBotUEuFWNQpMa5u2iQ2tfTZfZqHCtE7uFYIXKPN4ZyLl8h+hHBal9MGWHAlBEGNFqacDmn3LEW3T6TDgDhOj9nQB9oVEMqEXu4aiIiIupJOFPuwJlyai9BEFBUVmfviZ5xHZdzymFtEqCQSxDXT2vv6BKlg0al8HSpPR7HCpF7OFaI3MOZcqJeRCQSIUzrhzCtH+4abUR9oxUXs8pwNsOMM5lmnPyuBABgDFE6A3q0PgASMWfRiYiIyBVDOVEn8ZFLMXxgMIYPDIYgCMi7XmMP6Blm7Dqajc+PZMFPIUV8lBYJUfaQHuAv93TZRERE5AUYyom6gEgkgiFYCUOwEjPHRKK23ooL10rtS10yzUi/WAwRgH7hKsfuokHoF66CmC0XiYiI+iSGcqJu4OcjxahBIRg1KAQ2QUBOUbX9RtFMM7Z/fQ3bvr4GlZ8M8f3tLRfj+muh9JV5umwiIiLqJgzlRN1MLBIhMkyFyDAV7knpj6raRpy/WoozmfZZ9CPnCyESAdH6QCRG2UO6MUTJjYuIiIh6MXZfcWD3FfIGNpuAqwWV9paLmWZkFdq/PtRKuWOZiw5D+mnhq+hb/57mWCFyD8cKkXvYfYWIfpBYLEK0PhDR+kDMnRCFiuoGnM20z6Ifv1yCQ2cKIBGLMNAQiMToICRE6xCh8+MsOhERUQ/HmXIHzpSTt7M22ZCRV2Ff5pJhRm5JDQBAF+Bjb7kYrcNgkwYKucTDlXY+jhUi93CsELmHM+VE1GFSiRixJg1iTRrcP2kASivrnQH9m3OF2H8qD1KJGINMaiRE25e6hGr8PF02ERERuYEz5Q6cKaeezGK14bvccmdf9MLSWgBAqMbXGdBjjWrIpD1zFp1jhcg9HCtE7vHGmXKGcgeGcupNistq7WvRM8y4lF0Gi9UGuUyMIZFae0iP0kEX6OPpMt3GsULkHo4VIvd4Yyjn8hWiXihE44epI/0wdaQBDZYmXM4us3d0yTDj2++vAwD0Qf7OgD7AEAipROzhqomIiPouhnKiXk4hkyAxOgiJ0UEQBAGFpbXOgP7lsRzsTsuGr0KCIf20SIyy3zCqVio8XTYREVGfwlBO1IeIRCKE6/wRrvPH9CQT6hqsuJhln0U/m2nGicslAABTqBKJ0TokRgUhKiIAYjFbLhIREXUlhnKiPsxXIcWImGCMiAmGIAjILanBmYzrOJthxs4j2djxTRb8faSIj7Ivc4mL0iLAT+7psomIiHodhnIiAmCfRTeGKGEMUeLuO/qhpt6C81dLcdYxi552oQgiAP0jApzLXCLDVBBz4yIiIqLbxlBORK3y95EhaXAokgaHwiYIyCqssrdczDTjs8NXsfXwVQT4yZDgCOjx/bXw85F5umwiIqIeiaGciH6UWCRC//AA9A8PwL3j+qOythHnM0txJtPezeXrc4UQi0QYoA9w9EUPgiHYHyLOohMREbmFfcod2KecqGOabDZcza/CmczrOJNhRnZRNQBAo1IgIUqHodE6DO6ngY+843MAHCtE7uFYIXIP+5QTUa8jEYsxwBCIAYZAzJsQjbKqBpzLtC9zSb9YhK9O50MiFiHWpHauRQ/T+nEWnYiI6CacKXfgTDlR57M22fB9bgXOZJpxNsOMvOs1AIBgtQ8So4KQEK3DIJMacpnkB6/DsULkHo4VIvd440w5Q7kDQzlR17teUYezmfaOLheyStFosUEmFWNwpMZ5w2iI2td5/pHzhdh8MAOllQ3QBigwb2I07ogL8+BnQOTd+HOFyD0M5V6MoZyoe1msTbicU27fuCjDjKKyOgBAuM4PCVE6SCUi7D2ei0arzfkcuVSMx2YOYjAnagN/rhC5xxtDOdeUE5FHyKQSxPfXIb6/DpgGFJXWOpe5pJ7Mg7XJ1uI5jVYbNh/MYCgnIqJeh6GciLxCqNYPd2r9cOcoIxoam/DTPx9s9TxzZUM3V0ZERNT1xJ4ugIjoVgq5BLoARZuPr993BaWV9d1YERERUdfyaCgvLi7Gm2++iUcffRTDhw9HbGws0tLS3H6+zWbDRx99hHvuuQeJiYkYM2YMlixZguzs7C6smoi6w7yJ0ZBLXb9FySRiREcEYO/xXLz0/hGs3H4BucXVHqqQiIio83h0+crVq1excuVKREZGIjY2FqdOnWrX81988UXs3bsXCxYswKJFi1BdXY0zZ86gvLwcJpOpi6omou7QvG68te4r1yvq8MWxHBw6XYAj5wsRH6XFzORIDDKp2f+ciIh6JI92X6murobFYoFGo8HevXuxdOlSrFmzBsnJyT/63B07dmDZsmVYu3Ythg4detu1sPsKkfdqa6xU11mw/1Qe9h3PQWWtBZFhKsxMNmFkbDAkYq7Oo76HP1eI3MPuK7dQKtsu7Md8+OGHmDZtGoYOHQqr1QqLxQJfX98ffyIR9RpKXxnuGdsPM5KM+PpcIfak5+D9z84jKNAH05NMGJcYDsWPbExERETkDXrkVFJ1dTXOnj2L2NhY/PrXv8bw4cMxbNgwzJ49G4cPH/Z0eUTUzWRSCSYN0+P3TyZj6dwEBPrLsfbL7/A/K77B1kOZqKxt9HSJREREP6hHtkTMzs6GIAj44IMPEBgYiN/+9reQSCT45z//iaeffhoff/wxEhMTPV0mEXUzsViEkbHBGBEThCu5Fdidlo1tX1/DrrRsjEsMx/TRRoRo/DxdJhERUQs9MpTX1tYCAGpqarB161aEh4cDAMaPH49p06bh73//O/72t7+165o/tManKwUHqzzyukQ9TXvHSkhIAFJGGJFTVIUtB77H/hO5OHgqD3ckRmDepAGIMWm6qFIiz+LPFSL3eNtY6ZGhXKGw9y8eMWKEM5ADgE6nw9ixY3Hy5Ml2X5M3ehJ5r9sZKz5i4KEpAzAzyYi9x3Ox/1Qevj6dj0EmNWYkm5AQpWPHFuo1+HOFyD280bOThISEAACCgoJaPKbT6VBZWdndJRGRl1MrFVgwKRp33xGJr07n44tjOfjLp2egD/bHjCQTkoeEQirpkbfZEBFRL9AjQ3loaCiCgoJQVFTU4rGioiJoNPy1NBG1zlchxfQkE6aONCD9YhF2pWVj1ecXsfmrTNw5yoiJwyLgq+iR3xqJiKgH6xHTQtnZ2S126ZwxYwZOnTqFjIwM57Hc3Fx8/fXXGDt2bHeXSEQ9jFQixtj4cLz2RBJ+/sBQhGp8sWH/93hhxdf4dP/3KKtq8HSJRETUh3h08yAAWLFiBQAgIyMDO3bswPz582EwGBAQEIBHHnkEADBlyhQAQGpqqvN5xcXFmDt3LkQiER599FFIJBJ89NFHqKqqwubNmxEZGdmuOrimnMh7dddYuVpQid1p2Th+uRhikQh3xIVherIJ+iD/Ln9tos7AnytE7vHGNeUeD+WxsbGtHtfr9c4Q3looB4Br167hj3/8I9LT0yEIAkaMGIEXX3yxzWv+EIZyIu/V3WOluLwOX6Rn4/CZAjRabRgarcPMMZEYaAjkTaHk1fhzhcg9DOVejKGcyHt5aqxU1TYi9WQe9p3IRXWdBVERAZiZbMLwgcEQixnOyfvw5wqRe7wxlPNuJiKiNqj85Jgzrj9mJJvw9dkC7EnPxt+2nEOIxhfTk0xIiQ+DXCbxdJlERNQLcKbcgTPlRN7LW8aKzSbgxHcl2J2WhasFVVD5yTBtpAGTRxig9JV5ujwirxkrRN6OM+VERD2YWCzC6EEhGBUbjMvZ5didno0th67i86NZmJAYgbtGGxGk9vV0mURE1AMxlBMRtZNIJMKgSA0GRWqQW1KNPWnZ2H8qD6kn8zB6cAhmJJkQGeZd2zcTEZF3YygnIroNhmAllswegrkTorD3eC4OfJuHtAtFGNJPgxnJJsT107JjCxER/SiuKXfgmnIi79WTxkptvRUHv83DF8dzUFHdCGOIEjOSTRg9KARSSY/Yr416sJ40Vog8yRvXlDOUOzCUE3mvnjhWLFYbjl4oxJ70HORfr4EuQIE7R5swYWg4fOT8JSV1jZ44Vog8wRtDOX8yEBF1AZlUjPGJEUhJCMeZDDN2p2Vj/b4r2P71VUwarse0kQYEKhWeLpOIiLwEQzkRURcSi0QYNiAIwwYEISO/ArvTsrHzSBb2pOdgbHwYpicZEa7z93SZRETkYZ0Syq1WK/bt24eKigpMnjwZwcHBnXFZIqJeJToiEEvnJqCotBZ70rNx+GwhDp3Ox7CBQZiZHIkBhkBPl0hERB7S7lC+fPlypKWlYdOmTQAAQRDw+OOP4/jx4xAEAWq1Ghs2bIDJZOr0YomIeoNQrR8WzRiEOeOjsO9ELvafzMWpK9cxwBCImckmDB0QBDE7thAR9SntbgVw6NAhjBo1yvlxamoqjh07hiVLluCtt94CAPzjH//ovAqJiHqpQH855k2Iwpv/mYL/mDYQZZUNeHfTWfzqn2n46nQ+LFabp0skIqJu0u6Z8sLCQkRGRjo/3r9/PwwGA1544QUAwJUrV7B9+/bOq5CIqJdTyCWYNsqIySP0OH6pBLvSsvDBrkvY8lUmpo0yYNJwPfx9ZJ4uk4iIulC7Q7nFYoFUeuNpaWlpGDt2rPNjo9GIkpKSzqmOiKgPkYjFSB4SiqTBIbiQVYbdadnYdDATO45kYeLQCNw5yghdoI+nyyQioi7Q7uUrYWFhOHXqFAD7rHhOTg5Gjx7tfNxsNsPPz6/zKiQi6mNEIhHi+mnx/MJh+O3jozF8YBD2Hs/Fsr8fwcrtF5BTXO3pEomIqJO1e6b87rvvxooVK1BaWoorV65AqVRi4sSJzscvXrzImzyJiDqJKVSFn9wTh3kTovDlsVx8dTofR84XIj5Ki5lJJgyK1EDEm0KJiHq8dofyp59+GgUFBdi3bx+USiXeeOMNBAQEAACqqqqQmpqKxYsXd3adRER9WlCgLx6aNhD3juuH/SfzsPdELv60/ltEhqkwM9mEkbHBkIjb/ctPIiLyEiJBEDptb3mbzYaamhr4+PhAJutZNyWZzdWw2TrtrXALt0Mmcg/HSksWaxO+OVeI3ek5KCqtRVCgD6YnmTAuIRwKucTT5ZGHcKwQuccTY0UsFkGnU7b5eKeG8sbGRsjl8s66XLdiKCfyXhwrbbMJAr69ch270rKQkVcJfx8ppo40YMpIAwL8eub3Y+o4jhUi93hjKG/37zoPHjyId9991+XY2rVrMWLECAwbNgzPP/88LBZL+yslIqJ2E4tEGBETjF8+OgovPzICMUY1tn19Df+z4hv8e89lFJXVerpEIiJyQ7vXlK9atQo6nc75cUZGBv7whz/AaDTCYDBg586dSEhI4LpyIqJuNtCgxkCDGgXmGuxJz8ahM/k48G0eRsYEY0ZyJKIiAjxdIhERtaHdoTwzM9Ol28rOnTuhUCiwceNGKJVKPP/889i6dStDORGRh4Tr/LF45mDcNz4K+07kIvVkHo5fLkGsUY0ZySYkROsgZscWIiKv0u5QXlFRAY1G4/z4m2++wZgxY6BU2tfIJCUl4eDBg51XIRERdYhaqcD8idGYNSYSX53OxxfHcvDXjWegD/LH9CQTxsSFQiphxxYiIm/Q7u/GGo0G+fn5AIDq6mqcPXsWo0aNcj5utVrR1NTUeRUSEdFt8VVIMT3JhDeeuQNPzR4CkQj4186LeOn9I9idlo26BqunSyQi6vPaPVM+bNgwrF+/HgMGDMBXX32FpqYmTJgwwfl4VlYWQkJCOrVIIiK6fVKJGHfEh2FMXCjOXy3FrrRsbNj/PbZ/cxWThukxbZQRGpXC02USEfVJ7Q7lzz77LBYtWoT//u//BgDMnTsXAwYMAAAIgoC9e/ciOTm5c6skIqJOIxKJEB+lQ3yUDtcKK7E7LRu707PxxbEcjIkLxYwkE/TBbbftIiKiztehPuXl5eU4efIkVCoVRo8e7TxeUVGBrVu3Ijk5GYMGDerUQrsa+5QTeS+Ola5XUl6HL9JzcOhMPhqtNiRG6zAz2YQYoxoi3hTaY3CsELnHG/uUd+rmQT0ZQzmR9+JY6T5VtY3YfzIPe0/korrOgv7hAZiZbMKImGCIxQzn3o5jhcg93hjK2718pVl2djb27duHnJwcAIDRaMTUqVNhMpk6ekkiIvIwlZ8c947rj+nJJnxztgB70nOwYus5hGh8MT3JhJT4MMhlEk+XSUTU63Ropvwvf/kLVq5c2aLLilgsxtNPP43nnnuu0wrsLpwpJ/JeHCueY7MJOPldCXalZeFqQRVUfjJMHWnAlBEGKH1lni6PbsGxQuSeXjFTvnHjRrz//vsYPnw4nnzySQwcOBAAcOXKFaxatQrvv/8+jEYj5s2b1/GqiYjIK4jFIowaFIKRscH4Lqccu9KysfXQVew8moXxiRG4a7QRwWpfT5dJRNTjtXumfN68eZDJZFi7di2kUtdMb7Va8fDDD8NisWDz5s2dWmhX40w5kffiWPEueSXV2J2ejaPni2ATBIweFIKZyZGIDFN5urQ+j2OFyD3eOFPe7s2DMjIyMGvWrBaBHACkUilmzZqFjIyM9l6WiIh6CH2wEkvuHoLlPx2L6UkmnMkw49UPjuFPH5/CuUwz2D+AiKj92r18RSaToba2ts3Ha2pqIJNxnSERUW+nUSnwwOQBmH1HPxw8nYcvj+XgzxtOwxiixIwkE0YPDoFU0u65HyKiPqnd3y0TEhLwySef4Pr16y0eM5vN2LBhA4YOHdopxRERkffz85FiZnIklv90LJ6YNRhNNgErd1zAsr8fwRfp2ahrsHq6RCIir9fuNeXHjh3D4sWL4e/vj/nz5zt38/z++++xefNm1NTU4IMPPsCoUaO6pOCuwjXlRN6LY6VnsQkCzmaYsSstG9/llMNPIcXkEXpMG2lAoFLh6fJ6NY4VIvd445ryDrVETE1Nxeuvv46CggKX4xEREfj1r3+NSZMmtbtQT2MoJ/JeHCs9V0Z+BXanZePk5RJIJCKMjQ/D9CQTwnX+ni6tV+JYIXJPrwnlAGCz2XDu3Dnk5uYCsG8eFBcXhw0bNmDNmjXYuXNnxyr2EIZyIu/FsdLzFZXWYs+xHHx9tgBWqw3DBgZhZnIkBhgCPV1ar8KxQuQebwzlHd7RUywWIzExEYmJiS7Hy8rKcPXq1Y5eloiIeqFQrR8WTY/FfeP6Y9+JXKSezMWpK9cxQB+ImckmDB0YBLFI5OkyiYg8psOhvDMUFxdjzZo1OH36NM6dO4fa2lqsWbMGycnJP/rcZcuWYcuWLS2ODx06FBs2bOiKcomI6DYF+Msxd0IUZo2JxKEz+fjiWA7e3XwWYVo/zEg24Y64UMikEk+XSUTU7Twayq9evYqVK1ciMjISsbGxOHXqVLue7+vri1dffdXlmFar7cwSiYioCyjkEkwbZcTkEXocv1SC3WnZ+GDXJWz+KhN3jjJg0nA9/H3YXpeI+g6PhvK4uDgcPXoUGo0Ge/fuxdKlS9v1fKlUijlz5nRRdURE1NUkYjGSh4QiaXAILmaVYXdaNjYdzMSOb7IwcVgE7hxlhC7Qx9NlEhF1OY+GcqWy7cXu7mpqakJdXV2nXIuIiDxDJBJhSD8thvTTIruoCnvSs7H3eC72nchF0uAQzEiOhDGE3+eJqPdyK5SvXr3a7QuePHmyw8W0V01NDUaOHIm6ujqo1Wrcd999+H//7/9BoWAfXCKinsoUqsJT98Rh3oRofHk8BwdP5+PI+SLE99diRrIJgyM1EPGmUCLqZdwK5W+88Ua7Ltod3yyDg4Px5JNPYvDgwbDZbNi/fz8++OADZGRk4J///GeXvz4REXUtXaAPqMPdqgAAIABJREFUHpw6EPek9MOBU3n48ngu3lz/LSJDVZiRbMKoQcGQiNu9MTURkVdyq095enp6uy+clJTUrvOb15S7232lNcuXL8eqVavwr3/9CykpKR26BhEReadGSxP2n8jFlgNXkFdSgxCtH+6bEI07k0zwUXh0NSYR0W1z67tYewO2pzzxxBNYtWoVjhw50u5Qzs2DiLwXxwo1GxGtxbCoJJy+ch270rLxj61nsXb3RUwZYcDUkQYE+Ms9XaJHcawQuadXbR7kjYKCgiCTyVBRUeHpUoiIqIuIRSIMjwnG8JhgfJ9bgV1pWdjxzTXsTs9GSkI4picZEarx83SZRETt0qtCeWFhISwWC3uVExH1EQMMgfgvQyIKzDXYk56Nw2fycfBUHkbEBmNGsgnREYGeLpGIyC09IpRnZ2cDAEwmEwCgoaEBFoulRRvEFStWAADGjRvXvQUSEZFHhev8sXjmYNw3Pgr7TuRi/8k8nLhcghijGjOSTUiM1kHMji1E5MU8Hsqbg3RGRgYA4LPPPsOJEycQEBCARx55BACwePFiAEBqaioAoKSkBHPnzsXs2bMRFRXl7L5y5MgRzJo1C6NHj+7+T4SIiDxOrVRg/sRozBoTiUOn8/HF8Ry8s/EMIoL8MSPJhDFxoZBK2LGFiLyPW91XulJsbGyrx/V6vTOET5kyBcCNUF5ZWYnXX38dp0+fRnFxMWw2G/r164e5c+di0aJFkEgk7a6DN3oSeS+OFeooa5MNxy4WY1daNnJLqqFWynHnaCMmDtXDz8fj81KdjmOFyD3eeKOnx0O5t2AoJ/JeHCt0uwRBwPlrpdh1NBsXs8rgI5dg0nA97hxlhEbVezac41ghco83hvLeN01ARER0C5FIhPj+OsT31yGrsAq70rKwJz0bXx7LwZghoZiebIIhuO0flkREXY2hnIiI+pTIMBWemfP/27vvgCrPu//j73PYyB4KMhUVFBHBgWjUqKi4oklMTBPNrKkjtmmetk/H0z5pnybNLzHDmGhTbRtN0ySNe4DiStSIqLh3XAwBQVQQVGT9/kBpKEZRgfsAn9d/5z73+ILnkg831319u/LowKsk7cxgy/4svjmYQ7cQT0bEBNIpwK1ROlOLiHyXpq/coOkrIpZLY0UaUtHVUjbuzmRDaiaXr5TSzteFETGBRHfyxmxuWuFcY0WkbjR9RURExMI4OdjwUL92xPcO5JuDOazdkc6cZQdp7ebA8N4B9Ivwxdbm7hcQEBG5G7pTfoPulItYLo0VaUwVFZXsPp5HYko6p7MLcXa0YUi0P4N7+OPkYGN0ebelsSJSN7pTLiIiYuHMZhM9w1rTI9Sb4xmXSExJZ9nW0yRsT6N/t7YM6x2At5uD0WWKSDOjUC4iInILJpOJ0EB3QgPdOZtXxJod6Xy19ywb92TSK6w18TGBBPu4GF2miDQTCuUiIiJ34OftxAujuvDIgBDW78rgq71n2XEkl85B7oyICSS8nYdWbBGR+6I55TdoTrmI5dJYEUtz5VoZm/dlkbQznUtF1/H3diI+JoDendtgbWU2rC6NFZG6scQ55QrlNyiUi1gujRWxVGXlFaQcPsealHTOni/Gw8WOoT0DGBDZFge7xv9jtMaKSN1YYijX9BUREZF7ZG1lpl+EL7FdfThwMp81Kel8sfEEK745w+BoP+J6+OPqZGd0mSLSBCiUi4iI3CezyURkBy8iO3hxKquQNSlpJGxPY+2OdPp29WF470B8PVsZXaaIWDCFchERkXrUvq0L0x6O4NzFKyTtyGDrgWw278smqqMX8TGBdPR3M7pEEbFACuUiIiINoI27I5OGhzK2fzs2pmayITWTPd+eJ8TPhRExQXTv6IVZK7aIyA0K5SIiIg3IxdGWcf3bMyImiK0Hslm7I50PlhygjYcj8b0D6NvVBxtrK6PLFBGDafWVG7T6iojl0liR5qS8ooLUY3kkpqSTlnMZl1a2xPXwZ1C0H63sbe7r3BorInWj1VdERERaOCuzmd6d29ArrDVH0y6SuCOdJZtPsTo5jQGRbRnWKwBPV3ujyxSRRqZQLiIiYgCTyUTnYA86B3uQkVvEmpR0Nu6umnveu0tr4nsHEtjG2egyRaSRKJSLiIgYLKC1E5PHdOHRge1J2pnB1/uy2H7oHOHtPIiPCaRLkDsmPRQq0qxpTvkNmlMuYrk0VqSlKb5Wyld7zrJ+VyYFxdcJbOPEiJggeoZ5Y2U2f+9xGisidWOJc8oVym9QKBexXBor0lKVllWQfCiHNSnp5Fy4gperPUN7BTCgW1vsbGuv2KKxIlI3CuUWTKFcxHJprEhLV1FZyb4T50lMSedEZgGt7K0ZFO1PXA9/XFrZknwohyVfn+RCYQkeLnY8MjCE2HAfo8sWsVgK5RZMoVzEcmmsiPzbicwCElPS2PvteaytzXRo68KJrEJKyyqq97G1NvPMiDAFc5HvYYmhXA96ioiINCEd/F2Z4d+N7Pxi1u7IYPO+rFr7XC+rYMnXJxXKRZqQ739aRERERCyWr2crnh0R9r3v5xeWNGI1InK/FMpFRESaME8Xu1tutzKbSD2WS4VmqYo0CQrlIiIiTdgjA0Owta7549zKbMLRzpoPlx7kt/NT2Lo/m7Lyiu85g4hYAs0pFxERacJuzhv/z9VXYjq3YdexXFYnp/G3hCMs33qK4b0D6R/ZFjub2sspioixtPrKDVp9RcRyaayI1M2txkplZSUHTuWzOjmNbzMLcHa0Ia5nAEOi/XC0tzGoUhFjafUVERERaVQmk4luIV50C/HieMYlEransXTzKRK3pzEoyo9hvQJwdbr1vHQRaTwK5SIiIi1EpwA3OgW4kX7uMgnb01izI511uzLp382X+JhAvN0cjC5RpMVSKBcREWlhAts4M2VsVx4ecIXE7els2Z/F13uz6N2lNSP7BOHv/f1/YheRhqFQLiIi0kK1cXfk2RFhjH2gHUk70/lqTxbbD52jewcvRsUGEeLnanSJIi2GHvS8QQ96ilgujRWRurnfsVJ0tZQNqZms35VB8bUywgLdGBkbRHiwByaTqR4rFTGWHvQUERERi+XkYMPYB9oxvHcAm/dmsXZnBu98sY8gH2dG9QkiupM3ZrPCuUhDUCgXERGRGuxtrRnWO5BB0f4kH8ohYXsac5YdxMfDkRF9AokN98HaSv0HReqTQrmIiIjcko21mQGRbXkgwpddx3JJSE7j7wlHWb71NMN7BTIgsi12tmpEJFIfFMpFRETktsxmE707t6FXWGsOnr7A6uQ0PtvwLSu3nWFoT38G9/CnlRoRidwXhXIRERGpE5PJRER7TyLae/Jt5iVWJ6exdMtpElLSqxsRuakRkcg9MXRCWG5uLjNnzmTSpElERUURGhpKSkrKXZ+nvLycMWPGEBoayscff1z/hYqIiEgNHf3dePmxSH7/fG+6d/Bi7Y50fjE3mYVrjpJ76arR5Yk0OYaG8tOnTzNv3jzOnTtHaGjoPZ/n888/JzMzsx4rExERkboIaO3Ejx4K5/UX+9AvwoetB7L51UfJ/GXFITJzi4wuT6TJMDSUh4eHs337dpKSkvjhD394T+e4dOkS77//Pi+88EI9VyciIiJ11cbdkWfiw/h/U/oyvFcge06c53d/28GsL/dxIrPA6PJELJ6hodzJyQl3d/f7OsesWbPw9/dn7Nix9VSViIiI3Ct3ZzseH9yBt6b2ZVz/dpzMKuT1f6Tyxqe7OXgqH/UsFLm1Jv2g57Fjx/jiiy9YuHChOo2JiIhYECcHGx7q147hvQL5el8Wa3ek886/9hHYxolRscH0UCMikRqadCj/4x//SFxcHD179tScchEREQtkZ2vFsF4BDI72I/lgDgkp6cxddpA2Ho6MjAkktqsaEYlAEw7la9asYc+ePSQmJtbL+Tw9nerlPHfL29vZkOuKNDUaKyJ1Y8lj5REfV8YO7kTygSy+3PAtf088ysptZxj3YAeGxwRhb9dkY4k0QZY2Vprkp7+kpIQ333yTp59+moCAgHo5Z35+ERUVjTvPzdvbmby8y416TZGmSGNFpG6aylgJbevCbyZGc+hGI6L5yw/yedIx4npUNSJyclAjImlYRowVs9l025vATTKU//Of/+TixYs89NBD1dNWcnJyACgoKCAzM5M2bdpgY6NBLSIiYolMJhNd23vStb0nJ84WkJCcxrKtp0nckc6D3dsyrFcg7s5qRCQtR5MM5VlZWVy5cuWWK67MmTOHOXPmkJCQQEhIiAHViYiIyN3o4OfKj8d3IzO3iITtaSTtzGBDaib9InwZERNIa3dHo0sUaXBNIpSnp6cDEBgYCMD48eOJiYmpsU9+fj6/+93vePTRRxk8eDA+Pj6NXqeIiIjcO//WTrz4UDjjBrRnTUo6W/dns3lfFr3CWjMqNpiA1sY8/yXSGAwP5XPmzAHg5MmTACxfvpzU1FRcXFyYOHEiAM8++ywAGzduBCA0NLRWB9Cb01g6depEXFxcY5QuIiIiDaC1mwNPDw/loX7BJO3MYNOes+w4kku3EE9GxQbR0d/N6BJF6p3hoXzWrFk1Xi9evBgAPz+/6lAuIiIiLY+bkx2PD+rAqNggNqZmsm5XJn/6x246+bsyMjaYiPYe6lMizYapUq21AK2+ImLJNFZE6qa5j5WS6+Vs3p/FmpR0Ll4uIbC1EyNjg+gZ2lqNiOSuaPUVERERkXtkZ2vF0J4BDIryI/lQDonb0/nz8kO0cT/FiD5BxIb7YGOtRkTSNCmUi4iISJNibWWmf7e29Ovqy+7jeazensbHiUdZvvU0w3oFMLB7W+xtFXGkadEnVkRERJoks9lEz7DW9Aj15vCZi6xOPsMXG0+watsZ4noGMESNiKQJUSgXERGRJs1kMhHezoPwdh6cPFvA6uQ0lm89zZqUdAZ2b8vw3mpEJJZPoVxERESajZCbjYjyqhoRrd+VycbdmfTt6suIPoG0USMisVAK5SIiItLs+Hs78eKYcB7uX9WIaMv+bLbsr2pENLJPEIFtnI0uUaQGhXIRERFptrzdHJh0sxHRrgw27a5qRBTRvqoRUacANSISy6B1ym/QOuUilktjRaRuNFbu7Mq1UjbuPkvSzgyKrpbS0d+VUbFBRLT3VCOiFkTrlIuIiIgYyNHehtF9gxnaK4At+7JYsyOd977cT0BrJ0b2CaJXmBoRiTF0p/wG3SkXsVwaKyJ1o7Fy98rKK9h+6ByJKWlk51+htbsDI2IC6dvVV42ImjHdKRcRERGxINZWZh7o5kvfCB/2HM9jdXIaC9Ycu9GIKJAHo9SISBqHPmUiIiLS4plNJnqEtia6kzeH0y6SkJzGvzadYHXyGYb08CeuZ4AaEUmDUigXERERucFkMhEe7EF4sAcnswpISE5jxTdnWLMjnQe7+zGsVwAeLvZGlynNkEK5iIiIyC2EtHVlxqPdOJtXRML2dNbvymRDaiZ9u/owsk8QbTzUiEjqj0K5iIiIyG34eTsxeUwXHu7fjsQd6Wzdn83W/dn0vNGIKMhHjYjk/imUi4iIiNSBl5sDk4aF8lC/dqzbmcGmPZnsPJpL1/YejOpT1YhIa53LvdKSiDdoSUQRy6WxIlI3GiuN68q1MjbtySRpZwaXr5TSwc+VkbFBRIaoEZGls8QlERXKb7hTKL96tZiiokuUl5fV2zXNZjMVFRX1dj4xltlshbW1Lc7ObtjY2BpdTrOioCFSNxorxrheWs6W/dmsSUkjv7AEf28nRsYG0iusNVZmrXVuiRTKLdjtQvnVq8VcvnwRNzdvbGxs6+23X2trM2VlCuXNQWVlJRUV5ZSUXKWoqABnZ3ccHFoZXVazoaAhUjcaK8YqK68g5fA5ErZXNSLydrNnREwQ/SJ8sLG2Mro8+Q6Fcgt2u1Cel3cWV1cvbG3t6vWaCuXN0/Xr1ygsvICXV1ujS2k2FDRE6kZjxTJUVFay99vzrE4+w+nsy7g62TK8VyADu7fFwU6P81kCSwzl+mTUQXl5maYjSJ3Z2NhRVlZqdBkiImIQs8lEdCdvojp6cSTtIqu/04hocLQ/cT39cXZUrpCaFMrrSA9sSF3psyIiIlD186BLsAddgj04nV3I6uQ0Vm47w9qd6QyIbEt870A1IpJqCuUiIiIiDaydrwsvPRLB2fPFJG5PY2PqWTbtPkvsjUZEPmpE1OIplEuDeumlFwH44IO/NOqxIiIilsjPqxU/HN2Fcf3bsTYlg837s/hmfzY9Qr0ZFRusRkQtmEJ5C/XAAz3rtN+XX67A11cPLIqIiNQnL1cHnhrWiTH9glm3K4ONuzPZdSyP8HYejI5VI6KWSKuv3HC71VdyctLw8Qmq92saufrK2rUJNV7/61+fce5cNjNmvFJj+4ABg3BwcLjn65SWVj3waGNj06jHGq2hPjMtlVaUEKkbjZWm62YjonU7Myi8UkqInwuj+gTTrYMnZoXzeqfVV8RiDB8+ssbrr77aQEHBpVrb/9O1a9ewt6/7Qyn3E6ibYhgXERG5F4721oyKDWZozwC2HshmTUo67y/ej593K0b1CaJXZzUiau70ryvf66WXXuTZZ5/k8OGDTJ36AoMH9+PTTxcAsGXLV/z85z9h7Nh4Bg2K5fHHx/Lxx/MpLy+vdY6bc8MBdu/exQMP9OTrrzfy8cfzGTduBIMH9+UnP5lKZmZGvR0LsHjxv3jssbEMHtyPyZOfZt++PbXOKSIiYklsbawYHO3P6y/24YejO1NZCX9ZeZhf/2U7m/acpbSs/M4nkSZJd8oNknwohyWbT5FfcA1PFzseGRhCbLiP0WXVcunSRX7xi58ybFg88fGjaNOmqsaEhFU4ODgyYcJTODo6kJq6i/nz/0xxcTHTp//kjuddsOCvmM1WPPnk01y+XMhnn33C73//P8ybt6Bejl26dBHvvvsm3btHM2HCD8jOzuZXv/oZzs7OeHu3vvdviIiISCOwtjLTt6svfcJ92PfteVZvT+OTtcdYsfU0w3oF8GCUnxoRNTP61zRA8qEcFiQe5fqN+eT5hSUsSDwKYHHB/Pz5PH75y98yevTYGttfffWP2Nn9exrLuHHjeeut11m69EsmT56Kre3tmyKUlZXxt78twNq66iPo4uLKrFkzOXXqBO3bd7ivY0tLS5k/fy7h4RG8996c6v06dOjIa6+9qlAuIiJNhtlkIqqTN907enE0/RIJyWf48quTrE5OY3APP+J6BuCiRkTNgkL5ffjmQDZb92ff9XEnswooK6/5UOn1sgr+nnCEzXuz7vp8D3TzpV+E710fVxf29vbEx4+qtf27gfzKlWKuXy8lMjKK5cuXkJZ2ho4dO932vKNGPVQdlgEiI7sDkJV19o6h/E7HHj16mIKCAqZNe7jGfkOHxvP+++/c9twiIiKWyGQy0TnInc5B7pzOLiQhOY3V29JI2pFR1YgoRo2ImjqFcgP8ZyC/03YjeXu3rhFsbzp16iTz5s1l9+6dFBcX13ivuLjojue9OQ3mJmdnFwAuX77zk9B3OjYnp+oXJX//gBr7WVtb4+vbML+8iIiINJZ2vi5MfySCrPPFJKaksWnPWTbtOUuf8DaM7BOEr2cro0uUe6BQfh/6RdzbHeqfz/mG/MKSWts9Xez476ei66O0evPdO+I3Xb58mRkzXsTR0YkXXpiCn58/tra2HD9+lLlzZ1NRcedlHs1mq1tur8sKnfdzrIiISHPR1qsVL4zqwrgH2rNmRzpb9mWx7UAO0aHejIoNItjHxegS5S4olBvgkYEhNeaUA9ham3lkYIiBVdXdnj2pFBQU8Nprb9G9+79/icjOvvupNw3Bx6fqF6XMzAwiI6Oqt5eVlZGdnU1IyO2nx4iIiDQlnq72PDW0qhHR+l0ZbEg9S+qxPMKD3RkZG0xYoBoRNQVaEtEAseE+PDMiDE/XqrvQni52PDMizOIe8vw+5hvrpH73znRpaSlLl35pVEk1hIV1wdXVlRUrllJWVla9fd26NVy+XGhgZSIiIg3HxdGWRwaEMHNaXx57MISMvGLe+mwPr32Syp7jeVToL8oWTXfKDRIb7kP/yLaGdfS8HxER3XB2duG1115l/PgJmEwm1q5NwFLGuo2NDc8//yLvvvsWL788jUGDhpCdnU1i4kr8/Px1t0BERJo1BztrRvQJIq6nP1v3Z5OYks7sJQfw82rFyD5B9O6iRkSWSP8ictdcXd1488138fT0Yt68uXz22T/o2TOGadN+bHRp1R59dAIvv/wzcnKy+fDDWezbt4c33ngHJydnbG3tjC5PRESkwdlYWzEo2p8//agPk8d0ARPMW3WYX320nY27M7leqkZElsRUqafjAMjPL6Ki4tbfipycNHx8gur9mtbW5iZ5p7ypqqioYPTooQwcOIj//u//adBrNdRnpqXy9nYmL+/OK/OItHQaK3I7FZWV7DtxnoTkNE5mFeLSypZhvQIY1AIbERkxVsxmE56eTt/7fsv6F5AWo6SkBDu7mnfE16xZTWFhAVFRPQyqSkRExDhmk4mojt507+DFsfRLrN6exqKbjYii/RjaMwCXVmpEZBRDQ3lubi4LFy5k3759HDx4kCtXrrBw4UJiYmLueOyCBQtITEzkzJkzFBcX4+vry8CBA5k6dSoeHh6NUL1Ysv379zJ37mwefHAwLi6uHD9+lNWrV9C+fQiDBsUZXZ6IiIhhTCYTYUHuhAW5cyanqhFRQnIa63Zm0D+yLfG9A6sXo5DGY2goP336NPPmzSMoKIjQ0FD27NlT52MPHz5Mx44diY+Pp1WrVpw+fZp//etfbNmyhWXLlmFvrw9TS9a2rR9eXt4sWvQFhYUFuLi4Eh8/iilTXsLGxsbo8kRERCxCsI8L0x6OIDu/mMTt6Xy15yxf7TlLny5tGNEniLZeakTUWAydU15UVERpaSnu7u6sX7+e6dOn1/lO+a0kJSUxY8YMZs2aRXx8/F0dqznlUp80p7x+aZ6sSN1orMj9ulB4jTU70tm8N4vSsgqiOlU1Imrn27waEWlO+X9wcvr+wu5F27Ztgbq1ahcRERGRmjxc7HkyrhOj+wazflcmG1Mz2X08jy7B7ozqE0RYkLuWFm4gTf5BzwsXLlBeXk5aWhozZ87E2tqaXr16GV2WiIiISJNV1YioPSNiAvlq71mSdmTw1ud7aefrwqjYILp39MKscF6vmnQoLy4uJjY2tvq1j48Pb7/9NsHBwcYVJSIiItJMONhZMyImiLge/nxzIIeE7Wl8sOQAbb1aMbJPIL07t8HaSm1v6kOTDuX29vb8/e9/p6SkhKNHj5KUlERRUdE9net2c3xyc81YWzfMB66hzivGMpvNeHs7G11Gs6Lvp0jdaKxIQ3nM141HhnRiy74sFm/8lvmrjrDimzM88mAH4mKCsLOxMrrEu2JpY6VJh3IrKyv69u0LwKBBg+jbty+PP/44np6eDBo06K7OdbsHPSsqKhrkgUw96Nl8VVRU6GGreqSH10TqRmNFGkN4gCtdnu7BvpP5JCSn8eelB/jn2qMM7RXAoCh/HO0tP17qQc8GFhkZia+vLytXrrzrUC4iIiIidWMymejewYvIEE+OZ1xidXIai78+RcL2NAZH+6sR0T1oVqEcqjo5avUVERERkYZnMpkIDXQnNNCdtJzLrN5e1YgoaWcG/bv5Eh8TiJerg9FlNglNIpSnp6cDEBgYCFQF79LS0lpLKq5fv54LFy4QHh7e6DWKiIiItGRBPs5MG9e1qhFRSjpf783i671ZxNxoROSnRkS3ZXgonzNnDgAnT54EYPny5aSmpuLi4sLEiRMBePbZZwHYuHEjAHl5eTz88MOMGDGCkJAQrK2tOXToECtWrMDPz4+nn3668b+QFi4hYSWvv/57vvxyBb6+VevFjx8/hqioHvzmN6/e9bH3a/fuXfz4x1N4//0/Ex3ds17OKSIiInfm69mK50d2ZtwD7Vi7I4Ov951l28Ecojp6MSo2mPZtm1cjovpieCifNWtWjdeLFy8GwM/PrzqU/yc3NzfGjBlDSkoKK1eupLS0FF9fX5544gmmTZuGh4dHg9fd1P3iFz9l9+6drFy5DgeHW/9Z6ZVXXuLQoQOsWJGEnZ1dI1dYN+vXr+XChXwef/xJo0sRERGR7/BwsecHcR0Z3TeIDamZbEjNZM+3u+gc5M7I2CC6qBFRDYaH8mPHjt1xn5t3yG9ycnLid7/7XUOV1CIMHTqcbdu2sHXr1wwdGl/r/YsXL5CaupNhw0bccyD/5z8XYzY37JKPGzYk8e23x2uF8u7do9mw4RtsbGwa9PoiIiJye86Otozr357hvQP5em8Wa3em8/bne2nn68zIPsFEdVIjIgAtkt1C9e//IA4Ojqxfv/aW72/cuJ7y8nKGDasd2OvK1tYWa2tjfu8zm83Y2dk1+C8FIiIiUjcOdtbExwTy5pRYno4PpfhqGR8uPcBv56fwzYFsyspb9jLRht8pF2PY29vTv/9ANm1aT2FhIS4uNed3rV+/Fk9PTwICgpg58w1SU3dw7tw57O3tiY7uyfTpP7nj/O9bzSk/deok7733FgcPHsDV1ZWxYx/By8u71rFbtnzFihVLOX78GIWFBXh7t2bkyDFMmvQcVlZVzQleeulF9u7dDcADD1TNG/fx8WXRopXfO6d8w4Yk/vGPj0lLO4OjYyv69evP1Kk/xs3NrXqfl156kaKiIn73uz/wzjtvcuTIIZydXXjssSd46qln7u4bLSIiIjXYWFvxYHc/+nfzZdfRPFYnp/HX1UdYtuUU8TFBPNDNt8k1IqoPCuUG2ZGzm5Wn1nDh2iXc7dx4KCSe3j7RjVrD0KHxJCUl8tVXG3jooYert+fkZHPw4H7Gj3+CI0cOcfDgfuLihuPt3Zrs7CyWLVvMjBk/4h//+BJ7e/s6Xy8//zw//vEUKioqmDjxGeztHVixYuktp8ckJKzCwcGRRjFjAAASOUlEQVSRCROewtHRgdTUXcyf/2eKi4uZPv0nADzzzPNcvXqVc+eymTHjFQAcHBy/9/o3HygND49g6tQfk5t7jsWLv+DIkUPMm7ewRh2FhQX813/9mEGDhjBkyDA2bVrP3Lmzad++A7Gx/er8NYuIiMitWZnNxHRpQ+/Ordl/Mp/V29P4dN1xVnxzmqE9Axgc7YejfcuZhqpQboAdObv559HFlFaUAnCx5BL/PFr1gGtjBvNevWJwc3Nn/fq1NUL5+vVrqaysZOjQ4YSEdGDQoLgax/XrN4ApU57jq682EB8/qs7X+/TTBRQUXGL+/E8IDQ0DYMSI0fzgBw/X2vfVV/+Ind2/A/+4ceN5663XWbr0SyZPnoqtrS29evVhyZIvKSi4xPDhI2977bKyMubOnU2HDp2YPfsjbG2rGhqEhobx6qu/YeXKpYwf/0T1/rm55/jf//1j9Xz70aPHMn78aFavXq5QLiIiUo9MJhORHbyI7OBV3YhoyeZTJKak8WCUH8N6BeLaAhoRKZTfh5TsVJKzd971cacL0imrLKuxrbSilE+PLGJb1o67Pl+sby9ifHvc9XHW1tYMHhzHsmWLOX/+PF5eXgCsX5+Ev38AXbp0rbF/WVkZxcVF+PsH4OTkzPHjR+8qlCcnf0NERGR1IAdwd3dn6NARLF36ZY19vxvIr1wp5vr1UiIjo1i+fAlpaWfo2LHTXX2tR48e5uLFC9WB/qbBg4fy4Yez2Lbtmxqh3MnJibi44dWvbWxs6Nw5nKyss3d1XREREam7TgFudApwIy3nMgnb01izPZ11OzPpH+nLiN6BeLk130ZECuUG+M9AfqftDWno0HiWLPmSjRuTePzxJzlz5jQnThznuecmA1BSco1PPvmYhISV5OXlUllZWX1sUVHRXV3r3LkcIiIia20PDAyqte3UqZPMmzeX3bt3UlxcXOO94uK7uy5UTcm51bXMZjP+/gGcO5ddY3vr1m1qLdPk7OzCyZMn7vraIiIicneCfJyZOq4r5y5cITEljc17s/h6TxYxXVozsk8Qft5Odz5JE6NQfh9ifHvc0x3q//nmdS6WXKq13d3OjZejp9RHaXUWERGJr68f69at4fHHn2TdujUA1dM23n33LRISVvLYYz+ga9eIG11UTbz66q9rBPT6dPnyZWbMeBFHRydeeGEKfn7+2Nracvz4UebOnU1FRcM/nW023/oBk4b6mkVERKS2Nh6OPDuiM2MfaM/aHVVdQpMPnSOqoxcjY4MIaetqdIn1RqHcAA+FxNeYUw5gY7bhoZB7X37wfsTFDeOTT/5OZmYGGzYkERraufqO8s154zNm/LR6/5KSkru+Sw7Qpo0PmZkZtbanp6fVeL1nTyoFBQW89tpbdO/+7zn22dlZtzhr3dY19fHxrb7Wd89ZWVlJZmYG7dqF1Ok8IiIi0vjcne14YkhHRvcNZv2ujBuNiM4TFujGqNhgugQ3/UZEWsTZAL19onky7FE87KuW4XO3c+PJsEcbffWVm4YNGwHABx+8S2ZmRo21yW91x3jx4i8oLy+/6+vExvbjwIF9HDt2tHrbxYsXWbcuscZ+N9cW/+5d6dLS0lrzzgEcHBzq9AtCWFgX3N09WLZsEaWl//5laNOmDeTl5dK3rx7eFBERsXRODjaM69+et6b1ZcLgDuRcuMLbX+zlDwt2kXosl4om/Bdt3Sk3SG+faPr696SszPiF8tu1a0+HDp3YunUzZrOZIUP+/YBj374PsHZtAq1aOREc3I5Dhw6wa9cOXF3v/s9FTz75DGvXJvDKK9MZP/4J7OzsWbFiKW3a+FJU9G31fhER3XB2duG1115l/PgJmEwm1q5N4FbjLDQ0jKSkRGbPfoewsC44ODjywAMDau1nbW3N1KkzeP313zNjxo+IixtGbu45Fi36gvbtQxgzpvYKMCIiImKZ7G2tGd47kMHR/mw7mE1iSjofLj2Ir6cjI2KC6BPeBmurpnXvWaFcABg2LJ4TJ44TFdWjehUWgJ/85GeYzWbWrUukpOQ6ERGRvPfeh7zyyoy7voaXlxfvv/8R7777Jp988nGN5kFvvPF/1fu5urrx5pvv8sEH7zFv3lycnV0YNmwEPXv25pVXXqpxzrFjH+X48aMkJKziiy/+iY+P7y1DOcDIkWOwtbXl008X8OGHs2jVqhVDh8YzZcqMW66VLiIiIpbNxtrMwO5+9O/Wll3HclmdnMbfEo6wbOsphvcOZEBk2ybTiMhUqSfXAMjPL6Ki4tbfipycNHx8aq8Qcr+src0Wcadc6l9DfWZaKm9vZ/LyLhtdhojF01iRlq6yspIDpy6QkHyG45kFODnYMLRXAENuNCJKPpTDkq9PcqGwBA8XOx4ZGEJsuE+j1GY2m/D0/P5VY3SnXERERESaBZPJRLcQT7qFeHI84xIJ29NYuvkUidvT6BTgypG0S5TeuCGaX1jCgsSq59waK5jfjkK5iIiIiDQ7NxsRpZ+rakS040hurX2ul1Ww5OuTFhHKm9YMeBERERGRuxDYxpkpY7t+7/v5hSWNWM33UygXERERkWbP0+XWizp83/bGplAuIiIiIs3eIwNDsLWuGX1trc08MtAyGghqTrmIiIiINHs3540btfrKnSiU11FlZWWTb98qjUOrjIqIiFim2HAfYsN9LHL5UE1fqQMrK2tKS68bXYY0EaWlJVhb2xhdhoiIiDQhCuV14OTkxqVLeVy/XqK7oHJLlZWVlJeXUVx8mUuXztOqlavRJYmIiEgToukrdeDg0AqAgoLzlJeX1dt5zWYzFRXq6NlcmM1W2NjY4u7eGhsbW6PLERERkSZEobyOHBxaVYfz+mKJ85lEREREpPFp+oqIiIiIiMEUykVEREREDKZQLiIiIiJiMIVyERERERGDKZSLiIiIiBhMq6/cYDYb063TqOuKNDUaKyJ1o7EiUjeNPVbudD1TpbrhiIiIiIgYStNXREREREQMplAuIiIiImIwhXIREREREYMplIuIiIiIGEyhXERERETEYArlIiIiIiIGUygXERERETGYQrmIiIiIiMEUykVEREREDKZQLiIiIiJiMGujC2hpcnNzWbhwIfv27ePgwYNcuXKFhQsXEhMTY3RpIhZj//79LF26lJSUFLKysnBzcyMqKoqXX36ZoKAgo8sTsRgHDhzgz3/+M4cPHyY/Px9nZ2fCwsKYPn060dHRRpcnYtHmzZvHzJkzCQsLY/ny5UaXo1De2E6fPs28efMICgoiNDSUPXv2GF2SiMWZP38+u3fvJj4+ntDQUPLy8vj0008ZN24cixYtIiQkxOgSRSxCRkYG5eXlPPbYY3h7e3P58mVWrlzJxIkTmTdvHv369TO6RBGLlJeXx9y5c3F0dDS6lGqmysrKSqOLaEmKioooLS3F3d2d9evXM336dN0pF/kPu3fvpmvXrtja2lZvO3PmDGPGjGHUqFG88cYbBlYnYtmuXr1KXFwcXbt25aOPPjK6HBGL9Mtf/pKsrCwqKyspLCy0iDvlmlPeyJycnHB3dze6DBGLFh0dXSOQAwQHB9OxY0dOnjxpUFUiTYODgwMeHh4UFhYaXYqIRdq/fz8rVqzgV7/6ldGl1KBQLiJNQmVlJefPn9cvtSK3UFRUxIULFzh16hTvvPMOx48fJzY21uiyRCxOZWUl//d//8e4cePo3Lmz0eXUoDnlItIkrFixgnPnzvHTn/7U6FJELM6vf/1r1q5dC4CNjQ1PPPEEU6ZMMbgqEcuzbNkyTpw4wYcffmh0KbUolIuIxTt58iR/+MMf6NGjB2PHjjW6HBGLM336dCZMmEBOTg7Lly/n+vXrlJaW1poGJtKSFRUV8fbbb/Piiy/SunVro8upRdNXRMSi5eXl8aMf/QhXV1dmzZqF2az/tkT+U2hoKP369ePRRx/lr3/9K4cOHbK4+bIiRps7dy42NjY899xzRpdyS/rpJiIW6/Lly0yePJnLly8zf/58vL29jS5JxOLZ2NgwZMgQkpKSuHbtmtHliFiE3NxcFixYwJNPPsn58+fJzMwkMzOTkpISSktLyczMpKCgwNAaNX1FRCxSSUkJU6ZM4cyZM3z88ce0b9/e6JJEmoxr165RWVlJcXEx9vb2RpcjYrj8/HxKS0uZOXMmM2fOrPX+kCFDmDx5Mj/72c8MqK6KQrmIWJzy8nJefvll9u7dy5w5c+jevbvRJYlYpAsXLuDh4VFjW1FREWvXrsXX1xdPT0+DKhOxLP7+/rd8uPO9997jypUr/PrXvyY4OLjxC/sOhXIDzJkzB6B6veXly5eTmpqKi4sLEydONLI0EYvwxhtvsHHjRgYNGsSlS5dqNHVo1aoVcXFxBlYnYjlefvll7OzsiIqKwtvbm+zsbJYsWUJOTg7vvPOO0eWJWAxnZ+db/uxYsGABVlZWFvFzRR09DRAaGnrL7X5+fmzcuLGRqxGxPJMmTWLHjh23fE/jROTfFi1axPLlyzlx4gSFhYU4OzvTvXt3nn/+eXr37m10eSIWb9KkSRbT0VOhXERERETEYFp9RURERETEYArlIiIiIiIGUygXERERETGYQrmIiIiIiMEUykVEREREDKZQLiIiIiJiMIVyERERERGDKZSLiIhhJk2axODBg40uQ0TEcNZGFyAiIvUrJSWFp59++nvft7Ky4vDhw41YkYiI3IlCuYhIMzV69GgGDBhQa7vZrD+SiohYGoVyEZFmqkuXLowdO9boMkREpA50u0REpIXKzMwkNDSU2bNns2rVKsaMGUNERAQPPvggs2fPpqysrNYxR48eZfr06cTExBAREcHIkSOZN28e5eXltfbNy8vjj3/8I0OGDKFr167Exsby3HPP8c0339Ta99y5c7zyyiv06tWLyMhIXnjhBU6fPt0gX7eIiCXSnXIRkWbq6tWrXLhwodZ2W1tbnJycql9v3LiRjIwMnnrqKby8vNi4cSMffPABWVlZ/OlPf6re78CBA0yaNAlra+vqfTdt2sTMmTM5evQob7/9dvW+mZmZ/OAHPyA/P5+xY8fStWtXrl69yr59+9i2bRv9+vWr3vfKlStMnDiRyMhIfvrTn5KZmcnChQuZNm0aq1atwsrKqoG+QyIilkOhXESkmZo9ezazZ8+utf3BBx/ko48+qn599OhRFi1aRHh4OAATJ07kpZdeYsmSJUyYMIHu3bsD8Nprr3H9+nU+//xzwsLCqvd9+eWXWbVqFePHjyc2NhaA3//+9+Tm5jJ//nz69+9f4/oVFRU1Xl+8eJEXXniByZMnV2/z8PDgrbfeYtu2bbWOFxFpjhTKRUSaqQkTJhAfH19ru4eHR43Xffv2rQ7kACaTiR/+8IesX7+edevW0b17d/Lz89mzZw9Dhw6tDuQ39506dSpr1qxh3bp1xMbGcunSJbZs2UL//v1vGaj/80FTs9lca7WYPn36AJCWlqZQLiItgkK5iEgzFRQURN++fe+4X0hISK1tHTp0ACAjIwOomo7y3e3f1b59e8xmc/W+6enpVFZW0qVLlzrV2bp1a+zs7Gpsc3NzA+DSpUt1OoeISFOnBz1FRMRQt5szXllZ2YiViIgYR6FcRKSFO3nyZK1tJ06cACAgIAAAf3//Gtu/69SpU1RUVFTvGxgYiMlk4siRIw1VsohIs6NQLiLSwm3bto1Dhw5Vv66srGT+/PkAxMXFAeDp6UlUVBSbNm3i+PHjNfb9y1/+AsDQoUOBqqknAwYMYPPmzWzbtq3W9XT3W0SkNs0pFxFppg4fPszy5ctv+d7NsA0QFhbGM888w1NPPYW3tzcbNmxg27ZtjB07lqioqOr9fvOb3zBp0iSeeuopnnzySby9vdm0aRNbt25l9OjR1SuvAPz2t7/l8OHDTJ48mXHjxhEeHk5JSQn79u3Dz8+Pn//85w33hYuINEEK5SIizdSqVatYtWrVLd9LSkqqnss9ePBg2rVrx0cffcTp06fx9PRk2rRpTJs2rcYxERERfP7557z//vt89tlnXLlyhYCAAH72s5/x/PPP19g3ICCAxYsX8+GHH7J582aWL1+Oi4sLYWFhTJgwoWG+YBGRJsxUqb8jioi0SJmZmQwZMoSXXnqJGTNmGF2OiEiLpjnlIiIiIiIGUygXERERETGYQrmIiIiIiME0p1xERERExGC6Uy4iIiIiYjCFchERERERgymUi4iIiIgYTKFcRERERMRgCuUiIiIiIgZTKBcRERERMdj/B1+sJvrwwbqIAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ovoUEt8y7YS"
      },
      "source": [
        "We do appear to be severely over-fitting from the third epoch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HDNPfN7qSfAY"
      },
      "source": [
        "# Part III - Performance & Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkyubuJSOzg3"
      },
      "source": [
        "## S5. Performance On Test Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "huAaDukC1Orh"
      },
      "source": [
        "### 5.1. Baseline Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TELGvAm56e0T"
      },
      "source": [
        "\n",
        "The quick summary:\n",
        "* It uses `tf-idf` for vectorization.\n",
        "* It uses `multinomial naive bayes` for classification.\n",
        "* F1 score: `0.221`   (on the test set)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bcGu2FTQ1S7w",
        "outputId": "6c482469-34a3-44f5-b148-0f246dae3e8f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn import metrics\n",
        "\n",
        "# Rename our dataset variables to be consistent with the example.\n",
        "liar_liar_train = train\n",
        "liar_liar_test = test\n",
        "\n",
        "# Vectorize the text using tf-idf.\n",
        "vectorizer = TfidfVectorizer()\n",
        "\n",
        "# Before we can vectorize the text, the tf-idf model needs\n",
        "# to analyze the dataset to build a vocabulary and word counts.\n",
        "print('Analyzing dataset for tf-idf...\\n')\n",
        "vectorizer.fit(liar_liar_train.data)\n",
        "\n",
        "# Convert the text samples into tf-idf vectors.\n",
        "print('Vectorizing the text samples...\\n')\n",
        "vectors_train = vectorizer.transform(liar_liar_train.data)\n",
        "vectors_test = vectorizer.transform(liar_liar_test.data)\n",
        "\n",
        "print('{:>6,} train samples with {:<7,} features.'.format(vectors_train.shape[0], vectors_train.shape[1]))\n",
        "print('{:>6,}  test samples with {:<7,} features.'.format(vectors_test.shape[0], vectors_test.shape[1]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Analyzing dataset for tf-idf...\n",
            "\n",
            "Vectorizing the text samples...\n",
            "\n",
            "10,138 train samples with 13,915  features.\n",
            " 1,250  test samples with 13,915  features.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mvutv2D53bJo",
        "outputId": "f26833a9-5ea0-4ec0-e0fd-1dc6e75c2ed0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Following the example, we'll use a \"multinomial Naive Bayes\" classifier.\n",
        "clf = MultinomialNB(alpha=.01)\n",
        "\n",
        "# Train the classifier on the training set.\n",
        "print('Training the Naive Bays classifier...\\n')\n",
        "clf.fit(vectors_train, liar_liar_train.target)\n",
        "\n",
        "# Run prediction on the test set.\n",
        "print('Predicting labels for the test set...\\n')\n",
        "pred = clf.predict(vectors_test)\n",
        "\n",
        "# Use the F1 metric to score our classifier's performance on the test set.\n",
        "score = metrics.f1_score(liar_liar_test.target, pred, average='macro')\n",
        "\n",
        "# Print the F1 score!\n",
        "print('F1 score: {:.3}'.format(score))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training the Naive Bays classifier...\n",
            "\n",
            "Predicting labels for the test set...\n",
            "\n",
            "F1 score: 0.221\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "16lctEOyNFik"
      },
      "source": [
        "### 5.2. Evaluate BERT on Test Set\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhR99IISNMg9"
      },
      "source": [
        "\n",
        "Now we can apply our fine-tuned BERT model to generate predictions on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hba10sXR7Xi6",
        "outputId": "4a3f3c79-1c3c-4ebd-b5ec-e2a2bf23447f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Create the DataLoaders.\n",
        "\n",
        "# Combine the features into a dataset object.\n",
        "test_data = TensorDataset(test_input_ids, test_attention_masks, test_labels)\n",
        "\n",
        "# Create a sequential sampler--no need to randomize the order!\n",
        "test_sampler = SequentialSampler(test_data)\n",
        "\n",
        "# Create the data loader.\n",
        "test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)\n",
        "\n",
        "\n",
        "# Predict labels for all test set examples.\n",
        "\n",
        "print('Predicting labels for {:,} test comments...'.format(len(test_input_ids)))\n",
        "\n",
        "# Put model in evaluation mode\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables \n",
        "predictions , true_labels = [], []\n",
        "\n",
        "# Measure elapsed time.\n",
        "t0 = time.time()\n",
        "\n",
        "# Predict \n",
        "for (step, batch) in enumerate(test_dataloader):\n",
        "    \n",
        "    # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "  \n",
        "    # Progress update every 50 batches.\n",
        "    if step % 50 == 0 and not step == 0:\n",
        "        # Calculate elapsed time in minutes.\n",
        "        elapsed = format_time(time.time() - t0)\n",
        "        \n",
        "        # Report progress.\n",
        "        print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(test_dataloader), elapsed))\n",
        "\n",
        "\n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "  \n",
        "    # Telling the model not to compute or store the compute graph, saving memory\n",
        "    # and speeding up prediction\n",
        "    with torch.no_grad():\n",
        "        # Forward pass, calculate logit predictions\n",
        "        outputs = model(b_input_ids, token_type_ids=None, \n",
        "                        attention_mask=b_input_mask)\n",
        "\n",
        "    logits = outputs[0]\n",
        "\n",
        "    # Move logits and labels to CPU\n",
        "    logits = logits.detach().cpu().numpy()\n",
        "    label_ids = b_labels.to('cpu').numpy()\n",
        "  \n",
        "    # Store predictions and true labels\n",
        "    predictions.append(logits)\n",
        "    true_labels.append(label_ids)\n",
        "\n",
        "print('    DONE.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicting labels for 1,250 test comments...\n",
            "  Batch    50  of     79.    Elapsed: 0:00:08.\n",
            "    DONE.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JrBswmqmyH5m"
      },
      "source": [
        "To turn our model outputs into actual predictions, we need to:\n",
        "\n",
        "1. Re-combine the predictions from across all of the batches.\n",
        "2. Pick a class label for each prediction by choosing the class with the highest output value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WlBi2hc-IuVu",
        "outputId": "7d0ecb64-4d98-423b-89da-f5137c427b07",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Combine the results across the batches.\n",
        "predictions = np.concatenate(predictions, axis=0)\n",
        "true_labels = np.concatenate(true_labels, axis=0)\n",
        "\n",
        "# Take the highest scoring output as the predicted label.\n",
        "predicted_labels = np.argmax(predictions, axis=1)\n",
        "\n",
        "print('`predictions` has shape', predictions.shape)\n",
        "print('`predicted_labels` has shape', predicted_labels.shape)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "`predictions` has shape (1250, 6)\n",
            "`predicted_labels` has shape (1250,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMVyXXHlr2pt"
      },
      "source": [
        "Let's peek at the model's outputs for the first 10 test samples, along with the correct labels for each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_mjLBPw_LIE0",
        "outputId": "deda5e6b-4915-4e4f-cc63-f436b6839bc1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Reduce printing precision for legibility.\n",
        "np.set_printoptions(precision=2)\n",
        "\n",
        "print(\"Predicted:\", str(predicted_labels[0:10]))\n",
        "print(\"  Correct:\", str(true_labels[0:10]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicted: [4 3 5 3 2 0 2 5 0 0]\n",
            "  Correct: [0 4 4 3 5 0 0 2 0 2]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T3_xUMQEy2nC"
      },
      "source": [
        "The metric for the Liar Liar dataset used by the scikit-learn example is the F1 score. Let's see how BERT performed!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TAm_DiwRD6e5",
        "outputId": "ba57d8b6-b73f-4e16-fb60-023b3346c0a6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn import metrics\n",
        "\n",
        "# Use the F1 metric to score our classifier's performance on the test set.\n",
        "score = metrics.f1_score(true_labels, predicted_labels, average='macro')\n",
        "\n",
        "# Print the F1 score!\n",
        "print('F1 score: {:.3}'.format(score))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.284\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WrMJKgif08iu"
      },
      "source": [
        "At this point, we have the following F1 scores:\n",
        "\n",
        "Liar-Liar F1 Scores\n",
        "* Naive Bayes: `0.221`\n",
        "* BERT:  `0.284`\n",
        "\n",
        "We've outperformed the baseline! "
      ]
    }
  ]
}