notebooks/Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows you how to use the pre-processing functionality of the `ktext` library to prepare data for Keras.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preview Data\n",
    "\n",
    "Github Issues pulled from [Github Archive](https://www.githubarchive.org/) for illustration.  This data is not provided, but just used for illustration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>issue_url</th>\n",
       "      <th>body</th>\n",
       "      <th>issue_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25056</th>\n",
       "      <td>https://github.com/CitizensDev/Citizens2/issue...</td>\n",
       "      <td>update players based on navigating npc's: upda...</td>\n",
       "      <td>fix npe, improve navigating npc skins, fix tab...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82182</th>\n",
       "      <td>https://github.com/docker-library/tomcat/issue...</td>\n",
       "      <td>docker image from java:openjdk-8u92-jre-alpine...</td>\n",
       "      <td>add use alpine image dockerfile issue 33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14214</th>\n",
       "      <td>https://github.com/UnionOfRAD/lithium/issues/317</td>\n",
       "      <td>it would be a nice addition to the dispatcher:...</td>\n",
       "      <td>allow closure usage in dispatcher::config</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               issue_url  \\\n",
       "25056  https://github.com/CitizensDev/Citizens2/issue...   \n",
       "82182  https://github.com/docker-library/tomcat/issue...   \n",
       "14214   https://github.com/UnionOfRAD/lithium/issues/317   \n",
       "\n",
       "                                                    body  \\\n",
       "25056  update players based on navigating npc's: upda...   \n",
       "82182  docker image from java:openjdk-8u92-jre-alpine...   \n",
       "14214  it would be a nice addition to the dispatcher:...   \n",
       "\n",
       "                                             issue_title  \n",
       "25056  fix npe, improve navigating npc skins, fix tab...  \n",
       "82182           add use alpine image dockerfile issue 33  \n",
       "14214          allow closure usage in dispatcher::config  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_parquet('demo_df.parquet')\n",
    "train, test = train_test_split(df, test_size=0.2)\n",
    "train[['issue_url', 'body', 'issue_title']].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Background: The data shown above are [Github Issues](https://guides.github.com/features/issues/) - specifically the body and issue title.  Suppose you want to train a model to read the body of an issue , and predict what the issue title will be.  Therefore, we need to pre-processes the **body** and **issue_title** fields from this dataframe.   \n",
    "\n",
    "`ktext` operates on lists of strings (where each element in the list is a document).  Therefore, we can extract these text fields from the dataframe as lists. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "body = train.body.tolist()\n",
    "issue_title = train.issue_title.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the raw data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL:  https://github.com/CitizensDev/Citizens2/issues/514\n",
      "Body:\n",
      " update players based on navigating npc's: update player when npc navigates into players field of view. moved skin update tracker code to own class skinupdatetracker fix use incorrect setting for tab list\n",
      "\n",
      "Title:\n",
      " fix npe, improve navigating npc skins, fix tablist setting\n"
     ]
    }
   ],
   "source": [
    "print('URL: ', train.issue_url.iloc[0])\n",
    "print('Body:\\n', body[0])\n",
    "print('Title:\\n', issue_title[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process Data For Deep Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "from ktext.preprocess import processor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initilize **processor** object\n",
    "It is important to read the docstring so you can see all the options. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function __init__ in module ktext.preprocess:\n",
      "\n",
      "__init__(self, heuristic_pct_padding:float=0.9, append_indicators:bool=False, keep_n:int=150000, padding:str='pre', padding_maxlen:Union[int, NoneType]=None, truncating:str='post')\n",
      "    Parameters:\n",
      "    ----------\n",
      "    heuristic_pct_padding: float\n",
      "        This parameter is only used if `padding_maxlen` = None.  A histogram\n",
      "        of documents is calculated, and the maxlen is set heuristic_pct_padding.\n",
      "    append_indicators: bool\n",
      "        If True, will append the tokens '_start_' and '_end_' to the beginning\n",
      "        and end of your tokenized documents.  This can be useful when training\n",
      "        seq2seq models.\n",
      "    keep_n: int = 150000\n",
      "        This is the maximum size of your vocabulary (unique number of words\n",
      "        allowed).  Consider limiting this to a reasonable size based upon\n",
      "        your corpus.\n",
      "    padding : str\n",
      "        'pre' or 'post', pad either before or after each sequence.\n",
      "    padding_maxlen : int or None\n",
      "        Maximum sequence length, longer sequences are truncated and shorter\n",
      "        sequences are padded with zeros at the end.  Note if this is specified,\n",
      "        the `heuristic_pct_padding` is ignored.\n",
      "    truncating : str\n",
      "        'pre' or 'post', remove values from sequences larger than padding_maxlen\n",
      "        either in the beginning or in the end of the sequence.\n",
      "    \n",
      "    See https://keras.io/preprocessing/sequence/\n",
      "    \n",
      "    Attributes:\n",
      "    -----------\n",
      "    vocabulary : gensim.corpora.dictionary.Dictionary\n",
      "        This is a gensim object that is built after parsing all the tokens\n",
      "        in your corpus.\n",
      "    n_tokens : int\n",
      "        The total number of tokens in the corpus.  Will be less than or\n",
      "        equal to keep_n\n",
      "    id2token : dict\n",
      "        dict with { int : str} ex: {'the': 2, 'cat': 3}\n",
      "        this is used for converting tokens to integers.\n",
      "    token2id : dict\n",
      "        dict with {str: int} ex: {2: 'the', 3: 'cat'}\n",
      "        this is used for decoding predictions back to tokens\n",
      "    document_length_stats : pandas.DataFrame\n",
      "        histogram of document lengths.  Can be used to decide padding_maxlen.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(processor.__init__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets initiliaze our processor and let it truncate and pad our documents such such that all documents are equal to the 70th percentile of document lengths in the data set.  Furthermore, lets only keep the top 5,000 tokens in our vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "issue_body_proc = processor(heuristic_pct_padding=.7, keep_n=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `fit_transform` method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**fit_transform** will perform cleaning, tokenization, build a vocabulary and truncate and pad documents.  Look at the docstring to get a more detailed explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:Setting maximum document length to 90 based upon heuristic of 0.7 percentile.\n",
      " See full histogram by insepecting the `document_length_stats` attribute.\n",
      "WARNING:root:(1/3) done. 66 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/3) done. 29 sec\n",
      "WARNING:root:....consolidating corpus\n",
      "WARNING:root:(3/3) done. 1 sec\n",
      "WARNING:root:Finished parsing 80,000 documents.\n",
      "WARNING:root:...fit is finished, beginning transform\n",
      "WARNING:root:done. 27 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 22s, sys: 3.14 s, total: 1min 25s\n",
      "Wall time: 2min 4s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# prepare text on 8 core node\n",
    "train_result = issue_body_proc.fit_transform(body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of result (80000, 90)\n",
      "\n",
      "original string:\n",
      " update players based on navigating npc's: update player when npc navigates into players field of view. moved skin update tracker code to own class skinupdatetracker fix use incorrect setting for tab list\n",
      "\n",
      "after pre-processing:\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  2  3  4  5  6  1  7  8  2  9 10  1  1 11  3 12 13 14 15 16\n",
      "  1  2 17 18 19 20 21  1 22 23 24 25 26 27 28]\n"
     ]
    }
   ],
   "source": [
    "print('shape of result', train_result.shape)\n",
    "print('\\noriginal string:\\n', body[0])\n",
    "print('after pre-processing:\\n', train_result[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can see that the body of this issue has been converted to an array of integers and has been padded so that it is of length 90.  This is because we initialized the `processor` to calculate max_len by using the heuristic of 70th percentile of document length.  We can see the histogram of document lengths by inspecting the **`document_length_stats`** attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bin</th>\n",
       "      <th>doc_count</th>\n",
       "      <th>cumsum_pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>10</td>\n",
       "      <td>2205</td>\n",
       "      <td>0.027563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>20</td>\n",
       "      <td>8240</td>\n",
       "      <td>0.130562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30</td>\n",
       "      <td>6623</td>\n",
       "      <td>0.213350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40</td>\n",
       "      <td>7305</td>\n",
       "      <td>0.304663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>50</td>\n",
       "      <td>5845</td>\n",
       "      <td>0.377725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>7338</td>\n",
       "      <td>0.469450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>70</td>\n",
       "      <td>6437</td>\n",
       "      <td>0.549913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>80</td>\n",
       "      <td>8687</td>\n",
       "      <td>0.658500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>90</td>\n",
       "      <td>8802</td>\n",
       "      <td>0.768525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>10775</td>\n",
       "      <td>0.903212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>110</td>\n",
       "      <td>4660</td>\n",
       "      <td>0.961462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>120</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.986625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>130</td>\n",
       "      <td>482</td>\n",
       "      <td>0.992650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>140</td>\n",
       "      <td>259</td>\n",
       "      <td>0.995888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>150</td>\n",
       "      <td>114</td>\n",
       "      <td>0.997313</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    bin  doc_count  cumsum_pct\n",
       "13   10       2205    0.027563\n",
       "10   20       8240    0.130562\n",
       "2    30       6623    0.213350\n",
       "0    40       7305    0.304663\n",
       "7    50       5845    0.377725\n",
       "3    60       7338    0.469450\n",
       "4    70       6437    0.549913\n",
       "8    80       8687    0.658500\n",
       "6    90       8802    0.768525\n",
       "1   100      10775    0.903212\n",
       "12  110       4660    0.961462\n",
       "9   120       2013    0.986625\n",
       "11  130        482    0.992650\n",
       "14  140        259    0.995888\n",
       "16  150        114    0.997313"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = issue_body_proc.document_length_stats\n",
    "stats.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not like this behavior, you can your own desired maximum length manually by specifying the **`padding_max_len`** parameter (which will skip the process of building a histogram and will run faster!)  This will override the **`heuristic_pct_padding`** parameter if it is specified.  Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:(1/3) done. 40 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/3) done. 28 sec\n",
      "WARNING:root:....consolidating corpus\n",
      "WARNING:root:(3/3) done. 1 sec\n",
      "WARNING:root:Finished parsing 80,000 documents.\n",
      "WARNING:root:...fit is finished, beginning transform\n",
      "WARNING:root:done. 27 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 56.8 s, sys: 2.39 s, total: 59.2 s\n",
      "Wall time: 1min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "issue_body_proc2 = processor(keep_n=5000, padding_maxlen=85)\n",
    "train_result2 = issue_body_proc2.fit_transform(body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of result (80000, 85)\n",
      "\n",
      "original string:\n",
      " update players based on navigating npc's: update player when npc navigates into players field of view. moved skin update tracker code to own class skinupdatetracker fix use incorrect setting for tab list\n",
      "\n",
      "after pre-processing:\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  2  3  4  5  6  1  7  8  2  9 10  1  1 11  3 12 13 14 15 16  1  2 17 18 19\n",
      " 20 21  1 22 23 24 25 26 27 28]\n"
     ]
    }
   ],
   "source": [
    "print('shape of result', train_result2.shape)\n",
    "print('\\noriginal string:\\n', body[0])\n",
    "print('after pre-processing:\\n', train_result2[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `fit` method\n",
    "\n",
    "The same as the **`fit_transform`** method except no data is returned.  See docstring for further details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **`index2token`** attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **`index2token`** attribute contains all of the index to word mappings, which we can use to reverse the indexes back to tokens.  Remember that `0` is just padding and `1` is reserved for rare (below the `keep_n` threshold) or unknown words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"update players based on navigating 's : update player when into players field of view . moved update tracker code to own class fix use incorrect setting for tab list\""
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([issue_body_proc2.id2token[idx] for idx in train_result2[0] if idx >1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `token_count_pandas` method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use this method to see the top tokens in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>token</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>62447</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>58599</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>53646</td>\n",
       "      <td>,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>52374</td>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>47715</td>\n",
       "      <td>*</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>40674</td>\n",
       "      <td>:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>40232</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>38626</td>\n",
       "      <td>is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>36808</td>\n",
       "      <td>in</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>35813</td>\n",
       "      <td>and</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     count token\n",
       "13   62447     .\n",
       "42   58599   the\n",
       "71   53646     ,\n",
       "17   52374    to\n",
       "35   47715     *\n",
       "6    40674     :\n",
       "51   40232     a\n",
       "95   38626    is\n",
       "67   36808    in\n",
       "127  35813   and"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_count_df = issue_body_proc2.token_count_pandas()\n",
    "token_count_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `transform` method\n",
    "\n",
    "This method performs a transformation operation on new raw data, but doesn't use process based threading for parallelization, which is suitable if you do not have that much data to transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 32.8 s, sys: 28 ms, total: 32.8 s\n",
      "Wall time: 32.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "body_test = test.body.tolist()\n",
    "body_test_transformed = issue_body_proc2.transform(body_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of result (20000, 85)\n"
     ]
    }
   ],
   "source": [
    "print('shape of result', body_test_transformed.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `transform_parallel` method\n",
    "\n",
    "Same as the **`transform`** method but uses process-based threading for parallelization.  In this case, we can see an appreciable speedup because we are transforming 20k documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.46 s, sys: 340 ms, total: 1.8 s\n",
      "Wall time: 8.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "body_test_transformed = issue_body_proc2.transform_parallel(body_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,  136,  137, 1124,  840,  541,  180,   86,    1,  373,\n",
       "         19,  207,  123, 2611,   10,  288,  223,    1,   15,   65,  136,\n",
       "        774,    1,   44, 1678,  596,  476,   15,  123,  386, 1127,    1,\n",
       "         92,    1,   92, 1563, 1950,  241,    1,    8,  117,   73, 1563,\n",
       "       1376, 2616, 4449, 1563,  131,    1,   19,   53, 3782,   13, 1812,\n",
       "        596,    1,    8,   92,    1,    8,    1,   73], dtype=int32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "body_test_transformed[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `vocabulary` attribute\n",
    "\n",
    "This is a gensim object that is updated when you call **`fit`** or **`fit_transform`**\n",
    "\n",
    "see https://radimrehurek.com/gensim/corpora/dictionary.html for more details on this object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gensim.corpora.dictionary.Dictionary at 0x7f2c75a36550>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "issue_body_proc.vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Other methods under construction but not yet completed:\n",
    "\n",
    "1. set_tokenizer\n",
    "2. set_cleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Experimental: incremental parsing with `fit` or `fit_transform` \n",
    "\n",
    "If you call **`fit`** or **`fit_transform`** more than once using the same **`processor`** object, it will simply **append to the existing vocabulary** this can be useful if you only want to add to your vocabulary incrementally for some reason.  Care must be taken here as `keep_n` throws out tokens completely so set `keep_n` sufficiently large if you want to allow for new tokens to be added.  Also with incremental training you will likely want to use **`padding_max_len`** instead of the **`heuristic_pct_padding:float`** parameter.  Furthermore, everytime you call fit or fit_transform word indices are changed.  Use incremental training with extreme caution.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:(1/3) done. 39 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/3) done. 33 sec\n",
      "WARNING:root:....consolidating corpus\n",
      "WARNING:root:(3/3) done. 1 sec\n",
      "WARNING:root:Finished parsing 80,000 documents.\n"
     ]
    }
   ],
   "source": [
    "new_proc = processor(keep_n=10000, padding_maxlen=70)\n",
    "new_proc.fit(body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents:  80000\n"
     ]
    }
   ],
   "source": [
    "print('Number of documents: ', new_proc.vocabulary.num_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add 15k more documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:(1/3) done. 9 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/3) done. 5 sec\n",
      "WARNING:root:....consolidating corpus\n",
      "WARNING:root:(3/3) done. 0 sec\n",
      "WARNING:root:Finished parsing 95,000 documents.\n"
     ]
    }
   ],
   "source": [
    "new_data = body[:15000]\n",
    "new_proc.fit(new_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents:  95000\n"
     ]
    }
   ],
   "source": [
    "print('Number of documents: ', new_proc.vocabulary.num_docs)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}