3_model_building.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Model Building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook explores building the **customer churn model** and an **interpretability model** to explain each prediction.\n",
    "In addition to making a prediction of whether a customer will churn, we will also be able to answer the question, \"why are they expected to churn?\"\n",
    "\n",
    "The following work will look fairly standard to anyone having trained machine learning models using python Jupyter notebooks.\n",
    "The CML platform provides a **fully capable Jupyter notebook environment** that data scientists know and love.\n",
    "\n",
    "If you haven't yet, run through the initialization steps in the README file and Part 1. \n",
    "In Part 1, the data is imported into the `default.telco_churn` table in Hive. All data accesses fetch from Hive.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "We again start by creating a `SparkSession` to fetch the data using Spark SQL, only this time we convert to a pandas `DataFrame` since we saw earlier that there are only 7k records in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- customerID: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- SeniorCitizen: string (nullable = true)\n",
      " |-- Partner: string (nullable = true)\n",
      " |-- Dependents: string (nullable = true)\n",
      " |-- tenure: double (nullable = true)\n",
      " |-- PhoneService: string (nullable = true)\n",
      " |-- MultipleLines: string (nullable = true)\n",
      " |-- InternetService: string (nullable = true)\n",
      " |-- OnlineSecurity: string (nullable = true)\n",
      " |-- OnlineBackup: string (nullable = true)\n",
      " |-- DeviceProtection: string (nullable = true)\n",
      " |-- TechSupport: string (nullable = true)\n",
      " |-- StreamingTV: string (nullable = true)\n",
      " |-- StreamingMovies: string (nullable = true)\n",
      " |-- Contract: string (nullable = true)\n",
      " |-- PaperlessBilling: string (nullable = true)\n",
      " |-- PaymentMethod: string (nullable = true)\n",
      " |-- MonthlyCharges: double (nullable = true)\n",
      " |-- TotalCharges: double (nullable = true)\n",
      " |-- Churn: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "\n",
    "spark = SparkSession\\\n",
    "    .builder\\\n",
    "    .appName(\"PythonSQL\")\\\n",
    "    .master(\"local[*]\")\\\n",
    "    .getOrCreate()\n",
    "\n",
    "spark_df = spark.sql(\"SELECT * FROM default.telco_churn\")\n",
    "spark_df.printSchema()\n",
    "df = spark_df.toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If you don't have the Hive table, you can read the csv from the CML filesystem using pandas directly:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "data_dir = '/home/cdsw'\n",
    "df = pd.read_csv(os.path.join(data_dir, 'raw', 'WA_Fn-UseC_-Telco-Customer-Churn-.csv'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic feature engineering\n",
    "\n",
    "\n",
    "Next we munge the data into appropriate types for later steps. \n",
    "In particular, we want to convert all the binary and string columns into pandas `Categorical` types.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess, glob, sys\n",
    "import dill  # a better pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "data_dir = '/home/cdsw'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "idcol = 'customerID'  # ID column\n",
    "labelcol = 'Churn'  # label column\n",
    "cols = (('gender', True),  # (feature column, Categorical?)\n",
    "        ('SeniorCitizen', True),\n",
    "        ('Partner', True),\n",
    "        ('Dependents', True),\n",
    "        ('tenure', False),\n",
    "        ('PhoneService', True),\n",
    "        ('MultipleLines', True),\n",
    "        ('InternetService', True),\n",
    "        ('OnlineSecurity', True),\n",
    "        ('OnlineBackup', True),\n",
    "        ('DeviceProtection', True),\n",
    "        ('TechSupport', True),\n",
    "        ('StreamingTV', True),\n",
    "        ('StreamingMovies', True),\n",
    "        ('Contract', True),\n",
    "        ('PaperlessBilling', True),\n",
    "        ('PaymentMethod', True),\n",
    "        ('MonthlyCharges', False),\n",
    "        ('TotalCharges', False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>SeniorCitizen</th>\n",
       "      <th>Partner</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>tenure</th>\n",
       "      <th>PhoneService</th>\n",
       "      <th>MultipleLines</th>\n",
       "      <th>InternetService</th>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <th>OnlineBackup</th>\n",
       "      <th>DeviceProtection</th>\n",
       "      <th>TechSupport</th>\n",
       "      <th>StreamingTV</th>\n",
       "      <th>StreamingMovies</th>\n",
       "      <th>Contract</th>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <th>PaymentMethod</th>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <th>TotalCharges</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>1.0</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>29.85</td>\n",
       "      <td>29.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>34.0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>56.95</td>\n",
       "      <td>1889.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>53.85</td>\n",
       "      <td>108.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>45.0</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>42.30</td>\n",
       "      <td>1840.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>70.70</td>\n",
       "      <td>151.65</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    gender SeniorCitizen Partner Dependents  tenure PhoneService  \\\n",
       "id                                                                 \n",
       "0   Female             0     Yes         No     1.0           No   \n",
       "1     Male             0      No         No    34.0          Yes   \n",
       "2     Male             0      No         No     2.0          Yes   \n",
       "3     Male             0      No         No    45.0           No   \n",
       "4   Female             0      No         No     2.0          Yes   \n",
       "\n",
       "       MultipleLines InternetService OnlineSecurity OnlineBackup  \\\n",
       "id                                                                 \n",
       "0   No phone service             DSL             No          Yes   \n",
       "1                 No             DSL            Yes           No   \n",
       "2                 No             DSL            Yes          Yes   \n",
       "3   No phone service             DSL            Yes           No   \n",
       "4                 No     Fiber optic             No           No   \n",
       "\n",
       "   DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \\\n",
       "id                                                                            \n",
       "0                No          No          No              No  Month-to-month   \n",
       "1               Yes          No          No              No        One year   \n",
       "2                No          No          No              No  Month-to-month   \n",
       "3               Yes         Yes          No              No        One year   \n",
       "4                No          No          No              No  Month-to-month   \n",
       "\n",
       "   PaperlessBilling              PaymentMethod  MonthlyCharges  TotalCharges  \n",
       "id                                                                            \n",
       "0               Yes           Electronic check           29.85         29.85  \n",
       "1                No               Mailed check           56.95       1889.50  \n",
       "2               Yes               Mailed check           53.85        108.15  \n",
       "3                No  Bank transfer (automatic)           42.30       1840.75  \n",
       "4               Yes           Electronic check           70.70        151.65  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.replace(r'^\\s$', np.nan, regex=True).dropna().reset_index()  # drop blank rows\n",
    "df.index.name = 'id'  # name the index\n",
    "data, labels = df.drop(labelcol, axis=1), df[labelcol]  # separate out the labels\n",
    "data = data[[c for c, _ in cols]]  # only use the columns named in `cols`\n",
    "data = data.replace({'SeniorCitizen': {1: 'Yes', 0: 'No'}})  # Change 1/0 to Yes/No to match the other binary features\n",
    "\n",
    "# convert the categorical columns to pd.Categorical form\n",
    "for col, iscat in cols:\n",
    "    if iscat:\n",
    "        data[col] = pd.Categorical(data[col])\n",
    "labels = (labels == 'Yes')  # convert labels from str to bool\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine learning model\n",
    "\n",
    "This step follows a fairly standard ML workflow, which is to create a pipeline to:\n",
    "\n",
    "* Encode the categorical features as numeric\n",
    "* Normalize the numeric features\n",
    "* Train a classification model using these processed features\n",
    "\n",
    "We use *one-hot encoding*, *standardization*, and *logistic regression with cross-validation* for the three steps.\n",
    "Then we can evaluate the model's performance.\n",
    "\n",
    "Note: `CategoricalEncoder` and, later, `ExplainedModel` are helper classes pulled and edited from the original CFFL [interpretability report code](https://ff06-2020.fastforwardlabs.com/).\n",
    "You can inspect `churnexplainer.py` to see what they do under the hood.\n",
    "CML lets you continue to write modular code to keep things segregated and clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegressionCV, LogisticRegression\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "from churnexplainer import CategoricalEncoder  # convert Categorical columns into numeric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "ce = CategoricalEncoder()\n",
    "X = ce.fit_transform(data)  # Categorical columns now have values 0 to num_categories-1\n",
    "y = labels.values\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
    "\n",
    "cat_cols = list(ce.cat_columns_ix_.values())  # indices of the categorical columns (now numeric)\n",
    "ct = ColumnTransformer(\n",
    "    [('ohe', OneHotEncoder(), cat_cols)],\n",
    "    remainder='passthrough'\n",
    ")\n",
    "clf = LogisticRegressionCV(cv=5,solver='lbfgs', max_iter=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cdsw/.local/lib/python3.6/site-packages/sklearn/preprocessing/_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.\n",
      "If you want the future behaviour and silence this warning, you can specify \"categories='auto'\".\n",
      "In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.\n",
      "  warnings.warn(msg, FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train 0.8077360637087599\n",
      "test 0.7912400455062572\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       False       0.84      0.89      0.86      1300\n",
      "        True       0.62      0.52      0.56       458\n",
      "\n",
      "    accuracy                           0.79      1758\n",
      "   macro avg       0.73      0.70      0.71      1758\n",
      "weighted avg       0.78      0.79      0.79      1758\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pipe = Pipeline([('ct', ct),  # 1. Encode the categorical features as numeric\n",
    "                 ('scaler', StandardScaler()),  # 2. Normalize the numeric features\n",
    "                 ('clf', clf)])  # 3. Train a classification model using these processed features\n",
    "pipe.fit(X_train, y_train)\n",
    "train_score = pipe.score(X_train, y_train)\n",
    "test_score = pipe.score(X_test, y_test)\n",
    "print(\"train\",train_score)\n",
    "print(\"test\", test_score)    \n",
    "print(classification_report(y_test, pipe.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare with Random Forest\n",
    "Just for a comparison, lets compare this model to a Random Forest model.\n",
    "This is simpler since Random Forests do not need the categorical features encoded with a `OneHotEncoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train 0.9981039059537353\n",
      "test 0.7895335608646189\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       False       0.83      0.90      0.86      1300\n",
      "        True       0.63      0.47      0.54       458\n",
      "\n",
      "    accuracy                           0.79      1758\n",
      "   macro avg       0.73      0.69      0.70      1758\n",
      "weighted avg       0.78      0.79      0.78      1758\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "clf_rf = RandomForestClassifier(n_estimators=100)\n",
    "pipe_rf = Pipeline([('scaler', StandardScaler()),\n",
    "                 ('clf', clf_rf)])\n",
    "pipe_rf.fit(X_train, y_train)\n",
    "train_score = pipe_rf.score(X_train, y_train)\n",
    "test_score = pipe_rf.score(X_test, y_test)\n",
    "print(\"train\",train_score)\n",
    "print(\"test\", test_score)\n",
    "print(classification_report(y_test, pipe_rf.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot ROC Curve\n",
    "\n",
    "We can also generate an ROC Curve to visualize the model's performance and calculate the AUROC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic: AUROC=0.834\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHpCAYAAAC4HUuQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAHvBJREFUeJzt3f+rZfdd7/HXJDOj/R4kjVB7i+ZS1EhscxRyTXMtEvpDJXiFaz8/tOK1FGJABvqT+Ru8v1iJFOy113DjF/Ke5oLSNmJIqG0I5gfP/BA7hVpLrTcFk2grNJPpTJK5P8w54549+8zsc/Znf1l7PR5QmLNm7VmrfHqmz/OZz/qsY5cuXQoAANDHTeu+AQAA2CYCGwAAOhLYAADQkcAGAICOBDYAAHQksAEAoCOBDQAAHQlsAADoSGADAEBHAhsAADoS2AAA0JHABgCAjo6v+wYO66mnnrq07nsAAGAc7rvvvmOH/YwZbAAA6GhwM9j7dnZ2Vnq93d3dtVyX1TLO42Cct58xHgfjPA7rGuf96x6FGWwAAOhIYAMAQEcCGwAAOhLYAADQkcAGAICOBDYAAHQksAEAoCOBDQAAHQlsAADoSGADAEBHAhsAADoS2AAA0JHABgCAjo73+oNaa7+Q5ERVffmA3/+ZJL+U5HyStyT5XFW90Ov6AACwCRYO7Nba/Unek+TLSd5/wDnHknygqv5g4tgDST6z6PUBAGCT9Fgi8oWq+nRV/f11zvlAkqemjn2jtfbuDtcHAICNsfAMdlVdmuO091TVM1PHnk/yviT/7yjX3d3dPcrHjuzcuXNruS6rZZzHwThvP2M8DsZ5c3zqzIU8//IbS73GZzOcce62BvsGZs2Uv5zknSu6PgDAIK0iXulrVYF9jaq61Fq7+aif39nZ6Xk7N7T/0/Gqr8tqGedxMM7bzxiPw6aN86nTZ/PMN7+37ttYm3tvvyUPf+SO7n/uusZ5kX8ZWVVgX7OMZO/BRz+OAQBbYZlxvax4ZTlWFdizQvqdSV5a0fUBgC22SbPHZx66Z923wJqtKrD/ubX23qr6h4lj70vy9RVdHwBYg5WE75PPLvfPP4R7b79l3bfABlhJYFfVM621B5NMBvZPV9WTq7g+ALB6q55VtoyCTbG0wG6t3Z0kVfXc3qGvtNZOJXk1yTuSPL6sawPAGGzSsojr2baH3+BGugZ2Vf3JxK+fm/q9ryb5as/rAcAYDSWsE7PKjNPatukDAOZ3UFQLWNg8AhsANpiwhuER2ACwgWaFtaiGYRDYALBBhDUMn8AGgCXo8SCisIZhEtgA0Emv3T2ENQybwAaATiztABKBDQCHMs8s9ZmH7lnR3QCbSGADQPou7wDGTWADMGpHCWtLP4DrEdgAbKyFZ5WffPZQpwtnoAeBDcDG6rFkYx7CGuhJYAOwkU6dPnvl14d9aHB3dzdJsrOz0/WeAOYhsAFYm3mWgHhoEBgagQ3ASniYEBgLgQ1AV4cNaRENbBuBDcCBeu0NvU9MA2MgsAG4oufLVoQ0MFYCG2BErIMGWD6BDbDlrIkGWC2BDbAFDhPRAhpguQQ2wBaYZy9pUQ2wGgIbYIsc9o2HAPQnsAEGpvfWeQD0JbABNthh11YDsH4CG2CDzBvU1lQDbC6BDbBits0D2G4CG2AFbKMHMB4CG6ADs9IA7BPYAIewyA4eohpgHAQ2wBzmCWsBDUAisAEOdFBUC2kArkdgA0wR1gAsQmADRFQD0I/ABkiuiWthDcBRCWxglD515kKef/LZa46feeieNdwNANtEYAOjcnkpyPmZv3fv7bes+G4A2EYCG9hKttUDYF0ENrCVrhfXd956Uz5518ns7IhrAPoT2MBWm7Wmend3dw13AsBY3LTuGwDo7dTps+u+BQBGzAw2MHjX28MaAFZNYAODdb0HGT3ACMC6CGxgkKbjWlADsCkENjA4k3EtrAHYNAIbGIRZy0HENQCbyC4iwMYT1wAMiRlsYONZDgLAkAhsYCPM82pzcQ3AEAhsYC3mCepJ9rQGYCgENrBy13sxjFlqAIZOYAMrZYs9ALadXUSAlRLXAGw7M9jA0s1aEiKuAdhWZrCBpZu1hzUAbCsz2MDC5t0R5MxD96zgbgBgvQQ2cGSH2WrPrDUAYyGwgUOxxR4AXJ/ABg5l1npqYQ0A/0FgA9d10Iy19dQAMJvABq5xo7XV1lMDwMEENnCVWXFtGQgAzE9gA6IaADoS2DBC8ywBEdcAcDQCG0bkemEtqgGgD4ENIzIZ14IaAJZDYMNInDp99sqvbbEHAMuzcGC31k4keSDJhSQ3Jfl2VT0x47xfTvLje+e9PckTVfW1Ra8P3Njk0hBb7AHAcvWYwW5JHqmqV5KktXZ/a+1dVfWdKye09pNJ3qiqT08c+2Rr7RtVdbHDPQAzTK+5tiwEAJavR2Cf2I/rPV9M8rEkj04c+/mq+tOpz30pyU8leb7DPQB7DnqQUVwDwGrctMiHW2snk3x/8lhVvZHktalTj7XWpmP+Tbm8XAToaNZ+1mceukdcA8CKLDqDfVuSl2Ycnw73zyf5RGvtM1V1qbX2llye1X74qBfe3d096keP5Ny5c2u5Lqu1TeP82Q/98N6vzm/Ff5+etmmcmc0Yj4NxHochjvNCM9i5HOivzzh+bPKLqvpekq8m+dPW2u8k+aMkjy14bWDKp874RyEAWLdFZ7BfS3LzjU5qrb0/yW1V9dG9r08m+e29Ge1Xrv/p2XZ2do7ysSPb/6lp1ddltYY8zqdOn83zL59PcnlZyM6OJSEHGfI4Mx9jPA7GeRzWNc6LzJgvGtgvJrl7jvN2qup/739RVRdaa59O8t+T/NmC9wCjNeuBRg8zAsB6LbREpKr297S+orV2U5KTU6de8+/WVfWDRa8PYyeuAWDz9Ajci621t018fX+Sp6fOebW19mOTB1prH07ydx2uD6M0/WZGO4UAwGbosQ/2Y0kebK2dT3IiyQtV9a3W2t1JUlXPVdXjrbWP77318fUkP5Tk697kCIc36+UxAMDmWDiw997EeM12e1X13NTXf7zotWDsvJkRADZfjxlsYEm8lREAhkdgwwY5KKgniWsA2GwCG9bsRlEtqAFgWAQ2rJF9rAFg+whsWJPJuBbVALA9vOgF1kRcA8B2EtiwBpMviRHXALBdLBGBFbjednsAwHYxgw0rYC9rABgPM9iwJLNmrc88dM+a7gYAWBUz2LAEB22/BwBsPzPYsAR2CAGA8TKDDUskrgFgfAQ2dDa5BR8AMD4CGzqbXB4CAIyPwIYlsTwEAMZJYAMAQEcCGzqy/hoAENjQkfXXAIDAhiWw/hoAxsuLZmABs97YCACMmxlsWMCsuLY8BADGzQw2HMH0zPWZh+5Z490AAJvEDDYcwWRcm7EGACaZwYYFmLkGAKYJbJiTBxoBgHlYIgJzmo5rS0MAgFnMYMMhWRYCAFyPGWyYg1egAwDzEthwA5Nrry0LAQBuRGDDDUzGtVegAwA3IrBhTuIaAJiHhxxhBlvyAQBHJbBhz+WoPr/31fmrfs/aawBgXgIb9sza59qyEADgsAQ25Opt+D77oR/Ozs7OGu8GABgygc1ozVpnfeetnvsFABYjsBmtWUtCPv6fzx9wNgDAfAQ2ozf56vPd3d013gkAsA0ENqNjCz4AYJkENqMxK6xtvwcA9CawGYXpuLYFHwCwLAKbUdiPa2ENACybPcnYepN7XItrAGDZzGCztWYtCwEAWDYz2Gwla64BgHUxg81WsuYaAFgXM9hsHWuuAYB1EthsncnZawCAVRPYbBWz1wDAuglstorZawBg3QQ2W8PsNQCwCQQ2W8PsNQCwCQQ2W8HsNQCwKeyDzaB5WyMAsGnMYDNo3tYIAGwaM9hshTMP3bPuWwAASGIGmwGbXHcNALApBDaDNLn22rprAGCTCGwGZzqurbsGADaJwGZQxDUAsOkENoMhrgGAIRDYDIK4BgCGQmAzCOIaABgK+2Cz0abf1CiuAYBNZwabjeU16ADAEJnBZmNZFgIADNHCgd1aO5HkgSQXcnlG/NtV9cQB596R5ENJziX566r6p0Wvz3aafEujuAYAhqTHDHZL8khVvZIkrbX7W2vvqqrvXHVSaz+X5I6q+v0O12TLeUsjADBUPdZgn9iP6z1fTHLfjPM+WFWPdrgeI2L2GgAYmoUCu7V2Msn3J49V1RtJXps6784kf7vItRiPyeUhAABDs+gSkduSvDTj+HS47yT5i9baR5Oc3Lvu41X13aNeeHd396gfPZJz586t5bpj86kzF/L8y28kSe689SbjzFIY5+1njMfBOI/DEMd50SUix5O8PuP4samv35rkV5J8rqoeSfLHST6x94AkXDEZ15+86+Sa7wYA4PAWncF+LcnNc5x3LMmfV9XFJKmq11trn03y4SR/eZQL7+zsHOVjR7b/U9Oqrzs6Tz6bJPk/n/gva7m8cR4H47z9jPE4GOdxWNc4LzJjvugM9otJbp3jvO/ux/W+veUh71jw+mwRa68BgG2wUGBX1YUkb5881lq7KZfXWU96tbX25qnz3pTZy0sYKVvzAQDboMc2fRdba2+b+Pr+JE9PnfNXST42dezX944zYqdOn81dv/ts7vrdZ68cszUfADBkPV4081iSB1tr55OcSPJCVX2rtXZ3klTVc1V1rrX2pdbabyf5QZI3JTlTVf/W4foMzKnTZ6/MVk8zew0ADN3Cgb23tvrhGcefm/r6H5L8w6LXY9hmxfW9t99i1hoA2Bo9ZrBhbpPrrEU1ALCNeqzBhkMT1wDAtjKDzUpcb901AMA2MYPNSkzGtQcZAYBtZgabpZt8gcyZh+5Z450AACyfGWyWzgtkAIAxEdisjAcbAYAxENgAANCRwAYAgI485MjS2JoPABgjgU13B70OHQBgDAQ2XU3HtVeiAwBjI7DpanJLPmENAIyRhxxZCnENAIyVwAYAgI4ENt1MvhIdAGCsBDbdeCU6AIDAppPJ2WvrrwGAMRPYdGH2GgDgMoFNV2avAYCxE9gAANCRwAYAgI4ENguzPR8AwH/wqnSO7NTps1cebkw84AgAkJjBZgHTce0BRwAAM9gc0eSykDMP3bPGOwEA2CwCm0OxLAQA4PosEWFus+LashAAgKuZwWYuk3EtrAEADiawmWl6tnqfuAYAuD5LRJhJXAMAHI0ZbK5hhxAAgKMzg81VptdaAwBwOAKbKzzICACwOIHNFeIaAGBxAptriGsAgKMT2CS5+sFGAACOTmCTJB5sBADoRGBz1ey15SEAAIsR2CNnWz4AgL4E9sjZOQQAoC+BTRJLQwAAehHYI2bnEACA/gT2iFl7DQDQn8DG8hAAgI4ENgAAdCSwAQCgI4E9Uh5wBABYDoE9Ql4uAwCwPMfXfQOszmRYJ14uAwCwDGawR0JcAwCshhnsEZheEiKsAQCWxwz2CIhrAIDVEdhbbnK3EHENALB8AnvL2S0EAGC1BPYWM3sNALB6AnuLmb0GAFg9gT0CZq8BAFZHYAMAQEcCGwAAOhLYW2ryAUcAAFZHYG+h6Tc3AgCwOgJ7C3lzIwDA+gjsLSauAQBWT2BvGWuvAQDWS2BvGWuvAQDWS2BvKctDAADW4/iif0Br7USSB5JcyOVg/3ZVPXGDz/xqkrdW1Z8sen3+g+UhAADr12MGuyV5pKr+V1X9YZKbW2vvOvDk1n40HcKea1keAgCwfj0C+0RVvTLx9ReT3Hed81uSxztclwNYHgIAsD4LBXZr7WSS708eq6o3krx2wPkfSvJ0VV1a5LoAALCpFl2qcVuSl2YcvybcW2tvSfITVfXkgtdMkuzu7vb4Y+Z27ty5tVz3KIZwj5tqSOPM0Rnn7WeMx8E4j8MQx3nRJSLHk7w+4/ixGcc+nuTRBa/HAT515sK6bwEAgCw+g/1akptvdFJr7eeSfK2qXl3welfs7Oz0+qPmsv9T06qvO49Tp8/m+ZfPJ7n8gOPOjjXYR7XJ40w/xnn7GeNxMM7jsK5xXmTGfNHAfjHJ3XOc94tJvtta+82JY7/QWjue5M+qyvTrEZ06ffaq3UM84AgAsF4LBXZVXWitvX3yWGvtpiQnp877venPttZesw/24sQ1AMBm6bFN38XW2tsmvr4/ydMd/lxuYPLFMuIaAGAz9Hjhy2NJHmytnU9yIskLVfWt1trdSVJVz3W4BjN4sQwAwOZZOLCr6mKSh2ccv25YWx7Sj9lrAIDN0WOJCAAAsEdgAwBARwJ7oCYfcAQAYHMI7IHygCMAwGYS2ANkez4AgM0lsAfI7DUAwOYS2ANm9hoAYPMIbAAA6EhgAwBARwJ7YGzPBwCw2QT2gJw6fdYDjgAAG05gD8hkXHvAEQBgMwnsgbD3NQDAMAjsAbA0BABgOAT2hpuOa7PXAACbTWBvOHENADAsAnsgxDUAwDAI7A1mz2sAgOER2BvKg40AAMMksDeQBxsBAIZLYG8YcQ0AMGwCe8OIawCAYRPYG0pcAwAMk8AGAICOjq/7Brhscu01AADDJbDX6KCoti0fAMBwCew1uF5YW3sNADBsAnsNJuNaVAMAbBeBvUZnHrpn3bcAAEBnAnuFPMgIALD9bNO3QtNLQwAA2D5msNfA0hAAgO1lBntFTp0+u+5bAABgBQT2iuwvD7E0BABguwnsFbMlHwDAdhPYAADQkcAGAICOBDYAAHQksAEAoCOBDQAAHQlsAADoSGCvgJfMAACMh8BeAS+ZAQAYD4G9Ql4yAwCw/QQ2AAB0JLCXzPprAIBxEdhLZv01AMC4COwVsf4aAGAcjq/7BrbVqdNnr8xeAwAwHmawl2Qyri0PAQAYDzPYS3bmoXvWfQsAAKyQGewlsHMIAMB4CewlsHMIAMB4CewlsnMIAMD4CGwAAOhIYAMAQEcCGwAAOhLYndlBBABg3AR2Z3YQAQAYN4G9JHYQAQAYJ4ENAAAdCWwAAOhIYHfkAUcAAAR2Rx5wBABAYC+BBxwBAMZLYAMAQEcCGwAAOjq+6B/QWjuR5IEkF3I52L9dVU/MOO/+JO9OcjHJm5M8VlUvLnp9AADYJAsHdpKW5JGqeiW5HNKttXdV1XeunNDa+5P8e1V9fuLYbyX5ww7XBwCAjdFjiciJ/bje88Uk902dc2dVfWXq2Pc7XBsAADbKQoHdWjuZqVCuqjeSvDZ16hdmfPzmRa4NAACbaNElIrcleWnG8avCvar+bfLr1tpNWTCwd3d3F/n4oZ07d27u66763ujnMOPMcBnn7WeMx8E4j8MQx3nRJSLHk7w+4/ixG3zu15L85YLXBgCAjbPoDPZrOeRMdGvtZ5P8S1X96yIX3tnZWeTjh7b/U9N1r/vkszc+h4021zgzeMZ5+xnjcTDO47CucV5kxnzRGewXk9w678mttR9JcldV/c2C1wUAgI20UGBX1YUkb588tre++uT0uXvHfyPJo4tcEwAANlmPbfouttbeNvH1/UmennHeJ5I8urfLCAAAbKUeL5p5LMmDrbXzSU4keaGqvtVauztJquq51toHk9yV5FJrbfKzZ6rq7zrcAwAAbISFA7uqLiZ5eMbx5yZ+/TdJrLsGAGDr9VgiAgAA7BHYnZw6fXbdtwAAwAYQ2J08883vJUnuvf2WNd8JAADrJLA7e/gjd6z7FgAAWCOBDQAAHQlsAADoSGADAEBHAhsAADoS2B3Yog8AgH0CuwNb9AEAsE9gd2SLPgAABDYAAHQksAEAoCOBDQAAHQlsAADoSGADAEBHAhsAADoS2AAA0JHABgCAjgQ2AAB0JLABAKAjgQ0AAB0JbAAA6EhgAwBARwIbAAA6EtgLOnX67LpvAQCADSKwF/TMN7+XJLn39lvWfCcAAGwCgd3Jwx+5Y923AADABhDYAADQkcAGAICOBDYAAHQksAEAoCOBDQAAHQnsBdgDGwCAaQJ7AfbABgBgmsDuwB7YAADsE9gAANCRwD4i668BAJhFYB+R9dcAAMwisBdk/TUAAJMENgAAdCSwj8D6awAADiKwj8D6awAADiKwF2D9NQAA0wQ2AAB0JLABAKAjgQ0AAB0JbAAA6EhgAwBARwIbAAA6EtgAANCRwAYAgI4ENgAAdCSwD+nU6bPrvgUAADaYwD6kZ775vSTJvbffsuY7AQBgEwnsI3r4I3es+xYAANhAAhsAADoS2AAA0JHABgCAjgQ2AAB0JLABAKAjgX0InzpzYd23AADAhhPYh/D8y28ksQc2AAAHE9hHYA9sAAAOIrABAKAjgQ0AAB0JbAAA6Oj4on9Aa+1EkgeSXMjlYP92VT0x47yfSfJLSc4neUuSz1XVC4tef1XsIAIAwDwWDuwkLckjVfVKkrTW7m+tvauqvnPlhNaOJflAVf3BxLEHknymw/VXwg4iAADMo8cSkRP7cb3ni0numzrnA0memjr2jdbauztcf6XsIAIAwPUsFNittZNJvj95rKreSPLa1Knvqap/nDr2fJKfWuT6AACwaRZdInJbkpdmHJ8O91kh/3KSdx71wru7u0f96ELWdV1W49y5c0mM87YzztvPGI+DcR6HIY7zoktEjid5fcbxYzf6YFVdSnLzgtcHAICNsugM9muZL5IvTR/Ye/DxjaNeeGdn56gfPZLPZnct12W19n86Ns7bzThvP2M8DsZ5HNY1zovMmC86g/1iklvnOG9WSL8zs5eXAADAYC0U2FV1IcnbJ4+11m5KcnLq1H9urb136tj7knx9kesDAMCm6bFN38XW2tsmvr4/ydOTJ1TVM7l2676frqp/6nB9AADYGD1eNPNYkgdba+eTnEjyQlV9q7V2d5JU1XN7532ltXYqyatJ3pHk8Q7XBgCAjbJwYFfVxSQPzzj+3NTXX03y1UWvBwAAm6zHEhEAAGCPwAYAgI4ENgAAdCSwAQCgI4ENAAAdCWwAAOhIYAMAQEcCGwAAOhLYAADQkcAGAICOBDYAAHQksAEAoKNjly5dWvc9HMpTTz01rBsGAGCw7rvvvmOH/YwZbAAA6GhwM9gAALDJzGADAEBHAhsAADoS2AAA0JHABgCAjgQ2AAB0JLABAKAjgQ0AAB0JbAAA6EhgAwBARwIbAAA6Or7uG9g0rbUTSR5IciGXfwD5dlU9MeO8n0nyS0nOJ3lLks9V1QurvFeO7hDjfH+Sdye5mOTNSR6rqhdXea8czbxjPPWZX03y1qr6kxXcIh0cZpxba3ck+VCSc0n+uqr+aWU3ykIO8Xf2Lyf58b3z3p7kiar62gpvlQW11n4hyYmq+vIBvz+I/hLY12pJHqmqV5LLgdVae1dVfefKCa0dS/KBqvqDiWMPJPnMyu+Wo5pnnN+f5N+r6vMTx34ryR+u/G45ihuO8VUnt/aj8XfiEM01zq21n0tyR1X9/jpukoXN83f2TyZ5o6o+PXHsk621b1TVxdXfMoexN6H1niRfTvL+A84ZTH9ZInKtE/vfwHu+mOS+qXM+kOSpqWPfaK29e6l3Rk/zjPOdVfWVqWPfX+5t0dE8YzypJXl8ubfEEsw7zh+sqkdXdE/0N884/3xV/dXUsS8l+all3hjdfKGqPl1Vf3+dcwbTXwJ7QmvtZKYCqqreSPLa1Knvqap/nDr2fHwTD8IhxvkLMz5+87Lui34OMcb7538oydNVdWkFt0cn845za+3OJH+7wlujo0N8Px9rrU3/K9Sbcnm5CBtuzr9/B9Nf/jn0arcleWnG8ekfRGb9YPJyknd2vyOWYa5xrqp/m/y6tXZTBPZQzPu9nNbaW5L8RFU9ufS7ord5x3knyV+01j6a5GQu/3/f41X13SXfH33MO86fT/KJ1tpnqurS3vf2z1fVw0u/Q1ZlMP1lBvtqx5O8PuP4sRt9cO8nL/E1DEcd519L8pf9b4clOMwYfzyJpQPDNO84vzXJr+Tyw1CPJPnjXA6xE8u9PTqZa5yr6ntJvprkT1trv5Pkj5I8tvzbY502tb8E9tVey3yDdM0/Y+wtvH+j+x2xDPOO8xWttZ9N8i9V9a/LuSU6m2uM9x58+1pVvbr8W2IJ5v1ePpbkz6vqQpJU1etJPpvkw0u8N/qZ9/v5/Uluq6qPVtX/TPI/knxsbyab7TCY/hLYV3sxya1znDdrIN+Z2f+ExeaZd5yTJK21H0lyV1X9zfJuic7mHeNfTPKfWmu/uf+fJP9179cnl3qH9DDvOH93eheJveUh71jKXdHbvOO8U1X/d/+LvR+oPp3kvy3rxli5wfSXwJ6w98349slje+tup/+P9p9ba++dOva+JF9f4u3RySHGef/4b8QSgkGZd4yr6veq6pHJ/yT5yt6vPRi14Q7xvfxqa+3NU+e9KbOXHbBhDjHO13zPVtUPonW2yWD6y//ornWxtfa2ia/vT/L05AlV9Uyu3R7op720YFBuOM57PpHk0b0n1hmWeceYYZtnnP8qycemjv363nGGYZ5xfrW19mOTB1prH07yd8u+OVZjSP1lF5FrPZbkwdba+SQnkrxQVd9qrd2dJFX13N55X2mtnUryai7/M6P9c4flhuPcWvtgkruSXGqtTX72TFX5C3vzzfu9zLDdcJyr6lxr7Uuttd9O8oNc3rrtzPROQWy0ecb58dbax/ceXn09yQ8l+bo3OQ7bUPvr2KVLtn0FAIBeLBEBAICOBDYAAHQksAEAoCOBDQAAHQlsAADoSGADAEBHAhsAADoS2AAA0JHABgCAjgQ2AAB0JLABAKAjgQ0AAB0JbAAA6EhgAwBARwIbAAA6EtgAANDR/wcTWcaQAppCuAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5f5821160>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from matplotlib import pyplot\n",
    "\n",
    "logistic_regression_probabilities = pipe.predict_proba(X_test)\n",
    "logistic_regression_probabilities = logistic_regression_probabilities[:, 1]\n",
    "logistic_regression_auc = roc_auc_score(y_test, logistic_regression_probabilities)\n",
    "print('Logistic: AUROC=%.3f' % (logistic_regression_auc))\n",
    "logistic_regression_fpr, logistic_regression_tpr, _ = roc_curve(y_test, logistic_regression_probabilities)\n",
    "pyplot.plot(logistic_regression_fpr, logistic_regression_tpr, label='Logistic')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find an AUC of 0.83. Not bad for a quick exercise without fine tuning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretability model\n",
    "We use [lime](https://github.com/marcotcr/lime) (Local Interpretable Model-Agnostic Explanations) to explain the predictions.\n",
    "It is a method of determining which feature has the greatest effect on the predicted value,\n",
    "and is explained in depth in the the [FFL report](https://ff06-2020.fastforwardlabs.com/).\n",
    "For more information, refer to the [lime documentation](https://lime-ml.readthedocs.io/en/latest/lime.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lime.lime_tabular import LimeTabularExplainer\n",
    "\n",
    "data[labels.name + ' probability'] = pipe.predict_proba(X)[:, 1]\n",
    "\n",
    "# List of length number of features, containing names of features in order\n",
    "# in which they appear in X\n",
    "feature_names = list(ce.columns_)\n",
    "\n",
    "# List of indices of columns of X containing categorical features\n",
    "categorical_features = list(ce.cat_columns_ix_.values())\n",
    "\n",
    "# List of (index, [cat1, cat2...]) index-strings tuples, where each index\n",
    "# is that of a categorical column in X, and the list of strings are the\n",
    "# possible values it can take\n",
    "categorical_names = {i: ce.classes_[c]\n",
    "                     for c, i in ce.cat_columns_ix_.items()}\n",
    "class_names = ['No ' + labels.name, labels.name]\n",
    "explainer = LimeTabularExplainer(ce.transform(data),\n",
    "                                 feature_names=feature_names,\n",
    "                                 class_names=class_names,\n",
    "                                 categorical_features=categorical_features,\n",
    "                                 categorical_names=categorical_names)    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explaining a Single Prediction\n",
    "\n",
    "Let's look at how one specfic prediction would be interpreted.\n",
    "Lime explains the prediction by giving every feature a weight from -1 to 1.\n",
    "Features with weights closer to -1 have a stronger impact in coming up with a 0 prediction result (will not churn) and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>4809</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>gender</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>SeniorCitizen</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Partner</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Dependents</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tenure</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>PhoneService</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>MultipleLines</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>InternetService</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>OnlineSecurity</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>OnlineBackup</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>DeviceProtection</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>TechSupport</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>StreamingTV</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>StreamingMovies</td>\n",
       "      <td>No internet service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Contract</td>\n",
       "      <td>Month-to-month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>PaperlessBilling</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>PaymentMethod</td>\n",
       "      <td>Mailed check</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>MonthlyCharges</td>\n",
       "      <td>19.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>TotalCharges</td>\n",
       "      <td>19.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Churn probability</td>\n",
       "      <td>0.220148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "id                                4809\n",
       "gender                          Female\n",
       "SeniorCitizen                        0\n",
       "Partner                             No\n",
       "Dependents                          No\n",
       "tenure                               1\n",
       "PhoneService                       Yes\n",
       "MultipleLines                       No\n",
       "InternetService                     No\n",
       "OnlineSecurity     No internet service\n",
       "OnlineBackup       No internet service\n",
       "DeviceProtection   No internet service\n",
       "TechSupport        No internet service\n",
       "StreamingTV        No internet service\n",
       "StreamingMovies    No internet service\n",
       "Contract                Month-to-month\n",
       "PaperlessBilling                    No\n",
       "PaymentMethod             Mailed check\n",
       "MonthlyCharges                    19.9\n",
       "TotalCharges                      19.9\n",
       "Churn probability             0.220148"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sample().T  # reminder of the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('tenure > 55.00', -0.2764138466515261)\n",
      "('MonthlyCharges > 89.86', -0.24321978003513584)\n",
      "('InternetService=Fiber optic', 0.2096249701592442)\n",
      "('TotalCharges > 3794.74', 0.2031826086609449)\n",
      "('StreamingMovies=Yes', 0.08274884799449057)\n",
      "('StreamingTV=Yes', 0.07781839117828696)\n",
      "('PhoneService=Yes', 0.04962121848245511)\n",
      "('MultipleLines=Yes', 0.04446637536101756)\n",
      "('Contract=One year', -0.04392535067270691)\n",
      "('TechSupport=No', 0.04173749428961184)\n"
     ]
    }
   ],
   "source": [
    "exp = explainer.explain_instance(ce.transform(data.sample())[0],pipe.predict_proba)\n",
    "for cols in exp.as_list():\n",
    "    print(cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1AAAAIwCAYAAACImIrfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAIABJREFUeJzs3Xm8XVV99/HPTgKEMCOCIk6Viog4pFasUEES0eLYan84wAOOLVXRVh/b56l9OmifTpa2zsgsOP2wKghOTRAsiEyXwTJJHgoqKFOYEwhJ9vPHXgd2Dufcu29yk5Ob+3m/Xnlxz15rr7323ifhfO9ae52qrmskSZIkSRObNeoOSJIkSdJ0YYCSJEmSpI4MUJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkdzRt0BSZI0PVRV9STgvUAFXFDX9ekj7pIkbXBVXdej7oMkaR1UVfWxuq4/Mup+jKeqqgOAOXVdLxp1XwaZDtcQoKqqC4ED6rpePoJjbwYcAxxV1/X9G/r4w1RVdQRwXl3XS0bdl2GqqnoicCSwGTCbZgbQauCYuq7/X1/dafFelGYyR6AkSdrIVFX1V8DH6rpe2Vf00rquHxpBlwD2BXJjCk/TQVVVzwT+CPjLuq7vaW2fC3y0qqrT6rq+aGQdlDRpPgMlSdI0McLwBPBk4L9HePxpp6qqCngf8KF2eAKo6/pB4H8B76mqys9j0jTiX1hJktTFbGDVqDsxzewPfHvASCIAZfvZwG9t0F5JWidO4ZOkGaSqqlcDLweW0zyPsQXw2bqurxpQ9wjgeTTPamwJ3A/8bd80pM1ppiftSrOwwKzS9j/1/8a9Y/+eCryztDOv/PlsXdeXl/IXAK+o6/rvB+z7J8CP67r+UVVVTwMWApcC/wOoy5/ZwNF1Xf9sgn7sDLyf5v+Ts2mu09U0z6ysbtX7WF3XH6mq6pXAK4AV5Th3luOsatWdBRwO7NXqSw18vK7rX5Y6+wOvAV4CbFNVVQ18o67r89vH6+vrvkDw6D3dEji1rusf9dX7GPAXNPfrGaWvWwDn1nX9zXGuxW7AB0q/96uq6m5geV3Xf9Gq0+l9VfpwbLm2q4Gr67o+YZxjPxt4N7CynNds4OS6ri8Ytk/Zb3/g9cDDZZ8tgRPqur6kr96TgaNKX2YBc4Frgc/13btn07wvV9K8z7eiee7qS+P1AzgA+NgEdRYBeww4hy7vqaHPS/WXlWt/NPBhYHPg5rqu/7n3fCJwDfCeUraS5h4eXdf1zyfovzTjGKAkaYaoquoPgYfqun5/a9tc4J+qqvpiXdc/bm1/H3BdXdcntbY9BfhUVVVva/1G/ZM0H7Kua9V7BnA88MZJ9u/pwJuAv+61XxYu+HhVVcfVdf2Tuq4vq6rqoKqqXtR+bqSqqucBW/eFhhcCjwM+2As9VVXNA/6lqqoP1XV935B+7AT8X5ppV3e3tr8W+Mvyp13/EGBuXdd/3Nq2D/CHwKdbVf+OJgyd2Kq3C/D5qqoOqev6wbquzwXOHecZqP6+vp7mw/cft85xNvCRqqp2HhCM/ho4o67rT7fa+GBVVc+r6/qKQceo6/oXwIeGLdYwmfcVsA1NePpIXdfLJji3PWjC74da74c5wN9VVbVq2HNDVVW9Cdi+735sDnyuqqp/7oW6su1/0bw/lrfqvhz4KPC/y+udgbcDH27fj6qqDq+q6r11XX9qnNOoJ7qHdV3fDNzcdw5d31OTMRv4P8BH67q+s69sF5rg/hd1Xd9bjrcV8E80gVtSi1P4JGkGKOFkl/aHd3jkOYz3A39UntforRi2VV3X3++r+zPgZOC1pd5+wOJ2eCr1/h9wQfmt/WS8FfjH9gfOuq4fBj4EvKtV7+PAe8sHvN6H9Q/SBJS2Z9R1/Q/tEaPyof2faZ5LGeYPgD9vh6ey7xnAbn115wG/Vtf1yX11L2zXLSMdd/SFCeq6vhX4AnDgOP0ZqKqqbYADB5zjqrqu/xr4nVKnZx/gov5RGOATwBsme/zSh87vq+L1wN9MFJ6Kd9N8oG+/H1bShJ5Dx9lv37quP9fXnxXA/6QJZD3PBb7Xv6JhXdf/AWzd2vRS4JT+IFTu+TMnOIe1+UV1p/fUWngl8IkB4QmakdE/74WncrwHaML889fhmNImyQAlSTPDm4CBvykvH77PpFllDZoPucOmJl1AM0UL4EbgO0Pq/QR4YtfOle8X+ll7elKrfw/TTB/svV4F/A3NKBE0I0J/X+q1nTboWHVd/xTYcZzufKMEm0FuLSM8PdsBpwyp2+7PAzRBaZBJXauW1wEnjlN+fKnTs7qu6zP7Kw24bpMxmfcVNCNYdw+q31ZV1dbA0kF9K0Hm8nF2/+yQ/vQHh1uApw1p40Otn39OM+VxonqDVBOUD9L1PTVZV9d1fcOQsnPqwasr/hR4+jocU9okOYVPkmaGeUN+89xzHvB75b+7DXtGqPxW+qvl51+M094KJvf/mBcALx1n1OopVVXN6o201HW9pKqqa6qq+hfgv+u6vnrAPqsHbOuZPaxgSFs9K1jzQ/Evx7kOj/ySsq7rpZNos6tfB744TvnFrBmgLh6n7tr+QnUy7yvovorfHjTP5Aw03nNTE9y/h1v1bqmq6sGqqj5CM8J0U6tsRevnC6uq+vOqqj4AfKmu69sG1ZtCnd5Ta2G8az9sGfX7aAKdpBYDlCTNDBN9QL8D2KH83Pkb1sv0rJcDL+PRRQkAngIM/ZA7wI7AP9d1PfRD8wCn0jzw/5xJ7NMz0fM3WwKH0ISUmkf/f7kfEy8KMF67vwW8mkcf0ofmA+qla9Hc7Lquh96ruq7rsgjF+jSZ99VkbEXz4X2tlKmFb6UZLd2cR/vZntJIXdfHVFX1eOB1ZbGM1cBXB0xL/dvyDGCU+g/ShK7xfokgaRNlgJKkmWGiD7o7AL1Rkk4fuss0q8/SjIL8eb3m6nQHMLn/x9xJs5LfZALU/wF+t/z3TyaxH4xzPaqqeg7N8zL/1F5Eo5T91SSP09tvNs21WkSzSMaKVtnTaFYMnFIl3K7vZccn876ajPsZf5rlUGU1xs2BT/VPFywr0a2hruvbgeNK+Vzgf1ZVdWNd16f01fsZZbpiVVXbAX9ZVdW367peNE53VldVtUU9zvd3lYVE9p6gnWHWd0CWNIDPQEnSzPBAWV1umBcDvVXYbi6LHjxGVVVzqqr6jfLyfTQren23HZ7W0uVMYiGFsmz2FXVdXwz8qKqq35vk8cb74P9e4N11Xf/XJNsczyHA1+q6zimc9vXTqlnWfZjnAtdP0bGGmcz7ajJ+Cuw5rLCsSDdo+27ADnVd/32XZ636lZUQP8qaz20NqncPzcIlEy2+cQ7NCO14fotm+feptsXEVSStDQOUJM0MX2XIcsRlpOJVwPll0+k0058GeQ6wd/l5K4Z/QJ/UiEpZynm3Mj1qUB9ntX7eiea7oL5Y9v0a8NtVVT2hb7ffYICyyuDt43Rn6aARg9KHSa+WV+wCjA0pG3StHmLiD8CnA28bp/wI4IwJe7ZuJvO+6qwsaLBTWcZ+kLcM2f54hgS2qqr2ohnl7L3+yDjtt5c1/+shfaxppvKN5xzgNWX59WEW0CzOsjZW91ajbCtLtE/5qKakhgFKkmaAsrT4nVVVvb29vXyA/BhwUu95mrr5UtcHq6p6RV/d7WlGZ7JsOpfm+3HadWZXVfXe8rK9FHQXHwE+Xb53p93mQppllnv+qvyhb9tf9m17YlVV7WWre1O0/hQ4Zpx+3Fu+V6q93zbAPwDXMfnzAvg+cGT/xqqq3kzzob+/zSU0X2I8VBkFObeqqj/rC5hVVVV/DPxne1nq9WEy76u1cCzwt+2QU87t/cC3h+xzFXBQVVVrhM+qqubThK72dMJvAx/sW2adqqpeyJojlBdXVdV+//XqHQzc1r+9rYzM/ivN96et8SxYVVWbV1X1UeDL6zCCewrwF+1zKO/xfwSuXMs2JU3AZ6Akafp7WVVVfz+k7Ja6rj8BUNf1p6uqem1VVZ+hWUThYZpnRU7sn65W1/W/VlX1tqqq/o1mcYjVNKMif9r7Dp+6rv+jqqq3VFX1idLeLJoPnqfSfLCcaOrSGuq6/nlVVUcBHy6LH6ykWS3v8t73DJVAdFb/ym91Xd9TVdVpVVW9q67rY8vmM2mmI/4bzbNAq0v/Pl5WExzmn4E/q6rqsNKHWeXc/5Hmu5SeBExqelhd11dVVbVbVVWfplkcYVb5cyZNIP3jvl1OB44uUxOvrev6uCHt/nvVfB/XJ6uqWk5zT+fSTBec9MjP2pjM+2qS7V5bVdXJwD9XVdV7D84BTqvreuCITV3XK6qq+jjNly8/QHO/Z9E8W/cRmvDcqztW3md/V/7bW3DiF8CHW/XOrKrqZVVV/WOrD1vSBJRhf+/afbqmPHv1ZyUM9t6HNXBs/4IVk1FWo7wAOL6qqvta5/sJ1vzOK0lTqFr7XwxJkrRx6i3MMCx4SJK0tpzCJ0mSJEkdGaAkSZIkqSMDlCRJkiR15DNQkiRJktSRI1CSJEmS1JEBSpIkSZI6MkBJkiRJUkcGKEmSJEnqyAAlSZIkSR0ZoCRJkiSpozmj7oC0vi1evNi1+iVJkvQYCxYsqCa7jyNQkiRJktSRI1CaMebPnz/qLsxYY2NjgPdgJvLez1ze+5nLez9zTad73+vr2nAESpIkSZI6MkBJkiRJUkcGKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR1ZICSJEmSpI4MUJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHc0ZdQekDWWHHXccdRdmrAWj7oBGxns/cy0AFi9aNOpuSNKUcwRKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkcGKEmSJEnqaM6oO6DpKSIeB/w5cHVr868y88xS/g6g6tvtPzPzugFtvYMmzFfA3ZmZrbLdgN8DlgFbAosz8+r+NiRJkqQNwQC1gUTEoZl56qj7MYV2AU7NzLEh5Q91Od+IeA3w7cz8ZXk9PyJemJmXlCqvysxPtOq/mzVDmyRJkrTBOIVPa2tn4LYpaOdxvfAEUALZXgAR8RTg+r76Z0fES6bguJIkSdKkOQK1AUTEq4EDI2JrYElmLirb3wJsBywHNgNOyMxVEXEocDFwYCnbGvhKZt5R9ltjNKtMc9s9M88p+94M7AncmJnfjogdgMNppsFVwNWZ+Z8D+nkYcH5m3tDhtB4P/Gic8rpDGwAPD9i2ovz3WcBl7YLMXBIRvznBsSVJkqT1wgC1AWTmmRGxfV/oeQ3w415YiYidgTcASROm9s3Mz5ayCvgfwMkdDrcNsENmfqa17TDgE5m5urQXEbFzZvaPIJ0KvCoiXgt8f4JnjTYH9o+IJ9GEpata0+4AtisBcTOa0PaNzLxnQDv9z0nBoyOjOwF3jFMuSZIkbVAGqNHZvj3Sk5m3RUQvGOwMfLZVVkfEoJGaQeYC3+q9iIhn04wqrW7V+QbwWuDf2ztmZg30FoH4nYg4mGbRhjVGgYr/Bm7KzP8o9Q+IiN/KzAtK+a+A08uI2hzgfRHxycxc2eEceqFqVunTsHJJ0kZs2bJljI0Ne1RWm6ply5YBeO9noJly7w1Qo7NPRMzu27Zt+e/NmXlvX1nXUZfbM7MdtvYCtomIvfvqTfT80vnAk4DfoG8aHUBm/qjv9TkRcThwQXn99VbZyog4ETgYOKOvqfGm+q2KiGpIiJIkSZI2OAPU6FyYmadsgONUNKvlrZiwJhAROwK/T/Mc0pcz84FJHGvo6FJm3h0R2wwoGi8c3UEzje/2SewjSdpIzJs3j/nz54+6G9rAeqMP3vuZZzrd+3UZJfNZktHZYh327Q8QW45T9yfAb3ZpNCKOAF4FnJSZJ44XniLijcP6FRFPiIj9+urPAlYN2GfQdZhb/nst8Ny+dp4J/HxYvyRJkqT1yQC14SyPiHmt17f2T6sr4aCLrfpev2ZYxcy8BnhJRGzWOs7jy8p8/U7OzFMy86EOfZhTRqt6bc6jPJuUmb8CntdX/5XAY1b+A+4oC1H02pkPXFPa+TnQf01eMaQdSZIkab1zCt+Gcw7w/ohYkpmnZea3IuKNEfFimulyc3ns80HDXBAR76dZlnwuE3+x7DHAkRFxH80o0cOZ+cX+SpN81iiBt0dEb5/NgBNb5d+JiCNpzm0W8LPMvBkgIo7qfTluuQ7vKisNzgIe6Jva+K2I+ABwP81y7uf4TJQkSZJGpaprP4tq07Z48eIaYMHChaPuiiTNKIsXLZoWz0Joak2n52A0tabTve/1dcGCBZNe3dkpfJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkdzRt0BaUO5a+nSUXdhxhobGwNg/vz5I+6JNjTv/czVu/eStKlxBEqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR1ZICSJEmSpI4MUJIkSZLUkV+kK21Edthxx1F3Yb1YMOoOaGS89zPXAmDxokWj7oYkTTlHoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR1ZICSJEmSpI7mjOKgEfEZ4KOZ+csh5U8D/iQzj1oPxz40M09d2/JWvc2Bw4FVQAXMBb6XmUsm0850FRFvBralOf/tgS9n5s2lbGvg3cC9wObALZn5zSHtHAw8DVhR2vtOZl7TKp8DHAZsAfxqWDuSJEnShjCSAAVcDBwMHD+k/OXAJVNxoIg4AFiSmb+YivZa/gA4PjOXtY717oj4VWbeP8XH2qhExEuBazPzsvK6Aj4IfLxUeQfw6cx8qJS/LiKemZk/7WtnD2B1Zn6mte0DEbEkMx8um/4ncEJm3rp+z0qSJEma2Kim8D0MzC4fvNcQEbOBGli9wXvVURkhW9IOT8UpNMFwU/eUXngCyMwauD4ito2IbWlGnB5q1T8DeNmAdl6Ymd/t23YO8Cx4ZHTqG4YnSZIkbSxGNQIFcDbNh+qz+7YfBHwP2L+3ISJ2BN4GLKcJV/f1pseVEaZfAgtL+WbAhZl5eUTsD7wWuC0irsvMb5R9dgF+H3iAR6fe3dDuRES8MzOP69v2OuAi4KXAl/tPKDOXR8QVrfpDjxMRLwKeTzMFbhbwX5l5QSk7FLgZ2BO4MTO/HRE70EyLu48mXF4DPDEzv1L22R94DrAM2AY4NTOXlrK382hYfhzw+cy8q+/cnkRzP76WmQ/2n1ufQe+bLcq5PBu4ol2QmXVEDGqziog5mbmytW1LYGn5+SmZ+e0J+iJJkiRtMCNbRKI8K7T7gKLdMvPnfdv+CPhkZn4mMz8LjEXEm1rlbwY+l5knZOYxNMGEzDyXZvTj1F54Kl6fmZ/KzBNLey8d0I8bIuLpfdt2Lc9tVa0pZv3ndd1Ex4mIzYBnZ+bnM/P4zDwW2D0ievdjG2CHcr69APFG4Oiy7XM0gexxpb1nANtm5qcz80Tg08AhpWwBsDgzjyuB8F96ZX39vhk4Ezg8It5RnmMa5ocREb0XEbFTOf4DNOFu0PuqHrDtTOAdvZHIiNiKZlSqdw1XRcRvRsThEfG2iFg4Tp8kSZKk9W6UI1AAt0fETpl5B0BE7Eoz8vKIiNgb+EFmruhty8yry4hLz/mZuar1egXDzeaxo16D6p8DHAH8d+nHdkBv1GbVgPqTOc5q4Et9ZTcCuwK/oAlH3+rftx3aMvN7EfHk8vIlmXlKq2xVRNxepkOuphmV65WtiIgc1OHMvBs4pkzDe2vZ/0tle7veDRHxgog4Hvh/wB7AkaX4auD1wLW9+iUY7gN8of94EXEV8MWIuBx4AfD+VpX9aEYTTy7tPDciXpuZZwzq/0TGxsbWZrcNasGoOyBJU2jZsmXT4t9eTa1ly5onHLz3M89MufejXsb8LOB3Wq970/fa9gQuH7Bve6GGa/rKxjuvVZl5/UT1M3M1PLJAAsCrgMlMJxt6nBL2toyIIyLinRFxBPBKHg20t7fDUhnhuW3AMXrT4p5a2nrkD83Kdttm5g+AF0fEoRHx3HL8pQPaekRm3gucD2xX2llDRBwI3JGZ78jM/wu8FzgyImZl5nJgizIlsDfa9l7gsgHtPB/YOTPfkpn/SLOq4VvLSBTA5Zl5ZatfVwI7jtd3SZIkaX0a6QhUGQ3ZDB4NKn0jST2DRog2G7Btqv2QZtrducB2JVgA1AOe3QFg0GpzA+psQ/M802fKtDciYr9xdlnN+Itq3NQegerXel7suRHxAeCsAeGu17cX0oz8XJmZfzekyadl5gmt9u+LiK8BB9CMup0MvLmMZFU0z4vtP6Cd+X3trChL3L+BZoRuUGjsMvo30Pz589d2V0nSWpg3b57/9s5AvdEH7/3MM53u/bqMko16Ch/A+RHxEpppaz8YUH4NzdSui/q2r/cAlZlLIuKAiFhCM7Wu50KaYLXGFL2I2AJ4HjBugKIJE5/vhadi7jj9WBoRjx+nvS2GFUTEvN5qgWUE58qIOBK4vq/erjQLXlycmf86Qf8HBdq7gK3LcWpaUxQj4teA/ufaBraTmQ+1ngV7zCqNNN8rJUmSJI3EqKfw9RZd2AN4ambeNKD8J8D+5XkcACLiWTz6PNJE7qf5gta1dT/NAg6PTC0sC2DsPWChhbcC/ctyD3IXsFvvRQleb2D8ULh5b7Su7PMioBeqFkXEW9qVI+KZ5cffLe33ts9i8GjWLzPz3zLzRx36v3kZRWt7HfCYfSNiS+DlmXnhgHaW96b6ter/DnBpeflwRDyhVbYdG0folyRJ0gy1sXwYvZvxF344AXhfRPSWA7+3t7BAB1cAH46IfcoKdZN1FvBX7UUsis8C74yIh2kCyeY0q93dN1GDmXl++dLdF9GMsswBPgNsP85uXwc+UK7BauAm4NbS3o0R8YQysrSMJoj1nhv7GnBERKwox9mevsUcShuDVskb5lSa1fMeplldb0vgot5iIAAR8eayvQZObG3fpxzvwsz897K63mY0U/O2AH6amdeUOl8tz27NK7tvARwziX5KkiRJU6qq68l8bp55ymjXzpn5w1H3pa08M/bGzDxt1H3Z2C1evLiG6TEfd4cdXSND0qZj8aJF0+LfXk2t6fQcjKbWdLr3vb4uWLBg0CMj4xr5FL5p4GXAf466ExGxT2tFQGi+OLjLdDtJkiRJU2RjmcK30SlLfu8P/Nckp7etLzcB74+I3vLtN5Qvv5UkSZK0gRighuitWDfqfvRk5q+AiVbHkyRJkrQeOYVPkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6mjPqDkh61F1Ll466C+vF2NgYAPPnzx9xT7Shee9nrt69l6RNjSNQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR15PdASZKk9WLheQvhvFH3QiPjvZ+5JnHvlx41/b4D0xEoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkcGKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHU0Z112johDM/PUqerMhhIRuwCHAA8AmwHzgLMz8/IpPs5bgO9n5h1T2W7fMT4MLB1QdElmXh4RR2XmJ0rdjf5+RcQs4I8y81Oj7oskSZLUb50C1GRExAHAksz8xYY6ZuvY/cHhUODozKxbdY6MiJsz8/apOm5mfmmq2hrHLeOFol542pi1709mrgYMT5IkSdoozbgpfBGxPXB9OzwVxwIHj6BLkiRJkqaJKRuBKiNMvwQWAstppsZdWKaR7Q+8FrgtIq7LzG+UfV4DPBlYBmwNHJ+ZyyPiUOAqYD/gtsz8atn2K+DpwMPAFpl5TOv4+wPPKW1tA5wKPAQcBrwoIrYGzip9fHp//zNzZUScXtqaC7wTWAmsohnlOauUHUoTPOcClwC/kZnH9l2LtwFfAN7cG1mJiHnAkcCK0q+7MvO0UjYLeHu5Zg8BD07V6FXf6NtWEfEemus3D7gxM7/ZqvsWYDsevX8nZOaqcs43A3uWfb494DiHAdsDDwLbAidl5p2l7IjSXkVz3e7KzFMi4gnA63n0/nwlM+9u9zkiDgL2Au4v+349M2+eimsjSZIkTdZUT+F7M/DRzFwFj3xwvjwzz42IitYUvoh4MU04+lZ5PQ94C3B8aWvvzPxkq+1n0gSZY0v9J0fEgsxcHBHPALbNzE+XstnAuzPzs8DnIuL+9jS3iLgiIt5H84H9kSl7mXl3+fEI4LjMfLDUPyAi9srMq4B9gE9n5rWlbK+I2DIzl7f6ulkJHv3X5rOZuazs94KIeFFmXkQzpfDrmbm0lD07Ig7MzLMj4pXAE/qu8wO98DVJrwQiMx8ux3lDRDwxM39ZwuyPM/OGUrYz8AYgaQLpDpn5mUGNRsSbgEsz8+ryeg7wv4G/KVUWAkdm5n2l/EUR8RuZeSkD7k+r3Z2B7TPzX1rb/hD43FqcO2NjY2uzm6bAsmXLAO/BTOS9n7l6916SxjMd//8w1VP4zu+Fp2LFOHV/PTMv7L0oweKB8nI28L2++rMz8+xW/Z8Du5SXL+kFsVK2Cri9BKnHyMxzgOOAA8qzT4f06kbEdsDNvfDUqr93efmLXngqvgu8qvciIl4IDHonPNwLT6XNy4C5JViu6oWnUnY1sHP5+buZeVJ8DmdTAAAgAElEQVTfn3Z4elJE/GHfn98bdN7Amb3wVHwTWFB+3r4Xnspxb+PR98dc4FsMt0MvPJV9VwKLIuK5ZdPZvfBUyi8Cnj1Oez0HAf/et+3qQRUlSZKkDWGqR6Cu6Xs9XkDbq4xQte1a/rsqM2+doO12+08d0NZONFPJ7hp08DJi1JtC90TggxHxKWAP4FkR8bi+XXrhZ43pY5l5e5mK1vOi/pGaiNgJuG1AH35YRlmeGxGbDTm3idw8iZX1Huo7/qqI6D0Lts+AwLlt+e/tfcFr3HaLS2im513J4CC9asC2x+gL5GTmD7vsN8j8+fPXdleto95vl7wHM4/3fuaajr9VlrThjer/D+vyb9QGW4VvgKsy85Qpauumrm1FxLaZeW97W5nCdjTNlLUbgW9m5vWTOP5tZWn0O4HVA8rr8meQCliUmf8xpL+vAp7Yt/mBzPzyJPo3nl5Qu3Ad7seggNQfCPsNHB3sM+haSpIkSSMzygC1xYja2i8iruxfTr0sIrElcC3N1LHJBKizaJ7fuhlY1F+YmXeWkaY1RMSzaUbWnjKs4d7iFetDGXHqhZR1uR9zB2x7Ps1CIMN0GWGrImJWWdocaK5Ze7qgJEmStCFtyGXM7+fRKWEAP4mIhe0KEfHMtWx7UVlBblhb7fP8Ac3KfGsoi1rcmJn3ALtExLatsi0jYrdhB8/MB2jObY/MXDKk2uZloYxem7sDO5fl1FdGxJNaZbNK+VQ7uDxz1fMaHg18t0bE3u3Kk7gfd7f7W6YjvrQsugFwYERs3irfh2ZqX8+w9+F/AL/ft+0lHfskSZIkTbkNOQJ1BfDhiNgnM0/MzAsj4qCI+AOapa+3AM5Zm4Yz88aIeEJEHEnzrNJmwOWtKneU1dsuyMwrIuIrEfEhmuW6VwKbAz/LzNNL/c8D74iIh2hGaGYDJ0/QjauBp45T/mXgj0qby4F7MzNL2ReAw8uI0Cqaa7E+voT3QuDIiOgtY35D71mzzPxWRLyxBMkVNKNKZ3RpNDO/FhFvLYF4Vmn7860qP6a5nqtKu0v7ntt6OCLeTTOVcY2FLCLiroj4E+C+0vb31+7UJUmSpHVX1fWwR3OkqdH3XVQb3OLFi2vwIfZRciGBmct7P3ONjY2x8LyFE1eUNKMtPWrpxJXWg97/nxYsWFBNUPUxNuQUPkmSJEma1gxQkiRJktSRAUrr3Sin70mSJElTyQAlSZIkSR0ZoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6mjOqDsgSZI2TYv2W8T8+fNH3Q1tYGNjYwDe+xloptx7R6AkSZIkqSMDlCRJkiR1ZICSJEmSpI4MUJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHflFupIkab1YeN5COG/UvdDIeO9nrta9X3rU0tH1Yz1xBEqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR1ZICSJEmSpI4MUJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSepozigPHhH7Anu2Nr0Y+HH5+eHMPHk9HPPQzDy1/FwBhwKbl+ItgB9n5lgpPwBYkpm/mOp+bAwi4oU01/whYDPggsy8LCJ2AN4wYJd9gD/OzPsj4gnAW4F7gC2Bn2TmOUOO8wHgksw8r2/7k4FXDNjlxZn5zgHtvBR4fmZ+ous5SpIkSVNppAEqM88Hzu+9jogHe+Gmq3YgWgtHAGdm5u2t9g6JiNs21dDUExHbAntl5qda294eEVdl5l3AcQP2eSgz7y8vDwM+npl1KXtXRFyemXf37fMc4G4GyMyf9x+nhNrlA469JbA7sHQSpylJkiRNqRk7ha98IK/b4QkgM78KHDiaXm1QLwPO6Nt2MbDHoMoRsQdwbfl5T+BHvfBUfAl4Td8+s4ADgHMm2a8fDNh+GPDlSbQjSZIkTbmRjkB1FRGHAdsDDwLbAidl5p0R8WrgwIjYmmaq3aKI2Ibmw/aDZffVmXnSgGZfDPxwyCEvbP28VUS8h2ZUZDPgwsy8vPRrT+ClwEqaMHpTZn6/lB0KXAXsB9yWmV+NiC2A99BMmVsJ3AJslZlfKfs8r7S3DJgHnJGZN5Wy3y/XoAZ2AE7LzBv7rtM2QABfy8x7hl3P4oy+AEQ55rD99snML5Sfn0Nf+MrMB0pgajsEOI1mil9Xu2Xm2e0NEfFc4PrMXB4Rk2hKkiRJmlobfYCKiDcBl2bm1eX1HOB/A3+TmWdGxPZ9U/h+HzgmM1eV+jtHxO9k5nf6mn4CcB4DZOZ1rZdvBj7aau8I4PJS9tuZeUyrrwdHxONbo1p7Z+Yn26cDHJuZ95X6LwBeX37enub5nk+22nsP8OmI2B24NTNPa5W9D2i3TWbeFxFfAg6JiHnNprxjyDmuEZ5KCH1RX397ZXNZc1rdagaPXj7SZkQ8iSa83hoRTxvUhwHH2RW4uW/bLODAzPzXLm2MZ2xsbF2b0FpatmwZ4D2Yibz3M1fv3kua2TbFf/+nwxS+HXrhCSAzVwKLyqjEIF/phZ1S/zaaEZt+szPz4Q7HP7/dHrCi9XP/s1c/4dEpcLOB7w045n2tvl0G/Ly8PHhAe5dFxFNogslmfWVfHNTZzFxeRtyOB14VER8owWSgiJgfEX8GnAicPqTa7wDfbb2+BHhJXztbA3u3Nr0JyGHHHeJA4Oy+bW/FqXuSJEnaSGz0I1A00936XUIzcnNlf0FmLouIVwK7AatowsegKWSrI2JWZq6e4PjX9L1uh87VEfEWmqlvK2mC2mWlbFVm3tq37yoeqzfVcGfgsL4pavOAOzPzuoh4TkQcTvMc0sWZOe5iCpn5UEScS7Oa3rNopgsOqjcGjEXEbOAPImLQqNW2fcHvpoh4bURcnJn3RsRWwLt75x4RrwK+O2CK4FDl+HV7n7JK38oB13GtzJ8/fyqa0Vro/fbJezDzeO9nrk3xt86SJm9j/fd/Xf6Nmg4BasWAbf2jMY+IiHfSjBp9t7Xt0AFVbwOeSN+UsVL/mZn50/E6VaaWvR84LjPvLNt2o1kpbphBAarnzsw8ZVhhZp5ejrE7TdC5NDMvGtK3PWiWB78ROHpQkImIzdojcJm5KiI+R/P82MmtenszIKgCxwCHlue6VgLH8uiS5POBx0fEb5bXOwE7RMRTMvNLQ05xIbCob9tBNCH1iNa23yrTOM+YKERKkiRJU206BKi5A7Y9n2aBhkEezMz+UaMtBtS7gGYUa9BUuBcB4wYoYC/gO73wNE5f28abMllHxNzMfLC/ICLmZeYygMxcAiyJiCOBi/rqbQ28A7iuw3clHU7fEuKZuToiVvbVmz/o+7gycwVwQuvYv035Dq/M/Ghfv55GszjEwGfOiif0jzRl5vH9lSJi5TosWy9JkiStk+nwDNTdZdQFaEZOgJdmZi9ALS8LJvS0fyYi3ghs3d9oZj4AzC1fCNuu/3vAuV36BTyptV9F89zP0NExYFVZKa+3z6/x6BcJf4sm/LT78szy40ERsWNfW9WA9h/IzH9rj76N47KIWONLbCNif+DS1uutgPv7d+wXETsBT13b786KiKfSjJZJkiRJG7WNfgQqM78WEW+NiIU0gW8e8PlWlXOA90fEkrJK3VVldOZBYHOakaZhoeYE4G0llK0s9S4pX/A6Ub9+HhH7limDq2lGuU4CXjjObv8OvCciVpTj3UUZScvMeyLi7LLy3gPlXG+kGQk7HTi8TBucRbOU+1kD+tT5maPMvDQiDoqI9/Loc2Y3Zea1rWoHDzoOPBJk30qzWMZyhixqMWTfo/pGyA4AvjCkuiRJkrTRqOq682durQcRcUj58l6tJ4sXL65h432IcSZwIYGZy3s/c42NjbHwvIWj7oakEVt61Mb5yHrv/08LFiwYNKtrXNNhCt8moywZvnnr9fOY+FkrSZIkSRuJjX4K3ybmeuDIiHiAZnn1OzPzmyPukyRJkqSODFAbUPkupX8bdT8kSZIkrR2n8EmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI68ot0JUnSerFov0XMnz9/1N3QBjY2NgbgvZ+BZsq9dwRKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JFfpCtJktaLhecthPNG3QuNZ+lRS0fdBWnacQRKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkcGKEmSJEnqaM6oO9BFROwCHAI8AGwGzAPOzszLR9qxASLiqMz8xBS08xngo5n5yyHlTwP+JDOPWsfjvBf4TGauXpd2xmn/Q8CnMvPBAWUfyMx/XR/HlSRJktaHaRGggEOBozOz7m2IiCMj4mZgL2BJZv5iZL1rmYrwVFwMHAwcP6T85cAl63qQzPzUurYxgZOBtwGfbW+MiNcC31nPx5YkSZKm1EY/hS8itgeub4en4liagLGpehiYHRFVf0FEzAZqYL2MGk2lzLyd5jwe19sWEZsBu2fmdaPrmSRJkjR502EE6n7g6f0bM3NlRNwIvA64LSKuy8xvRMShwFXAfsBtmfnViJgLvBNYCawCbsnMswAiYk/gpaVsFnBTZn6/lB0KfK8cYwWwNXARcB+wb9lnO+CszLyht09mntravwI2pwk8t2fmt3rnEBGHA9sCD5V+PZiZX2yd5tnAy8p/2w4q/dq/1daONCM9y8ux7svMUyNiFnBEZp7QbiAi3pWZx/b1d7zr9JxyTR8GtgR+lpln9N+XIU4A3gf8Q3n9P4BHzrP08e000zMfKtfhS6XsyTTXfzmwRTmvUzoeV5IkSZpSG32AKkHpioh4H/CVMqLRKzu3jND0T+HbOzM/2Xp9BHBc7zmciDggIvbKzKuA387MY3oVI+LgiHh86zivzszjWuXvAZa1A0lEvBO4YUD3Xwt8IDNvKfV+NyK2y8x7ImIf4L8y89JSth3wKVrBIjOXRMSBPDZA7ZaZ34mI9rY/Av4xM1eU9p4dEW/KzK9ExBr3OSK2oAmE/ca7Tvtk5udabRwSETtn5m0R8UaacNl2a2Z+p5zHsoj4VUTsDtwBzMvMW1t1DwW+nplLW30/MDPPBg5qTzOMiP0iYn5mjg3ovyRJkrRebfQBCiAzz4mIC4FXR8ROwFLga5m5akD12cB3ey9KMLm5vYhBae9NNCNVp/bt/xNgD+D20tZ5feUPA1/v27Z8SNev74Wn4irg12meXdozM09q9emeiDh3QBu3R8ROmXlHOZ9dgZvbFSJib+AHvfBU2rs6InojVBdFxG9l5gXl9cG0rlFpY6LrNLuvX9+kvH8y82tDzr/tVOAvgXuBdhCrgFW98NTq+3PLy/5ppucD23c43mOMjZm5RmXZsmWA92Am8t7PXL17r43b+vi76d/7mWum3PtpEaAAMnM5cBpARDwR+GBEDFoAYVXf6MYewLPaz+AUvX/ZV0fEW2hW9lsJ7ABc1mrr+v79MvOevm2PeU6puGbAMXfttT2g/mNWqgPOolmBsDdt7aDWzz17lnr97gfIzMvLanu9ALVr3zWCia/TuRHxLuBXwLmZeS/NdLtOMnNVRFwK7JKZ97eKHg88tzwX1dYLTmdFxB/QBNofliB5V9fjSpIkSVNpow9QEbFt+bD+iMz8ZUQcDbwB6A8C/SrgmwOCUO/Zm/fTTFu7s2zbDdh9Sjo/vkEB6jEyc0UvXPQWlBgy8jZoSl47lDwcEZsD2wB3Dqg79DqVY14HXBcR2wCvKFMRjy/9OqS02/arzDyzb9sVwG4DjrsoM/9jyHFvAY4pz2ctiIhn0CyLPukFNObPnz/ZXTRFer+J8h7MPN77mWtT/w30pmJ9/N307/3MNZ3u/br8G7XRByhgv4i4sn+Z8vJs1JYd9r+WZtRmUDDYC/hOLzwVc9e+q5MymRUQz4+Il9D07QcDyq8BXkCzwEVbO0B9j2bq3g7AoCl3Q69TWfVvdmauyMz7gK9FxIER8czM/GlmfnUS59LvNuApwwojYl5mLitTC8+KiF8DFgLfX4djSpIkSWtlo1/GnCYwHNa/MSJeDNxIM01t22E7l+l2u0TEI3UiYssy0nQ38KTW9gp4E2sGj/Xl2oh4JJ73RlgGVSyjP3sAT83MmwaU/wTYvwSdXnvPojXVLTNvBJ4BbF9CUH8b412nWTTTCNu2YvizX52V5elXRkT7PswqC04AvLlvl21ovlBZkiRJ2uA2+hGozFweEV+JiA/RfGBfSbMs+M8y8/Qyve3DEbFPZp44pJnPA++IiIdovjtpNnByGcXat6yit5pmmeyTgBeu59MiMy+JiEMjYl+aZ4kGLVjRdjeDp+n1nAC8LyIeoBmpujczT+6rcw/jT3kc7zrdUK5TXdq/NTN/Pk5bk/EF4PASAFfR3IcvlbJL+u7Pit7UQUmSJGlDq+q6//tpNSoRccg6TofTAIsXL65heszH3VRNpznRmlre+5lrbGyMhectHHU3NIGlRy2duNIk+fd+5ppO977X1wULFgxbDG6o6TCFb5MUEbuWL4ntvZ5HM8oiSZIkaSO10U/h21Rl5i0R8bYyBXElzXS5k0bbK0mSJEnjMUCN0DjPbEmSJEnaCDmFT5IkSZI6MkBJkiRJUkcGKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRX6QrSZLWi0X7LWL+/Pmj7oYkTSlHoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkd+UW6kiRpvVh43kI4b9S9EMDSo5aOugvSJsMRKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHBihJkiRJ6sgAJUmSJEkdGaAkSZIkqSMDlCRJkiR1NGfUHdiYRMQuwCHAA8BmwDzg7My8fKQdGyAijsrMT6xjG/sCe7Y2vRj4cfn5YeCnwAOZeeWAfZ8BPD0zF43T/m7AyzPzxAFlLwPu2hivrSRJkjSMAWpNhwJHZ2bd2xARR0bEzcBewJLM/MXIeteyruGptHE+cH7vdUQ8mJmntl5XwLuAxwQo4ADgpAna/0VEbB8R22Tmfa12ZwEvyMyj1+0MJEmSpA3LKXxFRGwPXN8OT8WxwMEj6NLIDbgWbXMyc1WHZk4A3tG37RDg39e6Y5IkSdKIOAL1qPuBp/dvzMyVEXEj8Drgtoi4LjO/ERGHAlcB+wG3ZeZXI2Iu8E5gJbAKuCUzzwKIiD2Bl5ayWcBNmfn9UnYo8L1yjBXA1sBFwH3AvmWf7YCzMvOG3j690aKyfwVsDtTA7Zn5rd45RMThwLbAQ6VfD2bmFztelysj4rntaXxl+t71XXbOzHsiYllEPCkzb46IecAumXlTq73nlWuzjGba5Bm98oj4fWD7cl47AKdl5o0d+y5JkiRNKQNUUYLSFRHxPuArmXl7q+zcMp2tfwrf3pn5ydbrI4DjMvNBgIg4ICL2ysyrgN/OzGN6FSPi4Ih4fOs4r87M41rl7wGWZeYJrW3vBG4Y0P3XAh/IzFtKvd+NiO1KeNkH+K/MvLSUbQd8CugUoDLzxxHRP41vf+Dk0t7uNCGy3/cy85fl55OAPwU+Cry9t2/Zf3vg+e3rWM7906XtWzPztFbZ+4D2Ne9sbGxsbXbTFFi2bBngPZiJvPczV+/ea+OwIf8O+vd+5pop994A1ZKZ50TEhcCrI2InYCnwtSFT1WYD3+29KMHk5l54arX3JpqRqlP79v8JsAdwe2nrvL7yh4Gv921bPqTr1/fCU3EV8OvAJcCemXlSq0/3RMS5Q9rp6pHpe5m5BFgyXuXMXBERP42IVwArM/OuVvHBPPbaXBYRT6EZddqsr6zryJkkSZI05QxQfTJzOXAaQEQ8EfhgRHxqQNVVmXlr6/UewLMi4nF99Xq/glsdEW+hmaK2kmY62mWttvqnxC3LzHv6tlVDun3NgGPu2mt7QP0HB2wbz5UR8bzMvKKMCnWavtcnaa7rW/u27wwcFhHtbfOAOzPzuoh4TpmCeC1wcWYuXYtjAzB//vy13VXrqPebKO/BzOO9n7k29d9ATzcb8u+gf+9nrul079fl3ygDVBER22bmve1tmfnLiDgaeANw6+A9H1EB3xwQhHqrzr2fZnrfnWXbbsDuU9L58XVZ6GFcmXlhmT54Bc2zSu0peL9OM6Wv37fbo2KZWUfENzPzob56d2bmKeMc+/RynN2BP4iISzPzonU4HUmSJGmtGaAetV9EXNm/THl5NmrLDvtfCxzE4NGZvYDv9MJTMXftuzopU73S4uz2lMYSGNdmRKqnjoi57amPPRExLzOXleMsAZZExJE0C2xIkiRJG5zLmD/qB8Bh/Rsj4sXAjTSr9G07bOcy3W6XiHikTkRsWUaa7gae1NpeAW/isc/3rA/XRsQj46hlpcAFa9HOFRHxRpov151K36JvmfOIeGb58aCI2LGv/rBpjJIkSdJ65whUkZnLI+IrEfEhmsUaVtIsC/6zzDw9IjYDPhwR+2TmiUOa+Tzwjoh4CFhNszjEyWUUa98yDW41sAXNynQvXM+nRWZeEhGHRsS+NMuYD1qwoks7F0fEnwIxYeXJtXtPRJxdVt57gCbU30gT1E4HDi9TIGfRBNizpvL4kiRJ0mRUdT3ed6VqUxQRh2TmV0fdjw1l8eLFNUyPBxo3VdPpoVJNLe/9zDU2NsbC8xaOuhsqlh611mswTZp/72eu6XTve31dsGDBpGc3OYVvExcRu0bEk1uv59GMgkmSJEmaJKfwbeIy85aIeFuZgriSZgrfSaPtlSRJkjQ9GaBmgHGe2ZIkSZI0CU7hkyRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkcGKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHXkF+lKkqT1YtF+i5g/f/6ouyFJU8oRKEmSJEnqyAAlSZIkSR0ZoCRJkiSpIwOUJEmSJHVkgJIkSZKkjgxQkiRJktSRAUqSJEmSOjJASZIkSVJHfpGuJElaLxaetxDOG3UvRmPpUUtH3QVJ64kjUJIkSZLUkQFKkiRJkjoyQEmSJElSRwYoSZIkSerIACVJkiRJHRmgJEmSJKkjA5QkSZIkdWSAkiRJkqSODFCSJEmS1JEBSpIkSZI6MkBJkiRJUkdzRt2BUYuIDwNL/3979xplZ1XnefxbJCEQLuEiUZQR7VZREFujDTTYDZiIouBlxD+IEVFRGh1plz2r16w1L2bWmhezpteM3aN4QWgEAcW/2Cg0QmuCCGEExOKmIBcRBFQChktIQRKSmhdnH3h4cqrqqapz6lB1vp+1slJn7+fZzz5n10nqV3s/+1SKFgB3ZuaqiFiRmef1oU8vBo4F1pf+LAKuyMybunyd44EfZeYj3Wy30v4ngYsz86EOdZ8D/m9mjvbi2pIkSVIvDHyAAn5fD0kR8e6I2KdfHQJWAF+ohouIOCUiHszMh7t1kcz8VrfaGsM3gc8B/6taGBFvBm42PEmSJGm2cQlfB5l5KXBAP64dEbsAd3UIF2cA7+pDl6YsMzcA90bEa2tVyzPzJ/3okyRJkjQdzkCNbRSgzES9DXgK2BG4oL3kLSL2Bj4IPEkrjD6QmReXuhXAj0v9emA74N8z855SvyvwUWAEGAJuy8yrS1uvrHcmM5+JiB+Uc7cDTgKeATbTmkW7tHLdbcr1bgDenJlnVNuKiI/Rmh36UHv2LSIWAacAG4ENwKOZ+d1Stw3wcVrLCTcAT09i9iqB/wb899LW0cBltf4cDywur/EC4KzM3BwRi8trtJ7W9+p84CvOXEmSJKlfDFAdRMR8WiFkHnBIZn61lA8BJwDnRMS2wIeB/9n+gT4iDo+IwyuzK+/LzNMq7Z4I3FMefgT4YmZuKXUREUsyc01E3BwRn6UV1p5dspeZj5UvTwTOzMyny7mHRcR+mfkr4EDgy5n561K3X0Rsn5lPVZ7ighJQqk/7Q8BXM3OknPemiDggM6+ntaTwXzNzbanbNyLelplXRMQ7gZfUXsL17fCVmaMRsToiDgWuAV6fmZdUXpOjgWsrwXIJ8AFawet95blsLnV7A0cDF281aA0MDw9P5TR1wcjICOAYDCLHfnC1x35QDfL3vO/7wTUoY+8Svpoy2/I54FJgCXBhu64EpU3l4TuAc6qzISU4tZerzQOuqDW/sVxjX+CadngqLgL+urRzJXAmcFi59+nYiJhXzl0MPNgOT5Xj9y8PH2iHp+Jy4N2V5/cWoNN39aZ2eCpt3ghsV0Lj5nZ4KnW3ldeGzLw8M8+u/fluteHMXAm8HfgYUN+UY5d2eCrHruG578t5tXbuA1Z36LskSZI0I5yBgj3LzBC0ltItAM7PzD+VTRueqB3f/uF+j8x8sEN7T5a/N2fmXWOcux+wU0TsX6tf0/6izBi1l9DtCfx9RJwG7AO8NiJ2r53bDj/P61NmPhwR1RmiAzLzK9VjIuJF1WtXzr2qzAi9ISIWjPFcmroQ+GBm3l8rP7AdDit2rpxzUkQ8Cvy/zHygGuQma+nSpVM9VdPU/k2UYzB4HPvBNdd/Az2RQf6e930/uGbT2E/n3ygDFPxhiluVbxyjvB40OhkCzsvMrdqIiJ3roS0z/xARX6C1tO1e4Psdwtl41pSt0f8EbOlQP1r+jNXXlZn5406VEfFuYM9a8frM/Ha1IDNviojXd2jiusw8t1Pb5XU4vSyp/JuIOI7Wkr6nOh0vSf3+cLQAABnWSURBVJIk9ZoBauoejYiXZOYfa+VNAtStwF/Suieo7q0RcUtmPlAtLJtIbA/8GjgCmEyAuhQ4ntbs1Mp6ZZltW1IvL0sNbwdePlbD7c0rpmHhWBURsSgzRzLzGeCKiLieVoic8c/mkiRJksB7oKbjx7RCybPKRgm/7nz4czLzduDg6rK4iNij7Mz3E1obTDxPRBwE3JuZjwMvjoidK3XbR8Re41xvPa1lcftk5t1jHLZt2Ymv3eargCXlHq9nIuJllbptSn03PFRfyhgRrylffqh27CLgaSRJkqQ+cQZqijJzY0RcFBF/R+v+o+2B+zPzooZNnA6cEhHraC2f25SZ5wNExAUR8Z9pbev9DLAt8LvM/EE59+vAJyJiA60lefOAcya43m3A3uPUfxv4dGnzKeCJzMxS903go+Vepc20Zo268iG8mXlJRBxTAuJGWtuvt3fZuyIiPkVr445ty3VP69ySJEmS1HtDo6N+pI7mtlWrVo3C7Lihca6aTTeVqrsc+8E1PDzM8tXL+92Nvll76pT3PJr1fN8Prtk09u2+Llu2bGiy57qET5IkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhgxQkiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJamh+vzsgSZLmppVvXcnSpUv73Q1J6ipnoCRJkiSpIQOUJEmSJDVkgJIkSZKkhgxQkiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJasgAJUmSJEkN+UG6kiSpJ5avXg6r+92L3lp76tp+d0HSDHMGSpIkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhgxQkiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJasgAVRERX4mIPcepf0VEfLFBO4dFxF5j1L0oIo5v2J8V0z0uIk5t0kYvRMRbIuLIMeqOG++1liRJkl6IDFDP93PgXePUvx24YbKNVgNOZj6Smd+aQt+mJDMnDHw9vPYNwNKImFctj4gdgF0z8w/96ZkkSZI0NQao59sEzIuIoXpFCQGjwJYZ79Xs9m3gw7WyjwLn9aEvkiRJ0rTM73cHXoCuAA4vf1cdAfw7cCi0ZpUy83khoF4WEa8GlgEHRMSOwBmZubl9XEQcBgwB+wLPAIuByzPzlnqnImJXWsFjpJxzW2ZePdGTqfapzIStB3Yv13um1t+jgf9QrrEj8C+Z+VRELAA+BTxNK3QvBr6YmRsnun5m3hMRx0TEDpm5PiJeCoxk5rrKdQ8FXl+uuxNwXmauLXUf57mgvzvw9cx8dKLrSpIkSb1ggKrJzLsj4m1sHaD2yszLImIybd0F3BURT9bDVsUbM/Of2g8i4hRgqwAFfIRWaNlSjouIWJKZaxp3CA4GvpqZF5U2/ioi9snMOyLiIGBNZl5S6hYBxwP/AvxH4JzMfLLU7QIcC5wbEW8E3tjhWt+rhKSzgE8C/wycAPyfyvP9c2DnzPxyeTyPVlj7akQsA1Zl5n2lblvg48DXJvGcJUmSpK4xQHX2cES8KDMfASizJg/26Fo/qD2+JiL+IjNvbhdExL7ANe3wVFwEvAf43iSu9Whm3lp5PAwcBdwBvDozz21XZOZIRKwvD7ehNevVrnssIi4tX98E3DTeRTPzkYjYHBHvAO7IzE2V6oNr190cEQ+XILUFWFCp2xgROYnn+zzDw8NTPVXTNDIyAjgGg8ixH1ztsZ/r/N7emu/7wTUoY2+A6uxSygxLeXxE5etuqy+D+w2wHLi5UrYfsFNE7F87djKzTwC3Vx9k5oaI2L59jYg4sXb8S8vfFwIfjYgR4OeZeVd7id0knAl8MzM/WCvfu8N1X0RrVuonEbGizI7dkpm3TOG6kiRJUtcYoDooMx0LANobSmTm5hm6/Hxam1VUDdG6L2jCe46m4VfVmaCqMmN0ZnktDoyI99IKQ2siYimwtPNp+UTlwVMRUZ9tA7hvrOuW89r3b70hIj4HXFqWRk7a0qWduqmZ0P5NlGMweBz7wTXXfwPd5vf21nzfD67ZNPbT+TfKADW2ayLiYGA74Ccd6p8XciJiIZVlbtPwZ8Bva2W3An8JXNOF9seycKyKiFiUmSOZOQpcGxE/p3Uv0hmZOUxrKWDPrgtQNta4pdwjNqUAJUmSJE2X25iPITPvAPYB9m5vYlCzQ+3xUYwdoMZ7nd9Ze3xg7T4lMvN24OD2rBhAROxRdubrllsjYnm1ICJeU748tra1+0JaW753w8r6BwtXrvv+Ekzb5dvgNvKSJEnqI2egxvcYW9+j1HZZRHweeIJWoLgJeOUYx24fEZ8Evp+ZD9fq/lhmVbYAOwOXj9HG6cApEbGO1uzXpsw8v9S9NCJOqh3/cGZ2WjLXUWZeFxFHRMTJtLYrXwhcWap/AJwcERtofc/sAny5adsTXPfeiHhJeQ1GaG0a0d6U4kLgxIjYWLnuN7txXUmSJGkqhkZH67fbaKaUz4G6OzMf6Hdf5rJVq1aNwuxYjztXzaY10eoux35wDQ8Ps3z18okPnOXWnureRnW+7wfXbBr7dl+XLVs26VtwXMInSZIkSQ0ZoCRJkiSpIe+B6qPMvLLffZAkSZLUnDNQkiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJasgAJUmSJEkNGaAkSZIkqSEDlCRJkiQ1ZICSJEmSpIb8IF1JktQTK9+6kqVLl/a7G5LUVc5ASZIkSVJDBihJkiRJasgAJUmSJEkNGaAkSZIkqSEDlCRJkiQ1ZICSJEmSpIYMUJIkSZLUkAFKkiRJkhryg3QlSeqTXXfbrd9d6JllwKqVK/vdDUnqOmegJEmSJKkhA5QkSZIkNWSAkiRJkqSGDFCSJEmS1JABSpIkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhub3uwP9FhEHAfsDm2kFyqeAb2XmaF871gMRMQ/4MK1x31L+vi4zb+1rxyRJkqRZYqADVES8FliSmWdUyl4GHAd8uwvtHwbcnZkPTLetMdpfkZnnTeKUk2iFw3WVNo6NiMcy8/7u91CSJEmaWwZ9Cd/hmXlxtSAzHwS27VN/eiYiXg7cXw1PRQJH9qFLkiRJ0qwz0DNQwKYxyldHxLzM3BwRuwEfo7W0bxRY1571KTNMfwCWl/oFtJbE3RQRhwLvAdZExB2ZeVFErAB+BbwVWJOZ34mI1wF/AzxDK9Del5k/anckIj4B7Ag8CSwEzgB2B94HHBAROwIXZOZjEzzXw+gwq5aZoxGxuXK9FcAQrRA5CjycmZdU6o8HFlee71mZubnaZkSclJln1so+Rmv2a8NYbUTETsBHgKfLaVsy8+xy/iuANwCvBh4vz/nJCZ6zJEmS1FUGqA4y8zeVh58G/jEzNwJExL4RcVxmXlDqPwT8j3aIiIgTgZsy86cRMcTWS/j2z8wvVR7/dWae3n4QEe+KiD0y8+GIOAK4OjPvLHU7AZGZ5wNfi4gnK2FuR+CYDk/n5sy8ERjKzLECY7X8PcDnMvP3pd33R8TizHw8Io4Grs3Me0rdEuADtGaxqp6MiB1rAWe7Ep7Ga+ODwOmV13JJRByZmZeVNo4CTp6L96dJkiRpdhj0ALV5vMqI2B/4STs8AWTmbWV2qe2a2gzMRsY2D7i8Vla/h+lWYB/gYWDP6mxUZq6LiI73KpWwcvY41x7vuW6pfH1XOzwVv6I163MDsEs7+JRrromITstAf0hrWeB34dl7zW4vdeO1cUH1tSx1u1baXTWd8DQ8PDzVUzVNIyMjgGMwiBz78S3rdwd6bGRkxLEfQL7vB9egjP2gB6iJvA64tEN5dWbl9lrdePeVbc7Mh2plW8qStkW0lvHtCtzYPr7eQGZeNW6PxzY0Tt28ytf15zMCvLR8fWDZya9q53pjmflERCyuFB1Ka+nhuG1k5khEvBPYi9ZzHwW2rxz34DjPQZIkSeq5QQ9Q9R/kAYiIPwfuLQ87zSgt6MbFy8zL3wFnZuafStlewKvKIVvGOrdDWzvS2j2w7sbM/AWwOSLmZ+YzHY5p+nyuy8xzGx57T0S8MjN/S2v5YPu5jNlGRJxEa0bv8krZiobXm9DSpUu71ZQmqf2bKMdg8Dj2g23RokWO/QDyfT+4ZtPYT2eWbNAD1MIxyv8qM38TEbcDbwKur9V3JUAB+wGXtcNTsV3l663GJyL2zczb6uVlCd+Z9fKKn9HaSGJlrb0hxgiSHYz1enVyJXBiRLykXLtJG09nZn0GbDLXlCRJknpq0LcxXx0R768WRMSLaS2lo3zA7KHVJWflfp5HG7b/JB2WuFU8Brys0vYQrVmkdkD7Y0TsU6lfCLy5cn7j8SszQftExC61qg8Al3U4pZOHyn1hz4qI14xxvfaM05sy8+aGbSyqlR9DawdCSZIk6QVhoGegyoYQiyPiMzy3dfbTwLcqh50FfDYi1tOaHXoiM89peImbgX+IiAMz8xsdrn9/RBxSlq5toTXbcjbwllL/o4g4qdwXtI7W1uJnV5rYFBGfAlZWN2YYx+nAJ0pQ20Br5un6zPxdkyeTmZdExDERcRCtpY3bARePc8o1tHbOa9rGryLiFFpjsC2tmatuzfZJkiRJ0zY0OuqO0OqNiHgH8OvMvK+f/Vi1atUozI71uHPVbFoTre5y7Me362679bsLPbVq5UrHfgD5vh9cs2ns231dtmzZeButdTToS/jUW6/qd3iSJEmSummgl/CpNyLiEOBAYFW/+yJJkiR1kwFKXZeZ19C6/0mSJEmaU1zCJ0mSJEkNGaAkSZIkqSEDlCRJkiQ1ZICSJEmSpIYMUJIkSZLUkAFKkiRJkhoyQEmSJElSQwYoSZIkSWrIACVJkiRJDc3vdwckSRpUj65d2+8u9Mzw8HC/uyBJPeEMlCRJkiQ1ZICSJEmSpIYMUJIkSZLUkAFKkiRJkhoyQEmSJElSQwYoSZIkSWrIACVJkiRJDfk5UJIkqSeWr14Oq/vdi+5ae+rc/ewuSc04AyVJkiRJDRmgJEmSJKkhA5QkSZIkNWSAkiRJkqSGDFCSJEmS1JABSpIkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhub3uwMvdBFxCPC6StFBwLXl602Zec4U212RmeeNU/8q4EhgBFgIbA98LzPvncr1Xkgi4mvA5zNzpEPduK+LJEmS1E8GqAlk5jXANe3HEfF0r3/Aj4htgSMz80u18v8SEV/IzI29vP5kTSH03ACsAL7eoy5JkiRJPeESvhem1wE/7VB+PnD4DPelF54G7oyI/fvdEUmSJGkynIHqgog4HlgMPAUsAM7KzM0RMQ/4NDBEaynelsw8q3LePsDbynk7Ahdk5iPAH4B3ALdUr5OZ90fEE+XcrWZ92mURsQK4HTgQ2ALsApyfmfeX404s/RwCtgMezcxzK+28F9i79Gsn4F/bSwdL29uU824A3gIcFBE7Aldl5m1NXrPMvDIiPhsRv8zM0TFe1zeW12d96e8vM/PKJu1LkiRJvWCAmqaIOBq4NjPvKY+XAB8AEvgg8O0SioiIvSLi3Zl5Ka1AcEhmfrXUDQEnAOdk5pqIWBcRJwGZmU+0r5eZjzfo1hDwisz8SqXtk4GvlfrlwCmZua7UHxARb87MX0TEocC6zPxi5Tn+14j435m5gVYo+3Jm/rpUD0fESDXMRcQxtAJh1UOZeVmtLIEAvlN/AhGxO/A3mfmFStnxEfHayrUnZXh4eCqnqQtGRlq3uzkGg8exH1ztsZ9r/F6emO/7wTUoY+8SvunbpR2eADJzDc+9rgvb4anUPQCsKw+XABdW6kaBTZXH3we+C7wnIj5dgtpk/KDW9gMRsWspuqIdnkr99cC+5eG+mXlFra1zgLeXrx+YKMBk5oWZeXbtTz08kZkPAUMR8eIOzRwFnFE7/lvAYeNdW5IkSeolZ6Cm78CyVK9q5/L35vrBmXlV+fLB6sxSsU3t2MeB86C1K19E/D3wT5m5ZYI+jWbmM7Wy3wIvBx4FOm1C0e7rhg59fiAi2vdePTjBtScrgc8AX6qVz8vMpzocv36qF1q6dOlUT9U0tX8T5RgMHsd+cM3V30D7vTwx3/eDazaN/XT+jTJATd911fuHaiYKOh1FxM71cJWZd0fEWcA7gR9Oodn5QMd7jYp2CBxrh78FTS8UEcfSuneq6o+Z+W/1YzNzS0RcFRGH1e5vmnY/JEmSpG4zQE3fwnHqtnp9I2LfBhstHBMR36zPImXmoxGxR3n4vDAUEQtp3fs0lr2Bq8apb89+bdehzy8DHpqgz9V+bnVP0wTH3xwRfxsR11aKt4mIBZn57LLGci9XfbZPkiRJmjHeAzV9D9W3446I15QvN1UCT3tjhH0atHk5cFK9sOyOd315uEOt+iieC1DbRMTbK+cNAXtm5mOl6G3ls6ba9Qfy3I5/d0bEwbW2PwL8eJz+duP76LxynbZLgRNrxxxH5+3dJUmSpBnhDNQ0ZeYlEXFMRBxEa9nZdsDF7WrgMxEBrXt3RoFvNGjz9xHxs4j4PK3PTNpCa6br5sy8vRx2Wal/otTdBLyy1G0BNkTEKeWai2l9hlTbtcAnImJz6e/a9i56mXlVRBxdOXcn4DsTfHjv3RHxn4CbMnP1RM9vjOf8ZEQ8AOxeHv8pIm4o7W4Etgduzcw7p9K+JEmS1A1Do6Pj3Raj2ajTZ0RNpn6uWbVq1SjMjhsa56rZdFOpusuxH1zDw8MsX728393ourWnru13F17wfN8Prtk09u2+Llu2bLxbYDpyCZ8kSZIkNWSAkiRJkqSGDFBz0ETL8wZp+Z4kSZLUTQYoSZIkSWrIACVJkiRJDRmgJEmSJKkhA5QkSZIkNWSAkiRJkqSGDFCSJEmS1JABSpIkSZIaMkBJkiRJUkPz+90BSZI0N61860qWLl3a725IUlc5AyVJkiRJDRmgJEmSJKkhA5QkSZIkNWSAkiRJkqSGDFCSJEmS1JABSpIkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhgxQkiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJasgAJUmSJEkNGaAkSZIkqSEDlCRJkiQ1ZICSJEmSpIaGRkdH+90HqadWrVrlN7kkSZK2smzZsqHJnuMMlCRJkiQ15AyUJEmSJDXkDJQkSZIkNWSAkiRJkqSGDFCSJEmS1JABSpIkSZIaMkBJkiRJUkMGKEmSJElqyAAlSZIkSQ0ZoCRJkiSpIQOUJEmSJDVkgJIkSZKkhub3uwOS5p6IWAycCKwHtgVuzMyfjXHs8cBOwCZgEfCNzFw/Q11Vl01m7MvxuwKRmafPTA/VbRGxAPgUsJHWL2Z/l5mXdThuP+Bw4GlgB+DCzHxwJvuq7mo69uXYRcAHgEsy87GZ66V6YRLv+6OAvXju//jvZOaamexrLxigJPXCMcBpmbkZICJOiIgbM/Pp6kERcQTwi8y8ozzeFlgBnDXTHVbXNB37PYDjgIeBLTPfTXVRAGe3f/EREUdFxEsz8/fPHhAxBBySmadVyj4FfH3Ge6tuajL2C4FPAuuAx4EdAQPU7Ndk7N8IPJ6Z/1YpOxmY9b8wcwmfpK6KiO2A9e0foIvvA0d2OPzF7fAEkJkbgQ097qJ6ZJJj/0hmfikzLwCempEOqlcW1GaNfwgsqx1zCLCqVnZ3ROzV056p1yYc+8zckJmnZeY5GJzmkibv+/0z8+pa2ZO97dbMMEBJ6raXA7+pFmTmE8B2HY69tEPZvF50SjOi8dhn5uhMdUq9U2aNn/cDUWZuAZ6pHfryzPxNrexW4LU97J56aBJjrzlmEmM/Z/+PN0BJ6rYlQKf1zVv9e5OZa6uPI2IHWuupNTs1HnvNGUtoLcOsq495p++BR4A9ut4jzZSmY6+5p9HYd/g/fhvmSIDyHihJUxIR/wjsViu+C7gO2Lz1GQw1aPY44Lxpdk091qOx1+w0nymOeWaORsSc+GFqQE157DXrTXXsjwEu7n53Zp4BStKUZOY/dCqPiLcyhd8wRcRhwPWZ6T1QL3DdHnvNas/QbMy3WrJZNpZwA5HZq+nYa+6Z9NhHxBuAhzLzT73p0sxymlVStz1Ea3q/sYj4M2CXzLy1N13SDJn02GvWWwO8qMFxnYLSHnReBqTZoenYa+6Z1NhHxG7AmzLzp73r0swyQEnqtvuBV1ULImJnxthdr3w2yJGZ+f0Z6Jt6a1Jjr9mv7Jy5c7Ws3Oewbe3Q+yPi1bWyvwDu7GH31EOTGHvNMZMZ+1J+AnDuzPRuZhigJHVV+byfHSOiukT4GFpbnHZyMnBmzzumnpvC2Gtu2BQRO1UeHwVcUT0gM1ez9RbHr8vM+3rdOfXUhGOvOavp2H8COLfs0jdneA+UpF5I4LMRsY7WFta/rHzY3nuBmzLzvog4Adgd+EhEVM+/uvr5UJpVGo19PzuorvsO8LcR8TSwAHgwM++NiAMBMvO6ctzVEfFZWp/7tRj4Xl96q25qOvaaeyYc+4g4FHgTMFr7P/7GzPzFjPe4i4ZGR/0oDkmSJElqwiV8kiRJktSQAUqSJEmSGjJASZIkSVJDBihJkiRJasgAJUmSJEkNGaAkSZIkqSEDlCRJkiQ1ZICSJEmSpIYMUJIkSZLUkAFKkiRJkhoyQEmSJElSQwYoSZIkSWrIACVJkiRJDRmgJEmSJKkhA5QkSZIkNWSAkiRJkqSG/j8Q5gq6gSUXTQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd5f5821470>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp.as_pyplot_figure()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that one of the features that contributes most strongly to the positive prediction is the short tenure of the customer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the model\n",
    "Now that we've done all this work to build the models, we want to be able to use them later.\n",
    "The `ExplainedModel` class is a handy wrapper for using the `CategoricalEncoder`, the `Pipeline` object which *is* the churn model, and the Lime Explainer.\n",
    "Here, we use it to save these trained models for use in later parts of the Project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from churnexplainer import ExplainedModel\n",
    "explainedmodel = ExplainedModel(data=data, labels=labels, model_name='telco_linear',\n",
    "                                categoricalencoder=ce, pipeline=pipe,\n",
    "                                explainer=explainer,data_dir=data_dir)\n",
    "explainedmodel.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap up\n",
    "We've now covered all the steps to **building a machine learning model** including interpretability\n",
    "and saved our work for use in later sections.\n",
    "\n",
    "In the next part of the series we will explore how to use the **Experiments** feature of CML\n",
    "for when we want to test lots of combinations of hyperparameters to fine tune our models.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}