tutorials/bicycle_demand_monitoring_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Monitoring setup for Bicycle Sharing Demand Prediction"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "QgumzOZ5wgec"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This notebook shows how you can use the Evidently to:\n",
    "* calculate prerformance and data drift for the model, performed as batch checks \n",
    "* log models quality & data drift using MLflow Tracking\n",
    "* explore the result \n",
    "\n",
    "More examples are avaliable in the github: https://github.com/evidentlyai/evidently/tree/main/examples\n",
    "\n",
    "Evidently docs: https://docs.evidentlyai.com/\n",
    "\n",
    "Join our Discord: https://discord.com/invite/xZjKRaNp8b"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "6eM1BMvDwgee"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "try:\n",
    "    import evidently\n",
    "except:\n",
    "    !pip install git+https://github.com/evidentlyai/evidently.git"
   ],
   "outputs": [],
   "metadata": {
    "id": "oDEk9bFNwgeh"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import datetime\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import requests\n",
    "import zipfile\n",
    "import io\n",
    "import json\n",
    "\n",
    "from sklearn import datasets, ensemble, model_selection\n",
    "from scipy.stats import anderson_ksamp\n",
    "\n",
    "from evidently.metrics import RegressionQualityMetric, RegressionErrorPlot, RegressionErrorDistribution\n",
    "from evidently.metric_preset import DataDriftPreset, RegressionPreset\n",
    "from evidently.pipeline.column_mapping import ColumnMapping\n",
    "from evidently.report import Report"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "rByuPhg7wgei"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter('ignore')"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "HiiUl3p8wgej"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Bicycle Demand Data"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "zw5Tap_Xwgej"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "More information about the dataset can be found in UCI machine learning repository: https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset\n",
    "\n",
    "Acknowledgement: Fanaee-T, Hadi, and Gama, Joao, 'Event labeling combining ensemble detectors and background knowledge', Progress in Artificial Intelligence (2013): pp. 1-15, Springer Berlin Heidelberg"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "VqGH1Mr6wgej"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "content = requests.get(\"https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip\").content\n",
    "with zipfile.ZipFile(io.BytesIO(content)) as arc:\n",
    "    raw_data = pd.read_csv(arc.open(\"hour.csv\"), header=0, sep=',', parse_dates=['dteday']) "
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "dKX2YV19wgek"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "raw_data.index = raw_data.apply(lambda row: datetime.datetime.combine(row.dteday.date(), datetime.time(row.hr)),\n",
    "                                axis=1)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "36Gk-YMhwgek"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "raw_data.head()"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "N6oQxQKNwgek"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model training "
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "dhZOCJZ1wgel"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "target = 'cnt'\n",
    "prediction = 'prediction'\n",
    "numerical_features = ['temp', 'atemp', 'hum', 'windspeed', 'mnth', 'hr', 'weekday']\n",
    "categorical_features = ['season', 'holiday', 'workingday', ]#'weathersit']"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "_i8edS6Ewgem"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "reference = raw_data.loc['2011-01-01 00:00:00':'2011-01-28 23:00:00']\n",
    "current = raw_data.loc['2011-01-29 00:00:00':'2011-02-28 23:00:00']"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "vTMu0TS_wgem"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
    "    reference[numerical_features + categorical_features],\n",
    "    reference[target],\n",
    "    test_size=0.3\n",
    ")"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "dqiRpf2Swgem"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regressor = ensemble.RandomForestRegressor(random_state = 0, n_estimators = 50)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "PWJv-0UKwgen"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regressor.fit(X_train, y_train)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "UOWSxRh1wgen"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "preds_train = regressor.predict(X_train)\n",
    "preds_test = regressor.predict(X_test)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "TZKZ_biHwgen"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model validation"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "pRV2_SNSwgen"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "X_train['target'] = y_train\n",
    "X_train['prediction'] = preds_train\n",
    "\n",
    "X_test['target'] = y_test\n",
    "X_test['prediction'] = preds_test"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "JcEFJgEswgeo"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "column_mapping = ColumnMapping()\n",
    "\n",
    "column_mapping.target = 'target'\n",
    "column_mapping.prediction = 'prediction'\n",
    "column_mapping.numerical_features = numerical_features\n",
    "column_mapping.categorical_features = categorical_features"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "wEhoX-Xcwgeo"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regression_performance_report = Report(metrics=[\n",
    "    RegressionPreset(),\n",
    "])\n",
    "\n",
    "regression_performance_report.run(reference_data=X_train.sort_index(), current_data=X_test.sort_index(),\n",
    "                                  column_mapping=column_mapping)\n",
    "regression_performance_report"
   ],
   "outputs": [],
   "metadata": {
    "id": "iQk44skSwgeo"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Production model training"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "r4QPdp17wgeo"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regressor.fit(reference[numerical_features + categorical_features], reference[target])"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "9u3zyR-bwgep"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "column_mapping = ColumnMapping()\n",
    "\n",
    "column_mapping.target = target\n",
    "column_mapping.prediction = prediction\n",
    "column_mapping.numerical_features = numerical_features\n",
    "column_mapping.categorical_features = categorical_features"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "qH5FK4x8wgep"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "ref_prediction = regressor.predict(reference[numerical_features + categorical_features])\n",
    "reference['prediction'] = ref_prediction"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "vTCmG4Ntwgeq"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regression_performance_report = Report(metrics=[\n",
    "    RegressionPreset(),\n",
    "])\n",
    "\n",
    "regression_performance_report.run(reference_data=None, current_data=reference,\n",
    "                                  column_mapping=column_mapping)\n",
    "regression_performance_report"
   ],
   "outputs": [],
   "metadata": {
    "id": "uV1kyj3Owgeq"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Some time has passed - how is the model working?"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "2rJyfTJ3wger"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "current_prediction = regressor.predict(current[numerical_features + categorical_features])\n",
    "current['prediction'] = current_prediction"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "TqljYICLwger"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Week 1"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "7eBBRAOTwger"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'],\n",
    "                                  column_mapping=column_mapping)\n",
    "regression_performance_report"
   ],
   "outputs": [],
   "metadata": {
    "id": "LvuPle_Nwges"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Week 2"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "SbmG0hvSwges"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regression_performance_report = Report(metrics=[\n",
    "    RegressionQualityMetric(),\n",
    "    RegressionErrorPlot(),\n",
    "    RegressionErrorDistribution()\n",
    "])\n",
    "\n",
    "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-02-07 00:00:00':'2011-02-14 23:00:00'], \n",
    "                                            column_mapping=column_mapping)\n",
    "regression_performance_report.show()"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false,
    "id": "d6tBIZDUwget"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Week 3"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "Zj3AHeU-wget"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "regression_performance_report = Report(metrics=[\n",
    "    RegressionQualityMetric(),\n",
    "    RegressionErrorPlot(),\n",
    "    RegressionErrorDistribution()\n",
    "])\n",
    "\n",
    "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-02-15 00:00:00':'2011-02-21 23:00:00'], \n",
    "                                            column_mapping=column_mapping)\n",
    "regression_performance_report.show()"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "LF6ZFyx_wgeu"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Why the quality has dropped?"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "sFeB72oIwgev"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "column_mapping_drift = ColumnMapping()\n",
    "\n",
    "column_mapping_drift.target = target\n",
    "column_mapping_drift.prediction = prediction\n",
    "column_mapping_drift.numerical_features = numerical_features\n",
    "column_mapping_drift.categorical_features = []"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "ubmWrELOwgev"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "data_drift_report = Report(metrics=[\n",
    "    DataDriftPreset(),\n",
    "])\n",
    "\n",
    "data_drift_report.run(\n",
    "    reference_data=reference,\n",
    "    current_data=current.loc['2011-02-14 00:00:00':'2011-02-21 23:00:00'],\n",
    "    column_mapping=column_mapping_drift,\n",
    ")\n",
    "data_drift_report"
   ],
   "outputs": [],
   "metadata": {
    "id": "Uz1ajc22wgew"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Let's customize the report!"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "-4S20imIwgew"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from evidently.calculations.stattests import StatTest\n",
    "\n",
    "def _anderson_stat_test(reference_data: pd.Series, current_data: pd.Series, feature_type: str, threshold: float):\n",
    "    p_value = anderson_ksamp(np.array([reference_data, current_data]))[2]\n",
    "    return p_value, p_value < threshold\n",
    "\n",
    "anderson_stat_test = StatTest(\n",
    "    name=\"anderson\",\n",
    "    display_name=\"Anderson test (p_value)\",\n",
    "    func=_anderson_stat_test,\n",
    "    allowed_feature_types=[\"num\"]\n",
    ")\n",
    "\n",
    "# options = DataDriftOptions(feature_stattest_func=anderson_stat_test, all_features_threshold=0.9, nbinsx=20)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "zJWuGbzdwgew"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "the_report = Report(metrics=[\n",
    "    RegressionQualityMetric(),\n",
    "    RegressionErrorPlot(),\n",
    "    RegressionErrorDistribution(),\n",
    "    DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
    "])\n",
    "\n",
    "\n",
    "the_report.run(\n",
    "    reference_data=reference,\n",
    "    current_data=current.loc['2011-02-14 00:00:00':'2011-02-21 23:00:00'], \n",
    "    column_mapping=column_mapping_drift\n",
    ")\n",
    "the_report"
   ],
   "outputs": [],
   "metadata": {
    "id": "EDqb_S9Xwgex"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Jupyter vs Automated Run?"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "q5lW24Xzwgex"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "To run this part of the tutorial you need to install MLflow (if not installed yet)\n",
    "\n",
    "You can install MLflow with the following command: `pip install mlflow` or install MLflow with scikit-learn via `pip install mlflow[extras]`\n",
    "\n",
    "More details:https://mlflow.org/docs/latest/tutorials-and-examples/tutorial.html#id5"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "p-80D3Bbwgex"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "try:\n",
    "    import mlflow\n",
    "except:\n",
    "    !pip install mlflow"
   ],
   "outputs": [],
   "metadata": {
    "id": "bUoXVw40AKWj"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import mlflow\n",
    "#import mlflow.sklearn\n",
    "from mlflow.tracking import MlflowClient"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "I_IyYlM0wgey"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "experiment_batches = [\n",
    "    ('2011-01-29 00:00:00','2011-02-07 23:00:00'),\n",
    "    ('2011-02-07 00:00:00','2011-02-14 23:00:00'),\n",
    "    ('2011-02-15 00:00:00','2011-02-21 23:00:00'),  \n",
    "]"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "L6PKtAGEwgey"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "the_report = Report(metrics=[\n",
    "    RegressionQualityMetric(),\n",
    "    RegressionErrorPlot(),\n",
    "    RegressionErrorDistribution(),\n",
    "    DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
    "])\n",
    "\n",
    "the_report.run(\n",
    "    reference_data=reference, \n",
    "    current_data=current.loc[experiment_batches[0][0]:experiment_batches[0][1]],\n",
    "    column_mapping=column_mapping_drift\n",
    ")"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "nHegk6fAwgey"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "logged_json = json.loads(the_report.json())"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false,
    "id": "qMD5Ek7-wgey"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "logged_json"
   ],
   "outputs": [],
   "metadata": {
    "id": "xwOcfczR-b6d"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "[x['metric'] for x in logged_json['metrics']]"
   ],
   "outputs": [],
   "metadata": {
    "id": "cyVps9-Q-ZoB"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "logged_json['metrics'][0]['result']['current']['mean_error']"
   ],
   "outputs": [],
   "metadata": {
    "id": "yc0HhabLxYVj"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "logged_json['metrics'][3]['result']['drift_share']"
   ],
   "outputs": [],
   "metadata": {
    "id": "SolMtKPjyHw5"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "#log into MLflow\n",
    "client = MlflowClient()\n",
    "\n",
    "#set experiment\n",
    "mlflow.set_experiment('Model Quality Evaluation')\n",
    "\n",
    "#start new run\n",
    "for date in experiment_batches:\n",
    "    with mlflow.start_run() as run: #inside brackets run_name='test'\n",
    "        \n",
    "        # Log parameters\n",
    "        mlflow.log_param(\"begin\", date[0])\n",
    "        mlflow.log_param(\"end\", date[1])\n",
    "\n",
    "        # Get metrics\n",
    "        the_report = Report(metrics=[\n",
    "            RegressionQualityMetric(),\n",
    "            RegressionErrorPlot(),\n",
    "            RegressionErrorDistribution(),\n",
    "            DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
    "        ])\n",
    "        the_report.run(\n",
    "            reference_data=reference, \n",
    "            current_data=current.loc[date[0]:date[1]],\n",
    "            column_mapping=column_mapping_drift)\n",
    "        logged_json = json.loads(the_report.json())\n",
    "        \n",
    "        me = logged_json['metrics'][0]['result']['current']['mean_error']\n",
    "        mae = logged_json['metrics'][0]['result']['current'][\"mean_abs_error\"]\n",
    "        drift_share = logged_json['metrics'][3]['result']['drift_share']\n",
    "        \n",
    "        # Log metrics\n",
    "        mlflow.log_metric('me', round(me, 3))\n",
    "        mlflow.log_metric('mae', round(mae, 3))\n",
    "        mlflow.log_metric('drift_share', round(drift_share, 3))\n",
    "\n",
    "        print(run.info)"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "bTHU8eAqwgez"
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "#run MLflow UI (NOT recommented! It will be more convinient to run it directly from the TERMINAL)\n",
    "#!mlflow ui"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "IUYxYN5nwgez"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Support Evidently\n",
    "Enjoyed the tutorial? Star Evidently on GitHub to contribute back! This helps us continue creating free open-source tools for the community. https://github.com/evidentlyai/evidently"
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "colab": {
   "provenance": []
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}