feature_store.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LPZmAUydQIC9"
      },
      "source": [
        "<div align=\"center\">\n",
        "<h1><img width=\"30\" src=\"https://madewithml.com/static/images/rounded_logo.png\">&nbsp;<a href=\"https://madewithml.com/\">Made With ML</a></h1>\n",
        "Applied ML · MLOps · Production\n",
        "<br>\n",
        "Join 30K+ developers in learning how to responsibly <a href=\"https://madewithml.com/about/\">deliver value</a> with ML.\n",
        "    <br>\n",
        "</div>\n",
        "\n",
        "<br>\n",
        "\n",
        "<div align=\"center\">\n",
        "    <a target=\"_blank\" href=\"https://madewithml.com\"><img src=\"https://img.shields.io/badge/Subscribe-40K-brightgreen\"></a>&nbsp;\n",
        "    <a target=\"_blank\" href=\"https://github.com/GokuMohandas/Made-With-ML\"><img src=\"https://img.shields.io/github/stars/GokuMohandas/MadeWithML.svg?style=social&label=Star\"></a>&nbsp;\n",
        "    <a target=\"_blank\" href=\"https://www.linkedin.com/in/goku\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n",
        "    <a target=\"_blank\" href=\"https://twitter.com/GokuMohandas\"><img src=\"https://img.shields.io/twitter/follow/GokuMohandas.svg?label=Follow&style=social\"></a>\n",
        "    <br>\n",
        "    🔥&nbsp; Among the <a href=\"https://github.com/GokuMohandas/Made-With-ML\" target=\"_blank\">top MLOps</a> repositories on GitHub\n",
        "</div>\n",
        "\n",
        "<br>\n",
        "<hr>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "raojM2N08wEu"
      },
      "source": [
        "# Feature store\n",
        "\n",
        "👉 &nbsp;This notebook complements the [feature store lesson](https://madewithml.com/courses/mlops/feature-store/), where we use a feature store to connect the [DataOps](https://madewithml.com/courses/mlops/orchestration/#dataops) and [MLOps](https://madewithml.com/courses/mlops/orchestration/#mlops) workflows to enable collaborative teams to develop efficiently. All the concepts mentioned here are covered in much more detail and tied to software engineering best practices for building ML systems. So be sure to check out the [lesson](https://madewithml.com/courses/mlops/feature-store/) if you haven't already."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "APsX_Wil_CNF"
      },
      "source": [
        "<div align=\"left\">\n",
        "<a target=\"_blank\" href=\"https://madewithml.com/courses/mlops/feature-store/\"><img src=\"https://img.shields.io/badge/📖 Read-lesson-9cf\"></a>&nbsp;\n",
        "<a href=\"https://github.com/GokuMohandas/feature-store/blob/main/feature_store.ipynb\" role=\"button\"><img src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n",
        "<a href=\"https://colab.research.google.com/github/GokuMohandas/feature-store/blob/main/feature_store.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BpAuTtoDabke"
      },
      "source": [
        "# Feast"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HzdPBQCYMHVz"
      },
      "source": [
        "> We suggest you execute this notebook on [Google colab](https://colab.research.google.com/github/GokuMohandas/feature-store/blob/main/feature_store.ipynb) since we leverage many of the preinstalled packages. However, as always, we can also run this locally but we'll have to manually install any missing packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iVrXrdE7abkf"
      },
      "outputs": [],
      "source": [
        "# Install Feast\n",
        "!pip install feast==0.10.5 PyYAML==5.3.1 -q"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OQfkZu-uspWu"
      },
      "outputs": [],
      "source": [
        "# Restart runtime/kernel\n",
        "import os\n",
        "os.kill(os.getpid(), 9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TB4wsuJLRqNF"
      },
      "source": [
        "> The code cell above will restart the kernel since we're installing a new version of PyYAML and we can just keep going and execute the cells below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DKXgm3natDns",
        "outputId": "54c9246d-428f-4521-e9e6-ecc111a7504e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "PyYAML==5.3.1\n"
          ]
        }
      ],
      "source": [
        "!pip freeze | grep PyYAML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qy2wr-edabkh"
      },
      "source": [
        "We're going to create a feature repository at the root of our project. [Feast](https://feast.dev/) will create a configuration file for us and we're going to add an additional [features.py](https://github.com/GokuMohandas/mlops-course/blob/main/features/features.py) file to define our features.\n",
        "\n",
        "> Traditionally, the feature repository would be it's own isolated repository that other services will use to read/write features from."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rPvfof-habkh",
        "outputId": "8beadec5-bd16-485f-8ee0-a06bcbce9508"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/telemetry\n",
            "\n",
            "Creating a new Feast repository in /content/features.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "mkdir -p stores/feature\n",
        "mkdir -p data\n",
        "feast init --minimal --template local features\n",
        "cd features\n",
        "touch features.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_4asVc2abki"
      },
      "source": [
        "```bash\n",
        "features/\n",
        "├── feature_store.yaml  - configuration\n",
        "└── features.py         - feature definitions\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jmdObcwlabki"
      },
      "source": [
        "We're going to configure the locations for our registry and online store (SQLite) in our `feature_store.yaml` file.\n",
        "\n",
        "<div class=\"ai-center-all\">\n",
        "    <img src=\"https://madewithml.com/static/images/mlops/feature_store/batch.png\" width=\"700\" alt=\"batch processing\">\n",
        "</div>\n",
        "\n",
        "- **registry**: contains information about our feature repository, such as data sources, feature views, etc. Since it's in a DB, instead of a Python file, it can very quickly be accessed in production.\n",
        "- **online store**: DB (SQLite for local) that stores the (latest) features for defined entities to be used for online inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aCpMbEcHU1Kx"
      },
      "source": [
        "If all definitions look valid, Feast will sync the metadata about Feast objects to the registry. The registry is a tiny database storing most of the same information you have in the feature repository. This step is necessary because the production feature serving infrastructure won't be able to access Python files in the feature repository at run time, but it will be able to efficiently and securely read the feature definitions from the registry.\n",
        "\n",
        "```yaml\n",
        "project: features\n",
        "registry: ../stores/feature/registry.db\n",
        "provider: local\n",
        "online_store:\n",
        "    path: ../stores/feature/online_store.db\n",
        "```\n",
        "\n",
        "> When we run Feast locally, the offline store is effectively represented via Pandas point-in-time joins. Whereas, in production, the offline store can be something more robust like [Google BigQuery](https://cloud.google.com/bigquery), [Amazon RedShift](https://aws.amazon.com/redshift/), etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_vN5Xqyft9QQ",
        "outputId": "14c49c53-8965-4983-9b86-73cbd21fd0d4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "project: features\n",
            "registry: ../stores/feature/registry.db\n",
            "provider: local\n",
            "online_store:\n",
            "    path: ../stores/feature/online_store.db\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "FEATURE_STORE_YAML=features/feature_store.yaml\n",
        "if test -f $FEATURE_STORE_YAML; then\n",
        "    rm $FEATURE_STORE_YAML\n",
        "fi\n",
        "touch $FEATURE_STORE_YAML\n",
        "echo \"project: features\" >> $FEATURE_STORE_YAML\n",
        "echo \"registry: ../stores/feature/registry.db\" >> $FEATURE_STORE_YAML\n",
        "echo \"provider: local\" >> $FEATURE_STORE_YAML\n",
        "echo \"online_store:\" >> $FEATURE_STORE_YAML\n",
        "echo \"    path: ../stores/feature/online_store.db\" >> $FEATURE_STORE_YAML\n",
        "cat $FEATURE_STORE_YAML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o54FiYInabkj"
      },
      "source": [
        "## Data ingestion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mkw43e-kabkk"
      },
      "source": [
        "The first step is to establish connections with our data sources (databases, data warehouse, etc.). Feast requires it's [data sources](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/data_source.py) to either come from a file ([Parquet](https://databricks.com/glossary/what-is-parquet)), data warehouse ([BigQuery](https://cloud.google.com/bigquery)) or data stream ([Kafka](https://kafka.apache.org/) / [Kinesis](https://aws.amazon.com/kinesis/)). We'll convert our generated features file from the DataOps pipeline (`features.json`) into a Parquet file, which is a column-major data format that allows fast feature retrieval and caching benefits (contrary to row-major data formats such as CSV where we have to traverse every single row to collect feature values)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r3crpclyabkk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import json\n",
        "import pandas as pd\n",
        "from pathlib import Path\n",
        "from urllib.request import urlopen"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "WABSPCTxabkl",
        "outputId": "f3544b04-8eed-4892-c0f6-8f19e9c556d4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-5a9ddc97-98ac-4231-880d-aded5cd5c2d4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>created_on</th>\n",
              "      <th>tag</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>2020-02-20 06:43:18</td>\n",
              "      <td>computer-vision</td>\n",
              "      <td>Comparison between YOLO and RCNN on real world...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>2020-02-20 06:47:21</td>\n",
              "      <td>computer-vision</td>\n",
              "      <td>Show, Infer &amp; Tell: Contextual Inference for C...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>2020-02-24 16:24:45</td>\n",
              "      <td>graph-learning</td>\n",
              "      <td>Awesome Graph Classification A collection of i...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>15</td>\n",
              "      <td>2020-02-28 23:55:26</td>\n",
              "      <td>reinforcement-learning</td>\n",
              "      <td>Awesome Monte Carlo Tree Search A curated list...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>19</td>\n",
              "      <td>2020-03-03 13:54:31</td>\n",
              "      <td>graph-learning</td>\n",
              "      <td>Diffusion to Vector Reference implementation o...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5a9ddc97-98ac-4231-880d-aded5cd5c2d4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5a9ddc97-98ac-4231-880d-aded5cd5c2d4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5a9ddc97-98ac-4231-880d-aded5cd5c2d4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   id           created_on                     tag  \\\n",
              "0   6  2020-02-20 06:43:18         computer-vision   \n",
              "1   7  2020-02-20 06:47:21         computer-vision   \n",
              "2   9  2020-02-24 16:24:45          graph-learning   \n",
              "3  15  2020-02-28 23:55:26  reinforcement-learning   \n",
              "4  19  2020-03-03 13:54:31          graph-learning   \n",
              "\n",
              "                                                text  \n",
              "0  Comparison between YOLO and RCNN on real world...  \n",
              "1  Show, Infer & Tell: Contextual Inference for C...  \n",
              "2  Awesome Graph Classification A collection of i...  \n",
              "3  Awesome Monte Carlo Tree Search A curated list...  \n",
              "4  Diffusion to Vector Reference implementation o...  "
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Load labeled projects\n",
        "projects = pd.read_csv(\"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/projects.csv\")\n",
        "tags = pd.read_csv(\"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/tags.csv\")\n",
        "df = pd.merge(projects, tags, on=\"id\")\n",
        "df[\"text\"] = df.title + \" \" + df.description\n",
        "df.drop([\"title\", \"description\"], axis=1, inplace=True)\n",
        "df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RoEuJX5Labkl"
      },
      "outputs": [],
      "source": [
        "# Format timestamp\n",
        "df.created_on = pd.to_datetime(df.created_on)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c5LBKipwabkl"
      },
      "outputs": [],
      "source": [
        "# Convert to parquet\n",
        "DATA_DIR = Path(os.getcwd(), \"data\")\n",
        "df.to_parquet(\n",
        "    Path(DATA_DIR, \"features.parquet\"),\n",
        "    compression=None,\n",
        "    allow_truncated_timestamps=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NCbcHUkaabkm"
      },
      "source": [
        "## Feature definitions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEu4rt7Pabkm"
      },
      "source": [
        "Now that we have our data source prepared, we can define our features for the feature store."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vsKSLOI9abkm"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "from pathlib import Path\n",
        "from feast import Entity, Feature, FeatureView, ValueType\n",
        "from feast.data_source import FileSource\n",
        "from google.protobuf.duration_pb2 import Duration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LSucFkEhabkn"
      },
      "source": [
        "The first step is to define the location of the features (FileSource in our case) and the timestamp column for each data point."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LFUpb2cKabkn"
      },
      "outputs": [],
      "source": [
        "# Read data\n",
        "START_TIME = \"2020-02-17\"\n",
        "project_details = FileSource(\n",
        "    path=str(Path(DATA_DIR, \"features.parquet\")),\n",
        "    event_timestamp_column=\"created_on\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uy9uRFFlabkn"
      },
      "source": [
        "Next, we need to define the main entity that each data point pertains to. In our case, each project has a unique ID with features such as text and tags."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2rofYTGhabkn"
      },
      "outputs": [],
      "source": [
        "# Define an entity\n",
        "project = Entity(\n",
        "    name=\"id\",\n",
        "    value_type=ValueType.INT64,\n",
        "    description=\"project id\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6vODPOU1abko"
      },
      "source": [
        "Finally, we're ready to create a [FeatureView](https://docs.feast.dev/concepts/feature-views) that loads specific features (`features`), of various [value types](https://api.docs.feast.dev/python/feast.html?highlight=valuetype#feast.value_type.ValueType), from a data source (`input`) for a specific period of time (`ttl`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r8LGZh8Gabko"
      },
      "outputs": [],
      "source": [
        "# Define a Feature View for each project\n",
        "project_details_view = FeatureView(\n",
        "    name=\"project_details\",\n",
        "    entities=[\"id\"],\n",
        "    ttl=Duration(\n",
        "        seconds=(datetime.today() - datetime.strptime(START_TIME, \"%Y-%m-%d\")).days * 24 * 60 * 60\n",
        "    ),\n",
        "    features=[\n",
        "        Feature(name=\"text\", dtype=ValueType.STRING),\n",
        "        Feature(name=\"tag\", dtype=ValueType.STRING),\n",
        "    ],\n",
        "    online=True,\n",
        "    input=project_details,\n",
        "    tags={},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p9gqkI4D1Uqw"
      },
      "source": [
        "We need to place all of this code into our `features.py` file we created earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TuSh6rbo1whJ"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "FEATURES_PY=features/features.py\n",
        "if test -f $FEATURES_PY; then\n",
        "    rm $FEATURES_PY\n",
        "fi\n",
        "touch $FEATURES_PY\n",
        "echo -e 'from datetime import datetime\n",
        "from pathlib import Path\n",
        "\n",
        "from feast import Entity, Feature, FeatureView, ValueType\n",
        "from feast.data_source import FileSource\n",
        "from google.protobuf.duration_pb2 import Duration\n",
        "\n",
        "\n",
        "# Read data\n",
        "START_TIME = \"2020-02-17\"\n",
        "project_details = FileSource(\n",
        "    path=\"/content/data/features.parquet\",\n",
        "    event_timestamp_column=\"created_on\",\n",
        ")\n",
        "\n",
        "# Define an entity for the project\n",
        "project = Entity(\n",
        "    name=\"id\",\n",
        "    value_type=ValueType.INT64,\n",
        "    description=\"project id\",\n",
        ")\n",
        "\n",
        "# Define a Feature View for each project\n",
        "# Can be used for fetching historical data and online serving\n",
        "project_details_view = FeatureView(\n",
        "    name=\"project_details\",\n",
        "    entities=[\"id\"],\n",
        "    ttl=Duration(\n",
        "        seconds=(datetime.today() - datetime.strptime(START_TIME, \"%Y-%m-%d\")).days * 24 * 60 * 60\n",
        "    ),\n",
        "    features=[\n",
        "        Feature(name=\"text\", dtype=ValueType.STRING),\n",
        "        Feature(name=\"tag\", dtype=ValueType.STRING),\n",
        "    ],\n",
        "    online=True,\n",
        "    input=project_details,\n",
        "    tags={},\n",
        ")' >> $FEATURES_PY"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yoMjvONzabkp"
      },
      "source": [
        "Once we've defined our feature views, we can `apply` it to push a version controlled definition of our features to the registry for fast access. It will also configure our registry and online stores that we've defined in our `feature_store.yaml`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bMGjTvsBabkp",
        "outputId": "421888d9-a89a-422f-9c7a-8b8113440664"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Registered entity id\n",
            "Registered feature view project_details\n",
            "Deploying infrastructure for project_details\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "cd features\n",
        "feast apply"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LVim9_Qtabkp"
      },
      "source": [
        "## Historical features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c740ja3Nabkp"
      },
      "source": [
        "Once we've registered our feature definition, along with the data source, entity definition, etc., we can use it to fetch historical features. This is done via joins using the provided timestamps using pandas for our local setup or BigQuery, Hive, etc. as an offline DB for production."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F0ZmpvYkabkp"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from feast import FeatureStore"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "TKipeeLYabkq",
        "outputId": "64a55788-8298-4b5a-e9c2-0e1547a44f1d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-d8138134-eddb-488f-aa8f-b8bd47f6b5a4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>event_timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>2022-06-23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>2022-06-23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>2022-06-23</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d8138134-eddb-488f-aa8f-b8bd47f6b5a4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d8138134-eddb-488f-aa8f-b8bd47f6b5a4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d8138134-eddb-488f-aa8f-b8bd47f6b5a4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   id event_timestamp\n",
              "0   6      2022-06-23\n",
              "1   7      2022-06-23\n",
              "2   9      2022-06-23"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Identify entities\n",
        "project_ids = df.id[0:3].to_list()\n",
        "now = datetime.now()\n",
        "timestamps = [datetime(now.year, now.month, now.day)]*len(project_ids)\n",
        "entity_df = pd.DataFrame.from_dict({\"id\": project_ids, \"event_timestamp\": timestamps})\n",
        "entity_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "4tDYiYNsabkq",
        "outputId": "ca93bc0d-0bb0-4f2a-ed30-e2074db47c82"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-71e652f3-9580-48b7-a6fb-f12bac77e8e2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>event_timestamp</th>\n",
              "      <th>id</th>\n",
              "      <th>project_details__text</th>\n",
              "      <th>project_details__tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2022-06-23 00:00:00+00:00</td>\n",
              "      <td>6</td>\n",
              "      <td>Comparison between YOLO and RCNN on real world...</td>\n",
              "      <td>computer-vision</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2022-06-23 00:00:00+00:00</td>\n",
              "      <td>7</td>\n",
              "      <td>Show, Infer &amp; Tell: Contextual Inference for C...</td>\n",
              "      <td>computer-vision</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2022-06-23 00:00:00+00:00</td>\n",
              "      <td>9</td>\n",
              "      <td>Awesome Graph Classification A collection of i...</td>\n",
              "      <td>graph-learning</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-71e652f3-9580-48b7-a6fb-f12bac77e8e2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-71e652f3-9580-48b7-a6fb-f12bac77e8e2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-71e652f3-9580-48b7-a6fb-f12bac77e8e2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "            event_timestamp  id  \\\n",
              "0 2022-06-23 00:00:00+00:00   6   \n",
              "1 2022-06-23 00:00:00+00:00   7   \n",
              "2 2022-06-23 00:00:00+00:00   9   \n",
              "\n",
              "                               project_details__text project_details__tag  \n",
              "0  Comparison between YOLO and RCNN on real world...      computer-vision  \n",
              "1  Show, Infer & Tell: Contextual Inference for C...      computer-vision  \n",
              "2  Awesome Graph Classification A collection of i...       graph-learning  "
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Get historical features\n",
        "store = FeatureStore(repo_path=\"features\")\n",
        "training_df = store.get_historical_features(\n",
        "    entity_df=entity_df,\n",
        "    feature_refs=[\"project_details:text\", \"project_details:tag\"],\n",
        ").to_df()\n",
        "training_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-4X6k32oabkr"
      },
      "source": [
        "## Materialize"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "erWBblHVabkr"
      },
      "source": [
        "For online inference, we want to retrieve features very quickly via our online store, as opposed to fetching them from slow joins. However, the features are not in our online store just yet, so we'll need to [materialize](https://docs.feast.dev/quickstart#4-materializing-features-to-the-online-store) them first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PDB2fkb4abkr",
        "outputId": "592ec800-9af1-4667-cac5-d39c169fc096"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Materializing \u001b[1m\u001b[32m1\u001b[0m feature views to \u001b[1m\u001b[32m2022-06-23 19:16:05+00:00\u001b[0m into the \u001b[1m\u001b[32msqlite\u001b[0m online store.\n",
            "\n",
            "\u001b[1m\u001b[32mproject_details\u001b[0m from \u001b[1m\u001b[32m2020-02-17 19:16:06+00:00\u001b[0m to \u001b[1m\u001b[32m2022-06-23 19:16:05+00:00\u001b[0m:\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\r  0%|                                                                       | 0/955 [00:00<?, ?it/s]\r100%|██████████████████████████████████████████████████████████| 955/955 [00:00<00:00, 10596.97it/s]\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "cd features\n",
        "CURRENT_TIME=$(date -u +\"%Y-%m-%dT%H:%M:%S\")\n",
        "feast materialize-incremental $CURRENT_TIME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ayf1KQOzabkr"
      },
      "source": [
        "This has moved the features for all of our projects into the online store since this was first time materializing to the online store. When we subsequently run the [`materialize-incremental`](https://docs.feast.dev/how-to-guides/load-data-into-the-online-store#2-b-materialize-incremental-alternative) command, Feast keeps track of previous materializations and so we'll only materialize the new data since the last attempt."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1x_d_Yz0abks"
      },
      "source": [
        "## Online features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A00FJZe0abks",
        "outputId": "d32b4613-520a-4cef-dbaf-c84d30a47ba9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'id': [6],\n",
              " 'project_details__tag': ['computer-vision'],\n",
              " 'project_details__text': ['Comparison between YOLO and RCNN on real world videos Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.']}"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Get online features\n",
        "store = FeatureStore(repo_path=\"features\")\n",
        "feature_vector = store.get_online_features(\n",
        "    feature_refs=[\"project_details:text\", \"project_details:tag\"],\n",
        "    entity_rows=[{\"id\": 6}],\n",
        ").to_dict()\n",
        "feature_vector"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VsVuYEj3_w80"
      },
      "source": [
        "# Over-engineering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ieGlw3CC8VfS"
      },
      "source": [
        "Not all machine learning tasks require a feature store. In fact, our use case is a perfect example of a test that *does not* benefit from a feature store. All of our inputs are stateless and there is no entity that has changing features over time. The real utility of a feature store shines when we need to have up-to-date features for an entity that we continually generate predictions for. For example, a user's behavior (clicks, purchases, etc.) on an e-commerce platform or the deliveries a food runner recently made today, etc."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "feature_store.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.9.1 64-bit ('3.9.1')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.1"
    },
    "vscode": {
      "interpreter": {
        "hash": "86a18fab98e45cbbb46dd97b38b13914cb97c5aa653edcee9fd16b53f5a0e542"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}