Pinecone_Practice.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21ix0-ioBXC_"
      },
      "source": [
        "# Install and import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "8a9u4SFc4qnS"
      },
      "outputs": [],
      "source": [
        "!pip install -qU pinecone-client openai datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "uB4rP0PG7Qsf"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDWUgrM0BdU4"
      },
      "source": [
        "# Read FAQ data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "5beOQjwB7Uc-"
      },
      "outputs": [],
      "source": [
        "faq = pd.read_csv(\"FAQ.csv\")\n",
        "questions = [s.rstrip() for s in faq['Question']] # list of all the questions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "id": "O5p_XnQHC8Y_",
        "outputId": "5b1838e0-df1e-455e-88bf-13c9562340c6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Question</th>\n",
              "      <th>Answer</th>\n",
              "      <th>URL</th>\n",
              "      <th>Label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Does UT Dallas provide services for students w...</td>\n",
              "      <td>Yes, accommodations and services are provided ...</td>\n",
              "      <td>https://accessability.utdallas.edu/student-acc...</td>\n",
              "      <td>Student Accessibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Is there a separate admissions procedure for s...</td>\n",
              "      <td>No. The admissions process and criteria are th...</td>\n",
              "      <td>https://Accessibility.utdallas.edu/student-acc...</td>\n",
              "      <td>Student Accessibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Should I send my disability documentation with...</td>\n",
              "      <td>No. Documentation should be submitted to ARC o...</td>\n",
              "      <td>https://Accessibility.utdallas.edu/student-acc...</td>\n",
              "      <td>Student Accessibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>What documentation is required to receive disa...</td>\n",
              "      <td>Students requesting services must provide curr...</td>\n",
              "      <td>https://Accessibility.utdallas.edu/student-acc...</td>\n",
              "      <td>Student Accessibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Is there a deadline for submitting disability ...</td>\n",
              "      <td>To ensure that accommodations will be in place...</td>\n",
              "      <td>https://Accessibility.utdallas.edu/student-acc...</td>\n",
              "      <td>Student Accessibility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>When should I expect to receive my refund?</td>\n",
              "      <td>Please see the Bursar’s Office Refunds page fo...</td>\n",
              "      <td>https://finaid.utdallas.edu/receiving-aid/faq/</td>\n",
              "      <td>Scholarship, Cost &amp; AID</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>152</th>\n",
              "      <td>Where can I get information on the loans that ...</td>\n",
              "      <td>All of your federal loan information can be ob...</td>\n",
              "      <td>https://finaid.utdallas.edu/receiving-aid/faq/</td>\n",
              "      <td>Scholarship, Cost &amp; AID</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>153</th>\n",
              "      <td>I need to purchase books, but my financial aid...</td>\n",
              "      <td>If your financial aid application is complete ...</td>\n",
              "      <td>https://finaid.utdallas.edu/receiving-aid/faq/</td>\n",
              "      <td>Scholarship, Cost &amp; AID</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>154</th>\n",
              "      <td>What is considered a special circumstance?</td>\n",
              "      <td>A special circumstance is an event that causes...</td>\n",
              "      <td>https://finaid.utdallas.edu/receiving-aid/faq/</td>\n",
              "      <td>Scholarship, Cost &amp; AID</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>155</th>\n",
              "      <td>What types of situations qualify for a Depende...</td>\n",
              "      <td>Please see the cover sheet of the Dependency O...</td>\n",
              "      <td>https://finaid.utdallas.edu/receiving-aid/faq/</td>\n",
              "      <td>Scholarship, Cost &amp; AID</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>156 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              Question   \n",
              "0    Does UT Dallas provide services for students w...  \\\n",
              "1    Is there a separate admissions procedure for s...   \n",
              "2    Should I send my disability documentation with...   \n",
              "3    What documentation is required to receive disa...   \n",
              "4    Is there a deadline for submitting disability ...   \n",
              "..                                                 ...   \n",
              "151         When should I expect to receive my refund?   \n",
              "152  Where can I get information on the loans that ...   \n",
              "153  I need to purchase books, but my financial aid...   \n",
              "154         What is considered a special circumstance?   \n",
              "155  What types of situations qualify for a Depende...   \n",
              "\n",
              "                                                Answer   \n",
              "0    Yes, accommodations and services are provided ...  \\\n",
              "1    No. The admissions process and criteria are th...   \n",
              "2    No. Documentation should be submitted to ARC o...   \n",
              "3    Students requesting services must provide curr...   \n",
              "4    To ensure that accommodations will be in place...   \n",
              "..                                                 ...   \n",
              "151  Please see the Bursar’s Office Refunds page fo...   \n",
              "152  All of your federal loan information can be ob...   \n",
              "153  If your financial aid application is complete ...   \n",
              "154  A special circumstance is an event that causes...   \n",
              "155  Please see the cover sheet of the Dependency O...   \n",
              "\n",
              "                                                   URL   \n",
              "0    https://accessability.utdallas.edu/student-acc...  \\\n",
              "1    https://Accessibility.utdallas.edu/student-acc...   \n",
              "2    https://Accessibility.utdallas.edu/student-acc...   \n",
              "3    https://Accessibility.utdallas.edu/student-acc...   \n",
              "4    https://Accessibility.utdallas.edu/student-acc...   \n",
              "..                                                 ...   \n",
              "151     https://finaid.utdallas.edu/receiving-aid/faq/   \n",
              "152     https://finaid.utdallas.edu/receiving-aid/faq/   \n",
              "153     https://finaid.utdallas.edu/receiving-aid/faq/   \n",
              "154     https://finaid.utdallas.edu/receiving-aid/faq/   \n",
              "155     https://finaid.utdallas.edu/receiving-aid/faq/   \n",
              "\n",
              "                       Label  \n",
              "0      Student Accessibility  \n",
              "1      Student Accessibility  \n",
              "2      Student Accessibility  \n",
              "3      Student Accessibility  \n",
              "4      Student Accessibility  \n",
              "..                       ...  \n",
              "151  Scholarship, Cost & AID  \n",
              "152  Scholarship, Cost & AID  \n",
              "153  Scholarship, Cost & AID  \n",
              "154  Scholarship, Cost & AID  \n",
              "155  Scholarship, Cost & AID  \n",
              "\n",
              "[156 rows x 4 columns]"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "faq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DwTq_n-OBkQl"
      },
      "source": [
        "# Setup openai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "1FsjbyAY5_xO"
      },
      "outputs": [],
      "source": [
        "import openai\n",
        "import os\n",
        "\n",
        "openai.api_key = \"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\"\n",
        "# get API key from top-right dropdown on OpenAI website"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_piBzzyrBuix"
      },
      "source": [
        "# Turn questions into vectors (embedding)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "EdXWaJw16inl"
      },
      "outputs": [],
      "source": [
        "MODEL = \"text-embedding-ada-002\"\n",
        "\n",
        "res = openai.Embedding.create(\n",
        "    input=questions, engine=MODEL\n",
        ")\n",
        "embeds = [record['embedding'] for record in res['data']] # embeds is list of vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wg5meaO5B7gS"
      },
      "source": [
        "# Setup pinecone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "dfQygyul8Byc"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/dylankaneshiro/opt/anaconda3/envs/ais/lib/python3.8/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
            "  from tqdm.autonotebook import tqdm\n"
          ]
        }
      ],
      "source": [
        "import pinecone\n",
        "\n",
        "index_name = 'semantic-search-openai'\n",
        "\n",
        "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
        "pinecone.init(\n",
        "    api_key=\"d5580837-1cbd-46ba-8cd5-d1f9e3cff833\",\n",
        "    environment=\"eu-west1-gcp\"  # find next to api key in console\n",
        ")\n",
        "# check if 'openai' index already exists (only create index if not)\n",
        "if index_name not in pinecone.list_indexes():\n",
        "    pinecone.create_index(index_name, dimension=len(embeds[0]))\n",
        "# connect to index\n",
        "index = pinecone.Index(index_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-ZeB1-hCHBb"
      },
      "source": [
        "# Put questions into pinecone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "Cekwsm0R8pRB"
      },
      "outputs": [],
      "source": [
        "to_upsert = zip([str(n) for n in range(len(embeds))], embeds, [{'text': line, 'faq':True} for line in faq['Question']])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X9R0EWGN-Zbc",
        "outputId": "07939f87-8180-4111-8377-422438db3d30"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'upserted_count': 156}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index.upsert(vectors=list(to_upsert))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Haystack"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Set up Haystack"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [],
      "source": [
        "from haystack.pipelines import Pipeline\n",
        "from haystack.nodes import Crawler, PreProcessor\n",
        "from haystack.document_stores import InMemoryDocumentStore\n",
        "\n",
        "document_store = InMemoryDocumentStore(use_bm25=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [],
      "source": [
        "URLS = [\n",
        "    \"https://engineering.utdallas.edu/academics/undergraduate-majors/undergrad-advising/advising-faq/\",\n",
        "    \"https://engineering.utdallas.edu/engineering/academics/undergraduate-majors/undergrad-advising/graduate-courses/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-biomedical-engineering/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-computer-engineering/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-computer-science/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-electrical-engineering/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-mechanical-engineering/\",\n",
        "    \"https://www.utdallas.edu/fact-sheets/ecs/bs-software-engineering/\",\n",
        "    \"https://fye.utdallas.edu/orientation/\",\n",
        "    \"https://fye.utdallas.edu/orientation/faq/\",\n",
        "    \"https://oue.utdallas.edu/aleks-exam/\",\n",
        "    \"https://www.utdallas.edu/academics/academic-resources/\",\n",
        "    \"https://www.utdallas.edu/costs-scholarships-aid/faq/\",\n",
        "    \"https://accessability.utdallas.edu/student-accommodations/frequently-asked-questions/\",\n",
        "    \"https://housing.utdallas.edu/resources/frequently-asked-questions/\",\n",
        "    \"https://services.utdallas.edu/transit/parkfaq/\",\n",
        "    \"https://registrar.utdallas.edu/faq/\",\n",
        "    \"https://services.utdallas.edu/bookstore/faq/\",\n",
        "    \"https://isso.utdallas.edu/joining-ut-dallas/i20-receive/\",\n",
        "    \"https://finaid.utdallas.edu/receiving-aid/faq/\"\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/home\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/biomedical-engineering\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/computer-engineering\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/computer-science\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/data-science\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/electrical-engineering\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/mechanical-engineering\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/software-engineering\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/certificates\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/programs/ecs/minors\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/curriculum\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/curriculum/core-curriculum\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/curriculum/honors-programs\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/curriculum/other-degree-requirements\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid/excessive-hours\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid/tuition-refund\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid/other-fees\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid/financial-aid\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/tuition-and-financial-aid/types-of-financial-aid\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/academic\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/graduate-courses/fasttrack\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/honors-college\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/degree-plans\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/course-policies\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/graduate-courses\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/graduation\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/registration\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/military\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/student-travel\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/education-abroad-policies\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/policies/addendum\",\n",
        "    \"https://catalog.utdallas.edu/2022/undergraduate/resources\",\n",
        "    \"http://cs.utdallas.edu/admissions/admissions-faq/\",\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [],
      "source": [
        "from haystack.pipelines import Pipeline\n",
        "from haystack.nodes import Crawler, PreProcessor\n",
        "from haystack.document_stores import PineconeDocumentStore\n",
        "\n",
        "def run_crawler(depth = 0):\n",
        "    crawler = Crawler(\n",
        "        urls=URLS,   # Websites to crawl\n",
        "        crawler_depth=depth,    # How many links to follow\n",
        "        #output_dir=\"crawled_files\",  # The directory to store the crawled files, not very important, we don't use the files in this example\n",
        "        filter_urls=[\"utd\"],  # Only follow links that match this regex\n",
        "    )\n",
        "\n",
        "    preprocessor = PreProcessor(\n",
        "        clean_empty_lines=True,\n",
        "        clean_whitespace=True,\n",
        "        clean_header_footer=True,\n",
        "        split_by=\"word\",\n",
        "        split_length=300,\n",
        "        split_respect_sentence_boundary=False,\n",
        "        split_overlap=40\n",
        "    )\n",
        "\n",
        "    indexing_pipeline = Pipeline()\n",
        "    indexing_pipeline.add_node(component=crawler, name=\"crawler\", inputs=['File'])\n",
        "    indexing_pipeline.add_node(component=preprocessor, name=\"preprocessor\", inputs=['crawler'])\n",
        "\n",
        "    return indexing_pipeline.run()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 184,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Preprocessing: 100%|██████████| 41/41 [00:00<00:00, 705.80docs/s]\n",
            "Writing Documents: 1024it [00:05, 185.91it/s]                         \n"
          ]
        }
      ],
      "source": [
        "data = run_crawler()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 185,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1008"
            ]
          },
          "execution_count": 185,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(data['documents'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "data"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Embed text from website"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [],
      "source": [
        "MODEL = \"text-embedding-ada-002\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [],
      "source": [
        "import openai\n",
        "\n",
        "def get_embeddings_webtext(data):\n",
        "    webtext = [i.content for i in data]\n",
        "    res = openai.Embedding.create(input=webtext, engine=MODEL)\n",
        "    embeds = [record['embedding'] for record in res['data']] # embeds is list of vectors\n",
        "    return embeds"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 160,
      "metadata": {},
      "outputs": [],
      "source": [
        "embeds = get_embeddings_webtext(data['documents'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 161,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1009"
            ]
          },
          "execution_count": 161,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(embeds)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Put webtext into Pinecone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 166,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'\\n\\nSkip to Main Content.\\nSkip to Main Navigation.\\nSkip to Main Footer.\\n\\nThe University of Texas at Dallas\\n\\nErik Jonsson School\\nof Engineering and\\nComputer Science\\n\\nAbout\\nAcademics\\nInnovation\\nEngage\\nNews\\n\\nLeadership\\nCorporate Partners\\nAt a Glance\\n\\nFacilities\\nContact\\nCareers\\n\\nUndergraduate Majors\\nGraduate Education\\nProfessional Education\\n\\nResearch\\nResearch Awards\\nResearch Centers\\nResearch Institutes\\n\\nStudents\\nAlumni\\nCorporate Engagement\\nSocial Media Directory\\nGiving\\n\\nStories\\nPulsar\\nMedia Spotlight\\nNewsletter\\nMagazine\\n\\nCampus Map\\nGiving\\nContact\\n\\nMain Navigation\\n\\nAbout\\n\\nLeadership\\nCorporate Partners\\nAt a Glance\\n\\nFacilities\\nContact\\nCareers\\nJ. Erik Jonsson\\n\\nAcademics\\n\\nUndergraduate Majors\\nGraduate Education\\nCertificates\\n\\nInnovation\\n\\nResearch\\nResearch Awards\\nResearch Centers\\nResearch Institutes\\n\\nEngage\\n\\nStudents\\nAlumni\\nCorporate Engagement\\nSocial Media Directory\\nGiving\\n\\nNews\\n\\nStories\\nPulsar\\nMedia Spotlight\\nNewsletter\\nMagazine\\n\\nContent\\n\\nAdvising FAQ\\n\\nJonsson School Home\\n>  Academics\\n>  Undergraduate Majors\\n>  Office of Undergraduate Advising\\n>  Advising FAQ\\n\\n$(document).ready(function(){\\n$(\".accordion\").accordion({\\nactive: false,\\ncollapsible: true,\\nheightStyle: \\'content\\'\\n});\\n});\\n\\nAdvising Links\\n\\nAdvising\\nAdvisors\\nProspective Students\\nDegree Resources\\nGraduate Level Courses\\nAdvising FAQ\\nStudent Resources\\n\\nAdvising FAQ\\n\\nSection Links\\n\\nRegistration and Course Selection\\n\\nAcademic Standing\\n\\nTransferring Credits\\n\\nPolicies\\n\\nFast Track\\n\\nFirst Time Freshman\\n\\nMajor/Minor Requirements\\n\\nGraduation\\n\\nInternational and Study Abroad\\n\\nRegistration and Course Selection\\nBack To Top\\n\\nHow long until I graduate?+\\nThere are many'"
            ]
          },
          "execution_count": 166,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data['documents'][0].content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 167,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'https://engineering.utdallas.edu/academics/undergraduate-majors/undergrad-advising/advising-faq/'"
            ]
          },
          "execution_count": 167,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data['documents'][0].meta['url']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pinecone\n",
        "import uuid\n",
        "\n",
        "def upsert_data(index: pinecone.Index, embeds, data):\n",
        "    to_upsert = zip([str(uuid.uuid4()) for n in range(len(embeds))], embeds, [{'text': doc.content, 'faq':False, 'source':doc.meta['url']} for doc in data])\n",
        "    print(to_upsert)\n",
        "    index.upsert(vectors=list(to_upsert))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [],
      "source": [
        "def upsert_crawled_data():\n",
        "    results = run_crawler()\n",
        "    results = results['documents']\n",
        "    embeds = get_embeddings_webtext(results)\n",
        "    index = pinecone.Index('chatbot-faq')\n",
        "    for i in range(0, len(embeds), 32):\n",
        "        stopping_point = min(i+32, len(embeds))\n",
        "        upsert_data(index, embeds[i:stopping_point], results[i:stopping_point])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Preprocessing:   0%|          | 0/53 [00:00<?, ?docs/s]Document 70bf4d8118fe014453d22175dea30496 is 10407 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.\n",
            "Document 691b596936d3ad88dbb8b48e282e5b79 is 10399 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.\n",
            "Document 59693c5e8cd0462669328dbcf10107a7 is 10287 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.\n",
            "Document c253733f1a5f8e9599a3022516a27402 is 10406 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.\n",
            "Document db69005ff42ef7a22a2a5e3ef2a52007 is 10552 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.\n",
            "Preprocessing: 100%|██████████| 53/53 [00:00<00:00, 744.05docs/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<zip object at 0x7fe2d0fc8300>\n",
            "<zip object at 0x7fe2c7f6eb80>\n",
            "<zip object at 0x7fe2d0fce580>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n",
            "<zip object at 0x7fe2a261b640>\n"
          ]
        }
      ],
      "source": [
        "upsert_crawled_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gi2wGf7AHA1v"
      },
      "source": [
        "# Get most similar questions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DKURIeL9CMdA"
      },
      "source": [
        "## FUNCTION: input = question, output = 5 most similar questions in pinecone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "_ZSQfk-L-mXz"
      },
      "outputs": [],
      "source": [
        "# returns a list of questions\n",
        "def mostSimilarQs(query):\n",
        "  xq = openai.Embedding.create(input=query, engine=MODEL)['data'][0]['embedding']\n",
        "  res = index.query([xq], top_k=5, include_metadata=True)\n",
        "  mostSimilarQs = [[res['matches'][i]['metadata']['text'], res['matches'][i]['metadata']['faq']] for i, _ in enumerate(res['matches'])]\n",
        "  return mostSimilarQs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [],
      "source": [
        "def QtoA (question):\n",
        "  row = faq[faq['Question'] == question].values[0]\n",
        "  return row[1] + \"\\n\" + row[2]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [],
      "source": [
        "def QtoChunk(l):\n",
        "    if l[1]:\n",
        "        return \"Q: \" + l[0] + \"\\nA: \" + QtoA(l[0])\n",
        "    else:\n",
        "        return l[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AITDECx4_9m6",
        "outputId": "3fa3a630-87de-42ab-8a0b-3ebbf842f7b9"
      },
      "outputs": [],
      "source": [
        "def chunks(question):\n",
        "    similar = mostSimilarQs(question)\n",
        "    return [QtoChunk(l) for l in similar]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frQJHZVPChCP"
      },
      "source": [
        "# Build full query"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vqaLmo2jHJxM"
      },
      "source": [
        "## FUNCTION: input = question, output = full query"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "5fPrKr4qEApm"
      },
      "outputs": [],
      "source": [
        "def QtoQuery(question):\n",
        "  c = chunks(question)\n",
        "  query =  \"\"\"Please pretend you are an enthusiastic, knowledgable, friendly, and helpful counselor for the University of Texas at Dallas. \n",
        "  I will be asking you questions about the University of Texas at Dallas and you must respond with clear, concise answers. \n",
        "  Provide examples and explain procedures step by step in an ordered list. Please provide all information that would be helpful.\n",
        "\n",
        "  {c1}\n",
        "\n",
        "  {c2}\n",
        "\n",
        "  {c3}\n",
        "\n",
        "  {c4}\n",
        "\n",
        "  {c5}\n",
        "\n",
        "  Q: {q}\n",
        "  A: \"\"\".format(c1 = c[0], c2 = c[1], c3 = c[2], c4 = c[3], c5 = c[4],q=question)\n",
        "  return query\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rrybef6HSyD"
      },
      "source": [
        "# Example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NPff1x9nFbp-",
        "outputId": "eeacf2cd-a679-41b9-b317-b9fba56b6383"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Please pretend you are an enthusiastic, knowledgable, friendly, and helpful counselor for the University of Texas at Dallas. \n",
            "  I will be asking you questions about the University of Texas at Dallas and you must respond with clear, concise answers. \n",
            "  Provide examples and explain procedures step by step in an ordered list. Please provide all information that would be helpful.\n",
            "\n",
            "  Q: How do I audit a course?\n",
            "A: For more information on how to audit a course, please go to audit registration.\n",
            "https://registrar.utdallas.edu/faq/\n",
            "\n",
            "  Q: How do I reserve classroom space on campus?\n",
            "A: For information on reserving classroom space on campus, please see Special Event Room Reservations.\n",
            "https://registrar.utdallas.edu/faq/\n",
            "\n",
            "  Q: I would like to change my major. What do I need to do?\n",
            "A: Eligible students wishing to change their major/plan/program should review the policy listed in the Academic Catalog.\n",
            "\n",
            "For undergraduate students: catalog.utdallas.edu/current/undergraduate/policies/degree-plans.\n",
            "For graduate students: catalog.utdallas.edu/current/graduate/policies/policy#change-of-program.\n",
            "https://registrar.utdallas.edu/faq/\n",
            "\n",
            "  Q: My class has attendance requirements because students are graded while I observe what they are doing and they must access specialized lab equipment that cannot be moved. How can I allow remote learning? \n",
            "A: In rare cases, it can be a fundamental alteration of your class, which is not required even by accommodation. However, to make this decision it is essential that faculty consult with their department/program heads and the Accessibility Resource Center before denying an accommodation. \n",
            "\n",
            "\n",
            "https://Accessibility.utdallas.edu/student-accommodations/frequently-asked-questions/\n",
            "\n",
            "  Q: I am going to finish my degree in less time than listed on my I-20. Can you issue an I-20 with a shorter program length?\n",
            "A: The Department of Homeland Security requires UT Dallas to establish a program length for the I-20s that the ISSO issues. The program length is established by the university, and your I-20 reflects the standard program length. Please note that students at UT Dallas are able to pace their own program completion. The program length is an average. Some students may finish a program in less time, while others may take longer. However, the I-20 must reflect the average length of time.\n",
            "https://isso.utdallas.edu/joining-ut-dallas/i20-receive/#faq\n",
            "\n",
            "  Q: How can I accelerate my curriculum?\n",
            "  A: \n"
          ]
        }
      ],
      "source": [
        "print(QtoQuery(\"How can I accelerate my curriculum?\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "ZyLg6-9Samra"
      },
      "outputs": [],
      "source": [
        "def talk(question):\n",
        "  response = openai.Completion.create(\n",
        "    model=\"text-davinci-003\",\n",
        "    prompt=QtoQuery(question),\n",
        "    temperature=0.3,\n",
        "    max_tokens=256,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0\n",
        "  )\n",
        "  print(response['choices'][0]['text'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hFyU1GTVbGCa",
        "outputId": "d3252ac4-1b6d-4ef1-d401-eb1a00950922"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Fast Track is a program that allows undergraduate students to take graduate-level courses for undergraduate credit. To get started with Fast Track, you must first meet the eligibility requirements. Eligibility requirements include:\n",
            "\n",
            "1. Be a current UT Dallas undergraduate student in good academic standing\n",
            "2. Have a minimum UT Dallas GPA of 3.0\n",
            "3. Have completed at least 60 hours of undergraduate coursework\n",
            "4. Have the approval of your academic advisor\n",
            "\n",
            "Once you have met the eligibility requirements, you can download the Fast Track application and submit it to your academic advisor during the appropriate application timeframe. Your academic advisor will review the application and provide you with feedback. Once approved, you can register for the graduate-level courses.\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "print(talk(\"How do I get started with fast track?\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Please pretend you are an enthusiastic, knowledgable, friendly, and helpful counselor for the University of Texas at Dallas. \n",
            "  I will be asking you questions about the University of Texas at Dallas and you must respond with clear, concise answers. \n",
            "  Provide examples and explain procedures step by step in an ordered list. Please provide all information that would be helpful.\n",
            "\n",
            "  To determine a possible graduation timeline, consider the following:\n",
            "1.\tHow many hours of degree requirements remain?\n",
            "Using your degree plan, you can add up the hours of all classes that remain to figure out this number.\n",
            "2.\tHow many hours will you take each semester? Remember that 18 is the maximum for fall/spring, and 15 is the max for summer.\n",
            "3.\tCan I complete all necessary prerequisites by taking ___ hours each semester? \n",
            "\n",
            "  Q: I am going to finish my degree in less time than listed on my I-20. Can you issue an I-20 with a shorter program length?\n",
            "A: The Department of Homeland Security requires UT Dallas to establish a program length for the I-20s that the ISSO issues. The program length is established by the university, and your I-20 reflects the standard program length. Please note that students at UT Dallas are able to pace their own program completion. The program length is an average. Some students may finish a program in less time, while others may take longer. However, the I-20 must reflect the average length of time.\n",
            "https://isso.utdallas.edu/joining-ut-dallas/i20-receive/#faq\n",
            "\n",
            "  \n",
            "\n",
            "Skip to Main Content.\n",
            "Skip to Main Navigation.\n",
            "Skip to Main Footer.\n",
            "\n",
            "The University of Texas at Dallas\n",
            "\n",
            "Erik Jonsson School\n",
            "of Engineering and\n",
            "Computer Science\n",
            "\n",
            "About\n",
            "Academics\n",
            "Innovation\n",
            "Engage\n",
            "News\n",
            "\n",
            "Leadership\n",
            "Corporate Partners\n",
            "At a Glance\n",
            "\n",
            "Facilities\n",
            "Contact\n",
            "Careers\n",
            "\n",
            "Undergraduate Majors\n",
            "Graduate Education\n",
            "Professional Education\n",
            "\n",
            "Research\n",
            "Research Awards\n",
            "Research Centers\n",
            "Research Institutes\n",
            "\n",
            "Students\n",
            "Alumni\n",
            "Corporate Engagement\n",
            "Social Media Directory\n",
            "Giving\n",
            "\n",
            "Stories\n",
            "Pulsar\n",
            "Media Spotlight\n",
            "Newsletter\n",
            "Magazine\n",
            "\n",
            "Campus Map\n",
            "Giving\n",
            "Contact\n",
            "\n",
            "Main Navigation\n",
            "\n",
            "About\n",
            "\n",
            "Leadership\n",
            "Corporate Partners\n",
            "At a Glance\n",
            "\n",
            "Facilities\n",
            "Contact\n",
            "Careers\n",
            "J. Erik Jonsson\n",
            "\n",
            "Academics\n",
            "\n",
            "Undergraduate Majors\n",
            "Graduate Education\n",
            "Certificates\n",
            "\n",
            "Innovation\n",
            "\n",
            "Research\n",
            "Research Awards\n",
            "Research Centers\n",
            "Research Institutes\n",
            "\n",
            "Engage\n",
            "\n",
            "Students\n",
            "Alumni\n",
            "Corporate Engagement\n",
            "Social Media Directory\n",
            "Giving\n",
            "\n",
            "News\n",
            "\n",
            "Stories\n",
            "Pulsar\n",
            "Media Spotlight\n",
            "Newsletter\n",
            "Magazine\n",
            "\n",
            "Content\n",
            "\n",
            "Advising FAQ\n",
            "\n",
            "Jonsson School Home\n",
            ">  Academics\n",
            ">  Undergraduate Majors\n",
            ">  Office of Undergraduate Advising\n",
            ">  Advising FAQ\n",
            "\n",
            "$(document).ready(function(){\n",
            "$(\".accordion\").accordion({\n",
            "active: false,\n",
            "collapsible: true,\n",
            "heightStyle: 'content'\n",
            "});\n",
            "});\n",
            "\n",
            "Advising Links\n",
            "\n",
            "Advising\n",
            "Advisors\n",
            "Prospective Students\n",
            "Degree Resources\n",
            "Graduate Level Courses\n",
            "Advising FAQ\n",
            "Student Resources\n",
            "\n",
            "Advising FAQ\n",
            "\n",
            "Section Links\n",
            "\n",
            "Registration and Course Selection\n",
            "\n",
            "Academic Standing\n",
            "\n",
            "Transferring Credits\n",
            "\n",
            "Policies\n",
            "\n",
            "Fast Track\n",
            "\n",
            "First Time Freshman\n",
            "\n",
            "Major/Minor Requirements\n",
            "\n",
            "Graduation\n",
            "\n",
            "International and Study Abroad\n",
            "\n",
            "Registration and Course Selection\n",
            "Back To Top\n",
            "\n",
            "How long until I graduate?+\n",
            "There are many variables that impact your graduation timeline such as the number of classes you take each semester, enrollment in summer classes, having to repeat classes, missing prerequisites, etc. To determine a possible graduation timeline, consider the following:\n",
            "1. How many hours of degree requirements remain?\n",
            "Using your degree plan, you can add up the hours of all classes that remain to figure out this number. 2. How many hours will you take each semester? Remember that 18 is the maximum for fall/spring, and 15 is the max for summer.\n",
            "3. Can I complete all necessary prerequisites by taking ___ hours each semester? Using your flowchart, you can make a tentative plan for classes that you will take each semester until graduation to make sure you do not miss any prerequisites.\n",
            "4. Did I miss anything? Share your plans with your academic advisor to verify that you have made a reasonable plan to meet your graduation goal.\n",
            "What classes should I take next semester?+\n",
            "\n",
            "Your flowchart will help you to identify classes which you are eligible to take each semester.\n",
            "1. Download or print your degree flowchart.\n",
            "2. Pull up your course history in ORION Student Center > My Academics.\n",
            "3. On your flowchart, cross off all the classes you have\n",
            "\n",
            "  They can assist with health career options, recommended courses/majors, assist with applications, and more.\n",
            "To get started, download the Pre-Health Starter Kit. After reviewing this information, contact your assigned advisor to notify them of your intention to pursue pre-med: your advisor will help you build schedules that fit both your major and pre-med classes.\n",
            "\n",
            "Graduation\n",
            "Back To Top\n",
            "\n",
            "How long until I graduate?+\n",
            "\n",
            "There are many variables that impact your graduation timeline such as the number of classes you take each semester, enrollment in summer classes, having to repeat classes, missing prerequisites, etc. \n",
            "\n",
            "  Q: Will my grades come in the mail?\n",
            "A: No, semester and midterm grades can be viewed in your UTD Student Center.\n",
            "https://registrar.utdallas.edu/faq/\n",
            "\n",
            "  Q: How long until I graduate?\n",
            "  A: \n"
          ]
        }
      ],
      "source": [
        "print(QtoQuery(\"How long until I graduate?\"))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}