03-Quering-AOpenAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d59d527f-1100-45ff-b051-5f7c9029d94d",
   "metadata": {},
   "source": [
    "# Queries with and without Azure OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb9a9444-dc90-4fc3-aea7-8ee918301aba",
   "metadata": {},
   "source": [
    "Now that we have our Search Engine loaded **from two different data sources in two diferent indexes**, we are going to try some example queries and then use Azure OpenAI service to see if we can get even better results.\n",
    "\n",
    "The idea is that a user can ask a question about Computer Science (first datasource/index) or about Covid (second datasource/index), and the engine will respond accordingly.\n",
    "This **Multi-Index** demo, mimics the scenario where a company loads multiple type of documents of different types and about completly different topics and the search engine must respond with the most relevant results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f6c7e3-9037-4b1e-ae17-1deaa27b9c08",
   "metadata": {},
   "source": [
    "## Set up variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8e50b404-a061-49e7-a3c7-c6eabc98ff0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib\n",
    "import requests\n",
    "import random\n",
    "from collections import OrderedDict\n",
    "from IPython.display import display, HTML\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.llms import AzureOpenAI\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "\n",
    "from common.prompts import COMBINE_QUESTION_PROMPT, COMBINE_PROMPT\n",
    "from common.utils import model_tokens_limit, num_tokens_from_docs\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "# Demo Datasource Blob Storage. Change if using your own data\n",
    "DATASOURCE_SAS_TOKEN = \"?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "2f2c22f8-79ab-405c-95e8-77a1978e53bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9297d29b-1f61-4dce-858e-bf4272172dba",
   "metadata": {},
   "source": [
    "## Multi-Index Search queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5a46e2d3-298a-4708-83de-9e108b1a117a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Index that we are going to query (from Notebook 01 and 02)\n",
    "index1_name = \"cogsrch-index-files\"\n",
    "index2_name = \"cogsrch-index-csv\"\n",
    "indexes = [index1_name, index2_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c62ebb2-d7be-4bfb-b1ba-4db86c11839a",
   "metadata": {},
   "source": [
    "Try questions that you think might be answered or addressed in computer science papers in 2020-2021 or that can be addressed by medical publications about COVID in 2020-201. Try comparing the results with the open version of ChatGPT.<br>\n",
    "The idea is that the answers using Azure OpenAI only looks at the information contained on these publications.\n",
    "\n",
    "**Example Questions you can ask**:\n",
    "- What is CLP?\n",
    "- How Markov chains work?\n",
    "- What are some examples of reinforcement learning?\n",
    "- What are the main risk factors for Covid-19?\n",
    "- What medicine reduces inflamation in the lungs?\n",
    "- Why Covid doesn't affect kids that much compared to adults?\n",
    "- Does chloroquine really works against covid?\n",
    "- tell me Use cases where I can use deep learning to solve it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b9b53c14-19bd-451f-aa43-7ad27ccfeead",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"What is CLP?\" "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d925eb-7f9c-429e-a62a-4c37d7702caf",
   "metadata": {},
   "source": [
    "### Search on both indexes individually and aggragate results\n",
    "\n",
    "**Note**: In order to standarize the indexes we are setting 4 mandatory fields to be present on each index: id, title, content, pages, language. These fields must be present in each index so that each document can be treated the same along the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "faf2e30f-e71f-4533-ab52-27d048b80a89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://cog-search-4njhjc5ogtyfm.search.windows.net/indexes/cogsrch-index-files/docs?api-version=2021-04-30-Preview&search=What is CLP?&select=*&$top=10&queryLanguage=en-us&queryType=semantic&semanticConfiguration=my-semantic-config&$count=true&speller=lexicon&answers=extractive|count-3&captions=extractive|highlight-false\n",
      "200\n",
      "Results Found: 9789, Results Returned: 10\n",
      "https://cog-search-4njhjc5ogtyfm.search.windows.net/indexes/cogsrch-index-csv/docs?api-version=2021-04-30-Preview&search=What is CLP?&select=*&$top=10&queryLanguage=en-us&queryType=semantic&semanticConfiguration=my-semantic-config&$count=true&speller=lexicon&answers=extractive|count-3&captions=extractive|highlight-false\n",
      "200\n",
      "Results Found: 22435, Results Returned: 10\n"
     ]
    }
   ],
   "source": [
    "agg_search_results = []\n",
    "\n",
    "for index in indexes:\n",
    "    url = os.environ['AZURE_SEARCH_ENDPOINT'] + '/indexes/'+ index + '/docs'\n",
    "    url += '?api-version={}'.format(os.environ['AZURE_SEARCH_API_VERSION'])\n",
    "    url += '&search={}'.format(QUESTION)\n",
    "    url += '&select=*'\n",
    "    url += '&$top=10'  # You can change this to anything you need/want\n",
    "    url += '&queryLanguage=en-us'\n",
    "    url += '&queryType=semantic'\n",
    "    url += '&semanticConfiguration=my-semantic-config'\n",
    "    url += '&$count=true'\n",
    "    url += '&speller=lexicon'\n",
    "    url += '&answers=extractive|count-3'\n",
    "    url += '&captions=extractive|highlight-false'\n",
    "\n",
    "    resp = requests.get(url, headers=headers)\n",
    "    print(url)\n",
    "    print(resp.status_code)\n",
    "\n",
    "    search_results = resp.json()\n",
    "    agg_search_results.append(search_results)\n",
    "    print(\"Results Found: {}, Results Returned: {}\".format(search_results['@odata.count'], len(search_results['value'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fd0fe5-4ee0-42e2-a920-72b93a407389",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Display the top results (from both searches) based on the score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9e938337-602d-4b61-8141-b8c92a5d91da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Top Answers</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5>Answer - score: 0.99</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Constraint Logic Programming (CLP) is an emerging software technology with a growing number of applications. Data flow in constraint programs is not explicit, and for this reason the concepts of slice and the slicing techniques of imperative languages are not directly applicable. This paper formulates declarative notions of slice suitable for CLP."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5>Answer - score: 0.97</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The key notions of CLP are those of an algebra and an associated constraint solver over a class of constraints, namely a set of first order formulas including the always satisfiable constraint true, the un- satisfiable constraint false, and closed under variable renaming, conjunction and existential quantification."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5>Answer - score: 0.92</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A solution is an instantiation of the variables of X which satisfies all the constraints in R.  2.1 Constraint Logic Programming  Constraint logic programming (CLP) [7] is an extension of logic programming where some of the predicate and function symbols have a fixed interpretation over some subdomain (e.g. finite trees or real numbers)."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h4>Top Results</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0701/0701082v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">0701082v1.pdf</a> - score: 1.94</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The key notions of CLP are those of an algebra and an associated constraint solver over a class of constraints, namely a set of first order formulas including the always satisfiable constraint true, the un- satisfiable constraint false, and closed under variable renaming, conjunction and existential quantification."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0408/0408056v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">0408056v1.pdf</a> - score: 1.62</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "CLP(FD) languages have been suc-  cessfully used for solving a variety of industrial and academic problems. However,  in some constraint problems, where domain elements need to be acquired, it may  not be wise to perform the acquisition of the whole domains of variables before the  beginning of the constraint propagation process."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0508/0508108v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">0508108v1.pdf</a> - score: 1.62</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "CLP(FD) is an extension of logic programming. In CLP(FD) pro- grams, logical variables are assigned a domain and relations between vari- ables are described with constraints. A solution to a CLP(FD) program is a valuation of every variable in its own domain such that no constraint is falsified."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0402/0402019v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">0402019v1.pdf</a> - score: 1.57</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "CLP combines the advantages of two declarative  paradigms: logic programming (Prolog) and constraint solving. In logic program-  ming, problems are stated in a declarative way using rules to define relations (predi-  cates). Problems are solved using chronological backtrack search to explore choices."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0506/0506005v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">0506005v1.pdf</a> - score: 1.47</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A CLP(FD) system provides primitives for  accessing and updating attribute values. A CLP(FD) system provides equality (=), disequality (6=), and inequality con-  straints. In addition, a CLP(FD) system also provides some other constraints such  as global constraints."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0012/0012014v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0012014v1  [cs.SE]  18 Dec 2000</a> - score: 1.45</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Constraint Logic Programming (CLP) is an emerging software technology with a growing number of applications. Data flow in constraint programs is not explicit, and for this reason the concepts of slice and the slicing techniques of imperative languages are not directly applicable. This paper formulates declarative notions of slice suitable for CLP."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">Lock-down logistics in Consultation-Liaison Psychiatry.</a> - score: 1.34</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "We describe the adaptation of services to allow flexible and practical responses to the COVID-19 public health crisis by four Consultation-Liaison psychiatry (CLP) services; Galway University Hospital, Beaumont Hospital, University Hospital Waterford and St Vincent's University Hospital CLP Services."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">Effect of Thymoquinone on Acute Kidney Injury Induced by Sepsis in BALB/c Mice.</a> - score: 1.3</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "CLP was performed after 2 weeks of TQ gavage. After 48 h, we measured the histopathological alterations in the kidney tissue and the serum levels of creatinine (CRE) and blood urea nitrogen (BUN)."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0003/0003026v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0003026v1  [cs.LO]  8 Mar 2000</a> - score: 1.28</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A proof procedure for CLP is defined as an extension of standard resolution. A state is defined as a pair 〈← a, A || C〉 of a goal and a set of constraints. At each step of the computation, some literal a is selected from the current goal according to some selection function."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0106/0106008v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0106008v1  [cs.PL]  7 Jun 2001</a> - score: 1.26</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A distinguishing feature of CLP(Intervals) is that it decomposes equations, or other composite expressions, into primitive constraints. These primitive con- straints are the relational versions of the building blocks of expressions, which are admissible functions."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0008/0008036v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0008036v1  [cs.CL]  30 Aug 2000</a> - score: 1.26</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "To this end,  we discuss the formal basics of Constraint Logic Programming (CLP), which is used here  to provide an operational treatment of various declarative constraint-based grammars. This  is done by an embedding of the logical description languages of such grammars into a CLP  scheme, yielding Constraint Logic Grammars (CLGs)."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">Consultation-Liaison Psychiatry During COVID-19 Lockdown: A Retrospective Chart Review.</a> - score: 1.24</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A retrospective chart review of consultation-liaison psychiatry (CLP) case records was conducted for one month before and after the start of lockdown. Patients seen during lockdown were relatively younger; t = 1.8, p = 0.074."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0011/0011030v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0011030v1  [cs.AI]  21 Nov 2000</a> - score: 1.2</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A solution is an instantiation of the variables of X which satisfies all the constraints in R.  2.1 Constraint Logic Programming  Constraint logic programming (CLP) [7] is an extension of logic programming where some of the predicate and function symbols have a fixed interpretation over some subdomain (e.g. finite trees or real numbers)."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(HTML('<h4>Top Answers</h4>'))\n",
    "\n",
    "for search_results in agg_search_results:\n",
    "    for result in search_results['@search.answers']:\n",
    "        if result['score'] > 0.5: # Show answers that are at least 50% of the max possible score=1\n",
    "            display(HTML('<h5>' + 'Answer - score: ' + str(round(result['score'],2)) + '</h5>'))\n",
    "            display(HTML(result['text']))\n",
    "            \n",
    "print(\"\\n\\n\")\n",
    "display(HTML('<h4>Top Results</h4>'))\n",
    "\n",
    "content = dict()\n",
    "ordered_content = OrderedDict()\n",
    "\n",
    "\n",
    "for search_results in agg_search_results:\n",
    "    for result in search_results['value']:\n",
    "        if result['@search.rerankerScore'] > 1: # Filter results that are at least 25% of the max possible score=4\n",
    "            content[result['id']]={\n",
    "                                    \"title\": result['title'],\n",
    "                                    \"chunks\": result['pages'],\n",
    "                                    \"language\": result['language'], \n",
    "                                    \"caption\": result['@search.captions'][0]['text'],\n",
    "                                    \"score\": result['@search.rerankerScore'],\n",
    "                                    \"name\": result['metadata_storage_name'], \n",
    "                                    \"location\": result['metadata_storage_path']                  \n",
    "                                }\n",
    "    \n",
    "#After results have been filtered we will Sort and add them as an Ordered list\\n\",\n",
    "for id in sorted(content, key= lambda x: content[x][\"score\"], reverse=True):\n",
    "    ordered_content[id] = content[id]\n",
    "    url = ordered_content[id]['location'] + DATASOURCE_SAS_TOKEN\n",
    "    title = str(ordered_content[id]['title']) if (ordered_content[id]['title']) else ordered_content[id]['name']\n",
    "    score = str(round(ordered_content[id]['score'],2))\n",
    "    display(HTML('<h5><a href=\"'+ url + '\">' + title + '</a> - score: '+ score + '</h5>'))\n",
    "    display(HTML(ordered_content[id]['caption']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a6d3e6-afb2-4fa7-96d3-69bc2373ded5",
   "metadata": {},
   "source": [
    "## Comments on Query results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e02227-6a92-4944-86f8-6c1e38d90fe4",
   "metadata": {},
   "source": [
    "As seen above the semantic search feature of Azure Cognitive Search service is good. It gives us some answers and also the top results with the corresponding file and the paragraph where the answers is possible located.\n",
    "\n",
    "Let's see if we can make this better with Azure OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df3e6d4-9a09-4b0f-b328-238738ccfaec",
   "metadata": {},
   "source": [
    "# Using Azure OpenAI\n",
    "\n",
    "To use OpenAI to get a better answer to our question, the thought process is: let's send the the documents of the search result to the GPT model and let it understand the document's content and provide a better response.\n",
    "\n",
    "We will use a genius library call **LangChain** that wraps a lot of boiler plate code.\n",
    "Langchain is one library that does a lot of the prompt engineering for us under the hood, for more information see [here](https://python.langchain.com/en/latest/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e7c720e-ece1-45ad-9d01-2dfd15c182bb",
   "metadata": {},
   "source": [
    "## A gentle intro to chaining LLMs and prompt engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bcd7028-5a6c-4296-8c85-4f420d408d69",
   "metadata": {},
   "source": [
    "Chains are what you get by connecting one or more large language models (LLMs) in a logical way. (Chains can be built of entities other than LLMs but for now, let’s stick with this definition for simplicity).\n",
    "\n",
    "Azure OpenAI is a type of LLM (provider) that you can use but there are others like Cohere, Huggingface, etc.\n",
    "\n",
    "Chains can be simple (i.e. Generic) or specialized (i.e. Utility).\n",
    "\n",
    "* Generic — A single LLM is the simplest chain. It takes an input prompt and the name of the LLM and then uses the LLM for text generation (i.e. output for the prompt).\n",
    "\n",
    "Here’s an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "eea62a7d-7e0e-4a93-a89c-20c96560c665",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_BASE\"] = os.environ[\"AZURE_OPENAI_ENDPOINT\"]\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.environ[\"AZURE_OPENAI_API_KEY\"]\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "13df9247-e784-4e04-9475-55e672efea47",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create our LLM model\n",
    "# Make sure you have the deployment named \"gpt-35-turbo\" for the model \"gpt-35-turbo (0301)\". \n",
    "# Use \"gpt-4\" if you have it available.\n",
    "MODEL = \"gpt-35-turbo\" # options: gpt-35-turbo, gpt-4, gpt-4-32k\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0, max_tokens=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7b0520b9-83b2-49fd-ad84-624cb0f15ce1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question: \"What is CLP?\". Give your response in French\n"
     ]
    }
   ],
   "source": [
    "# Now we create a simple prompt template\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\", \"language\"],\n",
    "    template='Answer the following question: \"{question}\". Give your response in {language}',\n",
    ")\n",
    "\n",
    "print(prompt.format(question=QUESTION, language=\"French\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "dcc7dae3-6b88-4ea6-be43-b178ebc559dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'What is CLP?',\n",
       " 'language': 'French',\n",
       " 'text': \"CLP signifie Classification, Étiquetage et Emballage des substances et des mélanges. Il s'agit d'un système de classification et d'étiquetage harmonisé au niveau européen pour les produits chimiques dangereux. Le CLP vise à protéger la santé humaine et l'environnement en fournissant des informations claires et précises sur les dangers des produits chimiques.\"}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And finnaly we create our first generic chain\n",
    "chain_chat = LLMChain(llm=llm, prompt=prompt)\n",
    "chain_chat({\"question\": QUESTION, \"language\": \"French\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ed014c-0c6b-448c-b995-fe7970b92ad5",
   "metadata": {},
   "source": [
    "Great!!, now you know how to create a simple prompt and use a chain in order to answer a general question using ChatGPT knowledge!. \n",
    "\n",
    "It is important to note that we rarely use generic chains as standalone chains. More often they are used as building blocks for Utility chains (as we will see next). Also important to notice is that we are NOT using our documents or the result of the Azure Search yet, just the knowledge of ChatGPT on the data it was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c48038-b1af-4228-8ffb-720e554fd3b2",
   "metadata": {
    "tags": []
   },
   "source": [
    "**The second type of Chains are Utility:**\n",
    "\n",
    "* Utility — These are specialized chains, comprised of many LLMs to help solve a specific task. For example, LangChain supports some end-to-end chains (such as [QA_WITH_SOURCES](https://python.langchain.com/en/latest/modules/chains/index_examples/qa_with_sources.html) for QnA Doc retrieval, Summarization, etc) and some specific ones (such as GraphQnAChain for creating, querying, and saving graphs). \n",
    "\n",
    "We will look at one specific chain called **qa_with_sources** in this workshop for digging deeper and solve our use case of enhancing the results of Azure Cognitive Search."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0454ddb-44d8-4fa9-929a-5e5563dd28f8",
   "metadata": {},
   "source": [
    "\n",
    "But before dealing with the utility chain needed, we need to deal first with this problem: **the content of the search result files is or can be very lengthy, more than the allowed tokens allowed by the GPT Azure OpenAI models**. So what we need to do is: split in chunks, vectorize those chunks and do a vector semantic search to get the top chunks in order to provide the best and not too lenghy context to the LLM.\n",
    "\n",
    "Notice that **the documents chunks are already done in Azure Search**. *ordered_content* dictionary (created a few cells above) contains the pages (chunks) of each document. So we don't really need to chunk them again, but we still need to make sure that we can be as fast as possible and that we are below the max allowed input token limits of our selected OpenAI model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f7b41d2-65b0-4058-8a46-c76cf6960720",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of chunks: 148\n"
     ]
    }
   ],
   "source": [
    "# Iterate over each of the results chunks and create a LangChain Document class to use further in the pipeline\n",
    "docs = []\n",
    "for key,value in ordered_content.items():\n",
    "    for page in value[\"chunks\"]:\n",
    "        docs.append(Document(page_content=page, metadata={\"source\": value[\"location\"]}))\n",
    "        \n",
    "print(\"Number of chunks:\",len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "345d35f6-b7c8-4fda-a9e2-94a7da16a18e",
   "metadata": {},
   "source": [
    "We need now to calculate the number of tokens for all the chunks combined to decide what to do:\n",
    "1) Should we embed to vectors and do cosine similarity because there is too much data to fit on the prompt as context?\n",
    "2) What happens if the amount of chunks is too big? how do we keep the response time on-check?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "62bd5169-f273-4c66-a91b-6de990dad244",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Custom token limit for gpt-35-turbo : 3000\n",
      "Combined docs tokens count: 184936\n"
     ]
    }
   ],
   "source": [
    "# Calculate number of tokens of our docs\n",
    "if(len(docs)>0):\n",
    "    tokens_limit = model_tokens_limit(MODEL) # this is a custom function we created in common/utils.py\n",
    "    num_tokens = num_tokens_from_docs(docs) # this is a custom function we created in common/utils.py\n",
    "    print(\"Custom token limit for\", MODEL, \":\", tokens_limit)\n",
    "    print(\"Combined docs tokens count:\",num_tokens)\n",
    "        \n",
    "else:\n",
    "    print(\"NO RESULTS FROM AZURE SEARCH\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5403dee-a4c4-420c-9819-68151d973695",
   "metadata": {},
   "source": [
    "Now, depending of the amount of chunks/pages returned from the search result, which is very related to the size of the documents returned, we need to make some decisions to keep the speed of the response from Azure OpenAI at reasonable levels.\n",
    "\n",
    "**The logic is**: if there is less than X chunks (of 5000 chars each) to vectorize, then we use OpenAI models in all of them, which currently don't offer batch processing (but it will soon), but if there is more than X chunks we have to trim the number of chunks to the first X, otherwise large documents can take minutes to vectorize.\n",
    "\n",
    "**Note**, this is a temporary solution. Once Vector Capabilities in Azure Cognitive Search is in Public Preview, then we can vectorize once, store, then retrieve them again when needed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a03f1f10-32b0-4c1e-8a0e-eee1b1d29ce7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Number of chunks: 148\n",
      "Truncated Number of chunks: 100\n",
      "Token count after similarity search: 3836\n",
      "Chain Type selected: map_reduce\n",
      "CPU times: user 742 ms, sys: 12 ms, total: 754 ms\n",
      "Wall time: 6.02 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "chunks_limit = 100 # This is the limit of how many chunks are we willing to take in order to keep the response fairly quick\n",
    "\n",
    "if num_tokens > tokens_limit:\n",
    "    # Select the Embedder model\n",
    "    embedder = OpenAIEmbeddings(deployment=\"text-embedding-ada-002\", chunk_size=1) \n",
    "    print(\"Initial Number of chunks:\",len(docs))\n",
    "    if len(docs) > chunks_limit:\n",
    "        docs = docs[:chunks_limit]\n",
    "        print(\"Truncated Number of chunks:\",len(docs))\n",
    "    \n",
    "    # Create our in-memory vector database index from the chunks given by Azure Search.\n",
    "    # We are using FAISS. https://ai.facebook.com/tools/faiss/\n",
    "    db = FAISS.from_documents(docs, embedder)\n",
    "    top_docs = db.similarity_search(QUESTION, k=4)  # Return the top 4 documents\n",
    "    \n",
    "    # Now we need to recalculate the tokens count of the top results from similarity vector search\n",
    "    # in order to select the chain type: stuff (all chunks in one prompt) or \n",
    "    # map_reduce (multiple calls to the LLM to summarize/reduce the chunks and then combine them)\n",
    "    \n",
    "    num_tokens = num_tokens_from_docs(top_docs)\n",
    "    print(\"Token count after similarity search:\", num_tokens)\n",
    "    chain_type = \"map_reduce\" if num_tokens > tokens_limit else \"stuff\"\n",
    "    \n",
    "else:\n",
    "    # if total tokens is less than our limit, we don't need to vectorize and do similarity search\n",
    "    top_docs = docs\n",
    "    chain_type = \"stuff\"\n",
    "    \n",
    "print(\"Chain Type selected:\", chain_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17247488-7d14-4178-9add-31eb1afcbcbe",
   "metadata": {},
   "source": [
    "At this point we already have the top most similar chunks (in order of relevance) in **top_docs**\n",
    "\n",
    "Now we need Azure OpenAI GPT model to understand these top chunks and provide us an answer to the question."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8c6ad1-82b8-4fd8-80a3-0276b81d7231",
   "metadata": {},
   "source": [
    "For this task, we need to come back to the Utility Chain: **qa_with_sources** that we mentioned before. See [HERE](https://python.langchain.com/en/latest/modules/chains/index_examples/qa_with_sources.html) for reference.\n",
    "\n",
    "We created our own custom prompts so we can add translation to a specified language. But, for more information on the different types of prompts for this utility chain please see [HERE](https://github.com/hwchase17/langchain/tree/master/langchain/chains/question_answering)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3ab16c86-9863-4001-89af-6819c6f3240a",
   "metadata": {},
   "outputs": [],
   "source": [
    "if chain_type == \"stuff\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       prompt=COMBINE_PROMPT)\n",
    "elif chain_type == \"map_reduce\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       question_prompt=COMBINE_QUESTION_PROMPT,\n",
    "                                       combine_prompt=COMBINE_PROMPT,\n",
    "                                       return_intermediate_steps=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "28926219-74c2-4538-8493-129463ac40a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below line if you want to check our custom COMBINE_PROMPT\n",
    "# print(chain.combine_document_chain.llm_chain.prompt.template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a1e619b8-1dcf-431b-8aad-f1696a09c2ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 26.6 ms, sys: 647 µs, total: 27.2 ms\n",
      "Wall time: 20.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Try with other language as well\n",
    "response = chain({\"input_documents\": top_docs, \"question\": QUESTION, \"language\": \"English\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8cddb1cb-a4a0-4e2f-9f0c-4216b0f232b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = response['output_text']\n",
    "\n",
    "split_regex = re.compile(f\"sources?:?\\\\W*\", re.IGNORECASE)\n",
    "answer_text = split_regex.split(answer)[0]\n",
    "sources_list = split_regex.split(answer)[1].replace(\" \",\"\").split(\",\")\n",
    "\n",
    "sources_html = '<br><u>Sources</u>: '\n",
    "for index, value in enumerate(sources_list):\n",
    "    url = value + DATASOURCE_SAS_TOKEN\n",
    "    sources_html +='<sup><a href=\"'+ url + '\">[' + str(index+1) + ']</a></sup>'\n",
    "\n",
    "answer = answer_text + sources_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "681fe1de-e37c-4355-accc-6dd15b6a310a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "CLP stands for different things depending on the context: Consultation-Liaison psychiatry, Constraint Logic Programming, or both. \n",
       "<br><u>Sources</u>: <sup><a href=\"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup><sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0012/0012014v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[2]</a></sup><sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0408/0408056v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[3]</a></sup><sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0008/0008036v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[4]</a></sup>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(HTML(answer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "11345374-6420-4b36-b061-795d2a804c85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment if you want to inspect the results from map_reduce chain type, each top similar chunk summary (k=4 by default)\n",
    "\n",
    "# if chain_type == \"map_reduce\":\n",
    "#     for step in response['intermediate_steps']:\n",
    "#         display(HTML(\"<b>Chunk Summary:</b> \" + step))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f347373a-a5be-473d-b64e-0f6b6dbcd0e0",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### This answer is way better than taking just the result from Azure Cognitive Search. So the summary is:\n",
    "- Azure Cognitive Search give us the top results (context)\n",
    "- Azure OpenAI takes these results and understand the content and uses it as context to give the best answer\n",
    "- Best of two worlds!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc6e2fe-1c34-4952-99ad-14940f022379",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We just added a smart layer on top of Azure Cognitive Search. This is the backend for a GPT Smart Search Engine.\n",
    "\n",
    "However, we are missing something: **How to have a conversation with this engine?**\n",
    "\n",
    "On the next Notebook, we are going to understand the concept of **memory**. This is necessary in order to have a chatbot that can establish a conversation with the user. Without memory, there is no real conversation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}