02-rag-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "15bbda72-2f2a-43d3-a3ed-b7f8c2f244a9",
   "metadata": {},
   "source": [
    "<img src=\"images/ragna-logo.png\" width=\"200px\" align=\"right\"/>\n",
    "\n",
    "# Basics of RAG-powered chat app\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "feba6e1b-e8ad-47c9-a9e4-48648fd476f1",
   "metadata": {},
   "source": [
    "## What is Retrieval-augmented generation (RAG)?\n",
    "\n",
    "LLMs are trained on vast, but static datasets. This means, while they can predict answers for several general questions like:\n",
    "\n",
    "<img src=\"images/chatgpt-what-is-scipy.png\" width=60% style=\"border:1px solid black;\"/>\n",
    "\n",
    "They can't answer or hallucinate answers for recent events or specific topics:\n",
    "\n",
    "<img src=\"images/chatgpt-when-is-scipy.png\" width=50% style=\"border:1px solid black;\"/>\n",
    "\n",
    "**Retrieval-augmented generation (RAG)** is a method to augment foundational LLMs with **contextual data (documents)**, to reduce hallucinations and get around the limited space available in an LLM prompts (around 3,000 words for ChatGPT 3.5).\n",
    "\n",
    "As a basic example, we can provide some relevant information from the SciPy website:\n",
    "\n",
    "<img src=\"images/chatgpt-with-context.png\" width=50% style=\"border:1px solid black;\"/>\n",
    "\n",
    "<img src=\"images/RAG-new.png\" width=70%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae71cfe5-6de7-4ed6-9ede-ac4ff606f21d",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "\n",
    "Breaking down document text into component units like words, sub-words, characters, etc.\n",
    "\n",
    "<img src=\"images/openai-tokenization.png\" width=60% style=\"border:1px solid black;\"/>\n",
    "\n",
    "\n",
    "### Embedding\n",
    "\n",
    "Storing the tokens logically, usually as vectors, where related tokens are closer to each other.\n",
    "\n",
    "<img src=\"images/embedding-projector.png\" width=80% style=\"border:1px solid black;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a1f1a5-cdef-4349-a69d-44cf0d81c487",
   "metadata": {},
   "source": [
    "## What is Ragna?\n",
    "\n",
    "Open source library for RAG **Orchestration** with a Python API, REST API, and web UI.\n",
    "\n",
    "It gives you a convenience tools to quickly build RAG workflows and applications, with any LLM or source storage you prefer.\n",
    "\n",
    "<img src=\"images/ragna-architecture.png\" width=80%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94d45748-b244-4104-bc9a-3b744b4d10c9",
   "metadata": {},
   "source": [
    "## Build a chat function with Ragna"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b649a183-e6d9-4f92-96b1-c89d962fb3ca",
   "metadata": {},
   "source": [
    "### Step 0: Setup requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ca1a36-78dd-454c-a92b-317c4b53792d",
   "metadata": {},
   "source": [
    "To use builtin LLMs like OpenAI, you will need API keys. For this tutorial, we have included a key for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "050b4d2b-a758-46a6-92d1-6b099af640ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "dotenv_path = Path.home() / Path(\"shared/scipy/rags-to-riches/.env\")\n",
    "assert load_dotenv(dotenv_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cec7c65-7741-46ed-8454-115f97424784",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Side note: Local setup instructions 💻\n",
    "\n",
    "On local computers, follow these step to get started with Ragna:\n",
    "\n",
    "1. Install Ragna: `pip install 'ragna[all]'`\n",
    "2. Run `ragna init` to create the `ragna.toml` config file with a guided CLI. \n",
    "3. [Get an OpenAI API key](https://platform.openai.com/api-keys) and set the relevant environment variable `export OPENAI_API_KEY=\"XXX\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b96f3695-9e43-4b1a-8d33-649f3c4449ec",
   "metadata": {},
   "source": [
    "### Step 1: Select relevant documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14b187fe-31f6-449a-bfec-bc3398dd43af",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Let's use 10k report filings for Ford, General Motors, and Tesla.\n",
    "\n",
    "💡 **Tip:** There are more documents in the `/files` directory that you can explore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9865ffb-8436-4856-a46a-6f4448ed7d37",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "documents = [\n",
    "    \"files/10k-report-ford.pdf\",\n",
    "    \"files/10k-report-gm.pdf\",\n",
    "    \"files/10k-report-tesla.pdf\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e778ed8-3b35-48e0-9a01-ac17937b05fa",
   "metadata": {},
   "source": [
    "### Step 2: Select assistants and source storage\n",
    "\n",
    "🔗 [Check the available assistants in the docs](https://ragna.chat/en/stable/generated/tutorials/gallery_python_api/#step-3-select-an-assistant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc39c85-65a1-4a58-97d7-ed6d5dcf7b1d",
   "metadata": {},
   "source": [
    "We are selecting OpenAI's GPT-3.5 and GPT-4 LLMs, and Chroma and LanceDB source storages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4d44b6-8d52-4e5e-a5e1-da566cea4407",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ragna import Rag\n",
    "from ragna.assistants import Gpt4, Gpt35Turbo16k\n",
    "from ragna.source_storages import Chroma, LanceDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de4c14ab-d1e6-4c0a-a060-7dbefe10de15",
   "metadata": {},
   "outputs": [],
   "source": [
    "rag = Rag()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85a4c3dc-2474-430c-be12-ec13b41458e0",
   "metadata": {},
   "source": [
    "### Step 3: Start chat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "168f993c-3f5b-4973-bbec-27e17293d9c3",
   "metadata": {},
   "source": [
    "Ragna is async by design.\n",
    "\n",
    "This setup involves building the vector embedding, and therefore will take longer for large document sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eb51db8-c455-416e-acd5-0d6888fde5ad",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "chat = rag.chat(\n",
    "    documents=documents,\n",
    "    source_storage=Chroma,\n",
    "    assistant=Gpt4,\n",
    ")\n",
    "\n",
    "await chat.prepare()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea0f42c-d1fc-4e0f-899f-83a1e156557d",
   "metadata": {},
   "source": [
    "### Step 4: Ask questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4cc2cf6-b464-46f4-91f5-c123be434035",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = await chat.answer(\"How did Ford, General Motors, and Tesla's perform financially 2023?\")\n",
    "print(f\"\\nLLM Response: \\n\\n{answer.content}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c1f4dab-4a4c-49af-92af-ab3fbd675169",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Let's check the sources used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d88afda0-e505-4994-b300-2f5470fbccf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer.sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e68c8887-b10d-4b21-bea1-fa950534387d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for idx, source in enumerate(answer.sources, 1):\n",
    "    print(f\"{idx}.: {source.document.name}, page(s) {source.location}\\n\")\n",
    "    print(source.content)\n",
    "    print(\"#\" * 80 + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3397e92f-d28b-4e6f-a21c-872bc2939221",
   "metadata": {},
   "source": [
    "#### Streaming answers\n",
    "\n",
    "Ragna allows you to stream the answers, one chunk at a time, just set `stream=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "465bd78f-b700-43ec-923c-13329e725f05",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "answer = await chat.answer(\"What are the primary risk factors for Ford?\", stream=True)\n",
    "\n",
    "print(f\"\\nLLM Response: \\n\\n\")\n",
    "      \n",
    "async for chunk in answer:\n",
    "    print(chunk, end= \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b31dcb-6aa5-45df-b326-4b8dd7f0993f",
   "metadata": {},
   "source": [
    "#### Reducing hallucinations (errors)\n",
    "\n",
    "Ragna tries to ensure only the sources are used for answering questions.\n",
    "\n",
    "🔗 [For reference, see Ragna source code on GitHub highlighting the prompt.](https://github.com/Quansight/ragna/blob/3cef0f7da1f2ed90e5d0618bcad82f824d00dc5a/ragna/assistants/_openai.py#L25-L26)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35313b3f-913d-4393-b310-b1b866d4921e",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = await chat.answer(\"How much did Tesla earn in 2024?\")\n",
    "print(f\"\\nLLM Response: \\n\\n{answer.content}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cc06093-81ea-4a5f-a197-db25336698a5",
   "metadata": {},
   "source": [
    "### Advanced configuration\n",
    "\n",
    "`Rag().chat()` takes the following keyword arguments to help you optimize the quality of answers:\n",
    "\n",
    "* `chunk_size` - Size of each chunk (sections of the document that contain context) to use. Default is 500.\n",
    "* `chunk_overlap` - Size of the overlap with previous and next chunk for retrieving additional context for future prompts. Default is 250.\n",
    "* `num_tokens` - Maximum number of context tokens, and in turn the number of document chunks, pulled out of the vector database. Default is 1024.\n",
    "\n",
    "You can also set these configurations in the web app (which we will see later)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7205c89-ee6e-4b51-a18d-0bb454cd7c39",
   "metadata": {},
   "source": [
    "### Your turn ✨ \n",
    "\n",
    "1. Create a new chat with larger `chunk_size` and `num_tokens`\n",
    "2. Ask about the overall carbon emissions for each company"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03afd5a9-f217-4cd7-86ec-6e71afdb229e",
   "metadata": {},
   "outputs": [],
   "source": [
    "chat = rag.chat(\n",
    "    documents=...,\n",
    "    source_storage=...,\n",
    "    assistant=...,\n",
    "    chunck_size=...,\n",
    "    num_tokens=...,\n",
    ")\n",
    "\n",
    "await chat.prepare()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf1e107-72db-40bd-87af-a9ae6ceb566d",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = await chat.answer(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fedd4e7-95a5-4cc3-a97f-a9ed7ea562b6",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "_❗️ **Warning:** Make sure to stop the Jupyter Kernel (in the JupyterLab Menu Bar, click on \"Kernel\" -> \"Shut down Kernel\") before proceeding to prevent the \"insufficient VRAM\" error._\n",
    "\n",
    "<br>\n",
    "\n",
    "**✨ Next: [Use Local LLM with Ragna](03-RAG-local-llm.ipynb) →**\n",
    "\n",
    "<br>\n",
    "\n",
    "💬 _Wish to continue discussions after the tutorial? Contact the presenters: [@pavithraes](https://github.com/pavithraes), [@dharhas](https://github.com/dharhas), [@ahuang11](https://github.com/ahuang11)_\n",
    "\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-rags-to-riches",
   "language": "python",
   "name": "conda-env-scipy-scipy-rags-to-riches-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}