examples/bring_your_own_pdf/upload_explore_download_pdfs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8ee29891",
   "metadata": {},
   "source": [
    "# Bring Your Own PDFs\n",
    "\n",
    "In this example we combine the document conversion capabilities of Deep Search with its data query capabilities.\n",
    "From the Deep Search Workspace, we create a new project data index which can host our own PDF documents.\n",
    "Once the upload is completed, we will be able to query the documents, similar to the public data which we\n",
    "explored in the [Data query quick start example](../data_query_quick_start/). \n",
    "In the last steps of the example, we additionally export the converted documents as JSON files.\n",
    "\n",
    "\n",
    "Sections\n",
    "1. [Create data index and upload data](#Create-data-index-and-upload-data)\n",
    "2. [Query your data](#Query-your-data)\n",
    "3. [Download your data](#Download-your-data)\n",
    "4. Custom upload settings\n",
    "    1. [Enable OCR](#Enable-OCR)\n",
    "    2. [Enable raw PDF cells](#Enable-raw-PDF-cells)\n",
    "\n",
    "\n",
    "### Access required\n",
    "\n",
    "The content of this notebook requires access to Deep Search capabilities which are not\n",
    "available on the public access system.\n",
    "\n",
    "[Contact us](https://ds4sd.github.io) if you are interested in exploring\n",
    "these Deep Search capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc9a9f1",
   "metadata": {},
   "source": [
    "### Set notebook parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "13560ae0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:46:40.803078Z",
     "iopub.status.busy": "2024-05-27T10:46:40.802251Z",
     "iopub.status.idle": "2024-05-27T10:46:43.090868Z",
     "shell.execute_reply": "2024-05-27T10:46:43.090020Z",
     "shell.execute_reply.started": "2024-05-27T10:46:40.803044Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The example will be executed on the Deep Search instance ds-internal\n"
     ]
    }
   ],
   "source": [
    "from dsnotebooks.settings import ProjectNotebookSettings\n",
    "from pathlib import Path\n",
    "\n",
    "# notebook settings auto-loaded from .env / env vars\n",
    "notebook_settings = ProjectNotebookSettings()\n",
    "\n",
    "PROFILE_NAME = notebook_settings.profile  # profile to use\n",
    "PROJ_KEY = notebook_settings.proj_key  # project to use\n",
    "INDEX_NAME = notebook_settings.new_idx_name  # index to create\n",
    "CLEANUP = notebook_settings.cleanup  # whether to clean up\n",
    "INPUT_FILES_FOLDER = Path(\"../../data/samples/\")\n",
    "INPUT_OCR_FILE = Path(\"../../data/scanned-samples/2206.00785-7.png\")\n",
    "\n",
    "############\n",
    "_GARBAGE_COLLECTOR = (\n",
    "    []\n",
    ")  # list of resources to clean up at the end of the execution (if CLEANUP=True)\n",
    "\n",
    "print(f\"The example will be executed on the Deep Search instance {PROFILE_NAME}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0531c9c4",
   "metadata": {},
   "source": [
    "### Import example dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1ea3cd26",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:46:43.092408Z",
     "iopub.status.busy": "2024-05-27T10:46:43.091887Z",
     "iopub.status.idle": "2024-05-27T10:46:43.806261Z",
     "shell.execute_reply": "2024-05-27T10:46:43.805255Z",
     "shell.execute_reply.started": "2024-05-27T10:46:43.092381Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import standard dependenices\n",
    "from copy import deepcopy\n",
    "import json\n",
    "from tqdm.notebook import tqdm\n",
    "import pandas as pd\n",
    "import tempfile\n",
    "\n",
    "# IPython utilities\n",
    "from IPython.display import display, Markdown, HTML\n",
    "\n",
    "# Import the deepsearch-toolkit\n",
    "import deepsearch as ds\n",
    "from deepsearch.documents.core.export import export_to_markdown\n",
    "from deepsearch.cps.queries import DataQuery\n",
    "from deepsearch.cps.data_indices import utils as data_indices_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8be733",
   "metadata": {},
   "source": [
    "### Connect to Deep Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fae4dee3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:46:43.811749Z",
     "iopub.status.busy": "2024-05-27T10:46:43.810806Z",
     "iopub.status.idle": "2024-05-27T10:46:44.335265Z",
     "shell.execute_reply": "2024-05-27T10:46:44.334519Z",
     "shell.execute_reply.started": "2024-05-27T10:46:43.811713Z"
    }
   },
   "outputs": [],
   "source": [
    "api = ds.CpsApi.from_env(profile_name=PROFILE_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70fe44f7-89f8-45ec-b6a7-8367d171a1ba",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18957147",
   "metadata": {},
   "source": [
    "### Create data index and upload data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f5496626",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:46:44.336782Z",
     "iopub.status.busy": "2024-05-27T10:46:44.336292Z",
     "iopub.status.idle": "2024-05-27T10:46:46.230863Z",
     "shell.execute_reply": "2024-05-27T10:46:46.229796Z",
     "shell.execute_reply.started": "2024-05-27T10:46:44.336753Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a new data index in your project\n",
    "data_index = api.data_indices.create(proj_key=PROJ_KEY, name=INDEX_NAME)\n",
    "_GARBAGE_COLLECTOR.append(data_index)\n",
    "index_key = data_index.source.index_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "530fd67a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:46:46.232454Z",
     "iopub.status.busy": "2024-05-27T10:46:46.232000Z",
     "iopub.status.idle": "2024-05-27T10:47:47.618951Z",
     "shell.execute_reply": "2024-05-27T10:47:47.618065Z",
     "shell.execute_reply.started": "2024-05-27T10:46:46.232420Z"
    }
   },
   "outputs": [],
   "source": [
    "# Upload and convert documents\n",
    "data_indices_utils.upload_files(\n",
    "    api=api, coords=data_index.source, local_file=INPUT_FILES_FOLDER\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bfe83b90",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:47.620964Z",
     "iopub.status.busy": "2024-05-27T10:47:47.620402Z",
     "iopub.status.idle": "2024-05-27T10:47:47.632258Z",
     "shell.execute_reply": "2024-05-27T10:47:47.631093Z",
     "shell.execute_reply.started": "2024-05-27T10:47:47.620933Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The data is now available. You can query it programmatically (see next section) or access it via the Deep Search UI at <br />https://cps.foc-deepsearch.zurich.ibm.com//projects/e74892839ff04df16a87cae3e96248a24eec7756/library/private/44770b3d86403db9fe2afb8339c82db6478a3807"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(\n",
    "    Markdown(\n",
    "        f\"The data is now available. You can query it programmatically (see next section) or access it via the Deep Search UI at <br />{api.client.config.host}/projects/{PROJ_KEY}/library/private/{index_key}\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d650fffc-566e-46b8-b04b-07a58ff20847",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45ec4a93",
   "metadata": {},
   "source": [
    "### Query your data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6a9a31ab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:47.634245Z",
     "iopub.status.busy": "2024-05-27T10:47:47.633751Z",
     "iopub.status.idle": "2024-05-27T10:47:48.848024Z",
     "shell.execute_reply": "2024-05-27T10:47:48.847147Z",
     "shell.execute_reply.started": "2024-05-27T10:47:47.634215Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data index contains 3 entries.\n"
     ]
    }
   ],
   "source": [
    "# Count the documents in the data index\n",
    "query = DataQuery(\"*\", source=[\"\"], limit=0, coordinates=data_index.source)\n",
    "query_results = api.queries.run(query)\n",
    "num_results = query_results.outputs[\"data_count\"]\n",
    "print(f\"The data index contains {num_results} entries.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a669e5a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:48.849757Z",
     "iopub.status.busy": "2024-05-27T10:47:48.849158Z",
     "iopub.status.idle": "2024-05-27T10:47:50.722773Z",
     "shell.execute_reply": "2024-05-27T10:47:50.721734Z",
     "shell.execute_reply.started": "2024-05-27T10:47:48.849722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d8b53e1cc3554a8c88a04a8fdb064f50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Finished fetching all data. Total is 1 records.\n"
     ]
    }
   ],
   "source": [
    "# Find documents matching query\n",
    "search_query = \"speedup\"\n",
    "query = DataQuery(\n",
    "    search_query,\n",
    "    source=[\"file-info.filename\", \"description.title\", \"description.authors\"],\n",
    "    coordinates=data_index.source,\n",
    ")\n",
    "query_results = api.queries.run(query)\n",
    "\n",
    "all_results = []\n",
    "cursor = api.queries.run_paginated_query(query)\n",
    "for result_page in tqdm(cursor):\n",
    "    # Iterate through the results of a single page, and add to the total list\n",
    "    for row in result_page.outputs[\"data_outputs\"]:\n",
    "        print()\n",
    "        metadata = row[\"_source\"].get(\n",
    "            \"description\", {}\n",
    "        )  # setting default, in case no title and authors are detected\n",
    "        # Add row to results table\n",
    "        all_results.append(\n",
    "            {\n",
    "                \"Filename\": row[\"_source\"][\"file-info\"][\"filename\"],\n",
    "                \"Title\": metadata.get(\"title\", \"\"),\n",
    "                \"Authors\": \", \".join(\n",
    "                    [author[\"name\"] for author in metadata.get(\"authors\", [])]\n",
    "                ),\n",
    "            }\n",
    "        )\n",
    "\n",
    "num_results = len(all_results)\n",
    "print(f\"Finished fetching all data. Total is {num_results} records.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e997bbae",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:50.724461Z",
     "iopub.status.busy": "2024-05-27T10:47:50.724035Z",
     "iopub.status.idle": "2024-05-27T10:47:50.738077Z",
     "shell.execute_reply": "2024-05-27T10:47:50.737245Z",
     "shell.execute_reply.started": "2024-05-27T10:47:50.724431Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### Results\n",
       "Documents matching the search query 'speedup':"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Title</th>\n",
       "      <th>Authors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the table with all results\n",
    "df = pd.json_normalize(all_results)\n",
    "display(\n",
    "    Markdown(f\"#### Results\\nDocuments matching the search query '{search_query}':\")\n",
    ")\n",
    "display(HTML(df.head().to_html(render_links=True)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66c0c18-54ea-4279-a1c2-c9ff6a9493d5",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ee15caf-37c2-4985-883b-27bea5cb9229",
   "metadata": {},
   "source": [
    "### Download your data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b7addff1-b857-463a-b82c-02ed9dd5ef5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:50.739861Z",
     "iopub.status.busy": "2024-05-27T10:47:50.739531Z",
     "iopub.status.idle": "2024-05-27T10:47:52.447541Z",
     "shell.execute_reply": "2024-05-27T10:47:52.446707Z",
     "shell.execute_reply.started": "2024-05-27T10:47:50.739834Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e04a6352dc88483d9f30f2360047a5ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished fetching all data. Total is 3 records.\n",
      "Data downloaded in /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp/T/tmpzb6oosaa\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Title</th>\n",
       "      <th>JSON Path</th>\n",
       "      <th>Markdown Path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td></td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td></td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td></td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "      <td>/var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Filename Title                                          JSON Path  \\\n",
       "0   2206.01062.pdf        /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...   \n",
       "1   2206.00785.pdf        /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...   \n",
       "2  20140197356.pdf        /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...   \n",
       "\n",
       "                                       Markdown Path  \n",
       "0  /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...  \n",
       "1  /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...  \n",
       "2  /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run query\n",
    "query = DataQuery(search_query=\"*\", source=[\"*\"], coordinates=data_index.source)\n",
    "cursor = api.queries.run_paginated_query(query)\n",
    "\n",
    "# Using a temp dir for demo purposes; to persist instead, set output dir accordingly\n",
    "temp_dir = tempfile.TemporaryDirectory()\n",
    "output_dir = temp_dir.name\n",
    "\n",
    "# Iterate through query results\n",
    "all_results = []\n",
    "for result_page in tqdm(cursor):\n",
    "    for row in result_page.outputs[\"data_outputs\"]:\n",
    "        metadata = row[\"_source\"].get(\n",
    "            \"description\", {}\n",
    "        )  # setting default, in case no title and authors are detected\n",
    "\n",
    "        # Download JSON file\n",
    "        file_path_json = Path(output_dir) / f\"{row['_id']}.json\"\n",
    "        with open(file_path_json, \"w\") as outfile:\n",
    "            json.dump(row[\"_source\"], outfile, indent=2)\n",
    "\n",
    "        # Export JSON to Markdown\n",
    "        file_path_md = Path(output_dir) / f\"{row['_id']}.md\"\n",
    "        with open(file_path_md, \"w\") as outfile:\n",
    "            outfile.write(export_to_markdown(row[\"_source\"]))\n",
    "\n",
    "        all_results.append(\n",
    "            {\n",
    "                \"Filename\": row[\"_source\"][\"file-info\"][\"filename\"],\n",
    "                \"Title\": metadata.get(\"title\", \"\"),\n",
    "                \"JSON Path\": file_path_json,\n",
    "                \"Markdown Path\": file_path_md,\n",
    "            }\n",
    "        )\n",
    "\n",
    "print(f\"Finished fetching all data. Total is {len(all_results)} records.\")\n",
    "print(f\"Data downloaded in {output_dir}\")\n",
    "\n",
    "# Visualize a table listing document titles and locations\n",
    "df = pd.json_normalize(all_results)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fee869dd-6f23-4999-b262-b034d2161deb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:52.449246Z",
     "iopub.status.busy": "2024-05-27T10:47:52.448798Z",
     "iopub.status.idle": "2024-05-27T10:47:52.470073Z",
     "shell.execute_reply": "2024-05-27T10:47:52.468763Z",
     "shell.execute_reply.started": "2024-05-27T10:47:52.449214Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "## Markdown content"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "## DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis\n",
       "\n",
       "Birgit Pfitzmann IBM Research Rueschlikon, Switzerland bpf@zurich.ibm.com\n",
       "\n",
       "Christoph Auer IBM Research Rueschlikon, Switzerland cau@zurich.ibm.com\n",
       "\n",
       "Michele Dolfi IBM Research Rueschlikon, Switzerland dol@zurich.ibm.com\n",
       "\n",
       "Ahmed S. Nassar IBM Research Rueschlikon, Switzerland ahn@zurich.ibm.com\n",
       "\n",
       "Peter Staar IBM Research Rueschlikon, Switzerland taa@zurich.ibm.com\n",
       "\n",
       "## ABSTRACT\n",
       "\n",
       "Accurate document layout analysis is a key requirement for highquality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present DocLayNet, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double-and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNettrained models are more robust and thus the preferred choice for general-purpose document-layout analysis.\n",
       "\n",
       "## CCS CONCEPTS\n",
       "\n",
       "· Information systems → Document structure; · Applied computing → Document analysis; · Computing methodologies → Machine learning; Computer vision; Object detection;\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "## JSON content"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<code>{\n",
       "  \"_name\": \"2206.01062.pdf\",\n",
       "  \"bitmaps\": [],\n",
       "  \"description\": {\n",
       "    \"logs\": [\n",
       "      {\n",
       "        \"agent\": \"CCS\",\n",
       "        \"type\": \"parsing\",\n",
       "        \"comment\": \"CCS v10.5.0 parsing of documents\",\n",
       "        \"date\": \"2024-05-27T10:47:36.503487+00:00\"\n",
       "      }\n",
       "    ],\n",
       "    \"collection\": {\n",
       "      \"type\": \"Document\"\n",
       "    }\n",
       "  },\n",
       "  \"equations\": [],\n",
       "  \"figures\": [\n",
       "    {\n",
       "      \"confidence\": 0.0,\n",
       "</code>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Peek first lines of a downloaded file\n",
    "with open(df.iloc[0][\"Markdown Path\"]) as demo_file:\n",
    "    content = \"\"\n",
    "    for _ in range(20):\n",
    "        line = demo_file.readline()\n",
    "        content += line\n",
    "\n",
    "    display(Markdown(\"## Markdown content\"))\n",
    "    display(Markdown(content))\n",
    "\n",
    "with open(df.iloc[0][\"JSON Path\"]) as demo_file:\n",
    "    content = \"\"\n",
    "    for _ in range(20):\n",
    "        line = demo_file.readline()\n",
    "        content += line\n",
    "    display(Markdown(\"## JSON content\"))\n",
    "    display(Markdown(f\"<code>{content}</code>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f968b98-0346-49d8-a3ac-fbbadd35b104",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f92b2b1-101d-4e01-b482-e5d97586248e",
   "metadata": {},
   "source": [
    "## Enable OCR\n",
    "\n",
    "This section is using the `ConversionSettings` object to enable OCR when converting PDF documents.\n",
    "\n",
    "Refer to the [OCR settings documentation](https://ds4sd.github.io/deepsearch-toolkit/guide/convert_doc/#modify-ocr-settings) for more details. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "58c15840-b342-44e8-b654-32b3e35f3991",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:52.479311Z",
     "iopub.status.busy": "2024-05-27T10:47:52.478701Z",
     "iopub.status.idle": "2024-05-27T10:47:52.484591Z",
     "shell.execute_reply": "2024-05-27T10:47:52.483243Z",
     "shell.execute_reply.started": "2024-05-27T10:47:52.479274Z"
    }
   },
   "outputs": [],
   "source": [
    "from deepsearch.documents.core.models import ConversionSettings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d477507f-bcbe-45b5-9557-dd0e3109c81c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:52.488533Z",
     "iopub.status.busy": "2024-05-27T10:47:52.487580Z",
     "iopub.status.idle": "2024-05-27T10:47:53.904759Z",
     "shell.execute_reply": "2024-05-27T10:47:53.903917Z",
     "shell.execute_reply.started": "2024-05-27T10:47:52.488483Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a new data index to process with OCR\n",
    "data_index = api.data_indices.create(proj_key=PROJ_KEY, name=INDEX_NAME + \"-ocr\")\n",
    "_GARBAGE_COLLECTOR.append(data_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "57ac3ecb-f411-4dd2-936e-4e73825da323",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:47:53.906948Z",
     "iopub.status.busy": "2024-05-27T10:47:53.906165Z",
     "iopub.status.idle": "2024-05-27T10:48:42.217491Z",
     "shell.execute_reply": "2024-05-27T10:48:42.216788Z",
     "shell.execute_reply.started": "2024-05-27T10:47:53.906917Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### Results\n",
       "The data is now available. This file will now display the text from the scanned pages. Access it via the Deep Search UI at <br />https://cps.foc-deepsearch.zurich.ibm.com//projects/e74892839ff04df16a87cae3e96248a24eec7756/library/private/a9d5732282cacf822d94086f492992542a271692"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load conversion settings and enable OCR\n",
    "cs = ConversionSettings.from_project(api, proj_key=PROJ_KEY)\n",
    "cs.ocr.enabled = True  # Enable or disable OCR\n",
    "\n",
    "# Upload and convert documents with custom conversion settings\n",
    "data_indices_utils.upload_files(\n",
    "    api=api, coords=data_index.source, local_file=INPUT_OCR_FILE, conv_settings=cs\n",
    ")\n",
    "\n",
    "# Display message\n",
    "display(\n",
    "    Markdown(\n",
    "        f\"#### Results\\nThe data is now available. This file will now display the text from the scanned pages. Access it via the Deep Search UI at <br />{api.client.config.host}/projects/{data_index.source.proj_key}/library/private/{data_index.source.index_key}\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46950043-2687-4fc9-ab20-43bcc88d246d",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17cf77e1-125b-442a-812b-fdfc2b5eb259",
   "metadata": {},
   "source": [
    "## Enable raw PDF cells\n",
    "\n",
    "The document conversion pipeline is producing a JSON file corresponsing to the PDF documents, where all document components have been grouped, classified and further inspected (e.g. table structure) for a simple usage.\n",
    "\n",
    "However, in some use cases it is convenient to rely on the raw text cells contained in the PDF document.\n",
    "This an auxiliary file that Deep Search is making available on demand.\n",
    "To following section is demonstrating how this is enabled.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "16655fea-becc-4236-9ad3-ac1e7ba13c46",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:48:42.219795Z",
     "iopub.status.busy": "2024-05-27T10:48:42.219135Z",
     "iopub.status.idle": "2024-05-27T10:48:42.227066Z",
     "shell.execute_reply": "2024-05-27T10:48:42.223426Z",
     "shell.execute_reply.started": "2024-05-27T10:48:42.219745Z"
    }
   },
   "outputs": [],
   "source": [
    "from deepsearch.documents.core.models import TargetSettings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c3c8d9a-b06c-4966-98ee-335be9b659aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:48:42.236320Z",
     "iopub.status.busy": "2024-05-27T10:48:42.234796Z",
     "iopub.status.idle": "2024-05-27T10:48:43.223126Z",
     "shell.execute_reply": "2024-05-27T10:48:43.222336Z",
     "shell.execute_reply.started": "2024-05-27T10:48:42.236267Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a new data index to process with OCR\n",
    "data_index = api.data_indices.create(proj_key=PROJ_KEY, name=INDEX_NAME + \"-raw\")\n",
    "_GARBAGE_COLLECTOR.append(data_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d5060e1e-f8da-4948-8d34-56ed86f5ddb3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:48:43.224502Z",
     "iopub.status.busy": "2024-05-27T10:48:43.224100Z",
     "iopub.status.idle": "2024-05-27T10:49:37.944415Z",
     "shell.execute_reply": "2024-05-27T10:49:37.943677Z",
     "shell.execute_reply.started": "2024-05-27T10:48:43.224475Z"
    }
   },
   "outputs": [],
   "source": [
    "# Set custom target settings with raw pdf cells enabled\n",
    "tsettings = TargetSettings(add_raw_pages=True)\n",
    "\n",
    "# Upload and convert documents with custom conversion settings\n",
    "data_indices_utils.upload_files(\n",
    "    api=api,\n",
    "    coords=data_index.source,\n",
    "    local_file=INPUT_FILES_FOLDER,\n",
    "    target_settings=tsettings,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8d6dc87a-1501-4564-9914-f2787a85508b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:49:37.946123Z",
     "iopub.status.busy": "2024-05-27T10:49:37.945824Z",
     "iopub.status.idle": "2024-05-27T10:49:40.025489Z",
     "shell.execute_reply": "2024-05-27T10:49:40.024293Z",
     "shell.execute_reply.started": "2024-05-27T10:49:37.946098Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished fetching all data. Total is 31 records.\n",
      "Data downloaded in /var/folders/32/5m4rnj816kl4pxl4vm97_wbh0000kp/T/tmpzb6oosaa\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "#### Results\n",
       "Here is the list of the files uploaded and the urls where to download the raw pdf cells details."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Page</th>\n",
       "      <th>RAW file</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/3c76b6d3fd82865e42c51d5cbd7d1a9996dba7902643b919acc581e866b92716.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=7f4fb8b328f146fb689bc2a31a9b20d8088a3d5ef08a039cb137cfcd43c640fd'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>2</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/5ccfaddd314d3712cbabc857c8c0f33d1268341ce37b27089857cbf09f0522d4.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=23cf104fe43c53fc412a79ee273aaede30ee4e743d8e4c8c740153367c209af2'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>3</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/d2dc51ad0a01ee9486ffe248649ee1cd10ce35773de8e4b21abf30d310f4fc26.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1fe63c0fc7fe4c194367ec7addd089b744c15a20a2b4a4d25aa045b1e8bce865'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>4</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/310121977375f8f1106412189943bd70f121629b2b4d35394077233dedbfb041.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e7abd5ca84ed52bf70eb003926816fd6940d4b29e32eeb164415db9baaf87f46'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>5</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/09fa72b602eb0640669844acabc17ef494802a4a9188aeaaf0e0131c496e6951.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9b9a97905490ffc2f98456095994945febde87d17dc87627cf29a84a22f8abe9'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>6</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/ec3fa60f136f3d9f5fa790ab27f5d1c14e5622573c52377b909b591d0be0ea44.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d90bb7ce33c241f8482db4335c9fc795dddf6b538e0443f132dd156df3bdd086'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>7</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/ec1bc56fe581ce95615b1fab11c3ba8fc89662acf2f53446decd380a155b06dd.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=ff0223bd2a4b3a2833f5e0b9a3afaa55d2948afb0e5c9daa6264b95048ce6c7f'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>8</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/fbd2b06876dddc19ee08e0a9751d978c03e6943b74bedf1d83d6528cd4f8954d.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=3b65aaf7223cf3b88ee40da070100ba4437cf3c7614913ebb37f98f179d5764a'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2206.01062.pdf</td>\n",
       "      <td>9</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/6cfa4eb4410fa9972da289dbf8d8cc585d317a192e1214c778ddd7768e98f311.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=937f45d671c00650294b87d57d3117f743ddcd7a84786c396656723510ca21b6'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/dc8ab77215bdf5d1c50e4635bc886078ceb862131df9de406624cb73be373fc1.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=6387b7ed31edc30f01026822e9c431aac27d6c83b052d19804db1545954c0333'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>2</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/58474aac22030f7608c307aeb1aee7b69b8b4ca533c80ae41ab68153790d6ab0.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=97feaafdc44c9fab120a8757c5346b8819d0c432664356fefdcf872bafe3a068'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>3</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/1d96fb50e9a4d970cab5199fa3b76469b804c2457390c4bdda04890bc651b813.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=be6353440d565b36a02dffadbe5260599e5915bd41840a38be3a11f1a1dcd699'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>4</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/4dc4d130627b1f9841208469936e65cf4667f47730044ef31ecbf9664a41d706.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=f894184605aecfb42ec5c27ddde07cfbe5b12f9a8fb79326359b204ebd0926af'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>5</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/4ba309b0bcae706030066723f66989d89818f00305702bbd5935d6eca9d4db6d.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b259bb77cc35c5072ea50ff52efe2ba1f85936ed7229c4a7b3ae8ba3128d78c6'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>6</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/09357309f7ee2ad6efe2951fcc314bcdac7f6296b244724a78d9e9037bd071ff.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=bdf6b11171c1e47d3b1f2b135daa560245af02831b8d813d939fc7c71127606c'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>7</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/1b8ad8508fe45a56af6aafcaeefcefa168b0b0e8530b9b0c60be1542ce001c14.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1469c0502416084e93a03bff3164efc3e07eb661164438d0dc2e94de11d09584'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>8</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/8289d80b04962b29ea5f2967b69ca0ae43fb56535cef4fe36630c58a9edde998.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=8374f25f22b993580adc865c88000740f1a1537214b3e2853a97dd2fd4cf62c1'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>9</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/ba08e96ab334e61dfcde2a71203a96856d5261280e445df5f7d84d7e5974170e.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=00b7b9fc47b3e92c35bcfef269b469c894007bc01da1edafa693065d6a3aeb5e'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>10</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/f90176f27b3737c29840cf46f7c23ef86f607a2edd94f4c4201e931817f7d2ee.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9759da12050ee39c3abe5e9532f6703d5dfeb1f86459b0d1ca5c7e4456d70661'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2206.00785.pdf</td>\n",
       "      <td>11</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/5c71fa7ee304c4d871411bbb71734c39660173335fc28f955dc3ee8e044fd748.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b556e814ece0ad0d1861d902aff586c5d06adeec71a942f140230f8af5351faa'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/85d641ecde9849942cb4f1c4ecb7e7f3d4e4fdb1c9d26a66b2ce08482fb4f9e6.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=6a25f6c09aa89c7cefc2718f31cefe4fabc2a32e53c3fdac4a656eba729fe0f8'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>2</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/36956f9cf96313d6c405d166346538d186d8d0e0de6fe940c51611ee2071b46f.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=428a650d0e888aa3d8919a0a4354be07631ad9e9e2e4ab7908888d6d793bd675'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>3</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/720b85ce0a0f9d92674e949816bb47328eaf1f3f87eac2e56115c3715b71b2dc.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d8e40ee22aac7452ee193c397db019a24efcb1470c571007691de1b85023ccdb'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>4</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/5790543526a36735c70329d7cc00c936df0bb5d3c033442d92d62868009b951c.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=518d58f7f4286b3dc9925ae8b5b93a2c247dfd62820d20ab518d3b8378823044'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>5</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/ad8cb3b74d125828f71b8aded57c4f1cff54cb1cae6d6c7f5be6246fa21a60e8.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=c29a8d0c54d9b726715a6893f944db7c1ae90a3ac487196e0adebf784a636718'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>6</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/50c860f9d04699d87e543a171048ecb5ecdc45342a6c645ca56bdfa7c000bd50.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=56f8e7507c3ee962c89e58d2031b946fb9c1937b8b1de292f593eb8df24d0b18'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>7</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/192c6314a1ddcf8cdb4cd4eae56164fa5147ec3b733c531ae53982747f697a79.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=c2e41bdd8176e06edbbf7cdb554163b1f672097d966790523ac27cf44c2070a3'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>8</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/0d9b2b623eab94d057a4f2b7828e1a03ba22d0451a5659d7230c3c52aaf74d17.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=0448307c57eb798da710eb14c3cbb38846122e284fa98bdffbda3ca57afb8d9e'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>9</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/f9775f3168756fd61087943aa3255b7a1588dda5379bf912f948b066ae3d0c52.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2431da9edd7b34d16e5d89ef540073ad6b1364a81802faaffeb3d8f659e4dc53'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>10</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/810ca6de2f6fee1fa0e9bb03da042e2f89ef310e8edc7fd706e74d094e8d4987.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d2257720d6fb42682261da97d678dcf6c4f385b6eace34c2474a73af892e4594'>Link</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>20140197356.pdf</td>\n",
       "      <td>11</td>\n",
       "      <td><a target='_blank' href='https://s3.eu-de.cloud-object-storage.appdomain.cloud/foc-cps-foc-mvp-deepsearch-bags/foc-cps-foc-mvp-deepsearch-projdata/kouros_e74892/entities/foc-cps-foc-mvp-deepsearch-projdata70bc2a/RAW/137af51024bef23fb1fdc9c269fff327804a934331f3dd18d2b49fae7ceec536.raw?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b7d02de376e24e97904c2f698795e3d0%2F20240527%2Feu-de-standard%2Fs3%2Faws4_request&X-Amz-Date=20240527T104939Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e8bd1e9f9dc709d9030bb326324fb388a58c6449fca3351769135d02bac23660'>Link</a></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run query\n",
    "query = DataQuery(\n",
    "    search_query=\"*\",\n",
    "    source=[\"file-info.filename\", \"_s3_data.raw-pages\"],\n",
    "    coordinates=data_index.source,\n",
    ")\n",
    "cursor = api.queries.run_paginated_query(query)\n",
    "\n",
    "# Iterate through query results\n",
    "all_results = []\n",
    "for result_page in cursor:\n",
    "    for row in result_page.outputs[\"data_outputs\"]:\n",
    "        filename = row[\"_source\"][\"file-info\"][\"filename\"]\n",
    "        for raw_page in row[\"_source\"][\"_s3_data\"][\"raw-pages\"]:\n",
    "\n",
    "            all_results.append(\n",
    "                {\n",
    "                    \"Filename\": filename,\n",
    "                    \"Page\": raw_page[\"page\"],\n",
    "                    \"RAW file\": f\"<a target='_blank' href='{raw_page['url']}'>Link</a>\",\n",
    "                }\n",
    "            )\n",
    "\n",
    "print(f\"Finished fetching all data. Total is {len(all_results)} records.\")\n",
    "print(f\"Data downloaded in {output_dir}\")\n",
    "display(\n",
    "    Markdown(\n",
    "        \"#### Results\\nHere is the list of the files uploaded and the urls where to download the raw pdf cells details.\"\n",
    "    )\n",
    ")\n",
    "\n",
    "# Visualize a table listing document titles and locations\n",
    "df = pd.json_normalize(all_results)\n",
    "display(HTML(df.to_html(render_links=True, escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b13b5f9e",
   "metadata": {},
   "source": [
    "### Cleanup\n",
    "If enabled, we will delete all the resources created in the example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9a0b82b8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-27T10:49:40.028168Z",
     "iopub.status.busy": "2024-05-27T10:49:40.027334Z",
     "iopub.status.idle": "2024-05-27T10:49:47.131402Z",
     "shell.execute_reply": "2024-05-27T10:49:47.129835Z",
     "shell.execute_reply.started": "2024-05-27T10:49:40.028110Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data index tmp_20240527124643 deleted\n",
      "Data index tmp_20240527124643-ocr deleted\n",
      "Data index tmp_20240527124643-raw deleted\n"
     ]
    }
   ],
   "source": [
    "# Delete data index\n",
    "if CLEANUP:\n",
    "    for data_index in _GARBAGE_COLLECTOR:\n",
    "        api.data_indices.delete(data_index.source)\n",
    "        print(f\"Data index {data_index.name} deleted\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41e3df80-1073-437e-a890-36801453898d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}