examples/pyterrier_preferences_parallel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Parallel Computation of Preference Matrices\n",
    "\n",
    "The notebook below exemplifies how `ir_axioms` can be used to compute preference matrices for all runs submitted to the TREC 2020 Deep Learning passage ranking track."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Preparation\n",
    "\n",
    "Install the `ir_axioms` framework and [PyTerrier](https://github.com/terrier-org/pyterrier). In Google Colab, we do this automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sys import modules\n",
    "\n",
    "if 'google.colab' in modules:\n",
    "    !pip install -q ir_axioms[examples] python-terrier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We initialize PyTerrier and import all required libraries and load the data from [ir_datasets](https://ir-datasets.com/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n"
     ]
    }
   ],
   "source": [
    "from pyterrier import started, init\n",
    "\n",
    "if not started():\n",
    "    init(tqdm=\"auto\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Datasets and Index\n",
    "Using PyTerrier's `get_dataset()`, we load the MS MARCO passage ranking dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.datasets import get_dataset, Dataset\n",
    "\n",
    "# Load dataset.\n",
    "dataset_name = \"msmarco-passage/trec-dl-2020/judged\"\n",
    "dataset: Dataset = get_dataset(f\"irds:{dataset_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now define paths where we will store temporary files, datasets, and the search index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "cache_dir = Path(\"cache/\")\n",
    "index_dir = cache_dir / \"indices\" / dataset_name.split(\"/\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If the index is not ready yet, now is a good time to create it and index the MS MARCO passages.\n",
    "(Lean back and relax as this may take a while...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.index import IterDictIndexer\n",
    "\n",
    "if not index_dir.exists():\n",
    "    indexer = IterDictIndexer(str(index_dir.absolute()))\n",
    "    indexer.index(\n",
    "        dataset.get_corpus_iter(),\n",
    "        fields=[\"text\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Submitted Runs\n",
    "Define the path where you have stored the submitted run files.\n",
    "(You have to manually download the run files from TREC and store them in a directory of your choice. Then adjust the path below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "result_dir = Path(\n",
    "    \"/mnt/ceph/storage/data-in-progress/data-research\"\n",
    "    \"/web-search/web-search-trec/trec-system-runs\"\n",
    "    \"/trec29/deep.passages\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can now load the runs from the run file directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8da6f7f3bd1548169c68edbf86ed7e3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Load runs:   0%|          | 0/59 [00:00<?, ?run/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.auto import tqdm\n",
    "from pyterrier.io import read_results\n",
    "\n",
    "run_files = list(result_dir.iterdir())\n",
    "run_results = [\n",
    "    read_results(result_file)\n",
    "    for result_file in tqdm(\n",
    "        run_files,\n",
    "        desc=\"Load runs\",\n",
    "        unit=\"run\",\n",
    "        total=len(run_files)\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Concat the retrieved results from all runs in a single data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>docno</th>\n",
       "      <th>rank</th>\n",
       "      <th>score</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3505</td>\n",
       "      <td>1030851</td>\n",
       "      <td>0</td>\n",
       "      <td>-11.050011</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3505</td>\n",
       "      <td>3998753</td>\n",
       "      <td>1</td>\n",
       "      <td>-11.185551</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3505</td>\n",
       "      <td>2631212</td>\n",
       "      <td>2</td>\n",
       "      <td>-11.263526</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3505</td>\n",
       "      <td>329745</td>\n",
       "      <td>3</td>\n",
       "      <td>-11.579689</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3505</td>\n",
       "      <td>8285654</td>\n",
       "      <td>4</td>\n",
       "      <td>-11.582441</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199995</th>\n",
       "      <td>1136962</td>\n",
       "      <td>659447</td>\n",
       "      <td>996</td>\n",
       "      <td>0.006507</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199996</th>\n",
       "      <td>1136962</td>\n",
       "      <td>4101357</td>\n",
       "      <td>997</td>\n",
       "      <td>0.006501</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199997</th>\n",
       "      <td>1136962</td>\n",
       "      <td>7422963</td>\n",
       "      <td>998</td>\n",
       "      <td>0.006494</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199998</th>\n",
       "      <td>1136962</td>\n",
       "      <td>4297877</td>\n",
       "      <td>999</td>\n",
       "      <td>0.006488</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199999</th>\n",
       "      <td>1136962</td>\n",
       "      <td>4735960</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.006481</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11540491 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            qid    docno  rank      score           name\n",
       "0          3505  1030851     0 -11.050011  TUW-TK-2Layer\n",
       "1          3505  3998753     1 -11.185551  TUW-TK-2Layer\n",
       "2          3505  2631212     2 -11.263526  TUW-TK-2Layer\n",
       "3          3505   329745     3 -11.579689  TUW-TK-2Layer\n",
       "4          3505  8285654     4 -11.582441  TUW-TK-2Layer\n",
       "...         ...      ...   ...        ...            ...\n",
       "199995  1136962   659447   996   0.006507  p_d2q_rm3_duo\n",
       "199996  1136962  4101357   997   0.006501  p_d2q_rm3_duo\n",
       "199997  1136962  7422963   998   0.006494  p_d2q_rm3_duo\n",
       "199998  1136962  4297877   999   0.006488  p_d2q_rm3_duo\n",
       "199999  1136962  4735960  1000   0.006481  p_d2q_rm3_duo\n",
       "\n",
       "[11540491 rows x 5 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pandas import concat\n",
    "\n",
    "all_results = concat(run_results)\n",
    "all_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Import Axioms\n",
    "Here we're listing which axioms we want to compute preferences for.\n",
    "Because some axioms require API calls or are computationally expensive, we cache all axioms using `ir_axiom`'s tilde operator (`~`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import (\n",
    "    ArgUC, QTArg, QTPArg, aSL, PROX1, PROX2, PROX3, PROX4, PROX5, TFC1, TFC3, RS_TF, RS_TF_IDF, RS_BM25, RS_PL2, RS_QL,\n",
    "    AND, LEN_AND, M_AND, LEN_M_AND, DIV, LEN_DIV, M_TDC, LEN_M_TDC, STMC1, STMC1_f, STMC2, STMC2_f, LNC1, TF_LNC, LB1,\n",
    "    REG, ANTI_REG, ASPECT_REG, REG_f, ANTI_REG_f, ASPECT_REG_f\n",
    ")\n",
    "\n",
    "axioms = [\n",
    "    ArgUC(), QTArg(), QTPArg(), aSL(),\n",
    "    LNC1(), TF_LNC(), LB1(),\n",
    "    PROX1(), PROX2(), PROX3(), PROX4(), PROX5(),\n",
    "    REG(), REG_f(), ANTI_REG(), ANTI_REG_f(), ASPECT_REG(), ASPECT_REG_f(),\n",
    "    AND(), LEN_AND(), M_AND(), LEN_M_AND(), DIV(), LEN_DIV(),\n",
    "    RS_TF(), RS_TF_IDF(), RS_BM25(), RS_PL2(), RS_QL(),\n",
    "    TFC1(), TFC3(), M_TDC(), LEN_M_TDC(),\n",
    "    STMC1(), STMC1_f(), STMC2(), STMC2_f(),\n",
    "]\n",
    "axioms_cached = [~axiom for axiom in axioms]\n",
    "axiom_names = [axiom.name for axiom in axioms]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Preference Computation\n",
    "After having defined the axioms to compute, we create a new PyTerrier pipeline that computes preference matrices for the top-10 results of each system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier import Transformer\n",
    "from ir_axioms.backend.pyterrier.transformers import AxiomaticPreferences\n",
    "\n",
    "compute_preferences = Transformer.from_df(all_results) % 10 >> AxiomaticPreferences(\n",
    "    axioms=axioms,\n",
    "    # axioms=axioms_cached,\n",
    "    axiom_names=axiom_names,\n",
    "    index=index_dir,\n",
    "    dataset=dataset_name,\n",
    "    cache_dir=cache_dir,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To speed up computation, let's distribute preference matrix computation across 4 cores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "compute_preferences = compute_preferences.parallel(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In the next step, we parallely compute the preference matrices (line 5)\n",
    "and measure the elapsed time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n",
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n",
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n",
      "Computing query axiom preferences: 100%|██████████| 826/826 [05:31<00:00,  2.49query/s]\n",
      "Computing query axiom preferences: 100%|██████████| 708/708 [05:44<00:00,  2.06query/s]\n",
      "Computing query axiom preferences: 100%|██████████| 826/826 [06:09<00:00,  2.24query/s]\n",
      "Computing query axiom preferences: 100%|██████████| 826/826 [06:54<00:00,  2.00query/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed time: 445.94s\n",
      "Preferences per second: 601.71s\n"
     ]
    }
   ],
   "source": [
    "from time import perf_counter_ns\n",
    "\n",
    "time = perf_counter_ns()\n",
    "\n",
    "preferences = compute_preferences.transform(dataset.get_topics())\n",
    "\n",
    "elapsed_time = perf_counter_ns() - time\n",
    "elapsed_time_seconds = elapsed_time / 1_000_000_000\n",
    "print(f\"Elapsed time: {elapsed_time_seconds:.2f}s\")\n",
    "preferences_per_second = len(preferences) / elapsed_time_seconds\n",
    "print(f\"Preferences per second: {preferences_per_second:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here's the resulting Pandas `DataFrame` containing all preferences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>query</th>\n",
       "      <th>docno_a</th>\n",
       "      <th>rank_a</th>\n",
       "      <th>score_a</th>\n",
       "      <th>name</th>\n",
       "      <th>docno_b</th>\n",
       "      <th>rank_b</th>\n",
       "      <th>score_b</th>\n",
       "      <th>ArgUC_preference</th>\n",
       "      <th>...</th>\n",
       "      <th>RS-PL2_preference</th>\n",
       "      <th>RS-QL_preference</th>\n",
       "      <th>TFC1_preference</th>\n",
       "      <th>TFC3_preference</th>\n",
       "      <th>M-TDC_preference</th>\n",
       "      <th>LEN-M-TDC_preference</th>\n",
       "      <th>STMC1_preference</th>\n",
       "      <th>STMC1-fastText_preference</th>\n",
       "      <th>STMC2_preference</th>\n",
       "      <th>STMC2-fastText_preference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1030303</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1030303</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "      <td>8726435</td>\n",
       "      <td>1</td>\n",
       "      <td>-6.964480</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1030303</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "      <td>5681381</td>\n",
       "      <td>2</td>\n",
       "      <td>-7.851367</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1030303</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "      <td>7284047</td>\n",
       "      <td>3</td>\n",
       "      <td>-8.081900</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1030303</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726437</td>\n",
       "      <td>0</td>\n",
       "      <td>-6.537941</td>\n",
       "      <td>TUW-TK-2Layer</td>\n",
       "      <td>8726433</td>\n",
       "      <td>4</td>\n",
       "      <td>-8.227568</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59623</th>\n",
       "      <td>997622</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "      <td>73462</td>\n",
       "      <td>5</td>\n",
       "      <td>81.672453</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59624</th>\n",
       "      <td>997622</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "      <td>73460</td>\n",
       "      <td>6</td>\n",
       "      <td>81.004847</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59625</th>\n",
       "      <td>997622</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "      <td>4643397</td>\n",
       "      <td>7</td>\n",
       "      <td>80.117589</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59626</th>\n",
       "      <td>997622</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "      <td>73465</td>\n",
       "      <td>8</td>\n",
       "      <td>79.295991</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59627</th>\n",
       "      <td>997622</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>p_d2q_rm3_duo</td>\n",
       "      <td>4518222</td>\n",
       "      <td>9</td>\n",
       "      <td>78.546576</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>268326 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           qid                               query  docno_a  rank_a  \\\n",
       "0      1030303                  who is aziz hashim  8726437       0   \n",
       "1      1030303                  who is aziz hashim  8726437       0   \n",
       "2      1030303                  who is aziz hashim  8726437       0   \n",
       "3      1030303                  who is aziz hashim  8726437       0   \n",
       "4      1030303                  who is aziz hashim  8726437       0   \n",
       "...        ...                                 ...      ...     ...   \n",
       "59623   997622  where is the show shameless filmed  4518222       9   \n",
       "59624   997622  where is the show shameless filmed  4518222       9   \n",
       "59625   997622  where is the show shameless filmed  4518222       9   \n",
       "59626   997622  where is the show shameless filmed  4518222       9   \n",
       "59627   997622  where is the show shameless filmed  4518222       9   \n",
       "\n",
       "         score_a           name  docno_b  rank_b    score_b  ArgUC_preference  \\\n",
       "0      -6.537941  TUW-TK-2Layer  8726437       0  -6.537941                 0   \n",
       "1      -6.537941  TUW-TK-2Layer  8726435       1  -6.964480                 0   \n",
       "2      -6.537941  TUW-TK-2Layer  5681381       2  -7.851367                 0   \n",
       "3      -6.537941  TUW-TK-2Layer  7284047       3  -8.081900                 0   \n",
       "4      -6.537941  TUW-TK-2Layer  8726433       4  -8.227568                 0   \n",
       "...          ...            ...      ...     ...        ...               ...   \n",
       "59623  78.546576  p_d2q_rm3_duo    73462       5  81.672453                 0   \n",
       "59624  78.546576  p_d2q_rm3_duo    73460       6  81.004847                 0   \n",
       "59625  78.546576  p_d2q_rm3_duo  4643397       7  80.117589                 0   \n",
       "59626  78.546576  p_d2q_rm3_duo    73465       8  79.295991                 0   \n",
       "59627  78.546576  p_d2q_rm3_duo  4518222       9  78.546576                 0   \n",
       "\n",
       "       ...  RS-PL2_preference  RS-QL_preference  TFC1_preference  \\\n",
       "0      ...                  0                 0                0   \n",
       "1      ...                 -1                -1                0   \n",
       "2      ...                  1                 1                0   \n",
       "3      ...                 -1                 1                0   \n",
       "4      ...                 -1                -1                0   \n",
       "...    ...                ...               ...              ...   \n",
       "59623  ...                  1                 1                0   \n",
       "59624  ...                 -1                 1                0   \n",
       "59625  ...                 -1                 1                0   \n",
       "59626  ...                 -1                -1               -1   \n",
       "59627  ...                  0                 0                0   \n",
       "\n",
       "       TFC3_preference  M-TDC_preference  LEN-M-TDC_preference  \\\n",
       "0                    0                 0                     0   \n",
       "1                    0                 0                     0   \n",
       "2                    0                 0                     0   \n",
       "3                    0                 0                     0   \n",
       "4                    0                 0                     0   \n",
       "...                ...               ...                   ...   \n",
       "59623                0                 0                     0   \n",
       "59624                0                 0                     0   \n",
       "59625                0                 0                     0   \n",
       "59626                0                 0                     0   \n",
       "59627                0                 0                     0   \n",
       "\n",
       "       STMC1_preference  STMC1-fastText_preference  STMC2_preference  \\\n",
       "0                     0                          0                 0   \n",
       "1                     1                         -1                 0   \n",
       "2                     1                         -1                 0   \n",
       "3                    -1                         -1                 0   \n",
       "4                     1                         -1                 0   \n",
       "...                 ...                        ...               ...   \n",
       "59623                 1                          1                 0   \n",
       "59624                -1                         -1                 0   \n",
       "59625                -1                         -1                 0   \n",
       "59626                -1                         -1                 0   \n",
       "59627                 0                          0                 1   \n",
       "\n",
       "       STMC2-fastText_preference  \n",
       "0                              1  \n",
       "1                              0  \n",
       "2                              1  \n",
       "3                              0  \n",
       "4                              0  \n",
       "...                          ...  \n",
       "59623                          0  \n",
       "59624                          0  \n",
       "59625                          0  \n",
       "59626                          0  \n",
       "59627                          1  \n",
       "\n",
       "[268326 rows x 46 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preferences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Note About Backends\n",
    "The parallelization is implemented in [PyTerrier](https://pyterrier.readthedocs.io/en/latest/parallel.html).\n",
    "In this example we use the default [Joblib](https://joblib.readthedocs.io/en/latest/) backend which splits computation per query and runs on multiple cores on the same machine.\n",
    "\n",
    "However, you could also use the [Ray](https://www.ray.io/) backend.\n",
    "With Ray you can connect to remote clusters and distribute the workload across multiple machines\n",
    "(e.g. [Kubernetes](https://docs.ray.io/en/latest/cluster/kubernetes.html), [Hadoop/Spark](https://docs.ray.io/en/latest/cluster/yarn.html), or [Slurm](https://docs.ray.io/en/latest/cluster/slurm.html)).\n",
    "Please refer to the [Ray documentation](https://docs.ray.io/en/latest/) for detailed instructions on how to connect your cluster."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f1a2212cc5ec32f4daeb9f9834a8cd8a7db46f8766913eafbf4fe810a9194b3a"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}