examples/pyterrier_post_hoc_analysis_of_runs_and_qrels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Post-Hoc Analysis of Rankings and Relevance Judgments\n",
    "\n",
    "The notebook below examplifies how `ir_axioms` allows the post-hoc analysis of run files and qrels using the passage retrieval task of the TREC Deep Learning track in 2019 and 2020 as example (using a single BM25 run). In this notebook, we calculate the distribution of axiom preferences in rankings and evaluate the consistency of rankings and relevance judgments with retrieval axioms."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Preparation\n",
    "\n",
    "Install the `ir_axioms` framework and [PyTerrier](https://github.com/terrier-org/pyterrier). In Google Colab, we do this automatically."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sys import modules\n",
    "\n",
    "if 'google.colab' in modules:\n",
    "    !pip install -q ir_axioms[examples] python-terrier"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We initialize PyTerrier and import all required libraries and load the data from [ir_datasets](https://ir-datasets.com/)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from pyterrier import started, init\n",
    "\n",
    "if not started():\n",
    "    init(tqdm=\"auto\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Datasets and Index\n",
    "Using PyTerrier's `get_dataset()`, we load the MS MARCO passage ranking dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.datasets import get_dataset, Dataset\n",
    "\n",
    "# Load dataset.\n",
    "dataset_name = \"msmarco-passage\"\n",
    "dataset: Dataset = get_dataset(f\"irds:{dataset_name}\")\n",
    "dataset_train: Dataset = get_dataset(f\"irds:{dataset_name}/trec-dl-2019/judged\")\n",
    "dataset_test: Dataset = get_dataset(f\"irds:{dataset_name}/trec-dl-2020/judged\")"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now define paths where we will store temporary files, datasets, and the search index."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "cache_dir = Path(\"cache/\")\n",
    "index_dir = cache_dir / \"indices\" / dataset_name.split(\"/\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "If the index is not ready yet, now is a good time to create it and index the MS MARCO passages.\n",
    "(Lean back and relax as this may take a while...)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.index import IterDictIndexer\n",
    "\n",
    "if not index_dir.exists():\n",
    "    indexer = IterDictIndexer(str(index_dir.absolute()))\n",
    "    indexer.index(\n",
    "        dataset.get_corpus_iter(),\n",
    "        fields=[\"text\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Baseline Run\n",
    "\n",
    "We use PyTerrier's `BatchRetrieve` to create a baseline search pipeline for retrieving with BM25 from the index we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.batchretrieve import BatchRetrieve\n",
    "\n",
    "bm25 = BatchRetrieve(str(index_dir.absolute()), wmodel=\"BM25\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Import Axioms\n",
    "Here we're listing which axioms we want to use in our experiments.\n",
    "Because some axioms require API calls or are computationally expensive, we cache all axioms using `ir_axiom`'s tilde operator (`~`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import (\n",
    "    ArgUC, QTArg, QTPArg, aSL, PROX1, PROX2, PROX3, PROX4, PROX5, TFC1, TFC3, RS_TF, RS_TF_IDF, RS_BM25, RS_PL2, RS_QL,\n",
    "    AND, LEN_AND, M_AND, LEN_M_AND, DIV, LEN_DIV, M_TDC, LEN_M_TDC, STMC1, STMC1_f, STMC2, STMC2_f, LNC1, TF_LNC, LB1,\n",
    "    REG, ANTI_REG, REG_f, ANTI_REG_f, ASPECT_REG, ASPECT_REG_f\n",
    ")\n",
    "\n",
    "axioms = [\n",
    "    ~ArgUC(), ~QTArg(), ~QTPArg(), ~aSL(),\n",
    "    ~LNC1(), ~TF_LNC(), ~LB1(),\n",
    "    ~PROX1(), ~PROX2(), ~PROX3(), ~PROX4(), ~PROX5(),\n",
    "    ~REG(), ~REG_f(), ~ANTI_REG(), ~ANTI_REG_f(), ~ASPECT_REG(), ~ASPECT_REG_f(),\n",
    "    ~AND(), ~LEN_AND(), ~M_AND(), ~LEN_M_AND(), ~DIV(), ~LEN_DIV(),\n",
    "    ~RS_TF(), ~RS_TF_IDF(), ~RS_BM25(), ~RS_PL2(), ~RS_QL(),\n",
    "    ~TFC1(), ~TFC3(), ~M_TDC(), ~LEN_M_TDC(),\n",
    "    ~STMC1(), ~STMC1_f(), ~STMC2(), ~STMC2_f(),\n",
    "]\n",
    "axiom_names = [axiom.axiom.name for axiom in axioms]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Axiomatic Experiments\n",
    "\n",
    "The `AxiomaticExperiment` class provides the entry-point to the post-hoc analysis of rankings and relevance judgments.\n",
    "We pass it the retrieval pipelines to evaluate, dataset and index, and the axioms to compute preferences for.\n",
    "With the `depth` parameter you can control how much preferences are being computed.\n",
    "\n",
    "Computing time roughly scales with `len(retrieval_systems)` x `len(topics)` x `depth` x `depth` x `len(axioms)`.\n",
    "Additional parameters help optimize computational costs by filtering missing qrels and/or topics.\n",
    "In our experience, caching helps most with multiple runs for the same benchmark (like TREC runs), because different run's result sets often overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.backend.pyterrier.experiment import AxiomaticExperiment\n",
    "\n",
    "experiment = AxiomaticExperiment(\n",
    "    retrieval_systems=[bm25],\n",
    "    topics=dataset_test.get_topics(),\n",
    "    qrels=dataset_test.get_qrels(),\n",
    "    index=index_dir,\n",
    "    dataset=dataset_name,\n",
    "    axioms=axioms,\n",
    "    axiom_names=axiom_names,\n",
    "    depth=10,\n",
    "    filter_by_qrels=False,\n",
    "    filter_by_topics=False,\n",
    "    verbose=True,\n",
    "    cache_dir=cache_dir,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Calculate All Preferences\n",
    "With the `preference` member you can compute a `DataFrame` containing all pairwise preferences up to the specified depth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff4d4477ba894c9a97fcb9a2107e9c5e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing system axiomatic preferences:   0%|          | 0/1 [00:00<?, ?system/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "448be720339e4034be5dfe37a94b9bc4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing query axiom preferences:   0%|          | 0/54 [00:00<?, ?query/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>docid_a</th>\n",
       "      <th>docno_a</th>\n",
       "      <th>rank_a</th>\n",
       "      <th>score_a</th>\n",
       "      <th>query</th>\n",
       "      <th>label_a</th>\n",
       "      <th>iteration_a</th>\n",
       "      <th>docid_b</th>\n",
       "      <th>docno_b</th>\n",
       "      <th>...</th>\n",
       "      <th>RS-PL2_preference</th>\n",
       "      <th>RS-QL_preference</th>\n",
       "      <th>TFC1_preference</th>\n",
       "      <th>TFC3_preference</th>\n",
       "      <th>M-TDC_preference</th>\n",
       "      <th>LEN-M-TDC_preference</th>\n",
       "      <th>STMC1_preference</th>\n",
       "      <th>STMC1-fastText_preference</th>\n",
       "      <th>STMC2_preference</th>\n",
       "      <th>STMC2-fastText_preference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8726433</td>\n",
       "      <td>8726433</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8726435</td>\n",
       "      <td>8726435</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8726429</td>\n",
       "      <td>8726429</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8726437</td>\n",
       "      <td>8726437</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5395</th>\n",
       "      <td>997622</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>9</td>\n",
       "      <td>30.023091</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4643397</td>\n",
       "      <td>4643397</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5396</th>\n",
       "      <td>997622</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>9</td>\n",
       "      <td>30.023091</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4518222</td>\n",
       "      <td>4518222</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5397</th>\n",
       "      <td>997622</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>9</td>\n",
       "      <td>30.023091</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4810071</td>\n",
       "      <td>4810071</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5398</th>\n",
       "      <td>997622</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>9</td>\n",
       "      <td>30.023091</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4558331</td>\n",
       "      <td>4558331</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5399</th>\n",
       "      <td>997622</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>9</td>\n",
       "      <td>30.023091</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7965342</td>\n",
       "      <td>7965342</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5400 rows × 53 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          qid  docid_a  docno_a  rank_a    score_a  \\\n",
       "0     1030303  8726436  8726436       0  54.354218   \n",
       "1     1030303  8726436  8726436       0  54.354218   \n",
       "2     1030303  8726436  8726436       0  54.354218   \n",
       "3     1030303  8726436  8726436       0  54.354218   \n",
       "4     1030303  8726436  8726436       0  54.354218   \n",
       "...       ...      ...      ...     ...        ...   \n",
       "5395   997622  7965342  7965342       9  30.023091   \n",
       "5396   997622  7965342  7965342       9  30.023091   \n",
       "5397   997622  7965342  7965342       9  30.023091   \n",
       "5398   997622  7965342  7965342       9  30.023091   \n",
       "5399   997622  7965342  7965342       9  30.023091   \n",
       "\n",
       "                                   query  label_a iteration_a  docid_b  \\\n",
       "0                     who is aziz hashim      3.0           0  8726436   \n",
       "1                     who is aziz hashim      3.0           0  8726433   \n",
       "2                     who is aziz hashim      3.0           0  8726435   \n",
       "3                     who is aziz hashim      3.0           0  8726429   \n",
       "4                     who is aziz hashim      3.0           0  8726437   \n",
       "...                                  ...      ...         ...      ...   \n",
       "5395  where is the show shameless filmed      0.0           0  4643397   \n",
       "5396  where is the show shameless filmed      0.0           0  4518222   \n",
       "5397  where is the show shameless filmed      0.0           0  4810071   \n",
       "5398  where is the show shameless filmed      0.0           0  4558331   \n",
       "5399  where is the show shameless filmed      0.0           0  7965342   \n",
       "\n",
       "      docno_b  ...  RS-PL2_preference  RS-QL_preference  TFC1_preference  \\\n",
       "0     8726436  ...                  0                 0                0   \n",
       "1     8726433  ...                  1                 1                0   \n",
       "2     8726435  ...                  1                 1                0   \n",
       "3     8726429  ...                  1                 1               -1   \n",
       "4     8726437  ...                  1                 1                0   \n",
       "...       ...  ...                ...               ...              ...   \n",
       "5395  4643397  ...                  1                -1                0   \n",
       "5396  4518222  ...                  1                -1                0   \n",
       "5397  4810071  ...                 -1                -1                0   \n",
       "5398  4558331  ...                  1                -1                0   \n",
       "5399  7965342  ...                  0                 0                0   \n",
       "\n",
       "     TFC3_preference  M-TDC_preference  LEN-M-TDC_preference  \\\n",
       "0                  0                 0                     0   \n",
       "1                  0                 0                     0   \n",
       "2                  0                 0                     0   \n",
       "3                  0                 0                     0   \n",
       "4                  0                 0                     0   \n",
       "...              ...               ...                   ...   \n",
       "5395               0                 0                     0   \n",
       "5396               0                 0                     0   \n",
       "5397               0                 0                     0   \n",
       "5398               0                 0                     0   \n",
       "5399               0                 0                     0   \n",
       "\n",
       "      STMC1_preference  STMC1-fastText_preference  STMC2_preference  \\\n",
       "0                    0                          0                 0   \n",
       "1                    1                          1                 0   \n",
       "2                    1                          1                 0   \n",
       "3                    1                          1                 0   \n",
       "4                   -1                          1                 0   \n",
       "...                ...                        ...               ...   \n",
       "5395                -1                          1                 0   \n",
       "5396                 1                          1                 0   \n",
       "5397                -1                          1                 0   \n",
       "5398                 1                          1                 0   \n",
       "5399                 0                          0                 0   \n",
       "\n",
       "      STMC2-fastText_preference  \n",
       "0                             0  \n",
       "1                             0  \n",
       "2                             0  \n",
       "3                             0  \n",
       "4                            -1  \n",
       "...                         ...  \n",
       "5395                         -1  \n",
       "5396                          0  \n",
       "5397                          0  \n",
       "5398                          0  \n",
       "5399                          0  \n",
       "\n",
       "[5400 rows x 53 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preferences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evaluate Inconsistent Document Pairs\n",
    "Investigate document pairs that are inconsistent with retrieval axioms (and relevance judgments).\n",
    "These can provide insights to improve the evaluated ranking function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_74100/445467560.py:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  experiment.inconsistent_pairs.mean()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "qid                                        inf\n",
       "docid_a                           4.859517e+06\n",
       "docno_a                                    inf\n",
       "rank_a                            6.326316e+00\n",
       "score_a                           3.505220e+01\n",
       "label_a                           2.223158e+00\n",
       "iteration_a                       0.000000e+00\n",
       "docid_b                           4.742632e+06\n",
       "docno_b                                    inf\n",
       "rank_b                            2.810526e+00\n",
       "score_b                           3.751082e+01\n",
       "label_b                           4.757895e-01\n",
       "iteration_b                       0.000000e+00\n",
       "ORIG_preference                  -1.000000e+00\n",
       "ORACLE_preference                 1.000000e+00\n",
       "ArgUC_preference                 -2.105263e-02\n",
       "QTArg_preference                 -4.842105e-02\n",
       "QTPArg_preference                 2.105263e-02\n",
       "aSL_preference                    2.105263e-03\n",
       "LNC1_preference                  -2.105263e-03\n",
       "TF-LNC_preference                 1.263158e-02\n",
       "LB1_preference                   -6.736842e-02\n",
       "PROX1_preference                  2.105263e-03\n",
       "PROX2_preference                  6.526316e-02\n",
       "PROX3_preference                 -2.947368e-02\n",
       "PROX4_preference                 -1.052632e-02\n",
       "PROX5_preference                 -6.105263e-02\n",
       "REG_preference                   -2.042105e-01\n",
       "REG-fastText_preference          -2.084211e-01\n",
       "ANTI-REG_preference              -1.368421e-01\n",
       "ANTI-REG-fastText_preference     -9.052632e-02\n",
       "ASPECT-REG_preference             0.000000e+00\n",
       "ASPECT-REG-fastText_preference    0.000000e+00\n",
       "AND_preference                   -6.315789e-02\n",
       "LEN-AND_preference               -2.105263e-03\n",
       "M-AND_preference                 -1.010526e-01\n",
       "LEN-M-AND_preference              6.315789e-03\n",
       "DIV_preference                    2.736842e-01\n",
       "LEN-DIV_preference                8.000000e-02\n",
       "RS-TF_preference                 -1.957895e-01\n",
       "RS-TF-IDF_preference             -9.831579e-01\n",
       "RS-BM25_preference               -9.957895e-01\n",
       "RS-PL2_preference                -4.778947e-01\n",
       "RS-QL_preference                 -2.757895e-01\n",
       "TFC1_preference                  -3.789474e-02\n",
       "TFC3_preference                   2.105263e-03\n",
       "M-TDC_preference                 -2.105263e-03\n",
       "LEN-M-TDC_preference             -2.105263e-03\n",
       "STMC1_preference                 -7.368421e-02\n",
       "STMC1-fastText_preference        -2.063158e-01\n",
       "STMC2_preference                  2.105263e-03\n",
       "STMC2-fastText_preference         4.210526e-03\n",
       "dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.inconsistent_pairs.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evaluate Consistency of Runs with Axioms\n",
    "This evaluation shows how consistent axioms are with the run's original order (ORIG) and the relevance judgments (ORACLE).\n",
    "High consistency with the ORACLE axiom indicates that for your run, the axiom captures the relevance judgments well.\n",
    "High consistency with the ORIG axiom indicates that your run adheres to the axiom's constraint well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>axiom</th>\n",
       "      <th>ORIG_consistency</th>\n",
       "      <th>ORACLE_consistency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ArgUC</td>\n",
       "      <td>0.494475</td>\n",
       "      <td>0.494505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QTArg</td>\n",
       "      <td>0.662539</td>\n",
       "      <td>0.538922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>QTPArg</td>\n",
       "      <td>0.591623</td>\n",
       "      <td>0.673367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aSL</td>\n",
       "      <td>0.462185</td>\n",
       "      <td>0.531746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LNC1</td>\n",
       "      <td>0.578947</td>\n",
       "      <td>0.562500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>TF-LNC</td>\n",
       "      <td>0.580420</td>\n",
       "      <td>0.613333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>LB1</td>\n",
       "      <td>0.664957</td>\n",
       "      <td>0.625352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>PROX1</td>\n",
       "      <td>0.568596</td>\n",
       "      <td>0.605611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PROX2</td>\n",
       "      <td>0.592907</td>\n",
       "      <td>0.634740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PROX3</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.482759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>PROX4</td>\n",
       "      <td>0.618863</td>\n",
       "      <td>0.609195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>PROX5</td>\n",
       "      <td>0.617896</td>\n",
       "      <td>0.567982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>REG</td>\n",
       "      <td>0.607470</td>\n",
       "      <td>0.526520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>REG-fastText</td>\n",
       "      <td>0.591817</td>\n",
       "      <td>0.518106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ANTI-REG</td>\n",
       "      <td>0.614934</td>\n",
       "      <td>0.544321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ANTI-REG-fastText</td>\n",
       "      <td>0.577869</td>\n",
       "      <td>0.544534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ASPECT-REG</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ASPECT-REG-fastText</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>AND</td>\n",
       "      <td>0.812950</td>\n",
       "      <td>0.752809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>LEN-AND</td>\n",
       "      <td>0.811881</td>\n",
       "      <td>0.813559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>M-AND</td>\n",
       "      <td>0.799745</td>\n",
       "      <td>0.737288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>LEN-M-AND</td>\n",
       "      <td>0.821429</td>\n",
       "      <td>0.862500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>DIV</td>\n",
       "      <td>0.380179</td>\n",
       "      <td>0.475954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>LEN-DIV</td>\n",
       "      <td>0.330357</td>\n",
       "      <td>0.491379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>RS-TF</td>\n",
       "      <td>0.615836</td>\n",
       "      <td>0.515094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>RS-TF-IDF</td>\n",
       "      <td>0.992139</td>\n",
       "      <td>0.619775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>RS-BM25</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.619775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RS-PL2</td>\n",
       "      <td>0.708730</td>\n",
       "      <td>0.496785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RS-QL</td>\n",
       "      <td>0.687216</td>\n",
       "      <td>0.579582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>TFC1</td>\n",
       "      <td>0.627119</td>\n",
       "      <td>0.531646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>TFC3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>M-TDC</td>\n",
       "      <td>0.625000</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>LEN-M-TDC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>STMC1</td>\n",
       "      <td>0.530750</td>\n",
       "      <td>0.487264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>STMC1-fastText</td>\n",
       "      <td>0.567590</td>\n",
       "      <td>0.495582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>STMC2</td>\n",
       "      <td>0.560748</td>\n",
       "      <td>0.537190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>STMC2-fastText</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.515152</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  axiom  ORIG_consistency  ORACLE_consistency\n",
       "0                 ArgUC          0.494475            0.494505\n",
       "1                 QTArg          0.662539            0.538922\n",
       "2                QTPArg          0.591623            0.673367\n",
       "3                   aSL          0.462185            0.531746\n",
       "4                  LNC1          0.578947            0.562500\n",
       "5                TF-LNC          0.580420            0.613333\n",
       "6                   LB1          0.664957            0.625352\n",
       "7                 PROX1          0.568596            0.605611\n",
       "8                 PROX2          0.592907            0.634740\n",
       "9                 PROX3          0.666667            0.482759\n",
       "10                PROX4          0.618863            0.609195\n",
       "11                PROX5          0.617896            0.567982\n",
       "12                  REG          0.607470            0.526520\n",
       "13         REG-fastText          0.591817            0.518106\n",
       "14             ANTI-REG          0.614934            0.544321\n",
       "15    ANTI-REG-fastText          0.577869            0.544534\n",
       "16           ASPECT-REG               NaN                 NaN\n",
       "17  ASPECT-REG-fastText               NaN                 NaN\n",
       "18                  AND          0.812950            0.752809\n",
       "19              LEN-AND          0.811881            0.813559\n",
       "20                M-AND          0.799745            0.737288\n",
       "21            LEN-M-AND          0.821429            0.862500\n",
       "22                  DIV          0.380179            0.475954\n",
       "23              LEN-DIV          0.330357            0.491379\n",
       "24                RS-TF          0.615836            0.515094\n",
       "25            RS-TF-IDF          0.992139            0.619775\n",
       "26              RS-BM25          1.000000            0.619775\n",
       "27               RS-PL2          0.708730            0.496785\n",
       "28                RS-QL          0.687216            0.579582\n",
       "29                 TFC1          0.627119            0.531646\n",
       "30                 TFC3          0.000000            0.500000\n",
       "31                M-TDC          0.625000            0.666667\n",
       "32            LEN-M-TDC          1.000000            0.500000\n",
       "33                STMC1          0.530750            0.487264\n",
       "34       STMC1-fastText          0.567590            0.495582\n",
       "35                STMC2          0.560748            0.537190\n",
       "36       STMC2-fastText          0.500000            0.515152"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preference_consistency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evaluate Preference Distribution\n",
    "Calculating the distribution of each axiom's preferences compared to the original ranking order (ORIG) can help identify decisive axioms for your dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>axiom</th>\n",
       "      <th>axiom == 0</th>\n",
       "      <th>axiom == ORIG</th>\n",
       "      <th>axiom != ORIG</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ArgUC</td>\n",
       "      <td>2068</td>\n",
       "      <td>179</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QTArg</td>\n",
       "      <td>2107</td>\n",
       "      <td>214</td>\n",
       "      <td>109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>QTPArg</td>\n",
       "      <td>2048</td>\n",
       "      <td>226</td>\n",
       "      <td>156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aSL</td>\n",
       "      <td>2192</td>\n",
       "      <td>110</td>\n",
       "      <td>128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LNC1</td>\n",
       "      <td>2373</td>\n",
       "      <td>33</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>TF-LNC</td>\n",
       "      <td>2287</td>\n",
       "      <td>83</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>LB1</td>\n",
       "      <td>1845</td>\n",
       "      <td>389</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>PROX1</td>\n",
       "      <td>1169</td>\n",
       "      <td>717</td>\n",
       "      <td>544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PROX2</td>\n",
       "      <td>1133</td>\n",
       "      <td>769</td>\n",
       "      <td>528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PROX3</td>\n",
       "      <td>2340</td>\n",
       "      <td>60</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>PROX4</td>\n",
       "      <td>1656</td>\n",
       "      <td>479</td>\n",
       "      <td>295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>PROX5</td>\n",
       "      <td>1603</td>\n",
       "      <td>511</td>\n",
       "      <td>316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>REG</td>\n",
       "      <td>1011</td>\n",
       "      <td>862</td>\n",
       "      <td>557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>REG-fastText</td>\n",
       "      <td>328</td>\n",
       "      <td>1244</td>\n",
       "      <td>858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ANTI-REG</td>\n",
       "      <td>1064</td>\n",
       "      <td>840</td>\n",
       "      <td>526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ANTI-REG-fastText</td>\n",
       "      <td>478</td>\n",
       "      <td>1128</td>\n",
       "      <td>824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ASPECT-REG</td>\n",
       "      <td>2430</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ASPECT-REG-fastText</td>\n",
       "      <td>2430</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>AND</td>\n",
       "      <td>1874</td>\n",
       "      <td>452</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>LEN-AND</td>\n",
       "      <td>2329</td>\n",
       "      <td>82</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>M-AND</td>\n",
       "      <td>1646</td>\n",
       "      <td>627</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>LEN-M-AND</td>\n",
       "      <td>2290</td>\n",
       "      <td>115</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>DIV</td>\n",
       "      <td>89</td>\n",
       "      <td>890</td>\n",
       "      <td>1451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>LEN-DIV</td>\n",
       "      <td>1982</td>\n",
       "      <td>148</td>\n",
       "      <td>300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>RS-TF</td>\n",
       "      <td>384</td>\n",
       "      <td>1260</td>\n",
       "      <td>786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>RS-TF-IDF</td>\n",
       "      <td>13</td>\n",
       "      <td>2398</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>RS-BM25</td>\n",
       "      <td>13</td>\n",
       "      <td>2417</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RS-PL2</td>\n",
       "      <td>13</td>\n",
       "      <td>1713</td>\n",
       "      <td>704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RS-QL</td>\n",
       "      <td>13</td>\n",
       "      <td>1661</td>\n",
       "      <td>756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>TFC1</td>\n",
       "      <td>2135</td>\n",
       "      <td>185</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>TFC3</td>\n",
       "      <td>2428</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>M-TDC</td>\n",
       "      <td>2422</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>LEN-M-TDC</td>\n",
       "      <td>2428</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>STMC1</td>\n",
       "      <td>56</td>\n",
       "      <td>1260</td>\n",
       "      <td>1114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>STMC1-fastText</td>\n",
       "      <td>11</td>\n",
       "      <td>1373</td>\n",
       "      <td>1046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>STMC2</td>\n",
       "      <td>2216</td>\n",
       "      <td>120</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>STMC2-fastText</td>\n",
       "      <td>2124</td>\n",
       "      <td>153</td>\n",
       "      <td>153</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  axiom  axiom == 0  axiom == ORIG  axiom != ORIG\n",
       "0                 ArgUC        2068            179            183\n",
       "1                 QTArg        2107            214            109\n",
       "2                QTPArg        2048            226            156\n",
       "3                   aSL        2192            110            128\n",
       "4                  LNC1        2373             33             24\n",
       "5                TF-LNC        2287             83             60\n",
       "6                   LB1        1845            389            196\n",
       "7                 PROX1        1169            717            544\n",
       "8                 PROX2        1133            769            528\n",
       "9                 PROX3        2340             60             30\n",
       "10                PROX4        1656            479            295\n",
       "11                PROX5        1603            511            316\n",
       "12                  REG        1011            862            557\n",
       "13         REG-fastText         328           1244            858\n",
       "14             ANTI-REG        1064            840            526\n",
       "15    ANTI-REG-fastText         478           1128            824\n",
       "16           ASPECT-REG        2430              0              0\n",
       "17  ASPECT-REG-fastText        2430              0              0\n",
       "18                  AND        1874            452            104\n",
       "19              LEN-AND        2329             82             19\n",
       "20                M-AND        1646            627            157\n",
       "21            LEN-M-AND        2290            115             25\n",
       "22                  DIV          89            890           1451\n",
       "23              LEN-DIV        1982            148            300\n",
       "24                RS-TF         384           1260            786\n",
       "25            RS-TF-IDF          13           2398             19\n",
       "26              RS-BM25          13           2417              0\n",
       "27               RS-PL2          13           1713            704\n",
       "28                RS-QL          13           1661            756\n",
       "29                 TFC1        2135            185            110\n",
       "30                 TFC3        2428              0              2\n",
       "31                M-TDC        2422              5              3\n",
       "32            LEN-M-TDC        2428              2              0\n",
       "33                STMC1          56           1260           1114\n",
       "34       STMC1-fastText          11           1373           1046\n",
       "35                STMC2        2216            120             94\n",
       "36       STMC2-fastText        2124            153            153"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preference_distribution"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}