opr-error-modelling.ipynb

{
 "nbformat": 4,
 "nbformat_minor": 2,
 "metadata": {
  "language_info": {
   "name": "python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "version": "3.8.1-final"
  },
  "orig_nbformat": 2,
  "file_extension": ".py",
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3,
  "kernelspec": {
   "name": "python38164bit4481320a91e545d29888e4d0a4bc66ea",
   "display_name": "Python 3.8.1 64-bit"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In here we're going to explore a few different methods for extracting standard error from the component OPR model to see which ones hold up.\n",
    "\n",
    "This is _essential_ to creating any kind of model that we'll use to get match score predictions, as we need to know what the match score confidence interval looks like to say by what margin a team will win by.\n",
    "\n",
    "(I'd assume that in the future for that model we'll end up using some kind of \"sliding confidence interval\" to see at what point the intervals stop overlapping to determine a match victor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(a lot of this is transcribed from some research/brainstorming i've been doing)\n",
    "\n",
    "The main leads that I have for determining the error are:\n",
    "- Using some kind of bootstrapping to simulate matches and find the standard error of the _overall_ OPR model\n",
    "- Using the fact that $\\frac{1}{n}\\sqrt{\\Sigma ^n _{i=0} (O_i - AP)^2} = SE$ to find the standard error for a _single team_ -- (O is observed match score, AP is match score prediction from model (incidence * oPr))\n",
    "- Taking the above method and repeating it for all the teams, tuning the model being made as it goes, to eventaully get a better tuned model (and hope that it converges!) -- this seems a bit like an ML technique\n",
    "- Looking at the mean of the standard error for the OPR fit/residuals when doing the matrix math in order to get a general uncertainty for each team\n",
    "- A poor man's approach to calculating OPR (as outlined by [wgardner](https://www.chiefdelphi.com/t/standard-error-of-opr-values/144363/20)), which basically entails finding the variance and standard deviation of a team's OPR while omitting every match, one match at a time. This seems like less of a \"standard error\" metric, but still one that can be used to estimate match score variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first method that I'll be looking at is **wgardner's method**, as that seems like it'll be the simplest to implement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataLoader import load_data_event\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import json\n",
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>match_key</th>\n      <th>match_type</th>\n      <th>match_number</th>\n      <th>blue_1_key</th>\n      <th>blue_2_key</th>\n      <th>blue_3_key</th>\n      <th>blue_keys</th>\n      <th>blue_endgame_level</th>\n      <th>blue_foul_count</th>\n      <th>blue_points_scored</th>\n      <th>...</th>\n      <th>red_3_init_line</th>\n      <th>red_1_endgame</th>\n      <th>red_2_endgame</th>\n      <th>red_3_endgame</th>\n      <th>red_cells_bottom_auto</th>\n      <th>red_cells_bottom_teleop</th>\n      <th>red_cells_outer_auto</th>\n      <th>red_cells_outer_teleop</th>\n      <th>red_cells_inner_auto</th>\n      <th>red_cells_inner_teleop</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>11</th>\n      <td>2020ncwak_qm1</td>\n      <td>qm</td>\n      <td>1</td>\n      <td>frc435</td>\n      <td>frc6565</td>\n      <td>frc5511</td>\n      <td>[frc435, frc6565, frc5511]</td>\n      <td>True</td>\n      <td>1</td>\n      <td>81</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>2020ncwak_qm10</td>\n      <td>qm</td>\n      <td>10</td>\n      <td>frc5919</td>\n      <td>frc7265</td>\n      <td>frc4291</td>\n      <td>[frc5919, frc7265, frc4291]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>10</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>2020ncwak_qm11</td>\n      <td>qm</td>\n      <td>11</td>\n      <td>frc435</td>\n      <td>frc6240</td>\n      <td>frc6502</td>\n      <td>[frc435, frc6240, frc6502]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>73</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>9</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>2020ncwak_qm12</td>\n      <td>qm</td>\n      <td>12</td>\n      <td>frc3459</td>\n      <td>frc4828</td>\n      <td>frc5190</td>\n      <td>[frc3459, frc4828, frc5190]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>107</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>2020ncwak_qm13</td>\n      <td>qm</td>\n      <td>13</td>\n      <td>frc7890</td>\n      <td>frc5518</td>\n      <td>frc5511</td>\n      <td>[frc7890, frc5518, frc5511]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>92</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>4</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>2020ncwak_qm14</td>\n      <td>qm</td>\n      <td>14</td>\n      <td>frc5607</td>\n      <td>frc5160</td>\n      <td>frc2642</td>\n      <td>[frc5607, frc5160, frc2642]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>27</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>2020ncwak_qm15</td>\n      <td>qm</td>\n      <td>15</td>\n      <td>frc6500</td>\n      <td>frc3459</td>\n      <td>frc6496</td>\n      <td>[frc6500, frc3459, frc6496]</td>\n      <td>False</td>\n      <td>1</td>\n      <td>59</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>2020ncwak_qm16</td>\n      <td>qm</td>\n      <td>16</td>\n      <td>frc2059</td>\n      <td>frc5919</td>\n      <td>frc5762</td>\n      <td>[frc2059, frc5919, frc5762]</td>\n      <td>False</td>\n      <td>4</td>\n      <td>37</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>1</td>\n      <td>5</td>\n      <td>2</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>2020ncwak_qm17</td>\n      <td>qm</td>\n      <td>17</td>\n      <td>frc8090</td>\n      <td>frc7890</td>\n      <td>frc3229</td>\n      <td>[frc8090, frc7890, frc3229]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>50</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>2020ncwak_qm18</td>\n      <td>qm</td>\n      <td>18</td>\n      <td>frc7671</td>\n      <td>frc5160</td>\n      <td>frc7463</td>\n      <td>[frc7671, frc5160, frc7463]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>54</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>2020ncwak_qm19</td>\n      <td>qm</td>\n      <td>19</td>\n      <td>frc4816</td>\n      <td>frc4828</td>\n      <td>frc6500</td>\n      <td>[frc4816, frc4828, frc6500]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>60</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>13</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>2020ncwak_qm2</td>\n      <td>qm</td>\n      <td>2</td>\n      <td>frc6496</td>\n      <td>frc8090</td>\n      <td>frc2059</td>\n      <td>[frc6496, frc8090, frc2059]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>35</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>3</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>23</th>\n      <td>2020ncwak_qm20</td>\n      <td>qm</td>\n      <td>20</td>\n      <td>frc5190</td>\n      <td>frc5511</td>\n      <td>frc8090</td>\n      <td>[frc5190, frc5511, frc8090]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>69</td>\n      <td>...</td>\n      <td>0</td>\n      <td>25</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>3</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>2020ncwak_qm21</td>\n      <td>qm</td>\n      <td>21</td>\n      <td>frc4816</td>\n      <td>frc5607</td>\n      <td>frc7463</td>\n      <td>[frc4816, frc5607, frc7463]</td>\n      <td>False</td>\n      <td>1</td>\n      <td>24</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>25</th>\n      <td>2020ncwak_qm22</td>\n      <td>qm</td>\n      <td>22</td>\n      <td>frc7671</td>\n      <td>frc6500</td>\n      <td>frc4828</td>\n      <td>[frc7671, frc6500, frc4828]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>42</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>25</td>\n      <td>25</td>\n      <td>3</td>\n      <td>11</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>26</th>\n      <td>2020ncwak_qm23</td>\n      <td>qm</td>\n      <td>23</td>\n      <td>frc7890</td>\n      <td>frc5762</td>\n      <td>frc6565</td>\n      <td>[frc7890, frc5762, frc6565]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>41</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>2</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>27</th>\n      <td>2020ncwak_qm24</td>\n      <td>qm</td>\n      <td>24</td>\n      <td>frc6908</td>\n      <td>frc3459</td>\n      <td>frc5518</td>\n      <td>[frc6908, frc3459, frc5518]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>61</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>4</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>28</th>\n      <td>2020ncwak_qm25</td>\n      <td>qm</td>\n      <td>25</td>\n      <td>frc6502</td>\n      <td>frc7890</td>\n      <td>frc2642</td>\n      <td>[frc6502, frc7890, frc2642]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>86</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>25</td>\n      <td>3</td>\n      <td>12</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>29</th>\n      <td>2020ncwak_qm26</td>\n      <td>qm</td>\n      <td>26</td>\n      <td>frc7763</td>\n      <td>frc5919</td>\n      <td>frc5607</td>\n      <td>[frc7763, frc5919, frc5607]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>19</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>30</th>\n      <td>2020ncwak_qm27</td>\n      <td>qm</td>\n      <td>27</td>\n      <td>frc4561</td>\n      <td>frc6502</td>\n      <td>frc4816</td>\n      <td>[frc4561, frc6502, frc4816]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>114</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>2</td>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>31</th>\n      <td>2020ncwak_qm28</td>\n      <td>qm</td>\n      <td>28</td>\n      <td>frc4291</td>\n      <td>frc5511</td>\n      <td>frc6496</td>\n      <td>[frc4291, frc5511, frc6496]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>59</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>32</th>\n      <td>2020ncwak_qm29</td>\n      <td>qm</td>\n      <td>29</td>\n      <td>frc4828</td>\n      <td>frc6908</td>\n      <td>frc6004</td>\n      <td>[frc4828, frc6908, frc6004]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>114</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>5</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>33</th>\n      <td>2020ncwak_qm3</td>\n      <td>qm</td>\n      <td>3</td>\n      <td>frc4828</td>\n      <td>frc5160</td>\n      <td>frc6240</td>\n      <td>[frc4828, frc5160, frc6240]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>63</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>34</th>\n      <td>2020ncwak_qm30</td>\n      <td>qm</td>\n      <td>30</td>\n      <td>frc7265</td>\n      <td>frc7890</td>\n      <td>frc7671</td>\n      <td>[frc7265, frc7890, frc7671]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>44</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>7</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>35</th>\n      <td>2020ncwak_qm31</td>\n      <td>qm</td>\n      <td>31</td>\n      <td>frc5160</td>\n      <td>frc2059</td>\n      <td>frc4561</td>\n      <td>[frc5160, frc2059, frc4561]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>86</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>36</th>\n      <td>2020ncwak_qm32</td>\n      <td>qm</td>\n      <td>32</td>\n      <td>frc7463</td>\n      <td>frc6908</td>\n      <td>frc4291</td>\n      <td>[frc7463, frc6908, frc4291]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>39</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>37</th>\n      <td>2020ncwak_qm33</td>\n      <td>qm</td>\n      <td>33</td>\n      <td>frc3229</td>\n      <td>frc4816</td>\n      <td>frc435</td>\n      <td>[frc3229, frc4816, frc435]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>52</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>13</td>\n      <td>1</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>38</th>\n      <td>2020ncwak_qm34</td>\n      <td>qm</td>\n      <td>34</td>\n      <td>frc6565</td>\n      <td>frc3459</td>\n      <td>frc6502</td>\n      <td>[frc6565, frc3459, frc6502]</td>\n      <td>True</td>\n      <td>1</td>\n      <td>41</td>\n      <td>...</td>\n      <td>0</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>39</th>\n      <td>2020ncwak_qm35</td>\n      <td>qm</td>\n      <td>35</td>\n      <td>frc5190</td>\n      <td>frc2059</td>\n      <td>frc6908</td>\n      <td>[frc5190, frc2059, frc6908]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>35</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>5</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>40</th>\n      <td>2020ncwak_qm36</td>\n      <td>qm</td>\n      <td>36</td>\n      <td>frc3459</td>\n      <td>frc7763</td>\n      <td>frc2642</td>\n      <td>[frc3459, frc7763, frc2642]</td>\n      <td>True</td>\n      <td>2</td>\n      <td>95</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>11</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>41</th>\n      <td>2020ncwak_qm37</td>\n      <td>qm</td>\n      <td>37</td>\n      <td>frc8090</td>\n      <td>frc435</td>\n      <td>frc5160</td>\n      <td>[frc8090, frc435, frc5160]</td>\n      <td>True</td>\n      <td>1</td>\n      <td>100</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>3</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>42</th>\n      <td>2020ncwak_qm38</td>\n      <td>qm</td>\n      <td>38</td>\n      <td>frc4291</td>\n      <td>frc6565</td>\n      <td>frc6004</td>\n      <td>[frc4291, frc6565, frc6004]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>55</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>43</th>\n      <td>2020ncwak_qm39</td>\n      <td>qm</td>\n      <td>39</td>\n      <td>frc7463</td>\n      <td>frc4561</td>\n      <td>frc5607</td>\n      <td>[frc7463, frc4561, frc5607]</td>\n      <td>True</td>\n      <td>3</td>\n      <td>80</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>4</td>\n      <td>10</td>\n      <td>3</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>44</th>\n      <td>2020ncwak_qm4</td>\n      <td>qm</td>\n      <td>4</td>\n      <td>frc5518</td>\n      <td>frc7265</td>\n      <td>frc5762</td>\n      <td>[frc5518, frc7265, frc5762]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>20</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>45</th>\n      <td>2020ncwak_qm40</td>\n      <td>qm</td>\n      <td>40</td>\n      <td>frc6502</td>\n      <td>frc5518</td>\n      <td>frc2059</td>\n      <td>[frc6502, frc5518, frc2059]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>45</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>46</th>\n      <td>2020ncwak_qm41</td>\n      <td>qm</td>\n      <td>41</td>\n      <td>frc6496</td>\n      <td>frc3229</td>\n      <td>frc7463</td>\n      <td>[frc6496, frc3229, frc7463]</td>\n      <td>False</td>\n      <td>1</td>\n      <td>32</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>5</td>\n      <td>3</td>\n      <td>13</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>47</th>\n      <td>2020ncwak_qm42</td>\n      <td>qm</td>\n      <td>42</td>\n      <td>frc8090</td>\n      <td>frc6908</td>\n      <td>frc6500</td>\n      <td>[frc8090, frc6908, frc6500]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>89</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>48</th>\n      <td>2020ncwak_qm43</td>\n      <td>qm</td>\n      <td>43</td>\n      <td>frc6004</td>\n      <td>frc5190</td>\n      <td>frc7265</td>\n      <td>[frc6004, frc5190, frc7265]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>62</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>6</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>49</th>\n      <td>2020ncwak_qm44</td>\n      <td>qm</td>\n      <td>44</td>\n      <td>frc4291</td>\n      <td>frc7763</td>\n      <td>frc6502</td>\n      <td>[frc4291, frc7763, frc6502]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>106</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>25</td>\n      <td>2</td>\n      <td>12</td>\n      <td>2</td>\n      <td>9</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>50</th>\n      <td>2020ncwak_qm45</td>\n      <td>qm</td>\n      <td>45</td>\n      <td>frc5518</td>\n      <td>frc5190</td>\n      <td>frc5919</td>\n      <td>[frc5518, frc5190, frc5919]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>45</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>51</th>\n      <td>2020ncwak_qm46</td>\n      <td>qm</td>\n      <td>46</td>\n      <td>frc5511</td>\n      <td>frc7671</td>\n      <td>frc5607</td>\n      <td>[frc5511, frc7671, frc5607]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>62</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>3</td>\n      <td>0</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>52</th>\n      <td>2020ncwak_qm47</td>\n      <td>qm</td>\n      <td>47</td>\n      <td>frc5762</td>\n      <td>frc4561</td>\n      <td>frc6496</td>\n      <td>[frc5762, frc4561, frc6496]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>58</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>53</th>\n      <td>2020ncwak_qm48</td>\n      <td>qm</td>\n      <td>48</td>\n      <td>frc6500</td>\n      <td>frc435</td>\n      <td>frc7265</td>\n      <td>[frc6500, frc435, frc7265]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>24</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>54</th>\n      <td>2020ncwak_qm49</td>\n      <td>qm</td>\n      <td>49</td>\n      <td>frc6565</td>\n      <td>frc5518</td>\n      <td>frc5607</td>\n      <td>[frc6565, frc5518, frc5607]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>39</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>6</td>\n      <td>2</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2020ncwak_qm5</td>\n      <td>qm</td>\n      <td>5</td>\n      <td>frc5919</td>\n      <td>frc6502</td>\n      <td>frc3229</td>\n      <td>[frc5919, frc6502, frc3229]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>84</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>1</td>\n      <td>3</td>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>56</th>\n      <td>2020ncwak_qm50</td>\n      <td>qm</td>\n      <td>50</td>\n      <td>frc7763</td>\n      <td>frc3229</td>\n      <td>frc6908</td>\n      <td>[frc7763, frc3229, frc6908]</td>\n      <td>True</td>\n      <td>1</td>\n      <td>47</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>25</td>\n      <td>2</td>\n      <td>10</td>\n      <td>3</td>\n      <td>7</td>\n      <td>0</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>57</th>\n      <td>2020ncwak_qm51</td>\n      <td>qm</td>\n      <td>51</td>\n      <td>frc7265</td>\n      <td>frc7463</td>\n      <td>frc6240</td>\n      <td>[frc7265, frc7463, frc6240]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>24</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>25</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>7</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>58</th>\n      <td>2020ncwak_qm52</td>\n      <td>qm</td>\n      <td>52</td>\n      <td>frc5511</td>\n      <td>frc2642</td>\n      <td>frc4816</td>\n      <td>[frc5511, frc2642, frc4816]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>73</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>25</td>\n      <td>3</td>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>59</th>\n      <td>2020ncwak_qm53</td>\n      <td>qm</td>\n      <td>53</td>\n      <td>frc6004</td>\n      <td>frc6496</td>\n      <td>frc5919</td>\n      <td>[frc6004, frc6496, frc5919]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>68</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>9</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>60</th>\n      <td>2020ncwak_qm54</td>\n      <td>qm</td>\n      <td>54</td>\n      <td>frc6565</td>\n      <td>frc7671</td>\n      <td>frc4561</td>\n      <td>[frc6565, frc7671, frc4561]</td>\n      <td>True</td>\n      <td>2</td>\n      <td>79</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>61</th>\n      <td>2020ncwak_qm55</td>\n      <td>qm</td>\n      <td>55</td>\n      <td>frc6240</td>\n      <td>frc4291</td>\n      <td>frc2642</td>\n      <td>[frc6240, frc4291, frc2642]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>69</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>62</th>\n      <td>2020ncwak_qm56</td>\n      <td>qm</td>\n      <td>56</td>\n      <td>frc2059</td>\n      <td>frc435</td>\n      <td>frc4828</td>\n      <td>[frc2059, frc435, frc4828]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>52</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>4</td>\n      <td>12</td>\n      <td>2</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <th>63</th>\n      <td>2020ncwak_qm57</td>\n      <td>qm</td>\n      <td>57</td>\n      <td>frc6496</td>\n      <td>frc7890</td>\n      <td>frc5160</td>\n      <td>[frc6496, frc7890, frc5160]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>98</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>64</th>\n      <td>2020ncwak_qm58</td>\n      <td>qm</td>\n      <td>58</td>\n      <td>frc7763</td>\n      <td>frc6004</td>\n      <td>frc5762</td>\n      <td>[frc7763, frc6004, frc5762]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>73</td>\n      <td>...</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>4</td>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>65</th>\n      <td>2020ncwak_qm6</td>\n      <td>qm</td>\n      <td>6</td>\n      <td>frc5190</td>\n      <td>frc5762</td>\n      <td>frc6240</td>\n      <td>[frc5190, frc5762, frc6240]</td>\n      <td>True</td>\n      <td>4</td>\n      <td>32</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>25</td>\n      <td>0</td>\n      <td>11</td>\n      <td>1</td>\n      <td>8</td>\n      <td>2</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>66</th>\n      <td>2020ncwak_qm7</td>\n      <td>qm</td>\n      <td>7</td>\n      <td>frc6908</td>\n      <td>frc2642</td>\n      <td>frc6565</td>\n      <td>[frc6908, frc2642, frc6565]</td>\n      <td>True</td>\n      <td>0</td>\n      <td>37</td>\n      <td>...</td>\n      <td>5</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>67</th>\n      <td>2020ncwak_qm8</td>\n      <td>qm</td>\n      <td>8</td>\n      <td>frc4561</td>\n      <td>frc4828</td>\n      <td>frc7890</td>\n      <td>[frc4561, frc4828, frc7890]</td>\n      <td>False</td>\n      <td>1</td>\n      <td>45</td>\n      <td>...</td>\n      <td>5</td>\n      <td>25</td>\n      <td>0</td>\n      <td>5</td>\n      <td>0</td>\n      <td>6</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>68</th>\n      <td>2020ncwak_qm9</td>\n      <td>qm</td>\n      <td>9</td>\n      <td>frc6004</td>\n      <td>frc5607</td>\n      <td>frc2059</td>\n      <td>[frc6004, frc5607, frc2059]</td>\n      <td>False</td>\n      <td>0</td>\n      <td>40</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>5</td>\n      <td>0</td>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>58 rows × 47 columns</p>\n</div>",
      "text/plain": "   frc6908   \n57  2020ncwak_qm51         qm            51    frc7265    frc7463    frc6240   \n58  2020ncwak_qm52         qm            52    frc5511    frc2642    frc4816   \n59  2020ncwak_qm53         qm            53    frc6004    frc6496    frc5919   \n60  2020ncwak_qm54         qm            54    frc6565    frc7671    frc4561   \n61  2020ncwak_qm55         qm            55    frc6240    frc4291    frc2642   \n62  2020ncwak_qm56         qm            56    frc2059     frc435    frc4828   \n63  2020ncwak_qm57         qm            57    frc6496    frc7890    frc5160   \n64  2020ncwak_qm58         qm            58    frc7763    frc6004    frc5762   \n65   2020ncwak_qm6         qm             6    frc5190    frc5762    frc6240   \n66   2020ncwak_qm7         qm             7    frc6908    frc2642    frc6565   \n67   2020ncwak_qm8         qm             8    frc4561    frc4828    frc7890   \n68   2020ncwak_qm9         qm             9    frc6004    frc5607    frc2059   \n\n                      blue_keys  blue_endgame_level  blue_foul_count  \\\n11   [frc435, frc6565, frc5511]                True                1   \n12  [frc5919, frc7265, frc4291]                True                0   \n13   [frc435, frc6240, frc6502]               False                0   \n14  [frc3459, frc4828, frc5190]               False                0   \n15  [frc7890, frc5518, frc5511]                True                0   \n16  [frc5607, frc5160, frc2642]                True                0   \n17  [frc6500, frc3459, frc6496]               False                1   \n18  [frc2059, frc5919, frc5762]               False                4   \n19  [frc8090, frc7890, frc3229]                True                0   \n20  [frc7671, frc5160, frc7463]               False                0   \n21  [frc4816, frc4828, frc6500]                True                0   \n22  [frc6496, frc8090, frc2059]                True                0   \n23  [frc5190, frc5511, frc8090]               False                0   \n24  [frc4816, frc5607, frc7463]               False                1   \n25  [frc7671, frc6500, frc4828]               False                0   \n26  [frc7890, frc5762, frc6565]                True                0   \n27  [frc6908, frc3459, frc5518]               False                0   \n28  [frc6502, frc7890, frc2642]               False                0   \n29  [frc7763, frc5919, frc5607]               False                0   \n30  [frc4561, frc6502, frc4816]                True                0   \n31  [frc4291, frc5511, frc6496]               False                0   \n32  [frc4828, frc6908, frc6004]                True                0   \n33  [frc4828, frc5160, frc6240]                True                0   \n34  [frc7265, frc7890, frc7671]               False                0   \n35  [frc5160, frc2059, frc4561]               False                0   \n36  [frc7463, frc6908, frc4291]                True                0   \n37   [frc3229, frc4816, frc435]               False                0   \n38  [frc6565, frc3459, frc6502]                True                1   \n39  [frc5190, frc2059, frc6908]               False                0   \n40  [frc3459, frc7763, frc2642]                True                2   \n41   [frc8090, frc435, frc5160]                True                1   \n42  [frc4291, frc6565, frc6004]               False                0   \n43  [frc7463, frc4561, frc5607]                True                3   \n44  [frc5518, frc7265, frc5762]                True                0   \n45  [frc6502, frc5518, frc2059]               False                0   \n46  [frc6496, frc3229, frc7463]               False                1   \n47  [frc8090, frc6908, frc6500]                True                0   \n48  [frc6004, frc5190, frc7265]                True                0   \n49  [frc4291, frc7763, frc6502]                True                0   \n50  [frc5518, frc5190, frc5919]               False                0   \n51  [frc5511, frc7671, frc5607]               False                0   \n52  [frc5762, frc4561, frc6496]               False                0   \n53   [frc6500, frc435, frc7265]               False                0   \n54  [frc6565, frc5518, frc5607]                True                0   \n55  [frc5919, frc6502, frc3229]                True                0   \n56  [frc7763, frc3229, frc6908]                True                1   \n57  [frc7265, frc7463, frc6240]                True                0   \n58  [frc5511, frc2642, frc4816]               False                0   \n59  [frc6004, frc6496, frc5919]               False                0   \n60  [frc6565, frc7671, frc4561]                True                2   \n61  [frc6240, frc4291, frc2642]                True                0   \n62   [frc2059, frc435, frc4828]                True                0   \n63  [frc6496, frc7890, frc5160]               False                0   \n64  [frc7763, frc6004, frc5762]                True                0   \n65  [frc5190, frc5762, frc6240]                True                4   \n66  [frc6908, frc2642, frc6565]                True                0   \n67  [frc4561, frc4828, frc7890]               False                1   \n68  [frc6004, frc5607, frc2059]               False                0   \n\n    blue_points_scored  ...  red_3_init_line  red_1_endgame  red_2_endgame  \\\n11                  81  ...                5              0              5   \n12                  10  ...                5              0              0   \n13                  73  ...                5              5              5   \n14                 107  ...                5              5              5   \n15                  92  ...                5              5              5   \n16                  27  ...                0              5              0   \n17                  59  ...                5              5              5   \n18                  37  ...                5              5              5   \n19                  50  ...                5              5              5   \n20                  54  ...                0              5              5   \n21                  60  ...                5              5              5   \n22                  35  ...                5              0              5   \n23                  69  ...                0             25             25   \n24                  24  ...                5              5              0   \n25                  42  ...                0              0             25   \n26                  41  ...                0              0             25   \n27                  61  ...                5              5              5   \n28                  86  ...                5             25              0   \n29                  19  ...                5              5              0   \n30                 114  ...                5             25              5   \n31                  59  ...                0              5              0   \n32                 114  ...                5             25             25   \n33                  63  ...                5              0              5   \n34                  44  ...                5             25              5   \n35                  86  ...                0              5              0   \n36                  39  ...                5             25             25   \n37                  52  ...                5              5              0   \n38                  41  ...                0             25              5   \n39                  35  ...                5              0             25   \n40                  95  ...                5              0              5   \n41                 100  ...                5              5              5   \n42                  55  ...                0              0              5   \n43                  80  ...                5             25              0   \n44                  20  ...                5              0              5   \n45                  45  ...                5              0             25   \n46                  32  ...                5             25              0   \n47                  89  ...                5              0              5   \n48                  62  ...                5              5             25   \n49                 106  ...                5             25              0   \n50                  45  ...                0              0              5   \n51                  62  ...                5              5              0   \n52                  58  ...                5             25              0   \n53                  24  ...                5             25              5   \n54                  39  ...                5              0             25   \n55                  84  ...                0              5              5   \n56                  47  ...                5              0              5   \n57                  24  ...                5             25             25   \n58                  73  ...                5              0              5   \n59                  68  ...                5             25             25   \n60                  79  ...                5              0              5   \n61                  69  ...                0              0              5   \n62                  52  ...                5             25              5   \n63                  98  ...                5              5              5   \n64                  73  ...                0              5              0   \n65                  32  ...                5             25              0   \n66                  37  ...                5              0              5   \n67                  45  ...                5             25              0   \n68                  40  ...                0              0              5   \n\n    red_3_endgame  red_cells_bottom_auto  red_cells_bottom_teleop  \\\n11              5                      0                        0   \n12             25                      0                        0   \n13              5                      0                        9   \n14              0                      0                        0   \n15              0                      0                        4   \n16              5                      0                        0   \n17              5                      0                        0   \n18              5                      0                        5   \n19              5                      0                        0   \n20              0                      1                        1   \n21              5                      0                        0   \n22             25                      0                        0   \n23              5                      0                        6   \n24             25                      0                        0   \n25             25                      3                       11   \n26             25                      0                        2   \n27              5                      0                        4   \n28             25                      3                       12   \n29             25                      0                        0   \n30              0                      2                        2   \n31              5                      0                        0   \n32              0                      0                        0   \n33              5                      0                        0   \n34              5                      0                        0   \n35              5                      2                        3   \n36              0                      0                        1   \n37              0                      0                        1   \n38              0                      0                        2   \n39             25                      0                        5   \n40             25                      0                        0   \n41              5                      0                        3   \n42              0                      0                        0   \n43              5                      0                        0   \n44              5                      0                        0   \n45              0                      0                        1   \n46              5                      3                       13   \n47              5                      0                        6   \n48             25                      0                        0   \n49             25                      2                       12   \n50              0                      0                        0   \n51             25                      3                        0   \n52              5                      0                        0   \n53              5                      0                        1   \n54              5                      0                        0   \n55              5                      0                        1   \n56             25                      2                       10   \n57             25                      0                        0   \n58             25                      3                        3   \n59              5                      0                        0   \n60             25                      0                        0   \n61              0                      0                        0   \n62              0                      0                        0   \n63              5                      0                        0   \n64              0                      3                        4   \n65             25                      0                       11   \n66              5                      0                        0   \n67              5                      0                        6   \n68              5                      0                        4   \n\n    red_cells_outer_auto  red_cells_outer_teleop  red_cells_inner_auto  \\\n11                     0                       0                     0   \n12                     3                       0                     0   \n13                     0                       0                     0   \n14                     0                       0                     0   \n15                     0                       1                     0   \n16                     2                       1                     1   \n17                     3                       3                     0   \n18                     1                       5                     2   \n19                     0                       6                     0   \n20                     0                       0                     0   \n21                     6                      13                     0   \n22                     0                       2                     3   \n23                     0                       3                     0   \n24                     1                       5                     0   \n25                     0                       0                     0   \n26                     6                       0                     0   \n27                     1                       0                     0   \n28                     0                       0                     0   \n29                     2                       6                     0   \n30                     0                       0                     0   \n31                     0                       0                     0   \n32                     3                       5                     0   \n33                     5                       4                     0   \n34                     3                       7                     0   \n35                     0                       1                     0   \n36                     1                       3                     0   \n37                     1                      13                     1   \n38                     1                       0                     0   \n39                     1                       0                     0   \n40                     1                      11                     2   \n41                     0                       6                     0   \n42                     0                       0                     0   \n43                     4                      10                     3   \n44                     0                       5                     0   \n45                     0                       6                     0   \n46                     0                       0                     0   \n47                     0                       0                     0   \n48                     3                       6                     0   \n49                     2                       9                     1   \n50                     0                       1                     0   \n51                     2                       3                     0   \n52                     1                       5                     0   \n53                     1                       6                     0   \n54                     1                       6                     2   \n55                     3                       2                     0   \n56                     3                       7                     0   \n57                     6                       7                     0   \n58                     0                       0                     0   \n59                     3                       9                     0   \n60                     0                       0                     0   \n61                     1                       5                     0   \n62                     4                      12                     2   \n63                     3                       4                     0   \n64                     2                       1                     0   \n65                     1                       8                     2   \n66                     0                       0                     0   \n67                     0                       0                     0   \n68                     0                       0                     0   \n\n    red_cells_inner_teleop  \n11                       0  \n12                       0  \n13                       0  \n14                       0  \n15                       0  \n16                       1  \n17                       0  \n18                       0  \n19                       0  \n20                       0  \n21                       0  \n22                       1  \n23                       2  \n24                       0  \n25                       0  \n26                       0  \n27                       0  \n28                       0  \n29                       0  \n30                       0  \n31                       0  \n32                       1  \n33                       0  \n34                       1  \n35                       0  \n36                       0  \n37                       1  \n38                       0  \n39                       0  \n40                       1  \n41                       1  \n42                       0  \n43                       1  \n44                       0  \n45                       2  \n46                       0  \n47                       0  \n48                       2  \n49                       0  \n50                       1  \n51                       1  \n52                       0  \n53                       0  \n54                       0  \n55                       0  \n56                       4  \n57                       1  \n58                       0  \n59                       2  \n60                       0  \n61                       0  \n62                       5  \n63                       0  \n64                       0  \n65                       0  \n66                       0  \n67                       0  \n68                       0  \n\n[58 rows x 47 columns]"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load in the data -- for this, we'll only be looking at data from one qualifier event to see if the method works\n",
    "with open('data/week1/2020ncwak.json', 'r') as f:\n",
    "    json_data = json.load(f)\n",
    "\n",
    "team_keys = json_data['teams']\n",
    "\n",
    "qualification_matches, _, _, _ = load_data_event('week1/2020ncwak')\n",
    "qualification_matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# common function for calculating a team's OPR given a point objective to get the OPR for and matches to look at\n",
    "# still, thanks to [this](https://blog.thebluealliance.com/2017/10/05/the-math-behind-opr-an-introduction/)\n",
    "def calculate_oprs(matches, team_keys, point_objective):\n",
    "    match_matrix = np.zeros((len(matches)*2, len(team_keys)))\n",
    "    score_matrix = np.zeros((len(matches)*2, 1))\n",
    "\n",
    "    i = 0\n",
    "    for _, match in matches.iterrows():\n",
    "        for red_team in match['red_keys']:\n",
    "            match_matrix[i*2][team_keys.index(red_team)] = 1\n",
    "        score_matrix[i*2][0] = match[f'red_{point_objective}']\n",
    "\n",
    "        for blue_team in match['blue_keys']:\n",
    "            match_matrix[i*2+1][team_keys.index(blue_team)] = 1\n",
    "        score_matrix[i*2+1][0] = match[f'blue_{point_objective}']\n",
    "\n",
    "        i += 1\n",
    "\n",
    "    l_matrix = match_matrix.T.dot(match_matrix)\n",
    "    r_matrix = match_matrix.T.dot(score_matrix)\n",
    "    opr_matrix = np.linalg.pinv(l_matrix).dot(r_matrix)\n",
    "\n",
    "    return opr_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to implement wgardner's method, we'll need to iterate through teams and compute OPR while taking out matches that they're in\n",
    "def wgardner_team(team, matches, team_keys, point_objective):\n",
    "    # first get matches that the team is in {O(n)}\n",
    "    matches_playing = []\n",
    "    for i, match in matches.iterrows():\n",
    "        if team in match['red_keys'] or team in match['blue_keys']:\n",
    "            matches_playing.append(i)\n",
    "\n",
    "    # iterate through the matches, calculate OPR w/out that match being considered\n",
    "    oprs = []\n",
    "    for match_removing in matches_playing:\n",
    "        oprs.append(calculate_oprs(matches.drop(match_removing), team_keys, point_objective)[team_keys.index(team)][0])\n",
    "\n",
    "    # take the mean and standard deviation\n",
    "    std = np.std(oprs)\n",
    "    mean = np.mean(oprs)\n",
    "\n",
    "    return mean, std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>team_key</th>\n      <th>opr_mean</th>\n      <th>opr_std</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>frc2059</td>\n      <td>17.896851</td>\n      <td>2.428961</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>frc2642</td>\n      <td>22.791394</td>\n      <td>1.808675</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>frc3229</td>\n      <td>14.869575</td>\n      <td>0.864849</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>frc3459</td>\n      <td>36.204948</td>\n      <td>2.416582</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>frc4291</td>\n      <td>21.131777</td>\n      <td>2.353282</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>frc435</td>\n      <td>28.395997</td>\n      <td>2.545728</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>frc4561</td>\n      <td>33.783723</td>\n      <td>2.672946</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>frc4816</td>\n      <td>15.177462</td>\n      <td>1.763741</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>frc4828</td>\n      <td>30.759297</td>\n      <td>2.681994</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>frc5160</td>\n      <td>41.944778</td>\n      <td>1.908392</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>frc5190</td>\n      <td>23.695213</td>\n      <td>1.657400</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>frc5511</td>\n      <td>42.288735</td>\n      <td>1.527604</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>frc5518</td>\n      <td>7.659281</td>\n      <td>2.078717</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>frc5607</td>\n      <td>5.567661</td>\n      <td>2.252394</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>frc5762</td>\n      <td>8.975077</td>\n      <td>1.406170</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>frc5919</td>\n      <td>13.802305</td>\n      <td>1.831114</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>frc6004</td>\n      <td>33.360781</td>\n      <td>2.022743</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>frc6240</td>\n      <td>4.994076</td>\n      <td>1.973270</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>frc6496</td>\n      <td>18.297739</td>\n      <td>1.534121</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>frc6500</td>\n      <td>16.535880</td>\n      <td>2.319479</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>frc6502</td>\n      <td>43.562286</td>\n      <td>2.911898</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>frc6565</td>\n      <td>6.999122</td>\n      <td>2.069461</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>frc6908</td>\n      <td>14.187784</td>\n      <td>2.288888</td>\n    </tr>\n    <tr>\n      <th>23</th>\n      <td>frc7265</td>\n      <td>-3.986231</td>\n      <td>1.992065</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>frc7463</td>\n      <td>2.026216</td>\n      <td>1.960852</td>\n    </tr>\n    <tr>\n      <th>25</th>\n      <td>frc7671</td>\n      <td>24.479618</td>\n      <td>2.534583</td>\n    </tr>\n    <tr>\n      <th>26</th>\n      <td>frc7763</td>\n      <td>11.973122</td>\n      <td>2.039864</td>\n    </tr>\n    <tr>\n      <th>27</th>\n      <td>frc7890</td>\n      <td>24.577645</td>\n      <td>2.747796</td>\n    </tr>\n    <tr>\n      <th>28</th>\n      <td>frc8090</td>\n      <td>14.055447</td>\n      <td>2.451501</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
      "text/plain": "   team_key   opr_mean   opr_std\n0   frc2059  17.896851  2.428961\n1   frc2642  22.791394  1.808675\n2   frc3229  14.869575  0.864849\n3   frc3459  36.204948  2.416582\n4   frc4291  21.131777  2.353282\n5    frc435  28.395997  2.545728\n6   frc4561  33.783723  2.672946\n7   frc4816  15.177462  1.763741\n8   frc4828  30.759297  2.681994\n9   frc5160  41.944778  1.908392\n10  frc5190  23.695213  1.657400\n11  frc5511  42.288735  1.527604\n12  frc5518   7.659281  2.078717\n13  frc5607   5.567661  2.252394\n14  frc5762   8.975077  1.406170\n15  frc5919  13.802305  1.831114\n16  frc6004  33.360781  2.022743\n17  frc6240   4.994076  1.973270\n18  frc6496  18.297739  1.534121\n19  frc6500  16.535880  2.319479\n20  frc6502  43.562286  2.911898\n21  frc6565   6.999122  2.069461\n22  frc6908  14.187784  2.288888\n23  frc7265  -3.986231  1.992065\n24  frc7463   2.026216  1.960852\n25  frc7671  24.479618  2.534583\n26  frc7763  11.973122  2.039864\n27  frc7890  24.577645  2.747796\n28  frc8090  14.055447  2.451501"
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get a dataframe with the new OPR calculations\n",
    "def get_opr_std_method(matches, team_keys, point_objective, opr_function):\n",
    "    team_opr_data = []\n",
    "    for team in team_keys:\n",
    "        mean, std = opr_function(team, qualification_matches, team_keys, point_objective)\n",
    "        team_opr_data.append([team, mean, std])\n",
    "    return pd.DataFrame(team_opr_data, columns=['team_key', 'opr_mean', 'opr_std'])\n",
    "\n",
    "opr_wgardner_data = get_opr_std_method(qualification_matches, team_keys, 'points_scored', wgardner_team)\n",
    "opr_wgardner_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results seem to be pretty similar between them and I'd say that they line up with what I'd expect the STD to look like.\n",
    "\n",
    "I'll now be looking at the more mathematical approach of finding the residuals for all the matches that a team plays in and using that to compute standard error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for the math approach, we iterate through the matches that a team plays with initial OPR predictions, find the residual, and then use that in our error stuff\n",
    "def residual_team(team, matches, team_keys, point_objective):\n",
    "    # get initial OPR results\n",
    "    opr_predictions_initial = calculate_oprs(qualification_matches, team_keys, point_objective)\n",
    "\n",
    "    # go through and find residuals on a per match basis\n",
    "    predictions = []\n",
    "    targets = []\n",
    "    for _, match in matches.iterrows():\n",
    "        alliance_keys = observed_score = None\n",
    "        if team in match['red_keys']:\n",
    "            alliance_keys = match['red_keys']\n",
    "            observed_score = match[f'red_{point_objective}']\n",
    "        if team in match['blue_keys']:\n",
    "            aliance_keys = match['blue_keys']\n",
    "            observed_score = match[f'blue_{point_objective}']\n",
    "        \n",
    "        if alliance_keys != None:\n",
    "            predictions.append(sum([\n",
    "                opr_predictions_initial[team_keys.index(key)][0] for key in alliance_keys\n",
    "            ]))\n",
    "            targets.append(observed_score)\n",
    "    predictions = np.array(predictions)\n",
    "    targets = np.array(targets)\n",
    "\n",
    "    # calculate std from residuals\n",
    "    std = np.sqrt(np.mean((predictions-targets)**2))\n",
    "\n",
    "    return opr_predictions_initial[team_keys.index(team)][0], std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>team_key</th>\n      <th>opr_mean</th>\n      <th>opr_std</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>frc2059</td>\n      <td>17.894015</td>\n      <td>29.535132</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>frc2642</td>\n      <td>22.768461</td>\n      <td>10.053786</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>frc3229</td>\n      <td>14.960390</td>\n      <td>9.729637</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>frc3459</td>\n      <td>36.080602</td>\n      <td>19.882051</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>frc4291</td>\n      <td>21.166143</td>\n      <td>23.294821</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>frc435</td>\n      <td>28.398827</td>\n      <td>30.158297</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>frc4561</td>\n      <td>33.786561</td>\n      <td>20.391414</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>frc4816</td>\n      <td>15.268651</td>\n      <td>18.272789</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>frc4828</td>\n      <td>30.687604</td>\n      <td>22.782791</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>frc5160</td>\n      <td>42.045429</td>\n      <td>10.967039</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>frc5190</td>\n      <td>23.667173</td>\n      <td>15.072472</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>frc5511</td>\n      <td>42.259525</td>\n      <td>10.633653</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>frc5518</td>\n      <td>7.859717</td>\n      <td>19.375560</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>frc5607</td>\n      <td>5.552262</td>\n      <td>10.267063</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>frc5762</td>\n      <td>8.966863</td>\n      <td>13.309038</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>frc5919</td>\n      <td>13.692614</td>\n      <td>20.506555</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>frc6004</td>\n      <td>33.268494</td>\n      <td>19.319780</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>frc6240</td>\n      <td>4.929856</td>\n      <td>20.578337</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>frc6496</td>\n      <td>18.316304</td>\n      <td>14.620219</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>frc6500</td>\n      <td>16.389886</td>\n      <td>19.505194</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>frc6502</td>\n      <td>43.605208</td>\n      <td>28.011243</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>frc6565</td>\n      <td>7.200063</td>\n      <td>14.927467</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>frc6908</td>\n      <td>14.211545</td>\n      <td>19.099127</td>\n    </tr>\n    <tr>\n      <th>23</th>\n      <td>frc7265</td>\n      <td>-4.033774</td>\n      <td>21.027692</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>frc7463</td>\n      <td>2.088444</td>\n      <td>16.618299</td>\n    </tr>\n    <tr>\n      <th>25</th>\n      <td>frc7671</td>\n      <td>24.379804</td>\n      <td>27.709048</td>\n    </tr>\n    <tr>\n      <th>26</th>\n      <td>frc7763</td>\n      <td>12.007671</td>\n      <td>17.164093</td>\n    </tr>\n    <tr>\n      <th>27</th>\n      <td>frc7890</td>\n      <td>24.589682</td>\n      <td>31.156774</td>\n    </tr>\n    <tr>\n      <th>28</th>\n      <td>frc8090</td>\n      <td>13.825314</td>\n      <td>22.631138</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
      "text/plain": "   team_key   opr_mean    opr_std\n0   frc2059  17.894015  29.535132\n1   frc2642  22.768461  10.053786\n2   frc3229  14.960390   9.729637\n3   frc3459  36.080602  19.882051\n4   frc4291  21.166143  23.294821\n5    frc435  28.398827  30.158297\n6   frc4561  33.786561  20.391414\n7   frc4816  15.268651  18.272789\n8   frc4828  30.687604  22.782791\n9   frc5160  42.045429  10.967039\n10  frc5190  23.667173  15.072472\n11  frc5511  42.259525  10.633653\n12  frc5518   7.859717  19.375560\n13  frc5607   5.552262  10.267063\n14  frc5762   8.966863  13.309038\n15  frc5919  13.692614  20.506555\n16  frc6004  33.268494  19.319780\n17  frc6240   4.929856  20.578337\n18  frc6496  18.316304  14.620219\n19  frc6500  16.389886  19.505194\n20  frc6502  43.605208  28.011243\n21  frc6565   7.200063  14.927467\n22  frc6908  14.211545  19.099127\n23  frc7265  -4.033774  21.027692\n24  frc7463   2.088444  16.618299\n25  frc7671  24.379804  27.709048\n26  frc7763  12.007671  17.164093\n27  frc7890  24.589682  31.156774\n28  frc8090  13.825314  22.631138"
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opr_residual_data = get_opr_std_method(qualification_matches, team_keys, 'points_scored', residual_team)\n",
    "opr_residual_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this approach it looks like we get a bit (lit. a lot) more error than before, which could also make sense given that OPR is potentially bogus as a scoring metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the two metrics that I've implemented, I think that they both serve their own purposes. wgardner's method is more suited to looking at the variance of the OPR metric itself, while the residual method is more suited for looking at a team's performance while taking OPR into consideration. Between the two, I think that we'll likely get better results when using the residual method, since our models are mainly built off of the prediction of the metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}