reproduce_results_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import argparse\n",
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "from ppnp.pytorch import PPNP\n",
    "from ppnp.pytorch.training import train_model\n",
    "from ppnp.pytorch.earlystopping import stopping_args\n",
    "from ppnp.pytorch.propagation import PPRExact, PPRPowerIteration\n",
    "from ppnp.data.io import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.basicConfig(\n",
    "        format='%(asctime)s: %(message)s',\n",
    "        datefmt='%Y-%m-%d %H:%M:%S',\n",
    "        level=logging.INFO + 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch variant\n",
    "This is not the official code that should be used to reproduce the results from the paper (see `reproduce_results.ipynb` for this), but an adaptation of that code to PyTorch for better accessibility. This notebook reproduces the accuracy of the TensorFlow implementation, but has a longer computation time and varies in some details due to the change to PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Undirected, unweighted and connected SparseGraph with 15962 edges (no self-loops). Data: adj_matrix (2810x2810), attr_matrix (2810x2879), labels (2810), node_names (2810), attr_names (2879), class_names (7)>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graph_name = 'cora_ml'\n",
    "graph = load_dataset(graph_name)\n",
    "graph.standardize(select_lcc=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up data splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all we need to decide whether to use the test or validation set. Be mindful that we can only look at the test set exactly _once_ and then can't change any hyperparameters oder model details, no matter what. Everything else would cause overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the seeds for the dataset splits used in the paper for test/validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_seeds = [\n",
    "        2144199730,  794209841, 2985733717, 2282690970, 1901557222,\n",
    "        2009332812, 2266730407,  635625077, 3538425002,  960893189,\n",
    "        497096336, 3940842554, 3594628340,  948012117, 3305901371,\n",
    "        3644534211, 2297033685, 4092258879, 2590091101, 1694925034]\n",
    "val_seeds = [\n",
    "        2413340114, 3258769933, 1789234713, 2222151463, 2813247115,\n",
    "        1920426428, 4272044734, 2092442742, 841404887, 2188879532,\n",
    "        646784207, 1633698412, 2256863076,  374355442,  289680769,\n",
    "        4281139389, 4263036964,  900418539,  119332950, 1628837138]\n",
    "\n",
    "if test:\n",
    "    seeds = test_seeds\n",
    "else:\n",
    "    seeds = val_seeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can choose the remaining settings for the training/early stopping/validation(test) split. These are the ones chosen in the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "if graph_name == 'microsoft_academic':\n",
    "    nknown = 5000\n",
    "else:\n",
    "    nknown = 1500\n",
    "    \n",
    "idx_split_args = {'ntrain_per_class': 20, 'nstopping': 500, 'nknown': nknown}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up propagation\n",
    "\n",
    "Next we need to set up the proper pmropagation scheme. In the paper we've introduced the exact PPR propagation used in PPNP and the PPR power iteration propagation used in APPNP.\n",
    "\n",
    "We use the hyperparameters from the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "if graph_name == 'microsoft_academic':\n",
    "    alpha = 0.2\n",
    "else:\n",
    "    alpha = 0.1\n",
    "\n",
    "prop_ppnp = PPRExact(graph.adj_matrix, alpha=alpha)\n",
    "prop_appnp = PPRPowerIteration(graph.adj_matrix, alpha=alpha, niter=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Choose model hyperparameters\n",
    "\n",
    "Now we choose the hyperparameters. These are the ones used in the paper for all datasets.\n",
    "\n",
    "Note that we choose the propagation for APPNP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_args = {\n",
    "    'hiddenunits': [64],\n",
    "    'drop_prob': 0.5,\n",
    "    'propagation': prop_appnp}\n",
    "\n",
    "reg_lambda = 5e-3\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train model\n",
    "\n",
    "First we set the remaining settings for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "niter_per_seed = 5\n",
    "save_result = False\n",
    "print_interval = 100\n",
    "device = 'cuda'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use 20 different seeds for splitting and 5 iterations (different random initializations) per split, so we train 100 times altogether. This will take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-12-04 12:35:06: Iteration 1 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:35:06: PyTorch seed: 3735636244\n",
      "2019-12-04 12:37:01: Last epoch: 1641, best epoch: 1273 (112.727 sec)\n",
      "2019-12-04 12:37:01: Test accuracy: 83.4%\n",
      "2019-12-04 12:37:01: Iteration 2 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:37:01: PyTorch seed: 301655472\n",
      "2019-12-04 12:38:26: Last epoch: 1226, best epoch: 1126 (84.561 sec)\n",
      "2019-12-04 12:38:26: Test accuracy: 85.6%\n",
      "2019-12-04 12:38:26: Iteration 3 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:38:26: PyTorch seed: 2898121914\n",
      "2019-12-04 12:40:33: Last epoch: 1854, best epoch: 1680 (127.663 sec)\n",
      "2019-12-04 12:40:33: Test accuracy: 84.2%\n",
      "2019-12-04 12:40:34: Iteration 4 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:40:34: PyTorch seed: 1529719124\n",
      "2019-12-04 12:43:01: Last epoch: 2153, best epoch: 2053 (147.002 sec)\n",
      "2019-12-04 12:43:01: Test accuracy: 84.5%\n",
      "2019-12-04 12:43:01: Iteration 5 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:43:01: PyTorch seed: 3611559506\n",
      "2019-12-04 12:44:38: Last epoch: 1417, best epoch: 1317 (97.350 sec)\n",
      "2019-12-04 12:44:38: Test accuracy: 84.5%\n",
      "2019-12-04 12:44:38: Iteration 6 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:44:38: PyTorch seed: 4129579036\n",
      "2019-12-04 12:46:32: Last epoch: 1640, best epoch: 486 (113.348 sec)\n",
      "2019-12-04 12:46:32: Test accuracy: 83.4%\n",
      "2019-12-04 12:46:32: Iteration 7 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:46:32: PyTorch seed: 1621262813\n",
      "2019-12-04 12:48:21: Last epoch: 1591, best epoch: 543 (109.375 sec)\n",
      "2019-12-04 12:48:21: Test accuracy: 84.4%\n",
      "2019-12-04 12:48:21: Iteration 8 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:48:21: PyTorch seed: 2770793973\n",
      "2019-12-04 12:50:22: Last epoch: 1751, best epoch: 691 (120.446 sec)\n",
      "2019-12-04 12:50:22: Test accuracy: 82.6%\n",
      "2019-12-04 12:50:22: Iteration 9 of 100\n",
      "                     ------------------\n",
      "2019-12-04 12:50:22: PyTorch seed: 793788464\n",
      "2019-12-04 12:52:21: Last epoch: 1722, best epoch: 459 (118.687 sec)\n",
      "2019-12-04 12:52:21: Test accuracy: 83.8%\n",
      "2019-12-04 12:52:21: Iteration 10 of 100\n",
      "                     -------------------\n",
      "2019-12-04 12:52:21: PyTorch seed: 1832553900\n",
      "2019-12-04 12:54:02: Last epoch: 1468, best epoch: 1279 (100.710 sec)\n",
      "2019-12-04 12:54:02: Test accuracy: 84.9%\n",
      "2019-12-04 12:54:02: Iteration 11 of 100\n",
      "                     -------------------\n",
      "2019-12-04 12:54:02: PyTorch seed: 1667731880\n",
      "2019-12-04 12:56:15: Last epoch: 1929, best epoch: 1369 (132.785 sec)\n",
      "2019-12-04 12:56:15: Test accuracy: 85.8%\n",
      "2019-12-04 12:56:15: Iteration 12 of 100\n",
      "                     -------------------\n",
      "2019-12-04 12:56:15: PyTorch seed: 3759012365\n",
      "2019-12-04 12:58:20: Last epoch: 1824, best epoch: 1698 (125.336 sec)\n",
      "2019-12-04 12:58:20: Test accuracy: 86.3%\n",
      "2019-12-04 12:58:20: Iteration 13 of 100\n",
      "                     -------------------\n",
      "2019-12-04 12:58:20: PyTorch seed: 2819547057\n",
      "2019-12-04 13:01:19: Last epoch: 2609, best epoch: 2344 (178.874 sec)\n",
      "2019-12-04 13:01:19: Test accuracy: 86.3%\n",
      "2019-12-04 13:01:19: Iteration 14 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:01:19: PyTorch seed: 880641236\n",
      "2019-12-04 13:02:46: Last epoch: 1248, best epoch: 1075 (86.677 sec)\n",
      "2019-12-04 13:02:46: Test accuracy: 87.2%\n",
      "2019-12-04 13:02:46: Iteration 15 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:02:46: PyTorch seed: 1560109512\n",
      "2019-12-04 13:04:56: Last epoch: 1870, best epoch: 1656 (129.475 sec)\n",
      "2019-12-04 13:04:56: Test accuracy: 85.9%\n",
      "2019-12-04 13:04:56: Iteration 16 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:04:56: PyTorch seed: 2749644593\n",
      "2019-12-04 13:07:13: Last epoch: 2009, best epoch: 1908 (137.694 sec)\n",
      "2019-12-04 13:07:13: Test accuracy: 85.6%\n",
      "2019-12-04 13:07:14: Iteration 17 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:07:14: PyTorch seed: 3732160565\n",
      "2019-12-04 13:09:47: Last epoch: 2228, best epoch: 2128 (153.324 sec)\n",
      "2019-12-04 13:09:47: Test accuracy: 86.3%\n",
      "2019-12-04 13:09:47: Iteration 18 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:09:47: PyTorch seed: 2382395551\n",
      "2019-12-04 13:12:08: Last epoch: 2046, best epoch: 1135 (141.231 sec)\n",
      "2019-12-04 13:12:08: Test accuracy: 86.2%\n",
      "2019-12-04 13:12:08: Iteration 19 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:12:08: PyTorch seed: 2067348931\n",
      "2019-12-04 13:14:49: Last epoch: 2341, best epoch: 1382 (160.841 sec)\n",
      "2019-12-04 13:14:49: Test accuracy: 85.0%\n",
      "2019-12-04 13:14:49: Iteration 20 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:14:50: PyTorch seed: 1282230086\n",
      "2019-12-04 13:16:46: Last epoch: 1696, best epoch: 1531 (116.610 sec)\n",
      "2019-12-04 13:16:46: Test accuracy: 85.9%\n",
      "2019-12-04 13:16:46: Iteration 21 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:16:46: PyTorch seed: 469384778\n",
      "2019-12-04 13:18:59: Last epoch: 1915, best epoch: 1815 (132.686 sec)\n",
      "2019-12-04 13:18:59: Test accuracy: 86.0%\n",
      "2019-12-04 13:18:59: Iteration 22 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:18:59: PyTorch seed: 1998347281\n",
      "2019-12-04 13:22:12: Last epoch: 2798, best epoch: 2698 (192.562 sec)\n",
      "2019-12-04 13:22:12: Test accuracy: 86.3%\n",
      "2019-12-04 13:22:12: Iteration 23 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:22:12: PyTorch seed: 6505637\n",
      "2019-12-04 13:24:19: Last epoch: 1841, best epoch: 1741 (127.414 sec)\n",
      "2019-12-04 13:24:19: Test accuracy: 86.3%\n",
      "2019-12-04 13:24:19: Iteration 24 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:24:19: PyTorch seed: 1322593177\n",
      "2019-12-04 13:27:04: Last epoch: 2393, best epoch: 1928 (164.497 sec)\n",
      "2019-12-04 13:27:04: Test accuracy: 86.0%\n",
      "2019-12-04 13:27:04: Iteration 25 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:27:04: PyTorch seed: 1481164471\n",
      "2019-12-04 13:29:26: Last epoch: 2094, best epoch: 1501 (142.070 sec)\n",
      "2019-12-04 13:29:26: Test accuracy: 86.6%\n",
      "2019-12-04 13:29:26: Iteration 26 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:29:26: PyTorch seed: 2315609452\n",
      "2019-12-04 13:31:01: Last epoch: 1421, best epoch: 1321 (95.074 sec)\n",
      "2019-12-04 13:31:01: Test accuracy: 86.7%\n",
      "2019-12-04 13:31:01: Iteration 27 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:31:02: PyTorch seed: 134370138\n",
      "2019-12-04 13:33:36: Last epoch: 2236, best epoch: 1414 (154.081 sec)\n",
      "2019-12-04 13:33:36: Test accuracy: 86.1%\n",
      "2019-12-04 13:33:36: Iteration 28 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:33:36: PyTorch seed: 763854306\n",
      "2019-12-04 13:35:47: Last epoch: 1956, best epoch: 1855 (130.867 sec)\n",
      "2019-12-04 13:35:47: Test accuracy: 86.3%\n",
      "2019-12-04 13:35:47: Iteration 29 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:35:47: PyTorch seed: 1236976485\n",
      "2019-12-04 13:37:57: Last epoch: 1966, best epoch: 1866 (130.466 sec)\n",
      "2019-12-04 13:37:57: Test accuracy: 86.6%\n",
      "2019-12-04 13:37:57: Iteration 30 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:37:57: PyTorch seed: 3088245823\n",
      "2019-12-04 13:39:47: Last epoch: 1598, best epoch: 1103 (110.042 sec)\n",
      "2019-12-04 13:39:48: Test accuracy: 86.4%\n",
      "2019-12-04 13:39:48: Iteration 31 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:39:48: PyTorch seed: 2415526065\n",
      "2019-12-04 13:41:22: Last epoch: 1354, best epoch: 781 (94.161 sec)\n",
      "2019-12-04 13:41:22: Test accuracy: 83.0%\n",
      "2019-12-04 13:41:22: Iteration 32 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:41:22: PyTorch seed: 3025391654\n",
      "2019-12-04 13:43:00: Last epoch: 1420, best epoch: 431 (98.275 sec)\n",
      "2019-12-04 13:43:00: Test accuracy: 82.8%\n",
      "2019-12-04 13:43:00: Iteration 33 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:43:00: PyTorch seed: 2557734369\n",
      "2019-12-04 13:45:11: Last epoch: 1895, best epoch: 819 (130.202 sec)\n",
      "2019-12-04 13:45:11: Test accuracy: 83.6%\n",
      "2019-12-04 13:45:11: Iteration 34 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:45:11: PyTorch seed: 3192628045\n",
      "2019-12-04 13:47:35: Last epoch: 2104, best epoch: 1912 (144.018 sec)\n",
      "2019-12-04 13:47:35: Test accuracy: 83.1%\n",
      "2019-12-04 13:47:35: Iteration 35 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:47:35: PyTorch seed: 4016062332\n",
      "2019-12-04 13:49:10: Last epoch: 1386, best epoch: 1214 (95.527 sec)\n",
      "2019-12-04 13:49:11: Test accuracy: 83.7%\n",
      "2019-12-04 13:49:11: Iteration 36 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:49:11: PyTorch seed: 3805129701\n",
      "2019-12-04 13:51:02: Last epoch: 1604, best epoch: 1071 (110.960 sec)\n",
      "2019-12-04 13:51:02: Test accuracy: 85.3%\n",
      "2019-12-04 13:51:02: Iteration 37 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:51:02: PyTorch seed: 2077709091\n",
      "2019-12-04 13:53:13: Last epoch: 1944, best epoch: 1744 (131.659 sec)\n",
      "2019-12-04 13:53:14: Test accuracy: 85.2%\n",
      "2019-12-04 13:53:14: Iteration 38 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:53:14: PyTorch seed: 1920483596\n",
      "2019-12-04 13:54:57: Last epoch: 1491, best epoch: 1290 (103.465 sec)\n",
      "2019-12-04 13:54:57: Test accuracy: 85.1%\n",
      "2019-12-04 13:54:57: Iteration 39 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:54:57: PyTorch seed: 3690189808\n",
      "2019-12-04 13:56:40: Last epoch: 1484, best epoch: 1384 (102.904 sec)\n",
      "2019-12-04 13:56:40: Test accuracy: 84.7%\n",
      "2019-12-04 13:56:40: Iteration 40 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:56:40: PyTorch seed: 3128409866\n",
      "2019-12-04 13:58:40: Last epoch: 1744, best epoch: 1644 (119.849 sec)\n",
      "2019-12-04 13:58:40: Test accuracy: 84.1%\n",
      "2019-12-04 13:58:40: Iteration 41 of 100\n",
      "                     -------------------\n",
      "2019-12-04 13:58:40: PyTorch seed: 121192343\n",
      "2019-12-04 14:00:15: Last epoch: 1378, best epoch: 830 (94.705 sec)\n",
      "2019-12-04 14:00:15: Test accuracy: 84.4%\n",
      "2019-12-04 14:00:15: Iteration 42 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:00:15: PyTorch seed: 2567981569\n",
      "2019-12-04 14:01:59: Last epoch: 1501, best epoch: 1350 (103.583 sec)\n",
      "2019-12-04 14:01:59: Test accuracy: 85.4%\n",
      "2019-12-04 14:01:59: Iteration 43 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:01:59: PyTorch seed: 771847210\n",
      "2019-12-04 14:04:48: Last epoch: 2453, best epoch: 1141 (169.193 sec)\n",
      "2019-12-04 14:04:48: Test accuracy: 85.0%\n",
      "2019-12-04 14:04:48: Iteration 44 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:04:48: PyTorch seed: 914029014\n",
      "2019-12-04 14:06:59: Last epoch: 1901, best epoch: 1186 (131.001 sec)\n",
      "2019-12-04 14:06:59: Test accuracy: 84.9%\n",
      "2019-12-04 14:06:59: Iteration 45 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:06:59: PyTorch seed: 43768811\n",
      "2019-12-04 14:08:39: Last epoch: 1439, best epoch: 719 (99.129 sec)\n",
      "2019-12-04 14:08:39: Test accuracy: 85.2%\n",
      "2019-12-04 14:08:39: Iteration 46 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:08:39: PyTorch seed: 1997993430\n",
      "2019-12-04 14:10:29: Last epoch: 1584, best epoch: 1341 (109.806 sec)\n",
      "2019-12-04 14:10:29: Test accuracy: 83.3%\n",
      "2019-12-04 14:10:29: Iteration 47 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:10:29: PyTorch seed: 247169375\n",
      "2019-12-04 14:11:55: Last epoch: 1246, best epoch: 1027 (86.026 sec)\n",
      "2019-12-04 14:11:55: Test accuracy: 84.2%\n",
      "2019-12-04 14:11:55: Iteration 48 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:11:55: PyTorch seed: 377535166\n",
      "2019-12-04 14:13:47: Last epoch: 1627, best epoch: 1303 (111.783 sec)\n",
      "2019-12-04 14:13:47: Test accuracy: 84.2%\n",
      "2019-12-04 14:13:47: Iteration 49 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:13:47: PyTorch seed: 3410062402\n",
      "2019-12-04 14:16:04: Last epoch: 1991, best epoch: 1173 (136.617 sec)\n",
      "2019-12-04 14:16:04: Test accuracy: 83.4%\n",
      "2019-12-04 14:16:04: Iteration 50 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:16:04: PyTorch seed: 2394233274\n",
      "2019-12-04 14:17:43: Last epoch: 1503, best epoch: 1313 (99.076 sec)\n",
      "2019-12-04 14:17:43: Test accuracy: 84.4%\n",
      "2019-12-04 14:17:43: Iteration 51 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:17:43: PyTorch seed: 2642551977\n",
      "2019-12-04 14:19:27: Last epoch: 1509, best epoch: 1140 (104.406 sec)\n",
      "2019-12-04 14:19:27: Test accuracy: 84.2%\n",
      "2019-12-04 14:19:27: Iteration 52 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:19:27: PyTorch seed: 519427632\n",
      "2019-12-04 14:21:10: Last epoch: 1478, best epoch: 1343 (102.058 sec)\n",
      "2019-12-04 14:21:10: Test accuracy: 84.3%\n",
      "2019-12-04 14:21:10: Iteration 53 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:21:10: PyTorch seed: 2884376025\n",
      "2019-12-04 14:22:35: Last epoch: 1238, best epoch: 1138 (85.267 sec)\n",
      "2019-12-04 14:22:35: Test accuracy: 83.5%\n",
      "2019-12-04 14:22:35: Iteration 54 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:22:35: PyTorch seed: 3341859265\n",
      "2019-12-04 14:24:25: Last epoch: 1589, best epoch: 1472 (109.486 sec)\n",
      "2019-12-04 14:24:25: Test accuracy: 84.6%\n",
      "2019-12-04 14:24:25: Iteration 55 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:24:25: PyTorch seed: 1238206969\n",
      "2019-12-04 14:26:22: Last epoch: 1700, best epoch: 1192 (116.898 sec)\n",
      "2019-12-04 14:26:22: Test accuracy: 84.1%\n",
      "2019-12-04 14:26:22: Iteration 56 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:26:22: PyTorch seed: 1687791607\n",
      "2019-12-04 14:28:15: Last epoch: 1637, best epoch: 1305 (113.389 sec)\n",
      "2019-12-04 14:28:15: Test accuracy: 85.0%\n",
      "2019-12-04 14:28:15: Iteration 57 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:28:15: PyTorch seed: 319400591\n",
      "2019-12-04 14:30:30: Last epoch: 1958, best epoch: 1688 (134.276 sec)\n",
      "2019-12-04 14:30:30: Test accuracy: 84.7%\n",
      "2019-12-04 14:30:30: Iteration 58 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:30:30: PyTorch seed: 1326104933\n",
      "2019-12-04 14:31:42: Last epoch: 1048, best epoch: 486 (71.748 sec)\n",
      "2019-12-04 14:31:42: Test accuracy: 83.6%\n",
      "2019-12-04 14:31:42: Iteration 59 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:31:42: PyTorch seed: 3004365329\n",
      "2019-12-04 14:33:25: Last epoch: 1503, best epoch: 709 (103.013 sec)\n",
      "2019-12-04 14:33:25: Test accuracy: 85.3%\n",
      "2019-12-04 14:33:25: Iteration 60 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:33:25: PyTorch seed: 3559006409\n",
      "2019-12-04 14:35:35: Last epoch: 1907, best epoch: 1807 (130.259 sec)\n",
      "2019-12-04 14:35:35: Test accuracy: 84.1%\n",
      "2019-12-04 14:35:35: Iteration 61 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:35:35: PyTorch seed: 2285855002\n",
      "2019-12-04 14:37:31: Last epoch: 1693, best epoch: 1286 (115.765 sec)\n",
      "2019-12-04 14:37:31: Test accuracy: 84.4%\n",
      "2019-12-04 14:37:31: Iteration 62 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:37:31: PyTorch seed: 578082656\n",
      "2019-12-04 14:40:00: Last epoch: 2174, best epoch: 2023 (148.663 sec)\n",
      "2019-12-04 14:40:00: Test accuracy: 85.4%\n",
      "2019-12-04 14:40:00: Iteration 63 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:40:00: PyTorch seed: 57905702\n",
      "2019-12-04 14:42:21: Last epoch: 2063, best epoch: 1639 (141.149 sec)\n",
      "2019-12-04 14:42:21: Test accuracy: 85.3%\n",
      "2019-12-04 14:42:21: Iteration 64 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:42:21: PyTorch seed: 1505067973\n",
      "2019-12-04 14:44:16: Last epoch: 1685, best epoch: 1577 (114.222 sec)\n",
      "2019-12-04 14:44:16: Test accuracy: 85.5%\n",
      "2019-12-04 14:44:16: Iteration 65 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:44:16: PyTorch seed: 3523521412\n",
      "2019-12-04 14:45:49: Last epoch: 1405, best epoch: 1305 (93.210 sec)\n",
      "2019-12-04 14:45:49: Test accuracy: 84.7%\n",
      "2019-12-04 14:45:49: Iteration 66 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:45:49: PyTorch seed: 2528033453\n",
      "2019-12-04 14:48:24: Last epoch: 2287, best epoch: 2109 (154.557 sec)\n",
      "2019-12-04 14:48:24: Test accuracy: 86.9%\n",
      "2019-12-04 14:48:24: Iteration 67 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:48:24: PyTorch seed: 1559748535\n",
      "2019-12-04 14:50:25: Last epoch: 1768, best epoch: 1424 (120.801 sec)\n",
      "2019-12-04 14:50:25: Test accuracy: 86.9%\n",
      "2019-12-04 14:50:25: Iteration 68 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:50:25: PyTorch seed: 1412587798\n",
      "2019-12-04 14:52:07: Last epoch: 1516, best epoch: 1416 (101.620 sec)\n",
      "2019-12-04 14:52:07: Test accuracy: 88.4%\n",
      "2019-12-04 14:52:07: Iteration 69 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:52:07: PyTorch seed: 2839055120\n",
      "2019-12-04 14:54:54: Last epoch: 2485, best epoch: 2092 (167.668 sec)\n",
      "2019-12-04 14:54:54: Test accuracy: 87.2%\n",
      "2019-12-04 14:54:55: Iteration 70 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:54:55: PyTorch seed: 3710964622\n",
      "2019-12-04 14:57:53: Last epoch: 2657, best epoch: 2163 (178.291 sec)\n",
      "2019-12-04 14:57:53: Test accuracy: 87.9%\n",
      "2019-12-04 14:57:53: Iteration 71 of 100\n",
      "                     -------------------\n",
      "2019-12-04 14:57:53: PyTorch seed: 3990089082\n",
      "2019-12-04 15:00:10: Last epoch: 2033, best epoch: 1804 (136.497 sec)\n",
      "2019-12-04 15:00:10: Test accuracy: 87.0%\n",
      "2019-12-04 15:00:10: Iteration 72 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:00:10: PyTorch seed: 919565932\n",
      "2019-12-04 15:02:14: Last epoch: 1852, best epoch: 992 (124.620 sec)\n",
      "2019-12-04 15:02:14: Test accuracy: 86.8%\n",
      "2019-12-04 15:02:14: Iteration 73 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:02:14: PyTorch seed: 69724526\n",
      "2019-12-04 15:04:32: Last epoch: 2022, best epoch: 1108 (137.662 sec)\n",
      "2019-12-04 15:04:32: Test accuracy: 86.6%\n",
      "2019-12-04 15:04:32: Iteration 74 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:04:32: PyTorch seed: 1034149573\n",
      "2019-12-04 15:06:07: Last epoch: 1399, best epoch: 1299 (94.787 sec)\n",
      "2019-12-04 15:06:07: Test accuracy: 86.7%\n",
      "2019-12-04 15:06:07: Iteration 75 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:06:07: PyTorch seed: 388547972\n",
      "2019-12-04 15:08:08: Last epoch: 1821, best epoch: 1495 (120.803 sec)\n",
      "2019-12-04 15:08:08: Test accuracy: 86.9%\n",
      "2019-12-04 15:08:08: Iteration 76 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:08:08: PyTorch seed: 1367645262\n",
      "2019-12-04 15:09:38: Last epoch: 1326, best epoch: 1121 (89.510 sec)\n",
      "2019-12-04 15:09:38: Test accuracy: 84.8%\n",
      "2019-12-04 15:09:38: Iteration 77 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:09:38: PyTorch seed: 821176455\n",
      "2019-12-04 15:11:44: Last epoch: 1859, best epoch: 936 (126.533 sec)\n",
      "2019-12-04 15:11:45: Test accuracy: 85.3%\n",
      "2019-12-04 15:11:45: Iteration 78 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:11:45: PyTorch seed: 2445386884\n",
      "2019-12-04 15:13:36: Last epoch: 1615, best epoch: 1515 (111.391 sec)\n",
      "2019-12-04 15:13:36: Test accuracy: 84.7%\n",
      "2019-12-04 15:13:36: Iteration 79 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:13:36: PyTorch seed: 1041334648\n",
      "2019-12-04 15:15:26: Last epoch: 1587, best epoch: 1054 (109.655 sec)\n",
      "2019-12-04 15:15:26: Test accuracy: 85.0%\n",
      "2019-12-04 15:15:26: Iteration 80 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:15:26: PyTorch seed: 2845574661\n",
      "2019-12-04 15:17:33: Last epoch: 1832, best epoch: 1194 (126.665 sec)\n",
      "2019-12-04 15:17:33: Test accuracy: 85.1%\n",
      "2019-12-04 15:17:33: Iteration 81 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:17:33: PyTorch seed: 4200644897\n",
      "2019-12-04 15:19:32: Last epoch: 1724, best epoch: 1624 (119.162 sec)\n",
      "2019-12-04 15:19:32: Test accuracy: 84.5%\n",
      "2019-12-04 15:19:32: Iteration 82 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:19:32: PyTorch seed: 2905881480\n",
      "2019-12-04 15:21:44: Last epoch: 1907, best epoch: 1486 (132.009 sec)\n",
      "2019-12-04 15:21:44: Test accuracy: 84.4%\n",
      "2019-12-04 15:21:44: Iteration 83 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:21:44: PyTorch seed: 2968666547\n",
      "2019-12-04 15:23:34: Last epoch: 1586, best epoch: 1485 (109.847 sec)\n",
      "2019-12-04 15:23:34: Test accuracy: 84.4%\n",
      "2019-12-04 15:23:34: Iteration 84 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:23:34: PyTorch seed: 3239142148\n",
      "2019-12-04 15:24:48: Last epoch: 1069, best epoch: 969 (73.951 sec)\n",
      "2019-12-04 15:24:48: Test accuracy: 83.9%\n",
      "2019-12-04 15:24:48: Iteration 85 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:24:48: PyTorch seed: 586676192\n",
      "2019-12-04 15:26:43: Last epoch: 1655, best epoch: 1546 (114.341 sec)\n",
      "2019-12-04 15:26:43: Test accuracy: 84.1%\n",
      "2019-12-04 15:26:43: Iteration 86 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:26:43: PyTorch seed: 2152154544\n",
      "2019-12-04 15:28:20: Last epoch: 1406, best epoch: 1306 (97.311 sec)\n",
      "2019-12-04 15:28:20: Test accuracy: 84.0%\n",
      "2019-12-04 15:28:20: Iteration 87 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:28:20: PyTorch seed: 1038951480\n",
      "2019-12-04 15:30:09: Last epoch: 1569, best epoch: 1017 (108.336 sec)\n",
      "2019-12-04 15:30:09: Test accuracy: 82.0%\n",
      "2019-12-04 15:30:09: Iteration 88 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:30:09: PyTorch seed: 1165058153\n",
      "2019-12-04 15:31:51: Last epoch: 1469, best epoch: 1143 (101.586 sec)\n",
      "2019-12-04 15:31:51: Test accuracy: 83.2%\n",
      "2019-12-04 15:31:51: Iteration 89 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:31:51: PyTorch seed: 3274574999\n",
      "2019-12-04 15:33:33: Last epoch: 1481, best epoch: 1137 (101.821 sec)\n",
      "2019-12-04 15:33:33: Test accuracy: 82.2%\n",
      "2019-12-04 15:33:33: Iteration 90 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:33:33: PyTorch seed: 3761825190\n",
      "2019-12-04 15:35:16: Last epoch: 1487, best epoch: 1356 (102.932 sec)\n",
      "2019-12-04 15:35:16: Test accuracy: 82.5%\n",
      "2019-12-04 15:35:16: Iteration 91 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:35:16: PyTorch seed: 2649625274\n",
      "2019-12-04 15:36:56: Last epoch: 1457, best epoch: 1298 (100.460 sec)\n",
      "2019-12-04 15:36:56: Test accuracy: 86.5%\n",
      "2019-12-04 15:36:56: Iteration 92 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:36:56: PyTorch seed: 1950551320\n",
      "2019-12-04 15:38:49: Last epoch: 1634, best epoch: 745 (112.649 sec)\n",
      "2019-12-04 15:38:49: Test accuracy: 86.6%\n",
      "2019-12-04 15:38:49: Iteration 93 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:38:49: PyTorch seed: 856395625\n",
      "2019-12-04 15:41:15: Last epoch: 2109, best epoch: 1886 (146.015 sec)\n",
      "2019-12-04 15:41:15: Test accuracy: 86.5%\n",
      "2019-12-04 15:41:15: Iteration 94 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:41:15: PyTorch seed: 4028067342\n",
      "2019-12-04 15:43:09: Last epoch: 1633, best epoch: 1179 (113.466 sec)\n",
      "2019-12-04 15:43:09: Test accuracy: 86.0%\n",
      "2019-12-04 15:43:09: Iteration 95 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:43:09: PyTorch seed: 2685703419\n",
      "2019-12-04 15:45:08: Last epoch: 1722, best epoch: 1556 (119.130 sec)\n",
      "2019-12-04 15:45:08: Test accuracy: 87.0%\n",
      "2019-12-04 15:45:08: Iteration 96 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:45:08: PyTorch seed: 2917757789\n",
      "2019-12-04 15:47:18: Last epoch: 1870, best epoch: 1696 (129.509 sec)\n",
      "2019-12-04 15:47:18: Test accuracy: 85.3%\n",
      "2019-12-04 15:47:18: Iteration 97 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:47:18: PyTorch seed: 323093313\n",
      "2019-12-04 15:49:23: Last epoch: 1814, best epoch: 990 (124.675 sec)\n",
      "2019-12-04 15:49:23: Test accuracy: 85.3%\n",
      "2019-12-04 15:49:23: Iteration 98 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:49:23: PyTorch seed: 3769111585\n",
      "2019-12-04 15:51:12: Last epoch: 1575, best epoch: 1475 (108.614 sec)\n",
      "2019-12-04 15:51:12: Test accuracy: 84.8%\n",
      "2019-12-04 15:51:12: Iteration 99 of 100\n",
      "                     -------------------\n",
      "2019-12-04 15:51:12: PyTorch seed: 4264326989\n",
      "2019-12-04 15:53:38: Last epoch: 2171, best epoch: 2071 (146.324 sec)\n",
      "2019-12-04 15:53:38: Test accuracy: 84.7%\n",
      "2019-12-04 15:53:38: Iteration 100 of 100\n",
      "                     --------------------\n",
      "2019-12-04 15:53:38: PyTorch seed: 2259132929\n",
      "2019-12-04 15:55:01: Last epoch: 1213, best epoch: 979 (83.004 sec)\n",
      "2019-12-04 15:55:01: Test accuracy: 84.3%\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "niter_tot = niter_per_seed * len(seeds)\n",
    "i_tot = 0\n",
    "for seed in seeds:\n",
    "    idx_split_args['seed'] = seed\n",
    "    for _ in range(niter_per_seed):\n",
    "        i_tot += 1\n",
    "        logging_string = f\"Iteration {i_tot} of {niter_tot}\"\n",
    "        logging.log(22,\n",
    "                logging_string + \"\\n                     \"\n",
    "                + '-' * len(logging_string))\n",
    "        _, result = train_model(\n",
    "            graph_name, PPNP, graph, model_args, learning_rate, reg_lambda,\n",
    "            idx_split_args, stopping_args, test, device, None, print_interval)\n",
    "        results.append({})\n",
    "        results[-1]['stopping_accuracy'] = result['early_stopping']['accuracy']\n",
    "        results[-1]['valtest_accuracy'] = result['valtest']['accuracy']\n",
    "        results[-1]['runtime'] = result['runtime']\n",
    "        results[-1]['runtime_perepoch'] = result['runtime_perepoch']\n",
    "        results[-1]['split_seed'] = seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "To evaluate the data we use Pandas and Seaborn (for bootstrapping)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stopping_accuracy</th>\n",
       "      <th>valtest_accuracy</th>\n",
       "      <th>runtime</th>\n",
       "      <th>runtime_perepoch</th>\n",
       "      <th>split_seed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.832</td>\n",
       "      <td>0.833588</td>\n",
       "      <td>112.726716</td>\n",
       "      <td>0.068652</td>\n",
       "      <td>2144199730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.838</td>\n",
       "      <td>0.855725</td>\n",
       "      <td>84.561042</td>\n",
       "      <td>0.068917</td>\n",
       "      <td>2144199730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.830</td>\n",
       "      <td>0.841985</td>\n",
       "      <td>127.662649</td>\n",
       "      <td>0.068821</td>\n",
       "      <td>2144199730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.838</td>\n",
       "      <td>0.845038</td>\n",
       "      <td>147.002493</td>\n",
       "      <td>0.068246</td>\n",
       "      <td>2144199730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.836</td>\n",
       "      <td>0.845038</td>\n",
       "      <td>97.349578</td>\n",
       "      <td>0.068653</td>\n",
       "      <td>2144199730</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   stopping_accuracy  valtest_accuracy     runtime  runtime_perepoch  \\\n",
       "0              0.832          0.833588  112.726716          0.068652   \n",
       "1              0.838          0.855725   84.561042          0.068917   \n",
       "2              0.830          0.841985  127.662649          0.068821   \n",
       "3              0.838          0.845038  147.002493          0.068246   \n",
       "4              0.836          0.845038   97.349578          0.068653   \n",
       "\n",
       "   split_seed  \n",
       "0  2144199730  \n",
       "1  2144199730  \n",
       "2  2144199730  \n",
       "3  2144199730  \n",
       "4  2144199730  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_df = pd.DataFrame(results)\n",
    "result_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard deviation doesn't really say much about the uncertainty of our results and the standard error of the mean (SEM) assumes a normal distribution. So the best way to get a valid estimate for our results' uncertainty is via bootstrapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_uncertainty(values: np.ndarray, n_boot: int = 1000, ci: int = 95) -> dict:\n",
    "    stats = {}\n",
    "    stats['mean'] = values.mean()\n",
    "    boots_series = sns.algorithms.bootstrap(values, func=np.mean, n_boot=n_boot)\n",
    "    stats['CI'] = sns.utils.ci(boots_series, ci)\n",
    "    stats['uncertainty'] = np.max(np.abs(stats['CI'] - stats['mean']))\n",
    "    return stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopping_acc = calc_uncertainty(result_df['stopping_accuracy'])\n",
    "valtest_acc = calc_uncertainty(result_df['valtest_accuracy'])\n",
    "runtime = calc_uncertainty(result_df['runtime'])\n",
    "runtime_perepoch = calc_uncertainty(result_df['runtime_perepoch'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "APPNP\n",
      "Early stopping: Accuracy: 84.03 ± 0.40%\n",
      "Test: Accuracy: 85.07 ± 0.26%\n",
      "Runtime: 119.764 ± 4.501 sec, per epoch: 68.53 ± 0.14ms\n"
     ]
    }
   ],
   "source": [
    "print(\"APPNP\\n\"\n",
    "      \"Early stopping: Accuracy: {:.2f} ± {:.2f}%\\n\"\n",
    "      \"{}: Accuracy: {:.2f} ± {:.2f}%\\n\"\n",
    "      \"Runtime: {:.3f} ± {:.3f} sec, per epoch: {:.2f} ± {:.2f}ms\"\n",
    "      .format(\n",
    "          stopping_acc['mean'] * 100,\n",
    "          stopping_acc['uncertainty'] * 100,\n",
    "          'Test' if test else 'Validation',\n",
    "          valtest_acc['mean'] * 100,\n",
    "          valtest_acc['uncertainty'] * 100,\n",
    "          runtime['mean'],\n",
    "          runtime['uncertainty'],\n",
    "          runtime_perepoch['mean'] * 1e3,\n",
    "          runtime_perepoch['uncertainty'] * 1e3,\n",
    "      ))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch1.0]",
   "language": "python",
   "name": "conda-env-pytorch1.0-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}