cascor_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cascade-Correlation,  a Forgotten Learning Architecture\n",
    "\n",
    "#### How the first 'deep learning' approach outperformed classical neural networks in 1990.\n",
    "\n",
    "In 1990, a dynamic neural network architecture by Scott E. Fahlman and Christian Lebiere called \"Cascade-Correlation\" [1] outperformed static neural networks in training speed, accuracy, and size. In a [recent lecture](https://www.youtube.com/watch?v=k2mPEUZH978), Fahlman actually called it the first approach to use something like 'deep' learning.\n",
    "With this algorithm, they successfully competed on the 'two spirals' categorization task and artificial grammar learning problems.\n",
    "\n",
    "And yet their work was largely forgotten or treated as a side note of history.\n",
    "Even though the idea itself is quite close in spirit to what we, by now, refer to as the ['boosting'](https://en.wikipedia.org/wiki/Boosting_(machine_learning)) family of machine learning algorithms - techniques that perform [better than deep learning](https://www.youtube.com/watch?v=9GCEVv94udY) on certain sets of problems. These - just like Cascade-Correlation - are based on the idea of iteratively improving a classifier by stacking specialized learners, trained one at a time.\n",
    "\n",
    "More than enough reason to understand and implement Fahlman's approach - and put it to the test on various domains and challenges! Let's investigate if, where, and how it can still compete with modern machine learning tools.\n",
    "\n",
    "---\n",
    "\n",
    "_This article is part one of a series with a focus on (Recurrent) Cascade-Correlation._\n",
    "\n",
    "_In this first part, we'll look in detail on how a simple, forward-only Cascade-Correlation (or CasCor for short) network can be implemented using Python and PyTorch. We'll also see some results of applying it to a simplistic problem domain._\n",
    "\n",
    "_In the next parts, we'll investigate how to automate and optimize the algorithm using batching. We'll also implement and train a recurrent version of it and benchmark CasCor against existing solutions._\n",
    "\n",
    "_To follow along with this article, you should be familiar with how neural networks can be trained using back-propagation of the loss gradient (as of 2020, a widely used approach). That is, you should understand how the gradient is usually calculated and applied to the parameters of a network to try to iteratively achieve convergence of the loss to a global minimum._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Cascade Correlation\n",
    "\n",
    "Without further ado, let's implement a simplistic version of CasCor! Since we are working with Python, we need to import some basics and PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.random import Generator, PCG64\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "\n",
    "rnd = Generator(PCG64(12345))\n",
    "# To be used like: numpy.random, e.g. rnd.random(2, 3) produces a random (2,3) array\n",
    "# We'll use this to generate initial weights for our neurons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "The CasCor algorithm starts off with a one-layer network (only output neurons). It then follows these steps:\n",
    "\n",
    "1. Train (only!) the output neurons until reaching a plateau\n",
    "2. If the residual error is good enough, stop;\n",
    "3. Otherwise, freeze all weights of the existing network\n",
    "4. Then train a new hidden neuron, by optimizing the correlation between its output and the last measured residual error (see below)\n",
    "5. Go back to step 1 - rinse & repeat\n",
    "\n",
    "Step 4. is the most important and most complicated part of CasCor, illustrated also in the figure below. \n",
    "Until the training has finished, the hidden neuron is disconnected from the network.\n",
    "We statically feed all the original inputs and the values of earlier layers as a weighted sum to its activation function.\n",
    "Then the training algorithm optimizes the neuron's weights to achieve the best possible correlation of its output value with the sample set residual error, measured in an earlier iteration.\n",
    "\n",
    "To train for optimum correlation, we will need to use the correlation measure instead of some standard loss function, but more on that later.\n",
    "\n",
    "After the training finished, we can add the hidden neuron to the network. The output neurons will now receive its value as an additional input, and we need to train their new weights, so we jump back to step 1.\n",
    "\n",
    "Fahlman's paper illustrates the process of adding a hidden neuron, like this:\n",
    "\n",
    "![CasCor network with new hidden node](img/cascor_layer_add.png)\n",
    "\n",
    "#### When adding a new hidden neuron, only the new weights (red squares) are iterated. Here, the arrows are weighted sums fed into the circle depicting the activation function. (Graphic translated and adapted from [Wikipedia](https://de.wikipedia.org/wiki/Datei:Cascar.png))\n",
    "\n",
    "In this article, we focus on the two major parts of CasCor: Steps 1. and 4., i.e. training the outputs and training a hidden neuron. The rest we'll simply do manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Set\n",
    "\n",
    "For simplicity, we use a 2D categorization task for testing (It's easier to debug). Our network will thus have 2 input dimensions (the coordinates of the grid) and 1 output dimension (a value between 0 and 1).\n",
    "\n",
    "We'll train on the following data sets (without test sets for now), where 0 values are black and 1 values are white:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAALXklEQVR4nO3df6jdd33H8efL2K4yhbaYhdDW2WlBimgGd6Uy/+gyOrL+0wpSLEwyKNTBCgoy7PzHOiYoqN0/Q+gwa/5w1lKdLeJ+hC7g/Cc21ljTRm3sWkxIk0ottn/Ykfa9P8434xpvktPz6557388HHM73+/l+zz3vL7mve77ne04+71QVkja/N6x3AZIWw7BLTRh2qQnDLjVh2KUmDLvUxFRhT7IryU+SHE1y16yKkjR7mfRz9iRbgJ8CNwLHgEeB26rqyfM8pjyVkObnNaCqsta2N07xc68DjlbV0wBJ7gduBs4Z9jcAl0zxhJLO79fn2TbNC+0VwM9XrR8bxiQtoWle2ceS5A7gDoA1zy0kLcQ0YT8OXLVq/cph7DdU1b3AvQBbEr+IL62TaU7jHwWuSXJ1kouBDwEPz6YsSbM28St7VZ1OcifwH8AWYE9VPTGzyiTN1MQfvU1iS1JejZfm59fAq+f46M2PvaUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71MRUU0kneQZ4CXgVOF1VK7MoStLszWLe+D+pql/M4OdImiNP46Umpg17Af+Z5PtD5xdJS2ra0/j3V9XxJL8H7Evy46r6zuodbP8kLYeZzRuf5G7g5ar6/Ln2cd54ab7mMm98kt9N8pYzy8CfAYcn/XmS5mua0/htwL8mOfNz/qWq/n0mVUmaOds/SZuI7Z8kGXapC8MuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVxwbAn2ZPkVJLDq8YuT7IvyVPD/WXzLVPStMZ5Zb8P2HXW2F3AI1V1DfDIsC5piV0w7EOHlxfOGr4Z2Dss7wVumW1ZkmZt0nnjt1XViWH5OUZzyK/J9k/Scpj6Al2NJp4/5+TzVXVvVa1U1Yphl9bPpGE/mWQ7wHB/anYlSZqHScP+MLB7WN4NPDSbciTNywXbPyX5KnAD8FbgJPAp4JvAA8DbgGeBW6vq7It4v8X2T9J8na/9k73epE3EXm+SDLvUhWGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71MSk7Z/uTnI8yaHhdtN8y5Q0rUnbPwHcU1U7htu3Z1uWpFmbtP2TpA1mmvfsdyZ5fDjNt4urtOQmDfuXgHcAO4ATwBfOtWOSO5IcTHJwcZNWSzrbWPPGJ3k78K2qevfr2XY2542X5mvm88af6fM2+ABw+Fz7SloOF2zZvLr9U5JjjNo/3ZBkB6Purc8AH5lfiZJmwfZP0iZi+ydJhl3qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdamKc9k9XJdmf5MkkTyT56DB+eZJ9SZ4a7p07XlpiF5yDbphJdntVPZbkLcD3gVuAvwReqKrPJrkLuKyqPnG+n7WyslIHDx6cSeGSftvKygoHDx6cbA66qjpRVY8Nyy8BR4ArgJuBvcNuexn9AZC0pF7Xe/ahIcQfAgeAbVV1Ytj0HLBttqVJmqWxw57kzcDXgY9V1a9Wb6vRe4E13w+sbv/0/PPPT1WspMmNFfYkFzEK+leq6hvD8MkznWGG+1NrPbaq7q2qlapa2bp16yxqljSBca7GB/gycKSqvrhq08PA7mF5N/DQ7MuTNCsXbP8E/DHwYeBHSQ4NY58EPgs8kOR24Fng1rlUKGkmLhj2qvousOalfOBPZ1uOpHnxG3RSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWpimvZPdyc5nuTQcLtp/uVKmtQ4E06eBj6+uv1Tkn3Dtnuq6vPzK0/SrIwz4eQJ4MSw/FKSM+2fJG0g07R/ArgzyeNJ9tjFVVpu07R/+hLwDmAHo1f+L5zjcbZ/kpbAxO2fqupkVb1aVa8B/wRct9Zjbf8kLYeJ2z+d6fM2+ABwePblSZqVado/3ZZkB6Purc8AH5lDfZJmZJr2T9+efTmS5sVv0ElNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qYlxJpy8JMn3kvxwaP/06WH86iQHkhxN8rUkF8+/XEmTGueV/RVgZ1W9l9Ec8buSXA98jlH7p3cCvwRun1uVkqZ2wbDXyMvD6kXDrYCdwIPD+F7glnkUKGk2xm0SsWWYRvoUsA/4GfBiVZ0edjmG/d+kpTZW2IfOLzuAKxl1fnnXuE9g+ydpObyuq/FV9SKwH3gfcGmSM/POXwkcP8djbP8kLYFxrsZvTXLpsPwm4EbgCKPQf3DYbTfw0JxqlDQD47R/2g7sTbKF0R+HB6rqW0meBO5P8vfADxj1g5O0pMZp//Q4o57sZ48/zTk6t0paPn6DTmrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTUzT/um+JP+T5NBw2zH3aiVNbJwJJ8+0f3o5yUXAd5P827Dtb6rqwfM8VtKSGGfCyQLWav8kaQOZqP1TVR0YNn0myeNJ7knyO/MqUtL0Jmr/lOTdwN8yagP1R8DlwCfWeqztn6TlMGn7p11VdWLo8PoK8M+cYw552z9Jy2HS9k8/TrJ9GAujds2H51empGlN0/7pv5JsBQIcAv5qfmVKmtY07Z92zqUiSXPhN+ikJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSExl1d1rQkyXPA88Oq28FfrGwJ18cj2vj2UzH9vtVtWaDhoWG/TeeODlYVSvr8uRz5HFtPJv52FbzNF5qwrBLTaxn2O9dx+eeJ49r49nMx/b/1u09u6TF8jReamLhYU+yK8lPkhxNctein3+WkuxJcirJ4VVjlyfZl+Sp4f6y9axxEkmuSrI/yZNJnkjy0WF8Qx9bkkuSfC/JD4fj+vQwfnWSA8Pv5NeSXLzetc7DQsM+dIL9R+DPgWuB25Jcu8gaZuw+YNdZY3cBj1TVNcAjw/pGcxr4eFVdC1wP/PXw77TRj+0VYGdVvRfYAexKcj3wOeCeqnon8Evg9vUrcX4W/cp+HXC0qp6uqv8F7gduXnANM1NV3wFeOGv4ZmDvsLyXUe/6DaWqTlTVY8PyS8AR4Ao2+LHVyMvD6kXDrYCdwIPD+IY7rnEtOuxXAD9ftX5sGNtMtlXViWH5OWDbehYzrSRvZ9Sy+wCb4NiSbElyCDgF7AN+BrxYVaeHXTbj7yTgBbq5qtFHHRv2444kbwa+Dnysqn61ettGPbaqerWqdgBXMjrTfNf6VrQ4iw77ceCqVetXDmObyckk2wGG+1PrXM9EklzEKOhfqapvDMOb4tgAqupFYD/wPuDSJG8cNm3G30lg8WF/FLhmuPp5MfAh4OEF1zBvDwO7h+XdwEPrWMtEkgT4MnCkqr64atOGPrYkW5NcOiy/CbiR0fWI/cAHh9023HGNa+FfqklyE/APwBZgT1V9ZqEFzFCSrwI3MPpfUyeBTwHfBB4A3sbof/jdWlVnX8RbakneD/w38CPgtWH4k4zet2/YY0vyHkYX4LYweqF7oKr+LskfMLpYfDnwA+AvquqV9at0PvwGndSEF+ikJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjXxfxkLTCs/mJgnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "target1 = np.concatenate((np.zeros((20,40)), np.ones((20,40))), axis=0)\n",
    "                         \n",
    "plt.imshow(target1, cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAALaklEQVR4nO3df+hd9X3H8eerqc6yFqw0C6Juda2syJgRM7Gsf7h0jsx/tFBKhY0MBDuY0EIZzfpP27GChbXun1GwNDN/dLVi2ynD/QhO6ArDmtrURm2ndSlNiInSSvUfR/S9P+5J+Tbka673nnu/95v38wGXe87nnPu970O+r5xzzz3f805VIenc96aNLkDSchh2qQnDLjVh2KUmDLvUhGGXmpgr7El2JflRkmeS7BmrKEnjy6zfsyfZAvwPcANwBHgUuKWqnnyd15SHEpvL1ddcs9El6A04fPgwL7zwQs607M1z/NxrgWeq6lmAJPcANwHrhv1NwAVzvKGW78CBAxtdgt6AHTt2rLtsnh3tJcBP18wfGcYkraB59uxTSXIbcBvAGY8tJC3FPGE/Cly2Zv7SYexXVNVdwF0AWxIvxJc2yDyH8Y8CVyS5PMn5wIeBB8YpS9LYZt6zV9XJJLcD/w5sAfZW1ROjVSZpVHN9Zq+qB4EHR6pF0gL5tbfUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5qY67ZUSQ4DLwGvAierav071EvaUGPcN/4Pq+qFEX6OpAXyMF5qYt6wF/AfSb47dH6RtKLmPYx/X1UdTfIbwP4kP6yqb61dwfZP0mqYa89eVUeH5xPAN5l0dj19nbuqakdV7TDs0saZOexJfj3J205NA38MHBqrMEnjmucwfhvwzSSnfs4/VdW/jVKVpNHN0+vtWeCqEWuRtEB+9SY1YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9TEWcOeZG+SE0kOrRm7KMn+JE8Pz29fbJmS5jXNnv1uYNdpY3uAh6rqCuChYV7SCjtr2IcOLz87bfgmYN8wvQ+4edyyJI1t1ltJb6uqY8P0c0zuIX9Gtn+SVsPcJ+iqqpg0eFxvue2fpBUwa9iPJ7kYYHg+MV5JkhZh1rA/AOwepncD949TjqRFmeart68C/w38TpIjSW4F7gBuSPI08EfDvKQVdtYTdFV1yzqL3j9yLZIWyCvopCYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUxKztnz6d5GiSg8PjxsWWKWles7Z/ArizqrYPjwfHLUvS2GZt/yRpk5nnM/vtSR4fDvPt4iqtuFnD/kXgXcB24Bjw+fVWTHJbkgNJDqzbI0rSws0U9qo6XlWvVtVrwJeAa19nXXu9SStgprCf6vM2+ABwaL11Ja2Gs3aEGdo/XQ+8I8kR4FPA9Um2M+neehj4yOJKlDSGWds/fXkBtUhaIK+gk5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE9O0f7osycNJnkzyRJKPDuMXJdmf5Onh2XvHSyvsrPegA04CH6+qx5K8Dfhukv3AnwMPVdUdSfYAe4BPvN4Puvqaazhw4MC8NUuawTTtn45V1WPD9EvAU8AlwE3AvmG1fcDNC6pR0gje0Gf2JO8ErgYeAbZV1bFh0XPAtnFLkzSmqcOe5K3A14GPVdUv1i6rqmJyD/kzve6X7Z+ef/75uYqVNLupwp7kPCZB/0pVfWMYPn6qM8zwfOJMr13b/mnr1q1j1CxpBtOcjQ+TphBPVdUX1ix6ANg9TO8G7h+/PEljmeZs/B8Afwb8IMnBYeyTwB3AvUluBX4CfGghFUoaxTTtn74NrNeA9f3jliNpUbyCTmrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTczT/unTSY4mOTg8blx8uZJmNU/7J4A7q+rvFleepLFMc8PJY8CxYfqlJKfaP0naROZp/wRwe5LHk+y1i6u02uZp//RF4F3AdiZ7/s+v8zrbP0krYOb2T1V1vKperarXgC8B157ptbZ/klbDzO2fTvV5G3wAODR+eZLGMk/7p1uSbGfSvfUw8JEF1CdpJPO0f3pw/HIkLYpX0ElNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qYlpbjh5QZLvJPn+0P7pM8P45UkeSfJMkq8lOX/x5Uqa1TR79leAnVV1FZN7xO9Kch3wOSbtn94N/By4dWFVSprbWcNeEy8Ps+cNjwJ2AvcN4/uAmxdRoKRxTNskYstwG+kTwH7gx8CLVXVyWOUI9n+TVtpUYR86v2wHLmXS+eU9076B7Z+k1fCGzsZX1YvAw8B7gQuTnLrv/KXA0XVeY/snaQVMczZ+a5ILh+m3ADcATzEJ/QeH1XYD9y+oRkkjmKb908XAviRbmPzncG9V/UuSJ4F7kvwt8D0m/eAkrahp2j89zqQn++njz7JO51ZJq8cr6KQmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71MQ87Z/uTvK/SQ4Oj+0Lr1bSzKa54eSp9k8vJzkP+HaSfx2W/VVV3fc6r5W0Iqa54WQBZ2r/JGkTman9U1U9Miz6bJLHk9yZ5NcWVaSk+c3U/inJ7wJ/zaQN1O8DFwGfONNrbf8krYZZ2z/tqqpjQ4fXV4B/ZJ17yNv+SVoNs7Z/+mGSi4exMGnXfGhxZUqa1zztn/4zyVYgwEHgLxZXpqR5zdP+aedCKpK0EF5BJzVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5rIpLvTkt4seR74yTD7DuCFpb358rhdm8+5tG2/VVVnbNCw1LD/yhsnB6pqx4a8+QK5XZvPubxta3kYLzVh2KUmNjLsd23gey+S27X5nMvb9ksb9pld0nJ5GC81sfSwJ9mV5EdJnkmyZ9nvP6Yke5OcSHJozdhFSfYneXp4fvtG1jiLJJcleTjJk0meSPLRYXxTb1uSC5J8J8n3h+36zDB+eZJHht/JryU5f6NrXYSlhn3oBPsPwJ8AVwK3JLlymTWM7G5g12lje4CHquoK4KFhfrM5CXy8qq4ErgP+cvh32uzb9gqws6quArYDu5JcB3wOuLOq3g38HLh140pcnGXv2a8FnqmqZ6vq/4B7gJuWXMNoqupbwM9OG74J2DdM72PSu35TqapjVfXYMP0S8BRwCZt822ri5WH2vOFRwE7gvmF8023XtJYd9kuAn66ZPzKMnUu2VdWxYfo5YNtGFjOvJO9k0rL7Ec6BbUuyJclB4ASwH/gx8GJVnRxWORd/JwFP0C1UTb7q2LRfdyR5K/B14GNV9Yu1yzbrtlXVq1W1HbiUyZHmeza2ouVZdtiPApetmb90GDuXHE9yMcDwfGKD65lJkvOYBP0rVfWNYfic2DaAqnoReBh4L3BhkjcPi87F30lg+WF/FLhiOPt5PvBh4IEl17BoDwC7h+ndwP0bWMtMkgT4MvBUVX1hzaJNvW1Jtia5cJh+C3ADk/MRDwMfHFbbdNs1raVfVJPkRuDvgS3A3qr67FILGFGSrwLXM/mrqePAp4B/Bu4FfpPJX/h9qKpOP4m30pK8D/gv4AfAa8PwJ5l8bt+025bk95icgNvCZEd3b1X9TZLfZnKy+CLge8CfVtUrG1fpYngFndSEJ+ikJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjXx/0JQOOIGCcDiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "target2 = np.concatenate(\n",
    "    (np.concatenate((np.zeros((20,20)), np.ones((20,20))), axis=1),\n",
    "     np.ones((20,40))), axis=0)\n",
    "                         \n",
    "plt.imshow(target2, cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To actually feed the input and output into our training function, we also need to convert these to a 'long' form and add a static bias value of 1.\n",
    "\n",
    "Plus, testing showed that CasCor and quickprop perform better when the inputs are normalized, so let's do that as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_trainset(target):\n",
    "    idxs = np.asarray(list(np.ndindex(target.shape)))\n",
    "    # Normalize inputs\n",
    "    idxs = idxs / np.linalg.norm(idxs, axis=0)\n",
    "    # Add bias vector:\n",
    "    x = np.ones((idxs.shape[0], idxs.shape[1]+1))\n",
    "    x[:,:-1] = idxs\n",
    "\n",
    "    y = target.reshape((-1, 1))\n",
    "    \n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quickprop\n",
    "\n",
    "To train the units in the CasCor network, they use a technique that was also invented by Fahlman in 1988 [2] called quickprop.\n",
    "\n",
    "Quickprop is an alternative to back-propagation that uses a variation of [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method). For more info on this aside from the original paper, [this useful blog post](https://www.bonaccorso.eu/2017/09/15/quickprop-an-almost-forgotten-neural-training-algorithm/) by Giuseppe Bonaccorso also describes it quite well.\n",
    "\n",
    "_Note that quickprop is not strictly necessary to implement CasCor. However, to stick close to the original paper and for maximized learning, we'll use it here as well. It is actually an interesting topic all on its own, and I encourage you to investigate it!_\n",
    "\n",
    "_If you couldn't care less about quickprop, skip ahead to the next section and treat any further mention of it simply as 'training neuron weights based on given input & expected output pairs'._\n",
    "\n",
    "Our implementation is based on the blog post - but since we don't want to focus on quickprop in this article, we'll just peek at some adjustments to their code instead of diving into the maths.\n",
    "\n",
    "**Flexibility.** The code from the post uses a fixed activation and loss function and statically implements their gradient. For CasCor, we need to be a bit more flexible (at least when it comes to the loss) so we pass these functions as parameters.\n",
    "\n",
    "**Automatic Gradient Computation.** Since the activation and loss function are now variable, we'll run into trouble when trying to build their gradient. But, using PyTorch, we can easily skip over that and let the `autograd` do the heavy lifting.\n",
    "\n",
    "**Convergence.** Giuseppe's code tests the change in weights per iteration to determine convergence. Some quick tests found that to be troublesome since it often seems to get stuck on saddle points and local minima. So instead of that, we'll use the residual error.\n",
    "\n",
    "Specifically, we'll calculate a running mean of the residual error, and check if the difference in mean per iteration is smaller than a given `tolerance`.\n",
    "\n",
    "Last but not least, if the error diverges or converges too slowly, quickprop simply gives up after a certain amount of iterations (it runs out of `patience`, see the function parameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Param shapes: x_: (n,i), y_: (n,o), weights: (i,o)\n",
    "#   Where n is the size of the whole sample set, i is the input count, o is the output count\n",
    "#   We expect x_ to already include the bias\n",
    "# Returns: trained weights, last prediction, last iteration, last loss\n",
    "# NB: Differentiation is done via torch\n",
    "def quickprop(x_, y_, weights,\n",
    "              activation=torch.nn.Sigmoid(),\n",
    "              loss=torch.nn.MSELoss(),\n",
    "              learning_rate=1e-4,\n",
    "              tolerance=1e-6,\n",
    "              patience=20000,\n",
    "              debug=False):\n",
    "    # Box params as torch datatypes\n",
    "    x = torch.Tensor(x_)\n",
    "    y = torch.Tensor(y_)\n",
    "    w = torch.Tensor(weights)\n",
    "\n",
    "    # Keep track of mean residual error values (used to test for convergence)\n",
    "    L_mean = 1\n",
    "    L_mean_prev = 1\n",
    "    L_mean_diff = 1\n",
    "    \n",
    "    # Keep track of loss and weight gradients\n",
    "    dL = torch.zeros(w.shape)\n",
    "    dL_prev = torch.ones(w.shape)\n",
    "    dw_prev = torch.ones(w.shape)\n",
    "\n",
    "    i = 0\n",
    "    predicted = []\n",
    "\n",
    "    # This algorithm expects the mean losses to converge or the patience to run out...\n",
    "    while L_mean_diff > tolerance and i < patience:\n",
    "        # Prep iteration\n",
    "        i += 1\n",
    "        dL_prev = dL.clone()\n",
    "        # NB: The following can probably done better with torch.no_grad(), but I couldn't make it work\n",
    "        w_var = torch.autograd.Variable(torch.Tensor(w), requires_grad=True)\n",
    "        \n",
    "        # Calc forward and loss\n",
    "        predicted = activation(torch.mm(x, w_var))\n",
    "        L = loss(predicted, y)\n",
    "        \n",
    "        # Keep track of losses and use as convergence criterion if mean doesn't change much     \n",
    "        L_mean = L_mean + (1/(i+1))*(L.detach().numpy() - L_mean)\n",
    "        L_mean_diff = np.abs(L_mean_prev - L_mean)\n",
    "        L_mean_prev = L_mean\n",
    "        \n",
    "        # Calc differential and do the weight update\n",
    "        L.backward()\n",
    "        \n",
    "        dL = w_var.grad.detach() # =: partial(L) / partial(W)\n",
    "        dw = dw_prev * dL / (dL_prev - dL + 1e-10) # Prevent div/0\n",
    "        \n",
    "        dw_prev = dw.clone()\n",
    "        \n",
    "        w += learning_rate * dw\n",
    "        \n",
    "        if debug and i % 100 == 99:\n",
    "            print(\"Residual           \", L.detach().numpy())\n",
    "            print(\"Residual mean      \", L_mean)\n",
    "            print(\"Residual mean diff \", L_mean_diff)\n",
    "        \n",
    "    return w.detach().numpy(), predicted.detach().numpy(), i, L.detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output Training\n",
    "\n",
    "With quickprop implemented, let's get to the fun part!\n",
    "\n",
    "CasCor starts with a one layer network, i.e. we will be using a single output neuron and connect that to our input (and bias).\n",
    "\n",
    "To start training this neuron, we take sets of input (`x`) and output (`y`) samples and create newly initialized (random) weights, all of which we feed into quickprop.\n",
    "\n",
    "_Note that this approach does not care whether the network is single-layer or deeper - since we are_ only _training the output weights, we could also have run the inputs through a number of hidden layers and use that as the `x` for the training._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The vector x is the values from the earlier hidden/input layers per each sample\n",
    "# Parameter shapes: x - (n,i), y - (n,o)\n",
    "#   Where n is the size of the whole sample set, i is the input count, o is the output count\n",
    "#   We expect x_ to already include the bias\n",
    "def train_outputs(x, y):\n",
    "    # Next we need to create a weight vector with the right shape\n",
    "    n, i = x.shape\n",
    "    n, o = y.shape\n",
    "    \n",
    "    weights = rnd.uniform(-0.01, 0.01, size=(i, o))\n",
    "    \n",
    "    # And run through the training\n",
    "    weights, predicted, i, loss = quickprop(x, y, weights)\n",
    "    \n",
    "    return weights, predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test this with the training sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAMBElEQVR4nO3dXahl9XnH8e8vo1OlCahkOoiaaBJpkNBMYSqG5sLaWqbeaCBILC1TEEyhQgKhjc1NTGkggSTmJgQssc5FGiPmRQnpy2CENDfGiZmYUZNqrKLDOGMwEr2IZfTpxV5TjqfnzOzZb2ef83w/sNlrr7X3Xs865/zO2vt/9vk/qSokbX1v2ugCJC2GYZeaMOxSE4ZdasKwS00YdqmJqcKeZE+Snyd5MsktsypK0uxl0r+zJ9kG/BdwNfAc8BBwQ1U9dpLHlC8lpPl5HaiqrLXtjCme93Lgyap6CiDJXcC1wLphfxNw1hQ7lHRyvznJtmlOtBcAz664/dywTtISmubMPpYkNwE3Aaz52kLSQkwT9sPARStuXzise4Oquh24HWBb4gfxpQ0yzcv4h4BLk1ySZDvwIeC+2ZQladYmPrNX1fEkNwP/DmwD7qiqR2dWmaSZmvhPb5PYlpSj8dL8/AZ4bZ0/vflnb6kJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNTHVVNJJngZeBl4DjlfV7lkUJWn2ZjFv/B9V1S9n8DyS5siX8VIT04a9gP9I8qOh84ukJTXty/j3V9XhJL8D7E/ys6r6/so72P5JWg4zmzc+ya3AK1X1ufXu47zx0nzNZd74JL+d5C0nloE/BQ5N+nyS5mual/E7gW8lOfE8/1JV/zaTqiTNnO2fpC3E9k+SDLvUhWGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5o4ZdiT3JHkWJJDK9adl2R/kieG63PnW6akaY1zZr8T2LNq3S3A/VV1KXD/cFvSEjtl2IcOLy+uWn0tsG9Y3gdcN9uyJM3apPPG76yqI8Py84zmkF+T7Z+k5TD1AF2NJp5fd/L5qrq9qnZX1W7DLm2cScN+NMn5AMP1sdmVJGkeJg37fcDeYXkvcO9sypE0L6ds/5Tka8CVwFuBo8AngW8DdwNvA54Brq+q1YN4/4/tn6T5Oln7J3u9SVuIvd4kGXapC8MuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qwrBLTRh2qYlJ2z/dmuRwkoPD5Zr5lilpWpO2fwK4rap2DZfvzrYsSbM2afsnSZvMNO/Zb07yyPAy3y6u0pKbNOxfBt4J7AKOAJ9f745JbkpyIMmBxU1aLWm1seaNT3Ix8J2qes/pbFvNeeOl+Zr5vPEn+rwNPgAcWu++kpbDKVs2r2z/lOQ5Ru2frkyyi1H31qeBD8+vREmzYPsnaQux/ZMkwy51YdilJk45QDdr2xa9Q0mAZ3apDcMuNWHYpSYMu9TEQgfotgNvX+QO1/D6Bu9/0V7b6AKWQKevwbMn2eaZXWrCsEtNGHapCcMuNWHYpSYWOhr/DuCuVetmMTp+OqOt047Mnk690+5rFqPIi6x32u/lsn4f51XD6Tx+3Hr/7iTbPLNLTRh2qQnDLjUxTvuni5I8kOSxJI8m+ciw/rwk+5M8MVw7d7y0xE45B90wk+z5VfVwkrcAPwKuA/4KeLGqPpPkFuDcqvr4yZ5r9+7tdeDAjlVrj09a+wyfY6MfvyQ11JjDQIseOZy2hs02mjdFvbv/BA4cnHAOuqo6UlUPD8svA48DFwDXAvuGu+1j9AtA0pI6rffsQ0OI3wceBHZW1ZFh0/PAztmWJmmWxg57kjcD3wA+WlW/XrmtRu8F1nw/sLL90wsvdPufM2l5jBX2JGcyCvpXq+qbw+qjJzrDDNfH1npsVd1eVburaveOHQ7+SxtlnI4wAb4CPF5VX1ix6T5gL/CZ4freU+/ud4FvTVLnEpnFYNySWnNYZw3r/tRs4a/N2Db4a3DGn6+/aYyH/yHwl8BPkxwc1n2CUcjvTnIj8Axw/XRVSpqnU4a9qn7A+r/z/3i25UiaF99ES00YdqkJwy41seD2T9uBixe7S6mVs9fd4pldasKwS00YdqkJwy41seABuix+l1Ir63/m2TO71IRhl5ow7FIThl1qwrBLTRh2qQnDLjVh2KUmDLvUxDTtn25NcjjJweFyzfzLlTSpcT67ehz42Mr2T0n2D9tuq6rPza88SbMyzoSTR4Ajw/LLSU60f5K0iUzT/gng5iSPJLnDLq7Scpum/dOXgXcCuxid+T+/zuNWtH96YfqKJU1k4vZPVXW0ql6rqteBfwIuX+uxb2z/tLpds6RFGWc0fs32Tyf6vA0+AByafXmSZmWa9k83JNnFqHvr08CH51CfpBmZpv3Td2dfjqR58RN0UhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41YdilJgy71IRhl5ow7FIThl1qYpwJJ89K8sMkPxnaP31qWH9JkgeTPJnk60m2z79cSZMa58z+KnBVVb2X0Rzxe5JcAXyWUfundwG/Am6cW5WSpnbKsNfIK8PNM4dLAVcB9wzr9wHXzaNASbMxbpOIbcM00seA/cAvgJeq6vhwl+ew/5u01MYK+9D5ZRdwIaPOL+8edwe2f5KWw2mNxlfVS8ADwPuAc5KcmHf+QuDwOo+x/ZO0BMYZjd+R5Jxh+WzgauBxRqH/4HC3vcC9c6pR0gyM0/7pfGBfkm2MfjncXVXfSfIYcFeSfwR+zKgfnKQlNU77p0cY9WRfvf4p1uncKmn5+Ak6qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41MU37pzuT/HeSg8Nl19yrlTSxcSacPNH+6ZUkZwI/SPKvw7a/rap7TvJYSUtinAknC1ir/ZOkTWSi9k9V9eCw6dNJHklyW5LfmleRkqY3UfunJO8B/p5RG6g/AM4DPr7WY23/JC2HSds/7amqI0OH11eBf2adOeRt/yQth0nbP/0syfnDujBq13xofmVKmtY07Z++l2QHEOAg8NfzK1PStKZp/3TVXCqSNBd+gk5qwrBLTRh2qQnDLjVh2KUmDLvUhGGXmjDsUhOGXWrCsEtNGHapCcMuNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqkJwy41kVF3pwXtLHkBeGa4+Vbglwvb+eJ4XJvPVjq2t1fVmg0aFhr2N+w4OVBVuzdk53PkcW0+W/nYVvJlvNSEYZea2Miw376B+54nj2vz2crH9n827D27pMXyZbzUxMLDnmRPkp8neTLJLYve/ywluSPJsSSHVqw7L8n+JE8M1+duZI2TSHJRkgeSPJbk0SQfGdZv6mNLclaSHyb5yXBcnxrWX5LkweFn8utJtm90rfOw0LAPnWC/BPwZcBlwQ5LLFlnDjN0J7Fm17hbg/qq6FLh/uL3ZHAc+VlWXAVcAfzN8nzb7sb0KXFVV7wV2AXuSXAF8Fritqt4F/Aq4ceNKnJ9Fn9kvB56sqqeq6n+Au4BrF1zDzFTV94EXV62+Ftg3LO9j1Lt+U6mqI1X18LD8MvA4cAGb/Nhq5JXh5pnDpYCrgHuG9ZvuuMa16LBfADy74vZzw7qtZGdVHRmWnwd2bmQx00pyMaOW3Q+yBY4tybYkB4FjwH7gF8BLVXV8uMtW/JkEHKCbqxr9qWPT/rkjyZuBbwAfrapfr9y2WY+tql6rql3AhYxeab57YytanEWH/TBw0YrbFw7rtpKjSc4HGK6PbXA9E0lyJqOgf7Wqvjms3hLHBlBVLwEPAO8DzklyxrBpK/5MAosP+0PApcPo53bgQ8B9C65h3u4D9g7Le4F7N7CWiSQJ8BXg8ar6wopNm/rYkuxIcs6wfDZwNaPxiAeADw5323THNa6Ff6gmyTXAF4FtwB1V9emFFjBDSb4GXMnov6aOAp8Evg3cDbyN0X/4XV9VqwfxllqS9wP/CfwUeH1Y/QlG79s37bEl+T1GA3DbGJ3o7q6qf0jyDkaDxecBPwb+oqpe3bhK58NP0ElNOEAnNWHYpSYMu9SEYZeaMOxSE4ZdasKwS00YdqmJ/wVvEHM9kgZChQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = create_trainset(target1)\n",
    "w, pred = train_outputs(x, y)\n",
    "\n",
    "plt.imshow(pred.reshape(target1.shape), cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good for the simple one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPy0lEQVR4nO3df4gc93nH8fdHV8lSYrsXYVUVlls5iUkwaaOCKhKaP1ylLqopyIFg4tKigMEJxJDQUKLmnySlAReSuH+0pDhEtQppbOMktSnuD6EK0kBR7DiKI1tJrLgKkZB1Fo6xDcVF8tM/ZqSe9mZO393ZnZ2Z7+cFx+1+9zs73zntc7Pz3KN9FBGY2fCtmfcCzKwdDnazTDjYzTLhYDfLhIPdLBMOdrNMNAp2Sbsl/VjSCUn7prUoM5s+Tfp3dkkLwE+AW4FTwBPAnRHx7CrbRN/fSmhG21eNjzO37udaNb5QM7dqfG3F2C/VbH9V4hjA+qr9V+0MYEPF2JvGmFu1s6oxgDXrxthZ1dFVbQ/j/SSr/iXS/oVPnjzFuXMvVb506vaWYidwIiKeB5D0ILAHqA32NdT/jOep7sVfZZxfVlXPW7evqpdC3Wu/avzNNXOrXqbX1MxdrBj71YqxzTXbb6sYe2vN3Jsrxq6te+LfqBh79xhzq3b2jprtN2wdY2dvrxi7vmZu1fMu1sz95Yqxq2vmXj6+Y8dtNfOavY2/Hvj5svunqD9SM5uzJmf2JJLuBu6G5m+BzWxyTYL9NHDDsvtby7HLRMT9wP0AC5IL8c3mpEmwPwHcJOlGiiD/EPBHU1nVDM3i+rxpwqtuvG5u1fV5XQqp6vp8sWZu6vX5tprtq67Pqy6XAa6tuoStut6G6kvmurmp1+cb6rIJVTurujaH6qvWqgOD6p961bU5VF+f112zj2bB6l+1Ewd7RJyXdA/wbxSv6/0R8cykz2dms9Xomj0iHgcen9JazGyG+v5nbzNL5GA3y4SD3SwTM/87+7y4Kq6wWDFWlXWH9Mz7WFVxdcnpWVTFwRiZ9/5UxRXqak9HQ7i+msVndrNMONjNMuFgN8uEg90sE4NI0M07GTeEElhIT8Z1tgQWxkjG9akEFupD1Qk6MxvhYDfLhIPdLBMOdrNMONjNMtGrbHybWfe68aGWwEJ65r27JbB1O+x7CWzdWNW4s/Fm2XOwm2XCwW6WiUbX7JJOAq8CF4DzEbFjGosys+mbRoLudyPi3BSe5zLzLoGF9GTcEEpgYYxkXGdLYCE9GdenEtjV5qbz23izTDQN9gD+XdL3ys4vZtZRTd8bvC8iTkv6FeCgpB9FxLeXT3D7J7NuaHRmj4jT5fcl4FsUnV1H59wfETsiYoeD3Wx+Jg52SW+WdM3F28DvA8emtTAzm64mb+M3A9+SdPF5/jEi/nXcJ+lqCWzd+FBLYGGMzHtnS2AhPfO+WLN9F0tgV5M+t0mvt+ep/2c3s47xn97MMuFgN8uEg90sE63/f/bUhNy8S2DrxgdbAgvpybjOlsBCejKubyWwLpc1s0QOdrNMONjNMuFgN8uEg90sE3P/dNmulsBCeuZ9ECWwkJ5572wJLKRn3odRAutPlzWzFRzsZplwsJtlwsFulolWE3Qi/bfLvEtgIT0Zt1izfa9KYOvm9qoEFtKTcX0rgXW5rJklcrCbZcLBbpYJB7tZJq541S9pP/CHwFJEvKsc2wg8RJFXOgncERG/mGQBXa2Kg/Rk3CCq4up22KuqOEhPxvWpKm46Us7sDwC7R8b2AYci4ibgUHnfzDrsisFednh5aWR4D3CgvH0AuH26yzKzaZv0/cLmiDhT3n6B+j8fu/2TWUc0TtBFRFA0eKx7/FL7J2cDzeZn0vg7K2kLQPl9aXpLMrNZmPRt/GPAXuDe8vujqRuOZtm7WgIL6Zn3QZTAwhiZ966WwNaND7cEdhxXPLNL+jrwX8A7JJ2SdBdFkN8q6Tng98r7ZtZhV/zVEhF31jz0/imvxcxmyDkzs0w42M0y0fr/Zx9NyHW1BBbSk3HbarbvVQksjJGM62oJLKQn44ZRAjsOn9nNMuFgN8uEg90sEw52s0w42M0y0Xo2fjSj3tUSWEjPvA+jBLZuh30qgYX0zPtws+51fGY3y4SD3SwTDnazTDjYzTLR2QTdvEtgIT0ZN4wSWEhPxnW1BLZuPL9kXBWf2c0y4WA3y4SD3SwTDnazTKR8Bt1+SUuSji0b+6yk05KOll+3zXaZZtZUSjrxAeBvgH8YGb8vIr4wzs6qsvFdLYGF9Mz7MEpgIT3zvliz/bxLYFcbn3TeuHO7a9L2T2bWM02u2e+R9HT5Nv8tU1uRmc3EpMH+ZeBtwHbgDPDFuomS7pb0pKQnL0y4MzNrbqJgj4izEXEhIt4AvgLsXGXupV5vdd1fzGz2Jso8SNqyrIvrB4Bjq82/aA0rE3JdLYGFMZJxgyiBhfRknEtg++iKR1a2f7oFuE7SKeAzwC2StlN0bz0JfGR2SzSzaZi0/dNXZ7AWM5shV9CZZcLBbpYJB7tZJlpNPa5hZfa9qyWwMEbmfRAlsJCeee9qCew4c4ebda/jM7tZJhzsZplwsJtlwsFulolWsxQLrEzILdbMnXsJLKQn4wZRAgvpyTiXwPaRz+xmmXCwm2XCwW6WCQe7WSYc7GaZaD0bvzgy1tkSWEjPvA+iBBbSM+9dLYEdd25efGY3y4SD3SwTDnazTKS0f7pB0mFJz0p6RtLHy/GNkg5Keq787s+ON+uwlGzGeeCTEfGUpGuA70k6CHwYOBQR90raB+wDPrXaE61lZUKusyWwdXMHWwJbN+4S2KFIaf90JiKeKm+/ChyneBXuAQ6U0w4At89ojWY2BWNds0vaBvwWcATYvOyz41+g/iRtZh2QHOySrga+AXwiIl5Z/lhEBMVnyFdtd6n90/80WqqZNZEU7JLWUgT61yLim+XwWUlbyse3AEtV2y5v/7RhGis2s4mkdIQRRVOI4xHxpWUPPQbsBe4tvz+asrPR9/rbaubOvSquboeDrYqD9GScq+L6KOWn+DvAnwA/lHS0HPs0RZA/LOku4GfAHTNZoZlNRUr7p+8Aqnn4/dNdjpnNiivozDLhYDfLhIPdLBOtpjmvYmX2vbMlsDBG5n0IJbCQnnl31r2PfGY3y4SD3SwTDnazTDjYzTLReoJuNL3V2RJYGCMZN4QS2LpxJ+OGwmd2s0w42M0y4WA3y4SD3SwTDnazTLSaEl3PymR4d0tg63Y41BLY1cYnnTfuXJsln9nNMuFgN8uEg90sE03aP31W0mlJR8uv22a/XDObVJP2TwD3RcQXUne2sBauHf142c6WwEJ6Ms4lsOPPtbalfODkGeBMeftVSRfbP5lZjzRp/wRwj6SnJe13F1ezbmvS/unLwNuA7RRn/i/WbHep/dOLbzRfsJlNZuL2TxFxNiIuRMQbwFeAnVXbLm//tMm5f7O5ScnGV7Z/utjnrfQB4Nj0l2dm09Kk/dOdkrZTdG89CXzkis+0gZUZ9c6WwEJ65n2xZvs+lcCOM9dZ9z5q0v7p8ekvx8xmxVfRZplwsJtlwsFulol2My1vYmV+rLMlsJCejHMJrHWfz+xmmXCwm2XCwW6WCQe7WSYc7GaZaDfVWlUu29kSWEjPvA+hBHbcudY3PrObZcLBbpYJB7tZJhzsZploNyNT1f+psyWwkJ6McwmsdZ/P7GaZcLCbZcLBbpaJlA+cXC/pu5J+ULZ/+lw5fqOkI5JOSHpI0rrZL9fMJpWSqXkd2BURr5UfKf0dSf8C/ClF+6cHJf0dcBfFZ8nXW8/KhFxnq+IgPRnnqjjrviue2aPwWnl3bfkVwC7gkXL8AHD7LBZoZtOR2iRiofwY6SXgIPBT4OWIOF9OOYX7v5l1WlKwl51ftlO8N94JvDN1B5e1fzo32SLNrLmxsvER8TJwGHgvsCjp4gXhVuB0zTb/3/7puiZLNbMmUrLxmyQtlrc3ALcCxymC/oPltL3AozNao5lNQUqqdgtwQNICxS+HhyPinyU9Czwo6S+B71P0g1vdmnWwYTRL3tUS2Lpxl8BaP6W0f3qaoif76Pjz1HRuNbPucQWdWSYc7GaZcLCbZaLlrE5V/6eulsBCejLOJbDWfT6zm2XCwW6WCQe7WSYc7GaZcLCbZaLltO5VrMy+d7UEFtIz7866W/f5zG6WCQe7WSYc7GaZcLCbZaLlDNA6VibkuloCWzfuZJz1k8/sZplwsJtlwsFulokm7Z8ekPTfko6WX9tnvlozm1iT9k8AfxYRj6yyrZl1RMoHTgZQ1f5pAmtZmX1frJk77xLY1cYnnTfuXLPpmaj9U0QcKR/6vKSnJd0n6apZLdLMmpuo/ZOkdwF/TtEG6reBjcCnqra9rP3Ti69MZ9VmNrZJ2z/tjogzZYfX14G/p+Yz5C9r/7Tp2sYLNrPJTNr+6UeStpRjomjXfGx2yzSzppq0f/oPSZsAAUeBj6btbnFkzCWwZm1o0v5p10xWZGYz4Qo6s0w42M0y4WA3y4SD3SwTLaeLF1iZfe9qCew4c511t+7zmd0sEw52s0w42M0y4WA3y0TLmaU1rEy8uQTWrA0+s5tlwsFulgkHu1kmHOxmmXCwm2WiA9n4rpbAjjvXrNt8ZjfLhIPdLBMOdrNMONjNMqGiu1NLO5NeBH5W3r0OONfaztvj4+qfIR3br0fEpqoHWg32y3YsPRkRO+ay8xnycfXPkI9tOb+NN8uEg90sE/MM9vvnuO9Z8nH1z5CP7ZK5XbObWbv8Nt4sE60Hu6Tdkn4s6YSkfW3vf5ok7Ze0JOnYsrGNkg5Keq78/pZ5rnESkm6QdFjSs5KekfTxcrzXxyZpvaTvSvpBeVyfK8dvlHSkfE0+JGndvNc6C60Ge9kJ9m+BPwBuBu6UdHOba5iyB4DdI2P7gEMRcRNwqLzfN+eBT0bEzcB7gI+V/059P7bXgV0R8W5gO7Bb0nuAvwLui4i3A78A7prfEmen7TP7TuBERDwfEf8LPAjsaXkNUxMR3wZeGhneAxwobx+g6F3fKxFxJiKeKm+/ChwHrqfnxxaF18q7a8uvAHYBj5TjvTuuVG0H+/XAz5fdP1WODcnmiDhT3n4B2DzPxTQlaRtFy+4jDODYJC1IOgosAQeBnwIvR8T5csoQX5OAE3QzFcWfOnr75w5JVwPfAD4REa8sf6yvxxYRFyJiO7CV4p3mO+e7ova0HeyngRuW3d9ajg3JWUlbAMrvS3Nez0QkraUI9K9FxDfL4UEcG0BEvAwcBt4LLEq6+EklQ3xNAu0H+xPATWX2cx3wIeCxltcwa48Be8vbe4FH57iWiUgS8FXgeER8adlDvT42SZskLZa3NwC3UuQjDgMfLKf17rhStV5UI+k24K8pWrruj4jPt7qAKZL0deAWiv81dRb4DPBPwMPAr1H8D787ImI0iddpkt4H/CfwQ+CNcvjTFNftvT02Sb9JkYBboDjRPRwRfyHprRTJ4o3A94E/jojX57fS2XAFnVkmnKAzy4SD3SwTDnazTDjYzTLhYDfLhIPdLBMOdrNMONjNMvF/xl6MrEoBvNoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = create_trainset(target2)\n",
    "w, pred = train_outputs(x, y)\n",
    "\n",
    "plt.imshow(pred.reshape(target1.shape), cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsurprisingly, this doesn't match up so well. But it is a good enough approximation for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hidden Neuron Training\n",
    "\n",
    "As we have seen, our simple one-neuron model approximates the second shape with quite a bit of error.\n",
    "To achieve a better fit, we'll need to add hidden layer(s).\n",
    "\n",
    "When we add a hidden neuron, we:\n",
    "\n",
    "1. Freeze all other parameters (including output)\n",
    "2. Run the training sample forward through the net and use the input values and other hidden unit values as the input of the new unit\n",
    "3. Train the new neuron such that its value best correlates with the residual errors calculated in an earlier run\n",
    "\n",
    "The covariance function S (we'll use it instead of correlation; more details on that in [1]) is given by:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "V_p &= \\phi \\ (I_p \\cdot W) \\\\\n",
    "S &= \\sum_{o} \\left| \\sum_{p} (V_p - \\bar{V}) (E_{o,p} - \\bar{E_o}) \\right| \\\\\n",
    "\\end{align} \n",
    "$$\n",
    "\n",
    "Where phi is the activation function of the neuron, V is the value of it, and E is the residual error (the earlier prediction minus the actual target). The bar-terms are the means and o and p are indices of output values and samples respectively.\n",
    "\n",
    "With this as our 'loss' function, we can simply run quickprop again.\n",
    "\n",
    "_Note that in hindsight, using quickprop here was a bit dangerous, since it uses an approximation of the second-order derivative under the hood. This fails to be a good solution, since the first-order derivative of $S$ is not continuous due to the magnitude calculation. In the CasCor paper, Fahlman et al. actually use gradient ascent for training the neuron._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_hidden(x, y, predicted, debug=False):\n",
    "    # Figure out how many weights we need\n",
    "    n, i = x.shape\n",
    "    \n",
    "    # And initialize a weights matrix\n",
    "    weights = torch.Tensor(rnd.uniform(-0.01, 0.01, size=(i, 1)))\n",
    "    \n",
    "    # Calculate the residuals for correlation\n",
    "    err = torch.Tensor(y - predicted)\n",
    "    err_mean = torch.mean(err, axis=0)\n",
    "    err_corr = (err - err_mean)\n",
    "    \n",
    "    if debug:\n",
    "        plt.imshow(err_corr.reshape(target1.shape), cmap='hot', interpolation='nearest')\n",
    "        plt.show()\n",
    "    \n",
    "    # Create a custom loss function (S)\n",
    "    def covariance(pred, target):\n",
    "        pred_mean = torch.mean(pred, axis=0)\n",
    "        # We want to try to maximize the absolute covariance, but quickprop will minimize its loss function\n",
    "        # Therefore, we need to multiply by (-1) to guide the optimizer correctly\n",
    "        loss = -torch.sum(torch.abs(torch.sum((pred - pred_mean)*(target), axis=0)), axis=0)\n",
    "        return loss\n",
    "        \n",
    "    # Use quickprop to generate the weights based on the special loss function\n",
    "    # We also need to pass in the residual errors as a target\n",
    "    weights, predicted, i, loss = quickprop(x, err_corr, weights, loss=covariance)\n",
    "    \n",
    "    return weights, predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run this with the one-layer net predictions (`pred`) of the last sample set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQKUlEQVR4nO3df+hd9X3H8edrmc7QWkwwC8HYxXV2ImXNaCYt6x8unSOTQSwU0bGRgWAHE1pWRrP+03as4KCt+2Ojo8XMDLqq2HbKsNtCFugKIzW1qY3GanRfabKYRGpWbW30m7z3xzmp3++953y/597z455zP68HfPne+/meH5+T733n3PP+vu95KyIws/n3C7OegJl1w8FulggHu1kiHOxmiXCwmyXCwW6WiFrBLmmHpB9IOiZpd1OTMrPmadq/s0taAzwD3AQcBx4Dbo+Ip8rWuXKtYsvbptrdbHRZgjDJvoqWLVv/Qs1l3/megsHFkg28UTD2esmy5yqOAfxsgmULxoumVXYIRePnS5YtGi/6NywbL/s9TPL7HRlf+D946aehokV/sWQTVdwAHIuI5wEk3Q/sBEqDfcvb4NBtNfbYtbJfchvrF70YytYvevEWjUFxrBXFDhTHz75DBYMvlWzgxYKxF0qWXSgYe65k2aerLxvPjI8VTetMya5+VDB2tmTZVwrGflKy7E8Lxsp+Z5P8nznyutl2b8ly1HsbfxXwwyXPj+djZtZDrSfoJN0p6ZCkQ2dea3tvZlamTrCfAK5e8nxzPrZMRHwxIrZFxLYNa2vszcxqqXPN/hhwraRryIL8NuAPG5lV12Z9bV62jSau6SbJbf24aLDo+rzoIhiKr88XSpYtuuYuujYvWbbo2hyqX58XXZtD8fV50bU5FF+fF12bQ/08yySvmxJTB3tELEq6C/h3YA2wJyKenHZ7ZtauOmd2IuJR4NGG5mJmLXIFnVkiHOxmiXCwmyWi1jX7IM06896HqrjCrDslWecBVcVB9cz72ZL1e1gVB1Qv2V2h7NpndrNEONjNEuFgN0uEg90sEfOboKubiJtkG0MrgS0r/ywsIR1QCSxUT8YNrQS27LVYto0CPrObJcLBbpYIB7tZIhzsZolwsJslYj6y8S6BnawEtuzGDYXjCwVjPS2BheqZ9yGVwJatPyGf2c0S4WA3S4SD3SwRta7ZJS2QXRGdBxYjYlsTkzKz5jWRoPudiChrEdKsLktgoXpSZWglsCXJrZcL5rtuSCWwUD0ZNyclsP48u5mNqRvsAfyHpO9IurOJCZlZO+q+jX9/RJyQ9MvAPklPR8Q3ly6Q/ydwJ8DbL6+5NzObWq0ze0ScyL+fBr5O1tl1dBm3fzLrgamDXdJbJF1+8THwe8CRpiZmZs2q8zZ+I/B1SRe3888R8W+NzApmXwJbto05KIEtyroDnCoYWzekElionnkfWglsA3+JqtPr7Xng3fWnYGZd8J/ezBLhYDdLhIPdLBGz/zx7X0tgoXoybmAlsEWJOID/LRi7bkglsFA9GTekEtiG+MxulggHu1kiHOxmiXCwmyXCwW6WiG6z8UG9TGOXJbBl43NQAluUdQc4UTQ4pBJYqP8XlIGVwE7CZ3azRDjYzRLhYDdLhIPdLBGzL5ctM+sSWKiejBtYCWxhIq5sG0MqgS0bn+MS2En4zG6WCAe7WSIc7GaJcLCbJWLVBJ2kPcAfAKcj4l352HrgAWALWQPvWyPi5alm0NeqOKiejBtYVVxZMq+wMG5IVXFQPRk3J1Vxk6hyZr8P2DEythvYHxHXAvvz52bWY6sGe97hZfT/8p3A3vzxXuCWZqdlZk2b9pp9Y0SczB+/SHYP+UKS7pR0SNKhM69NuTczq612gi4ighUaxbr9k1k/TBvspyRtAsi/n25uSmbWhmnLZR8BdgF3598frrxm1WzlrEtgoXrmfWAlsJN8HH1QJbBQ/XUzJyWwk8x31TO7pK8A/w38uqTjku4gC/KbJD0L/G7+3Mx6bNUze0TcXvKjDzQ8FzNrkSvozBLhYDdLxOw/z97XElionowbWAnsJB9HH1QJLFRPxs1xIq6Mz+xmiXCwmyXCwW6WCAe7WSIc7GaJ6L7902hWsa8lsFA98z6wEthJ7j0xqBLYsvEEM+9FfGY3S4SD3SwRDnazRDjYzRLRfbnsaAKkryWwUDkZN7QS2LMlyxb+MwypBHalbVRdv0stJeLK+MxulggHu1kiHOxmiXCwmyWiyj3o9kg6LenIkrFPSToh6XD+dXO70zSzuqpk4+8D/g74p5HxeyLisxPtLRjPzva1BLZkfB5KYMv++FA47hLYZnSceS8ybfsnMxuYOtfsd0l6In+bv66xGZlZK6YN9i8A7wC2AieBz5UtuKzXW9nbcDNr3VTBHhGnIuJ8RFwAvgTcsMKyb/Z6u2zaaZpZXVOVy0ratKSL6weBIyst/3NFCbqelsBC9WTc0Epgy/5pCt94DakEdqVtdKUHibgyqwZ73v7pRuBKSceBTwI3StpKFr4LwIfbm6KZNWHa9k/3tjAXM2uRK+jMEuFgN0uEg90sEd3evOIC4xnbnpbAQvXM+9BKYMv+yStn410Cm+lx5r2Iz+xmiXCwmyXCwW6WCAe7WSK6b/80mgXqaQksVE/GDa0EtixBV5indAns4BJxZXxmN0uEg90sEQ52s0Q42M0S4WA3S0T35bKj2feelsBC9cz70Epgy+49UZj0dgns3PCZ3SwRDnazRDjYzRJRpf3T1ZIOSHpK0pOSPpKPr5e0T9Kz+XffO96sx6ok6BaBj0XE45IuB74jaR/wJ8D+iLhb0m5gN/DxFbf0zvfAvkMjgy+VLFyU9nqhZNmFsZF1PFe45DqeHhu7rmRZ4plq02qrBvYnJcvOuiWTS2AHqUr7p5MR8Xj++BXgKHAVsBPYmy+2F7ilpTmaWQMmumaXtAX4TeAgsHHJveNfBDY2OzUza1LlYJf0VuCrwEcjYtlfxyMiyD7TVrTem+2fzpS93zWztlUKdkmXkAX6lyPia/nwKUmb8p9vAk4Xrbus/dOGDU3M2cymUKUjjMiaQhyNiM8v+dEjwC7g7vz7w6vvbpHxhFxZ/VlRMm6hZNmiBNt4Iq502aJEHFRPxjVRFleUjCtKxMHsWzINKREHSSbjilTJxv828MfA9yUdzsc+QRbkD0q6gywyb21lhmbWiCrtn74FqOTHH2h2OmbWFlfQmSXCwW6WCAe7WSK6/Tw7bzCe4q5eAlucdYfizHvNElionnk/W7L+kEpgy8adeZ8bPrObJcLBbpYIB7tZIhzsZonoOEH3OuMJuYWSZWdcAgvVk3HzUAK70jaqrt8lJ+Im5jO7WSIc7GaJcLCbJcLBbpYIB7tZIjrOxp9jPPve0xJYqJ55dwlsu5x5b4TP7GaJcLCbJcLBbpaIOu2fPiXphKTD+dfN7U/XzKZVp/0TwD0R8dnquzvHeJKtpyWwUD0ZNw8lsCttoytOxLWqyg0nTwIn88evSLrY/snMBqRO+yeAuyQ9IWmPu7ia9Vud9k9fAN4BbCU783+uZL0l7Z9eqz9jM5vK1O2fIuJURJyPiAvAl4AbitZd3v5pbVPzNrMJVcnGF7Z/utjnLfdB4Ejz0zOzptRp/3S7pK1k3VsXgA+vvqmfMZ5972kJLFTPvLsEdnLOvHeuTvunR5ufjpm1xRV0ZolwsJslwsFulogZfJ59JCHX1xJYqJ6McwlsOSfiesNndrNEONjNEuFgN0uEg90sEQ52s0R0n40fzb73tQQWqmfeXQKbcea913xmN0uEg90sEQ52s0Q42M0S0W2C7g3GE3J9LYEtG3cJrBNxA+Uzu1kiHOxmiXCwmyWiyg0nL5P0bUnfy9s/fTofv0bSQUnHJD0g6dL2p2tm06qSoDsHbI+IV/NbSn9L0jeAPydr/3S/pH8A7iC7l3y5RcYTcn2tioPqyThXxdkArHpmj8yr+dNL8q8AtgMP5eN7gVvamKCZNaNqk4g1+W2kTwP7yG43czYiFvNFjuP+b2a9VinY884vW4HNZJ1frqu6g2Xtn16ebpJmVt9E2fiIOAscAN4HXCHp4jX/ZuBEyTpvtn9y60ezmamSjd8g6Yr88VrgJuAoWdB/KF9sF/BwS3M0swZUycZvAvZKWkP2n8ODEfGvkp4C7pf018B3yfrBrWyR8ez72ZJlZ10CC9Uz7/NQAgvOvM+5Ku2fniDryT46/jwlnVvNrH9cQWeWCAe7WSIc7GaJ6Pbz7OcZT8j1tQQWqifjnIizAfCZ3SwRDnazRDjYzRLhYDdLhIPdLBHdZ+NHs+99LYEtG3fm3QbKZ3azRDjYzRLhYDdLhIPdLBHdJuguMJ6Q62sJ7ErbqLp+l5yIs1X4zG6WCAe7WSIc7GaJqNP+6T5J/yPpcP61tfXZmtnU6rR/AviLiHhohXXNrCeq3HAygKL2T5O7wHj23SWwk3Pm3aYwVfuniDiY/+gzkp6QdI+kX2prkmZW31TtnyS9C/hLsjZQvwWsBz5etO6y9k9lH3oxs9ZN2/5pR0SczDu8ngP+kZJ7yC9r//SW2vM1sylN2/7paUmb8jGRtWs+0t40zayuOu2f/lPSBkDAYeBPV91SMJ5462sJ7Erb6IoTcdagOu2ftrcyIzNrhSvozBLhYDdLhIPdLBEOdrNEdHvziqJsvEtgM868W8t8ZjdLhIPdLBEOdrNEONjNEtF9gm40IecSWLNO+MxulggHu1kiHOxmiXCwmyXCwW6WiO6z8aPZaJfAmnXCZ3azRDjYzRLhYDdLhIPdLBHKujt1tDPpDPBC/vRK4KXOdt4dH9fwzNOx/UpEbCj6QafBvmzH0qGI2DaTnbfIxzU883xsS/ltvFkiHOxmiZhlsH9xhvtuk49reOb52H5uZtfsZtYtv403S0TnwS5ph6QfSDomaXfX+2+SpD2STks6smRsvaR9kp7Nv6+b5RynIelqSQckPSXpSUkfyccHfWySLpP0bUnfy4/r0/n4NZIO5q/JByRdOuu5tqHTYM87wf498PvA9cDtkq7vcg4Nuw/YMTK2G9gfEdcC+/PnQ7MIfCwirgfeC/xZ/nsa+rGdA7ZHxLuBrcAOSe8F/ga4JyJ+DXgZuGN2U2xP12f2G4BjEfF8RLwO3A/s7HgOjYmIbwI/GhneCezNH+8l610/KBFxMiIezx+/AhwFrmLgxxaZV/Onl+RfAWwHHsrHB3dcVXUd7FcBP1zy/Hg+Nk82RsTJ/PGLwMZZTqYuSVvIWnYfZA6OTdIaSYeB08A+4DngbEQs5ovM42sScIKuVZH9qWOwf+6Q9Fbgq8BHI+LHS3821GOLiPMRsRXYTPZO87rZzqg7XQf7CeDqJc8352Pz5JSkTQD599Mzns9UJF1CFuhfjoiv5cNzcWwAEXEWOAC8D7hC0sUbuczjaxLoPtgfA67Ns5+XArcBj3Q8h7Y9AuzKH+8CHp7hXKYiScC9wNGI+PySHw362CRtkHRF/ngtcBNZPuIA8KF8scEdV1WdF9VIuhn4W2ANsCciPtPpBBok6SvAjWSfmjoFfBL4F+BB4O1kn/C7NSJGk3i9Jun9wH8B3+fNm2t9guy6fbDHJuk3yBJwa8hOdA9GxF9J+lWyZPF64LvAH0XEudnNtB2uoDNLhBN0ZolwsJslwsFulggHu1kiHOxmiXCwmyXCwW6WCAe7WSL+HwhyE/kiC5pGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAMH0lEQVR4nO3df6jdd33H8efLmK4yO9pgFkJTZ6cFKTIzuOsqE9ZldGT9pxWkWNjIoFAHFhRkmPmPdUxQULt/hpBh1vzhrKE6G8T9CF3A+U9srLGmSWvTrp3J0qRdzZpstF3S9/4434zb6znJyfl1z72f5wMO53s+3+855/0l95Vzzuee+3mnqpC0+r1luQuQNBuGXWqEYZcaYdilRhh2qRGGXWrEWGFPsjXJU0mOJtk+qaIkTV5G/T17kjXAT4FbgWPAo8BdVXX4Ivcp30pI0/MGUFXpt++tYzzuTcDRqnoWIMmDwO3AwLC/BbhyjCeUdHGvXmTfOC+01wI/W3T7WDcmaQ6N88o+lCT3APcA9H1vIWkmxgn7ceC6Rbc3dWNvUlU7gB0AaxK/iC8tk3Hexj8K3JDk+iRXAB8B9kymLEmTNvIre1WdS3Iv8E/AGmBnVT0xscokTdTIv3obxZqknI2XpudV4PyAX735a2+pEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdasRYS0kneQ44A5wHzlXVwiSKkjR5k1g3/veq6qUJPI6kKfJtvNSIccNewD8n+WHX+UXSnBr3bfwHq+p4kl8F9iZ5sqq+t/gA2z9J82Fi68YnuQ84W1VfHHSM68ZL0zWVdeOT/HKSqy5sA38AHBr18SRN1zhv4zcAf5/kwuP8XVX940SqkjRxtn+SVhHbP0ky7FIrDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUiEuGPcnOJKeSHFo0ti7J3iRPd9fXTLdMSeMa5pX9AWDrkrHtwCNVdQPwSHdb0hy7ZNi7Di8vLxm+HdjVbe8C7phsWZImbdR14zdU1Ylu+wV6a8j3ZfsnaT6MPUFXvYXnBy4+X1U7qmqhqhYMu7R8Rg37ySQbAbrrU5MrSdI0jBr2PcC2bnsb8PBkypE0LZds/5Tk68AtwDuAk8BngG8Du4F3As8Dd1bV0km8X2D7J2m6Ltb+yV5v0ipirzdJhl1qhWGXGmHYpUYYdqkRhl1qhGGXGmHYpUYYdqkRhl1qhGGXGmHYpUYYdqkRhl1qhGGXGmHYpUYYdqkRhl1qxKjtn+5LcjzJwe5y23TLlDSuUds/AdxfVZu7y3cnW5akSRu1/ZOkFWacz+z3Jnm8e5tvF1dpzo0a9q8A7wY2AyeALw06MMk9SQ4kOTC7RaslLTXUuvFJ3gV8p6redzn7lnLdeGm6Jr5u/IU+b50PAYcGHStpPlyyZfPi9k9JjtFr/3RLks30urc+B3x0eiVKmgTbP0mriO2fJBl2qRWGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGX/Hv2SboK+N0lY2cGHPs/fcb+e8Cxr/YZ+98Bx/YbP38Zx/Yz6P5vDHn/y31caRS+skuNMOxSIwy71Ihh2j9dl2RfksNJnkjy8W58XZK9SZ7url07Xppjl1yDrltJdmNVPZbkKuCHwB3AnwAvV9Xnk2wHrqmqT13ssRYWFurAgQNjlHtujPuO8hj9jh12DPpPHfYbAzg75BjAf/UZe2nAscf6jB0fcOzRPmNP/eLQK//e/+7P9hl7csBT9XlYDg84tt/4gGOf6TMr2q+s5wY81X/0GTs54Nj/7DN2esCx/Sai+01Cw+VNOL++5PZJ4PVR16CrqhNV9Vi3fQY4AlwL3A7s6g7bRe8/AElz6rI+s3cNIX4T2A9sqKoT3a4XgA2TLU3SJA0d9iRvB74JfKKqXlm8r3qfBfp+Hljc/unFF18cq1hJoxsq7EnW0gv616rqW93wyQudYbrrU/3uW1U7qmqhqhbWr18/iZoljWCYjjABvgocqaovL9q1B9gGfL67fngqFb7JJL7wN9MvDa5OvzJgfPOQY1P07iHH5tvoE9ELC789cN8wP/m/A/wx8JMkB7uxT9ML+e4kdwPPA3eOXKGkqbtk2Kvq+0DfqXzg9ydbjqRp8Rt0UiMMu9QIwy41wqlpae6ME8tB02u+skvNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSI8Zp/3RfkuNJDnaX26ZfrqRRDfOHs+eATy5u/5Rkb7fv/qr64vTKkzQpwyw4eQI40W2fSXKh/ZOkFWSc9k8A9yZ5PMlOu7hK822c9k9fobf+/mZ6r/xfGnA/2z9Jc2Dk9k9VdbKqzlfVG8DfADf1u6/tn6T5MMxsfN/2Txf6vHU+BByafHmSJmWc9k93JdlMr3vrc8BHp1CfpAkZp/3TdydfjqRp8Rt0UiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9SIYRacvDLJD5L8uGv/9Nlu/Pok+5McTfKNJFdMv1xJoxrmlf01YEtVvZ/eGvFbk9wMfIFe+6f3AD8H7p5alZLGdsmwV8/Z7uba7lLAFuChbnwXcMc0CpQ0GcM2iVjTLSN9CtgLPAOcrqpz3SHHsP+bNNeGCnvX+WUzsIle55f3DvsEtn+S5sNlzcZX1WlgH/AB4OokF9ad3wQcH3Af2z9Jc2CY2fj1Sa7utt8G3AocoRf6D3eHbQMenlKNkiZgmPZPG4FdSdbQ+89hd1V9J8lh4MEkfwn8iF4/OElzapj2T4/T68m+dPxZBnRulTR//Aad1AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjVinPZPDyT5tyQHu8vmqVcraWTDLDh5of3T2SRrge8n+Ydu359V1UMXua+kOTHMgpMF9Gv/JGkFGan9U1Xt73Z9LsnjSe5P8kvTKlLS+EZq/5TkfcCf02sD9VvAOuBT/e5r+ydpPoza/mlrVZ3oOry+BvwtA9aQt/2TNB9Gbf/0ZJKN3VjotWs+NL0yJY1rnPZP/5JkPRDgIPCn0ytT0rjGaf+0ZSoVSZoKv0EnNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUiPS6O83oyZIXgee7m+8AXprZk8+O57XyrKZz+7Wq6tugYaZhf9MTJweqamFZnnyKPK+VZzWf22K+jZcaYdilRixn2Hcs43NPk+e18qzmc/t/y/aZXdJs+TZeasTMw55ka5KnkhxNsn3Wzz9JSXYmOZXk0KKxdUn2Jnm6u75mOWscRZLrkuxLcjjJE0k+3o2v6HNLcmWSHyT5cXden+3Gr0+yv/uZ/EaSK5a71mmYadi7TrB/DfwhcCNwV5IbZ1nDhD0AbF0yth14pKpuAB7pbq8054BPVtWNwM3Ax7p/p5V+bq8BW6rq/cBmYGuSm4EvAPdX1XuAnwN3L1+J0zPrV/abgKNV9WxVvQ48CNw+4xompqq+B7y8ZPh2YFe3vYte7/oVpapOVNVj3fYZ4AhwLSv83KrnbHdzbXcpYAvwUDe+4s5rWLMO+7XAzxbdPtaNrSYbqupEt/0CsGE5ixlXknfRa9m9n1VwbknWJDkInAL2As8Ap6vqXHfIavyZBJygm6rq/apjxf66I8nbgW8Cn6iqVxbvW6nnVlXnq2ozsIneO833Lm9FszPrsB8Hrlt0e1M3tpqcTLIRoLs+tcz1jCTJWnpB/1pVfasbXhXnBlBVp4F9wAeAq5O8tdu1Gn8mgdmH/VHghm728wrgI8CeGdcwbXuAbd32NuDhZaxlJEkCfBU4UlVfXrRrRZ9bkvVJru623wbcSm8+Yh/w4e6wFXdew5r5l2qS3Ab8FbAG2FlVn5tpAROU5OvALfT+auok8Bng28Bu4J30/sLvzqpaOok315J8EPhX4CfAG93wp+l9bl+x55bkN+hNwK2h90K3u6r+Ismv05ssXgf8CPijqnpt+SqdDr9BJzXCCTqpEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVG/B81gHwdCEDkCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "neuron_w, neuron_value = train_hidden(x, y, pred, debug=True)\n",
    "\n",
    "plt.imshow(neuron_value.reshape(target1.shape), cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows us pretty nicely where the error currently is biggest (very dark or very bright spots in the first plot) and what the neuron tries to do to correlate with that (second plot).\n",
    "\n",
    "### Combining Hidden & Output Neurons\n",
    "\n",
    "As the last step (of this article) we now train our output layer again, additionally based on the values (`neuron_value`) computed by our newly trained neuron.\n",
    "\n",
    "To do so, we need to include these values as input to the output neurons (`x2`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAL50lEQVR4nO3df+hdd33H8efLmKyyymowC6Xp1k7DXBlrpFmozEEX15H1n1QQsbCRPwp1YEFBxjr/UccEhWn3zxBazJo/nLVUXcPofoQYcMKIjTXWtNU1dhET0qRFi+0/HUne++OehK/Z/Ta395776/t5PuBwz/mcc7/3fdr7yrnfc8/3vFNVSFr73jTvAiTNhmGXGmHYpUYYdqkRhl1qhGGXGjFR2JPsSvKjJMeT3NdXUZL6l3G/Z0+yDvhv4HbgJPAEcFdVPfM6zyk/Sszf+lXGbxgydvUt64aM/s4qP2HDkLGMUpJ6cuLECV566aWh/9HfPMHP3QEcr6rnAZI8DOwGVg37m4CrJnhBvXHDorp5lW0fHDL2h0d+bcjoN1b5CTcMGZvkLaY3avv27auum+RAex3w0xXLJ7sxSQto6v/sJrkHuAf8QCfN0yRhPwVcv2J5Szf2S6rqAeABgHWJF+JLczLJx/gngK1JbkyyAfgQsL+fsjQP54dMcG7IpGU09pG9qs4luRf4dwbngfZW1dO9VSapVxP9zl5VjwOP91SLpCnya2+pEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVG+PeHuuTC0FEvj10rPLJLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjZjoCrokJ4BXGNyb8FxVrX6Heklz1cflsn9UVS/18HMkTZEf46VGTBr2Av4jyXe7zi+SFtSkH+PfW1Wnkvw6cCDJD6vqWys3sP2TtBgmOrJX1anu8SyD1p47hmzzQFVtr6rthl2an7HDnuRXk7z14jzwJ8CxvgqT1K9JPsZvBr6R5OLP+aeq+rdeqtJcnB866s0r1opJer09D9zcYy2Spsiv3qRGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVG2OtNl3i57NrmkV1qhGGXGmHYpUYYdqkRhl1qhGGXGmHYpUYYdqkRhl1qxBXDnmRvkrNJjq0Y25jkQJLnuse3TbdMzcKFIZPWjlGO7A8Buy4buw84WFVbgYPdsqQFdsWwdx1efnbZ8G5gXze/D7iz37Ik9W3cP4TZXFWnu/kXGNxDfijbP0mLYeITdFVVDBo8rrbe9k/SAhg37GeSXAvQPZ7tryRJ0zBu2PcDe7r5PcBj/ZSjWRh21n3VM+914f9PWkqjfPX2FeC/gN9OcjLJ3cBngduTPAf8cbcsaYFd8QRdVd21yqr39VyLpCnyCjqpEYZdaoQ3nNQYvAnlMvLILjXCsEuNMOxSIwy71AjDLjXCs/G6ZGj7p2GDvmuWkkd2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRHjtn/6VJJTSY520x3TLVPSpMZt/wRwf1Vt66bH+y1L83B+yGQDuLVj3PZPkpbMJL+z35vkqe5jvl1cpQU3bti/CLwD2AacBj6/2oZJ7klyJMmRVXtESZq6scJeVWeq6nxVXQAeBHa8zrb2epMWwFhhv9jnrfN+4Nhq20paDFe8DUHX/uk24O1JTgKfBG5Lso1B99YTwIenV6KkPozb/ulLU6hF0hR5BZ3UCMMuNcKwS43wPqG6ZOiVsENvOatl5JFdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxox08tl333LLRw5cmSWLymp45FdaoRhlxph2KVGjNL+6fokh5I8k+TpJB/txjcmOZDkue7Re8dLC2yUI/s54ONVdRNwK/CRJDcB9wEHq2orcLBblrSgRmn/dLqqnuzmXwGeBa4DdgP7us32AXdOqUZJPXhDv7MnuQF4N3AY2FxVp7tVLwCb+y1NUp9GDnuSq4GvAR+rql+sXFdVxeAe8sOed6n904svvjhRsZLGN1LYk6xnEPQvV9XXu+EzFzvDdI9nhz13ZfunTZs29VGzpDGMcjY+DJpCPFtVX1ixaj+wp5vfAzzWf3mS+jLK5bJ/APw58IMkR7uxTwCfBR5JcjfwE+CDU6lQUi9Gaf/0bWC1Bqzv67ccSdPiFXRSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71AjDLjXCsEuNMOxSIwy71IhJ2j99KsmpJEe76Y7plytpXKPccPJi+6cnk7wV+G6SA926+6vq76ZXnqS+jHLDydPA6W7+lSQX2z9JWiKTtH8CuDfJU0n22sVVWmyTtH/6IvAOYBuDI//nV3me7Z+kBTB2+6eqOlNV56vqAvAgsGPYc23/JC2Gsds/Xezz1nk/cKz/8iT1ZZL2T3cl2cage+sJ4MNTqE9STyZp//R4/+VImhavoJMaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRoxyw8mrknwnyfe79k+f7sZvTHI4yfEkX02yYfrlShrXKEf214CdVXUzg3vE70pyK/A5Bu2f3gn8HLh7alVKmtgVw14Dr3aL67upgJ3Ao934PuDOaRQoqR+jNolY191G+ixwAPgx8HJVnes2OYn936SFNlLYu84v24AtDDq/vGvUF7D9k7QY3tDZ+Kp6GTgEvAe4JsnF+85vAU6t8hzbP0kLYJSz8ZuSXNPNvwW4HXiWQeg/0G22B3hsSjVK6sEo7Z+uBfYlWcfgH4dHqupfkjwDPJzkb4HvMegHJ2lBjdL+6SkGPdkvH3+eVTq3Slo8XkEnNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS42YpP3TQ0n+J8nRbto29WoljW2UG05ebP/0apL1wLeT/Gu37i+r6tHXea6kBTHKDScLGNb+SdISGav9U1Ud7lZ9JslTSe5P8ivTKlLS5MZq/5Tkd4G/ZtAG6veBjcBfDXuu7Z+kxTBu+6ddVXW66/D6GvCPrHIPeds/SYth3PZPP0xybTcWBu2aj02vTEmTmqT90zeTbAICHAX+YnplSprUJO2fdk6lIklT4RV0UiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjcigu9OMXix5EfhJt/h24KWZvfjsuF/LZy3t229W1dAGDTMN+y+9cHKkqrbP5cWnyP1aPmt531byY7zUCMMuNWKeYX9gjq89Te7X8lnL+3bJ3H5nlzRbfoyXGjHzsCfZleRHSY4nuW/Wr9+nJHuTnE1ybMXYxiQHkjzXPb5tnjWOI8n1SQ4leSbJ00k+2o0v9b4luSrJd5J8v9uvT3fjNyY53L0nv5pkw7xrnYaZhr3rBPsPwJ8CNwF3JblpljX07CFg12Vj9wEHq2orcLBbXjbngI9X1U3ArcBHuv9Py75vrwE7q+pmYBuwK8mtwOeA+6vqncDPgbvnV+L0zPrIvgM4XlXPV9X/Ag8Du2dcQ2+q6lvAzy4b3g3s6+b3Mehdv1Sq6nRVPdnNvwI8C1zHku9bDbzaLa7vpgJ2Ao9240u3X6OaddivA366YvlkN7aWbK6q0938C8DmeRYzqSQ3MGjZfZg1sG9J1iU5CpwFDgA/Bl6uqnPdJmvxPQl4gm6qavBVx9J+3ZHkauBrwMeq6hcr1y3rvlXV+araBmxh8EnzXfOtaHZmHfZTwPUrlrd0Y2vJmSTXAnSPZ+dcz1iSrGcQ9C9X1de74TWxbwBV9TJwCHgPcE2SN3er1uJ7Eph92J8AtnZnPzcAHwL2z7iGadsP7Onm9wCPzbGWsSQJ8CXg2ar6wopVS71vSTYluaabfwtwO4PzEYeAD3SbLd1+jWrmF9UkuQP4e2AdsLeqPjPTAnqU5CvAbQz+auoM8Engn4FHgN9g8Bd+H6yqy0/iLbQk7wX+E/gBcKEb/gSD39uXdt+S/B6DE3DrGBzoHqmqv0nyWwxOFm8Evgf8WVW9Nr9Kp8Mr6KRGeIJOaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEf8HX7lquYWkamoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x2 = np.concatenate((x, neuron_value), axis=1)\n",
    "\n",
    "w2, pred2 = train_outputs(x2, y)\n",
    "\n",
    "plt.imshow(pred2.reshape(target1.shape), cmap='hot', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty!\n",
    "\n",
    "The output neuron can now approximate the shape because it can base its regression on the additional value of the hidden neuron.\n",
    "And since that correlates with the error we got from the earlier optimum, adding it has the effect of reducing the problem the output neuron has to solve by one dimension.\n",
    "\n",
    "### Caveats\n",
    "\n",
    "All the pictures are the result of executing the code snippets, however, what you cannot see is how often they were run until they generated those images.\n",
    "Particularly, training the neuron only converges with a very low probability to the weights that produce the displayed values, i.e. I had to run the algorithm a couple of times until I got that particular output.  \n",
    "This can presumably be attributed in part to the usage of quickprop here, since it relies on some properties of the loss function, like being continuous in its first derivative. As I mentioned above, Fahlman et. al. actually use gradient ascent in [1] to train the hidden neurons.\n",
    "\n",
    "But there are more problems with quickprop that the simple implementation above does not address properly. From Fahlman's paper [2] and another, more recent comparison of quickprop with back-propagation [3] it becomes clear that a) it will need some more adjustments to yield good results more reliably, and b) it doesn't perform equally well on all domains. Specifically, it fails to compete with back-prop on bias and variance in some real-world image classification problems.\n",
    "\n",
    "That said, a comparable network structure (1 hidden, 1 output neuron) trained with standard back-propagation and using the Adam optimizer never even converged to the above result during the test runs (it needed at least another neuron to converge to that), but that could be bad luck since it wasn't a controlled test setup.\n",
    "\n",
    "Another problem of CasCor in general is the thinness of the network vs. its depth.\n",
    "Since computers are very good at linear algebra (through the usage of GPUs), it is faster to run through broader networks that are less deep (i.e. that can better be represented as matrices), as compared to thinner and deeper networks, such as CasCor generates by default.\n",
    "\n",
    "This is of course not so much of a problem with our little example, but it might become a problem when the generated network solves a more complicated task.\n",
    "\n",
    "Which brings us to the next section:\n",
    "\n",
    "### Future Improvements\n",
    "\n",
    "As we have seen, this basic implementation of CasCor in fact works! :)\n",
    "\n",
    "However, we are still missing a lot of boilerplate code that automates the process, and some optimization of the training method, to find a global optimum with higher probability.\n",
    "\n",
    "That is why, in the next parts of this series, we'll see how we can:\n",
    "\n",
    "- Automate the _output -> hidden -> output -> hidden -> ..._ training cycle\n",
    "- Change quickprop to deliver more stable results and to train a set of new nodes instead of only one (and pick the best)\n",
    "- Further improve that process to pick more than one node (i.e. have 'broader' layers)\n",
    "- Change the problem domain to some more interesting/challenging ones (e.g. domains that can only be solved with deeper and/or recurrent networks)\n",
    "- Benchmark it against other machine learning methods in a controlled setting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "As mentioned earler, Quickprop deserves a more detailed look all on its own. For this reason, I wrote [a more in-depth article](https://towardsdatascience.com/quickprop-an-alternative-to-back-propagation-d9a78069e2a7) covering the maths behind it, a possible implementation and some improvements.\n",
    "\n",
    "Part 2 of the series is currently in progress and will be linked here when it is published.  \n",
    "All finished notebooks and code of this series are also [available on Github](https://github.com/ephe-meral/cascor).\n",
    "Please feel encouraged to leave feedback and suggest improvements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "[1] S. E. Fahlman and C. Lebiere, [The cascade-correlation learning architecture](http://web.cs.iastate.edu/~honavar/fahlman.pdf) (1990), Advances in neural information processing systems (pp. 524–532)\n",
    "\n",
    "[2] S. E. Fahlman, [An empirical study of learning speed in back-propagation networks](http://www.it.uu.se/edu/course/homepage/mil/vt11/handouts/fahlman.quickprop-tr.pdf) (1988), Carnegie Mellon University, Computer Science Department\n",
    "\n",
    "[3] C. A. Brust, S. Sickert, M. Simon, E. Rodner and J. Denzler, [Neither Quick Nor Proper - Evaluation of QuickProp for Learning Deep Neural Networks](https://arxiv.org/pdf/1606.04333.pdf) (2016), arXiv preprint arXiv:1606.04333"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}