01_Intro_To_Distributed_Systems.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#  Introduction\n",
    "\n",
    "To understand how Blockchain works, we must first discuss how distributed systems work.\n",
    "\n",
    "The goal of this notebook is to explain the building blocks for any distributed system, and specifically a peer-to-peer system. P2P environment is a network where peers are connected, exchange information and run applications on top of it.\n",
    "\n",
    "The goals are as follows:\n",
    "- We will build a system block by block.\n",
    "- Distributed systems are hard as there are many random errors: race conditions, nasty bugs, deadlocks.\n",
    "- We will simplify the world to get to the juiciest parts without leaving out the essentials.  \n",
    "\n",
    "\n",
    "We will simulate the P2P environment locally to get an idea of how such systems would work and understand the design decision that developers make.\n",
    "\n",
    "To show the different tradeoffs and why developers and researchers choose specific designs, we need a way to simulate such an environment. Here we will simulate the network and message exchange via a  discrete event simulator: [SimPy](simpy.readthedocs.io/en/latest/).\n",
    "\n",
    "For the simplicity of use, you can use wrapper for SimPY implement a P2P network simulation: `p2psimpy`. \n",
    "\n",
    "\n",
    "# Simulation of P2P environments\n",
    "\n",
    "We start the notebook by introducing the backbone for such a simulation: `BaseSimulation`.\n",
    "This class represents the network of peers with their physical representations in the world, such as physical `location`, [`bandwidth`](https://en.wikipedia.org/wiki/Bandwidth_(computing)). \n",
    "\n",
    "These properties both define and restrict the capabilities of how fast you can transfer messages and how message travel through the network (which peers see the message, which channels are used).\n",
    "\n",
    "\n",
    "`BaseSimulation` consists of 3 main parameters: \n",
    " - *Locations*\n",
    " - *Network Topology* \n",
    " - *Services and Message Processors* \n",
    "\n",
    "We will explain them one by one. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Simulating latencies between locations\n",
    "\n",
    "The simulator simulates connection delays (`latency`) between the nodes in the physical space (`location`).\n",
    "This is a very rough representation of reality but is sufficient for us. You can read more about jitter and latency [here](https://www.tpx.com/blog/latency-jitter-and-packet-loss/) or [here](https://www.youtube.com/watch?v=WdbJdUh6W08).   \n",
    "\n",
    "\n",
    "\n",
    "Locations are defined as a `Config` class. Variables can be constant, or probabilistic (with a probability distribution using scipy).\n",
    "This class can be used as generators of parameters, saved and loaded as a [YAML](https://en.wikipedia.org/wiki/YAML) file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Location configuration should contain at least two fields: `locations` and `latencies`. \n",
    "- `locations` is an array or a tuple with names of locations.\n",
    "- `latencies` is a dictionary (matrix) with latencies of pairwise connections.\n",
    "\n",
    "Depending on the experiments, latencies can be constant, but a more realistic model is a probabilistic distribution.\n",
    "Such a distribution is represented with the `Dist` class. \n",
    "\n",
    "`Dist` class is a wrap-around [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html), which has a rich collection of distributions. Use the [link](https://docs.scipy.org/doc/scipy/reference/stats.html) as reference documentation when choosing a distribution function.   \n",
    "\n",
    "Finally,  `Config.get()` samples all the parameters from distributions and returns them as a dictionary.  \n",
    "\n",
    "Let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.256784Z",
     "start_time": "2020-03-27T09:17:34.199645Z"
    },
    "recordTiming": true
   },
   "outputs": [],
   "source": [
    "from p2psimpy.config import *\n",
    "\n",
    "\n",
    "class ConstLocations(Config):\n",
    "    '''A configuration with 2 locations: LocA and LocB with constant latencies.'''\n",
    "    locations = ['LocA', 'LocB']\n",
    "    latencies = {\n",
    "        'LocA': {'LocB': 10, 'LocA': 2},\n",
    "        'LocB': {'LocB': 1}\n",
    "    }    \n",
    "\n",
    "class DistLocations(Config):\n",
    "    '''A configuration with 2 locations: LocA and LocB with variable latencies defined with a statistical function.'''\n",
    "    locations = ['LocA', 'LocB']\n",
    "    latencies = {\n",
    "        'LocB': {'LocB': Dist('gamma', (1, 1, 1))},\n",
    "        'LocA': {'LocB': Dist('norm', (12, 2)), 'LocA': Dist('norm', (2, 0.5))},\n",
    "    }    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.274429Z",
     "start_time": "2020-03-27T09:17:35.259195Z"
    }
   },
   "outputs": [],
   "source": [
    "ConstLocations.get()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.287131Z",
     "start_time": "2020-03-27T09:17:35.278297Z"
    }
   },
   "outputs": [],
   "source": [
    "DistLocations.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every time you call `Config.get()` it will return new sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.304760Z",
     "start_time": "2020-03-27T09:17:35.294903Z"
    }
   },
   "outputs": [],
   "source": [
    "DistLocations.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize it we can represent latencies sample as pandas dataframe. It provides a readable table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.896505Z",
     "start_time": "2020-03-27T09:17:35.307036Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "\n",
    "lat_sample = pd.DataFrame(DistLocations.get()['latencies'])\n",
    "lat_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T16:34:34.176798Z",
     "start_time": "2020-03-16T16:34:34.165723Z"
    }
   },
   "source": [
    "*Notice that table has **NaN** values. They will not be automatically filled with symmetric value if not specified.*\n",
    "\n",
    "We can transform the table to fill them up symmetrically. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:35.940332Z",
     "start_time": "2020-03-27T09:17:35.904410Z"
    }
   },
   "outputs": [],
   "source": [
    "lat_sample[pd.isnull(lat_sample)] = lat_sample.T[pd.isnull(lat_sample)]\n",
    "lat_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Visualizing the data\n",
    "\n",
    "\n",
    " 1. Prepare data: \n",
    "  - Trasfer to pandas dataframe\n",
    "  - Fill null values with the symmetic counter-part\n",
    "  - Transform into list of tuples: 'from', 'to', 'value' \n",
    "  \n",
    " 2. Get 100 samples from the configuration with latencies\n",
    " \n",
    " 3. Visualize in a grid of histograms\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:41.570964Z",
     "start_time": "2020-03-27T09:17:35.952523Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set(style='darkgrid')\n",
    "\n",
    "# 1. Data prepare\n",
    "def prepare(x):\n",
    "    x = pd.DataFrame(x)\n",
    "    x[pd.isnull(x)] = x.T[pd.isnull(x)]\n",
    "    return x.stack().reset_index()\\\n",
    "                    .rename(columns={'level_0':'from', 'level_1':'to', 0: 'value'})\n",
    "                    \n",
    "# 2. Get 100 samples\n",
    "df = pd.concat(prepare(DistLocations.get()['latencies']) for _ in range(100))\n",
    "\n",
    "# 3. Visualize in a grid of histograms\n",
    "g = sns.FacetGrid(df, col='to', row='from', height=4, aspect=1)\n",
    "g = g.map(sns.distplot, 'value')\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visulations shows the latency matrix for our two locations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Peers and Topologies \n",
    "\n",
    "Usually when designing your own system, the architect should think about the network topology: *How are peers connected to each other?*. \n",
    "\n",
    "\n",
    "Simulation can work in two modes: \n",
    "\n",
    "1. **NetworkX**. You know what topology you want to simulate, pass a network topology to the simulation.\n",
    "\n",
    "2. **Emergent**. Peers connect to each other without a pre-defined topology. A certain topology emerges through local rules, for example, peers connect with each other through the bootstrap peers and exchange data with each other. Typically, unstructured topologies of a peer-to-peer network fall in this category. You can read more [here](https://en.wikipedia.org/wiki/Bootstrapping_node), or  read how [Bitcoin network](https://en.bitcoin.it/wiki/Network) is formed.  \n",
    "\n",
    "\n",
    "### Networkx topology\n",
    "\n",
    "Let's first see how we can form a network topology.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:43.066290Z",
     "start_time": "2020-03-27T09:17:41.577831Z"
    },
    "recordTiming": true
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "\n",
    "peer_num = 10\n",
    "\n",
    "# Generate a random network network topology \n",
    "G = nx.erdos_renyi_graph(10, 0.5)\n",
    "\n",
    "# Assign a peer type to the peers with a type. Type if later used to tell which services to run.\n",
    "nx.set_node_attributes(G, {k: 'basic' for k in G.nodes()}, 'type')\n",
    "\n",
    "sns.set(style='dark')\n",
    "nx.draw_networkx(G)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NetworkX is a library with rich semantics to work with graphs. You can read more [here](https://networkx.github.io/documentation/stable/reference/generators.html).  \n",
    "\n",
    "You can choose any graph, generate them and put `type` as node attributes. `type` is a name for the peer type. We will explain later on how it will be used in a simulation.  \n",
    "\n",
    "\n",
    "\n",
    "### Emergent topology\n",
    "\n",
    "What if topology is not known?\n",
    "\n",
    "In this case we can model the peer discovery mechanism through *bootstrapping* peers. The general bootstrap process is the following: \n",
    "1. Bootstrap nodes are first created and introduced to the network. The goal of bootstrap nodes is to maintain list of peers and connect them with each other upon request. Bootstrap nodes are usually known in advance, for example, hardcoded in the supplied code. \n",
    "2. The bootstrap procedure is as follows: a) new peer joins the network, b) peer sends a `Hello` message to one of bootstrap nodes, c) a bootstrap node responds to the peer with a random sample of known online peers. \n",
    "3. The peer connect with other peers from the given sample list.\n",
    "\n",
    "\n",
    "To run default bootstrap you should pass as a parameter `topology`, a dictionary with type `peer_id` -> {`type`: type_name}\n",
    "This is all abstracted in simulation, but you can specify your own bootstrap logic later. \n",
    "\n",
    "\n",
    "<!-- ### Explain more\n",
    "\n",
    "Discovery  -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------\n",
    "## Peer types and services\n",
    "\n",
    "Finally, our simulation needs messages and service to process these messages. In `p2psimpy` we model it through Manager Classes and `PeerType`.\n",
    "\n",
    "Below we show an example of a simple map with one type of the peer: `basic`: \n",
    "\n",
    "- Peer is described with a `PeerType(PeerConfig, services)`. \n",
    "    - `PeerConfig` describes a physical capacities of the node: bandwidth for the messages that go through (upload and download) and location (one the specified before)\n",
    "    - `services` is a list of service that peer should run. There some standard services that you can use in your simulation. But you will also implement your own later!\n",
    "- Minimally peer should have at least a connection service to connect to the network and respond to the introduction messages. These are the standard connection services available for you: \n",
    "  - `BaseConnectionManager`: a simple connection service that can connect, ping other peers and disconnect unresponsive peers. This is the recommended service for a **Networkx** topology approach.  \n",
    "  - `P2PConnectionManager`: an extended BaseConnectionManger that additionally keeps the number of connections between specified `min_peers` and `max_peers`. If the number of local connections is lower than `min_peers`, service will actievly pool and ask other peers for new connection. If the number of local connections is higher than `max_peers`, service will refuse all new connection and disconnect the slowest peers. This is a recommended service for an **Emergent** topology approach. \n",
    "\n",
    "Let's see how it looks in code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:43.208204Z",
     "start_time": "2020-03-27T09:17:43.085495Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.config import *\n",
    "from p2psimpy.consts import *\n",
    "    \n",
    "class PeerConfig(Config):\n",
    "    # Location of the peer - random location from the locations specified earlier \n",
    "    location = Dist('sample', DistLocations.locations)\n",
    "    # Bandwidth is normally distributed with average of 50 and var 10 Mbit\n",
    "    bandwidth_ul = Dist( 'norm', (50*MBit, 10*MBit))\n",
    "    bandwidth_dl = Dist( 'norm', (50*MBit, 10*MBit))\n",
    "\n",
    "# Let's add ConnectionManager - that will periodically ping neighbours and check if they are online.\n",
    "# We use BaseConnectionManager - that will periodically ping peer and disconnect unresponsive peers.\n",
    "from p2psimpy.services.connection_manager import BaseConnectionManager\n",
    "\n",
    "services = (BaseConnectionManager,)\n",
    "# We have on peer role: basic\n",
    "peer_types = {'basic': PeerType(PeerConfig, services)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to use these configurations? Configs are used to get value. \n",
    "They work as generators and attribute descriptors.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Putting it all together\n",
    "\n",
    "\n",
    "Let's combine all you learned so far to run a simple simulation:\n",
    "\n",
    "1. Define locations and simulation world parameters.\n",
    "2. Define topology and number of peers with their types.  \n",
    "3. Define what each peer type is with a `PeerConfig` and what it does with services. \n",
    "\n",
    "\n",
    "After we defined all this we can create a simulation object and run it with `.run(time)`.\n",
    "\n",
    "Simulation has it's internal clock and scheduler for all events. The parameter `time` is a time until which the simulation is run. \n",
    "\n",
    "\n",
    "***The time is defined in milliseconds**!\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-11T09:50:33.374122Z",
     "start_time": "2020-03-11T09:50:33.347834Z"
    }
   },
   "source": [
    "## Random given topology \n",
    "\n",
    "Let us first try out with a random given topology generated as a networkx graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:43.889018Z",
     "start_time": "2020-03-27T09:17:43.214432Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.config import *\n",
    "from p2psimpy.consts import *\n",
    "import networkx as nx\n",
    "\n",
    "class Locations(Config):\n",
    "    locations = ['LocA', 'LocB']\n",
    "    latencies = {\n",
    "        'LocB': {'LocB': Dist('gamma', (1, 1, 1))},\n",
    "        'LocA': {'LocB': Dist('norm', (12, 2)), 'LocA': Dist('norm', (2, 0.5))},\n",
    "    } \n",
    "\n",
    "# Number of nodes\n",
    "N = 10\n",
    "    \n",
    "# Generate network topology \n",
    "G = nx.erdos_renyi_graph(N, 0.5)\n",
    "# Assign a peer type to the peers \n",
    "nx.set_node_attributes(G, {k: 'basic' for k in G.nodes()}, 'type')\n",
    "\n",
    "class PeerConfig(Config):\n",
    "    location = Dist('sample', Locations.locations)\n",
    "    bandwidth_ul = Dist( 'norm', (50*MBit, 10*MBit))\n",
    "    bandwidth_dl = Dist( 'norm', (50*MBit, 10*MBit))\n",
    "\n",
    "# Let's add ConnectionManager - that will periodically ping neighbours and check if they are online \n",
    "from p2psimpy.services.connection_manager import BaseConnectionManager\n",
    "# For each service you can define own configuration, or use default values.   \n",
    "# Lets use base connection manager - that will periodically ping peer and disconnect unresponsive peers.\n",
    "\n",
    "services = (BaseConnectionManager,)\n",
    "peer_types = {'basic': PeerType(PeerConfig, services)}\n",
    "\n",
    "# Display the topology \n",
    "nx.draw_networkx(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:44.329539Z",
     "start_time": "2020-03-27T09:17:43.895337Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.simulation import BaseSimulation\n",
    "\n",
    "# Create BaseSimulation with enabled logger and save in the logs directory. \n",
    "sim = BaseSimulation(Locations, G, peer_types, enable_logger=True, logger_dir='logs')\n",
    "\n",
    "# Let's run the simulation for 5 seconds\n",
    "sim.run(5_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-17T12:40:15.632268Z",
     "start_time": "2020-03-17T12:40:15.613568Z"
    }
   },
   "source": [
    "----------------------------\n",
    "\n",
    "What's next? There is no output.\n",
    "\n",
    "We run the simulator with logger enabled to see the all the messages exchanged in the network. \n",
    "\n",
    "The output is written to the logs in `\\logs` directory. Let's see what is the output. \n",
    "Each peer writes its own log with events happening like received message, connecting to a peer etc. \n",
    "\n",
    "For example, here is the log of peer 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:44.413024Z",
     "start_time": "2020-03-27T09:17:44.345639Z"
    }
   },
   "outputs": [],
   "source": [
    "with open('logs/Peer_1:basic.log') as s:\n",
    "    print(s.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Emergent topology\n",
    "\n",
    "Now let's try an emergent topology, we will use the same configuration, but use bootstrap peer for discovery and building a network. For the connection manager we will use `P2PConnectionManager`.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:44.430653Z",
     "start_time": "2020-03-27T09:17:44.415134Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.services.connection_manager import P2PConnectionManager\n",
    "\n",
    "num_peers = 10\n",
    "\n",
    "topology_specs = {i:{'type': 'basic'} for i in range(1, num_peers+1)}\n",
    "\n",
    "class ConnectionConfig(Config):\n",
    "    min_peers = 4\n",
    "    max_peers = 8\n",
    "\n",
    "peer_types = {'basic': PeerType(PeerConfig, {P2PConnectionManager: ConnectionConfig})}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:44.729428Z",
     "start_time": "2020-03-27T09:17:44.442513Z"
    }
   },
   "outputs": [],
   "source": [
    "sim = BaseSimulation(Locations, topology_specs, peer_types, logger_dir='logs2')\n",
    "\n",
    "# Let's run the simulation for 5 seconds\n",
    "sim.run(5_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The peers form themselves a random topology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:46.596898Z",
     "start_time": "2020-03-27T09:17:44.765123Z"
    }
   },
   "outputs": [],
   "source": [
    "G1 = sim.get_graph()\n",
    "nx.draw_networkx(G1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T09:17:46.609528Z",
     "start_time": "2020-03-27T09:17:46.600756Z"
    }
   },
   "outputs": [],
   "source": [
    "with open('logs2/Peer_1:basic.log') as s:\n",
    "    print(s.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------\n",
    "\n",
    "Now you get how to work a simulation, in the next notebook we will take a look into services and implement our own.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your experiments here \n",
    "\n",
    "\n",
    "1. Change the latency between the locations and look at the logs.  \n",
    "2. Change the rules for the discovery\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "524px",
    "left": "441px",
    "top": "355px",
    "width": "256px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}