notebooks/8 - Data Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Visualization\n",
    "\n",
    "Jay Urbain, PhD\n",
    "\n",
    "6/9/2018, 9/16/2018, 9/18/2018\n",
    "\n",
    "Topics:  \n",
    "- Data Visualization  \n",
    "- Introduction to Matplotlib  \n",
    "- Line Plot  \n",
    "- Bar Chart  \n",
    "- Histogram Plot  \n",
    "- Box and Whisker Plot  \n",
    "- Scatter Plot  \n",
    "\n",
    "References:  \n",
    "https://matplotlib.org/users/index.html   \n",
    "Python Data Science Handbook, Jake VanderPlas, 2017.    \n",
    "The Visual Display of Quantitative Information, 2001.  \n",
    "Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, 2017.\n",
    "Applied Multivariate Statistical Analysis, 2015.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data visualization is an important skill in applied statistics, data science, and machine learning.\n",
    "\n",
    "Statistics focuses on quantitative descriptions and estimations of data. Data visualization provides tools for gaining a qualitative understanding of these descriptions.\n",
    "\n",
    "Visually exploring and getting to know a dataset and can help with identifying patterns, corrupt data, and outliers, and better understand relationships in the data. \n",
    "\n",
    "Data visualizations can be used to express and demonstrate key relationships in plots and charts that are more intuitive to data scientists and stakeholders than raw measures of association or significance.\n",
    "\n",
    "Data visualization and exploratory data analysis are large fields themselves and this tutorial will just provide an introduction.\n",
    "\n",
    "There are five key plots for basic data visualization:\n",
    "\n",
    "- Line Plot  \n",
    "- Bar Chart  \n",
    "- Histogram Plot  \n",
    "- Box and Whisker Plot  \n",
    "- Scatter Plot  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matplotlib\n",
    "\n",
    "There are several excellent plotting libraries in Python.\n",
    "\n",
    "For quick and exploratory plots intended for your initial EDA, start with the matplotlib library. It is the foundation for many other plotting libraries and plotting support in higher-level libraries such as Pandas.\n",
    "\n",
    "Recent Matplotlib versions make it relatively easy to set new global plotting styles (customizing Style Sheets), and people have been developing new packages that build on its powerful internals to drive Matplotlib via cleaner, more modern APIs. For example, Seaborn, ggpy, HoloViews, Altair, and even Pandas can be used as wrappers around Matplotlib's API. \n",
    "\n",
    "The matplotlib provides a context, one in which one or more plots can be drawn before the image is shown or saved to file. The context can be accessed via functions on pyplot. By convention, the context can be imported as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib as mpl \n",
    "import matplotlib.pyplot as plt # plt interface is what's usually needed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting Styles\n",
    "\n",
    "Use the ``plt.style`` directive to choose appropriate aesthetic styles for our figures. \n",
    "\n",
    "https://matplotlib.org/api/style_api.html   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This site provides a nice illustration of the different styles:  \n",
    "https://tonysyu.github.io/raw_content/matplotlib-style-gallery/gallery.html    \n",
    "\n",
    "Here we will set the ``classic`` style, which ensures that the plots we create use the classic Matplotlib style. Please feel free to experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use('classic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting from an IPython shell\n",
    "\n",
    "It can be convenient to use Matplotlib interactively within an IPython shell.\n",
    "IPython is built to work well with Matplotlib if you specify Matplotlib mode.\n",
    "To enable this mode, you can use the ``%matplotlib`` magic command after starting ``ipython``:\n",
    "\n",
    "```ipython\n",
    "In [1]: %matplotlib\n",
    "Using matplotlib backend: TkAgg\n",
    "\n",
    "In [2]: import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "At this point, any ``plt`` plot command will cause a figure window to open, and further commands can be run to update the plot.\n",
    "Some changes (such as modifying properties of lines that are already drawn) will not draw automatically: to force an update, use ``plt.draw()``.\n",
    "Using ``plt.show()`` in Matplotlib mode is not required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting from an Jupyter notebook\n",
    "\n",
    "The Jupyter notebook is a browser-based interactive data analysis tool that can combine narrative, code, graphics, HTML elements, and more into a single executable document.\n",
    "\n",
    "Plotting interactively within an notebook can be done with the ``%matplotlib`` magic command, and works in a similar way to the IPython shell.\n",
    "In the Jupyter notebook, you also have the option of embedding graphics directly in the notebook, with two possible options:\n",
    "\n",
    "- ``%matplotlib notebook`` will lead to *interactive* plots embedded within the notebook\n",
    "- ``%matplotlib inline`` will lead to *static* images of your plot embedded in the notebook\n",
    "\n",
    "Generally opt for ``%matplotlib inline``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running this command (it needs to be done only once per kernel/session), any cell within the notebook that creates a plot will embed a PNG image of the resulting graphic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.linspace(0, 10, 100)\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.plot(x, np.sin(x), '-', label=\"sin\")\n",
    "plt.plot(x, np.cos(x), '--', label=\"cos\");\n",
    "# fig.legend()\n",
    "plt.ylabel(\"magnitude\")\n",
    "plt.xlabel(\"sin and cosine\")\n",
    "plt.title(\"waves\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that when saving your figure, it's not necessary to use ``plt.show()`` or related commands discussed earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving Figures to File\n",
    "\n",
    "One nice feature of Matplotlib is the ability to save figures in a wide variety of formats.\n",
    "Saving a figure can be done using the ``savefig()`` command.\n",
    "For example, to save the previous figure as a PNG file, you can run this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig.savefig('my_figure.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a file called ``my_figure.png`` in the current working directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lh my_figure.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To confirm that it contains what we think it contains, let's use the IPython ``Image`` object to display the contents of this file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image('my_figure.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In ``savefig()``, the file format is inferred from the extension of the given filename.\n",
    "Depending on what backends you have installed, many different file formats are available.\n",
    "The list of supported file types can be found for your system by using the following method of the figure canvas object:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig.canvas.get_supported_filetypes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Interfaces \n",
    "A confusing feature of Matplotlib is its dual interfaces:   \n",
    "- a convenient MATLAB-style state-based interface, and   \n",
    "- a more powerful object-oriented interface. \n",
    "    \n",
    "**MATLAB-style Interface**\n",
    "\n",
    "Matplotlib was originally written as a Python alternative for MATLAB users, and much of its syntax reflects that fact.\n",
    "The MATLAB-style tools are contained in the pyplot (``plt``) interface.\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()  # create a plot figure\n",
    "\n",
    "# create the first of two panels and set current axis\n",
    "plt.subplot(2, 1, 1) # (rows, columns, panel number)\n",
    "plt.plot(x, np.sin(x))\n",
    "\n",
    "# create the second panel and set current axis\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.plot(x, np.cos(x));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This interface is *stateful*: it keeps track of the \"current\" figure and axes, which are where all ``plt`` commands are applied.\n",
    "\n",
    "You can get a reference to these using the ``plt.gcf()`` (get current figure) and ``plt.gca()`` (get current axes) routines.\n",
    "\n",
    "While this stateful interface is fast and convenient for simple plots, it is easy to run into problems.\n",
    "For example, once the second panel is created, how can we go back and add something to the first?\n",
    "This is possible within the MATLAB-style interface, but there is a better way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Object-oriented interface\n",
    "\n",
    "The object-oriented interface is available for these more complicated situations, and for when you want more control over your figure.\n",
    "Rather than depending on some notion of an \"active\" figure or axes, in the object-oriented interface the plotting functions are *methods* of explicit ``Figure`` and ``Axes`` objects.\n",
    "To re-create the previous plot using this style of plotting, you might do the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Charts and plots are made by making and calling the context:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "pyplot.plot(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drawings on the context can be shown in a new window by calling the `show()` function:\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# display the plot\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, the drawings on the context can be saved to file using the `savefig()` function."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "pyplot.savefig('my_image.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Line Plot\n",
    "\n",
    "A line plot is generally used to present observations collected at regular intervals.\n",
    "\n",
    "The `x-axis` represents the regular interval, such as time. The `y-axis` shows the observations, ordered by the x-axis and connected by a line.\n",
    "\n",
    "A line plot can be created by calling the plot() function and passing the `x-axis` data for the regular interval, and `y-axis` for the observations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ipython magic command to plot inline\n",
    "%matplotlib inline \n",
    "\n",
    "from numpy import sin\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# generate consistent interval for x-axis\n",
    "x = [x*0.1 for x in range(100)]\n",
    "\n",
    "# apply function of x to generate y-axis data\n",
    "y = sin(x)\n",
    "\n",
    "# create line plot\n",
    "plt.plot(x, y)\n",
    "\n",
    "# show line plot\n",
    "#plt.show() # not needed for notebook\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bar Chart\n",
    "\n",
    "A bar chart is generally used to present relative quantities for multiple categories.\n",
    "\n",
    "The `x-axis` represents the categories and are spaced evenly. The y-axis represents the quantity for each category and is drawn as a bar from the baseline to the appropriate level on the y-axis.\n",
    "\n",
    "A bar chart can be created by calling the `bar()` function and passing the category names for the x-axis and the quantities for the y-axis."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# create bar chart\n",
    "plt.bar(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example of a bar chart\n",
    "from random import seed\n",
    "from random import randint\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# seed the random number generator\n",
    "seed(42)\n",
    "\n",
    "# names for categories\n",
    "x = ['apples', 'bananas', 'grapes']\n",
    "\n",
    "# quantities for each category\n",
    "y = [randint(0, 100), randint(0, 100), randint(0, 100)]\n",
    "\n",
    "# create bar chart\n",
    "plt.bar(x, y)\n",
    "\n",
    "# show line plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Histogram Plot\n",
    "\n",
    "A histogram plot is used to summarize the distribution of a data sample.\n",
    "\n",
    "The `x-axis` represents discrete bins or intervals for the observations. \n",
    "For example, observations with values between 1 and 10 may be split into five bins, the values $[1,2]$ would be allocated to the first bin, $[3,4]$ would be allocated to the second bin, and so on.\n",
    "\n",
    "The `y-axis` represents the frequency or count of the number of observations in the dataset that belong to each bin.\n",
    "\n",
    "Essentially, a data sample is transformed into a bar chart where each category on the `x-axis` represents an interval of observation values.\n",
    "\n",
    "A histogram plot can be created by calling the hist() function and passing in a list or array that represents the data sample."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# create histogram plot\n",
    "plt.hist(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example below creates a dataset of 1,000 random numbers drawn from a standard Gaussian distribution, then plots the dataset as a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example of a histogram plot\n",
    "from numpy.random import seed\n",
    "from numpy.random import randn\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# seed the random number generator\n",
    "seed(42)\n",
    "\n",
    "# random numbers drawn from a Gaussian distribution\n",
    "x = randn(10000)\n",
    "\n",
    "# create histogram plot\n",
    "plt.hist(x, bins=50)\n",
    "\n",
    "# show line plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# controlling the number of bins in a historgram\n",
    "# create histogram plot\n",
    "plt.hist(x, bins=100)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Box (and Whisker) Plot\n",
    "\n",
    "A box and whisker plot, or boxplot for short, is used to summarize the distribution of a data sample.\n",
    "\n",
    "The `x-axis` is used to represent the data sample, where multiple boxplots can be drawn side by side on the x-axis if desired.\n",
    "\n",
    "The `y-axis` represents the observation values. A box is drawn to summarize the middle $50%$ of the dataset starting at the observation at the $25th$ percentile and ending at the $75th$ percentile. The median, or `50th percentile`, is drawn with a line. A value called the *interquartile range*, or IQR, is calculated as *1.5 * the difference between the 75th and 25th percentiles. Lines called whiskers are drawn extending from both ends of the box with the length of the IQR to demonstrate the expected range of sensible values in the distribution. Observations outside the whiskers might be outliers and are drawn with small circles.\n",
    "\n",
    "Boxplots can be drawn by calling the boxplot() function passing in the data sample as an array or list."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# create box and whisker plot\n",
    "plt.boxplot(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplots are useful to summarize the distribution of a data sample as an alternative to the histogram. \n",
    "\n",
    "They can help to quickly get an idea of the range of common and sensible values in the box and in the whisker respectively. Because we are not looking at the shape of the distribution explicitly, this method is often used when the data has an unknown or unusual distribution, such as data  that is non-Gaussian.\n",
    "\n",
    "The example below creates three boxplots in one chart, each summarizing a data sample drawn from a slightly different Gaussian distribution. Each data sample is created as an array and all three data sample arrays are added to a list that is padded to the plotting function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example of a box and whisker plot\n",
    "from numpy.random import seed\n",
    "from numpy.random import randn\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# seed the random number generator\n",
    "seed(42)\n",
    "\n",
    "# random numbers drawn from a Gaussian distribution\n",
    "x = [5*randn(1000), 7.5 * randn(1000), 10 * randn(1000)]\n",
    "\n",
    "# create box and whisker plot\n",
    "plt.boxplot(x)\n",
    "\n",
    "# show box plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scatter Plot\n",
    "\n",
    "A scatter plot is used to summarize the relationship between two paired data samples.\n",
    "\n",
    "Paired data samples means that two measures were recorded for a given observation, such as the weight and height of a person.\n",
    "\n",
    "The `x-axis` represents observation values for the first sample, and the `y-axis` represents the observation values for the second sample. Each point on the plot represents a single observation.\n",
    "\n",
    "Scatterplots are basically bivariate or trivariate plots of variables against each other. They help us understand relationships among the variables of a dataset. A downward-sloping scatter indicates that as we increase the variable on the horizontal axis, the variable on the vertical axis decreases.\n",
    "\n",
    "Scatter plots can be created by calling the scatter() function and passing the two data sample arrays."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# create scatter plot\n",
    "pyplot.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example of a scatter plot\n",
    "from numpy.random import seed\n",
    "from numpy.random import randn\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# seed the random number generator\n",
    "seed(42)\n",
    "\n",
    "# first variable\n",
    "x = 20 * randn(1000) + 100\n",
    "\n",
    "# second variable\n",
    "y = x + (10 * randn(1000) + 50)\n",
    "\n",
    "# create scatter plot\n",
    "plt.scatter(x, y)\n",
    "\n",
    "# show line plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding a regression line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.polynomial.polynomial import polyfit\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Sample data\n",
    "# x = np.arange(10)\n",
    "# y = 5 * x + 10\n",
    "\n",
    "# Fit with polyfit\n",
    "b, m = polyfit(x, y, 1)\n",
    "\n",
    "plt.plot(x, y, '.')\n",
    "plt.plot(x, b + m * x, '-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "polyfit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pandas\n",
    "\n",
    "Univariate historgrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Univariate Histograms\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas\n",
    "\n",
    "names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']\n",
    "data = pandas.read_csv('data/pima-indians-diabetes.data.csv', names=names)\n",
    "ax = data.hist(bins=10, grid=False, figsize=(12,8), layout=(3,3), sharex=True, rwidth=0.9)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow]",
   "language": "python",
   "name": "conda-env-tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}