Intro to Machine Learning for Software Engineers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BTiXKbFZfv1y"
   },
   "source": [
    "# Introduction to Machine Learning for Software Engineers\n",
    "\n",
    "In this notebook we will get acquainted with [machine learning](https://en.wikipedia.org/wiki/Machine_learning) in relation to the practice of [software engineering](https://en.wikipedia.org/wiki/Software_engineer). For the sake of simplicity, all code examples in this notebook will be written in Python. We will be using TensorFlow (2.3) for practical machine learning demonstrations.\n",
    "\n",
    "This notebook is written in collaboration with [Yassine Yousfi](https://yassineyousfi.github.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction to [machine learning](https://en.wikipedia.org/wiki/Machine_learning)\n",
    "\n",
    "> Machine learning is the study of computer [algorithms](https://en.wikipedia.org/wiki/Algorithm) that improve automatically through experience. It is seen as a part of [artificial intelligence](https://en.wikipedia.org/wiki/Artificial_intelligence). Machine learning algorithms build a model based on sample data, known as '[training data](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#training_set)', in order to make predictions or decisions without being explicitly programmed to do so. Machine learning algorithms are used in a wide variety of applications, such as [email filtering](https://en.wikipedia.org/wiki/Email_filtering) and [computer vision](https://en.wikipedia.org/wiki/Computer_vision), where it is difficult or unfeasible to develop conventional algorithms to perform the needed tasks.\n",
    "\n",
    "What is the goal of machine learning? Typically we are looking to train an algorithm so that it produces useful output from a given input. Take traditional programming for instance. The following input: `4`, passed to the following Python algorithm:\n",
    "\n",
    "```py\n",
    "def cube(x):\n",
    "    return x**3\n",
    "```\n",
    "Produces the following useful output: `64`.\n",
    "\n",
    "Likewise, one machine learning goal might be to take an input equal to an image of a horse, pass it to a function equal to a machine learning algorithm, and get an output equal to the same horse, but transformed into a zebra.\n",
    "\n",
    "Another goal might be to take an input equal to historical electricity consumption, pass it to a function equal to a machine learning algorithm, and get an output equal to the future year's electricity consumption.\n",
    "\n",
    "Yet another goal might be to take an input equal to a physical structure's attributes (say, data about a given house: size, location, damage, etc.), pass it to a function equal to a machine learning algorithm, and get an output equal to the value of the structure.\n",
    "\n",
    "Finally, another goal might be to take an input equal to a set of geo-locations representing residences in a city, pass this set to a function which is equal to a machine learning algorithm, and get an output equal to a the bounded geo-locations grouped according to proximity.\n",
    "\n",
    "What do all these goals have in common? \n",
    "\n",
    "For one, they can all actually be represented by the same formula: `f(x) = y`. Where `x` and `y` are input and output data respectively. \n",
    "\n",
    "So what about the function `f`?\n",
    "\n",
    "Consider again our traditional programming example, what would represent `f` in that case? Yes, the `cube` algorithm! In the same way, in machine learning, we are searching for the _algorithm `f`_. Of course, in traditional software development, we can simply design an algorithm ourselves. But how does this process work programmatically in machine learning?\n",
    "\n",
    "In the next section, we will consider two [types of machine learning algorithms](https://en.wikipedia.org/wiki/Machine_learning#Types_of_learning_algorithms) which are designed to accomplish just that; [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning), and [unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning). Afterward, we can look at some practical examples of _supervised learning_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Machine learning algorithms\n",
    "\n",
    "> The types of machine learning algorithms differ in their approach, the type of data they input and output, and the type of task or problem that they are intended to solve.\n",
    "\n",
    "Their are plenty of algorithm types to discover. Some of the common ones include the above mentioned supervised and unsupervised learning algorithms, as well as [reinforcement learning algorithms](https://en.wikipedia.org/wiki/Reinforcement_learning). As brought out in the linked material, one type of algorithm can differ drastically from another type.\n",
    "\n",
    "We cannot cover _all_ the different types of algorithms in this notebook, unfortunately. However, you are strongly encouraged to read more about them by visiting the links above. As previously mentioned, we will only cover _two_ types of machine learning algorithms in this notebook, only _one_ of which will be demonstrated in detail.\n",
    "\n",
    "With that, let's consider supervised and unsupervised learning in the cells below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Supervised learning\n",
    "\n",
    "> Supervised learning is the [machine learning](https://en.wikipedia.org/wiki/Machine_learning) task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from labeled [training data](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#training_set) consisting of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal). A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. An optimal scenario will allow for the algorithm to correctly determine the class labels for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a \"reasonable\" way (see [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias)).\n",
    "\n",
    "In supervised learning, we have a set of 'correct answers' (often called `labels`) corresponding to true outputs: `y`. \n",
    "\n",
    "Of course, these `labels` do not necessarily correspond to all the possible outputs ever. Rather, they correspond to a decent set of outputs that allow an algorithm to generalize to a problem domain.\n",
    "\n",
    "Consider our previous formula: `f(x) = y`. If `y` is known during the training process, then we are dealing with a supervised learning problem.\n",
    "\n",
    "The algorithm, `f`, is learned by training on input-output data pairs. Those inputs (often called `features`), therefore, must correspond to `x`, representing the input values.\n",
    "\n",
    "So what do we mean by _**training**_ anyway? We can again simplify the idea by considering traditional programming. Say you were tasked with building an algorithm that would calculate the _mean_ of a list:\n",
    "\n",
    "```py\n",
    "def calculate_mean(data):\n",
    "    \"\"\"\n",
    "    Function to calculate the mean of the data set.\n",
    "\n",
    "    Args: \n",
    "        data: list. A list of int.\n",
    "\n",
    "    Returns: \n",
    "        mean: float. Mean of data.\n",
    "    \"\"\"\n",
    "    pass\n",
    "```\n",
    "\n",
    "How would you go about building such an algorithm? Ideally, if you already knew how to compute the mean of a list, you would simply implement the solution immediately, possibly as so:\n",
    "\n",
    "```py\n",
    "def calculate_mean(data):\n",
    "    \"\"\"\n",
    "    Function to calculate the mean of the data set.\n",
    "\n",
    "    Args: \n",
    "        data: list. A list of int.\n",
    "\n",
    "    Returns: \n",
    "        mean: float. Mean of data.\n",
    "    \"\"\"\n",
    "    return sum(data) / len(data)\n",
    "```\n",
    "\n",
    "But say you did not know how to compute the mean of a list. Then what would you do? And what if you were in a void, with no access to outside material? You'd be in a tough situation!\n",
    "\n",
    "But say someone gave you one-thousand input-output pairs, each demonstrating an input represented by a list of integers, and an output represented by the true mean of that list. Now what would you do? Indeed, trial and error. Possibly, you could start with a simple algorithm, your best guess, and test the product of your algorithm against your set of true means. When your outputs are far off from the truth, you make large changes to your algorithm, and when your outputs are close to the truth, you make smaller changes. With each iteration drawing you closer and closer to the true solution.\n",
    "\n",
    "In the same way, machine learning algorithms are learned by iterating over many input-output pairs with the goal of training the algorithm to output values equal to the true outputs.\n",
    "\n",
    "Does this process different in unsupervised learning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Unsupervised learning\n",
    "\n",
    "> Unsupervised learning (UL) is a type of algorithm that learns patterns from untagged data. The hope is that through mimicry, the machine is forced to build a compact internal representation of its world. In contrast to Supervised Learning (SL) where data is tagged by a human, eg. as \"car\" or \"fish\" etc, UL exhibits self-organization that captures patterns as neuronal predilections or probability densities.\n",
    "\n",
    "We can observe that, while sharing certain attributes, supervised and unsupervised learning are not equal. For one, unsupervised learning has _no true outputs_.\n",
    "\n",
    "When we consider our formula: `f(x) = y`. It is evident that unsupervised learning problems dictate that, while `y` exists, it cannot be derived from input-output pairs. In other words, while we have `features`, we _do not_ have `labels`.\n",
    "\n",
    "Therefore, the way in which the algorithm `f` is learned is no longer by training on `x` and `y` pairs as in supervised learning. Rather, we are now attempting to learn an optimal _pattern_ directly from the input `x` itself.\n",
    "\n",
    "This is akin to our geo-location problem above, in which no _true_ boundaries ever actually exist. Rather, we can only infer that an _optimal pattern may exist_ and we must rely on the training process to discover such a pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll find plenty of practical applications of both supervised and unsupervised learning. However, it seems that, of the two, supervised learning relates more so to the popular applications of machine learning in the workplace than unsupervised learning, at least for now. Further, other types of machine learning algorithms, such as reinforcement learning, are too complex to tackle without first having a solid foundation in supervised learning.\n",
    "\n",
    "Therefore, as this notebook serves only as an introduction to machine learning, we will concentrate primarily on _supervised machine learning_ since it is the most relevant of the two, and possibly the most relevant in general. You are encouraged to read more about unsupervised learning in your own time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Machine learning models\n",
    "\n",
    "In short, the goal of machine learning is to _fit an algorithm_ (that is, the process of causing an algorithm to be learned) to data. Once the algorithm is fit, we can [derive a _statistical model_ from the rules, numbers, and any other algorithm-specific data structures required to make _predictions_](https://machinelearningmastery.com/difference-between-algorithm-and-model-in-machine-learning/).\n",
    "\n",
    "With that model, we can then generate new outputs (_predictions_) for seen and unseen input.\n",
    "\n",
    "As we are already familiar with types of algorithms, lets take this concept one step further and consider [_types of models_](https://en.wikipedia.org/wiki/Machine_learning#Models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Types of machine learning models\n",
    "\n",
    "> Performing machine learning involves creating a [model](https://en.wikipedia.org/wiki/Statistical_model), which is trained on some training data and then can process additional data to make predictions. Various types of models have been used and researched for machine learning systems.\n",
    "\n",
    "As with [machine learning algorithms](#2.-Types-of-machine-learning-algorithms), machine learning _models_ also vary in  their approach, the type of data they input and output, and the type of task or problem that they are intended to solve. Even within the context of the _same type of algorithm_, models can really differ from one to another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models include [decision trees](https://en.wikipedia.org/wiki/Decision_tree_learning), [support vector machines](https://en.wikipedia.org/wiki/Support-vector_machine), [regression models](https://en.wikipedia.org/wiki/Regression_analysis), and more.\n",
    "\n",
    "In this notebook we will focus primarily on just one type of model, possibly the most popular; the [_artificial neural network_](https://en.wikipedia.org/wiki/Artificial_neural_network)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1.1 Artificial neural networks\n",
    "\n",
    "> Artificial neural networks (ANNs), usually simply called neural networks (NNs), are computing systems vaguely inspired by the [biological neural networks](https://en.wikipedia.org/wiki/Neural_circuit) that constitute animal brains.\n",
    "\n",
    "> An ANN is based on a collection of connected units or nodes called [artificial neurons](https://en.wikipedia.org/wiki/Artificial_neuron), which loosely model the [neurons](https://en.wikipedia.org/wiki/Neuron) in a biological brain. Each connection, like the [synapses](https://en.wikipedia.org/wiki/Synapse) in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it. The \"signal\" at a connection is a [real number](https://en.wikipedia.org/wiki/Real_number), and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a [weight](https://en.wikipedia.org/wiki/Weighting) that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times.\n",
    "\n",
    "If you are not already familiar with the _concept of artificial neural networks_, [**see this tutorial**](http://jalammar.github.io/visual-interactive-guide-basics-neural-networks/) for a comprehensive, visual introduction.\n",
    "\n",
    "Now that we have learned the basic ideas behind machine learning, we can move onto the topic of practical machine learning with Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Machine learning with Python\n",
    "\n",
    "[Python is currently the most popular programming language for machine learning engineers](https://www.eduplusnow.com/blog/the-top-10-machine-learning-languages-to-know-in-2019). Hosting libraries such as [NumPy](https://numpy.org/), [SciPy](https://www.scipy.org/), [Matplotlib](https://matplotlib.org/), [Pandas](https://pandas.pydata.org/), [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), [Scikit-learn](https://scikit-learn.org/), and more; it is hard to deny that Python is one of the most mature languages in the machine learning space at this time.\n",
    "\n",
    "You are likely already familiar with at least some of these tools.\n",
    "\n",
    "Some of them drastically simplify the process of designing and fitting [machine learning models and algorithms](https://en.wikipedia.org/wiki/Machine_learning#Models). Other libraries, such as NumPy, add invaluable functionality to Python by means of modules, classes, and functions that make it easier to work with data.\n",
    "\n",
    "In this notebook, we will be using some of these very libraries to start building machine learning models today. \n",
    "\n",
    "To begin, let's get acquainted with machine learning in Python using [TensorFlow](https://www.tensorflow.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 TensorFlow\n",
    "\n",
    "> TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.\n",
    "\n",
    "This section is designed to introduce you to basic machine learning with TensorFlow. It is adapted from [these tutorials](https://www.tensorflow.org/tutorials/https://www.tensorflow.org/tutorials/https://www.tensorflow.org/tutorials/). If you would like to work more with TensorFlow, you are strongly encouraged to go through more of their tutorials in your own time.\n",
    "\n",
    "Since we will now consider practical code examples, you should ensure that you have properly [installed TensorFlow](https://www.tensorflow.org/install), [NumPy](https://numpy.org/install/), and [Matplotlib](https://matplotlib.org/users/installing.html#installing-an-official-release) in your workspace.\n",
    "\n",
    "You may also optionally install [`pydot`](https://pypi.org/project/pydot/), [`pydotplus`](https://pypi.org/project/pydotplus/), and [Graphviz](https://graphviz.gitlab.io/download/). Some cells will not work correctly if you do not install these tools, however, you will still be able to complete this notebook without them.\n",
    "\n",
    "Once you've installed the required libraries you should be able to run the following cell without issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SqRnfJAFauo2"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our environment prepared, we can start our journey by exploring the namesake of TensorFlow; the `tensor` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HFzD2Cwvm86e"
   },
   "source": [
    "#### 4.1.1 [Tensors](https://en.wikipedia.org/wiki/Tensor)\n",
    "\n",
    "In TensorFlow, data isn’t stored as integers, floats, or strings. Rather, these values are encapsulated in a `tensor` object.\n",
    "\n",
    "Consider the tensors below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 5967,
     "status": "ok",
     "timestamp": 1600474912825,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "BhBvcoH3a0So",
    "outputId": "e789825b-558e-4b5b-a22b-b085426bb91b"
   },
   "outputs": [],
   "source": [
    "i = tf.constant(1)\n",
    "print(i)\n",
    "\n",
    "x = tf.constant([1, 2, 3])\n",
    "print(x)\n",
    "\n",
    "y = x**2\n",
    "print(y)\n",
    "\n",
    "x = x + y\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we observe that tensors behave much like we would expect them to. For example, we can perform vector operations on vector like tensors as we would with NumPy arrays. Further we observe that tensors have underlying shapes and data types (`dtype`).\n",
    "\n",
    "In Python, it is normal to cast one type to another, for instance we could cast an `int` to a `float` as so: `float(10)`. What about a TensorFlow `tensor`?\n",
    "\n",
    "Consider the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 4778,
     "status": "ok",
     "timestamp": 1600474912827,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "ShvBze17bDij",
    "outputId": "3794b0f2-685a-4f4c-dbf8-e74a33e44d83"
   },
   "outputs": [],
   "source": [
    "x = tf.cast(x, dtype=tf.float32)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While syntactically different from the type casting we might be familiar with, we can observe above that it is arbitrary to cast a `tensor` of one `dtype` to a tensor of another `dtype`.\n",
    "\n",
    "Say though, we don't want to just derive a `tensor` of a different `dtype`, say we want to get a different type of object all together, like a list or a `numpy` `int32`. This can also be accomplished arbitrarily.\n",
    "\n",
    "Consider the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 4153,
     "status": "ok",
     "timestamp": 1600474912828,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "NQP2MYG-bEh0",
    "outputId": "ae6680f1-b57f-49d2-b5d5-7e84108c2981"
   },
   "outputs": [],
   "source": [
    "print(x.numpy().tolist())\n",
    "print(i.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, like NumPy arrays, tensors can also be generated and reshaped as demonstrated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "executionInfo": {
     "elapsed": 3423,
     "status": "ok",
     "timestamp": 1600474912829,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "XjsClPAT3hiV",
    "outputId": "5f1dd33d-b91a-4b83-bb9e-d320654dd2ea"
   },
   "outputs": [],
   "source": [
    "j = tf.ones((10,10))\n",
    "k = tf.zeros((10,10))\n",
    "j, k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "executionInfo": {
     "elapsed": 2450,
     "status": "ok",
     "timestamp": 1600474912829,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "UAdf6gyU3uzX",
    "outputId": "068e1acf-cb5e-4980-c19b-a13290a07264"
   },
   "outputs": [],
   "source": [
    "print(tf.reshape(j ,shape=(1,-1)))\n",
    "print(k[0::2,0::2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you've observed, tensors can be zero or more dimensions, and behave as expected according to their `dtype` and `shape`.\n",
    "\n",
    "One idea to consider is, if a `tensor` can already be described by other data types, (such as a NumPy `array`) _why do we need tensors in the first place?_. [This resource may help](https://www.tensorflow.org/guide/tensor). In summary, it is for convenience with TensorFlow.\n",
    "\n",
    "[`tf.constant`](https://www.tensorflow.org/api_docs/python/tf/constant), [`tf.cast`](https://www.tensorflow.org/api_docs/python/tf/cast), [`tf.ones`](https://www.tensorflow.org/api_docs/python/tf/ones), [`tf.zeros`](https://www.tensorflow.org/api_docs/python/tf/zeros), and [`tf.reshape`](https://www.tensorflow.org/api_docs/python/tf/reshape) are but a few of the many functions offered by TensorFlow.\n",
    "\n",
    "In the next sections you will see _quite a few more_. If one is not explained in enough detail, you can do your own research in the [TensorFlow documentation here](https://www.tensorflow.org/api_docs/python/tf) as you follow along in this notebook.\n",
    "\n",
    "Excellent, now that we are familiar with the `tensor` type and a few TensorFlow functions, we can dive into beginner level machine learning with TensorFlow; creating and training [Neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SN7G7bovm618"
   },
   "source": [
    "#### 4.1.2 Neural networks with TensorFlow\n",
    "\n",
    "With a simple dataset, TensorFlow makes neural network modeling fairly straight forward. We can observe some simple steps in the cells below.\n",
    "\n",
    "We will begin by importing, [loading](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist/load_data) and normalizing the [MNIST dataset](https://www.tensorflow.org/datasets/catalog/mnist), MNIST is available through `tf.keras.datasets`, and we can normalize by simply dividing the features (corresponding to pixel values) by 255. [We want all the pixels to be in a range of zero and one](https://en.wikipedia.org/wiki/Feature_scaling), unscaled features make fitting an algorithm more difficult.\n",
    "\n",
    "If you are not familiar with the MNIST dataset, please see the relevant link above.\n",
    "\n",
    "This is a [classification problem](https://en.wikipedia.org/wiki/Statistical_classification), which we are tackling using a supervised learning algorithm which will produce a artificial neural network model. Our goal in this section is to train our artificial neural network, so that the derived model is able to correctly classify MNIST images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "executionInfo": {
     "elapsed": 1169,
     "status": "ok",
     "timestamp": 1600444174795,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "jZXgxGh-baa9",
    "outputId": "70beb016-bf54-4d61-b6cb-2234aaac1d23"
   },
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This MNIST dataset features 70,000 examples of hand written digits. Each 28 pixels by 28 pixels. The digits have been size-normalized and centered. MNIST is generally a good starting point for an introduction to machine learning.\n",
    "\n",
    "We have effectively split this data into two sets, a 'train set' consisting of 60,000 images, and a 'test set' consisting of 10,000 images.\n",
    "\n",
    "Below, we should find the shape of our test set, which will be 10,000 (images), by 28 (height in pixels), by 28 (width in pixels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 353,
     "status": "ok",
     "timestamp": 1600444293895,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "xoXgTMHa6bwc",
    "outputId": "f5dd722d-86a5-4192-b7de-72c5a3b3fec9"
   },
   "outputs": [],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If that was successful, we can then view the data of the first `test` image, `x_test[0]`, and observe that the pixel values are indeed between zero and one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x_test[0], x_test[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print out a reconstructed visualization of the first image using [`plt.imshow`](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imshow.html) along with it's associated label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(x_test[0], cmap=\"gray\")\n",
    "print(y_test[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we have proved that our features, which correspond to the pixel values of our MNIST images, are indeed scaled to values between zero and one, and that when converted to an image, portray the reality of what our data is representing; hand written digits.\n",
    "\n",
    "Further we see that our labels (`y`, remember `f(x) = y`) correspond to the expected output of our input data.\n",
    "\n",
    "We see too that we have a few sets of data now, derived from MNIST; `x_train`, `y_train`, `x_test`, and `y_test`. These represent the input-output (`x`-`y`) pairs as previously described in our section on [supervised learning](#2.1-Supervised-learning)\n",
    "\n",
    "Now we can move onto constructing TensorFlow models. TensorFlow allows construction of models both by means of a [`Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) API as well as by means of a [`Functional`](https://www.tensorflow.org/guide/keras/functional) API. We can begin by constructing a model sequentially, and then do the same functionally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 4.1.2.1 Building sequential models with TensorFlow\n",
    "\n",
    "To define our model, we will use the [`Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) API. The building blocks are available in [`tf.keras.layers`](https://www.tensorflow.org/api_docs/python/tf/keras/layers).\n",
    "\n",
    "Namely, we will only be using the [`Dense` layer](https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/keras/layers/Dense) preceded by a [`Flatten` step](https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/keras/layers/Flatten).\n",
    "\n",
    "Our output is `10`, corresponding to the number of possible output classes, corresponding to the hand-written digits, zero through nine.\n",
    "\n",
    "The steps taken here are quite straight forward.\n",
    "\n",
    "We flatten our image, resulting in a vector of length `784` (`28 * 28`). Then we create subsequent layers in our neural network, the first containing `256` units, the next containing `128` units, and the final containing our above mentioned `10` output units.\n",
    "\n",
    "Each unit is of course a neuron as described previously in [[](#3.1.1 Artificial neural networks](#3.1.1-Artificial-neural-networks).\n",
    "\n",
    "One idea that may be new is the [`activation` function](https://en.wikipedia.org/wiki/Activation_function). Below we tell TensorFlow to use the [rectified linear unit activation function (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) and [softmax activation function](https://en.wikipedia.org/wiki/Softmax_function). An activation function is just a mathematical function that operates on the output of our units. Activation functions are critical in machine learning. You should definitely read more about activation functions if you want to dive deeper into machine learning. For this lesson though, it is enough to just have a general understanding of what they do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4dVdi-Dfffxr"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "                                    tf.keras.layers.Dense(256, activation='relu'),\n",
    "                                    tf.keras.layers.Dense(128, activation='relu'),\n",
    "                                    tf.keras.layers.Dense(10, activation='softmax')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our model, we can pass values into it as so `f(x)` to generate an output `y`.\n",
    "\n",
    "Of course, this model is not trained at all, so we can expect meaningless output. Our input below is the same test image that we sampled above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "executionInfo": {
     "elapsed": 352,
     "status": "ok",
     "timestamp": 1600444482446,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "fa-P-pcqfijg",
    "outputId": "ff21dd98-2f70-4c77-b9e0-da9e6057ba24"
   },
   "outputs": [],
   "source": [
    "predictions = model(x_test[:1])\n",
    "print(predictions)\n",
    "print(predictions.numpy().argmax())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can print a summary of our model using [tf.keras.utils.plot_model](https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/keras/utils/plot_model) and [model.summary](https://www.tensorflow.org/api_docs/python/tf/keras/Model#summary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 533
    },
    "executionInfo": {
     "elapsed": 623,
     "status": "ok",
     "timestamp": 1600444541638,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "x4fT4GN3G8ku",
    "outputId": "2f062b33-e5fa-4430-8048-4e1cdc352396"
   },
   "outputs": [],
   "source": [
    "tf.keras.utils.plot_model(model, show_shapes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "executionInfo": {
     "elapsed": 804,
     "status": "ok",
     "timestamp": 1600444685787,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "-wDGItFY78eF",
    "outputId": "c6167cc5-cdc8-473a-905e-1abd1e0d4ee6"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l7vEsdstmOLn"
   },
   "source": [
    "##### 4.1.2.2 Training sequential models with TensorFlow\n",
    "\n",
    "Obviously, we're going to want to _train_ our model. Otherwise it will not produce _useful output_. Before our model is ready for training, however, we need to set a few [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)). These are added during the model's [`compile`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile) step:\n",
    "\n",
    "We are going to add a:\n",
    "\n",
    "- [Loss function](https://en.wikipedia.org/wiki/Loss_function): [SparseCategoricalCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy).\n",
    "- [Optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers): [Adam](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam).\n",
    "- [Metric](https://www.tensorflow.org/api_docs/python/tf/keras/metrics): [Accuracy](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Accuracy).\n",
    "\n",
    "We can observe how this in the cell below. Also note, that in our new model, we are no longer using the `softmax` activation on the output since this is built into cross entropy loss functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MMz_VDhEfvBz"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "                                    tf.keras.layers.Dense(256, activation='relu'),\n",
    "                                    tf.keras.layers.Dense(128, activation='relu'),\n",
    "                                    tf.keras.layers.Dense(10)])\n",
    "\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "\n",
    "model.compile(optimizer='adam', # OR tf.keras.optimizers.Adam()\n",
    "              loss=loss_fn,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will now [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) our model in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "executionInfo": {
     "elapsed": 20052,
     "status": "ok",
     "timestamp": 1600445092343,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "ZjTJn4ssf_VX",
    "outputId": "3596c1db-778b-48e9-fc71-3abc526ba278"
   },
   "outputs": [],
   "source": [
    "# Training the model\n",
    "history = model.fit(x_train, y_train, validation_split=0.1, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How has our model changed over time?\n",
    "\n",
    "Run the following cells to observe the model metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 266
    },
    "executionInfo": {
     "elapsed": 713,
     "status": "ok",
     "timestamp": 1600445213471,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "dR4Jne4Bg5I_",
    "outputId": "8ed389fb-2383-4777-f3c6-e88c2242d3aa"
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(10,4))\n",
    "\n",
    "axes[0].plot(history.epoch, history.history['loss'], label='train')\n",
    "axes[0].plot(history.epoch, history.history['val_loss'], label='val')\n",
    "axes[0].set_ylabel('Loss')\n",
    "axes[0].legend()\n",
    "axes[1].plot(history.epoch, history.history['accuracy'], label='train')\n",
    "axes[1].plot(history.epoch, history.history['val_accuracy'], label='val')\n",
    "axes[1].set_ylabel('Accuracy')\n",
    "axes[1].legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 1178,
     "status": "ok",
     "timestamp": 1600445443945,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "G_GSlP3egBH-",
    "outputId": "c24aa7a8-7ac1-4319-c5f0-5c4ac0be35b1"
   },
   "outputs": [],
   "source": [
    "eval_metrics = model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we have successfully trained our model. It now produces useful results!\n",
    "\n",
    "Below, you can see how a model can be trained using the functional approach.\n",
    "\n",
    "The steps are mostly the same, so feel free to reference the **above** explanations when reading the **below** code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UMLVNM_cnDg5"
   },
   "source": [
    "##### 4.1.2.3 Training sequential models with TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "executionInfo": {
     "elapsed": 19998,
     "status": "ok",
     "timestamp": 1600445702880,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "F7b-nZ8anEL8",
    "outputId": "fbe12a5b-9e27-4fc2-e921-1c35b4e9bdb9"
   },
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(28,28)) # This is a symbolic input, a \"placeholder\"\n",
    "x = tf.keras.layers.Flatten()(inputs)\n",
    "x = tf.keras.layers.Dense(256, activation='relu')(x)\n",
    "x = tf.keras.layers.Dense(128, activation='relu')(x)\n",
    "outputs = tf.keras.layers.Dense(10)(x)\n",
    "\n",
    "model = tf.keras.models.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "model.compile(optimizer='adam',\n",
    "              loss=loss_fn,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x_train, y_train, validation_split=0.1, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 329
    },
    "executionInfo": {
     "elapsed": 476,
     "status": "ok",
     "timestamp": 1600445722945,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "49GDdknAnxd0",
    "outputId": "d9376e19-62ac-4d73-ade8-a4d6db1eae61"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the below cell, we demonstrate a few things we can do with the [Functional API](https://www.tensorflow.org/guide/keras/functional)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "executionInfo": {
     "elapsed": 21309,
     "status": "ok",
     "timestamp": 1600445841044,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "bDxqjBrIpUvl",
    "outputId": "e577e5f4-9b14-4eec-bf9d-416e6971278d",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(28,28))\n",
    "x = tf.keras.layers.Flatten()(inputs)\n",
    "x = tf.keras.layers.Dense(256, activation='relu')(x)\n",
    "x = tf.keras.layers.Dense(128, activation='relu')(x) + tf.keras.layers.Dense(128, activation='swish')(x) # Notice here\n",
    "outputs = tf.keras.layers.Dense(10)(x)\n",
    "\n",
    "model =  tf.keras.models.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "model.compile(optimizer='adam',\n",
    "              loss=loss_fn,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x_train, y_train, validation_split=0.1, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 416
    },
    "executionInfo": {
     "elapsed": 348,
     "status": "ok",
     "timestamp": 1600445858527,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "DyZ7JQNupkgm",
    "outputId": "32da5e21-181c-4283-b826-8fc03d878e4c"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 644
    },
    "executionInfo": {
     "elapsed": 409,
     "status": "ok",
     "timestamp": 1600445865058,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "lsVjg8ZcqH8Y",
    "outputId": "9354a578-1e0a-423c-eee2-f9d590e2f9e4"
   },
   "outputs": [],
   "source": [
    "tf.keras.utils.plot_model(model, show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8z4Xi476yq3l"
   },
   "source": [
    "#### 4.1.3 Loading data with TensorFlow\n",
    "\n",
    "In this section we look now at loading our own data into TensorFlow.\n",
    "\n",
    "We will start by using [`tf.keras.preprocessing.image_dataset_from_directory`](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image_dataset_from_directory). Then we will explore the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "executionInfo": {
     "elapsed": 4736,
     "status": "ok",
     "timestamp": 1600475114996,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "aPPUtejiwQmv",
    "outputId": "dfdafe1f-6605-4ee4-cd80-70be892e0bf2"
   },
   "outputs": [],
   "source": [
    "import pathlib\n",
    "import PIL\n",
    "\n",
    "dataset_url =\"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
    "data_dir = tf.keras.utils.get_file(origin=dataset_url, \n",
    "                                   fname='flower_photos', \n",
    "                                   untar=True,\n",
    "                                   cache_dir='/content/')\n",
    "\n",
    "data_dir = pathlib.Path(data_dir)\n",
    "\n",
    "#pathlib.Path('/content/datasets/flower_photos.tar.gz').unlink() # remove the tgz file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 402,
     "status": "ok",
     "timestamp": 1600475241796,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "BzyIrCf7x1CB",
    "outputId": "853913d6-d262-4ca9-8082-fde14544250f"
   },
   "outputs": [],
   "source": [
    "# pathlib is a great way to deal with paths in python\n",
    "data_dir/'folder'/'folder'/'file.jpg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 395,
     "status": "ok",
     "timestamp": 1600475244965,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "entafNgPHajO",
    "outputId": "001ff7dd-84df-4eaf-db1c-c5014e49b05d"
   },
   "outputs": [],
   "source": [
    "data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 437,
     "status": "ok",
     "timestamp": 1600475320114,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "50-_wnqey2Xi",
    "outputId": "65192e82-2798-4789-8e2b-df1da63504b4"
   },
   "outputs": [],
   "source": [
    "image_count = len(list(data_dir.glob('*/*.jpg')))\n",
    "print(image_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 257
    },
    "executionInfo": {
     "elapsed": 639,
     "status": "ok",
     "timestamp": 1600475375974,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "QNwzOJ-Czo5e",
    "outputId": "ed0a9927-9f46-4741-84f9-8356a2b08cf7"
   },
   "outputs": [],
   "source": [
    "roses = list(data_dir.glob('roses/*.jpg'))\n",
    "PIL.Image.open(str(roses[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's try to produce datasets from data, like we did in MNIST. First we will again set some basic hyperparameters. Afterward, we will use the above mentioned `image_dataset_from_directory` to build two datasets. [One for training, and one for validation](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0JQ7dDYD11S0"
   },
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "img_height = 28\n",
    "img_width = 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "executionInfo": {
     "elapsed": 970,
     "status": "ok",
     "timestamp": 1600475532086,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "0pTZbX0K1D1i",
    "outputId": "b2e97f8d-4a89-44d4-cd3a-0549e6eb8a4c"
   },
   "outputs": [],
   "source": [
    "train_ds = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "    data_dir,\n",
    "    validation_split=0.2,\n",
    "    subset=\"training\",\n",
    "    seed=123,\n",
    "    image_size=(img_height, img_width),\n",
    "    batch_size=batch_size)\n",
    "\n",
    "val_ds = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "    data_dir,\n",
    "    validation_split=0.2,\n",
    "    subset=\"validation\",\n",
    "    seed=123,\n",
    "    image_size=(img_height, img_width),\n",
    "    batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our datasets, we can move onto the training process! So what are we predicting?\n",
    "\n",
    "Below, we observe that just as in MNIST, we have a number of classes which we want to predict in our custom dataset.\n",
    "\n",
    "Therefore, our goal is very much the same. Once you observe the classes. Go ahead and start training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 386,
     "status": "ok",
     "timestamp": 1600475767071,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "v-gx4jAh3Ex_",
    "outputId": "6407189a-3608-4904-951e-aee28db50063"
   },
   "outputs": [],
   "source": [
    "class_names = train_ds.class_names\n",
    "print(class_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ST7Zch6uFNyN"
   },
   "outputs": [],
   "source": [
    "def get_model():\n",
    "    inputs = tf.keras.layers.Input(shape=(28,28,3))\n",
    "    \n",
    "    x = tf.keras.layers.experimental.preprocessing.Rescaling(1./255)(inputs)\n",
    "    x = tf.keras.layers.Flatten()(x)\n",
    "    x = tf.keras.layers.Dense(256, activation='relu')(x)\n",
    "    x = tf.keras.layers.Dense(128, activation='relu')(x)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dense(5)(x)\n",
    "    model =  tf.keras.models.Model(inputs=inputs, outputs=outputs)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "executionInfo": {
     "elapsed": 44942,
     "status": "ok",
     "timestamp": 1600475994877,
     "user": {
      "displayName": "Yassine Yousfi",
      "photoUrl": "",
      "userId": "06317191099946336363"
     },
     "user_tz": 240
    },
    "id": "Q_EamVob3k9x",
    "outputId": "b6407c78-bbbc-4068-fd3a-0ab86c59ebcf"
   },
   "outputs": [],
   "source": [
    "model = get_model()\n",
    "\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "model.compile(optimizer='adam',\n",
    "              loss=loss_fn,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=train_ds, validation_data=val_ds, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z7u5IHFg5Aa9"
   },
   "source": [
    "In the above cells, we effectively used `tf.keras.preprocessing.image_dataset_from_directory` together with pathlib to quickly create a dataset from a directory structure. \n",
    "\n",
    "Note that this simplicity restricts many aspects of the data loading:\n",
    "- How to read the files\n",
    "- How to resize them\n",
    "- They must be in a subdirectory structure\n",
    "- Sampling is uniform \n",
    "- ...\n",
    "\n",
    "However, for a beginner, this is more than enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that, we can conclude this lesson on an introduction to machine learning for software engineers. What a journey!\n",
    "\n",
    "You now have a strong foundation on which to continue growing as an engineer in the machine learning domain!\n",
    "\n",
    "Of course, you still have much to learn, we have really only scratched the surface. I strongly encourage you to keep experimenting with the concepts you learned about in this notebook. [More learning resources can be found here](https://www.tensorflow.org/learn)."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyMYHuX46zfV/PKmn5O5Ov8t",
   "collapsed_sections": [],
   "mount_file_id": "1Av88aSJczcHGijiyiXptY2Z7Y23V0Npg",
   "name": "intro2TF2x.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}