FFNN from Scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e47faf11",
   "metadata": {},
   "source": [
    "# Building a Feedforward Neural Network (FFNN) from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ea38165",
   "metadata": {},
   "source": [
    "### 1.Building a Feedforward Neural Network (FFNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ddf7d4d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Task 1: Build a Feedforward Neural Network (FFNN)\n",
    "input_size = 2\n",
    "hidden_size = 2\n",
    "output_size = 1\n",
    "\n",
    "# Initialize random weights and biases for the hidden and output layers\n",
    "hidden_weights = np.random.randn(input_size, hidden_size)\n",
    "hidden_biases = np.random.randn(1, hidden_size)\n",
    "output_weights = np.random.randn(hidden_size, output_size)\n",
    "output_biases = np.random.randn(1, output_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f67ddfa",
   "metadata": {},
   "source": [
    "### 2.Implementing Forward Propagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "6d207912",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task 2: Implement Forward Propagation\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def forward_propagation(input_data):\n",
    "    hidden_layer_input = np.dot(input_data, hidden_weights) + hidden_biases\n",
    "    hidden_layer_output = sigmoid(hidden_layer_input)\n",
    "    output_layer_input = np.dot(hidden_layer_output, output_weights) + output_biases\n",
    "    output_layer_output = sigmoid(output_layer_input)\n",
    "    return output_layer_output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eb51e02",
   "metadata": {},
   "source": [
    "### 3. Defining a Loss Function (Mean Squared Error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ecf50470",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task 3: Define a Loss Function (Mean Squared Error)\n",
    "def mean_squared_error(predicted, actual):\n",
    "    return np.mean((predicted - actual) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d29aec6",
   "metadata": {},
   "source": [
    "### 4.Initializing Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "9697b281",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task 4: Initialize Hyperparameters\n",
    "learning_rate = 0.1\n",
    "num_epochs = 10000\n",
    "batch_size = 4  # XOR problem dataset size\n",
    "\n",
    "# XOR problem dataset\n",
    "X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
    "y = np.array([[0], [1], [1], [0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "985657e0",
   "metadata": {},
   "source": [
    "#### Learning Rate:\n",
    "\n",
    "Importance: The learning rate controls the step size of weight updates during training. It is arguably one of the most critical hyperparameters in training neural networks.\n",
    "Impact: A learning rate that is too high can cause the model to converge quickly but may result in overshooting the optimal weights, leading to instability. A learning rate that is too low can lead to slow convergence and potential convergence to suboptimal solutions.\n",
    "Tuning: The learning rate needs to be tuned carefully. Common strategies include trying a range of values (e.g., 0.1, 0.01, 0.001) and using learning rate schedules (e.g., reducing the learning rate over time) to balance convergence speed and stability.\n",
    "\n",
    "#### Number of Training Epochs:\n",
    "\n",
    "Importance: The number of training epochs determines how many times the entire dataset is processed during training. It controls the duration of training.\n",
    "Impact: Too few epochs may result in underfitting, where the model doesn't capture the underlying patterns in the data. Too many epochs can lead to overfitting, where the model memorizes the training data but doesn't generalize well to new data.\n",
    "Tuning: The number of epochs depends on the complexity of the problem and the dataset. Techniques like early stopping can help determine when to stop training based on validation performance.\n",
    "\n",
    "#### Batch Size:\n",
    "\n",
    "Importance: The batch size defines how many samples are used in each iteration of training. It affects both memory usage and convergence behavior.\n",
    "Impact: Smaller batch sizes introduce more noise into the gradient estimates but can lead to better convergence and generalization. Larger batch sizes provide more stable gradient estimates but may slow down training and lead to suboptimal convergence.\n",
    "Tuning: The choice of batch size depends on factors like the dataset size, available memory, and model architecture. Values like 32, 64, or 128 are commonly used starting points.\n",
    "These hyperparameters are crucial because they directly influence how the neural network learns and generalizes from the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83b8129",
   "metadata": {},
   "source": [
    "### 5 & 6.Implementing Backpropagation Algorithm & Monitoring loss during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2700604e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, Loss: 0.4074865958576914\n",
      "Epoch 1000, Loss: 0.24994441838112022\n",
      "Epoch 2000, Loss: 0.24739232166248906\n",
      "Epoch 3000, Loss: 0.22970976328254944\n",
      "Epoch 4000, Loss: 0.18396367752620552\n",
      "Epoch 5000, Loss: 0.151759826701205\n",
      "Epoch 6000, Loss: 0.06278094996135467\n",
      "Epoch 7000, Loss: 0.015358949083753171\n",
      "Epoch 8000, Loss: 0.00791742059911669\n",
      "Epoch 9000, Loss: 0.005210586222572294\n"
     ]
    }
   ],
   "source": [
    "# Task 5: Implement Backpropagation Algorithm\n",
    "for epoch in range(num_epochs):\n",
    "    # Mini-batch training (in this case, the whole dataset is one batch)\n",
    "    input_data = X\n",
    "    target_output = y\n",
    "\n",
    "    # Forward propagation\n",
    "    hidden_layer_input = np.dot(input_data, hidden_weights) + hidden_biases\n",
    "    hidden_layer_output = sigmoid(hidden_layer_input)\n",
    "    output_layer_input = np.dot(hidden_layer_output, output_weights) + output_biases\n",
    "    output_layer_output = sigmoid(output_layer_input)\n",
    "\n",
    "    # Loss calculation\n",
    "    loss = mean_squared_error(output_layer_output, target_output)\n",
    "\n",
    "    # Backpropagation\n",
    "    output_error = target_output - output_layer_output\n",
    "    output_delta = output_error * output_layer_output * (1 - output_layer_output)\n",
    "    \n",
    "    hidden_layer_error = output_delta.dot(output_weights.T)\n",
    "    hidden_layer_delta = hidden_layer_error * hidden_layer_output * (1 - hidden_layer_output)\n",
    "\n",
    "    # Parameter updates\n",
    "    output_weights += hidden_layer_output.T.dot(output_delta) * learning_rate\n",
    "    output_biases += np.sum(output_delta, axis=0, keepdims=True) * learning_rate\n",
    "    hidden_weights += input_data.T.dot(hidden_layer_delta) * learning_rate\n",
    "    hidden_biases += np.sum(hidden_layer_delta, axis=0, keepdims=True) * learning_rate\n",
    "    \n",
    " # Task 6: Monitor loss during training\n",
    "    if epoch % 1000 == 0:\n",
    "        print(f\"Epoch {epoch}, Loss: {loss}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea49b4f",
   "metadata": {},
   "source": [
    "### 7. Printing and Visualizing Weights (printing weights after training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "fa28cc67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hidden Layer Weights:\n",
      "[[-4.408388   -5.4296859 ]\n",
      " [ 4.87632863  6.05797701]]\n",
      "Output Layer Weights:\n",
      "[[-6.75596773]\n",
      " [ 7.14608832]]\n"
     ]
    }
   ],
   "source": [
    "# Task 7: Print and Visualize Weights (printing weights after training)\n",
    "print(\"Hidden Layer Weights:\")\n",
    "print(hidden_weights)\n",
    "print(\"Output Layer Weights:\")\n",
    "print(output_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c914a64c",
   "metadata": {},
   "source": [
    "### 8. Printing and Visualizing Weights (printing weights after training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "805c918c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 100.0 %\n"
     ]
    }
   ],
   "source": [
    "# Task 8: Test the Trained Network\n",
    "test_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
    "expected_output = np.array([[0], [1], [1], [0]])\n",
    "\n",
    "# Forward pass on the test data\n",
    "predicted_output = forward_propagation(test_data)\n",
    "\n",
    "# Calculate accuracy for XOR problem\n",
    "correct_predictions = np.round(predicted_output)\n",
    "accuracy = np.mean(correct_predictions == expected_output) * 100\n",
    "\n",
    "print(\"Test Accuracy:\", accuracy, \"%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a33ced2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e220ed39",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "141fa3f9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}