MNIST Multi-Layer Perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sLiTOS8M4KUT"
   },
   "source": [
    "# 1) Loading packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23268,
     "status": "ok",
     "timestamp": 1574470263898,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "SUFFMksf4h6B",
    "outputId": "e7e31672-0871-4e2e-f16e-e6f36ca7d7f4"
   },
   "outputs": [],
   "source": [
    "import struct\n",
    "import gzip\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IVPVO6QT5cou"
   },
   "source": [
    "# 2) Importing and transforming image data\n",
    "\n",
    "In Machine Learning applications, we need data to be represented in the form of vectors/arrays/matrices/tensors depending on the dimensionality of our data. \n",
    "\n",
    "Tensors = Containers\n",
    "\n",
    "A tensor is the basic building block of modern machine learning.\n",
    "At its core it’s a data container. Mostly it contains numbers. Sometimes it even contains strings, but that’s rare. \n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/2000/1*_D5ZvufDS38WkhK9rK32hQ.jpeg)\n",
    "![alt text](https://cdn-images-1.medium.com/max/2000/1*_D5ZvufDS38WkhK9rK32hQ.jpeg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XRb43O3y8R5f"
   },
   "source": [
    "##        a) Examples of tensors using the mighty NumPy package\n",
    "\n",
    "In the case of images, our data will be imported as a 4-D Tensor represented as\n",
    "\n",
    " (Number of images, Width of each image in pixels, Height of each image in pixels, Depth of image)\n",
    "\n",
    "\n",
    " 3D Tensor= Time series\n",
    "\n",
    "4D Tensor= Images\n",
    "\n",
    "5D Tensor= Videos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pNK6AhDd_uc_"
   },
   "source": [
    "### 0-D Tensor A.K.A \"Scalar\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23255,
     "status": "ok",
     "timestamp": 1574470263900,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "N6hL4NLO_uu6",
    "outputId": "86c9e020-3cec-414c-af2b-5758de15a12f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar = np.array(5)\n",
    "scalar.ndim # Using \"ndim\" to get number of dimensions of the object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BS7rZuIC_huG"
   },
   "source": [
    "### 1-D Tensor A.K.A \"Vector\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23247,
     "status": "ok",
     "timestamp": 1574470263901,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "8RIf0Vvc_ii-",
    "outputId": "8a959b66-5fe3-4e4e-cab4-858842ffdbcc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector = np.array([1,2,3,4])\n",
    "vector.ndim # Using \"ndim\" to get number of dimensions of the object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TwhwSNW48SQD"
   },
   "source": [
    "### 2-D Tensor A.K.A \"Matrix\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23235,
     "status": "ok",
     "timestamp": 1574470263901,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "oYdrEsK88VVK",
    "outputId": "35c524dc-0a55-4d8b-873e-e79d9a87f451"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = np.array([[5,10,15,30,25],\n",
    "              [20,30,65,70,90],\n",
    "              [7,80,95,20,30]])\n",
    "matrix.ndim # Using \"ndim\" to get number of dimensions of the object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kvFCK6sb8SpO"
   },
   "source": [
    "### 3-D Tensor A.K.A .... a cube of numbers :) That's when tensors start being useful\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23221,
     "status": "ok",
     "timestamp": 1574470263902,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "6ePL06Iy8V55",
    "outputId": "8045d87e-3b70-49c0-a57d-2e721f1c84e0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threedtensor = np.array([[[5,10,15,30,25],\n",
    "               [20,30,65,70,90],\n",
    "               [7,80,95,20,30]],\n",
    "\n",
    "               [[3,0,5,0,45],\n",
    "               [12,-2,6,7,90],\n",
    "               [18,-9,95,120,30]],\n",
    "               \n",
    "               [[17,13,25,30,15],\n",
    "               [23,36,9,7,80],\n",
    "               [1,-7,-5,22,3]]])\n",
    "threedtensor.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1YvdfFCU_3Ym"
   },
   "source": [
    "### Now, we can proceed to load our binary data using a function which I found online. The function is \"defined\" as read_idx. The function components are as follows:\n",
    "a) gzip: is a package to open gzip files\n",
    "\n",
    "b) open: is a function that opens files to read, write, or append. In that function we specify 'rb' which means \"read binary\"\n",
    "\n",
    "c) \"with\" gzip and open we will \"unpack\" the binary file using a package called \"struct\" based on the order and format of the binary data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25623,
     "status": "ok",
     "timestamp": 1574470266317,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "WNjVWR1c4z4E",
    "outputId": "e6aa66c6-c38f-49dc-e0ce-0d4b6e927a0d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\oshaat\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:5: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "def read_idx(filename):\n",
    "    with gzip.open(filename, 'rb') as f:\n",
    "        zero, data_type, dims = struct.unpack('>HBB', f.read(4))\n",
    "        shape = tuple(struct.unpack('>I', f.read(4))[0] for d in range(dims))\n",
    "        return np.fromstring(f.read(), dtype=np.uint8).reshape(shape)\n",
    "\n",
    "images_in_binary_format = read_idx('train-images-idx3-ubyte.gz')\n",
    "labels_of_the_images = read_idx('train-labels-idx1-ubyte.gz')\n",
    "\n",
    "images_in_binary_format = images_in_binary_format/255 # To make calculations easier we will make values range from 0 - 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xt80uLzgEck7"
   },
   "source": [
    "### Lets look at the \"Shape\" of the data that we loaded in by using the \"Shape\" function from NumPy and figure out what we need to do in order to further prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25608,
     "status": "ok",
     "timestamp": 1574470266319,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "j9TatRmKEc0g",
    "outputId": "6262a2df-5ef7-40f5-a43d-d05a9466da5b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_in_binary_format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25596,
     "status": "ok",
     "timestamp": 1574470266320,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "H2rDOOJFEyf0",
    "outputId": "dce111d5-6889-4fb6-9c0d-6fec2d39e384"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_in_binary_format.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25578,
     "status": "ok",
     "timestamp": 1574470266320,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "VEuvqnewEvv6",
    "outputId": "6342378a-eb7b-4841-fc17-586c013e5900"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 4, ..., 5, 6, 8], dtype=uint8)"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_of_the_images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25568,
     "status": "ok",
     "timestamp": 1574470266321,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "TnaTDYJkEzBV",
    "outputId": "fdd08016-c76e-4c25-b3b2-3c8c08a07d06"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_of_the_images.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rDux-S92EE2z"
   },
   "source": [
    "## B) To fully prepare the data for our Neural Network, we will \"flatten\" the image into a Vector of pixels using the \"Reshape\" function which can be used to manipulated matrices, arrays, and tensors\n",
    "![alt text](https://miro.medium.com/max/940/0*teIkFyoG1mj8Ul-r)\n",
    "![alt text](https://miro.medium.com/max/940/0*teIkFyoG1mj8Ul-r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mN46LfLkDezG"
   },
   "outputs": [],
   "source": [
    "flattened_image = images_in_binary_format.reshape(60000,784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PlXeAD8mDfHY"
   },
   "source": [
    "## Furthermore, we need to find a way to have the network predict and perform calculations on specific digit labels. For that we will create a prediction class which will organize potential label values (0-9) and will show us a \"1\" in place of the image's label in a form of an array.\n",
    "\n",
    "For that we use np.eye which generates an identity matrix of dimensions equivalent to the number of classes/digits we have\n",
    "\n",
    "Return a 2-D array with ones on the diagonal and zeros elsewhere.\n",
    "\n",
    "Example:\n",
    "\n",
    "np.eye(3)\n",
    "array([[ 1.,  0.,  0.],\n",
    "\n",
    "   [ 0.,  1.,  0.],\n",
    "\n",
    "   [ 0.,  0.,  1.]])\n",
    "   \n",
    "\n",
    "# When a [label] is placed after np.eye, np.eye acts as an array element index. So with only one element in it, it returns given number of rows element as array. This method is referred to as \"One-Hot Encoding\"\n",
    "\n",
    "Examples:\n",
    "\n",
    "np.eye(3)[1]\n",
    "\n",
    "array([ 0.,  1.,  0.])\n",
    "\n",
    "\n",
    "np.eye(3)[2]\n",
    "\n",
    "array([ 0.,  0.,  1.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25538,
     "status": "ok",
     "timestamp": 1574470266322,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "7p3jIEd0DfQR",
    "outputId": "27c5aa33-595d-4a61-826a-24c7c5dc83f8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 1., 0.]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_digits = 10 #Number of digits to classify (0-9)\n",
    "images_labels_classification_array = np.eye(n_digits)[labels_of_the_images.astype('int32')] #Creating a 60,000 x 10 matrix to show labels in an array format\n",
    "images_labels_classification_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25526,
     "status": "ok",
     "timestamp": 1574470266322,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "pI7oRB_gDfeU",
    "outputId": "9fbe2abb-c6ce-4ec7-cd19-c9fbab780ceb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10)"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_labels_classification_array.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lZGHtHY7Df6d"
   },
   "source": [
    "# 3) Fun part - Creating a plain vanilla \"Fully Connected Neural Network\" A.K.A \"Multi-Layer Perceptron\n",
    "\n",
    "## A) How are Neurons connected? What are the layers in the middle?\n",
    "\n",
    "  1. The layers in the middle: are called \"Hidden Layers\" which the user specifies. There is no specific rule to how many neurons should be in each hidden layers or what exactly goes into it. However, what goes into the hidden layers is referred to as the \"Architecture\" of the network, and it is considered an art in the field.\n",
    "\n",
    "  2. The layers are connected: by inputing all 784 pixels into the first layer, with each neuron representing the value which corresponds to the brightness of each image. The \"pattern\" in the neurons' brightness causes a specific pattern in the hidden layers, which lead to a final output/prediction.\n",
    "\n",
    "  3. The last layer represents the possible digits from (0-9). The values/activation within the neurons in the last layer represent the probability of the image fed into the network having a label of 0-1-2-3-4-5-6-7-8- or 9. Of course, the network classifies the digit based on the neuron with the highest probability.\n",
    "\n",
    "![alt text](https://thumbs.gfycat.com/DeadlyDeafeningAtlanticblackgoby-size_restricted.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YXrRAFnUZJnW"
   },
   "outputs": [],
   "source": [
    "number_of_node_in_hidden_layer = 64 # Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K_qPDQUbKeoh"
   },
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "an1-7tdxS2ba"
   },
   "source": [
    "\n",
    "##B) What is a neuron and How are the Neurons fed into the network?\n",
    "\n",
    "\n",
    "   It's just a thing that holds a number (represented as a circle). In our case, the number is a value from 0-1 (which was originally a value from 0-255). And for EACH IMAGE we have a total of 784 neurons (28 x 28 flattened images) as the INPUT to the network. The value for the MNIST dataset represents the \"greyscale\" value of the corresponding pixel.\n",
    "\n",
    "![alt text](https://miro.medium.com/max/1446/1*cLsTCWtUL1GYBUv8vnbOxw.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ead3nmzJSbU1"
   },
   "outputs": [],
   "source": [
    "number_of_input_neurons = flattened_image.shape[1] # This is 28 x 28 = 784"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SYIo8T3WY1TW"
   },
   "source": [
    "##C) How do the layers in the middle work exactly? (This is an explanation just to expalain intuition behind it)\n",
    "\n",
    "- Simple answer: The middle layers try to identify specific \"knobs\"/\"parameters\" to identify:\n",
    "\n",
    "  1.   patterns, loops, and maybe edges in specific regions within the handwritten image\n",
    "  2.   In this case the \"parameters\" are weights assigned to each \"connection\" between the images' neurons and the neurons in the first hidden layer. The network learns by adjusting the weights in areas where there is handwriting\n",
    "\n",
    "  3. The weighted sum is then computed.\n",
    "\n",
    "\n",
    "![alt text](https://thumbs.gfycat.com/BabyishGeneralFruitfly-small.gif)\n",
    "\n",
    "- We start off by \"initializing\" the parameters and get the network to start learning. We initialize using NumPy's \"np.random.rand\" function for random weights, and \"np.zeros\" for zero initial bias terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4n7G-gnZI9vd"
   },
   "outputs": [],
   "source": [
    "# 2 layers before passing into activation function which squishes values into 0-1 range\n",
    "W1 = np.random.randn(number_of_node_in_hidden_layer, number_of_input_neurons)# Connection weights layer 1\n",
    "b1 = np.zeros((number_of_node_in_hidden_layer,1)) # Neurons bias terms output layer 1\n",
    "W2 = np.random.randn(n_digits, number_of_node_in_hidden_layer) # Connection weights ourput layer\n",
    "b2 = np.zeros((n_digits,1)) # Neurons bias terms output layer\n",
    "\n",
    "# The network has the following number of parameters:\n",
    "# W1 = 784 * 64 => 4704 weights\n",
    "# b1 = 64 => 64 biases\n",
    "# W2 = 64 x 10 => 640 weights\n",
    "# b2 = 10 => 10 biases\n",
    "# A TOTAL OF 5,418 parameters need to be tweaked while putting into account the relationship between each of them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25497,
     "status": "ok",
     "timestamp": 1574470266324,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "tfKnSZ24ajG_",
    "outputId": "c4a3b390-4270-4add-d174-412fb77fb0fc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(64, 784)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W1.shape # Shape of weights matrix (64 connections per input neuron into first hidden layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25609,
     "status": "ok",
     "timestamp": 1574470266445,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "UqufnuvYavv0",
    "outputId": "c802336a-648a-486b-b0af-d271aeef31b8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(64, 1)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b1.shape # Shape of bias terms vector (64 neurons in first hidden layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QpNEMV3KZBjw"
   },
   "source": [
    "##D) How does the final output/prediction computed?\n",
    "\n",
    "  - After taking the weighted sum, we pass each neuron's output through an \"Activation Function\" which squishes the value (again) to 0-1 to represent probability. For this we will use a \"sigmoid\" function which is pretty much a logistic curve. However, we dont pass it through the sigmoid untill we add a \"bias\" term which allows us to specify exactly when we want the neuron to meaningfully light up or be activated\n",
    "\n",
    "![alt text](https://www.researchgate.net/profile/Tali_Leibovich-Raveh/publication/325868989/figure/fig2/AS:639475206074368@1529474178211/A-Basic-sigmoid-function-with-two-parameters-c1-and-c2-as-commonly-used-for-subitizing.png)\n",
    "\n",
    "  - We will define this function using NumPy as well. We will use the \"np.exp\" to get the exponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MPClqjTbZCXn"
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    sig = 1/(1+np.exp(-z))\n",
    "    return sig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZDvwigphbHEE"
   },
   "source": [
    "##E) How does the network learn what's right and what's wrong?\n",
    "\n",
    " - It compared what it predicted VS. what the actual label was and computes the \"loss\". This is similar to the Least Squares in regression and each problem would have a unique Loss Function (Objective Function)\n",
    "\n",
    "![alt text](https://images.slideplayer.com/16/4913205/slides/slide_6.jpg)\n",
    "\n",
    " - For the loss function, we get the sum of the element wise multiplication. Therefore we get the \"np.sum\" of the element wise products of the elements within \"np.multiply\". Which in this case are Y = Actual Value and Y_hat = Predicted Value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "z288a_3CbLF8"
   },
   "outputs": [],
   "source": [
    "def compute_multiclass_loss(Y, Y_hat):\n",
    "    L_sum = np.sum(np.multiply(Y, np.log(Y_hat)))\n",
    "    m = 60000\n",
    "    L = -(1/m) * L_sum\n",
    "    return L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q_gKcueJdccW"
   },
   "source": [
    "- The network tries to minimize the loss as much as it can by taking the derivative of the function \n",
    "\n",
    "![alt text](https://miro.medium.com/max/1204/1*t6OiVIMKw3SBjNzj-lp_Fw.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E_-iUFRhcSNx"
   },
   "source": [
    "##F) Lets write a loop to start training the model\n",
    "\n",
    "  - Steps within a loop:\n",
    "  1. Create \"skeleton of model\n",
    "      - Create empty input neurons\n",
    "      - Specify number of neurons in hidden layers\n",
    "  2. Input image data into skeleton\n",
    "  3. Multiply weights by image data and add bias terms to identify patterns\n",
    "  4. Pass weighted sums through \"Sigmoid activation function\" to get probabilities\n",
    "  5. Compute cost/loss based on probability output versus actual label using loss function\n",
    "  6. Fix weights by using BackPropagation\n",
    "\n",
    "![alt text](http://hmkcode.github.io/images/ai/bp_update_formula.png)\n",
    "\n",
    "A more mathematical point of view\n",
    "![alt text](https://slideplayer.com/slide/13458681/80/images/26/Cross-Entropy+Avoids+the+Slow+Learning.jpg)\n",
    "![alt text](https://i.stack.imgur.com/pYVzl.png)\n",
    "  \n",
    "  7. Finally, use learned parameters to predict output values\n",
    "\n",
    "\n",
    " RECAP OF VARIABLES WE DEFINED BEFORE:\n",
    "- W1 = (784 * 64) => 4704 weights (initialized randomly)\n",
    "- b1 = (64) => 64 biases (initialized at zero)\n",
    "- W2 = (64) x 10 => 640 weights (initialized randomly)\n",
    "- b2 = (10) => 10 biases (initialized at zero)\n",
    "- flattened_image (60000,784) 60 thousand images with 784 pixels in form of a vector \n",
    "- images_labels_classification_array (One hot encoded classification labels)\n",
    "- number_of_node_in_hidden_layer (Number of nodes we specific as 64 in hidden layer)\n",
    "- number_of_input_neurons (The number of neurons/pixels of the image, we start it off by being empty)\n",
    "- sigmoid (The function that squishes output into 0-1 range)\n",
    "- compute_multiclass_loss (The cross-entropy function that tries to minimize the error between prediction and actual label)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 231
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 359,
     "status": "error",
     "timestamp": 1574470344077,
     "user": {
      "displayName": "Omar Khalil",
      "photoUrl": "",
      "userId": "01975206412095305585"
     },
     "user_tz": 300
    },
    "id": "62_7SrcIdXZd",
    "outputId": "d7461e20-2774-4562-cea2-813750fd0f5e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 cost:  9.084404420526242\n",
      "Epoch 5 cost:  7.629835317341987\n",
      "Epoch 10 cost:  1.4751061828881564\n",
      "Epoch 15 cost:  1.1597604518580427\n",
      "Epoch 20 cost:  0.9813871470527742\n",
      "Epoch 25 cost:  0.8583677807581396\n",
      "Epoch 30 cost:  0.7743159138250205\n",
      "Epoch 35 cost:  0.7123862817493861\n",
      "Epoch 40 cost:  0.6646142143400833\n",
      "Epoch 45 cost:  0.6269977787360498\n",
      "Epoch 50 cost:  0.5962393957064208\n",
      "Epoch 55 cost:  0.5702965291277824\n",
      "Epoch 60 cost:  0.5480569906556133\n",
      "Epoch 65 cost:  0.5285646340029159\n",
      "Epoch 70 cost:  0.5112060650601117\n",
      "Epoch 75 cost:  0.49560729190849884\n",
      "Epoch 80 cost:  0.48149326300123846\n",
      "Epoch 85 cost:  0.4686426886254633\n",
      "Epoch 90 cost:  0.45684520713255256\n",
      "Epoch 95 cost:  0.445925117785683\n",
      "Epoch 100 cost:  0.43575783227262277\n",
      "Epoch 105 cost:  0.42625964276645106\n",
      "Epoch 110 cost:  0.41737134173777735\n",
      "Epoch 115 cost:  0.40903359423853847\n",
      "Epoch 120 cost:  0.40118828894847824\n",
      "Epoch 125 cost:  0.39378924010589705\n",
      "Epoch 130 cost:  0.386803322225323\n",
      "Epoch 135 cost:  0.38020462138788735\n",
      "Epoch 140 cost:  0.37396966637453066\n",
      "Epoch 145 cost:  0.36807147204151897\n",
      "Epoch 150 cost:  0.36247728339543284\n",
      "Epoch 155 cost:  0.35715262901015427\n",
      "Epoch 160 cost:  0.3520676657957559\n",
      "Epoch 165 cost:  0.34720194720552744\n",
      "Epoch 170 cost:  0.3425428524693422\n",
      "Epoch 175 cost:  0.33808150301211565\n",
      "Epoch 180 cost:  0.33380999597136585\n",
      "Epoch 185 cost:  0.3297194100687712\n",
      "Epoch 190 cost:  0.3257977546021313\n",
      "Epoch 195 cost:  0.3220308966278811\n",
      "Epoch 200 cost:  0.31840598258408404\n",
      "Epoch 205 cost:  0.3149130620728412\n",
      "Epoch 210 cost:  0.31154418552921986\n",
      "Epoch 215 cost:  0.30829167994334744\n",
      "Epoch 220 cost:  0.30514746757634514\n",
      "Epoch 225 cost:  0.30210357615360184\n",
      "Epoch 230 cost:  0.29915276342539476\n",
      "Epoch 235 cost:  0.2962886893056853\n",
      "Epoch 240 cost:  0.2935057668487555\n",
      "Epoch 245 cost:  0.29079896025160085\n",
      "Epoch 250 cost:  0.2881636811049325\n",
      "Epoch 255 cost:  0.2855958158641745\n",
      "Epoch 260 cost:  0.28309183738884863\n",
      "Epoch 265 cost:  0.28064889479862914\n",
      "Epoch 270 cost:  0.27826476680353956\n",
      "Epoch 275 cost:  0.2759376461533603\n",
      "Epoch 280 cost:  0.2736658496092603\n",
      "Epoch 285 cost:  0.27144760171671417\n",
      "Epoch 290 cost:  0.2692809685684808\n",
      "Epoch 295 cost:  0.2671638934323072\n",
      "Epoch 300 cost:  0.2650942431038569\n",
      "Epoch 305 cost:  0.26306984207732237\n",
      "Epoch 310 cost:  0.2610885337683667\n",
      "Epoch 315 cost:  0.2591482872214434\n",
      "Epoch 320 cost:  0.25724731754030045\n",
      "Epoch 325 cost:  0.2553841651581562\n",
      "Epoch 330 cost:  0.25355768449519617\n",
      "Epoch 335 cost:  0.2517669286301112\n",
      "Epoch 340 cost:  0.25001097799971733\n",
      "Epoch 345 cost:  0.24828879678949659\n",
      "Epoch 350 cost:  0.24659917283620475\n",
      "Epoch 355 cost:  0.2449407408468759\n",
      "Epoch 360 cost:  0.24331205507518955\n",
      "Epoch 365 cost:  0.24171167381800335\n",
      "Epoch 370 cost:  0.2401382290371998\n",
      "Epoch 375 cost:  0.2385904694123812\n",
      "Epoch 380 cost:  0.23706727791205384\n",
      "Epoch 385 cost:  0.2355676708182551\n",
      "Epoch 390 cost:  0.23409078462121274\n",
      "Epoch 395 cost:  0.23263585555792435\n",
      "Epoch 400 cost:  0.23120219674596326\n",
      "Epoch 405 cost:  0.2297891779488106\n",
      "Epoch 410 cost:  0.22839621113025657\n",
      "Epoch 415 cost:  0.22702274251492707\n",
      "Epoch 420 cost:  0.22566825089087575\n",
      "Epoch 425 cost:  0.2243322521916485\n",
      "Epoch 430 cost:  0.22301431044041048\n",
      "Epoch 435 cost:  0.22171405401345598\n",
      "Epoch 440 cost:  0.22043119417268445\n",
      "Epoch 445 cost:  0.21916554079309802\n",
      "Epoch 450 cost:  0.21791700896661367\n",
      "Epoch 455 cost:  0.21668561050149585\n",
      "Epoch 460 cost:  0.21547142736750055\n",
      "Epoch 465 cost:  0.21427457032252742\n",
      "Epoch 470 cost:  0.213095133050498\n",
      "Epoch 475 cost:  0.21193315520981065\n",
      "Epoch 480 cost:  0.21078860348934608\n",
      "Epoch 485 cost:  0.20966137064721513\n",
      "Epoch 490 cost:  0.2085512851747652\n",
      "Epoch 495 cost:  0.20745812302640546\n",
      "Epoch 500 cost:  0.2063816166119506\n",
      "Epoch 505 cost:  0.20532146075900434\n",
      "Epoch 510 cost:  0.20427731756600143\n",
      "Epoch 515 cost:  0.20324882180437487\n",
      "Epoch 520 cost:  0.2022355873143837\n",
      "Epoch 525 cost:  0.20123721397959385\n",
      "Epoch 530 cost:  0.20025329466521555\n",
      "Epoch 535 cost:  0.1992834216604262\n",
      "Epoch 540 cost:  0.19832719235776872\n",
      "Epoch 545 cost:  0.1973842140344686\n",
      "Epoch 550 cost:  0.1964541077082345\n",
      "Epoch 555 cost:  0.19553651115332393\n",
      "Epoch 560 cost:  0.19463108124417186\n",
      "Epoch 565 cost:  0.19373749578186544\n",
      "Epoch 570 cost:  0.19285545484002564\n",
      "Epoch 575 cost:  0.19198468150296638\n",
      "Epoch 580 cost:  0.19112492175967447\n",
      "Epoch 585 cost:  0.1902759433409017\n",
      "Epoch 590 cost:  0.18943753346200695\n",
      "Epoch 595 cost:  0.1886094957058052\n",
      "Epoch 600 cost:  0.18779164653153504\n",
      "Epoch 605 cost:  0.18698381199938363\n",
      "Epoch 610 cost:  0.18618582518615392\n",
      "Epoch 615 cost:  0.1853975244793673\n",
      "Epoch 620 cost:  0.18461875261206928\n",
      "Epoch 625 cost:  0.18384935609104028\n",
      "Epoch 630 cost:  0.1830891846548988\n",
      "Epoch 635 cost:  0.1823380905427487\n",
      "Epoch 640 cost:  0.1815959275491187\n",
      "Epoch 645 cost:  0.1808625499775242\n",
      "Epoch 650 cost:  0.1801378116382205\n",
      "Epoch 655 cost:  0.179421564996409\n",
      "Epoch 660 cost:  0.17871366052882515\n",
      "Epoch 665 cost:  0.1780139463301358\n",
      "Epoch 670 cost:  0.1773222680208682\n",
      "Epoch 675 cost:  0.17663846901340086\n",
      "Epoch 680 cost:  0.17596239116604545\n",
      "Epoch 685 cost:  0.17529387579654274\n",
      "Epoch 690 cost:  0.17463276495486552\n",
      "Epoch 695 cost:  0.17397890279514422\n",
      "Epoch 700 cost:  0.17333213685431215\n",
      "Epoch 705 cost:  0.17269231904691182\n",
      "Epoch 710 cost:  0.17205930622103238\n",
      "Epoch 715 cost:  0.17143296018667634\n",
      "Epoch 720 cost:  0.17081314722009563\n",
      "Epoch 725 cost:  0.1701997371548059\n",
      "Epoch 730 cost:  0.16959260227064138\n",
      "Epoch 735 cost:  0.16899161625533177\n",
      "Epoch 740 cost:  0.16839665350908908\n",
      "Epoch 745 cost:  0.16780758898094358\n",
      "Epoch 750 cost:  0.16722429858637994\n",
      "Epoch 755 cost:  0.16664666010505455\n",
      "Epoch 760 cost:  0.16607455434429638\n",
      "Epoch 765 cost:  0.16550786630765524\n",
      "Epoch 770 cost:  0.16494648612736817\n",
      "Epoch 775 cost:  0.16439030958415016\n",
      "Epoch 780 cost:  0.16383923812192538\n",
      "Epoch 785 cost:  0.16329317835078322\n",
      "Epoch 790 cost:  0.16275204110798094\n",
      "Epoch 795 cost:  0.16221574020603652\n",
      "Epoch 800 cost:  0.1616841910301096\n",
      "Epoch 805 cost:  0.16115730914799956\n",
      "Epoch 810 cost:  0.16063500906813113\n",
      "Epoch 815 cost:  0.16011720323739817\n",
      "Epoch 820 cost:  0.1596038013298637\n",
      "Epoch 825 cost:  0.15909470985257415\n",
      "Epoch 830 cost:  0.1585898320896719\n",
      "Epoch 835 cost:  0.1580890684160327\n",
      "Epoch 840 cost:  0.15759231702983723\n",
      "Epoch 845 cost:  0.1570994751738584\n",
      "Epoch 850 cost:  0.15661044093089418\n",
      "Epoch 855 cost:  0.15612511567583637\n",
      "Epoch 860 cost:  0.15564340721900102\n",
      "Epoch 865 cost:  0.1551652335451578\n",
      "Epoch 870 cost:  0.15469052680875106\n",
      "Epoch 875 cost:  0.15421923690344547\n",
      "Epoch 880 cost:  0.15375133360217044\n",
      "Epoch 885 cost:  0.15328680620465907\n",
      "Epoch 890 cost:  0.15282566009035659\n",
      "Epoch 895 cost:  0.15236791056758064\n",
      "Epoch 900 cost:  0.1519135754996104\n",
      "Epoch 905 cost:  0.15146266869052016\n",
      "Epoch 910 cost:  0.1510151955693849\n",
      "Epoch 915 cost:  0.15057115164288065\n",
      "Epoch 920 cost:  0.15013052318259593\n",
      "Epoch 925 cost:  0.1496932891315374\n",
      "Epoch 930 cost:  0.14925942325104025\n",
      "Epoch 935 cost:  0.14882889583672015\n",
      "Epoch 940 cost:  0.14840167469015042\n",
      "Epoch 945 cost:  0.14797772533716633\n",
      "Epoch 950 cost:  0.1475570106998984\n",
      "Epoch 955 cost:  0.14713949054069278\n",
      "Epoch 960 cost:  0.14672512099388538\n",
      "Epoch 965 cost:  0.14631385440413078\n",
      "Epoch 970 cost:  0.14590563954729285\n",
      "Epoch 975 cost:  0.14550042218174683\n",
      "Epoch 980 cost:  0.14509814580475403\n",
      "Epoch 985 cost:  0.14469875247697958\n",
      "Epoch 990 cost:  0.14430218360903269\n",
      "Epoch 995 cost:  0.1439083806503104\n",
      "Final cost: 0.14359529077012503\n"
     ]
    }
   ],
   "source": [
    "n_x = X.shape[1]\n",
    "n_h = 64 # Architecture\n",
    "learning_rate = 10\n",
    "W1 = np.random.randn(n_h, n_x)# Neurons weights\n",
    "b1 = np.zeros((n_h,1)) # Neurons intercepts\n",
    "W2 = np.random.randn(n_digits, n_h) # Probability/weights of prediction\n",
    "b2 = np.zeros((n_digits,1))\n",
    "epochs = 1000\n",
    "for epoch in range(epochs):\n",
    "    Z1 = np.matmul(W1,X.T)+b1 # Feed forward prop\n",
    "    A1 = sigmoid(Z1) # Feedforward activation\n",
    "    Z2 = np.matmul(W2,A1)+b2 \n",
    "    A2 = np.exp(Z2) / np.sum(np.exp(Z2), axis=0) #Softmax\n",
    "    \n",
    "    cost = compute_multiclass_loss(images_labels_classification_array,A2.T)\n",
    "    \n",
    "    # Cross entropy cost\n",
    "    dZ2 = A2.T - images_labels_classification_array\n",
    "    dW2 = (1./y.shape[0])*np.matmul(A1,dZ2)\n",
    "    db2 = (1./y.shape[0])*np.sum(dZ2,axis=0,keepdims=True)\n",
    "    \n",
    "    dA1 = np.matmul(dZ2,W2)\n",
    "    dZ1 = dA1.T*sigmoid(Z1)*(1-sigmoid(Z1))\n",
    "    dW1 = (1./y.shape[0]) * np.matmul(dZ1,X)\n",
    "    db1 = (1./y.shape[0]) * np.sum(dZ1, axis=0, keepdims=True)\n",
    "    \n",
    "    W2 = W2 - learning_rate*dW2.T\n",
    "    b2 = b2 - learning_rate*db2.T\n",
    "    W1 = W1 - learning_rate*dW1\n",
    "    b1 = b1 - learning_rate*db1\n",
    "    \n",
    "    if(epoch % 5 == 0):\n",
    "        print('Epoch', epoch, 'cost: ', cost)\n",
    "\n",
    "print(\"Final cost:\", cost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95865"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z1 = np.matmul(W1,X.T)+b1 # First layer output\n",
    "A1 = sigmoid(Z1) # First layer activation\n",
    "Z2 = np.matmul(W2,A1.T)+b2 # Output layer output\n",
    "A2 = sigmoid(Z2) #Output layer output\n",
    "prediction = np.array([i.argmax() for i in A2.T]) #Predictions by taking the label with the maximum probability in the final layer A2\n",
    "(60000-(prediction!=y).sum())/60000 # Simple accuracy measure"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "pNK6AhDd_uc_",
    "BS7rZuIC_huG",
    "TwhwSNW48SQD",
    "kvFCK6sb8SpO",
    "Xt80uLzgEck7"
   ],
   "machine_shape": "hm",
   "name": "MNIST Multi-Layer Perceptron.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}