week5-neural-networks/Neural Network Full Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Recap: The Learning Problem\n",
    "\n",
    "We have a dataset of $(x, y)$ pairs where $x$ denotes a vector of features and $y$ denotes the label for that feature vector. We want to learn a function $h(x)$ that maps features to labels, with good generalization accuracy. We do this by minimizing a loss function computed on our dataset: $ \\sum_{i=1}^{N} L(y_i, h(x_i)) $. There are many loss functions we can choose. We have gone over the cross-entropy loss and variants of the squared error loss functions in previous workshops, and we will once again consider those today. \n",
    "\n",
    "\n",
    "### Review: A Single \"Neuron\", aka the Perceptron\n",
    "\n",
    "![perceptron](https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/perceptron.png)\n",
    "\n",
    "Previously, we sow how to calculate a **weighted sum** of our inputs, and then apply a **nonlinear** activation function to it to obtain a prediction between $0$ and $1$, which can be interpreted as a probability. \n",
    "\n",
    "First, we multiply each of our features $(x_1, x_2, ... x_n) \\in x$ with an associated weight $(w_1, w_2, ... w_n)$ .\n",
    "\n",
    "Next, we applied the sigmoid nonlinearity to this output. It turns out that if you're only aiming for binary classification, there's an even simpler approach than taking the sigmoid nonlinearity - just take the sign of the output as your predicted label (make an arbitary choice for zero). This model is called a **perceptron**, and the prediction function is given by $h(x) = sign(w^Tx + b) $.\n",
    "\n",
    "Previously, we saw an even more general approach to classification than just taking the sign - using the sigmoid ($\\sigma$) function. As a reminder, here's the sigmoid function: \n",
    "\n",
    "![sigmoid](https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/600px-Logistic-curve.svg.png)\n",
    "\n",
    "Therefore, the function we compute for logistic regression is $h(x) = \\sigma (w^Tx + b)$. As we get into neural networks, it's useful to start thinking about this classification model - taking a linear combination of our weights and passing them through a nonlinearity - as a **single unit** of computation. We will see how these individual units work together to make up a neural network. Here's a representation of such a unit: \n",
    "![sigmoidunit](https://raw.githubusercontent.com/uclaacmai/tf-workshop-series-fall17/master/week5-neural-networks/sigmoid_unit.png)\n",
    "\n",
    "The sigmoid function is commonly referred to as an \"activation\" function. When we say that a \"neuron computes an activation function\", it means that a standard linear combination is calculated ($w^Tx + b$) and then we apply a _non linear_ function to it, such as the sigmoid function. \n",
    "\n",
    "Here are a few other common activation functions: \n",
    "\n",
    "![tanh](http://www.dplot.com/functions/tanh.png)\n",
    "![relu](https://i.stack.imgur.com/8CGlM.png)\n",
    "\n",
    "We will see that when we build up our neural network, we'll have to make a choice as to which activation function we should use. It turns out that each particular activation function has its benefits and disadvantages.\n",
    "\n",
    "\n",
    "### Review: From binary to multi-class classification\n",
    "\n",
    "The most important change in moving from a binary (negative/positive) classification model to one that can classify training instances into many different classes (say, 10, for MNIST) is that our vector of weights $w$ changes into a matrix $W$. \n",
    "\n",
    "Each row of weights we learn represents the parameters for a certain class: \n",
    "\n",
    "![weights](https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/imagemap.jpg)\n",
    "\n",
    "We also want to take our output and normalize the results so that they all sum to one, so that we can interpret them as probabilities. This is commonly done using the _softmax_ function, which takes in a vector and returns another vector whose elements sum to 1, and each element is proportional in scale to what it was in the original vector. In binary classification we used the sigmoid function to compute probabilities. Now since we have a vector, we use the softmax function.  \n",
    "\n",
    "Here is our current model of learning, then:\n",
    "\n",
    "$h(x) = softmax(Wx + b) $.\n",
    "\n",
    "Looking at our example in the picture, we've computed $Wx + b$. Applying softmax to the output vector, $[-96.8,437.5, 61.95]$ produces $[9.04603615e^{-233},1.00000000000,7.95622985e^{-164}]$. Our prediction is taken as the index of the max value in this array, so we'd output 1 in this case.\n",
    "\n",
    "### Building up the neural network\n",
    "\n",
    "Now that we've figured out how to linearly model multi-class classification, we can create a basic neural network. Consider what happens when we combine the idea of artificial neurons with our softmax classifier. Instead of computing a linear function  $Wx + b$ and immediately passing the output to a softmax function, we have an intermediate step: pass the output of our linear combination to a vector of artificial neurons, which each compute a nonlinear function. \n",
    "\n",
    "The output of this \"layer\" of neurons can be multiplied with a matrix of weights again, and we can apply our softmax function to this result to produce our predictions. \n",
    "\n",
    "** Original function **: $h(x) = softmax(Wx + b)$\n",
    "\n",
    "** Neural Network function **: $h(x) = softmax(W_2(nonlin(W_1x + b_1)) + b_2)$\n",
    "\n",
    "The key differences are that we have more biases and weights, as well as a larger composition of functions. This function is harder to optimize, and introduces a few interesting ideas about learning the weights with an algorithm known as backpropagation.\n",
    "\n",
    "This “intermediate step” is actually known as a hidden layer, and we have complete control over it, meaning that among other things, we can vary the number of parameters or connections between weights and neurons to obtain an optimal network. It’s also important to notice that we can stack an arbitrary amount of these hidden layers between the input and output of our network, and we can tune these layers individually. This lets us make our network as deep as we want it. For example, here’s what a neural network with two hidden layers would look like:\n",
    "\n",
    "![neuralnet](https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/neuralnet.png)\n",
    "\n",
    "We're now ready to start implementing a basic neural network in Tensorflow. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEKFJREFUeJzt3X2MXNV9xvHvE3B4MYja9cqsePECsireVBMGEgGiboEU\nDAiiChRLCQuhMUKgFgFVgSpgVUS1UAgFtUKYQDGUAqEkwlAbcK0UHJUSBmqMsdVCLTtgbLwuQdgU\n8WJ+/WMu0WJ2zqxn7swd73k+0mpn7+/eub8d7+M7c8+dOYoIzCw/X6m6ATOrhsNvlimH3yxTDr9Z\nphx+s0w5/GaZcvgzJunfJP1pr7e1/uDwTwCS1ks6veo+mpE0LOklSe9LekvSLZL2rLqv3Dn81gv7\nAlcB04CvA6cB11bakTn8E5mkKZKelDQi6TfF7YN3Wu0ISb8qjsqPS5o6avtvSPp3Se9JekXS7Hb6\niIg7I2JFRHwcERuBB4GT2//NrAwO/8T2FeAfgBnAocCHwN/ttM5FwPeAQeBT4A4ASQcB/wLcDEyl\ncaR+TNLAzjuRdGjxH8Sh4+zrVOC1Xf5trFQO/wQWEf8bEY9FxP9FxDbgh8Af7LTaAxGxOiI+AH4A\nXChpD+A7wJKIWBIRn0XEMqAOzBljP7+OiN+JiF+36knS94Aa8KMOfz3rkE+6TGCS9gVuA84EphSL\n95e0R0TsKH5+c9QmG4BJNF6bzwAukHTuqPok4Bcd9HM+8DfA6RGxtd37sXI4/BPbNcDvAV+PiM2S\nZgH/CWjUOoeMun0o8AmwlcZ/Cg9ExPfLaETSmcDdwNkR8WoZ92md8dP+iWOSpL1Hfe0J7E/jdf57\nxYm8m8bY7juSjiqeJfw18M/Fs4J/BM6V9MeS9ijuc/YYJwxbkvRHNE7y/UlE/Krt39BK5fBPHEto\nBP3zr/nA3wL70DiS/wfw1BjbPQDcB2wG9gb+DCAi3gTOA24ARmg8E/gLxvibKU74bU+c8PsBcACw\npFhvu6Slbf2WVhr5wzzM8uQjv1mmHH6zTDn8Zply+M0y1dNx/mnTpsXQ0FAvd2mWlfXr17N161a1\nXrPD8BcXbtwO7AH8JCIWpNYfGhqiXq93skszS6jVauNet+2n/cX1338PnAUcBcyVdFS792dmvdXJ\na/4TgTciYl1EfAw8TOOiEDPbDXQS/oP44ptC3iqWfYGkeZLqkuojIyMd7M7MytT1s/0RsTAiahFR\nGxj40lvBzawinYR/I198R9jBxTIz2w10Ev4XgZmSDpP0VeDbwOJy2jKzbmt7qC8iPpV0JfA0jaG+\neyPCH81ktpvoaJw/IpbQeCupme1mfHmvWaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/\nWaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TD\nb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlqqNZeq0cmzdvTtaXLl2arK9du7Zpbc2aNcltlyxJ\nT7J8zTXXJOtz5sxJ1o888simtX322Se57QEHHJCs79ixI1m///77m9Y++OCD5LaXXXZZsj5p0qRk\nfXfQUfglrQe2ATuATyOiVkZTZtZ9ZRz5/zAitpZwP2bWQ37Nb5apTsMfwDOSXpI0b6wVJM2TVJdU\nHxkZ6XB3ZlaWTsN/SkR8DTgLuELSqTuvEBELI6IWEbWBgYEOd2dmZeko/BGxsfi+Bfg5cGIZTZlZ\n97UdfkmTJe3/+W3gm8Dqshozs+5SRLS3oXQ4jaM9NEYN/ikifpjaplarRb1eb2t/u7NFixYl65dc\nckmyLqnMdnZJq7+PTno7/PDDk/XUOD3AihUrkvXrr79+l3v63MqVK5P1Y489tu377qZarUa9Xh/X\nP0rbQ30RsQ74/Xa3N7NqeajPLFMOv1mmHH6zTDn8Zply+M0y5bf0luDtt99O1q+++uoedbLrZsyY\nkayvX7++a/tet25dsn7KKack650MQ06bNi257X777ZesTwQ+8ptlyuE3y5TDb5Yph98sUw6/WaYc\nfrNMOfxmmfI4fwmWL1+erL/33nsd3f/555+frM+fP7/t+2413t3qo9da/W4XX3xx09qGDRuS23Zq\n6tSpTWuPPPJIctvDDjus7Hb6jo/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmPM5fgjvvvLOj\n7SdPnpysH3fcccn6008/3fa+TzjhhGT9xRdfTNafeOKJZL3bY/kpBx98cNPa7Nmze9dIn/KR3yxT\nDr9Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlMf5S3Dqqacm663Gyrdv356s33jjjbvcU1m6OUX33nvv\nnaxfd911yfott9ySrL/yyitNa8uWLUtue8YZZyTrE0HLI7+keyVtkbR61LKpkpZJer34PqW7bZpZ\n2cbztP8+4Mydll0HLI+ImcDy4mcz2420DH9EPAe8u9Pi84BFxe1FQPpzpsys77R7wm96RGwqbm8G\npjdbUdI8SXVJ9VafB2dmvdPx2f5onBFqelYoIhZGRC0iagMDA53uzsxK0m7435E0CFB831JeS2bW\nC+2GfzEwXNweBh4vpx0z65WW4/ySHgJmA9MkvQXcBCwAfirpUmADcGE3m+x3CxYsSNa3bEk/Mbrv\nvvuS9U7G0rttxowZyXrqswiuvfba5LYnnXRSst7q+ohbb721aW3p0qXJbXMY528Z/oiY26R0Wsm9\nmFkP+fJes0w5/GaZcvjNMuXwm2XK4TfLlN/S2wOt3pI7d26zAZX+d8wxxyTrg4ODPepk16xdu7bq\nFirnI79Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlimP8/fA0NBQR3Ub25o1a9re9sgjjyyxk92T\nj/xmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaY8zm99q9XU5kuWLEnWp09vOoscl19+eVs9TSQ+\n8ptlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmfI4v1Xmww8/TNbPOeecZD0ikvUDDzywaW3mzJnJ\nbXPQ8sgv6V5JWyStHrVsvqSNklYWX3O626aZlW08T/vvA84cY/ltETGr+EpfamVmfadl+CPiOeDd\nHvRiZj3UyQm/KyWtKl4WTGm2kqR5kuqS6iMjIx3szszK1G747wSOAGYBm4Bbm60YEQsjohYRtYGB\ngTZ3Z2Zlayv8EfFOROyIiM+Au4ETy23LzLqtrfBLGj3v8reA1c3WNbP+1HKcX9JDwGxgmqS3gJuA\n2ZJmAQGsBy7rYo+2G9u2bVvT2vDwcHLbrVu3JuuSkvWzzz47Wc9dy/BHxNwxFt/ThV7MrId8ea9Z\nphx+s0w5/GaZcvjNMuXwm2XKb+m1rnr44Yeb1h5//PGO7nvWrFnJ+hVXXNHR/U90PvKbZcrhN8uU\nw2+WKYffLFMOv1mmHH6zTDn8ZpnyOL8ltfp47UsvvTRZbzWNdsrRRx+drN98883J+uDgYLKeOx/5\nzTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMeZy/BIsWLUrWly5dmqw/++yzZbazS0444YRk/YUX\nXkjWW328dicWL16crA8NDXVt3znwkd8sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y9R4pug+BLgf\nmE5jSu6FEXG7pKnAI8AQjWm6L4yI33Sv1Wo988wzTWuXXHJJR/cdEcl6q6moO/Hkk08m6532ttde\nezWt3XXXXcltPY7fXeM58n8KXBMRRwHfAK6QdBRwHbA8ImYCy4ufzWw30TL8EbEpIl4ubm8D1gIH\nAecBn1/atgg4v1tNmln5duk1v6Qh4DjgBWB6RGwqSptpvCwws93EuMMvaT/gMeCqiHh/dC0aLwzH\nfHEoaZ6kuqT6yMhIR82aWXnGFX5Jk2gE/8GI+Fmx+B1Jg0V9ENgy1rYRsTAiahFRGxgYKKNnMytB\ny/CrcTr3HmBtRPx4VGkxMFzcHgY6m3LVzHpqPG/pPRn4LvCqpJXFshuABcBPJV0KbAAu7E6L/eGp\np55qWuvmUFwv7r8TrXo7/vjjm9Zmz55dcje2K1qGPyJ+CTT7Fz6t3HbMrFd8hZ9Zphx+s0w5/GaZ\ncvjNMuXwm2XK4TfLlD+6u7Bt27ZkfcWKFT3qZGJ5/vnnm9ZOPvnk5LarVq1K1qdMmdJWT9bgI79Z\nphx+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlimP8xfWrVuXrL/88ss96qRc5557brJ++umnJ+utPrr7\njjvuSNZTj+vbb7+d3PaTTz5J1q0zPvKbZcrhN8uUw2+WKYffLFMOv1mmHH6zTDn8ZpnyOH+h1WxC\nF1xwQdPao48+2tG+J0+enKzfeOONyfrw8HDT2tSpU5Pb7rlnZ38CF110UbL+0UcfNa299tpryW33\n3Xfftnqy8fGR3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlFq9X1vSIcD9wHQggIURcbuk+cD3\ngZFi1RsiYknqvmq1WtTr9Y6bNrOx1Wo16vW6xrPueK7w+BS4JiJelrQ/8JKkZUXttoj4UbuNmll1\nWoY/IjYBm4rb2yStBQ7qdmNm1l279Jpf0hBwHPBCsehKSask3StpzLmTJM2TVJdUHxkZGWsVM6vA\nuMMvaT/gMeCqiHgfuBM4AphF45nBrWNtFxELI6IWEbVW18+bWe+MK/ySJtEI/oMR8TOAiHgnInZE\nxGfA3cCJ3WvTzMrWMvySBNwDrI2IH49aPjhqtW8Bq8tvz8y6ZTxn+08Gvgu8KmllsewGYK6kWTSG\n/9YDl3WlQzPrivGc7f8lMNa4YXJM38z6m6/wM8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5fCb\nZcrhN8uUw2+WKYffLFMOv1mmHH6zTDn8Zplq+dHdpe5MGgE2jFo0DdjaswZ2Tb/21q99gXtrV5m9\nzYiIcX1eXk/D/6WdS/WIqFXWQEK/9tavfYF7a1dVvflpv1mmHH6zTFUd/oUV7z+lX3vr177AvbWr\nkt4qfc1vZtWp+shvZhVx+M0yVUn4JZ0p6b8kvSHpuip6aEbSekmvSlopqdL5xIs5ELdIWj1q2VRJ\nyyS9Xnwfc47EinqbL2lj8ditlDSnot4OkfQLSWskvSbpz4vllT52ib4qedx6/ppf0h7AfwNnAG8B\nLwJzI2JNTxtpQtJ6oBYRlV8QIulUYDtwf0QcUyy7BXg3IhYU/3FOiYi/7JPe5gPbq562vZhNanD0\ntPLA+cDFVPjYJfq6kAoetyqO/CcCb0TEuoj4GHgYOK+CPvpeRDwHvLvT4vOARcXtRTT+eHquSW99\nISI2RcTLxe1twOfTylf62CX6qkQV4T8IeHPUz29R4QMwhgCekfSSpHlVNzOG6RGxqbi9GZheZTNj\naDltey/tNK183zx27Ux3Xzaf8PuyUyLia8BZwBXF09u+FI3XbP00Vjuuadt7ZYxp5X+ryseu3enu\ny1ZF+DcCh4z6+eBiWV+IiI3F9y3Az+m/qcff+XyG5OL7lor7+a1+mrZ9rGnl6YPHrp+mu68i/C8C\nMyUdJumrwLeBxRX08SWSJhcnYpA0Gfgm/Tf1+GJguLg9DDxeYS9f0C/TtjebVp6KH7u+m+4+Inr+\nBcyhccb/f4C/qqKHJn0dDrxSfL1WdW/AQzSeBn5C49zIpcDvAsuB14F/Bab2UW8PAK8Cq2gEbbCi\n3k6h8ZR+FbCy+JpT9WOX6KuSx82X95plyif8zDLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNM/T9b\nzAyO+W6w7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11ba947f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEHtJREFUeJzt3XusHPV5xvHnwbEJIUChPhhjDAZkKnFRnWhFwt0tNDVX\nE1VAjABDoA43tSAKRZgAqloVQRKSQoU4FGpCKSHCQZjihriIQlFp8IE6YAIFigzYGHwMRNhgLrbf\n/rHj9Nic/e3x7uzO2r/vR1qd3Xlndt5d+fHszszOzxEhAPnZpuoGAFSD8AOZIvxApgg/kCnCD2SK\n8AOZIvwZs/3vts/r9rLoDYR/K2B7ie1jqu6jEdszbT9j+wPbS23fYPsLVfeVO8KPbviSpEskjZX0\nNUlHS/qLSjsC4d+a2d7Z9r/YHrT9fnF/j01m29f208VW+UHbuwxZ/uu2/9P2b2z/yvbUVvqIiFsj\n4j8i4tOIWCbpHkmHtf7KUAbCv3XbRtI/StpL0p6S1ki6ZZN5zpL0bUnjJa2V9HeSZHuCpIcl/bWk\nXVTfUs+13bfpSmzvWfwHsecI+zpS0gub/WpQKsK/FYuIdyNibkR8FBGrJP2NpKM2me3uiFgcER9K\n+q6kU22PknSGpPkRMT8i1kfEAkkDko4bZj1vRMTvRMQbzXqy/W1JNUnfa/PloU3sdNmK2f6SpJsk\nTZO0czF5B9ujImJd8fjNIYu8Lmm06t/N95J0iu0Th9RHS3qsjX5OlvS3ko6JiJWtPg/KQfi3bpdJ\n+j1JX4uIt21PkfTfkjxknolD7u8p6TNJK1X/T+HuiPjTMhqxPU3S7ZKOj4jny3hOtIeP/VuP0ba/\nOOT2BUk7qP49/zfFjrxrh1nuDNv7F58S/krS/cWngn+SdKLtP7Y9qnjOqcPsMGzK9h+qvpPvTyLi\n6ZZfIUpF+Lce81UP+obbdZJ+KGk71bfk/yXp58Msd7ekOZLelvRFSX8mSRHxpqTpkq6SNKj6J4HL\nNcy/mWKH3+rEDr/vStpJ0vxivtW2/7WlV4nSmIt5AHliyw9kivADmSL8QKYIP5Cprh7nHzt2bEya\nNKmbqwSysmTJEq1cudLN52wz/MWJGz+SNErSP0TE9an5J02apIGBgXZWCSChVquNeN6WP/YX53//\nvaRjJe0vaYbt/Vt9PgDd1c53/oMlvRoRr0XEp5J+ovpJIQC2AO2Ef4I2/lHI0mLaRmzPsj1ge2Bw\ncLCN1QEoU8f39kdEf0TUIqLW1/e5n4IDqEg74V+mjX8RtkcxDcAWoJ3wL5Q02fbetsdI+pakeeW0\nBaDTWj7UFxFrbV8s6RHVD/XdGRFcmgnYQrR1nD8i5qv+U1IAWxhO7wUyRfiBTBF+IFOEH8gU4Qcy\nRfiBTBF+IFOEH8gU4QcyRfiBTBF+IFOEH8gU4QcyxRDdW7n169cn60uWLGnr+efMmZOsT5kypWHt\nkEMOSS47fvz4VlrCCLHlBzJF+IFMEX4gU4QfyBThBzJF+IFMEX4gUxzn3wosXLiwYe2GG25ILjt3\n7tyy29lIRDSs7brrrsllm/V2+OGHt9QT6tjyA5ki/ECmCD+QKcIPZIrwA5ki/ECmCD+QKY7z94A1\na9Yk62eccUay/sgjjzSsffTRRy31tMHxxx+frDf7zf2qVasa1u67777kstOnT0/Wly5dmqxvt912\nyXru2gq/7SWSVklaJ2ltRNTKaApA55Wx5f+DiFhZwvMA6CK+8wOZajf8IekXtp+xPWu4GWzPsj1g\ne2BwcLDN1QEoS7vhPzwivirpWEkX2T5y0xkioj8iahFR6+vra3N1AMrSVvgjYlnxd4WkByQdXEZT\nADqv5fDb3t72DhvuS/qGpMVlNQags9rZ2z9O0gO2NzzPP0fEz0vpKjPNjmc//vjjyfoFF1zQsHbC\nCScklz300EOT9W233TZZHzVqVLKeGjdg3bp1yWXvv//+ZP2WW25J1i+//PJkPXcthz8iXpP0+yX2\nAqCLONQHZIrwA5ki/ECmCD+QKcIPZIqf9PaAp556KlmfPXt2sn7NNdeU2U6pttmm8fbliiuuSC47\nf/78ZH3t2rUt9YQ6tvxApgg/kCnCD2SK8AOZIvxApgg/kCnCD2SK4/xbgP3226/qFjqiVktf7Hmf\nffbpUid5YssPZIrwA5ki/ECmCD+QKcIPZIrwA5ki/ECmOM7fA/r7+5P1xYu3zuEQnnjiiWT9pZde\n6lIneWLLD2SK8AOZIvxApgg/kCnCD2SK8AOZIvxApjjO3wNmzJhRdQuV+Oyzz5J1rsvfWU23/Lbv\ntL3C9uIh03axvcD2K8XfnTvbJoCyjeRj/xxJ0zaZdqWkRyNisqRHi8cAtiBNwx8RT0h6b5PJ0yXd\nVdy/S9LJJfcFoMNa3eE3LiKWF/ffljSu0Yy2Z9kesD0wODjY4uoAlK3tvf0REZIiUe+PiFpE1Pr6\n+tpdHYCStBr+d2yPl6Ti74ryWgLQDa2Gf56kmcX9mZIeLKcdAN3S9Di/7XslTZU01vZSSddKul7S\nT22fK+l1Sad2sklsnR544IGqW8ha0/BHRKMzUI4uuRcAXcTpvUCmCD+QKcIPZIrwA5ki/ECm+Ekv\nOuqNN95oWJszZ05bzz1lypS2ls8dW34gU4QfyBThBzJF+IFMEX4gU4QfyBThBzLFcX501HvvbXr5\nx//34YcfJpedNm3T68Zu7JhjjmmpJ9Sx5QcyRfiBTBF+IFOEH8gU4QcyRfiBTBF+IFMc50dbnnzy\nyWT9rLPOaliznVz2xhtvTNZHjx6drCONLT+QKcIPZIrwA5ki/ECmCD+QKcIPZIrwA5niOP8WYMWK\nFcn6vHnzGtZuv/325LJr1qxpqacNXn755WT9k08+aVhrdpx/7ty5yfratWuTda7rn9Z0y2/7Ttsr\nbC8eMu0628tsLypux3W2TQBlG8nH/jmShrukyk0RMaW4zS+3LQCd1jT8EfGEpMbXYgKwRWpnh9/F\ntp8rvhbs3Ggm27NsD9geGBwcbGN1AMrUavhvlbSvpCmSlkv6fqMZI6I/ImoRUevr62txdQDK1lL4\nI+KdiFgXEesl3S7p4HLbAtBpLYXf9vghD78paXGjeQH0JkdEegb7XklTJY2V9I6ka4vHUySFpCWS\nvhMRy5utrFarxcDAQFsNb4maHUu/9tprk/Wbb745WU8dS69a6lj+mDFjkss2e13Nlj/xxBMb1k44\n4YTksmeeeWayvn79+mR94cKFyfrSpUsb1k455ZTksim1Wk0DAwPpEygKTU/yiYgZw0y+Y7O7AtBT\nOL0XyBThBzJF+IFMEX4gU4QfyBQ/6S3B4sXp0xzOO++8ZP3pp59O1idPnpysH3rooQ1rU6dOTS57\nzjnnJOvN7Lbbbsn6bbfd1rC20047JZdtdunuRYsWJeupnwQ3+7nwW2+9lay///77yfodd6QPiJ12\n2mkNa+0c6tscbPmBTBF+IFOEH8gU4QcyRfiBTBF+IFOEH8gUx/lH6LHHHmtYu+iii5LLvvTSS8n6\n2Wefnaw3O969evXqhrWZM2cml21mjz32SNYffvjhZP2ggw5qed1HHnlksv7mm28m6wsWLGhYa3bu\nxezZs5P1ZucoXH311cn6ZZddlqx3A1t+IFOEH8gU4QcyRfiBTBF+IFOEH8gU4QcyxXH+EUr9/rvZ\ncfwDDzwwWd99992T9QMOOCBZ//jjjxvWPvjgg+Sye+21V7L+0EMPJevNXlsnTZw4MVlPXX672aW7\n+/v7k/WjjjoqWT/iiCOS9V7Alh/IFOEHMkX4gUwRfiBThB/IFOEHMkX4gUyNZIjuiZJ+LGmc6kNy\n90fEj2zvIuk+SZNUH6b71IhIXsx8Sx6iu6+vr2Ht3Xff7WInn7fvvvs2rDX73fj5559fdjuo0OYM\n0T2SLf9aSZdFxP6Svi7pItv7S7pS0qMRMVnSo8VjAFuIpuGPiOUR8Wxxf5WkFyVNkDRd0l3FbHdJ\nOrlTTQIo32Z957c9SdJXJP1S0riIWF6U3lb9awGALcSIw2/7y5LmSrokIjY6YTzqOw6G3Xlge5bt\nAdsDg4ODbTULoDwjCr/t0aoH/56I+Fkx+R3b44v6eEkrhls2IvojohYRtdROMwDd1TT8ti3pDkkv\nRsQPhpTmSdpwadiZkh4svz0AnTKSn/QeJulMSc/b3jAm8lWSrpf0U9vnSnpd0qmdabE3pC7FfOml\nlyaXbTZM9o477pisT5s2LVk//fTTW35u5Ktp+CPiSUmNjhseXW47ALqFM/yATBF+IFOEH8gU4Qcy\nRfiBTBF+IFNcunuELrzwwoa1k046KbnshAkTkvUxY8a01BPQDrb8QKYIP5Apwg9kivADmSL8QKYI\nP5Apwg9kiuP8IzR69OiGtb333ruLnQDlYMsPZIrwA5ki/ECmCD+QKcIPZIrwA5ki/ECmCD+QKcIP\nZIrwA5ki/ECmCD+QKcIPZIrwA5ki/ECmmobf9kTbj9n+te0XbP95Mf0628tsLypux3W+XQBlGcnF\nPNZKuiwinrW9g6RnbC8oajdFxPc61x6ATmka/ohYLml5cX+V7RclpYegAdDzNus7v+1Jkr4i6ZfF\npIttP2f7Tts7N1hmlu0B2wODg4NtNQugPCMOv+0vS5or6ZKI+EDSrZL2lTRF9U8G3x9uuYjoj4ha\nRNT6+vpKaBlAGUYUftujVQ/+PRHxM0mKiHciYl1ErJd0u6SDO9cmgLKNZG+/Jd0h6cWI+MGQ6eOH\nzPZNSYvLbw9Ap4xkb/9hks6U9LztRcW0qyTNsD1FUkhaIuk7HekQQEeMZG//k5I8TGl++e0A6BbO\n8AMyRfiBTBF+IFOEH8gU4QcyRfiBTBF+IFOEH8gU4QcyRfiBTBF+IFOEH8gU4QcyRfiBTDkiurcy\ne1DS60MmjZW0smsNbJ5e7a1X+5LorVVl9rZXRIzoenldDf/nVm4PREStsgYSerW3Xu1LordWVdUb\nH/uBTBF+IFNVh7+/4vWn9GpvvdqXRG+tqqS3Sr/zA6hO1Vt+ABUh/ECmKgm/7Wm2/8f2q7avrKKH\nRmwvsf18Mez4QMW93Gl7he3FQ6btYnuB7VeKv8OOkVhRbz0xbHtiWPlK37teG+6+69/5bY+S9LKk\nP5K0VNJCSTMi4tddbaQB20sk1SKi8hNCbB8pabWkH0fEgcW0GyS9FxHXF/9x7hwRf9kjvV0naXXV\nw7YXo0mNHzqsvKSTJZ2tCt+7RF+nqoL3rYot/8GSXo2I1yLiU0k/kTS9gj56XkQ8Iem9TSZPl3RX\ncf8u1f/xdF2D3npCRCyPiGeL+6skbRhWvtL3LtFXJaoI/wRJbw55vFQVvgHDCEm/sP2M7VlVNzOM\ncRGxvLj/tqRxVTYzjKbDtnfTJsPK98x718pw92Vjh9/nHR4RX5V0rKSLio+3PSnq39l66VjtiIZt\n75ZhhpX/rSrfu1aHuy9bFeFfJmnikMd7FNN6QkQsK/6ukPSAem/o8Xc2jJBc/F1RcT+/1UvDtg83\nrLx64L3rpeHuqwj/QkmTbe9te4ykb0maV0Efn2N7+2JHjGxvL+kb6r2hx+dJmlncnynpwQp72Uiv\nDNveaFh5Vfze9dxw9xHR9Zuk41Tf4/+/kmZX0UODvvaR9Kvi9kLVvUm6V/WPgZ+pvm/kXEm/K+lR\nSa9I+jdJu/RQb3dLel7Sc6oHbXxFvR2u+kf65yQtKm7HVf3eJfqq5H3j9F4gU+zwAzJF+IFMEX4g\nU4QfyBThBzJF+IFMEX4gU/8H0tAKNmgLMIUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11baa54a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEBRJREFUeJzt3X+MVWV+x/H3R1eji1qhjEgQnK0xTYxR1kxcf5dWu4rN\nBjZYf2W31IXiH04sKo3GqqDWFM2udruumrFSWbBuNyrRWKqCItY03WVUFJS0KhnWmYAMuiLWiot+\n+8c9bEac+9zx/obn80pu5sz5nuee79zw4dx7zr33UURgZvnZr9UNmFlrOPxmmXL4zTLl8JtlyuE3\ny5TDb5Yphz9jkp6XNLvZY609OPz7AEl9ks5pdR/lSJop6SVJH0rql3SHpK+1uq/cOfzWDF8H5gJj\ngW8BZwPzWtqROfz7MkmjJT0paVDSb4rlo/bY7BhJvyqOyo9LGjNk/CmS/lPSB5JelTSlmj4i4t6I\n+I+I+DQiBoCHgNOr/8usHhz+fdt+wD8DRwOTgP8D7t5jm78AfgCMB3YB/wggaQLwb8DfAWMoHakf\nldSx504kTSr+g5g0wr7OAl7/yn+N1ZXDvw+LiPci4tGI+DgidgC3AX+0x2ZLImJ9RPwvcCNwoaT9\nge8ByyNieUR8HhErgF7g/GH28+uIODwifl2pJ0k/ALqAH9b451mNfNJlHybp68BdwHnA6GL1oZL2\nj4jPit/fGTJkE3AApdfmRwN/Luk7Q+oHAKtq6Gc68PfAORGxrdr7sfpw+Pdt1wB/CHwrIrZImgy8\nAmjINhOHLE8Cfgtso/SfwpKI+Kt6NCLpPOB+4M8iYl097tNq46f9+44DJB005PY14FBKr/M/KE7k\nzR9m3PckHVc8S7gFeKR4VrAU+I6kcyXtX9znlGFOGFYk6U8oneSbERG/qvovtLpy+PcdyykFffdt\nAfAPwMGUjuT/BTw1zLglwIPAFuAg4EqAiHgHmAZcDwxSeibwNwzzb6Y44fdR4oTfjcDvAcuL7T6S\n9O9V/ZVWN/KXeZjlyUd+s0w5/GaZcvjNMuXwm2Wqqdf5x44dG52dnc3cpVlW+vr62LZtmypvWWP4\nizdu/BjYH/iniFiY2r6zs5Pe3t5admlmCV1dXSPetuqn/cX7v38KTAWOAy6RdFy192dmzVXLa/6T\ngbciYmNEfAr8nNKbQsxsL1BL+CfwxQ+F9BfrvkDSHEm9knoHBwdr2J2Z1VPDz/ZHRE9EdEVEV0fH\nlz4KbmYtUkv4B/jiJ8KOKtaZ2V6glvCvAY6V9A1JBwIXA0/Upy0za7SqL/VFxC5J3cDTlC71LYoI\nfzWT2V6ipuv8EbGc0kdJzWwv47f3mmXK4TfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2XK\n4TfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlimH3yxTDr9Zphx+s0w5/GaZ\ncvjNMuXwm2XK4TfLlMNvlimH3yxTDr9ZpmqapdeskXbu3JmsP/XUU8n6aaedVrbW0dFRVU/7kprC\nL6kP2AF8BuyKiK56NGVmjVePI/8fR8S2OtyPmTWRX/ObZarW8AfwjKSXJM0ZbgNJcyT1SuodHBys\ncXdmVi+1hv+MiDgJmApcIemsPTeIiJ6I6IqILp9kMWsfNYU/IgaKn1uBZcDJ9WjKzBqv6vBLGiXp\n0N3LwLeB9fVqzMwaq5az/eOAZZJ238+/RET6wqtl58UXXyxbu/3225NjV65cmax/8sknyfrll19e\ntnbfffclxy5btixZP/XUU5P1I488MllvB1WHPyI2AifWsRczayJf6jPLlMNvlimH3yxTDr9Zphx+\ns0z5I71Wk3Xr1iXrV155Zdnahg0bkmNnzZqVrH/44YfJ+qJFi8rWNm3alBxb6TJj6uPCAKtXr07W\n24GP/GaZcvjNMuXwm2XK4TfLlMNvlimH3yxTDr9Zpnyd35I+/vjjZH3evHnJ+iuvvFK2Nnv27OTY\nu+++O1l/9dVXk/UVK1aUrVX62u/io+plnX322cn63sBHfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yp\nh98sU77Ob0k/+clPkvVnnnkmWe/u7i5bW7hwYXLs0qVLa9r3li1bkvWUGTNmJOs33XRT1ffdLnzk\nN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5ev8mUtNoQ1w2223JevTp09P1i+77LKytUrX0p9+\n+ulkvRannHJKst7T09OwfbeLikd+SYskbZW0fsi6MZJWSHqz+Dm6sW2aWb2N5Gn/g8B5e6y7Dng2\nIo4Fni1+N7O9SMXwR8QLwPt7rJ4GLC6WFwPp535m1naqPeE3LiI2F8tbgHHlNpQ0R1KvpN7BwcEq\nd2dm9Vbz2f6ICCAS9Z6I6IqIro6Ojlp3Z2Z1Um3435U0HqD4ubV+LZlZM1Qb/ieAmcXyTODx+rRj\nZs1S8Tq/pIeBKcBYSf3AfGAh8AtJs4BNwIWNbNIaZ8OGDcn6jh07kvVVq1Yl688//3zZ2gcffJAc\n20gHH3xwsj569L5/9bpi+CPikjKlvX/WArOM+e29Zply+M0y5fCbZcrhN8uUw2+WKX+k12qyffv2\nqsceccQRyfqCBQuS9dRlRIBHHnmkbO3MM89Mjs2Bj/xmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/\nWaZ8nT9zO3fuTNYPO+ywZP3cc89N1i+44IKytSlTpiTHbty4MVm/4447kvVUbzfffHNybA585DfL\nlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXr/Jnr7u5O1qdNm5asT5w4sep9V/ra70svvTRZ37o1\nPVfMwoULv3JPOfGR3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlK/zW1It1/EBBgYGytbmzp2b\nHLtly5Zkffbs2cn6RRddlKznruKRX9IiSVslrR+yboGkAUlri9v5jW3TzOptJE/7HwTOG2b9XREx\nubgtr29bZtZoFcMfES8A7zehFzNrolpO+HVLeq14WTC63EaS5kjqldQ7ODhYw+7MrJ6qDf+9wDHA\nZGAz8KNyG0ZET0R0RURXR0dHlbszs3qrKvwR8W5EfBYRnwP3AyfXty0za7Sqwi9p/JBfvwusL7et\nmbWnitf5JT0MTAHGSuoH5gNTJE0GAugDLm9gj21h27ZtZWu1zBMP8Pbbb1fT0ogcfvjhyfrVV1+d\nrFe6zn/nnXcm6ytWrChb6+/vT4498cQTa9q3pVUMf0RcMszqBxrQi5k1kd/ea5Yph98sUw6/WaYc\nfrNMOfxmmfJHeguPPfZYsp76Gug1a9bUu52mWblyZatbKOuNN95I1u+5555k/dprr61nO/scH/nN\nMuXwm2XK4TfLlMNvlimH3yxTDr9Zphx+s0xlc51//vz5yfott9xS9X1PnTo1WZ83b16yXul9Au+9\n916yvn379rK1np6e5NhajRo1qqZ6LdauXduw+86Bj/xmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/\nWaayuc7/5JNPJusTJkxI1m+99daytRkzZiTH7tq1K1nv6+tL1levXp2sr1q1KlmvRaVpri+++OJk\nffr06fVsx+rIR36zTDn8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMjmaJ7IvAzYBylKbl7IuLHksYA\n/wp0Upqm+8KI+E3jWq3N4OBgsj4wMJCsL126tGztxhtvrOm+a3XQQQeVrU2aNCk59qqrrkrWZ8+e\nnawfcsghybq1r5Ec+XcB10TEccApwBWSjgOuA56NiGOBZ4vfzWwvUTH8EbE5Il4ulncAG4AJwDRg\ncbHZYsBv5TLbi3yl1/ySOoFvAr8ExkXE5qK0hdLLAjPbS4w4/JIOAR4F5kbEh0NrERGUzgcMN26O\npF5JvZVed5tZ84wo/JIOoBT8hyJi94yW70oaX9THA1uHGxsRPRHRFRFdHR0d9ejZzOqgYvglCXgA\n2BARdw4pPQHMLJZnAo/Xvz0za5SRfKT3dOD7wDpJu78r+XpgIfALSbOATcCFjWmxPs4555xkfcmS\nJcn6c889V/W+DzzwwGR9v/3S/wdPmzYtWb/hhhvK1o4//vjkWMtXxfBHxIuAypTPrm87ZtYsfoef\nWaYcfrNMOfxmmXL4zTLl8JtlyuE3y1Q2X929aNGiZL27uztZ7+/vr3rfJ5xwQrLe2dlZ9X2bVctH\nfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sU9lc56/kpJNOqqlutrfxkd8sUw6/WaYcfrNMOfxm\nmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y1TF8EuaKGmVpDck\nvS7pr4v1CyQNSFpb3M5vfLtmVi8j+TKPXcA1EfGypEOBlyStKGp3RcQPG9eemTVKxfBHxGZgc7G8\nQ9IGYEKjGzOzxvpKr/kldQLfBH5ZrOqW9JqkRZJGlxkzR1KvpN7BwcGamjWz+hlx+CUdAjwKzI2I\nD4F7gWOAyZSeGfxouHER0RMRXRHR1dHRUYeWzaweRhR+SQdQCv5DEfEYQES8GxGfRcTnwP3AyY1r\n08zqbSRn+wU8AGyIiDuHrB8/ZLPvAuvr356ZNcpIzvafDnwfWCdpbbHueuASSZOBAPqAyxvSoZk1\nxEjO9r8IaJjS8vq3Y2bN4nf4mWXK4TfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2XK4TfL\nlMNvlimH3yxTDr9Zphx+s0wpIpq3M2kQ2DRk1VhgW9Ma+Gratbd27QvcW7Xq2dvRETGi78travi/\ntHOpNyK6WtZAQrv21q59gXurVqt689N+s0w5/GaZanX4e1q8/5R27a1d+wL3Vq2W9NbS1/xm1jqt\nPvKbWYs4/GaZakn4JZ0n6b8lvSXpulb0UI6kPknrimnHe1vcyyJJWyWtH7JujKQVkt4sfg47R2KL\nemuLadsT08q39LFrt+num/6aX9L+wP8Afwr0A2uASyLijaY2UoakPqArIlr+hhBJZwEfAT+LiOOL\ndXcA70fEwuI/ztERcW2b9LYA+KjV07YXs0mNHzqtPDAd+Eta+Ngl+rqQFjxurTjynwy8FREbI+JT\n4OfAtBb00fYi4gXg/T1WTwMWF8uLKf3jaboyvbWFiNgcES8XyzuA3dPKt/SxS/TVEq0I/wTgnSG/\n99PCB2AYATwj6SVJc1rdzDDGRcTmYnkLMK6VzQyj4rTtzbTHtPJt89hVM919vfmE35edEREnAVOB\nK4qnt20pSq/Z2ula7YimbW+WYaaV/51WPnbVTndfb60I/wAwccjvRxXr2kJEDBQ/twLLaL+px9/d\nPUNy8XNri/v5nXaatn24aeVpg8eunaa7b0X41wDHSvqGpAOBi4EnWtDHl0gaVZyIQdIo4Nu039Tj\nTwAzi+WZwOMt7OUL2mXa9nLTytPix67tpruPiKbfgPMpnfF/G/jbVvRQpq8/AF4tbq+3ujfgYUpP\nA39L6dzILOD3gWeBN4GVwJg26m0JsA54jVLQxreotzMoPaV/DVhb3M5v9WOX6Kslj5vf3muWKZ/w\nM8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y9f/CuvGALEBUDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b922f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEAxJREFUeJzt3W2MXOV5xvHr4iUKEKu18cqybGOnsKqELHCiEUQyooBT\nCkYRjgoWxqSuoTgfjACLVryU8FJR1UJJaFALkk2IHZc6DSWA1bphXUgFCEg8YGLsQAtGJtgs3jUm\nYOqCA777YQ7R2t45s8ycmTO7z/8njXb23HPmuXfky2fmPDPzOCIEID1HlN0AgHIQfiBRhB9IFOEH\nEkX4gUQRfiBRhD9htv/L9l90el90B8I/BtjebvurZfdRj+1Ftp+3/b7tHbbvtH1U2X2ljvCjE46V\ndK2kiZJOlzRH0l+W2hEI/1hme7ztf7M9aPvd7PrUQ252ou1fZEflR21PGLL/V2w/Y/s3tn9p+6xm\n+oiIeyPiqYjYHxE7JT0gaXbzfxmKQPjHtiMk/UDSdEknSPo/Sf9wyG3+TNLlkiZL+ljS3ZJke4qk\nf5d0h6QJqh2pH7Ldc+ggtk/I/oM4YYR9nSlp62f+a1Aowj+GRcQ7EfFQROyLiL2S/lbSHx1yszUR\nsSUi/lfStyTNt32kpMskrY+I9RFxICI2SKpKmjvMOL+OiN+PiF836sn25ZIqkr7d4p+HFnHSZQyz\nfaykuySdJ2l8tnmc7SMj4pPs9zeH7PKGpKNVe20+XdLFtr82pH60pJ+10M88SX8n6asRsbvZ+0Ex\nCP/Ydp2kP5R0ekS8bXuWpE2SPOQ204ZcP0HSbyXtVu0/hTURcWURjdg+T9JKSRdExEtF3Cdaw9P+\nseNo258fcjlK0jjVXuf/JjuRd+sw+11m++TsWcLfSPrX7FnBP0n6mu0/sX1kdp9nDXPCsCHb56h2\nku9PI+IXTf+FKBThHzvWqxb0Ty+3Sfp7SceodiR/TtJPh9lvjaRVkt6W9HlJV0tSRLwp6UJJN0ka\nVO2ZwF9pmH8z2Qm/D3JO+H1L0u9JWp/d7gPb/9HUX4nCmC/zANLEkR9IFOEHEkX4gUQRfiBRHZ3n\nnzhxYsyYMaOTQwJJ2b59u3bv3u3Gt2wx/NkbN74n6UhJ90XE8rzbz5gxQ9VqtZUhAeSoVCojvm3T\nT/uz93//o6TzJZ0saYHtk5u9PwCd1cpr/tMkvRYRr0fEfkk/Uu1NIQBGgVbCP0UHfyhkR7btILaX\n2K7arg4ODrYwHIAitf1sf0SsiIhKRFR6eg77KDiAkrQS/p06+BNhU7NtAEaBVsK/UVKv7S/a/pyk\nSyStK6YtAO3W9FRfRHxs+ypJj6k21Xd/RPDVTMAo0dI8f0SsV+2jpABGGd7eCySK8AOJIvxAogg/\nkCjCDySK8AOJIvxAogg/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJIvxA\nogg/kCjCDySqo0t0j2b9/f11a4888khbx37nnXdy67fccktbx88TEbl1u/5q0YsXL87dd8mSJbn1\n008/PbeOfBz5gUQRfiBRhB9IFOEHEkX4gUQRfiBRhB9IFPP8I3T55ZfXrfX19XWwk8PlzaWXLa+3\nVatW5e67fn3+AtBz5szJrd999911axMmTMjdNwUthd/2dkl7JX0i6eOIqBTRFID2K+LIf3ZE7C7g\nfgB0EK/5gUS1Gv6Q1Gf7edvDvhHb9hLbVdvVwcHBFocDUJRWw39GRHxZ0vmSlto+89AbRMSKiKhE\nRKWnp6fF4QAUpaXwR8TO7OeApIclnVZEUwDar+nw2z7O9rhPr0s6V9KWohoD0F6tnO2fJOnhbB73\nKEn/HBE/LaSrLpT3eX60x8DAQG597dq1ufVLL720bm3u3LlN9TSWNB3+iHhd0qkF9gKgg5jqAxJF\n+IFEEX4gUYQfSBThBxLFR3pHKO8jvffdd1/uvlu3bs2t9/b25tYvuuii3HqZDhw4kFt/66236tbW\nrFlTdDsH2bBhQ90aU30c+YFkEX4gUYQfSBThBxJF+IFEEX4gUYQfSBTz/CN09dVX160tXLgwd999\n+/bl1o855pjc+sSJE3Pr3ezpp5+uW2v3PP+2bdvaev+jHUd+IFGEH0gU4QcSRfiBRBF+IFGEH0gU\n4QcSxTx/AY4//viW6qPZ3r17c+t5y2S36thjj82tL1u2rG1jjwUc+YFEEX4gUYQfSBThBxJF+IFE\nEX4gUYQfSBTz/GjJ/Pnzc+t9fX1tG/ucc87JrZ999tltG3ssaHjkt32/7QHbW4Zsm2B7g+1Xs5/j\n29smgKKN5Gn/KknnHbLtBkmPR0SvpMez3wGMIg3DHxFPStpzyOYLJa3Orq+WNK/gvgC0WbMn/CZF\nRH92/W1Jk+rd0PYS21Xb1cHBwSaHA1C0ls/2R0RIipz6ioioRESlp6en1eEAFKTZ8O+yPVmSsp8D\nxbUEoBOaDf86SYuy64skPVpMOwA6peE8v+21ks6SNNH2Dkm3Slou6ce2r5D0hqT8yV6MWhs3bsyt\nP/bYY7l1202PPXPmzNz6Pffc0/R9YwThj4gFdUpzCu4FQAfx9l4gUYQfSBThBxJF+IFEEX4gUXyk\nN3FPPfVUbn3BgnqTPa0bN25cbn3dunW59SlTphTZTnI48gOJIvxAogg/kCjCDySK8AOJIvxAogg/\nkCjm+RN3880359b7+/tz64309vbWrV1//fW5+06fPr2lsZGPIz+QKMIPJIrwA4ki/ECiCD+QKMIP\nJIrwA4linn8MeO655+rWGi1j/eGHH7Y0dm3BpvouvvjiurXFixe3NDZaw5EfSBThBxJF+IFEEX4g\nUYQfSBThBxJF+IFEMc8/CjT6bv358+uvkP7RRx/l7tvKEtqStHnz5tz6SSed1NL9o30aHvlt3297\nwPaWIdtus73T9ovZZW572wRQtJE87V8l6bxhtt8VEbOyy/pi2wLQbg3DHxFPStrTgV4AdFArJ/yu\nsr05e1kwvt6NbC+xXbVdHRwcbGE4AEVqNvz3SjpR0ixJ/ZK+U++GEbEiIioRUenp6WlyOABFayr8\nEbErIj6JiAOSVko6rdi2ALRbU+G3PXnIr1+XtKXebQF0p4bz/LbXSjpL0kTbOyTdKuks27MkhaTt\nkr7Zxh7HvE2bNuXWFy5cmFsfGBhoeuyjjsr/J3DllVfm1mfOnNn02ChXw/BHxIJhNn+/Db0A6CDe\n3gskivADiSL8QKIIP5Aowg8kio/0dsCzzz6bW1+5cmVufefOnUW2c5Bly5bl1pcvX962sdvtwQcf\nrFvbtm1bS/d92WWX5danTp3a0v13Akd+IFGEH0gU4QcSRfiBRBF+IFGEH0gU4QcSxTx/Ad5///3c\n+gUXXJBbf++994ps5yDz5s3Lrd9+++1tG1uS9uyp//WPzzzzTEv3vXXr1tz6HXfcUbe2b9++lsbu\n6+vLrT/xxBMt3X8ncOQHEkX4gUQRfiBRhB9IFOEHEkX4gUQRfiBRzPMXICJy6+2cx2/kkksuya3v\n2rWrpftfunRpbv3dd9+tW2v0PQfdLO/vGi048gOJIvxAogg/kCjCDySK8AOJIvxAogg/kKiRLNE9\nTdIPJU1SbUnuFRHxPdsTJP2LpBmqLdM9PyJG/+TnGNNonr9Vjd7jYLut46N5Iznyfyzpuog4WdJX\nJC21fbKkGyQ9HhG9kh7PfgcwSjQMf0T0R8QL2fW9kl6WNEXShZJWZzdbLSn/K2MAdJXP9Jrf9gxJ\nX5L0c0mTIqI/K72t2ssCAKPEiMNv+wuSHpJ0bUQc9KV1UXvhN+yLP9tLbFdtVwcHB1tqFkBxRhR+\n20erFvwHIuIn2eZdtidn9cmSBobbNyJWREQlIio9PT1F9AygAA3D79rp2u9LejkivjuktE7Souz6\nIkmPFt8egHYZyUd6Z0v6hqSXbL+YbbtJ0nJJP7Z9haQ3JM1vT4tI1cyZM3Prp556aoc6OdxoWIK7\nkYbhj4inJdWbrJ1TbDsAOoV3+AGJIvxAogg/kCjCDySK8AOJIvxAovjq7gIccUT+/6GnnHJKbv2V\nV17Jre/fv/8z91SURvPZ48ePz6339vbWrV1zzTW5+06bNi23Pn369Nw68nHkBxJF+IFEEX4gUYQf\nSBThBxJF+IFEEX4gUczzF2DcuHG59U2bNuXWb7zxxtz6nXfemVufN6/+d6eee+65ufs2Mnv27Nx6\no8/co3tx5AcSRfiBRBF+IFGEH0gU4QcSRfiBRBF+IFFutMRykSqVSlSr1Y6NB6SmUqmoWq2OaF10\njvxAogg/kCjCDySK8AOJIvxAogg/kCjCDySqYfhtT7P9M9u/sr3V9jXZ9tts77T9YnaZ2/52ARRl\nJF/m8bGk6yLiBdvjJD1ve0NWuysivt2+9gC0S8PwR0S/pP7s+l7bL0ua0u7GALTXZ3rNb3uGpC9J\n+nm26Srbm23fb3vYdZtsL7FdtV0dHBxsqVkAxRlx+G1/QdJDkq6NiPcl3SvpREmzVHtm8J3h9ouI\nFRFRiYhKT09PAS0DKMKIwm/7aNWC/0BE/ESSImJXRHwSEQckrZR0WvvaBFC0kZztt6TvS3o5Ir47\nZPvkITf7uqQtxbcHoF1GcrZ/tqRvSHrJ9ovZtpskLbA9S1JI2i7pm23pEEBbjORs/9OShvt88Pri\n2wHQKbzDD0gU4QcSRfiBRBF+IFGEH0gU4QcSRfiBRBF+IFGEH0gU4QcSRfiBRBF+IFGEH0gU4QcS\n1dElum0PSnpjyKaJknZ3rIHPplt769a+JHprVpG9TY+IEX1fXkfDf9jgdjUiKqU1kKNbe+vWviR6\na1ZZvfG0H0gU4QcSVXb4V5Q8fp5u7a1b+5LorVml9Fbqa34A5Sn7yA+gJIQfSFQp4bd9nu3/tv2a\n7RvK6KEe29ttv5QtO14tuZf7bQ/Y3jJk2wTbG2y/mv0cdo3EknrrimXbc5aVL/Wx67bl7jv+mt/2\nkZL+R9IfS9ohaaOkBRHxq442Uoft7ZIqEVH6G0JsnynpA0k/jIiZ2bY7Je2JiOXZf5zjI+L6Lunt\nNkkflL1se7aa1OShy8pLmifpz1XiY5fT13yV8LiVceQ/TdJrEfF6ROyX9CNJF5bQR9eLiCcl7Tlk\n84WSVmfXV6v2j6fj6vTWFSKiPyJeyK7vlfTpsvKlPnY5fZWijPBPkfTmkN93qMQHYBghqc/287aX\nlN3MMCZFRH92/W1Jk8psZhgNl23vpEOWle+ax66Z5e6Lxgm/w50REV+WdL6kpdnT264Utdds3TRX\nO6Jl2ztlmGXlf6fMx67Z5e6LVkb4d0qaNuT3qdm2rhARO7OfA5IeVvctPb7r0xWSs58DJffzO920\nbPtwy8qrCx67blruvozwb5TUa/uLtj8n6RJJ60ro4zC2j8tOxMj2cZLOVfctPb5O0qLs+iJJj5bY\ny0G6Zdn2esvKq+THruuWu4+Ijl8kzVXtjP82SX9dRg91+voDSb/MLlvL7k3SWtWeBv5WtXMjV0g6\nXtLjkl6V9J+SJnRRb2skvSRps2pBm1xSb2eo9pR+s6QXs8vcsh+7nL5Kedx4ey+QKE74AYki/ECi\nCD+QKMIPJIrwA4ki/ECiCD+QqP8HTOvZNI5NafAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b82f780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD+ZJREFUeJzt3X2MVXV+x/HPZ9UVeUgLZSAoD7MS08Q0kTUTnQS1iusK\nNquupUbjbmm0Hf/Q+BDbYGw2mEYjWXfX0rTR4BMsKGp0jaRSu2rW4KbpymhVYE1BzaAgDkPd1ZFW\ncODbP+7RjDj33OHec++5w+/9Sm7mzPmeh+/c8OHce8659+eIEID0fKPsBgCUg/ADiSL8QKIIP5Ao\nwg8kivADiSL8CbP9ku2/bvW6aA+E/yhgu8/2d8ruoxrbS2y/avsT2ztt/9j2sWX3lTrCj1YYL+km\nSVMlnSnpfEl/W2pHIPxHM9uTbf+r7QHbv8umZx622Fzbr2RH5WdsTxm2frft/7D9e9tv2D63nj4i\n4t6IeDkiDkTELkmPSJpf/1+GIhD+o9s3JD0saY6k2ZL+T9I/H7bMX0q6WtIMSUOS/kmSbJ8k6VlJ\nd0iaosqR+inbHYfvxPbs7D+I2aPs6xxJW4/4r0GhCP9RLCL+JyKeioj/jYhBSXdK+tPDFlsTEVsi\nYp+kH0m63PYxkn4gaUNEbIiIQxHxvKReSReNsJ/3IuIPI+K9Wj3ZvlpSl6SfNPjnoUGcdDmK2R4v\n6R5JCyVNzmZPsn1MRBzMfn9/2Co7JB2nynvzOZL+wvb3htWPk/SrBvq5VNJdkr4TEXvr3Q6KQfiP\nbrdI+mNJZ0bEh7bnSfovSR62zKxh07MlfS5pryr/KayJiL8pohHbCyXdL+nPImJzEdtEY3jZf/Q4\nzva4YY9jJU1S5X3+77MTectGWO8Htk/NXiX8g6Qns1cFayV9z/aFto/JtnnuCCcMa7K9QJWTfH8e\nEa/U/ReiUIT/6LFBlaB/8bhd0j9KOkGVI/l/SnpuhPXWSFol6UNJ4yTdIEkR8b6kSyTdJmlAlVcC\nf6cR/s1kJ/w+zTnh9yNJfyBpQ7bcp7b/ra6/EoUxX+YBpIkjP5Aowg8kivADiSL8QKJaep1/6tSp\n0dnZ2cpdAknp6+vT3r17XXvJBsOf3bixQtIxkh6IiOV5y3d2dqq3t7eRXQLI0dXVNepl637Zn93/\n/S+SFkk6VdKVtk+td3sAWquR9/xnSHo7It6NiAOSHlPlphAAY0Aj4T9JX/1QyM5s3lfY7rHda7t3\nYGCggd0BKFLTz/ZHxMqI6IqIro6Or30UHEBJGgn/Ln31E2Ezs3kAxoBGwr9J0im2v2X7m5KukLS+\nmLYANFvdl/oiYsj29ZL+XZVLfQ9FBF/NBIwRDV3nj4gNqnyUFMAYw+29QKIIP5Aowg8kivADiSL8\nQKIIP5Aowg8kivADiSL8QKIIP5Aowg8kivADiSL8QKIYovso9/HHH+fWH3744dz6G2+8kVtftWrV\nkbb0pXHjxuXWd+zYkVufNm1a3fsGR34gWYQfSBThBxJF+IFEEX4gUYQfSBThBxLFdf4xYNu2bbn1\nJUuWVK319fXlrtvf319PS6NmVx8tev/+/bnrbty4Mbe+ePHiunpCBUd+IFGEH0gU4QcSRfiBRBF+\nIFGEH0gU4QcSxXX+FnjnnXdy68uXL8+tP/bYY7n1ffv2HXFPXzjzzDNz67Wupc+fPz+3vnTp0qq1\nl19+OXddNFdD4bfdJ2lQ0kFJQxHRVURTAJqviCP/eRGxt4DtAGgh3vMDiWo0/CHpl7Zftd0z0gK2\ne2z32u4dGBhocHcAitJo+M+KiNMlLZJ0ne1zDl8gIlZGRFdEdHV0dDS4OwBFaSj8EbEr+7lH0tOS\nziiiKQDNV3f4bU+wPemLaUnflbSlqMYANFcjZ/unS3o6+7z2sZIejYjnCulqjBkaGsqtX3XVVbn1\nV155Jbc+d+7c3PqiRYuq1s4552vvxL7iwgsvzK1PmjQpt15L3v5r/d3d3d0N7Rv56g5/RLwr6bQC\newHQQlzqAxJF+IFEEX4gUYQfSBThBxLFR3oLcOyx+U/jCy+8kFs/cOBAbr3WUNbjx4/PrTfT4OBg\nbn3NmjVVa7Wet5kzZ9bVE0aHIz+QKMIPJIrwA4ki/ECiCD+QKMIPJIrwA4niOn8LTJw4sewWmmbT\npk259ffee69qbcKECUW3gyPAkR9IFOEHEkX4gUQRfiBRhB9IFOEHEkX4gURxnR+lmTNnTtktJI0j\nP5Aowg8kivADiSL8QKIIP5Aowg8kivADieI6PxqyYsWKutddvHhxgZ3gSNU88tt+yPYe21uGzZti\n+3nb27Ofk5vbJoCijeZl/ypJCw+bd6ukFyPiFEkvZr8DGENqhj8iNkr66LDZl0hanU2vlnRpwX0B\naLJ6T/hNj4jd2fSHkqZXW9B2j+1e270DAwN17g5A0Ro+2x8RISly6isjoisiujo6OhrdHYCC1Bv+\nftszJCn7uae4lgC0Qr3hXy9pSTa9RNIzxbQDoFVqXue3vU7SuZKm2t4paZmk5ZKesH2NpB2SLm9m\nkyjP2rVrc+vr16/PrZ988slVa0uXLq2rJxSjZvgj4soqpfML7gVAC3F7L5Aowg8kivADiSL8QKII\nP5AoPtKbuH379uXWly9f3tD2b7jhhqq1E044oaFtozEc+YFEEX4gUYQfSBThBxJF+IFEEX4gUYQf\nSBTX+RN366353726devW3PqCBQty6zfeeOMR94TW4MgPJIrwA4ki/ECiCD+QKMIPJIrwA4ki/ECi\nuM5/lHvggQdy6ytXrsyt286t33nnnUfcE9oDR34gUYQfSBThBxJF+IFEEX4gUYQfSBThBxLFdf6j\nwNDQUNXao48+mrvu559/nlu/+OKLc+vd3d25dbSvmkd+2w/Z3mN7y7B5t9veZfv17HFRc9sEULTR\nvOxfJWnhCPPviYh52WNDsW0BaLaa4Y+IjZI+akEvAFqokRN+19t+M3tbMLnaQrZ7bPfa7h0YGGhg\ndwCKVG/475U0V9I8Sbsl/bTaghGxMiK6IqKro6Ojzt0BKFpd4Y+I/og4GBGHJN0v6Yxi2wLQbHWF\n3/aMYb9+X9KWassCaE81r/PbXifpXElTbe+UtEzSubbnSQpJfZKubWKPqOHmm2+uWnvppZdy1z3t\ntNNy67U+74+xq2b4I+LKEWY/2IReALQQt/cCiSL8QKIIP5Aowg8kivADieIjvWPA1VdfnVtft25d\n1drEiRNz173rrrty69OmTcutY+ziyA8kivADiSL8QKIIP5Aowg8kivADiSL8QKK4zt8G7rjjjtz6\nE088Ufe2165dm1tfuHCk72Ztjc8++yy3vm/fvtx63v0NUu2vJc/T09OTW58wYULd224XHPmBRBF+\nIFGEH0gU4QcSRfiBRBF+IFGEH0gU1/lb4IMPPsit33333bn1gwcP5tYff/zxqrVaQ2w3qr+/P7e+\nYsWKqrVnn302d93NmzfX1VMRxo8fn1u/9tqx/231HPmBRBF+IFGEH0gU4QcSRfiBRBF+IFGEH0jU\naIboniXp55KmqzIk98qIWGF7iqTHJXWqMkz35RHxu+a12r4OHDiQW1+wYEFufXBwMLc+bty43Ppz\nzz1XtbZs2bLcdbdv355br6XWPQj79+9vaPt5Ojs7c+snnnhi1drs2bNz1z377LPraWlMGc2Rf0jS\nLRFxqqRuSdfZPlXSrZJejIhTJL2Y/Q5gjKgZ/ojYHRGvZdODkt6SdJKkSyStzhZbLenSZjUJoHhH\n9J7fdqekb0v6jaTpEbE7K32oytsCAGPEqMNve6KkpyTdFBGfDK9FRKhyPmCk9Xps99ruHRgYaKhZ\nAMUZVfhtH6dK8B+JiF9ks/ttz8jqMyTtGWndiFgZEV0R0dXR0VFEzwAKUDP8ti3pQUlvRcTPhpXW\nS1qSTS+R9Ezx7QFoltF8pHe+pB9K2mz79WzebZKWS3rC9jWSdki6vDkttr8nn3wyt75t27aGtl/r\nK67vu+++hrbfiFmzZuXWTz/99Kq1yy67LHfd7u7u3PrUqVNz65MnT86tp65m+CPi15JcpXx+se0A\naBXu8AMSRfiBRBF+IFGEH0gU4QcSRfiBRPHV3QVo9M7FK664Ird+/PHH173tWr0tWrSo7m1L0nnn\nndfQ+igPR34gUYQfSBThBxJF+IFEEX4gUYQfSBThBxLFdf4CXHDBBbn1Q4cOtagTYPQ48gOJIvxA\nogg/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJIvxAogg/kKia4bc9y/av\nbP/W9lbbN2bzb7e9y/br2eOi5rcLoCij+TKPIUm3RMRrtidJetX281ntnoj4SfPaA9AsNcMfEbsl\n7c6mB22/JemkZjcGoLmO6D2/7U5J35b0m2zW9bbftP2Q7clV1umx3Wu7d2BgoKFmARRn1OG3PVHS\nU5JuiohPJN0raa6keaq8MvjpSOtFxMqI6IqIrkbHtANQnFGF3/ZxqgT/kYj4hSRFRH9EHIyIQ5Lu\nl3RG89oEULTRnO23pAclvRURPxs2f8awxb4vaUvx7QFoltGc7Z8v6YeSNtt+PZt3m6Qrbc+TFJL6\nJF3blA4BNMVozvb/WpJHKG0ovh0ArcIdfkCiCD+QKMIPJIrwA4ki/ECiCD+QKMIPJIrwA4ki/ECi\nCD+QKMIPJIrwA4ki/ECiCD+QKEdE63ZmD0jaMWzWVEl7W9bAkWnX3tq1L4ne6lVkb3MiYlTfl9fS\n8H9t53ZvRHSV1kCOdu2tXfuS6K1eZfXGy34gUYQfSFTZ4V9Z8v7ztGtv7dqXRG/1KqW3Ut/zAyhP\n2Ud+ACUh/ECiSgm/7YW2/9v227ZvLaOHamz32d6cDTveW3IvD9neY3vLsHlTbD9ve3v2c8QxEkvq\nrS2Gbc8ZVr7U567dhrtv+Xt+28dI2ibpAkk7JW2SdGVE/LaljVRhu09SV0SUfkOI7XMkfSrp5xHx\nJ9m8H0v6KCKWZ/9xTo6IpW3S2+2SPi172PZsNKkZw4eVl3SppL9Sic9dTl+Xq4TnrYwj/xmS3o6I\ndyPigKTHJF1SQh9tLyI2SvrosNmXSFqdTa9W5R9Py1XprS1ExO6IeC2bHpT0xbDypT53OX2Voozw\nnyTp/WG/71SJT8AIQtIvbb9qu6fsZkYwPSJ2Z9MfSppeZjMjqDlseysdNqx82zx39Qx3XzRO+H3d\nWRFxuqRFkq7LXt62pai8Z2una7WjGra9VUYYVv5LZT539Q53X7Qywr9L0qxhv8/M5rWFiNiV/dwj\n6Wm139Dj/V+MkJz93FNyP19qp2HbRxpWXm3w3LXTcPdlhH+TpFNsf8v2NyVdIWl9CX18je0J2YkY\n2Z4g6btqv6HH10takk0vkfRMib18RbsM215tWHmV/Ny13XD3EdHyh6SLVDnj/46kvy+jhyp9nSzp\njeyxtezeJK1T5WXg56qcG7lG0h9JelHSdkkvSJrSRr2tkbRZ0puqBG1GSb2dpcpL+jclvZ49Lir7\nucvpq5Tnjdt7gURxwg9IFOEHEkX4gUQRfiBRhB9IFOEHEkX4gUT9PzuYxZKVvu7WAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b825470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEH9JREFUeJzt3XuMXOV9xvHvE8fBhZjay66MuTpFXF0oiQaDFIMpuKlx\niSBCtcLF3RKoQYAoglYgUACVFlAIoUatABuMCbc0qgO2qEkxKAiiQsKYclkCFIrsYGPsNVdDQcT4\n1z/mEC1m55313L3v85FGe/b8zjvntyM/PjPnzMyriMDM8vOlTjdgZp3h8JtlyuE3y5TDb5Yph98s\nUw6/WaYc/oxJelTSme0ea93B4R8FJK2SNLPTfVQjqV/SSknvS1oj6QeSvtzpvnLn8Fs77AhcAPQC\nhwPHAn/X0Y7M4R/NJE2U9ICkQUnvFMt7bLXZPpJ+XRyVl0rqGTL+CEn/JeldSc9KOrqePiLipoh4\nPCI+iYi1wN3AN+v/y6wZHP7R7UvA7cDewF7AR8C/bLXNXwHfAyYDm4EbASTtDvwH8I9AD5Uj9RJJ\nfVvvRNJexX8Qe42wr6OAF7b5r7GmcvhHsYh4KyKWRMT/RcQm4J+AGVttdmdEDETEh8D3gTmSxgCn\nAcsjYnlEbImIFUAZmD3Mfn4bERMi4re1epL0PaAE/LDBP88a5JMuo5ikHYEbgFnAxGL1eEljIuLT\n4vfXhwxZDYyl8tp8b+AvJX17SH0s8IsG+jkRuAaYGREb670faw6Hf3S7CNgfODwi3pR0KPDfgIZs\ns+eQ5b2A3wEbqfyncGdE/E0zGpE0C1gI/EVEPN+M+7TG+Gn/6DFW0rghty8D46m8zn+3OJF3xTDj\nTpN0UPEs4R+Afy+eFdwFfFvSn0saU9zn0cOcMKxJ0jFUTvKdFBG/rvsvtKZy+EeP5VSC/tntSuCf\ngT+gciR/Evj5MOPuBBYDbwLjgPMBIuJ14ATgUmCQyjOBv2eYfzPFCb8PEif8vg/8IbC82O4DSQ/W\n9Vda08hf5mGWJx/5zTLl8JtlyuE3y5TDb5aptl7n7+3tjSlTprRzl2ZZWbVqFRs3blTtLRsMf/HG\njfnAGODWiLg2tf2UKVMol8uN7NLMEkql0oi3rftpf/H+738FjgMOAk6WdFC992dm7dXIa/5pwKsR\n8VpEfAL8hMqbQsxsO9BI+Hfn8x8KWVOs+xxJ8ySVJZUHBwcb2J2ZNVPLz/ZHxIKIKEVEqa/vCx8F\nN7MOaST8a/n8J8L2KNaZ2XagkfA/Bewr6WuSvgJ8F1jWnLbMrNXqvtQXEZslnQf8J5VLfYsiwl/N\nZLadaOg6f0Qsp/JRUjPbzvjtvWaZcvjNMuXwm2XK4TfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjN\nMuXwm2XK4TfLlMNvlilP0W0NeeCBB5L1/v7+qrVDDjkkOfbmm29O1vfff/9k3dJ85DfLlMNvlimH\n3yxTDr9Zphx+s0w5/GaZcvjNMuXr/Jn7+OOPk/Xzzz8/Wb/tttuS9S1btlStPfroo8mxs2bNStYf\nfPDBZP2AAw5I1nPnI79Zphx+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlilf58/csmXLkvWFCxcm6xMm\nTEjWjz/++Kq1xx9/PDl21apVyfoNN9yQrN9yyy3Jeu4aCr+kVcAm4FNgc0SUmtGUmbVeM478fxoR\nG5twP2bWRn7Nb5apRsMfwEOSVkqaN9wGkuZJKksqDw4ONrg7M2uWRsM/PSK+ARwHnCvpqK03iIgF\nEVGKiFJfX1+DuzOzZmko/BGxtvi5AbgPmNaMpsys9eoOv6SdJI3/bBn4FjDQrMbMrLUaOds/CbhP\n0mf3c09E/LwpXVnbvPHGG8n6xIkTk/VXX301We/p6alaW7x4cXLs6aefnqwvWrSo7vFHHHFEcmwO\n6g5/RLwG/EkTezGzNvKlPrNMOfxmmXL4zTLl8JtlyuE3y5Q/0pu5++67L1k/8sgjk/XUpTxIfz13\nra8Fr2Xz5s3J+v3331+15kt9PvKbZcvhN8uUw2+WKYffLFMOv1mmHH6zTDn8Zpnydf7MHX744cn6\nyy+/nKy/9957yfoZZ5xRtbZp06bk2ClTpiTrtb7ae+XKlcl67nzkN8uUw2+WKYffLFMOv1mmHH6z\nTDn8Zply+M0y5ev8mdttt92S9euvvz5Znz59erL+2muvVa3NmDEjOfass85K1k855ZRk3dJ85DfL\nlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXr/Ja0ZcuWZH1gYCBZ7+3trVqbP39+cuy4ceOSdWtM\nzSO/pEWSNkgaGLKuR9IKSa8UP9OTuJtZ1xnJ0/7FwKyt1l0CPBIR+wKPFL+b2XakZvgj4jHg7a1W\nnwDcUSzfAZzY5L7MrMXqPeE3KSLWFctvApOqbShpnqSypPLg4GCduzOzZmv4bH9EBBCJ+oKIKEVE\nqa+vr9HdmVmT1Bv+9ZImAxQ/NzSvJTNrh3rDvwzoL5b7gaXNacfM2qXmdX5J9wJHA72S1gBXANcC\nP5V0BrAamNPKJq11Zs6c2dD4uXPnJus33nhj1dqECROSY2vNGWCNqRn+iDi5SunYJvdiZm3kt/ea\nZcrhN8uUw2+WKYffLFMOv1mm/JHezE2dOjVZrzUF9/jx45N1Sdvc02fuvvvuusdabT7ym2XK4TfL\nlMNvlimH3yxTDr9Zphx+s0w5/GaZ8nX+zNW6Dr/zzju3qZMvevbZZxsaf9JJJzWpk9HJR36zTDn8\nZply+M0y5fCbZcrhN8uUw2+WKYffLFO+zm8ds3RperqHhx56KFkfO3Zssn722Wdvc0858ZHfLFMO\nv1mmHH6zTDn8Zply+M0y5fCbZcrhN8uUr/MX3n333WT96quvrlobN25ccuycOekZzHfbbbdkvaen\nJ1nfXt1zzz3J+scff5ys9/f3N7Od7NQ88ktaJGmDpIEh666UtFbSM8VtdmvbNLNmG8nT/sXArGHW\n3xARhxa35c1ty8xarWb4I+Ix4O029GJmbdTICb/zJD1XvCyYWG0jSfMklSWVBwcHG9idmTVTveG/\nCdgHOBRYB1xfbcOIWBARpYgo9fX11bk7M2u2usIfEesj4tOI2AIsBKY1ty0za7W6wi9p8pBfvwMM\nVNvWzLpTzev8ku4FjgZ6Ja0BrgCOlnQoEMAq4KwW9tgW999/f7J+3XXX1X3fV111VbK+9957J+vH\nHHNMsn7xxRdXre23337JsbW+t79R5XK5am3FihXJsb29vcn6OeecU1dPVlEz/BFx8jCrb2tBL2bW\nRn57r1mmHH6zTDn8Zply+M0y5fCbZSqbj/Ru3LgxWb/kkkva1MkXrV69Olm//fbb666feeaZybEX\nXnhhsn7ggQcm6y+99FKyfuqpp1atvfPOO8mx11xzTbI+bZrfW9YIH/nNMuXwm2XK4TfLlMNvlimH\n3yxTDr9Zphx+s0xlc53/o48+StbXr1+frB922GFVa8uWLUuOrfXR1SVLliTrTzzxRLK+YcOGqrVb\nb701Ofbee+9N1sePH5+sb9q0KVn/8MMPk/WU1NelA7z11lvJeiMfw86Bj/xmmXL4zTLl8JtlyuE3\ny5TDb5Yph98sUw6/Waayuc7fqDFjxlStTZo0KTl27ty5DdXfeOONZP2yyy6rWlu8eHFybK3r8I1c\np29UrfcQ7Lrrrm3qZHTykd8sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y9RIpujeE/gxMInKlNwL\nImK+pB7g34ApVKbpnhMR6S9i76Cenp5kferUqcn6k08+WbV2+eWXJ8fOmDEjWa9lYGAgWX/44Ycb\nuv9OOfjgg5P1OXPmJOsXXXRRM9vJzkiO/JuBiyLiIOAI4FxJBwGXAI9ExL7AI8XvZradqBn+iFgX\nEU8Xy5uAF4HdgROAO4rN7gBObFWTZtZ82/SaX9IU4OvAr4BJEbGuKL1J5WWBmW0nRhx+SV8FlgAX\nRMT7Q2sREVTOBww3bp6ksqTy4OBgQ82aWfOMKPySxlIJ/t0R8bNi9XpJk4v6ZGDYb5GMiAURUYqI\nUl9fXzN6NrMmqBl+SQJuA16MiB8NKS0D+ovlfmBp89szs1ZR5Rl7YgNpOvA48DywpVh9KZXX/T8F\n9gJWU7nU93bqvkqlUpTL5UZ7bom77rorWa/1sdtutcMOOyTrs2fPTtbnz5/fzHY+Z5dddknWd9xx\nx5bte7QqlUqUy2WNZNua1/kj4pdAtTs7dlsaM7Pu4Xf4mWXK4TfLlMNvlimH3yxTDr9Zphx+s0z5\nq7sLp512WrK+Zs2aqrVrr702Ofa9996rq6eRmjlzZtVaf39/1RrU/rtt9PKR3yxTDr9Zphx+s0w5\n/GaZcvjNMuXwm2XK4TfLVM3P8zdTN3+e32w02JbP8/vIb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl\n8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8Jtlqmb4Je0p6ReSfiPpBUl/W6y/UtJa\nSc8Ut/RE72bWVUYyacdm4KKIeFrSeGClpBVF7YaI+GHr2jOzVqkZ/ohYB6wrljdJehHYvdWNmVlr\nbdNrfklTgK8DvypWnSfpOUmLJE2sMmaepLKk8uDgYEPNmlnzjDj8kr4KLAEuiIj3gZuAfYBDqTwz\nuH64cRGxICJKEVHq6+trQstm1gwjCr+ksVSCf3dE/AwgItZHxKcRsQVYCExrXZtm1mwjOdsv4Dbg\nxYj40ZD1k4ds9h1goPntmVmrjORs/zeBucDzkp4p1l0KnCzpUCCAVcBZLenQzFpiJGf7fwkM9z3g\ny5vfjpm1i9/hZ5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNM\nOfxmmXL4zTKliGjfzqRBYPWQVb3AxrY1sG26tbdu7QvcW72a2dveETGi78tra/i/sHOpHBGljjWQ\n0K29dWtf4N7q1ane/LTfLFMOv1mmOh3+BR3ef0q39tatfYF7q1dHeuvoa34z65xOH/nNrEMcfrNM\ndST8kmZJelnSq5Iu6UQP1UhaJen5Ytrxcod7WSRpg6SBIet6JK2Q9Erxc9g5EjvUW1dM256YVr6j\nj123TXff9tf8ksYA/wP8GbAGeAo4OSJ+09ZGqpC0CihFRMffECLpKOAD4McR8cfFuh8Ab0fEtcV/\nnBMj4uIu6e1K4INOT9tezCY1eei08sCJwF/Twccu0dccOvC4deLIPw14NSJei4hPgJ8AJ3Sgj64X\nEY8Bb2+1+gTgjmL5Dir/eNquSm9dISLWRcTTxfIm4LNp5Tv62CX66ohOhH934PUhv6+hgw/AMAJ4\nSNJKSfM63cwwJkXEumL5TWBSJ5sZRs1p29tpq2nlu+axq2e6+2bzCb8vmh4R3wCOA84tnt52pai8\nZuuma7Ujmra9XYaZVv73OvnY1TvdfbN1IvxrgT2H/L5Hsa4rRMTa4ucG4D66b+rx9Z/NkFz83NDh\nfn6vm6ZtH25aebrgseum6e47Ef6ngH0lfU3SV4DvAss60McXSNqpOBGDpJ2Ab9F9U48vA/qL5X5g\naQd7+Zxumba92rTydPix67rp7iOi7TdgNpUz/v8LXNaJHqr09UfAs8XthU73BtxL5Wng76icGzkD\n2AV4BHgFeBjo6aLe7gSeB56jErTJHeptOpWn9M8BzxS32Z1+7BJ9deRx89t7zTLlE35mmXL4zTLl\n8JtlyuE3y5TDb5Yph98sUw6/Wab+H+LICaw0MGwNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11bd3ef60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True) # reads in the MNIST dataset\n",
    "\n",
    "\n",
    "# a function that shows examples from the dataset. If num is specified (between 0 and 9), then only pictures with those labels will beused\n",
    "def show_pics(mnist, num = None):\n",
    "    to_show = list(range(10)) if not num else [num]*10 # figure out which numbers we should show\n",
    "    for i in range(100):\n",
    "        batch = mnist.train.next_batch(1) # gets some examples\n",
    "        pic, label = batch[0], batch[1]\n",
    "        if np.argmax(label) in to_show:\n",
    "            # use matplotlib to plot it\n",
    "            pic = pic.reshape((28,28))\n",
    "            plt.title(\"Label: {}\".format(np.argmax(label)))\n",
    "            plt.imshow(pic, cmap = 'binary')\n",
    "            plt.show()\n",
    "            to_show.remove(np.argmax(label))\n",
    "            \n",
    "            \n",
    "\n",
    "#show_pics(mnist)\n",
    "show_pics(mnist, 2)\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As usual, we would like to define several variables to represent our weight matrices and our biases. We will also need to create placeholders to hold our actual data. Anytime we want to create variables or placeholders, we must have a sense of the **shape** of our data so that Tensorflow has no issues in carrying out the numerical computations. \n",
    "\n",
    "In addition, neural networks rely on various hyperparameters, some of which will be defined below. Two important ones are the ** learning rate ** and the number of neurons in our hidden layer. Depending on these settings, the accuracy of the network may greatly change. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# some functions for quick variable creation\n",
    "def weight_variable(shape):\n",
    "    return tf.Variable(tf.truncated_normal(shape, stddev = 0.1))\n",
    "\n",
    "def bias_variable(shape):\n",
    "    return tf.Variable(tf.constant(0.1, shape = shape))\n",
    "\n",
    "# hyperparameters we will use\n",
    "learning_rate = 0.1\n",
    "hidden_layer_neurons = 50\n",
    "num_iterations = 5000\n",
    "\n",
    "# placeholder variables\n",
    "x = tf.placeholder(tf.float32, shape = [None, 784]) # none = the size of that dimension doesn't matter. why is that okay here? \n",
    "y_ = tf.placeholder(tf.float32, shape = [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will now actually create all of the variables we need, and define our neural network as a series of function computations. \n",
    "\n",
    "In our first layer, we take our inputs that have dimension $n * 784$, and multiply them with weights that have dimension $ 784 * k $, where $k$ is the number of neurons in the hidden layer. We then add the biases to this result, which also have a dimension of $k$. \n",
    "\n",
    "Finally, we apply a nonlinearity to our result. There are, as discussed, several choices, three of which are tanh, sigmoid, and rectifier. We have chosen to use the rectifier (also known as relu, standing for Rectified Linear Unit), since it has been shown in both research and practice that they tend to outperform and learn faster than other activation functions. \n",
    "\n",
    "Therefore, the \"activations\" of our hidden layer are given by $h_1 = relu(Wx + b)$. \n",
    "\n",
    "We follow a similar procedure for our output layer. Our activations have a shape $n * k$, where $n$ is the number of training examples we input into our network and $k$ is the number of neurons in our hidden layer. \n",
    "\n",
    "We want our final outputs to have dimension $n * 10$ (in the case of MNIST) since we have 10 classes. Therefore, it makes sense for our second matrix of weights to have dimension $k * 10$ and the bias to have dimension $10$. \n",
    "\n",
    "After taking the linear combination $W_2(h_1) + b$, we would then apply the softmax function. However, applying the softmax function and then writing out the cross-entropy loss ourself could result in numerical unstability, so we will instead use a library call that computes both the softmax outputs and the cross entropy loss. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# create our weights and biases for our first hidden layer\n",
    "W_1, b_1 = weight_variable([784, hidden_layer_neurons]), bias_variable([hidden_layer_neurons])\n",
    "\n",
    "# compute activations of the hidden layer\n",
    "h_1 = tf.nn.relu(tf.matmul(x, W_1) + b_1)\n",
    "\n",
    "W_2_hidden = weight_variable([hidden_layer_neurons, 30])\n",
    "b_2_hidden = bias_variable([30])\n",
    "h_2 = tf.nn.relu(tf.matmul(h_1, W_2_hidden) + b_2_hidden)\n",
    "# create our weights and biases for our output layer\n",
    "W_2, b_2 = weight_variable([30, 10]), bias_variable([10])\n",
    "# compute the of the output layer\n",
    "y = tf.matmul(h_2,W_2) + b_2\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The cross entropy loss function is a commonly used loss function. For a single prediction/label pair, it is given by $C(h(x), y) = -\\sum_i y_i log(h(x_i))$.*\n",
    "\n",
    "Here, $y$ is a specific one-hot encoded label vector, meaning that it is a column vector that has a 1 at the index corresponding to its label, and is zero everywhere else. $ h(x) $ is the output of our prediction function whose elements sum to 1. As an example, we may have: \n",
    "\n",
    "$$y = \\begin{bmatrix}\n",
    "           1 \\\\\n",
    "           0 \\\\\n",
    "           0 \n",
    "         \\end{bmatrix}, h(x_i) = \\begin{bmatrix}\n",
    "           0.2 \\\\\n",
    "           0.7 \\\\\n",
    "           0.1 \n",
    "         \\end{bmatrix} \\longrightarrow{} C(y, h(x)) = -\\sum_{i=1}^{N}y_ilog(h(x_i)) = -log(0.2) = 0.61 $$\n",
    "         \n",
    "The contribution to the entire training data's loss by this pair was 0.61. To contrast, we can swap the first two probabilities in our softmax vector. We then end up with a lower loss: \n",
    "\n",
    "$$y = \\begin{bmatrix}\n",
    "           1 \\\\\n",
    "           0 \\\\\n",
    "           0 \n",
    "         \\end{bmatrix}, h(x) = \\begin{bmatrix}\n",
    "           0.7 \\\\\n",
    "           0.2 \\\\\n",
    "           0.1 \n",
    "         \\end{bmatrix} \\longrightarrow{} C(y, h(x)) = -\\sum_{i=1}^{N}y_ilog(h(x_i)) = -log(0.7) = 0.15 $$\n",
    "\n",
    "So our cross-entropy loss makes intuitive sense: it is lower when our softmax vector has a high probability at the index of the true label, and it is higher when our probabilities indicate a wrong or uncertain choice.\n",
    "\n",
    "**Sanity check: why do we need the negative sign outside the sum?**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# define our loss function as the cross entropy loss\n",
    "cross_entropy_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y_, logits = y))\n",
    "\n",
    "# create an optimizer to minimize our cross entropy loss\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cross_entropy_loss)\n",
    "\n",
    "# functions that allow us to gauge accuracy of our model\n",
    "correct_predictions = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)) # creates a vector where each element is T or F, denoting whether our prediction was right\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32)) # maps the boolean values to 1.0 or 0.0 and calculates the accuracy\n",
    "\n",
    "# we will need to run this in our session to initialize our weights and biases. \n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With all of our variables created and computation graph defined, we can now launch the graph in a session and begin training. It is important to remember that since we declared the $x$ and $y$ variables as placeholders, we will need to feed in data to run our optimizer that minimizes the cross entropy loss. \n",
    "\n",
    "The data we will feed in (by passing into our function a dictionary *feed_dict*) will come from the MNIST dataset. To randomly sample 100 training examples, we can use a wrapper provided by Tensorflow: ```mnnist.train.next_batch(100)```. \n",
    "\n",
    "When we run the optimizer with the call ```optimizer.run(..)``` Tensorflow calculates a forward pass for us (essentially propagating our data through the graph we have described), and then uses the loss function we created to evaluate the loss, and then computes partial derivatives with respect to each set of weights and updates the weights according to the partial derivatives. This is called the backpropagation algorithm, and it involves significant application of the chain rule. CS 231N provides an [excellent explanation](http://cs231n.github.io/optimization-2/) of backpropagation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, accuracy: 0.07999999821186066, loss: 2.2931833267211914\n",
      "Epoch: 100, accuracy: 0.8399999737739563, loss: 0.6990350484848022\n",
      "Epoch: 200, accuracy: 0.8700000047683716, loss: 0.35569435358047485\n",
      "Epoch: 300, accuracy: 0.9300000071525574, loss: 0.26591774821281433\n",
      "Epoch: 400, accuracy: 0.8999999761581421, loss: 0.3307000696659088\n",
      "Epoch: 500, accuracy: 0.9399999976158142, loss: 0.23977749049663544\n",
      "Epoch: 600, accuracy: 0.9800000190734863, loss: 0.09397666901350021\n",
      "Epoch: 700, accuracy: 0.9200000166893005, loss: 0.2931550145149231\n",
      "Epoch: 800, accuracy: 0.9399999976158142, loss: 0.20180968940258026\n",
      "Epoch: 900, accuracy: 0.949999988079071, loss: 0.18461622297763824\n",
      "Epoch: 1000, accuracy: 0.9700000286102295, loss: 0.18968147039413452\n",
      "Epoch: 1100, accuracy: 0.9599999785423279, loss: 0.14828498661518097\n",
      "Epoch: 1200, accuracy: 0.949999988079071, loss: 0.1613173633813858\n",
      "Epoch: 1300, accuracy: 0.9800000190734863, loss: 0.10008890926837921\n",
      "Epoch: 1400, accuracy: 0.9900000095367432, loss: 0.07440848648548126\n",
      "Epoch: 1500, accuracy: 0.9599999785423279, loss: 0.1167958676815033\n",
      "Epoch: 1600, accuracy: 0.9100000262260437, loss: 0.1591644138097763\n",
      "Epoch: 1700, accuracy: 0.9599999785423279, loss: 0.10022231936454773\n",
      "Epoch: 1800, accuracy: 0.9700000286102295, loss: 0.1086776852607727\n",
      "Epoch: 1900, accuracy: 0.9700000286102295, loss: 0.15659521520137787\n",
      "Epoch: 2000, accuracy: 0.9599999785423279, loss: 0.09391114860773087\n",
      "Epoch: 2100, accuracy: 0.9800000190734863, loss: 0.09786181151866913\n",
      "Epoch: 2200, accuracy: 0.9700000286102295, loss: 0.11428779363632202\n",
      "Epoch: 2300, accuracy: 0.9900000095367432, loss: 0.07231700420379639\n",
      "Epoch: 2400, accuracy: 0.9700000286102295, loss: 0.09908157587051392\n",
      "Epoch: 2500, accuracy: 0.9599999785423279, loss: 0.15657338500022888\n",
      "Epoch: 2600, accuracy: 0.9900000095367432, loss: 0.07787769287824631\n",
      "Epoch: 2700, accuracy: 0.9800000190734863, loss: 0.07373256981372833\n",
      "Epoch: 2800, accuracy: 0.9700000286102295, loss: 0.062044695019721985\n",
      "Epoch: 2900, accuracy: 0.9700000286102295, loss: 0.12512363493442535\n",
      "Epoch: 3000, accuracy: 0.9900000095367432, loss: 0.11000598967075348\n",
      "Epoch: 3100, accuracy: 0.9700000286102295, loss: 0.20609986782073975\n",
      "Epoch: 3200, accuracy: 0.9800000190734863, loss: 0.09811186045408249\n",
      "Epoch: 3300, accuracy: 0.9700000286102295, loss: 0.09816547483205795\n",
      "Epoch: 3400, accuracy: 0.9700000286102295, loss: 0.10826745629310608\n",
      "Epoch: 3500, accuracy: 0.9900000095367432, loss: 0.0645124614238739\n",
      "Epoch: 3600, accuracy: 0.9700000286102295, loss: 0.1555529236793518\n",
      "Epoch: 3700, accuracy: 0.9700000286102295, loss: 0.06963416188955307\n",
      "Epoch: 3800, accuracy: 0.9900000095367432, loss: 0.08054723590612411\n",
      "Epoch: 3900, accuracy: 0.9800000190734863, loss: 0.06120322644710541\n",
      "Epoch: 4000, accuracy: 0.9900000095367432, loss: 0.06058483570814133\n",
      "Epoch: 4100, accuracy: 0.9700000286102295, loss: 0.11490124464035034\n",
      "Epoch: 4200, accuracy: 0.9700000286102295, loss: 0.10046141594648361\n",
      "Epoch: 4300, accuracy: 0.9800000190734863, loss: 0.04671316221356392\n",
      "Epoch: 4400, accuracy: 0.9900000095367432, loss: 0.052477456629276276\n",
      "Epoch: 4500, accuracy: 0.9800000190734863, loss: 0.08245706558227539\n",
      "Epoch: 4600, accuracy: 0.9900000095367432, loss: 0.041497569531202316\n",
      "Epoch: 4700, accuracy: 0.9900000095367432, loss: 0.050769224762916565\n",
      "Epoch: 4800, accuracy: 0.9900000095367432, loss: 0.039090484380722046\n",
      "Epoch: 4900, accuracy: 0.9900000095367432, loss: 0.0564178042113781\n",
      "testing accuracy: 0.9653000235557556\n"
     ]
    }
   ],
   "source": [
    "# launch a session to run our graph defined above. \n",
    "with tf.Session() as sess:\n",
    "    sess.run(init) # initializes our variables\n",
    "    for i in range(num_iterations):\n",
    "        # get a sample of the dataset and run the optimizer, which calculates a forward pass and then runs the backpropagation algorithm to improve the weights\n",
    "        batch = mnist.train.next_batch(100)\n",
    "        optimizer.run(feed_dict = {x: batch[0], y_: batch[1]})\n",
    "        # every 100 iterations, print out the accuracy\n",
    "        if i % 100 == 0:\n",
    "            # accuracy and loss are both functions that take (x, y) pairs as input, and run a forward pass through the network to obtain a prediction, and then compares the prediction with the actual y.\n",
    "            acc = accuracy.eval(feed_dict = {x: batch[0], y_: batch[1]})\n",
    "            loss = cross_entropy_loss.eval(feed_dict = {x: batch[0], y_: batch[1]})\n",
    "            print(\"Epoch: {}, accuracy: {}, loss: {}\".format(i, acc, loss))\n",
    "            \n",
    "     # evaluate our testing accuracy       \n",
    "    acc = accuracy.eval(feed_dict = {x: mnist.test.images, y_: mnist.test.labels})\n",
    "    print(\"testing accuracy: {}\".format(acc))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Questions to Ponder\n",
    "\n",
    "- Why bother with neural networks in the first place? Clearly, we saw much higher accuracy on the MNIST dataset than we were able to do with logistic regression, but why is this? How do additional layers and nonlinear activation functions chance the decision boundary that we learn, and why can this lead to better accuracy?\n",
    "- Why is the test accuracy lower than the (final) training accuracy ?\n",
    "- How can we tune our hyperparameters? In practice, is it okay to continually search for the best performance on the test dataset? \n",
    "- When we add additional layers to our learning model, how does gradient descent change? In particular, why is it important to be extra careful about our learning rate, have sensible initialization, and tune hyperparameters?\n",
    "- Why do we use only 100 examples in each iteration, as opposed to the entire dataset of 50,000 examples? \n",
    "\n",
    "### Exercises\n",
    "1. Using different activation functions. Consult the Tensorflow documentation on `tanh` and `sigmoid`, and use that as the activation function instead of `relu`. Gauge the resulting changes in accuracy. \n",
    "2. Varying the number of neurons - as mentioned, we have complete control over the number of neurons in our hidden layer. How does the testing accuracy change with a small number of neurons versus a large number of neurons? What about the generalization accuracy (with respect to the testing accuracy?)\n",
    "3. Using different loss functions - we have discussed the cross entropy loss. Another common loss function used in neural networks is the MSE loss. Consult the Tensorflow documentation and implement the ```MSELoss()``` function. \n",
    "4. Addition of another hidden layer - We can create a deeper neural network with additional hidden layers. Similar to how we created our original hidden layer, you will have to figure out the dimensions for the weights (and biases) by looking at the dimension of the previous layer, and deciding on the number of neurons you would like to use. Once you have decided this, you can simply insert another layer into the network with only a few lines of code: \n",
    "    1. Use ```weight_variable()``` and ```bias_variable()``` to create new variables for the additional layer (remember to specify the shape correctly). \n",
    "    2. Similar to computing the activations for the first layer, ```h_1 = tf.nn.relu(...)```, compute the activations for your additional hidden layer. \n",
    "    3. Remember to change your output weight dimensions in order to reflect the number of neurons in the previous layer. \n",
    "\n",
    "### Additional Challenges\n",
    "5. [Regularization](http://neuralnetworksanddeeplearning.com/chap3.html) is a popular approach to mitigage the overfitting issue in machine learning. Add L2-regularization to this model (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/nn/l2_loss)), by adding an extra expression that calculates the squared norm of the weights. What does this change with regards to the optimization problem we are now solving? \n",
    "6. A technique to prevent overfitting that has worked very well in recent years is [dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf), a method that was only introdcued in 2014. Dropout is essentially discarding some proportion (usually half) of a layer's activations (setting them to zero). Once you've added dropout to your network, how does your training and testing accuracy change? \n",
    "6. The [momentum method](https://www.coursera.org/learn/neural-networks/lecture/Oya9a/the-momentum-method) is an increasingly popular approach to train deep neural networks. Momentum takes into account the previous gradients computed earlier on in the main learning loop, and uses those gradients to \"guide\" the descent towards a local minimum. Instead of using the `GradientDescentOptimizer` as we did, use the `MomentumOptimizer`. Momentum also introduces another hyperparameter that must be tuned. What does this hyperparameter mean, and how can we find a sensible value for it? \n",
    "7. Decaying the learning rate across time is another useful method to increase our algorithm's accuracy. Intuitively, we'd expect our algorithm, in the beginning training steps, to make large descents towards a local minima and rapidly adjust it's weights, but when our accuracy is very high, we'd only want to make very small, incremental improvements towards the local minima, since we are already around it. Add learning rate decay to the network. As a hint, we will no longer be able to set the learning rate as a constant, rather, we'd have to feed it in using `feed_dict`, much like we feed in our training inputs. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    " *Technical note: The way this loss function is presented is such that activations corresponding to a label of zero are not penalized at all. The full form of the cross-entropy loss is given by $C(y, h(x)) = \\sum_i y_i log(h(x_i)) + (1 - y_i)(log(1 - h(x_i)) $. However, the previously presented function works just as well in environments with larger amounts of data samples and training for many epochs (passes through the dataset), which is typically the case for neural networks. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}