6_Classification_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Classification, Regression\n",
    "This section will cover logistic regression (which is a classification model, not regression) and linear regression. Once you can train both of these models, you know how to do both classification and regression in their simplest way possible. I first demonstrate how to train a logistic regression using the Iris dataset. As an exercise, you will train a linear regression on the Boston housing dataset.\n",
    "\n",
    "This section will put together what we've learned so far, e.g. `session`, `placeholder`, `variable`, etc. This section might seem hard at first, but the only new concept that will be introduced here is the `GradientDescentOptimizer`. As long as you understand the code step by step, it should be ok."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "This example is partially adopted from [Deep MNIST for Experts](https://www.tensorflow.org/versions/r0.12/tutorials/mnist/pros/) tutorial from TensorFlow official website.\n",
    "\n",
    "### Task\n",
    "\n",
    "Classify a dataset of hand-written digits (0 through 9).\n",
    "\n",
    "### Data\n",
    "\n",
    "Let's load the data first. Thankfully, TensorFlow provided us a nice API. The data is also already split into train, validation and test set. In general, the data is much less clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Import MINST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the first image to get a sense of what is inside. Note that `next_batch` is an API provided by TensorFlow to get the next `n` images from the dataset. You generally have to code this up yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.45490199,  0.9450981 ,  0.68235296,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.3137255 ,  0.50196081,\n",
       "         0.89019614,  0.42352945,  0.20000002,  0.67058825,  0.99215692,\n",
       "         0.99215692,  0.81176478,  0.09019608,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.2392157 ,  0.96862751,  0.99215692,  0.99215692,  0.99215692,\n",
       "         0.94117653,  1.        ,  0.99215692,  0.99215692,  0.99215692,\n",
       "         0.67843139,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.04313726,  0.77254909,\n",
       "         0.99215692,  0.99215692,  0.99215692,  0.99215692,  0.99607849,\n",
       "         0.99215692,  0.99215692,  0.99215692,  0.72941178,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.59607846,  0.99215692,  0.96470594,\n",
       "         0.74509805,  0.85490203,  0.8588236 ,  0.74117649,  0.54901963,\n",
       "         0.99215692,  0.72941178,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.05490196,  0.25882354,  0.23137257,  0.01568628,  0.12156864,\n",
       "         0.1254902 ,  0.01568628,  0.48235297,  0.99215692,  0.72941178,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.89411771,  0.99215692,  0.37254903,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.89411771,  0.99215692,\n",
       "         0.20784315,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.24705884,  0.94901967,  0.80784321,  0.08627451,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.42352945,  0.99215692,\n",
       "         0.67843139,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.227451  ,  0.97254908,  0.99607849,  0.15686275,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.47450984,  0.99215692,\n",
       "         0.83529419,  0.09019608,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.9450981 ,  0.99215692,  0.15294118,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.99607849,\n",
       "         0.99215692,  0.10196079,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.01568628,  0.26666668,  0.99607849,  0.65882355,  0.02352941,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.16470589,  0.99215692,\n",
       "         0.89803928,  0.17647059,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.43529415,  0.86274517,  0.99215692,  0.44313729,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.15686275,  0.8588236 ,  0.99215692,\n",
       "         0.93725497,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.68235296,  0.99215692,  0.96078438,  0.19607845,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.40784317,  0.99215692,\n",
       "         0.66274512,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ]], dtype=float32)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = mnist.train.next_batch(1) # get one image from the data\n",
    "plotData = batch[0]\n",
    "plotData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are $28\\times 28=784$ pixels. $0$ means black and $1$ means white. Let's reshape it and use `matplotlib` to make it an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADNxJREFUeJzt3W+IXfWdx/HPZzV5koQQKaYxya6xyLqikC6DiMqaRazuUkiqVJoHJYuh6YMqVioaAtI8WdHFtttHhSmGRmhtKm01StlNiAW3oMEYtZM0ppWaTWaNMw2KGQUNmu8+mJMyiXN/93rvuffcyff9gjD3nu8593w5+plz7pw/P0eEAOTzN003AKAZhB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFIXDnJltrmcEOiziHAn8/W057d9q+3Dtt+wvbmXzwIwWO722n7bF0j6o6SbJY1LeknS+oj4Q2EZ9vxAnw1iz3+NpDci4s8RcUrSzyWt7eHzAAxQL+FfLunYjPfj1bSz2N5ke5/tfT2sC0DNevmD32yHFp86rI+IUUmjEof9wDDpZc8/LmnljPcrJL3VWzsABqWX8L8k6XLbq2zPl/Q1STvraQtAv3V92B8RH9u+S9J/S7pA0raIOFhbZwD6qutTfV2tjO/8QN8N5CIfAHMX4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0l1PUS3JNk+ImlK0ieSPo6IkTqaAtB/PYW/8s8RcaKGzwEwQBz2A0n1Gv6QtMv2y7Y31dEQgMHo9bD/+oh4y/bFknbbfj0inp85Q/VLgV8MwJBxRNTzQfZWSe9HxKOFeepZGYCWIsKdzNf1Yb/tBbYXnXkt6UuSDnT7eQAGq5fD/qWSfm37zOf8LCL+q5auAPRdbYf9Ha2Mw36g7/p+2A9gbiP8QFKEH0iK8ANJEX4gKcIPJFXHXX1I7L777ivWH3zwwZa1PXv2FJe97bbbuuoJnWHPDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJcZ5/CKxbt65YX716dbG+cePGlrVHHnmkuOwLL7xQrG/evLlY7+Vc/KpVq4r1pUuXFusTExNdrxvs+YG0CD+QFOEHkiL8QFKEH0iK8ANJEX4gKR7dPQA33HBDsb579+5iff78+XW2c5apqaliffHixcX66dOn62znLE899VSxfvvtt/dt3XMZj+4GUET4gaQIP5AU4QeSIvxAUoQfSIrwA0m1vZ/f9jZJX5Y0GRFXVdMukrRD0qWSjki6IyLe7V+bc9srr7xSrO/fv79Yv/baa+ts5yyLFi0q1gd5Hci52vWG3nSy5/+JpFvPmbZZ0p6IuFzSnuo9gDmkbfgj4nlJ75wzea2k7dXr7ZLKj6IBMHS6/c6/NCKOS1L18+L6WgIwCH1/hp/tTZI29Xs9AD6bbvf8E7aXSVL1c7LVjBExGhEjETHS5boA9EG34d8paUP1eoOkp+tpB8CgtA2/7SckvSDp722P294o6WFJN9v+k6Sbq/cA5pC23/kjYn2L0k0193Le+uCDD4r1sbGxYr3def6PPvqoZe3FF18sLrtixYpi3S7fGn706NFivXSNw7333ltcFv3FFX5AUoQfSIrwA0kRfiApwg8kRfiBpBiiewjcf//9xfqOHTuK9VOnTrWs7d27t7js8uXLi/V2p/rGx8eL9QceeKBYR3PY8wNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUgzRjb46duxYy9oll1xSXPbOO+8s1rdv316sZ8UQ3QCKCD+QFOEHkiL8QFKEH0iK8ANJEX4gKe7nx9A6ePBg0y2c19jzA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBSbc/z294m6cuSJiPiqmraVknfkPSXarYtEfGbfjWJ4XXjjTcW64sXL25Ze+2114rLthv+G73pZM//E0m3zjL9BxGxuvpH8IE5pm34I+J5Se8MoBcAA9TLd/67bP/e9jbbS2rrCMBAdBv+H0n6gqTVko5L+l6rGW1vsr3P9r4u1wWgD7oKf0RMRMQnEXFa0o8lXVOYdzQiRiJipNsmAdSvq/DbXjbj7VckHainHQCD0smpvickrZH0Odvjkr4raY3t1ZJC0hFJ3+xjjwD6oG34I2L9LJMf60MvmIOuvvrqYn3BggUta2+++WZx2cnJya56Qme4wg9IivADSRF+ICnCDyRF+IGkCD+QFEN0o+i6664r1p977rliff78+S1rV155ZXHZ119/vVjH7BiiG0AR4QeSIvxAUoQfSIrwA0kRfiApwg8kxRDdKGr3aO558+YV64cOHWpZe/fdd7vqCfVgzw8kRfiBpAg/kBThB5Ii/EBShB9IivADSXE/P4ree++9Yn3hwoXFeume/cOHD3fVE8q4nx9AEeEHkiL8QFKEH0iK8ANJEX4gKcIPJNX2fn7bKyU9Lunzkk5LGo2IH9q+SNIOSZdKOiLpjojgBu3zzKJFi4r1dteJTE1N1dkOatTJnv9jSd+JiH+QdK2kb9m+UtJmSXsi4nJJe6r3AOaItuGPiOMRsb96PSXpkKTlktZK2l7Ntl3Sun41CaB+n+k7v+1LJX1R0l5JSyPiuDT9C0LSxXU3B6B/On6Gn+2Fkn4p6dsRcdLu6PJh2d4kaVN37QHol472/LbnaTr4P42IX1WTJ2wvq+rLJE3OtmxEjEbESESM1NEwgHq0Db+nd/GPSToUEd+fUdopaUP1eoOkp+tvD0C/dHLYf72kr0sas/1qNW2LpIcl/cL2RklHJX21Py2iFxdeWP5PvGbNmp4+/8knnyzWT5w40dPno3/ahj8ifiep1Rf8m+ptB8CgcIUfkBThB5Ii/EBShB9IivADSRF+ICke3X2eKz06W5LGxsaK9bfffrtYv+WWW4r1AwcOFOuoH4/uBlBE+IGkCD+QFOEHkiL8QFKEH0iK8ANJdfwYLwyve+65p2Xt7rvv7umzH3300WKd8/hzF3t+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK+/nngCuuuKJYf+aZZ1rWLrvssuKyJ0+eLNaXLFlSrGP4cD8/gCLCDyRF+IGkCD+QFOEHkiL8QFKEH0iq7f38tldKelzS5yWdljQaET+0vVXSNyT9pZp1S0T8pl+NZvbQQw8V66Vz+R9++GFx2ZtuYpT1rDp5mMfHkr4TEfttL5L0su3dVe0HEVF+2gOAodQ2/BFxXNLx6vWU7UOSlve7MQD99Zm+89u+VNIXJe2tJt1l+/e2t9me9TpQ25ts77O9r6dOAdSq4/DbXijpl5K+HREnJf1I0hckrdb0kcH3ZlsuIkYjYiQiRmroF0BNOgq/7XmaDv5PI+JXkhQRExHxSUSclvRjSdf0r00AdWsbftuW9JikQxHx/RnTl82Y7SuSeIwrMId08tf+6yV9XdKY7VeraVskrbe9WlJIOiLpm33pENq1a1exvnbt2pa1Z599trjs/v37u+oJc18nf+3/naTZ7g/mnD4wh3GFH5AU4QeSIvxAUoQfSIrwA0kRfiApHt0NnGd4dDeAIsIPJEX4gaQIP5AU4QeSIvxAUoQfSKqT+/nrdELS/854/7lq2jAa1t6GtS+J3rpVZ29/1+mMA73I51Mrt/cN67P9hrW3Ye1LorduNdUbh/1AUoQfSKrp8I82vP6SYe1tWPuS6K1bjfTW6Hd+AM1pes8PoCGNhN/2rbYP237D9uYmemjF9hHbY7ZfbXqIsWoYtEnbB2ZMu8j2btt/qn7OOkxaQ71ttf1/1bZ71fa/NtTbStu/tX3I9kHb91TTG912hb4a2W4DP+y3fYGkP0q6WdK4pJckrY+IPwy0kRZsH5E0EhGNnxO2/U+S3pf0eERcVU37D0nvRMTD1S/OJRHxwJD0tlXS+02P3FwNKLNs5sjSktZJ+jc1uO0Kfd2hBrZbE3v+ayS9ERF/johTkn4uqfWoE4lFxPOS3jln8lpJ26vX2zX9P8/AtehtKETE8YjYX72eknRmZOlGt12hr0Y0Ef7lko7NeD+u4RryOyTtsv2y7U1NNzOLpdWw6WeGT7+44X7O1Xbk5kE6Z2Tpodl23Yx4Xbcmwj/bI4aG6ZTD9RHxj5L+RdK3qsNbdKajkZsHZZaRpYdCtyNe162J8I9LWjnj/QpJbzXQx6wi4q3q56SkX2v4Rh+eODNIavVzsuF+/mqYRm6ebWRpDcG2G6YRr5sI/0uSLre9yvZ8SV+TtLOBPj7F9oLqDzGyvUDSlzR8ow/vlLSher1B0tMN9nKWYRm5udXI0mp42w3biNeNXORTncr4T0kXSNoWEf8+8CZmYfsyTe/tpek7Hn/WZG+2n5C0RtN3fU1I+q6kpyT9QtLfSjoq6asRMfA/vLXobY2mD13/OnLzme/YA+7tBkn/I2lM0ulq8hZNf79ubNsV+lqvBrYbV/gBSXGFH5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpP4fwQHLFAcm+PsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x181d514dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plotData = plotData.reshape(28, 28) # reshape to make image a matirx\n",
    "plt.gray() # use this line if you don't want to see it in color\n",
    "plt.imshow(plotData)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will build a softmax regression. Softmax regression is just a multi-class generalization of the logistic regression model:\n",
    "\n",
    "$Y=softmax(XW+b)$\n",
    "where\n",
    "\n",
    "$Y: N \\times K$. Labels. The labels are one-hot encoded.\n",
    "\n",
    "$X: N \\times M$. Features.\n",
    "\n",
    "$W: M \\times K$. Weights.\n",
    "\n",
    "$b: K$. Bias\n",
    "\n",
    "Here, $N$ can either be the train, validation, test sample size or the mini-batch size, depending on which part of the process you're in. $M$ is the number of features (784 in this case) and $K$ is the number of classes (10 in this case).\n",
    "\n",
    "I really like to write down all the matrix dimensions before I start any coding. This way, I can avoid matrix dimension errors later on. This is kind of like creating an outline of your essay before you start drafting it.\n",
    "\n",
    "The loss we would like to optimize is the cross-entropy loss, averaged over the sample size:\n",
    "\n",
    "$$\\frac{-\\sum_{n=1}^N \\sum_{k=1}^K y_{n,k} log(\\hat{y}_{n,k})}{N}$$\n",
    "\n",
    "We will train this using gradient descent. \n",
    "\n",
    "Coding of this model is a six step process:\n",
    "1. Define variables and placeholders.\n",
    "2. Define the model.\n",
    "3. Define the loss function.\n",
    "4. Define the optimizer.\n",
    "5. Train the model, i.e. initialize variables and run optimizer.\n",
    "6. Evaluate the model.\n",
    "\n",
    "This is the same for any other models. Neural nets will have these steps too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and Placeholders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Y,X$ will be a placeholder, because we will feed in datasets with different shapes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = 784\n",
    "K = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.placeholder(tf.float32, shape=[None, M])\n",
    "Y = tf.placeholder(tf.float32, shape=[None, K])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$W,b$ will be variables, because we would like to mutate them every epoch via the gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([M, K])) # the variables will be all zeros initially\n",
    "b = tf.Variable(tf.zeros([K]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model\n",
    "Softmax regression model is a one liner!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Yhat = tf.nn.softmax(X @ W + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss\n",
    "The cross-entropy loss is also a one liner! Make sure to compare it with the equation above and check that it's doing what it's supposed to. We will also define accuracy for evaluating the result on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss               = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(Yhat), axis=1))\n",
    "# choose predictions from 10 classes and compare them with the true labels\n",
    "correct_prediction = tf.equal(tf.argmax(Yhat, 1), tf.argmax(Y, 1))           \n",
    "accuracy           = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "Here's somethig new to learn! `GradientDescentOptimizer` takes in the objective that we would like to minimize. When we run this optimizer in a session, it will look for all trainable variables (or the specified list of variables, if any) and update them. There are many other optimizers e.g. `AdagradOptimizer`, `AdamOptimizer`. Find your favorite!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01 # this is a hyperparameter to tune\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate) # first initialize an optimizer\n",
    "update = optimizer.minimize(loss) # then pass in the objective you would like to minimize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize variables and run optimizer, evaluate the model\n",
    "The final step is to train the model, and then to evaluate it. We do these in one go. \n",
    "We have 25 epochs. Within each epoch, we go through the dataset batch by batch. In each bath, we `sess.run` two variables: `update` and `loss`. Running `update` applies backpropagation to the graph and updates the variables relevant to minimizing the objective. In this case `W` and `b` are updated. Running `loss` just extracts the objective so that we can keep track of how well our model is being trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 Cost: 1.1837944513\n",
      "Epoch: 1 Cost: 0.665343354128\n",
      "Epoch: 2 Cost: 0.552922634807\n",
      "Epoch: 3 Cost: 0.498772344047\n",
      "Epoch: 4 Cost: 0.465558629795\n",
      "Epoch: 5 Cost: 0.442648443471\n",
      "Epoch: 6 Cost: 0.425598685687\n",
      "Epoch: 7 Cost: 0.412232228843\n",
      "Epoch: 8 Cost: 0.401439684359\n",
      "Epoch: 9 Cost: 0.392432119738\n",
      "Epoch: 10 Cost: 0.384814032018\n",
      "Epoch: 11 Cost: 0.378210317005\n",
      "Epoch: 12 Cost: 0.372440685115\n",
      "Epoch: 13 Cost: 0.367294720736\n",
      "Epoch: 14 Cost: 0.362738664773\n",
      "Epoch: 15 Cost: 0.358654151478\n",
      "Epoch: 16 Cost: 0.354902698181\n",
      "Epoch: 17 Cost: 0.35149155083\n",
      "Epoch: 18 Cost: 0.348347141445\n",
      "Epoch: 19 Cost: 0.345404652953\n",
      "Epoch: 20 Cost: 0.342739603519\n",
      "Epoch: 21 Cost: 0.340292761651\n",
      "Epoch: 22 Cost: 0.337950857038\n",
      "Epoch: 23 Cost: 0.335738606128\n",
      "Epoch: 24 Cost: 0.33371479877\n",
      "Test Accuracy: 0.9138\n"
     ]
    }
   ],
   "source": [
    "num_epochs  = 25\n",
    "batch_size  = 100\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer()) # remember, you need to initialize variables first\n",
    "    total_batch = int(mnist.train.num_examples/batch_size)\n",
    "    for epoch in range(num_epochs):\n",
    "        average_cost = 0 # print cost per epoch\n",
    "        for _ in range(total_batch):\n",
    "            batch_X, batch_Y = mnist.train.next_batch(batch_size)\n",
    "            _, c = sess.run([update, loss], feed_dict={X: batch_X, \n",
    "                                                          Y: batch_Y}) # evaluate `update` and `loss`\n",
    "            average_cost += c / total_batch\n",
    "        print(\"Epoch:\",epoch,\"Cost:\",average_cost)\n",
    "        \n",
    "    print(\"Test Accuracy:\", accuracy.eval({X: mnist.test.images, Y: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Done!\n",
    "We have 92% ish accuracy. How good is this? To cite the `TensorFlow` documentation:\n",
    "\n",
    "> Getting 92% accuracy on MNIST is bad. It's almost embarrassingly bad. In this section, we'll fix that, jumping from a very simple model to something moderately sophisticated: a small convolutional neural network. This will get us to around 99.2% accuracy -- not state of the art, but respectable.\n",
    "\n",
    "We are not going to introduce CNNs in this workshop, but you should take a look to aim for 99+% accuracy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "Train a linear regression using gradient descent (no closed form!) that predicts housing prices in Boston from other covariates provided. Follow the six steps above. Plot the loss and observe that it goes down.\n",
    "\n",
    "(Optional) Fit a LASSO. Fit a Ridge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "import pandas as pd\n",
    "boston = load_boston()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(boston.data)\n",
    "df.columns = boston.feature_names\n",
    "df['PRICE'] = boston.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>PRICE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  PRICE  \n",
       "0     15.3  396.90   4.98   24.0  \n",
       "1     17.8  396.90   9.14   21.6  \n",
       "2     17.8  392.83   4.03   34.7  \n",
       "3     18.7  394.63   2.94   33.4  \n",
       "4     18.7  396.90   5.33   36.2  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = df.shape[1]\n",
    "N = df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.placeholder(tf.float32, shape=[None, M])\n",
    "Y = tf.placeholder(tf.float32, shape=[None, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.ones([M, 1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "Yhat = X @ W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean((Y-Yhat)**2)\n",
    "# regularizer = tf.contrib.layers.l1_regularizer(scale=0.1)\n",
    "# regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)\n",
    "# regularization_penalty = tf.contrib.layers.apply_regularization(regularizer, [W])\n",
    "# loss += regularization_penalty\n",
    "optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "update = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_np = df.iloc[:,:-1].as_matrix()\n",
    "X_np = (X_np - np.mean(X_np,axis=0)) / np.std(X_np,axis=0)\n",
    "X_np = np.hstack((X_np,np.ones([X_np.shape[0],1])))\n",
    "y_np = df.iloc[:,-1].as_matrix()[:,np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "train_indices = np.random.rand(X_np.shape[0]) < 0.80\n",
    "X_train = X_np[train_indices]\n",
    "X_test  = X_np[~train_indices]\n",
    "y_train = y_np[train_indices]\n",
    "y_test  = y_np[~train_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 Cost: 604.989\n",
      "Epoch: 1 Cost: 566.648\n",
      "Epoch: 2 Cost: 533.349\n",
      "Epoch: 3 Cost: 504.086\n",
      "Epoch: 4 Cost: 478.079\n",
      "Epoch: 5 Cost: 454.72\n",
      "Epoch: 6 Cost: 433.534\n",
      "Epoch: 7 Cost: 414.152\n",
      "Epoch: 8 Cost: 396.281\n",
      "Epoch: 9 Cost: 379.692\n",
      "Epoch: 10 Cost: 364.203\n",
      "Epoch: 11 Cost: 349.67\n",
      "Epoch: 12 Cost: 335.978\n",
      "Epoch: 13 Cost: 323.032\n",
      "Epoch: 14 Cost: 310.756\n",
      "Epoch: 15 Cost: 299.089\n",
      "Epoch: 16 Cost: 287.978\n",
      "Epoch: 17 Cost: 277.381\n",
      "Epoch: 18 Cost: 267.259\n",
      "Epoch: 19 Cost: 257.581\n",
      "Epoch: 20 Cost: 248.321\n",
      "Epoch: 21 Cost: 239.453\n",
      "Epoch: 22 Cost: 230.957\n",
      "Epoch: 23 Cost: 222.812\n",
      "Epoch: 24 Cost: 215.002\n",
      "Epoch: 25 Cost: 207.51\n",
      "Epoch: 26 Cost: 200.322\n",
      "Epoch: 27 Cost: 193.424\n",
      "Epoch: 28 Cost: 186.802\n",
      "Epoch: 29 Cost: 180.446\n",
      "Epoch: 30 Cost: 174.343\n",
      "Epoch: 31 Cost: 168.483\n",
      "Epoch: 32 Cost: 162.856\n",
      "Epoch: 33 Cost: 157.452\n",
      "Epoch: 34 Cost: 152.263\n",
      "Epoch: 35 Cost: 147.278\n",
      "Epoch: 36 Cost: 142.491\n",
      "Epoch: 37 Cost: 137.893\n",
      "Epoch: 38 Cost: 133.476\n",
      "Epoch: 39 Cost: 129.233\n",
      "Epoch: 40 Cost: 125.157\n",
      "Epoch: 41 Cost: 121.242\n",
      "Epoch: 42 Cost: 117.48\n",
      "Epoch: 43 Cost: 113.867\n",
      "Epoch: 44 Cost: 110.395\n",
      "Epoch: 45 Cost: 107.06\n",
      "Epoch: 46 Cost: 103.855\n",
      "Epoch: 47 Cost: 100.777\n",
      "Epoch: 48 Cost: 97.8185\n",
      "Epoch: 49 Cost: 94.9762\n",
      "Epoch: 50 Cost: 92.2452\n",
      "Epoch: 51 Cost: 89.6211\n",
      "Epoch: 52 Cost: 87.0995\n",
      "Epoch: 53 Cost: 84.6765\n",
      "Epoch: 54 Cost: 82.3482\n",
      "Epoch: 55 Cost: 80.1107\n",
      "Epoch: 56 Cost: 77.9605\n",
      "Epoch: 57 Cost: 75.8941\n",
      "Epoch: 58 Cost: 73.9082\n",
      "Epoch: 59 Cost: 71.9997\n",
      "Epoch: 60 Cost: 70.1654\n",
      "Epoch: 61 Cost: 68.4025\n",
      "Epoch: 62 Cost: 66.7081\n",
      "Epoch: 63 Cost: 65.0794\n",
      "Epoch: 64 Cost: 63.514\n",
      "Epoch: 65 Cost: 62.0093\n",
      "Epoch: 66 Cost: 60.5629\n",
      "Epoch: 67 Cost: 59.1725\n",
      "Epoch: 68 Cost: 57.836\n",
      "Epoch: 69 Cost: 56.5511\n",
      "Epoch: 70 Cost: 55.3159\n",
      "Epoch: 71 Cost: 54.1284\n",
      "Epoch: 72 Cost: 52.9866\n",
      "Epoch: 73 Cost: 51.889\n",
      "Epoch: 74 Cost: 50.8335\n",
      "Epoch: 75 Cost: 49.8187\n",
      "Epoch: 76 Cost: 48.843\n",
      "Epoch: 77 Cost: 47.9047\n",
      "Epoch: 78 Cost: 47.0024\n",
      "Epoch: 79 Cost: 46.1348\n",
      "Epoch: 80 Cost: 45.3004\n",
      "Epoch: 81 Cost: 44.4979\n",
      "Epoch: 82 Cost: 43.7261\n",
      "Epoch: 83 Cost: 42.9838\n",
      "Epoch: 84 Cost: 42.2698\n",
      "Epoch: 85 Cost: 41.5831\n",
      "Epoch: 86 Cost: 40.9225\n",
      "Epoch: 87 Cost: 40.287\n",
      "Epoch: 88 Cost: 39.6757\n",
      "Epoch: 89 Cost: 39.0875\n",
      "Epoch: 90 Cost: 38.5217\n",
      "Epoch: 91 Cost: 37.9773\n",
      "Epoch: 92 Cost: 37.4534\n",
      "Epoch: 93 Cost: 36.9494\n",
      "Epoch: 94 Cost: 36.4643\n",
      "Epoch: 95 Cost: 35.9976\n",
      "Epoch: 96 Cost: 35.5484\n",
      "Epoch: 97 Cost: 35.116\n",
      "Epoch: 98 Cost: 34.6999\n",
      "Epoch: 99 Cost: 34.2993\n",
      "Epoch: 100 Cost: 33.9138\n",
      "Epoch: 101 Cost: 33.5426\n",
      "Epoch: 102 Cost: 33.1853\n",
      "Epoch: 103 Cost: 32.8412\n",
      "Epoch: 104 Cost: 32.51\n",
      "Epoch: 105 Cost: 32.191\n",
      "Epoch: 106 Cost: 31.8838\n",
      "Epoch: 107 Cost: 31.588\n",
      "Epoch: 108 Cost: 31.3031\n",
      "Epoch: 109 Cost: 31.0287\n",
      "Epoch: 110 Cost: 30.7643\n",
      "Epoch: 111 Cost: 30.5097\n",
      "Epoch: 112 Cost: 30.2643\n",
      "Epoch: 113 Cost: 30.0279\n",
      "Epoch: 114 Cost: 29.8002\n",
      "Epoch: 115 Cost: 29.5807\n",
      "Epoch: 116 Cost: 29.3692\n",
      "Epoch: 117 Cost: 29.1653\n",
      "Epoch: 118 Cost: 28.9688\n",
      "Epoch: 119 Cost: 28.7793\n",
      "Epoch: 120 Cost: 28.5967\n",
      "Epoch: 121 Cost: 28.4206\n",
      "Epoch: 122 Cost: 28.2508\n",
      "Epoch: 123 Cost: 28.087\n",
      "Epoch: 124 Cost: 27.9291\n",
      "Epoch: 125 Cost: 27.7768\n",
      "Epoch: 126 Cost: 27.6298\n",
      "Epoch: 127 Cost: 27.488\n",
      "Epoch: 128 Cost: 27.3512\n",
      "Epoch: 129 Cost: 27.2192\n",
      "Epoch: 130 Cost: 27.0917\n",
      "Epoch: 131 Cost: 26.9688\n",
      "Epoch: 132 Cost: 26.85\n",
      "Epoch: 133 Cost: 26.7354\n",
      "Epoch: 134 Cost: 26.6247\n",
      "Epoch: 135 Cost: 26.5178\n",
      "Epoch: 136 Cost: 26.4145\n",
      "Epoch: 137 Cost: 26.3148\n",
      "Epoch: 138 Cost: 26.2184\n",
      "Epoch: 139 Cost: 26.1253\n",
      "Epoch: 140 Cost: 26.0353\n",
      "Epoch: 141 Cost: 25.9484\n",
      "Epoch: 142 Cost: 25.8643\n",
      "Epoch: 143 Cost: 25.783\n",
      "Epoch: 144 Cost: 25.7044\n",
      "Epoch: 145 Cost: 25.6283\n",
      "Epoch: 146 Cost: 25.5548\n",
      "Epoch: 147 Cost: 25.4836\n",
      "Epoch: 148 Cost: 25.4148\n",
      "Epoch: 149 Cost: 25.3482\n",
      "Epoch: 150 Cost: 25.2837\n",
      "Epoch: 151 Cost: 25.2212\n",
      "Epoch: 152 Cost: 25.1608\n",
      "Epoch: 153 Cost: 25.1022\n",
      "Epoch: 154 Cost: 25.0455\n",
      "Epoch: 155 Cost: 24.9905\n",
      "Epoch: 156 Cost: 24.9373\n",
      "Epoch: 157 Cost: 24.8856\n",
      "Epoch: 158 Cost: 24.8356\n",
      "Epoch: 159 Cost: 24.7871\n",
      "Epoch: 160 Cost: 24.74\n",
      "Epoch: 161 Cost: 24.6944\n",
      "Epoch: 162 Cost: 24.6501\n",
      "Epoch: 163 Cost: 24.6071\n",
      "Epoch: 164 Cost: 24.5654\n",
      "Epoch: 165 Cost: 24.5249\n",
      "Epoch: 166 Cost: 24.4856\n",
      "Epoch: 167 Cost: 24.4474\n",
      "Epoch: 168 Cost: 24.4103\n",
      "Epoch: 169 Cost: 24.3743\n",
      "Epoch: 170 Cost: 24.3392\n",
      "Epoch: 171 Cost: 24.3052\n",
      "Epoch: 172 Cost: 24.2721\n",
      "Epoch: 173 Cost: 24.2399\n",
      "Epoch: 174 Cost: 24.2085\n",
      "Epoch: 175 Cost: 24.178\n",
      "Epoch: 176 Cost: 24.1484\n",
      "Epoch: 177 Cost: 24.1195\n",
      "Epoch: 178 Cost: 24.0914\n",
      "Epoch: 179 Cost: 24.064\n",
      "Epoch: 180 Cost: 24.0373\n",
      "Epoch: 181 Cost: 24.0113\n",
      "Epoch: 182 Cost: 23.9859\n",
      "Epoch: 183 Cost: 23.9612\n",
      "Epoch: 184 Cost: 23.9371\n",
      "Epoch: 185 Cost: 23.9136\n",
      "Epoch: 186 Cost: 23.8907\n",
      "Epoch: 187 Cost: 23.8683\n",
      "Epoch: 188 Cost: 23.8465\n",
      "Epoch: 189 Cost: 23.8252\n",
      "Epoch: 190 Cost: 23.8043\n",
      "Epoch: 191 Cost: 23.784\n",
      "Epoch: 192 Cost: 23.7641\n",
      "Epoch: 193 Cost: 23.7447\n",
      "Epoch: 194 Cost: 23.7257\n",
      "Epoch: 195 Cost: 23.7071\n",
      "Epoch: 196 Cost: 23.689\n",
      "Epoch: 197 Cost: 23.6712\n",
      "Epoch: 198 Cost: 23.6538\n",
      "Epoch: 199 Cost: 23.6368\n",
      "Epoch: 200 Cost: 23.6202\n",
      "Epoch: 201 Cost: 23.6038\n",
      "Epoch: 202 Cost: 23.5879\n",
      "Epoch: 203 Cost: 23.5722\n",
      "Epoch: 204 Cost: 23.5569\n",
      "Epoch: 205 Cost: 23.5418\n",
      "Epoch: 206 Cost: 23.5271\n",
      "Epoch: 207 Cost: 23.5126\n",
      "Epoch: 208 Cost: 23.4985\n",
      "Epoch: 209 Cost: 23.4845\n",
      "Epoch: 210 Cost: 23.4709\n",
      "Epoch: 211 Cost: 23.4575\n",
      "Epoch: 212 Cost: 23.4443\n",
      "Epoch: 213 Cost: 23.4314\n",
      "Epoch: 214 Cost: 23.4187\n",
      "Epoch: 215 Cost: 23.4063\n",
      "Epoch: 216 Cost: 23.394\n",
      "Epoch: 217 Cost: 23.382\n",
      "Epoch: 218 Cost: 23.3701\n",
      "Epoch: 219 Cost: 23.3585\n",
      "Epoch: 220 Cost: 23.3471\n",
      "Epoch: 221 Cost: 23.3358\n",
      "Epoch: 222 Cost: 23.3247\n",
      "Epoch: 223 Cost: 23.3138\n",
      "Epoch: 224 Cost: 23.3031\n",
      "Epoch: 225 Cost: 23.2926\n",
      "Epoch: 226 Cost: 23.2822\n",
      "Epoch: 227 Cost: 23.2719\n",
      "Epoch: 228 Cost: 23.2619\n",
      "Epoch: 229 Cost: 23.2519\n",
      "Epoch: 230 Cost: 23.2421\n",
      "Epoch: 231 Cost: 23.2325\n",
      "Epoch: 232 Cost: 23.223\n",
      "Epoch: 233 Cost: 23.2136\n",
      "Epoch: 234 Cost: 23.2044\n",
      "Epoch: 235 Cost: 23.1953\n",
      "Epoch: 236 Cost: 23.1863\n",
      "Epoch: 237 Cost: 23.1774\n",
      "Epoch: 238 Cost: 23.1687\n",
      "Epoch: 239 Cost: 23.16\n",
      "Epoch: 240 Cost: 23.1515\n",
      "Epoch: 241 Cost: 23.1431\n",
      "Epoch: 242 Cost: 23.1348\n",
      "Epoch: 243 Cost: 23.1266\n",
      "Epoch: 244 Cost: 23.1185\n",
      "Epoch: 245 Cost: 23.1105\n",
      "Epoch: 246 Cost: 23.1027\n",
      "Epoch: 247 Cost: 23.0949\n",
      "Epoch: 248 Cost: 23.0872\n",
      "Epoch: 249 Cost: 23.0795\n",
      "Epoch: 250 Cost: 23.072\n",
      "Epoch: 251 Cost: 23.0646\n",
      "Epoch: 252 Cost: 23.0572\n",
      "Epoch: 253 Cost: 23.05\n",
      "Epoch: 254 Cost: 23.0428\n",
      "Epoch: 255 Cost: 23.0357\n",
      "Epoch: 256 Cost: 23.0287\n",
      "Epoch: 257 Cost: 23.0217\n",
      "Epoch: 258 Cost: 23.0149\n",
      "Epoch: 259 Cost: 23.0081\n",
      "Epoch: 260 Cost: 23.0014\n",
      "Epoch: 261 Cost: 22.9947\n",
      "Epoch: 262 Cost: 22.9881\n",
      "Epoch: 263 Cost: 22.9816\n",
      "Epoch: 264 Cost: 22.9752\n",
      "Epoch: 265 Cost: 22.9688\n",
      "Epoch: 266 Cost: 22.9625\n",
      "Epoch: 267 Cost: 22.9562\n",
      "Epoch: 268 Cost: 22.95\n",
      "Epoch: 269 Cost: 22.9439\n",
      "Epoch: 270 Cost: 22.9379\n",
      "Epoch: 271 Cost: 22.9319\n",
      "Epoch: 272 Cost: 22.9259\n",
      "Epoch: 273 Cost: 22.92\n",
      "Epoch: 274 Cost: 22.9142\n",
      "Epoch: 275 Cost: 22.9084\n",
      "Epoch: 276 Cost: 22.9027\n",
      "Epoch: 277 Cost: 22.897\n",
      "Epoch: 278 Cost: 22.8914\n",
      "Epoch: 279 Cost: 22.8858\n",
      "Epoch: 280 Cost: 22.8803\n",
      "Epoch: 281 Cost: 22.8749\n",
      "Epoch: 282 Cost: 22.8694\n",
      "Epoch: 283 Cost: 22.8641\n",
      "Epoch: 284 Cost: 22.8588\n",
      "Epoch: 285 Cost: 22.8535\n",
      "Epoch: 286 Cost: 22.8483\n",
      "Epoch: 287 Cost: 22.8431\n",
      "Epoch: 288 Cost: 22.838\n",
      "Epoch: 289 Cost: 22.8329\n",
      "Epoch: 290 Cost: 22.8278\n",
      "Epoch: 291 Cost: 22.8228\n",
      "Epoch: 292 Cost: 22.8179\n",
      "Epoch: 293 Cost: 22.813\n",
      "Epoch: 294 Cost: 22.8081\n",
      "Epoch: 295 Cost: 22.8032\n",
      "Epoch: 296 Cost: 22.7985\n",
      "Epoch: 297 Cost: 22.7937\n",
      "Epoch: 298 Cost: 22.789\n",
      "Epoch: 299 Cost: 22.7843\n",
      "Epoch: 300 Cost: 22.7797\n",
      "Epoch: 301 Cost: 22.7751\n",
      "Epoch: 302 Cost: 22.7705\n",
      "Epoch: 303 Cost: 22.766\n",
      "Epoch: 304 Cost: 22.7615\n",
      "Epoch: 305 Cost: 22.7571\n",
      "Epoch: 306 Cost: 22.7527\n",
      "Epoch: 307 Cost: 22.7483\n",
      "Epoch: 308 Cost: 22.7439\n",
      "Epoch: 309 Cost: 22.7396\n",
      "Epoch: 310 Cost: 22.7353\n",
      "Epoch: 311 Cost: 22.7311\n",
      "Epoch: 312 Cost: 22.7269\n",
      "Epoch: 313 Cost: 22.7227\n",
      "Epoch: 314 Cost: 22.7186\n",
      "Epoch: 315 Cost: 22.7145\n",
      "Epoch: 316 Cost: 22.7104\n",
      "Epoch: 317 Cost: 22.7063\n",
      "Epoch: 318 Cost: 22.7023\n",
      "Epoch: 319 Cost: 22.6983\n",
      "Epoch: 320 Cost: 22.6944\n",
      "Epoch: 321 Cost: 22.6904\n",
      "Epoch: 322 Cost: 22.6865\n",
      "Epoch: 323 Cost: 22.6827\n",
      "Epoch: 324 Cost: 22.6788\n",
      "Epoch: 325 Cost: 22.675\n",
      "Epoch: 326 Cost: 22.6712\n",
      "Epoch: 327 Cost: 22.6675\n",
      "Epoch: 328 Cost: 22.6638\n",
      "Epoch: 329 Cost: 22.6601\n",
      "Epoch: 330 Cost: 22.6564\n",
      "Epoch: 331 Cost: 22.6527\n",
      "Epoch: 332 Cost: 22.6491\n",
      "Epoch: 333 Cost: 22.6455\n",
      "Epoch: 334 Cost: 22.642\n",
      "Epoch: 335 Cost: 22.6384\n",
      "Epoch: 336 Cost: 22.6349\n",
      "Epoch: 337 Cost: 22.6314\n",
      "Epoch: 338 Cost: 22.628\n",
      "Epoch: 339 Cost: 22.6245\n",
      "Epoch: 340 Cost: 22.6211\n",
      "Epoch: 341 Cost: 22.6177\n",
      "Epoch: 342 Cost: 22.6144\n",
      "Epoch: 343 Cost: 22.611\n",
      "Epoch: 344 Cost: 22.6077\n",
      "Epoch: 345 Cost: 22.6044\n",
      "Epoch: 346 Cost: 22.6012\n",
      "Epoch: 347 Cost: 22.5979\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 348 Cost: 22.5947\n",
      "Epoch: 349 Cost: 22.5915\n",
      "Epoch: 350 Cost: 22.5883\n",
      "Epoch: 351 Cost: 22.5852\n",
      "Epoch: 352 Cost: 22.582\n",
      "Epoch: 353 Cost: 22.5789\n",
      "Epoch: 354 Cost: 22.5758\n",
      "Epoch: 355 Cost: 22.5728\n",
      "Epoch: 356 Cost: 22.5697\n",
      "Epoch: 357 Cost: 22.5667\n",
      "Epoch: 358 Cost: 22.5637\n",
      "Epoch: 359 Cost: 22.5607\n",
      "Epoch: 360 Cost: 22.5578\n",
      "Epoch: 361 Cost: 22.5548\n",
      "Epoch: 362 Cost: 22.5519\n",
      "Epoch: 363 Cost: 22.549\n",
      "Epoch: 364 Cost: 22.5461\n",
      "Epoch: 365 Cost: 22.5433\n",
      "Epoch: 366 Cost: 22.5404\n",
      "Epoch: 367 Cost: 22.5376\n",
      "Epoch: 368 Cost: 22.5348\n",
      "Epoch: 369 Cost: 22.532\n",
      "Epoch: 370 Cost: 22.5292\n",
      "Epoch: 371 Cost: 22.5265\n",
      "Epoch: 372 Cost: 22.5238\n",
      "Epoch: 373 Cost: 22.5211\n",
      "Epoch: 374 Cost: 22.5184\n",
      "Epoch: 375 Cost: 22.5157\n",
      "Epoch: 376 Cost: 22.513\n",
      "Epoch: 377 Cost: 22.5104\n",
      "Epoch: 378 Cost: 22.5078\n",
      "Epoch: 379 Cost: 22.5052\n",
      "Epoch: 380 Cost: 22.5026\n",
      "Epoch: 381 Cost: 22.5\n",
      "Epoch: 382 Cost: 22.4975\n",
      "Epoch: 383 Cost: 22.495\n",
      "Epoch: 384 Cost: 22.4924\n",
      "Epoch: 385 Cost: 22.4899\n",
      "Epoch: 386 Cost: 22.4875\n",
      "Epoch: 387 Cost: 22.485\n",
      "Epoch: 388 Cost: 22.4825\n",
      "Epoch: 389 Cost: 22.4801\n",
      "Epoch: 390 Cost: 22.4777\n",
      "Epoch: 391 Cost: 22.4753\n",
      "Epoch: 392 Cost: 22.4729\n",
      "Epoch: 393 Cost: 22.4705\n",
      "Epoch: 394 Cost: 22.4682\n",
      "Epoch: 395 Cost: 22.4659\n",
      "Epoch: 396 Cost: 22.4635\n",
      "Epoch: 397 Cost: 22.4612\n",
      "Epoch: 398 Cost: 22.4589\n",
      "Epoch: 399 Cost: 22.4567\n",
      "Epoch: 400 Cost: 22.4544\n",
      "Epoch: 401 Cost: 22.4521\n",
      "Epoch: 402 Cost: 22.4499\n",
      "Epoch: 403 Cost: 22.4477\n",
      "Epoch: 404 Cost: 22.4455\n",
      "Epoch: 405 Cost: 22.4433\n",
      "Epoch: 406 Cost: 22.4411\n",
      "Epoch: 407 Cost: 22.439\n",
      "Epoch: 408 Cost: 22.4368\n",
      "Epoch: 409 Cost: 22.4347\n",
      "Epoch: 410 Cost: 22.4326\n",
      "Epoch: 411 Cost: 22.4305\n",
      "Epoch: 412 Cost: 22.4284\n",
      "Epoch: 413 Cost: 22.4263\n",
      "Epoch: 414 Cost: 22.4242\n",
      "Epoch: 415 Cost: 22.4222\n",
      "Epoch: 416 Cost: 22.4201\n",
      "Epoch: 417 Cost: 22.4181\n",
      "Epoch: 418 Cost: 22.4161\n",
      "Epoch: 419 Cost: 22.4141\n",
      "Epoch: 420 Cost: 22.4121\n",
      "Epoch: 421 Cost: 22.4101\n",
      "Epoch: 422 Cost: 22.4082\n",
      "Epoch: 423 Cost: 22.4062\n",
      "Epoch: 424 Cost: 22.4043\n",
      "Epoch: 425 Cost: 22.4023\n",
      "Epoch: 426 Cost: 22.4004\n",
      "Epoch: 427 Cost: 22.3985\n",
      "Epoch: 428 Cost: 22.3966\n",
      "Epoch: 429 Cost: 22.3948\n",
      "Epoch: 430 Cost: 22.3929\n",
      "Epoch: 431 Cost: 22.391\n",
      "Epoch: 432 Cost: 22.3892\n",
      "Epoch: 433 Cost: 22.3874\n",
      "Epoch: 434 Cost: 22.3856\n",
      "Epoch: 435 Cost: 22.3837\n",
      "Epoch: 436 Cost: 22.3819\n",
      "Epoch: 437 Cost: 22.3802\n",
      "Epoch: 438 Cost: 22.3784\n",
      "Epoch: 439 Cost: 22.3766\n",
      "Epoch: 440 Cost: 22.3749\n",
      "Epoch: 441 Cost: 22.3731\n",
      "Epoch: 442 Cost: 22.3714\n",
      "Epoch: 443 Cost: 22.3697\n",
      "Epoch: 444 Cost: 22.368\n",
      "Epoch: 445 Cost: 22.3663\n",
      "Epoch: 446 Cost: 22.3646\n",
      "Epoch: 447 Cost: 22.3629\n",
      "Epoch: 448 Cost: 22.3612\n",
      "Epoch: 449 Cost: 22.3596\n",
      "Epoch: 450 Cost: 22.3579\n",
      "Epoch: 451 Cost: 22.3563\n",
      "Epoch: 452 Cost: 22.3547\n",
      "Epoch: 453 Cost: 22.3531\n",
      "Epoch: 454 Cost: 22.3515\n",
      "Epoch: 455 Cost: 22.3499\n",
      "Epoch: 456 Cost: 22.3483\n",
      "Epoch: 457 Cost: 22.3467\n",
      "Epoch: 458 Cost: 22.3451\n",
      "Epoch: 459 Cost: 22.3436\n",
      "Epoch: 460 Cost: 22.342\n",
      "Epoch: 461 Cost: 22.3405\n",
      "Epoch: 462 Cost: 22.339\n",
      "Epoch: 463 Cost: 22.3374\n",
      "Epoch: 464 Cost: 22.3359\n",
      "Epoch: 465 Cost: 22.3344\n",
      "Epoch: 466 Cost: 22.3329\n",
      "Epoch: 467 Cost: 22.3314\n",
      "Epoch: 468 Cost: 22.33\n",
      "Epoch: 469 Cost: 22.3285\n",
      "Epoch: 470 Cost: 22.3271\n",
      "Epoch: 471 Cost: 22.3256\n",
      "Epoch: 472 Cost: 22.3242\n",
      "Epoch: 473 Cost: 22.3227\n",
      "Epoch: 474 Cost: 22.3213\n",
      "Epoch: 475 Cost: 22.3199\n",
      "Epoch: 476 Cost: 22.3185\n",
      "Epoch: 477 Cost: 22.3171\n",
      "Epoch: 478 Cost: 22.3157\n",
      "Epoch: 479 Cost: 22.3143\n",
      "Epoch: 480 Cost: 22.3129\n",
      "Epoch: 481 Cost: 22.3116\n",
      "Epoch: 482 Cost: 22.3102\n",
      "Epoch: 483 Cost: 22.3089\n",
      "Epoch: 484 Cost: 22.3075\n",
      "Epoch: 485 Cost: 22.3062\n",
      "Epoch: 486 Cost: 22.3049\n",
      "Epoch: 487 Cost: 22.3036\n",
      "Epoch: 488 Cost: 22.3023\n",
      "Epoch: 489 Cost: 22.301\n",
      "Epoch: 490 Cost: 22.2997\n",
      "Epoch: 491 Cost: 22.2984\n",
      "Epoch: 492 Cost: 22.2971\n",
      "Epoch: 493 Cost: 22.2958\n",
      "Epoch: 494 Cost: 22.2946\n",
      "Epoch: 495 Cost: 22.2933\n",
      "Epoch: 496 Cost: 22.2921\n",
      "Epoch: 497 Cost: 22.2908\n",
      "Epoch: 498 Cost: 22.2896\n",
      "Epoch: 499 Cost: 22.2884\n",
      "Epoch: 500 Cost: 22.2872\n",
      "Epoch: 501 Cost: 22.2859\n",
      "Epoch: 502 Cost: 22.2847\n",
      "Epoch: 503 Cost: 22.2835\n",
      "Epoch: 504 Cost: 22.2824\n",
      "Epoch: 505 Cost: 22.2812\n",
      "Epoch: 506 Cost: 22.28\n",
      "Epoch: 507 Cost: 22.2788\n",
      "Epoch: 508 Cost: 22.2777\n",
      "Epoch: 509 Cost: 22.2765\n",
      "Epoch: 510 Cost: 22.2754\n",
      "Epoch: 511 Cost: 22.2742\n",
      "Epoch: 512 Cost: 22.2731\n",
      "Epoch: 513 Cost: 22.2719\n",
      "Epoch: 514 Cost: 22.2708\n",
      "Epoch: 515 Cost: 22.2697\n",
      "Epoch: 516 Cost: 22.2686\n",
      "Epoch: 517 Cost: 22.2675\n",
      "Epoch: 518 Cost: 22.2664\n",
      "Epoch: 519 Cost: 22.2653\n",
      "Epoch: 520 Cost: 22.2642\n",
      "Epoch: 521 Cost: 22.2631\n",
      "Epoch: 522 Cost: 22.2621\n",
      "Epoch: 523 Cost: 22.261\n",
      "Epoch: 524 Cost: 22.2599\n",
      "Epoch: 525 Cost: 22.2589\n",
      "Epoch: 526 Cost: 22.2578\n",
      "Epoch: 527 Cost: 22.2568\n",
      "Epoch: 528 Cost: 22.2557\n",
      "Epoch: 529 Cost: 22.2547\n",
      "Epoch: 530 Cost: 22.2537\n",
      "Epoch: 531 Cost: 22.2527\n",
      "Epoch: 532 Cost: 22.2517\n",
      "Epoch: 533 Cost: 22.2507\n",
      "Epoch: 534 Cost: 22.2496\n",
      "Epoch: 535 Cost: 22.2487\n",
      "Epoch: 536 Cost: 22.2477\n",
      "Epoch: 537 Cost: 22.2467\n",
      "Epoch: 538 Cost: 22.2457\n",
      "Epoch: 539 Cost: 22.2447\n",
      "Epoch: 540 Cost: 22.2438\n",
      "Epoch: 541 Cost: 22.2428\n",
      "Epoch: 542 Cost: 22.2418\n",
      "Epoch: 543 Cost: 22.2409\n",
      "Epoch: 544 Cost: 22.2399\n",
      "Epoch: 545 Cost: 22.239\n",
      "Epoch: 546 Cost: 22.2381\n",
      "Epoch: 547 Cost: 22.2371\n",
      "Epoch: 548 Cost: 22.2362\n",
      "Epoch: 549 Cost: 22.2353\n",
      "Epoch: 550 Cost: 22.2344\n",
      "Epoch: 551 Cost: 22.2334\n",
      "Epoch: 552 Cost: 22.2325\n",
      "Epoch: 553 Cost: 22.2316\n",
      "Epoch: 554 Cost: 22.2307\n",
      "Epoch: 555 Cost: 22.2298\n",
      "Epoch: 556 Cost: 22.229\n",
      "Epoch: 557 Cost: 22.2281\n",
      "Epoch: 558 Cost: 22.2272\n",
      "Epoch: 559 Cost: 22.2263\n",
      "Epoch: 560 Cost: 22.2255\n",
      "Epoch: 561 Cost: 22.2246\n",
      "Epoch: 562 Cost: 22.2237\n",
      "Epoch: 563 Cost: 22.2229\n",
      "Epoch: 564 Cost: 22.222\n",
      "Epoch: 565 Cost: 22.2212\n",
      "Epoch: 566 Cost: 22.2204\n",
      "Epoch: 567 Cost: 22.2195\n",
      "Epoch: 568 Cost: 22.2187\n",
      "Epoch: 569 Cost: 22.2179\n",
      "Epoch: 570 Cost: 22.217\n",
      "Epoch: 571 Cost: 22.2162\n",
      "Epoch: 572 Cost: 22.2154\n",
      "Epoch: 573 Cost: 22.2146\n",
      "Epoch: 574 Cost: 22.2138\n",
      "Epoch: 575 Cost: 22.213\n",
      "Epoch: 576 Cost: 22.2122\n",
      "Epoch: 577 Cost: 22.2114\n",
      "Epoch: 578 Cost: 22.2106\n",
      "Epoch: 579 Cost: 22.2099\n",
      "Epoch: 580 Cost: 22.2091\n",
      "Epoch: 581 Cost: 22.2083\n",
      "Epoch: 582 Cost: 22.2075\n",
      "Epoch: 583 Cost: 22.2068\n",
      "Epoch: 584 Cost: 22.206\n",
      "Epoch: 585 Cost: 22.2052\n",
      "Epoch: 586 Cost: 22.2045\n",
      "Epoch: 587 Cost: 22.2037\n",
      "Epoch: 588 Cost: 22.203\n",
      "Epoch: 589 Cost: 22.2023\n",
      "Epoch: 590 Cost: 22.2015\n",
      "Epoch: 591 Cost: 22.2008\n",
      "Epoch: 592 Cost: 22.2001\n",
      "Epoch: 593 Cost: 22.1993\n",
      "Epoch: 594 Cost: 22.1986\n",
      "Epoch: 595 Cost: 22.1979\n",
      "Epoch: 596 Cost: 22.1972\n",
      "Epoch: 597 Cost: 22.1965\n",
      "Epoch: 598 Cost: 22.1958\n",
      "Epoch: 599 Cost: 22.1951\n",
      "Epoch: 600 Cost: 22.1944\n",
      "Epoch: 601 Cost: 22.1937\n",
      "Epoch: 602 Cost: 22.193\n",
      "Epoch: 603 Cost: 22.1923\n",
      "Epoch: 604 Cost: 22.1916\n",
      "Epoch: 605 Cost: 22.1909\n",
      "Epoch: 606 Cost: 22.1902\n",
      "Epoch: 607 Cost: 22.1896\n",
      "Epoch: 608 Cost: 22.1889\n",
      "Epoch: 609 Cost: 22.1882\n",
      "Epoch: 610 Cost: 22.1876\n",
      "Epoch: 611 Cost: 22.1869\n",
      "Epoch: 612 Cost: 22.1862\n",
      "Epoch: 613 Cost: 22.1856\n",
      "Epoch: 614 Cost: 22.1849\n",
      "Epoch: 615 Cost: 22.1843\n",
      "Epoch: 616 Cost: 22.1836\n",
      "Epoch: 617 Cost: 22.183\n",
      "Epoch: 618 Cost: 22.1824\n",
      "Epoch: 619 Cost: 22.1817\n",
      "Epoch: 620 Cost: 22.1811\n",
      "Epoch: 621 Cost: 22.1805\n",
      "Epoch: 622 Cost: 22.1799\n",
      "Epoch: 623 Cost: 22.1792\n",
      "Epoch: 624 Cost: 22.1786\n",
      "Epoch: 625 Cost: 22.178\n",
      "Epoch: 626 Cost: 22.1774\n",
      "Epoch: 627 Cost: 22.1768\n",
      "Epoch: 628 Cost: 22.1762\n",
      "Epoch: 629 Cost: 22.1756\n",
      "Epoch: 630 Cost: 22.175\n",
      "Epoch: 631 Cost: 22.1744\n",
      "Epoch: 632 Cost: 22.1738\n",
      "Epoch: 633 Cost: 22.1732\n",
      "Epoch: 634 Cost: 22.1726\n",
      "Epoch: 635 Cost: 22.172\n",
      "Epoch: 636 Cost: 22.1714\n",
      "Epoch: 637 Cost: 22.1709\n",
      "Epoch: 638 Cost: 22.1703\n",
      "Epoch: 639 Cost: 22.1697\n",
      "Epoch: 640 Cost: 22.1691\n",
      "Epoch: 641 Cost: 22.1686\n",
      "Epoch: 642 Cost: 22.168\n",
      "Epoch: 643 Cost: 22.1674\n",
      "Epoch: 644 Cost: 22.1669\n",
      "Epoch: 645 Cost: 22.1663\n",
      "Epoch: 646 Cost: 22.1658\n",
      "Epoch: 647 Cost: 22.1652\n",
      "Epoch: 648 Cost: 22.1647\n",
      "Epoch: 649 Cost: 22.1641\n",
      "Epoch: 650 Cost: 22.1636\n",
      "Epoch: 651 Cost: 22.1631\n",
      "Epoch: 652 Cost: 22.1625\n",
      "Epoch: 653 Cost: 22.162\n",
      "Epoch: 654 Cost: 22.1615\n",
      "Epoch: 655 Cost: 22.1609\n",
      "Epoch: 656 Cost: 22.1604\n",
      "Epoch: 657 Cost: 22.1599\n",
      "Epoch: 658 Cost: 22.1594\n",
      "Epoch: 659 Cost: 22.1588\n",
      "Epoch: 660 Cost: 22.1583\n",
      "Epoch: 661 Cost: 22.1578\n",
      "Epoch: 662 Cost: 22.1573\n",
      "Epoch: 663 Cost: 22.1568\n",
      "Epoch: 664 Cost: 22.1563\n",
      "Epoch: 665 Cost: 22.1558\n",
      "Epoch: 666 Cost: 22.1553\n",
      "Epoch: 667 Cost: 22.1548\n",
      "Epoch: 668 Cost: 22.1543\n",
      "Epoch: 669 Cost: 22.1538\n",
      "Epoch: 670 Cost: 22.1533\n",
      "Epoch: 671 Cost: 22.1528\n",
      "Epoch: 672 Cost: 22.1523\n",
      "Epoch: 673 Cost: 22.1518\n",
      "Epoch: 674 Cost: 22.1514\n",
      "Epoch: 675 Cost: 22.1509\n",
      "Epoch: 676 Cost: 22.1504\n",
      "Epoch: 677 Cost: 22.1499\n",
      "Epoch: 678 Cost: 22.1495\n",
      "Epoch: 679 Cost: 22.149\n",
      "Epoch: 680 Cost: 22.1485\n",
      "Epoch: 681 Cost: 22.148\n",
      "Epoch: 682 Cost: 22.1476\n",
      "Epoch: 683 Cost: 22.1471\n",
      "Epoch: 684 Cost: 22.1467\n",
      "Epoch: 685 Cost: 22.1462\n",
      "Epoch: 686 Cost: 22.1458\n",
      "Epoch: 687 Cost: 22.1453\n",
      "Epoch: 688 Cost: 22.1448\n",
      "Epoch: 689 Cost: 22.1444\n",
      "Epoch: 690 Cost: 22.1439\n",
      "Epoch: 691 Cost: 22.1435\n",
      "Epoch: 692 Cost: 22.1431\n",
      "Epoch: 693 Cost: 22.1426\n",
      "Epoch: 694 Cost: 22.1422\n",
      "Epoch: 695 Cost: 22.1417\n",
      "Epoch: 696 Cost: 22.1413\n",
      "Epoch: 697 Cost: 22.1409\n",
      "Epoch: 698 Cost: 22.1405\n",
      "Epoch: 699 Cost: 22.14\n",
      "Epoch: 700 Cost: 22.1396\n",
      "Epoch: 701 Cost: 22.1392\n",
      "Epoch: 702 Cost: 22.1387\n",
      "Epoch: 703 Cost: 22.1383\n",
      "Epoch: 704 Cost: 22.1379\n",
      "Epoch: 705 Cost: 22.1375\n",
      "Epoch: 706 Cost: 22.1371\n",
      "Epoch: 707 Cost: 22.1367\n",
      "Epoch: 708 Cost: 22.1363\n",
      "Epoch: 709 Cost: 22.1359\n",
      "Epoch: 710 Cost: 22.1354\n",
      "Epoch: 711 Cost: 22.135\n",
      "Epoch: 712 Cost: 22.1346\n",
      "Epoch: 713 Cost: 22.1342\n",
      "Epoch: 714 Cost: 22.1338\n",
      "Epoch: 715 Cost: 22.1334\n",
      "Epoch: 716 Cost: 22.133\n",
      "Epoch: 717 Cost: 22.1326\n",
      "Epoch: 718 Cost: 22.1323\n",
      "Epoch: 719 Cost: 22.1319\n",
      "Epoch: 720 Cost: 22.1315\n",
      "Epoch: 721 Cost: 22.1311\n",
      "Epoch: 722 Cost: 22.1307\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 723 Cost: 22.1303\n",
      "Epoch: 724 Cost: 22.1299\n",
      "Epoch: 725 Cost: 22.1296\n",
      "Epoch: 726 Cost: 22.1292\n",
      "Epoch: 727 Cost: 22.1288\n",
      "Epoch: 728 Cost: 22.1284\n",
      "Epoch: 729 Cost: 22.1281\n",
      "Epoch: 730 Cost: 22.1277\n",
      "Epoch: 731 Cost: 22.1273\n",
      "Epoch: 732 Cost: 22.1269\n",
      "Epoch: 733 Cost: 22.1266\n",
      "Epoch: 734 Cost: 22.1262\n",
      "Epoch: 735 Cost: 22.1259\n",
      "Epoch: 736 Cost: 22.1255\n",
      "Epoch: 737 Cost: 22.1251\n",
      "Epoch: 738 Cost: 22.1248\n",
      "Epoch: 739 Cost: 22.1244\n",
      "Epoch: 740 Cost: 22.1241\n",
      "Epoch: 741 Cost: 22.1237\n",
      "Epoch: 742 Cost: 22.1234\n",
      "Epoch: 743 Cost: 22.123\n",
      "Epoch: 744 Cost: 22.1227\n",
      "Epoch: 745 Cost: 22.1223\n",
      "Epoch: 746 Cost: 22.122\n",
      "Epoch: 747 Cost: 22.1216\n",
      "Epoch: 748 Cost: 22.1213\n",
      "Epoch: 749 Cost: 22.1209\n",
      "Epoch: 750 Cost: 22.1206\n",
      "Epoch: 751 Cost: 22.1203\n",
      "Epoch: 752 Cost: 22.1199\n",
      "Epoch: 753 Cost: 22.1196\n",
      "Epoch: 754 Cost: 22.1193\n",
      "Epoch: 755 Cost: 22.1189\n",
      "Epoch: 756 Cost: 22.1186\n",
      "Epoch: 757 Cost: 22.1183\n",
      "Epoch: 758 Cost: 22.1179\n",
      "Epoch: 759 Cost: 22.1176\n",
      "Epoch: 760 Cost: 22.1173\n",
      "Epoch: 761 Cost: 22.117\n",
      "Epoch: 762 Cost: 22.1166\n",
      "Epoch: 763 Cost: 22.1163\n",
      "Epoch: 764 Cost: 22.116\n",
      "Epoch: 765 Cost: 22.1157\n",
      "Epoch: 766 Cost: 22.1154\n",
      "Epoch: 767 Cost: 22.1151\n",
      "Epoch: 768 Cost: 22.1147\n",
      "Epoch: 769 Cost: 22.1144\n",
      "Epoch: 770 Cost: 22.1141\n",
      "Epoch: 771 Cost: 22.1138\n",
      "Epoch: 772 Cost: 22.1135\n",
      "Epoch: 773 Cost: 22.1132\n",
      "Epoch: 774 Cost: 22.1129\n",
      "Epoch: 775 Cost: 22.1126\n",
      "Epoch: 776 Cost: 22.1123\n",
      "Epoch: 777 Cost: 22.112\n",
      "Epoch: 778 Cost: 22.1117\n",
      "Epoch: 779 Cost: 22.1114\n",
      "Epoch: 780 Cost: 22.1111\n",
      "Epoch: 781 Cost: 22.1108\n",
      "Epoch: 782 Cost: 22.1105\n",
      "Epoch: 783 Cost: 22.1102\n",
      "Epoch: 784 Cost: 22.1099\n",
      "Epoch: 785 Cost: 22.1096\n",
      "Epoch: 786 Cost: 22.1093\n",
      "Epoch: 787 Cost: 22.109\n",
      "Epoch: 788 Cost: 22.1087\n",
      "Epoch: 789 Cost: 22.1084\n",
      "Epoch: 790 Cost: 22.1082\n",
      "Epoch: 791 Cost: 22.1079\n",
      "Epoch: 792 Cost: 22.1076\n",
      "Epoch: 793 Cost: 22.1073\n",
      "Epoch: 794 Cost: 22.107\n",
      "Epoch: 795 Cost: 22.1067\n",
      "Epoch: 796 Cost: 22.1065\n",
      "Epoch: 797 Cost: 22.1062\n",
      "Epoch: 798 Cost: 22.1059\n",
      "Epoch: 799 Cost: 22.1056\n",
      "Epoch: 800 Cost: 22.1054\n",
      "Epoch: 801 Cost: 22.1051\n",
      "Epoch: 802 Cost: 22.1048\n",
      "Epoch: 803 Cost: 22.1045\n",
      "Epoch: 804 Cost: 22.1043\n",
      "Epoch: 805 Cost: 22.104\n",
      "Epoch: 806 Cost: 22.1037\n",
      "Epoch: 807 Cost: 22.1035\n",
      "Epoch: 808 Cost: 22.1032\n",
      "Epoch: 809 Cost: 22.1029\n",
      "Epoch: 810 Cost: 22.1027\n",
      "Epoch: 811 Cost: 22.1024\n",
      "Epoch: 812 Cost: 22.1022\n",
      "Epoch: 813 Cost: 22.1019\n",
      "Epoch: 814 Cost: 22.1016\n",
      "Epoch: 815 Cost: 22.1014\n",
      "Epoch: 816 Cost: 22.1011\n",
      "Epoch: 817 Cost: 22.1009\n",
      "Epoch: 818 Cost: 22.1006\n",
      "Epoch: 819 Cost: 22.1004\n",
      "Epoch: 820 Cost: 22.1001\n",
      "Epoch: 821 Cost: 22.0999\n",
      "Epoch: 822 Cost: 22.0996\n",
      "Epoch: 823 Cost: 22.0994\n",
      "Epoch: 824 Cost: 22.0991\n",
      "Epoch: 825 Cost: 22.0989\n",
      "Epoch: 826 Cost: 22.0986\n",
      "Epoch: 827 Cost: 22.0984\n",
      "Epoch: 828 Cost: 22.0981\n",
      "Epoch: 829 Cost: 22.0979\n",
      "Epoch: 830 Cost: 22.0976\n",
      "Epoch: 831 Cost: 22.0974\n",
      "Epoch: 832 Cost: 22.0972\n",
      "Epoch: 833 Cost: 22.0969\n",
      "Epoch: 834 Cost: 22.0967\n",
      "Epoch: 835 Cost: 22.0964\n",
      "Epoch: 836 Cost: 22.0962\n",
      "Epoch: 837 Cost: 22.096\n",
      "Epoch: 838 Cost: 22.0957\n",
      "Epoch: 839 Cost: 22.0955\n",
      "Epoch: 840 Cost: 22.0953\n",
      "Epoch: 841 Cost: 22.095\n",
      "Epoch: 842 Cost: 22.0948\n",
      "Epoch: 843 Cost: 22.0946\n",
      "Epoch: 844 Cost: 22.0943\n",
      "Epoch: 845 Cost: 22.0941\n",
      "Epoch: 846 Cost: 22.0939\n",
      "Epoch: 847 Cost: 22.0937\n",
      "Epoch: 848 Cost: 22.0934\n",
      "Epoch: 849 Cost: 22.0932\n",
      "Epoch: 850 Cost: 22.093\n",
      "Epoch: 851 Cost: 22.0928\n",
      "Epoch: 852 Cost: 22.0925\n",
      "Epoch: 853 Cost: 22.0923\n",
      "Epoch: 854 Cost: 22.0921\n",
      "Epoch: 855 Cost: 22.0919\n",
      "Epoch: 856 Cost: 22.0917\n",
      "Epoch: 857 Cost: 22.0914\n",
      "Epoch: 858 Cost: 22.0912\n",
      "Epoch: 859 Cost: 22.091\n",
      "Epoch: 860 Cost: 22.0908\n",
      "Epoch: 861 Cost: 22.0906\n",
      "Epoch: 862 Cost: 22.0904\n",
      "Epoch: 863 Cost: 22.0901\n",
      "Epoch: 864 Cost: 22.0899\n",
      "Epoch: 865 Cost: 22.0897\n",
      "Epoch: 866 Cost: 22.0895\n",
      "Epoch: 867 Cost: 22.0893\n",
      "Epoch: 868 Cost: 22.0891\n",
      "Epoch: 869 Cost: 22.0889\n",
      "Epoch: 870 Cost: 22.0887\n",
      "Epoch: 871 Cost: 22.0885\n",
      "Epoch: 872 Cost: 22.0883\n",
      "Epoch: 873 Cost: 22.0881\n",
      "Epoch: 874 Cost: 22.0879\n",
      "Epoch: 875 Cost: 22.0877\n",
      "Epoch: 876 Cost: 22.0874\n",
      "Epoch: 877 Cost: 22.0872\n",
      "Epoch: 878 Cost: 22.087\n",
      "Epoch: 879 Cost: 22.0868\n",
      "Epoch: 880 Cost: 22.0866\n",
      "Epoch: 881 Cost: 22.0864\n",
      "Epoch: 882 Cost: 22.0862\n",
      "Epoch: 883 Cost: 22.0861\n",
      "Epoch: 884 Cost: 22.0859\n",
      "Epoch: 885 Cost: 22.0857\n",
      "Epoch: 886 Cost: 22.0855\n",
      "Epoch: 887 Cost: 22.0853\n",
      "Epoch: 888 Cost: 22.0851\n",
      "Epoch: 889 Cost: 22.0849\n",
      "Epoch: 890 Cost: 22.0847\n",
      "Epoch: 891 Cost: 22.0845\n",
      "Epoch: 892 Cost: 22.0843\n",
      "Epoch: 893 Cost: 22.0841\n",
      "Epoch: 894 Cost: 22.0839\n",
      "Epoch: 895 Cost: 22.0837\n",
      "Epoch: 896 Cost: 22.0835\n",
      "Epoch: 897 Cost: 22.0834\n",
      "Epoch: 898 Cost: 22.0832\n",
      "Epoch: 899 Cost: 22.083\n",
      "Epoch: 900 Cost: 22.0828\n",
      "Epoch: 901 Cost: 22.0826\n",
      "Epoch: 902 Cost: 22.0824\n",
      "Epoch: 903 Cost: 22.0823\n",
      "Epoch: 904 Cost: 22.0821\n",
      "Epoch: 905 Cost: 22.0819\n",
      "Epoch: 906 Cost: 22.0817\n",
      "Epoch: 907 Cost: 22.0815\n",
      "Epoch: 908 Cost: 22.0813\n",
      "Epoch: 909 Cost: 22.0812\n",
      "Epoch: 910 Cost: 22.081\n",
      "Epoch: 911 Cost: 22.0808\n",
      "Epoch: 912 Cost: 22.0806\n",
      "Epoch: 913 Cost: 22.0805\n",
      "Epoch: 914 Cost: 22.0803\n",
      "Epoch: 915 Cost: 22.0801\n",
      "Epoch: 916 Cost: 22.0799\n",
      "Epoch: 917 Cost: 22.0798\n",
      "Epoch: 918 Cost: 22.0796\n",
      "Epoch: 919 Cost: 22.0794\n",
      "Epoch: 920 Cost: 22.0792\n",
      "Epoch: 921 Cost: 22.0791\n",
      "Epoch: 922 Cost: 22.0789\n",
      "Epoch: 923 Cost: 22.0787\n",
      "Epoch: 924 Cost: 22.0786\n",
      "Epoch: 925 Cost: 22.0784\n",
      "Epoch: 926 Cost: 22.0782\n",
      "Epoch: 927 Cost: 22.078\n",
      "Epoch: 928 Cost: 22.0779\n",
      "Epoch: 929 Cost: 22.0777\n",
      "Epoch: 930 Cost: 22.0775\n",
      "Epoch: 931 Cost: 22.0774\n",
      "Epoch: 932 Cost: 22.0772\n",
      "Epoch: 933 Cost: 22.077\n",
      "Epoch: 934 Cost: 22.0769\n",
      "Epoch: 935 Cost: 22.0767\n",
      "Epoch: 936 Cost: 22.0766\n",
      "Epoch: 937 Cost: 22.0764\n",
      "Epoch: 938 Cost: 22.0762\n",
      "Epoch: 939 Cost: 22.0761\n",
      "Epoch: 940 Cost: 22.0759\n",
      "Epoch: 941 Cost: 22.0758\n",
      "Epoch: 942 Cost: 22.0756\n",
      "Epoch: 943 Cost: 22.0754\n",
      "Epoch: 944 Cost: 22.0753\n",
      "Epoch: 945 Cost: 22.0751\n",
      "Epoch: 946 Cost: 22.075\n",
      "Epoch: 947 Cost: 22.0748\n",
      "Epoch: 948 Cost: 22.0747\n",
      "Epoch: 949 Cost: 22.0745\n",
      "Epoch: 950 Cost: 22.0743\n",
      "Epoch: 951 Cost: 22.0742\n",
      "Epoch: 952 Cost: 22.074\n",
      "Epoch: 953 Cost: 22.0739\n",
      "Epoch: 954 Cost: 22.0737\n",
      "Epoch: 955 Cost: 22.0736\n",
      "Epoch: 956 Cost: 22.0734\n",
      "Epoch: 957 Cost: 22.0733\n",
      "Epoch: 958 Cost: 22.0731\n",
      "Epoch: 959 Cost: 22.073\n",
      "Epoch: 960 Cost: 22.0728\n",
      "Epoch: 961 Cost: 22.0727\n",
      "Epoch: 962 Cost: 22.0725\n",
      "Epoch: 963 Cost: 22.0724\n",
      "Epoch: 964 Cost: 22.0722\n",
      "Epoch: 965 Cost: 22.0721\n",
      "Epoch: 966 Cost: 22.0719\n",
      "Epoch: 967 Cost: 22.0718\n",
      "Epoch: 968 Cost: 22.0717\n",
      "Epoch: 969 Cost: 22.0715\n",
      "Epoch: 970 Cost: 22.0714\n",
      "Epoch: 971 Cost: 22.0712\n",
      "Epoch: 972 Cost: 22.0711\n",
      "Epoch: 973 Cost: 22.0709\n",
      "Epoch: 974 Cost: 22.0708\n",
      "Epoch: 975 Cost: 22.0707\n",
      "Epoch: 976 Cost: 22.0705\n",
      "Epoch: 977 Cost: 22.0704\n",
      "Epoch: 978 Cost: 22.0702\n",
      "Epoch: 979 Cost: 22.0701\n",
      "Epoch: 980 Cost: 22.0699\n",
      "Epoch: 981 Cost: 22.0698\n",
      "Epoch: 982 Cost: 22.0697\n",
      "Epoch: 983 Cost: 22.0695\n",
      "Epoch: 984 Cost: 22.0694\n",
      "Epoch: 985 Cost: 22.0693\n",
      "Epoch: 986 Cost: 22.0691\n",
      "Epoch: 987 Cost: 22.069\n",
      "Epoch: 988 Cost: 22.0688\n",
      "Epoch: 989 Cost: 22.0687\n",
      "Epoch: 990 Cost: 22.0686\n",
      "Epoch: 991 Cost: 22.0684\n",
      "Epoch: 992 Cost: 22.0683\n",
      "Epoch: 993 Cost: 22.0682\n",
      "Epoch: 994 Cost: 22.068\n",
      "Epoch: 995 Cost: 22.0679\n",
      "Epoch: 996 Cost: 22.0678\n",
      "Epoch: 997 Cost: 22.0676\n",
      "Epoch: 998 Cost: 22.0675\n",
      "Epoch: 999 Cost: 22.0674\n",
      "Test Loss: 22.3642\n"
     ]
    }
   ],
   "source": [
    "num_epochs  = 1000\n",
    "losses = []\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for epoch in range(num_epochs):\n",
    "        _, c = sess.run([update, loss], feed_dict={X: X_train, \n",
    "                                                      Y: y_train})\n",
    "        losses.append(c)\n",
    "        print(\"Epoch:\",epoch,\"Cost:\",c)\n",
    "    test_loss = sess.run(loss, feed_dict={X: X_test, \n",
    "                                          Y: y_test})\n",
    "    print(\"Test Loss:\",test_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x182cece390>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGhVJREFUeJzt3WtsXOd95/Hvf268iiJpUTItyqVsq77AjWMtq8h10TZWksaOt3LRGEha1IIjVEDr7rrbAlmnfVEEW2wboKkTt4VRN3YiF2kSr5LUquEm8cpOu20TxVTsKHZkW4ykWIxkkbqRuvDO/744z0gjcsgZUjMczZnfBxicc57zzJnn6Ai/eficy5i7IyIi8ZWodANERKS8FPQiIjGnoBcRiTkFvYhIzCnoRURiTkEvIhJzCnoRkZhT0IuIxJyCXkQk5lKVbgDAihUrvLu7u9LNEBGpKnv27Dnu7h2F6l0RQd/d3U1vb2+lmyEiUlXM7CfF1NPQjYhIzCnoRURiTkEvIhJzCnoRkZhT0IuIxJyCXkQk5ooKejNrNbMdZvaGme0zszvMrN3MXjCz/WHaFuqamT1mZn1mttfM1pd3F0REZD7F9ug/C3zD3W8CbgP2AY8Au9x9HbArLAPcDawLr23A4yVtcY6XD53k0996k8mp6XJ9hIhI1SsY9GbWAvwS8CSAu4+7+2lgM7A9VNsO3BfmNwNPe+S7QKuZdZa85cArb5/ir1/sY2xSQS8iMpdievTXAYPA583sFTP7nJk1Aavc/ShAmK4M9VcDh3Pe3x/KSi6djJo/rqAXEZlTMUGfAtYDj7v77cA5Lg7T5GN5ynxWJbNtZtZrZr2Dg4NFNXambNBPaOhGRGROxQR9P9Dv7rvD8g6i4D+WHZIJ04Gc+mty3t8FHJm5UXd/wt173L2no6PgM3nyyqSi5mvoRkRkbgWD3t3fAQ6b2Y2haBPwI2AnsCWUbQGeDfM7gQfC1TcbgaHsEE+pZdSjFxEpqNinV/434ItmlgEOAA8SfUk8Y2ZbgbeB+0Pd54F7gD7gfKhbFtke/cTUrJEhEREJigp6d38V6MmzalOeug48dJntKopOxoqIFFbVd8Zme/TjGroREZlTVQd9Ohld4KMevYjI3Ko66HUyVkSksOoO+pTG6EVECqnqoNcNUyIihVV10OtkrIhIYdUd9Lq8UkSkoOoOet0wJSJSUFUH/cUbpqYq3BIRkStXlQd9dB29evQiInOr6qDXyVgRkcKqOujTCZ2MFREppKqDPpEw0knTdfQiIvOo6qCH6ISsevQiInOr+qDPpBLq0YuIzKPqgz6dTOhkrIjIPKo+6DPJBOOTurxSRGQu1R/0KfXoRUTmU/VBX5dKMDqhO2NFROZS/UGfTjKmq25EROZU9UFfrx69iMi8qj/o00nGFPQiInOq+qCPxug1dCMiMpeqD/r6dJIxPaZYRGROMQh69ehFROZTVNCb2SEz+6GZvWpmvaGs3cxeMLP9YdoWys3MHjOzPjPba2bry7kD9ekko+rRi4jMaSE9+ve6+7vdvScsPwLscvd1wK6wDHA3sC68tgGPl6qx+dSnk7rqRkRkHpczdLMZ2B7mtwP35ZQ/7ZHvAq1m1nkZnzOv7MlYdz0GQUQkn2KD3oFvmdkeM9sWyla5+1GAMF0ZylcDh3Pe2x/KyqI+nQT0K1MiInNJFVnvTnc/YmYrgRfM7I156lqeslnd7fCFsQ3g2muvLbIZs9WFnxMcnZimLpVc9HZEROKqqB69ux8J0wHg68AG4Fh2SCZMB0L1fmBNztu7gCN5tvmEu/e4e09HR8eidyDbo9dNUyIi+RUMejNrMrNl2XngA8BrwE5gS6i2BXg2zO8EHghX32wEhrJDPOWQ26MXEZHZihm6WQV83cyy9f/R3b9hZi8Dz5jZVuBt4P5Q/3ngHqAPOA88WPJW57jQo9clliIieRUMenc/ANyWp/wEsClPuQMPlaR1RcgGvXr0IiL5xeLOWEA3TYmIzCEGQZ/t0SvoRUTyqfqg18lYEZH5VX3Q62SsiMj8qj/oUzoZKyIyn+oP+uzJWI3Ri4jkVfVBX5fSyVgRkflUf9CHHv3YpIZuRETyqf6gTyUw07NuRETmUvVBb2bRM+nVoxcRyavqgx6gIZ1kZFw9ehGRfGIR9I2ZFOcV9CIiecUk6JOcH5+sdDNERK5I8Qj6uhTn1KMXEckrFkHflElyfkw9ehGRfGIR9I0Z9ehFROYSk6BPMqIxehGRvGIR9E11SfXoRUTmEIugb8ykNEYvIjKHWAR9UybJ+Ykppqe90k0REbnixCLoG+tSuOt3Y0VE8olH0GeiRxXr7lgRkdliEvQpAM6PKehFRGaKRdA3hR79OV1iKSIySyyCvrEu9OgV9CIisxQd9GaWNLNXzOy5sLzWzHab2X4z+4qZZUJ5XVjuC+u7y9P0iy706DV0IyIyy0J69A8D+3KWPwU86u7rgFPA1lC+FTjl7jcAj4Z6ZdVw4WSsevQiIjMVFfRm1gV8CPhcWDbgLmBHqLIduC/Mbw7LhPWbQv2yacqejNVVNyIisxTbo/8M8HEg+3t9VwGn3T3bhe4HVof51cBhgLB+KNQvm8a67MlYBb2IyEwFg97M7gUG3H1PbnGeql7EutztbjOzXjPrHRwcLKqxc8n26M/pMQgiIrMU06O/E/g1MzsEfJloyOYzQKuZpUKdLuBImO8H1gCE9cuBkzM36u5PuHuPu/d0dHRc1k40pJOYKehFRPIpGPTu/gl373L3buAjwIvu/lvAS8CHQ7UtwLNhfmdYJqx/0d3L+hCaRMJorktxZlRBLyIy0+VcR/8/gT80sz6iMfgnQ/mTwFWh/A+BRy6vicVpqU8zPDqxFB8lIlJVUoWrXOTu3wa+HeYPABvy1BkF7i9B2xZkWX2K4RH16EVEZorFnbEQ9ejPqEcvIjJLbIJ+Wb3G6EVE8olX0I+pRy8iMlNsgr6lIa0xehGRPGIT9MvqU5wdm6TMV3KKiFSdGAV9mqlp1/NuRERmiFHQR1eK6oSsiMilYhP0LfVpAN00JSIyQ2yC/mKPXkEvIpIrRkGf7dFr6EZEJFdsgr4l9OiHR9SjFxHJFZ+gb4h69DoZKyJyqdgEva66ERHJLzZB35BOkkyYTsaKiMwQm6A3s+hRxQp6EZFLxCboIfuoYg3diIjkilfQN6R01Y2IyAyxCvrWhgynzivoRURyxSvoG9OcPj9e6WaIiFxRYhX0bY3q0YuIzBSvoG/KMDw6wdS0nkkvIpIVr6BvTOMOQzohKyJyQcyCPgPAKY3Ti4hcEKugb22MnnejE7IiIhfFKujbm0KP/pyGbkREsgoGvZnVm9n3zOwHZva6mX0ylK81s91mtt/MvmJmmVBeF5b7wvru8u7CRdmhm5Pq0YuIXFBMj34MuMvdbwPeDXzQzDYCnwIedfd1wClga6i/FTjl7jcAj4Z6S0JDNyIisxUMeo+cDYvp8HLgLmBHKN8O3BfmN4dlwvpNZmYla/E8mutSpBKma+lFRHIUNUZvZkkzexUYAF4AfgycdvfsE8T6gdVhfjVwGCCsHwKuKmWj52knrY0Z9ehFRHIUFfTuPuXu7wa6gA3AzfmqhWm+3vusO5jMbJuZ9ZpZ7+DgYLHtLai9Ka2TsSIiORZ01Y27nwa+DWwEWs0sFVZ1AUfCfD+wBiCsXw6czLOtJ9y9x917Ojo6Ftf6PFobM7qOXkQkRzFX3XSYWWuYbwDeB+wDXgI+HKptAZ4N8zvDMmH9i+6+ZM8kaGtMc1pj9CIiF6QKV6ET2G5mSaIvhmfc/Tkz+xHwZTP7M+AV4MlQ/0ngH8ysj6gn/5EytHtObY0Zvn/+9FJ+pIjIFa1g0Lv7XuD2POUHiMbrZ5aPAveXpHWLcFVzhpPnxpmedhKJJbnYR0TkiharO2MBVjTXMTXtnNaDzUREgBgGfceyOgCOnx2rcEtERK4MsQv6Fc1R0A+eUdCLiECMg149ehGRSOyCvkM9ehGRS8Qu6FsaUmSSCY6f1U1TIiIQw6A3M65qzmjoRkQkiF3QQ3TljYZuREQisQz6Fc116tGLiAQxDXoN3YiIZMU06Os4cTZ6DIKISK2LbdBP6jEIIiJATIN+ZUt0Lf3AmdEKt0REpPJiGfRXt9QD8M6Qgl5EJJ5Bv1xBLyKSFcugX7ksBP2wgl5EJJZBn0klWNFcpx69iAgxDXqAzuX16tGLiBDjoF/VUq8evYgIMQ76zuX1HFXQi4jEN+ivXl7P0MgEI+NTlW6KiEhFxTfoW3TljYgIxDjoO8O19EeHRircEhGRyopt0Gdvmjp6Wj16EaltsQ36a1obAOg/pR69iNS2gkFvZmvM7CUz22dmr5vZw6G83cxeMLP9YdoWys3MHjOzPjPba2bry70T+dSnk6xcVkf/qfOV+HgRkStGMT36SeCP3P1mYCPwkJndAjwC7HL3dcCusAxwN7AuvLYBj5e81UVa097IYQW9iNS4gkHv7kfd/fth/gywD1gNbAa2h2rbgfvC/GbgaY98F2g1s86St7wIXW0NGroRkZq3oDF6M+sGbgd2A6vc/ShEXwbAylBtNXA45239oWzJrWlr5OjQKJNT05X4eBGRK0LRQW9mzcBXgT9w9+H5quYpm/Wbfma2zcx6zax3cHCw2GYsyJr2BqamXXfIikhNKyrozSxNFPJfdPevheJj2SGZMB0I5f3Ampy3dwFHZm7T3Z9w9x537+no6Fhs++fV1dYIoHF6EalpxVx1Y8CTwD53/6ucVTuBLWF+C/BsTvkD4eqbjcBQdohnqa0JQd9/UuP0IlK7UkXUuRP4beCHZvZqKPtj4C+AZ8xsK/A2cH9Y9zxwD9AHnAceLGmLF6CztZ6EqUcvIrWtYNC7+7+Tf9wdYFOe+g48dJntKol0MsHqtgZ+ckJBLyK1K7Z3xmatXdHMwePnKt0MEZGKiX3QX7eiiQODZ4n+0BARqT3xD/qOJs6NTzF4ZqzSTRERqYjYB/3aFU0AHNDwjYjUqNoJ+kEFvYjUptgH/TXLG6hLJTh4/GylmyIiUhGxD/pEwli7oklX3ohIzYp90ANc39HM/gH16EWkNtVE0N949TLePnme8+OTlW6KiMiSq5mgd4e3jqlXLyK1pyaC/qarlwHwxtH5nq4sIhJPNRH0a9oaacwkeeOdM5VuiojIkquJoE8kjHWrlvGmgl5EalBNBD3ATauW8eaxM3rmjYjUnJoJ+luuaeHkuXH9rKCI1JyaCfp3dS0HYG//6Qq3RERkadVM0N/c2UIqYfygf6jSTRERWVI1E/T16SQ3d7aoRy8iNadmgh6i4Zu9/UNMT+uErIjUjpoK+tu6WjkzOsnBE3rAmYjUjpoK+vU/0wrAnkOnKtwSEZGlU1NBf31HM+1NGXYfPFnppoiILJmaCnozY0N3O987dKLSTRERWTI1FfQAG9a2c/jkCEdOj1S6KSIiS6Lmgv4917UDsPugevUiUhsKBr2ZPWVmA2b2Wk5Zu5m9YGb7w7QtlJuZPWZmfWa218zWl7Pxi3HT1S20Nab5f28dr3RTRESWRDE9+i8AH5xR9giwy93XAbvCMsDdwLrw2gY8Xppmlk4yYfzyz3bw7bcGmdL19CJSAwoGvbv/GzDzMpXNwPYwvx24L6f8aY98F2g1s85SNbZU3nvTSk6eG+cHuktWRGrAYsfoV7n7UYAwXRnKVwOHc+r1h7JZzGybmfWaWe/g4OAim7E4v/yzHSQMXnpjYEk/V0SkEkp9MtbylOUdH3H3J9y9x917Ojo6StyM+bU2Zlh/bRsvKuhFpAYsNuiPZYdkwjSbmP3Ampx6XcCRxTevfN53yypePzLM2yfOV7opIiJltdig3wlsCfNbgGdzyh8IV99sBIayQzxXmnvfFZ06ePbVn1a4JSIi5VXM5ZVfAr4D3Ghm/Wa2FfgL4P1mth94f1gGeB44APQBfw/8XllaXQJdbY1s6G7nn179qX5eUERiLVWogrt/dI5Vm/LUdeChy23UUtl8+zX8yddf4/Ujw9y6enmlmyMiUhY1d2dsrg/9XCeZZIJneg8XriwiUqVqOuhbGzPce1snX93Tz/DoRKWbIyJSFjUd9AAfu3Mt58aneOZl9epFJJ5qPuhvXb2cn+9uY/t3DjE5NV3p5oiIlFzNBz3Atl+6nsMnR9ixp7/STRERKTkFPfC+m1dy+7WtfHbXfkYnpirdHBGRklLQE/3y1Md/9SaODo3y1H8crHRzRERKSkEf3HH9Vbz/llU8tms/PzlxrtLNEREpGQV9jv+1+VbSiQSPfPWHTOtZ9SISEwr6HFcvr+ePP3Qz3zlwgr95qa/SzRERKQkF/Qwf+fk1/Prtq3n0/77Fi28cq3RzREQum4J+BjPjf//6z3FLZwu/98Xvs/uAfkRcRKqbgj6PhkyS7R/bwOrWBj72hZf5t7eW9hewRERKSUE/hxXNdfzj72xkTXsjD37hZT7/Hwf1OGMRqUoK+nmsaqlnx+/+Au+9sYNP/vOPeOCp79F/Sr9IJSLVRUFfQHNdir9/oIc/u+9Weg+d4q5P/yt//i/7GDwzVummiYgUxa6E4Yienh7v7e2tdDMKOnJ6hE9/6y2+9ko/6USCe9/VyW/8ly7es7adVFLfmSKytMxsj7v3FKynoF+4Hw+e5en/PMSOPf2cG5+irTHNe29ayYbudnq627m+owkzq3QzRSTmFPRLYGR8in99a4B/ee0d/n3/cU6cGwegKZPkhpXN3LByGdd1NNG5vJ6rW+pZtbyeVS31NGWS+iIQkcumoF9i7s6B4+foPXSSfUfP0Ddwlv0DZzg2PHssP5UwWhrStNSnaGlIs7whTWMmSV0qSX06kXdal06QMCOVMJIJI5W0sJyIlhNGMmkk89QxMxIGifDlEpVF04SBWXT/gJEts1B2aV2DC9u6MMWwxMX12foQtotdmAfIfr1lv+guLqMvP5EFKjboC/44uBTHzLi+o5nrO5ovKR8Zn+Kd4VGODo1wbHiUY8NjDI9MMDw6wdDIJMMjEwyNTDAwPMbY5BSjE9MXpqOTU1wB38MVk+/LwWatu7TSJV8cRX7JYHnWzfMey31j3rpF7BvzVypuG0XUKdGXZ6HNlGKfi99OEXWK2FDBGkvUloc3reO/3nZNEVtaPAV9mTVkkqxd0cTaFU0Lfq+7MzHljE1OMTY5zfS0M+XO5JQzNe1MTkfT7GtyenrGcjR1HHeYdpj2aN7dcaLl6eyyg+NMT4d6oQ3R+hll09n3X3zvdM42c7+gsn81Zsv8Qnl22Wetyxbk1nUKbye74uK64j97VnuLaOfF9/qM5bkVquMU3khRn1O4SpHbKdjgErWliP0uajtF1FmithRTaXlDupgtXRYF/RXMzMikjEwqwbJKN0ZEqpauCRQRiTkFvYhIzJUl6M3sg2b2ppn1mdkj5fgMEREpTsmD3sySwN8CdwO3AB81s1tK/TkiIlKccvToNwB97n7A3ceBLwOby/A5IiJShHIE/WrgcM5yfyi7hJltM7NeM+sdHNTz3kVEyqUcQZ/v7oBZV5O6+xPu3uPuPR0dHWVohoiIQHmCvh9Yk7PcBRwpw+eIiEgRSv6sGzNLAW8Bm4CfAi8Dv+nur8/znkHgJ4v8yBXA8UW+t1ppn2uD9rk2XM4+/4y7FxwSKfmdse4+aWa/D3wTSAJPzRfy4T2LHrsxs95iHuoTJ9rn2qB9rg1Lsc9leQSCuz8PPF+ObYuIyMLozlgRkZiLQ9A/UekGVID2uTZon2tD2ff5ivjhERERKZ849OhFRGQeVR30cX14mpmtMbOXzGyfmb1uZg+H8nYze8HM9odpWyg3M3ss/DvsNbP1ld2DxTGzpJm9YmbPheW1ZrY77O9XzCwTyuvCcl9Y313Jdi+WmbWa2Q4zeyMc6ztq4Bj/j/B/+jUz+5KZ1cfxOJvZU2Y2YGav5ZQt+Nia2ZZQf7+ZbVlse6o26GP+8LRJ4I/c/WZgI/BQ2LdHgF3uvg7YFZYh+jdYF17bgMeXvskl8TCwL2f5U8CjYX9PAVtD+VbglLvfADwa6lWjzwLfcPebgNuI9j22x9jMVgP/Hehx91uJLr/+CPE8zl8APjijbEHH1szagT8F3kP0DLE/zX45LFj0M3DV9wLuAL6Zs/wJ4BOVbleZ9vVZ4P3Am0BnKOsE3gzzfwd8NKf+hXrV8iK6g3oXcBfwHNGjNI4DqZnHm+gejTvCfCrUs0rvwwL3twU4OLPdMT/G2edgtYfj9hzwq3E9zkA38Npijy3wUeDvcsovqbeQV9X26Cny4WnVLvy5ejuwG1jl7kcBwnRlqBaHf4vPAB8HpsPyVcBpd58My7n7dGF/w/qhUL+aXAcMAp8Pw1WfM7MmYnyM3f2nwF8CbwNHiY7bHuJ9nHMt9NiW7JhXc9AX9fC0amZmzcBXgT9w9+H5quYpq5p/CzO7Fxhw9z25xXmqehHrqkUKWA887u63A+e4+Kd8PlW/z2HYYTOwFrgGaCIatpgpTse5GHPtZ8n2v5qDPtYPTzOzNFHIf9HdvxaKj5lZZ1jfCQyE8mr/t7gT+DUzO0T0+wV3EfXwW8Ozk+DSfbqwv2H9cuDkUja4BPqBfnffHZZ3EAV/XI8xwPuAg+4+6O4TwNeAXyDexznXQo9tyY55NQf9y8C6cMY+Q3RSZ2eF21QSZmbAk8A+d/+rnFU7geyZ9y1EY/fZ8gfC2fuNwFD2T8Rq4O6fcPcud+8mOo4vuvtvAS8BHw7VZu5v9t/hw6F+VfX03P0d4LCZ3RiKNgE/IqbHOHgb2GhmjeH/eHafY3ucZ1josf0m8AEzawt/DX0glC1cpU9YXObJjnuInpT5Y+BPKt2eEu7XLxL9ibYXeDW87iEan9wF7A/T9lDfiK5A+jHwQ6KrGiq+H4vc918Bngvz1wHfA/qA/wPUhfL6sNwX1l9X6XYvcl/fDfSG4/xPQFvcjzHwSeAN4DXgH4C6OB5n4EtE5yEmiHrmWxdzbIGPhf3vAx5cbHt0Z6yISMxV89CNiIgUQUEvIhJzCnoRkZhT0IuIxJyCXkQk5hT0IiIxp6AXEYk5Bb2ISMz9f9AG4opsBOwoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1829a10a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow_workshop]",
   "language": "python",
   "name": "conda-env-tensorflow_workshop-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}