Policy Gradients/Cartpole/Cartpole REINFORCE Monte Carlo Policy Gradients.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cartpole: REINFORCE Monte Carlo Policy Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll implement an agent <b>that plays Cartpole </b>\n",
    "\n",
    "<img src=\"http://neuro-educator.com/wp-content/uploads/2017/09/DQN.gif\" alt=\"Cartpole gif\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This is a notebook from [Deep Reinforcement Learning Course with Tensorflow](https://simoninithomas.github.io/Deep_reinforcement_learning_Course/)\n",
    "<img src=\"https://raw.githubusercontent.com/simoninithomas/Deep_reinforcement_learning_Course/master/docs/assets/img/DRLC%20Environments.png\" alt=\"Deep Reinforcement Course\"/>\n",
    "<br>\n",
    "<p>  Deep Reinforcement Learning Course is a free series of articles and videos tutorials 🆕 about Deep Reinforcement Learning, where **we'll learn the main algorithms (Q-learning, Deep Q Nets, Dueling Deep Q Nets, Policy Gradients, A2C, Proximal Policy Gradients…), and how to implement them with Tensorflow.**\n",
    "<br><br>\n",
    "    \n",
    "📜The articles explain the architectures from the big picture to the mathematical details behind them.\n",
    "<br>\n",
    "📹 The videos explain how to build the agents with Tensorflow </b></p>\n",
    "<br>\n",
    "This course will give you a **solid foundation for understanding and implementing the future state of the art algorithms**. And, you'll build a strong professional portfolio by creating **agents that learn to play awesome environments**: Doom© 👹, Space invaders 👾, Outrun, Sonic the Hedgehog©, Michael Jackson’s Moonwalker, agents that will be able to navigate in 3D environments with DeepMindLab (Quake) and able to walk with Mujoco. \n",
    "<br><br>\n",
    "</p> \n",
    "\n",
    "## 📚 The complete [Syllabus HERE](https://simoninithomas.github.io/Deep_reinforcement_learning_Course/)\n",
    "\n",
    "\n",
    "## Any questions 👨‍💻\n",
    "<p> If you have any questions, feel free to ask me: </p>\n",
    "<p> 📧: <a href=\"mailto:hello@simoninithomas.com\">hello@simoninithomas.com</a>  </p>\n",
    "<p> Github: https://github.com/simoninithomas/Deep_reinforcement_learning_Course </p>\n",
    "<p> 🌐 : https://simoninithomas.github.io/Deep_reinforcement_learning_Course/ </p>\n",
    "<p> Twitter: <a href=\"https://twitter.com/ThomasSimonini\">@ThomasSimonini</a> </p>\n",
    "<p> Don't forget to <b> follow me on <a href=\"https://twitter.com/ThomasSimonini\">twitter</a>, <a href=\"https://github.com/simoninithomas/Deep_reinforcement_learning_Course\">github</a> and <a href=\"https://medium.com/@thomassimonini\">Medium</a> to be alerted of the new articles that I publish </b></p>\n",
    "    \n",
    "## How to help  🙌\n",
    "3 ways:\n",
    "- **Clap our articles and like our videos a lot**:Clapping in Medium means that you really like our articles. And the more claps we have, the more our article is shared Liking our videos help them to be much more visible to the deep learning community.\n",
    "- **Share and speak about our articles and videos**: By sharing our articles and videos you help us to spread the word. \n",
    "- **Improve our notebooks**: if you found a bug or **a better implementation** you can send a pull request.\n",
    "<br>\n",
    "\n",
    "## Important note 🤔\n",
    "<b> You can run it on your computer but it's better to run it on GPU based services</b>, personally I use Microsoft Azure and their Deep Learning Virtual Machine (they offer 170$)\n",
    "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning\n",
    "<br>\n",
    "⚠️ I don't have any business relations with them. I just loved their excellent customer service.\n",
    "\n",
    "If you have some troubles to use Microsoft Azure follow the explainations of this excellent article here (without last the part fast.ai): https://medium.com/@manikantayadunanda/setting-up-deeplearning-machine-and-fast-ai-on-azure-a22eb6bd6429"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites 🏗️\n",
    "Before diving on the notebook **you need to understand**:\n",
    "- The foundations of Reinforcement learning (MC, TD, Rewards hypothesis...) [Article](https://medium.freecodecamp.org/an-introduction-to-reinforcement-learning-4339519de419)\n",
    "- Policy gradients [Article](https://medium.freecodecamp.org/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Import the libraries 📚"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create our environment 🎮\n",
    "This time we use <a href=\"https://gym.openai.com/\">OpenAI Gym</a> which has a lot of great environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "env = env.unwrapped\n",
    "# Policy gradient has high variance, seed for reproducability\n",
    "env.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Set up our hyperparameters ⚗️"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## ENVIRONMENT Hyperparameters\n",
    "state_size = 4\n",
    "action_size = env.action_space.n\n",
    "\n",
    "## TRAINING Hyperparameters\n",
    "max_episodes = 300\n",
    "learning_rate = 0.01\n",
    "gamma = 0.95 # Discount rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 : Define the preprocessing functions ⚙️\n",
    "This function takes <b>the rewards and perform discounting.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_and_normalize_rewards(episode_rewards):\n",
    "    discounted_episode_rewards = np.zeros_like(episode_rewards)\n",
    "    cumulative = 0.0\n",
    "    for i in reversed(range(len(episode_rewards))):\n",
    "        cumulative = cumulative * gamma + episode_rewards[i]\n",
    "        discounted_episode_rewards[i] = cumulative\n",
    "    \n",
    "    mean = np.mean(discounted_episode_rewards)\n",
    "    std = np.std(discounted_episode_rewards)\n",
    "    discounted_episode_rewards = (discounted_episode_rewards - mean) / (std)\n",
    "    \n",
    "    return discounted_episode_rewards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Create our Policy Gradient Neural Network model 🧠"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/simoninithomas/Deep_reinforcement_learning_Course/master/Policy%20Gradients/Cartpole/assets/catpole.png\">\n",
    "\n",
    "The idea is simple:\n",
    "- Our state which is an array of 4 values will be used as an input.\n",
    "- Our NN is 3 fully connected layers.\n",
    "- Our output activation function is softmax that squashes the outputs to a probability distribution (for instance if we have 4, 2, 6 --> softmax --> (0.4, 0.2, 0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope(\"inputs\"):\n",
    "    input_ = tf.placeholder(tf.float32, [None, state_size], name=\"input_\")\n",
    "    actions = tf.placeholder(tf.int32, [None, action_size], name=\"actions\")\n",
    "    discounted_episode_rewards_ = tf.placeholder(tf.float32, [None,], name=\"discounted_episode_rewards\")\n",
    "    \n",
    "    # Add this placeholder for having this variable in tensorboard\n",
    "    mean_reward_ = tf.placeholder(tf.float32 , name=\"mean_reward\")\n",
    "\n",
    "    with tf.name_scope(\"fc1\"):\n",
    "        fc1 = tf.contrib.layers.fully_connected(inputs = input_,\n",
    "                                                num_outputs = 10,\n",
    "                                                activation_fn=tf.nn.relu,\n",
    "                                                weights_initializer=tf.contrib.layers.xavier_initializer())\n",
    "\n",
    "    with tf.name_scope(\"fc2\"):\n",
    "        fc2 = tf.contrib.layers.fully_connected(inputs = fc1,\n",
    "                                                num_outputs = action_size,\n",
    "                                                activation_fn= tf.nn.relu,\n",
    "                                                weights_initializer=tf.contrib.layers.xavier_initializer())\n",
    "    \n",
    "    with tf.name_scope(\"fc3\"):\n",
    "        fc3 = tf.contrib.layers.fully_connected(inputs = fc2,\n",
    "                                                num_outputs = action_size,\n",
    "                                                activation_fn= None,\n",
    "                                                weights_initializer=tf.contrib.layers.xavier_initializer())\n",
    "\n",
    "    with tf.name_scope(\"softmax\"):\n",
    "        action_distribution = tf.nn.softmax(fc3)\n",
    "\n",
    "    with tf.name_scope(\"loss\"):\n",
    "        # tf.nn.softmax_cross_entropy_with_logits computes the cross entropy of the result after applying the softmax function\n",
    "        # If you have single-class labels, where an object can only belong to one class, you might now consider using \n",
    "        # tf.nn.sparse_softmax_cross_entropy_with_logits so that you don't have to convert your labels to a dense one-hot array. \n",
    "        neg_log_prob = tf.nn.softmax_cross_entropy_with_logits_v2(logits = fc3, labels = actions)\n",
    "        loss = tf.reduce_mean(neg_log_prob * discounted_episode_rewards_) \n",
    "        \n",
    "    \n",
    "    with tf.name_scope(\"train\"):\n",
    "        train_opt = tf.train.AdamOptimizer(learning_rate).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Set up Tensorboard 📊\n",
    "For more information about tensorboard, please watch this <a href=\"https://www.youtube.com/embed/eBbEDRsCmv4\">excellent 30min tutorial</a> <br><br>\n",
    "To launch tensorboard : `tensorboard --logdir=/tensorboard/pg/1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup TensorBoard Writer\n",
    "writer = tf.summary.FileWriter(\"/tensorboard/pg/1\")\n",
    "\n",
    "## Losses\n",
    "tf.summary.scalar(\"Loss\", loss)\n",
    "\n",
    "## Reward mean\n",
    "tf.summary.scalar(\"Reward_mean\", mean_reward_)\n",
    "\n",
    "write_op = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Train our Agent 🏃‍♂️"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Create the NN\n",
    "maxReward = 0 # Keep track of maximum reward\n",
    "For episode in range(max_episodes):\n",
    "    episode + 1\n",
    "    reset environment\n",
    "    reset stores (states, actions, rewards)\n",
    "    \n",
    "    For each step:\n",
    "        Choose action a\n",
    "        Perform action a\n",
    "        Store s, a, r\n",
    "        If done:\n",
    "            Calculate sum reward\n",
    "            Calculate gamma Gt\n",
    "            Optimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "Episode:  0\n",
      "Reward:  10.0\n",
      "Mean Reward 10.0\n",
      "Max reward so far:  10.0\n",
      "Model saved\n",
      "==========================================\n",
      "Episode:  1\n",
      "Reward:  10.0\n",
      "Mean Reward 10.0\n",
      "Max reward so far:  10.0\n",
      "==========================================\n",
      "Episode:  2\n",
      "Reward:  21.0\n",
      "Mean Reward 13.6666666667\n",
      "Max reward so far:  21.0\n",
      "==========================================\n",
      "Episode:  3\n",
      "Reward:  14.0\n",
      "Mean Reward 13.75\n",
      "Max reward so far:  21.0\n",
      "==========================================\n",
      "Episode:  4\n",
      "Reward:  21.0\n",
      "Mean Reward 15.2\n",
      "Max reward so far:  21.0\n",
      "==========================================\n",
      "Episode:  5\n",
      "Reward:  13.0\n",
      "Mean Reward 14.8333333333\n",
      "Max reward so far:  21.0\n",
      "==========================================\n",
      "Episode:  6\n",
      "Reward:  34.0\n",
      "Mean Reward 17.5714285714\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  7\n",
      "Reward:  19.0\n",
      "Mean Reward 17.75\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  8\n",
      "Reward:  12.0\n",
      "Mean Reward 17.1111111111\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  9\n",
      "Reward:  31.0\n",
      "Mean Reward 18.5\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  10\n",
      "Reward:  14.0\n",
      "Mean Reward 18.0909090909\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  11\n",
      "Reward:  10.0\n",
      "Mean Reward 17.4166666667\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  12\n",
      "Reward:  24.0\n",
      "Mean Reward 17.9230769231\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  13\n",
      "Reward:  12.0\n",
      "Mean Reward 17.5\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  14\n",
      "Reward:  15.0\n",
      "Mean Reward 17.3333333333\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  15\n",
      "Reward:  19.0\n",
      "Mean Reward 17.4375\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  16\n",
      "Reward:  34.0\n",
      "Mean Reward 18.4117647059\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  17\n",
      "Reward:  28.0\n",
      "Mean Reward 18.9444444444\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  18\n",
      "Reward:  11.0\n",
      "Mean Reward 18.5263157895\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  19\n",
      "Reward:  21.0\n",
      "Mean Reward 18.65\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  20\n",
      "Reward:  11.0\n",
      "Mean Reward 18.2857142857\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  21\n",
      "Reward:  21.0\n",
      "Mean Reward 18.4090909091\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  22\n",
      "Reward:  10.0\n",
      "Mean Reward 18.0434782609\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  23\n",
      "Reward:  31.0\n",
      "Mean Reward 18.5833333333\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  24\n",
      "Reward:  9.0\n",
      "Mean Reward 18.2\n",
      "Max reward so far:  34.0\n",
      "==========================================\n",
      "Episode:  25\n",
      "Reward:  47.0\n",
      "Mean Reward 19.3076923077\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  26\n",
      "Reward:  12.0\n",
      "Mean Reward 19.037037037\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  27\n",
      "Reward:  17.0\n",
      "Mean Reward 18.9642857143\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  28\n",
      "Reward:  17.0\n",
      "Mean Reward 18.8965517241\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  29\n",
      "Reward:  27.0\n",
      "Mean Reward 19.1666666667\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  30\n",
      "Reward:  12.0\n",
      "Mean Reward 18.935483871\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  31\n",
      "Reward:  43.0\n",
      "Mean Reward 19.6875\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  32\n",
      "Reward:  26.0\n",
      "Mean Reward 19.8787878788\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  33\n",
      "Reward:  20.0\n",
      "Mean Reward 19.8823529412\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  34\n",
      "Reward:  20.0\n",
      "Mean Reward 19.8857142857\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  35\n",
      "Reward:  42.0\n",
      "Mean Reward 20.5\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  36\n",
      "Reward:  33.0\n",
      "Mean Reward 20.8378378378\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  37\n",
      "Reward:  19.0\n",
      "Mean Reward 20.7894736842\n",
      "Max reward so far:  47.0\n",
      "==========================================\n",
      "Episode:  38\n",
      "Reward:  56.0\n",
      "Mean Reward 21.6923076923\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  39\n",
      "Reward:  16.0\n",
      "Mean Reward 21.55\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  40\n",
      "Reward:  12.0\n",
      "Mean Reward 21.3170731707\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  41\n",
      "Reward:  22.0\n",
      "Mean Reward 21.3333333333\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  42\n",
      "Reward:  25.0\n",
      "Mean Reward 21.4186046512\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  43\n",
      "Reward:  13.0\n",
      "Mean Reward 21.2272727273\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  44\n",
      "Reward:  23.0\n",
      "Mean Reward 21.2666666667\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  45\n",
      "Reward:  21.0\n",
      "Mean Reward 21.2608695652\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  46\n",
      "Reward:  22.0\n",
      "Mean Reward 21.2765957447\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  47\n",
      "Reward:  21.0\n",
      "Mean Reward 21.2708333333\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  48\n",
      "Reward:  11.0\n",
      "Mean Reward 21.0612244898\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  49\n",
      "Reward:  12.0\n",
      "Mean Reward 20.88\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  50\n",
      "Reward:  20.0\n",
      "Mean Reward 20.862745098\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  51\n",
      "Reward:  11.0\n",
      "Mean Reward 20.6730769231\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  52\n",
      "Reward:  21.0\n",
      "Mean Reward 20.679245283\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  53\n",
      "Reward:  26.0\n",
      "Mean Reward 20.7777777778\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  54\n",
      "Reward:  17.0\n",
      "Mean Reward 20.7090909091\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  55\n",
      "Reward:  11.0\n",
      "Mean Reward 20.5357142857\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  56\n",
      "Reward:  35.0\n",
      "Mean Reward 20.7894736842\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  57\n",
      "Reward:  13.0\n",
      "Mean Reward 20.6551724138\n",
      "Max reward so far:  56.0\n",
      "==========================================\n",
      "Episode:  58\n",
      "Reward:  59.0\n",
      "Mean Reward 21.3050847458\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  59\n",
      "Reward:  19.0\n",
      "Mean Reward 21.2666666667\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  60\n",
      "Reward:  23.0\n",
      "Mean Reward 21.2950819672\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  61\n",
      "Reward:  11.0\n",
      "Mean Reward 21.1290322581\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  62\n",
      "Reward:  22.0\n",
      "Mean Reward 21.1428571429\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  63\n",
      "Reward:  41.0\n",
      "Mean Reward 21.453125\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  64\n",
      "Reward:  23.0\n",
      "Mean Reward 21.4769230769\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  65\n",
      "Reward:  14.0\n",
      "Mean Reward 21.3636363636\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  66\n",
      "Reward:  10.0\n",
      "Mean Reward 21.1940298507\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  67\n",
      "Reward:  14.0\n",
      "Mean Reward 21.0882352941\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  68\n",
      "Reward:  18.0\n",
      "Mean Reward 21.0434782609\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  69\n",
      "Reward:  36.0\n",
      "Mean Reward 21.2571428571\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  70\n",
      "Reward:  29.0\n",
      "Mean Reward 21.3661971831\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  71\n",
      "Reward:  13.0\n",
      "Mean Reward 21.25\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  72\n",
      "Reward:  22.0\n",
      "Mean Reward 21.2602739726\n",
      "Max reward so far:  59.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "Episode:  73\n",
      "Reward:  46.0\n",
      "Mean Reward 21.5945945946\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  74\n",
      "Reward:  17.0\n",
      "Mean Reward 21.5333333333\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  75\n",
      "Reward:  24.0\n",
      "Mean Reward 21.5657894737\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  76\n",
      "Reward:  15.0\n",
      "Mean Reward 21.4805194805\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  77\n",
      "Reward:  16.0\n",
      "Mean Reward 21.4102564103\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  78\n",
      "Reward:  32.0\n",
      "Mean Reward 21.5443037975\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  79\n",
      "Reward:  14.0\n",
      "Mean Reward 21.45\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  80\n",
      "Reward:  17.0\n",
      "Mean Reward 21.3950617284\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  81\n",
      "Reward:  39.0\n",
      "Mean Reward 21.6097560976\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  82\n",
      "Reward:  14.0\n",
      "Mean Reward 21.5180722892\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  83\n",
      "Reward:  40.0\n",
      "Mean Reward 21.7380952381\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  84\n",
      "Reward:  24.0\n",
      "Mean Reward 21.7647058824\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  85\n",
      "Reward:  15.0\n",
      "Mean Reward 21.6860465116\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  86\n",
      "Reward:  15.0\n",
      "Mean Reward 21.6091954023\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  87\n",
      "Reward:  40.0\n",
      "Mean Reward 21.8181818182\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  88\n",
      "Reward:  18.0\n",
      "Mean Reward 21.7752808989\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  89\n",
      "Reward:  13.0\n",
      "Mean Reward 21.6777777778\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  90\n",
      "Reward:  18.0\n",
      "Mean Reward 21.6373626374\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  91\n",
      "Reward:  9.0\n",
      "Mean Reward 21.5\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  92\n",
      "Reward:  25.0\n",
      "Mean Reward 21.5376344086\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  93\n",
      "Reward:  24.0\n",
      "Mean Reward 21.5638297872\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  94\n",
      "Reward:  19.0\n",
      "Mean Reward 21.5368421053\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  95\n",
      "Reward:  23.0\n",
      "Mean Reward 21.5520833333\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  96\n",
      "Reward:  36.0\n",
      "Mean Reward 21.7010309278\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  97\n",
      "Reward:  16.0\n",
      "Mean Reward 21.6428571429\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  98\n",
      "Reward:  50.0\n",
      "Mean Reward 21.9292929293\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  99\n",
      "Reward:  18.0\n",
      "Mean Reward 21.89\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  100\n",
      "Reward:  40.0\n",
      "Mean Reward 22.0693069307\n",
      "Max reward so far:  59.0\n",
      "Model saved\n",
      "==========================================\n",
      "Episode:  101\n",
      "Reward:  18.0\n",
      "Mean Reward 22.0294117647\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  102\n",
      "Reward:  21.0\n",
      "Mean Reward 22.0194174757\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  103\n",
      "Reward:  29.0\n",
      "Mean Reward 22.0865384615\n",
      "Max reward so far:  59.0\n",
      "==========================================\n",
      "Episode:  104\n",
      "Reward:  62.0\n",
      "Mean Reward 22.4666666667\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  105\n",
      "Reward:  18.0\n",
      "Mean Reward 22.4245283019\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  106\n",
      "Reward:  22.0\n",
      "Mean Reward 22.4205607477\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  107\n",
      "Reward:  12.0\n",
      "Mean Reward 22.3240740741\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  108\n",
      "Reward:  21.0\n",
      "Mean Reward 22.3119266055\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  109\n",
      "Reward:  24.0\n",
      "Mean Reward 22.3272727273\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  110\n",
      "Reward:  32.0\n",
      "Mean Reward 22.4144144144\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  111\n",
      "Reward:  40.0\n",
      "Mean Reward 22.5714285714\n",
      "Max reward so far:  62.0\n",
      "==========================================\n",
      "Episode:  112\n",
      "Reward:  63.0\n",
      "Mean Reward 22.9292035398\n",
      "Max reward so far:  63.0\n",
      "==========================================\n",
      "Episode:  113\n",
      "Reward:  59.0\n",
      "Mean Reward 23.2456140351\n",
      "Max reward so far:  63.0\n",
      "==========================================\n",
      "Episode:  114\n",
      "Reward:  32.0\n",
      "Mean Reward 23.3217391304\n",
      "Max reward so far:  63.0\n",
      "==========================================\n",
      "Episode:  115\n",
      "Reward:  67.0\n",
      "Mean Reward 23.6982758621\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  116\n",
      "Reward:  57.0\n",
      "Mean Reward 23.9829059829\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  117\n",
      "Reward:  46.0\n",
      "Mean Reward 24.1694915254\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  118\n",
      "Reward:  12.0\n",
      "Mean Reward 24.0672268908\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  119\n",
      "Reward:  21.0\n",
      "Mean Reward 24.0416666667\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  120\n",
      "Reward:  20.0\n",
      "Mean Reward 24.0082644628\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  121\n",
      "Reward:  14.0\n",
      "Mean Reward 23.9262295082\n",
      "Max reward so far:  67.0\n",
      "==========================================\n",
      "Episode:  122\n",
      "Reward:  90.0\n",
      "Mean Reward 24.4634146341\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  123\n",
      "Reward:  12.0\n",
      "Mean Reward 24.3629032258\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  124\n",
      "Reward:  28.0\n",
      "Mean Reward 24.392\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  125\n",
      "Reward:  27.0\n",
      "Mean Reward 24.4126984127\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  126\n",
      "Reward:  56.0\n",
      "Mean Reward 24.6614173228\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  127\n",
      "Reward:  51.0\n",
      "Mean Reward 24.8671875\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  128\n",
      "Reward:  22.0\n",
      "Mean Reward 24.8449612403\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  129\n",
      "Reward:  51.0\n",
      "Mean Reward 25.0461538462\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  130\n",
      "Reward:  51.0\n",
      "Mean Reward 25.2442748092\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  131\n",
      "Reward:  44.0\n",
      "Mean Reward 25.3863636364\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  132\n",
      "Reward:  18.0\n",
      "Mean Reward 25.3308270677\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  133\n",
      "Reward:  35.0\n",
      "Mean Reward 25.4029850746\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  134\n",
      "Reward:  48.0\n",
      "Mean Reward 25.5703703704\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  135\n",
      "Reward:  15.0\n",
      "Mean Reward 25.4926470588\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  136\n",
      "Reward:  32.0\n",
      "Mean Reward 25.5401459854\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  137\n",
      "Reward:  74.0\n",
      "Mean Reward 25.8913043478\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  138\n",
      "Reward:  43.0\n",
      "Mean Reward 26.0143884892\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  139\n",
      "Reward:  27.0\n",
      "Mean Reward 26.0214285714\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  140\n",
      "Reward:  39.0\n",
      "Mean Reward 26.1134751773\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  141\n",
      "Reward:  26.0\n",
      "Mean Reward 26.1126760563\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  142\n",
      "Reward:  68.0\n",
      "Mean Reward 26.4055944056\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  143\n",
      "Reward:  53.0\n",
      "Mean Reward 26.5902777778\n",
      "Max reward so far:  90.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "Episode:  144\n",
      "Reward:  72.0\n",
      "Mean Reward 26.9034482759\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  145\n",
      "Reward:  16.0\n",
      "Mean Reward 26.8287671233\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  146\n",
      "Reward:  64.0\n",
      "Mean Reward 27.0816326531\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  147\n",
      "Reward:  38.0\n",
      "Mean Reward 27.1554054054\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  148\n",
      "Reward:  76.0\n",
      "Mean Reward 27.4832214765\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  149\n",
      "Reward:  59.0\n",
      "Mean Reward 27.6933333333\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  150\n",
      "Reward:  14.0\n",
      "Mean Reward 27.6026490066\n",
      "Max reward so far:  90.0\n",
      "==========================================\n",
      "Episode:  151\n",
      "Reward:  221.0\n",
      "Mean Reward 28.875\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  152\n",
      "Reward:  91.0\n",
      "Mean Reward 29.2810457516\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  153\n",
      "Reward:  56.0\n",
      "Mean Reward 29.4545454545\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  154\n",
      "Reward:  117.0\n",
      "Mean Reward 30.0193548387\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  155\n",
      "Reward:  65.0\n",
      "Mean Reward 30.2435897436\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  156\n",
      "Reward:  118.0\n",
      "Mean Reward 30.8025477707\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  157\n",
      "Reward:  13.0\n",
      "Mean Reward 30.6898734177\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  158\n",
      "Reward:  54.0\n",
      "Mean Reward 30.8364779874\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  159\n",
      "Reward:  37.0\n",
      "Mean Reward 30.875\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  160\n",
      "Reward:  19.0\n",
      "Mean Reward 30.801242236\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  161\n",
      "Reward:  28.0\n",
      "Mean Reward 30.7839506173\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  162\n",
      "Reward:  176.0\n",
      "Mean Reward 31.6748466258\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  163\n",
      "Reward:  99.0\n",
      "Mean Reward 32.0853658537\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  164\n",
      "Reward:  61.0\n",
      "Mean Reward 32.2606060606\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  165\n",
      "Reward:  21.0\n",
      "Mean Reward 32.1927710843\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  166\n",
      "Reward:  47.0\n",
      "Mean Reward 32.2814371257\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  167\n",
      "Reward:  199.0\n",
      "Mean Reward 33.2738095238\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  168\n",
      "Reward:  60.0\n",
      "Mean Reward 33.4319526627\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  169\n",
      "Reward:  95.0\n",
      "Mean Reward 33.7941176471\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  170\n",
      "Reward:  34.0\n",
      "Mean Reward 33.7953216374\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  171\n",
      "Reward:  161.0\n",
      "Mean Reward 34.5348837209\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  172\n",
      "Reward:  56.0\n",
      "Mean Reward 34.6589595376\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  173\n",
      "Reward:  76.0\n",
      "Mean Reward 34.8965517241\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  174\n",
      "Reward:  158.0\n",
      "Mean Reward 35.6\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  175\n",
      "Reward:  98.0\n",
      "Mean Reward 35.9545454545\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  176\n",
      "Reward:  69.0\n",
      "Mean Reward 36.1412429379\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  177\n",
      "Reward:  81.0\n",
      "Mean Reward 36.393258427\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  178\n",
      "Reward:  49.0\n",
      "Mean Reward 36.4636871508\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  179\n",
      "Reward:  185.0\n",
      "Mean Reward 37.2888888889\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  180\n",
      "Reward:  171.0\n",
      "Mean Reward 38.0276243094\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  181\n",
      "Reward:  142.0\n",
      "Mean Reward 38.5989010989\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  182\n",
      "Reward:  156.0\n",
      "Mean Reward 39.2404371585\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  183\n",
      "Reward:  149.0\n",
      "Mean Reward 39.8369565217\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  184\n",
      "Reward:  50.0\n",
      "Mean Reward 39.8918918919\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  185\n",
      "Reward:  149.0\n",
      "Mean Reward 40.4784946237\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  186\n",
      "Reward:  183.0\n",
      "Mean Reward 41.2406417112\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  187\n",
      "Reward:  177.0\n",
      "Mean Reward 41.9627659574\n",
      "Max reward so far:  221.0\n",
      "==========================================\n",
      "Episode:  188\n",
      "Reward:  263.0\n",
      "Mean Reward 43.1322751323\n",
      "Max reward so far:  263.0\n",
      "==========================================\n",
      "Episode:  189\n",
      "Reward:  321.0\n",
      "Mean Reward 44.5947368421\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  190\n",
      "Reward:  174.0\n",
      "Mean Reward 45.2722513089\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  191\n",
      "Reward:  203.0\n",
      "Mean Reward 46.09375\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  192\n",
      "Reward:  258.0\n",
      "Mean Reward 47.1917098446\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  193\n",
      "Reward:  183.0\n",
      "Mean Reward 47.8917525773\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  194\n",
      "Reward:  176.0\n",
      "Mean Reward 48.5487179487\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  195\n",
      "Reward:  258.0\n",
      "Mean Reward 49.6173469388\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  196\n",
      "Reward:  166.0\n",
      "Mean Reward 50.2081218274\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  197\n",
      "Reward:  215.0\n",
      "Mean Reward 51.0404040404\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  198\n",
      "Reward:  123.0\n",
      "Mean Reward 51.4020100503\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  199\n",
      "Reward:  248.0\n",
      "Mean Reward 52.385\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  200\n",
      "Reward:  231.0\n",
      "Mean Reward 53.2736318408\n",
      "Max reward so far:  321.0\n",
      "Model saved\n",
      "==========================================\n",
      "Episode:  201\n",
      "Reward:  226.0\n",
      "Mean Reward 54.1287128713\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  202\n",
      "Reward:  236.0\n",
      "Mean Reward 55.0246305419\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  203\n",
      "Reward:  129.0\n",
      "Mean Reward 55.387254902\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  204\n",
      "Reward:  226.0\n",
      "Mean Reward 56.2195121951\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  205\n",
      "Reward:  288.0\n",
      "Mean Reward 57.3446601942\n",
      "Max reward so far:  321.0\n",
      "==========================================\n",
      "Episode:  206\n",
      "Reward:  329.0\n",
      "Mean Reward 58.6570048309\n",
      "Max reward so far:  329.0\n",
      "==========================================\n",
      "Episode:  207\n",
      "Reward:  172.0\n",
      "Mean Reward 59.2019230769\n",
      "Max reward so far:  329.0\n",
      "==========================================\n",
      "Episode:  208\n",
      "Reward:  165.0\n",
      "Mean Reward 59.7081339713\n",
      "Max reward so far:  329.0\n",
      "==========================================\n",
      "Episode:  209\n",
      "Reward:  396.0\n",
      "Mean Reward 61.3095238095\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  210\n",
      "Reward:  174.0\n",
      "Mean Reward 61.8436018957\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  211\n",
      "Reward:  153.0\n",
      "Mean Reward 62.2735849057\n",
      "Max reward so far:  396.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "Episode:  212\n",
      "Reward:  228.0\n",
      "Mean Reward 63.0516431925\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  213\n",
      "Reward:  106.0\n",
      "Mean Reward 63.2523364486\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  214\n",
      "Reward:  147.0\n",
      "Mean Reward 63.6418604651\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  215\n",
      "Reward:  147.0\n",
      "Mean Reward 64.0277777778\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  216\n",
      "Reward:  175.0\n",
      "Mean Reward 64.5391705069\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  217\n",
      "Reward:  105.0\n",
      "Mean Reward 64.7247706422\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  218\n",
      "Reward:  97.0\n",
      "Mean Reward 64.8721461187\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  219\n",
      "Reward:  92.0\n",
      "Mean Reward 64.9954545455\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  220\n",
      "Reward:  103.0\n",
      "Mean Reward 65.1674208145\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  221\n",
      "Reward:  90.0\n",
      "Mean Reward 65.2792792793\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  222\n",
      "Reward:  165.0\n",
      "Mean Reward 65.7264573991\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  223\n",
      "Reward:  182.0\n",
      "Mean Reward 66.2455357143\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  224\n",
      "Reward:  138.0\n",
      "Mean Reward 66.5644444444\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  225\n",
      "Reward:  113.0\n",
      "Mean Reward 66.7699115044\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  226\n",
      "Reward:  124.0\n",
      "Mean Reward 67.0220264317\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  227\n",
      "Reward:  82.0\n",
      "Mean Reward 67.0877192982\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  228\n",
      "Reward:  121.0\n",
      "Mean Reward 67.3231441048\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  229\n",
      "Reward:  60.0\n",
      "Mean Reward 67.2913043478\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  230\n",
      "Reward:  99.0\n",
      "Mean Reward 67.4285714286\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  231\n",
      "Reward:  108.0\n",
      "Mean Reward 67.6034482759\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  232\n",
      "Reward:  131.0\n",
      "Mean Reward 67.8755364807\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  233\n",
      "Reward:  94.0\n",
      "Mean Reward 67.9871794872\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  234\n",
      "Reward:  45.0\n",
      "Mean Reward 67.8893617021\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  235\n",
      "Reward:  111.0\n",
      "Mean Reward 68.0720338983\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  236\n",
      "Reward:  172.0\n",
      "Mean Reward 68.5105485232\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  237\n",
      "Reward:  273.0\n",
      "Mean Reward 69.3697478992\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  238\n",
      "Reward:  139.0\n",
      "Mean Reward 69.6610878661\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  239\n",
      "Reward:  145.0\n",
      "Mean Reward 69.975\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  240\n",
      "Reward:  170.0\n",
      "Mean Reward 70.3900414938\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  241\n",
      "Reward:  153.0\n",
      "Mean Reward 70.7314049587\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  242\n",
      "Reward:  212.0\n",
      "Mean Reward 71.3127572016\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  243\n",
      "Reward:  162.0\n",
      "Mean Reward 71.6844262295\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  244\n",
      "Reward:  111.0\n",
      "Mean Reward 71.8448979592\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  245\n",
      "Reward:  138.0\n",
      "Mean Reward 72.1138211382\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  246\n",
      "Reward:  241.0\n",
      "Mean Reward 72.7975708502\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  247\n",
      "Reward:  183.0\n",
      "Mean Reward 73.2419354839\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  248\n",
      "Reward:  189.0\n",
      "Mean Reward 73.7068273092\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  249\n",
      "Reward:  162.0\n",
      "Mean Reward 74.06\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  250\n",
      "Reward:  181.0\n",
      "Mean Reward 74.4860557769\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  251\n",
      "Reward:  195.0\n",
      "Mean Reward 74.9642857143\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  252\n",
      "Reward:  245.0\n",
      "Mean Reward 75.6363636364\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  253\n",
      "Reward:  219.0\n",
      "Mean Reward 76.2007874016\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  254\n",
      "Reward:  364.0\n",
      "Mean Reward 77.3294117647\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  255\n",
      "Reward:  255.0\n",
      "Mean Reward 78.0234375\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  256\n",
      "Reward:  194.0\n",
      "Mean Reward 78.4747081712\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  257\n",
      "Reward:  245.0\n",
      "Mean Reward 79.1201550388\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  258\n",
      "Reward:  342.0\n",
      "Mean Reward 80.1351351351\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  259\n",
      "Reward:  262.0\n",
      "Mean Reward 80.8346153846\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  260\n",
      "Reward:  289.0\n",
      "Mean Reward 81.632183908\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  261\n",
      "Reward:  254.0\n",
      "Mean Reward 82.2900763359\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  262\n",
      "Reward:  228.0\n",
      "Mean Reward 82.8441064639\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  263\n",
      "Reward:  186.0\n",
      "Mean Reward 83.2348484848\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  264\n",
      "Reward:  285.0\n",
      "Mean Reward 83.9962264151\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  265\n",
      "Reward:  262.0\n",
      "Mean Reward 84.6654135338\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  266\n",
      "Reward:  210.0\n",
      "Mean Reward 85.1348314607\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  267\n",
      "Reward:  270.0\n",
      "Mean Reward 85.8246268657\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  268\n",
      "Reward:  362.0\n",
      "Mean Reward 86.8513011152\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  269\n",
      "Reward:  249.0\n",
      "Mean Reward 87.4518518519\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  270\n",
      "Reward:  192.0\n",
      "Mean Reward 87.8376383764\n",
      "Max reward so far:  396.0\n",
      "==========================================\n",
      "Episode:  271\n",
      "Reward:  533.0\n",
      "Mean Reward 89.4742647059\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  272\n",
      "Reward:  264.0\n",
      "Mean Reward 90.1135531136\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  273\n",
      "Reward:  262.0\n",
      "Mean Reward 90.7408759124\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  274\n",
      "Reward:  427.0\n",
      "Mean Reward 91.9636363636\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  275\n",
      "Reward:  190.0\n",
      "Mean Reward 92.3188405797\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  276\n",
      "Reward:  201.0\n",
      "Mean Reward 92.7111913357\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  277\n",
      "Reward:  200.0\n",
      "Mean Reward 93.0971223022\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  278\n",
      "Reward:  222.0\n",
      "Mean Reward 93.5591397849\n",
      "Max reward so far:  533.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "Episode:  279\n",
      "Reward:  195.0\n",
      "Mean Reward 93.9214285714\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  280\n",
      "Reward:  181.0\n",
      "Mean Reward 94.231316726\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  281\n",
      "Reward:  204.0\n",
      "Mean Reward 94.6205673759\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  282\n",
      "Reward:  205.0\n",
      "Mean Reward 95.0106007067\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  283\n",
      "Reward:  221.0\n",
      "Mean Reward 95.4542253521\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  284\n",
      "Reward:  152.0\n",
      "Mean Reward 95.6526315789\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  285\n",
      "Reward:  203.0\n",
      "Mean Reward 96.027972028\n",
      "Max reward so far:  533.0\n",
      "==========================================\n",
      "Episode:  286\n",
      "Reward:  737.0\n",
      "Mean Reward 98.2613240418\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  287\n",
      "Reward:  173.0\n",
      "Mean Reward 98.5208333333\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  288\n",
      "Reward:  134.0\n",
      "Mean Reward 98.6435986159\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  289\n",
      "Reward:  163.0\n",
      "Mean Reward 98.8655172414\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  290\n",
      "Reward:  148.0\n",
      "Mean Reward 99.0343642612\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  291\n",
      "Reward:  205.0\n",
      "Mean Reward 99.397260274\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  292\n",
      "Reward:  138.0\n",
      "Mean Reward 99.5290102389\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  293\n",
      "Reward:  142.0\n",
      "Mean Reward 99.6734693878\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  294\n",
      "Reward:  185.0\n",
      "Mean Reward 99.9627118644\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  295\n",
      "Reward:  181.0\n",
      "Mean Reward 100.236486486\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  296\n",
      "Reward:  145.0\n",
      "Mean Reward 100.387205387\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  297\n",
      "Reward:  147.0\n",
      "Mean Reward 100.543624161\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  298\n",
      "Reward:  155.0\n",
      "Mean Reward 100.725752508\n",
      "Max reward so far:  737.0\n",
      "==========================================\n",
      "Episode:  299\n",
      "Reward:  176.0\n",
      "Mean Reward 100.976666667\n",
      "Max reward so far:  737.0\n"
     ]
    }
   ],
   "source": [
    "allRewards = []\n",
    "total_rewards = 0\n",
    "maximumRewardRecorded = 0\n",
    "episode = 0\n",
    "episode_states, episode_actions, episode_rewards = [],[],[]\n",
    "\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    for episode in range(max_episodes):\n",
    "        \n",
    "        episode_rewards_sum = 0\n",
    "\n",
    "        # Launch the game\n",
    "        state = env.reset()\n",
    "        \n",
    "        env.render()\n",
    "           \n",
    "        while True:\n",
    "            \n",
    "            # Choose action a, remember WE'RE NOT IN A DETERMINISTIC ENVIRONMENT, WE'RE OUTPUT PROBABILITIES.\n",
    "            action_probability_distribution = sess.run(action_distribution, feed_dict={input_: state.reshape([1,4])})\n",
    "            \n",
    "            action = np.random.choice(range(action_probability_distribution.shape[1]), p=action_probability_distribution.ravel())  # select action w.r.t the actions prob\n",
    "\n",
    "            # Perform a\n",
    "            new_state, reward, done, info = env.step(action)\n",
    "\n",
    "            # Store s, a, r\n",
    "            episode_states.append(state)\n",
    "                        \n",
    "            # For actions because we output only one (the index) we need 2 (1 is for the action taken)\n",
    "            # We need [0., 1.] (if we take right) not just the index\n",
    "            action_ = np.zeros(action_size)\n",
    "            action_[action] = 1\n",
    "            \n",
    "            episode_actions.append(action_)\n",
    "            \n",
    "            episode_rewards.append(reward)\n",
    "            if done:\n",
    "                # Calculate sum reward\n",
    "                episode_rewards_sum = np.sum(episode_rewards)\n",
    "                \n",
    "                allRewards.append(episode_rewards_sum)\n",
    "                \n",
    "                total_rewards = np.sum(allRewards)\n",
    "                \n",
    "                # Mean reward\n",
    "                mean_reward = np.divide(total_rewards, episode+1)\n",
    "                \n",
    "                \n",
    "                maximumRewardRecorded = np.amax(allRewards)\n",
    "                \n",
    "                print(\"==========================================\")\n",
    "                print(\"Episode: \", episode)\n",
    "                print(\"Reward: \", episode_rewards_sum)\n",
    "                print(\"Mean Reward\", mean_reward)\n",
    "                print(\"Max reward so far: \", maximumRewardRecorded)\n",
    "                \n",
    "                # Calculate discounted reward\n",
    "                discounted_episode_rewards = discount_and_normalize_rewards(episode_rewards)\n",
    "                                \n",
    "                # Feedforward, gradient and backpropagation\n",
    "                loss_, _ = sess.run([loss, train_opt], feed_dict={input_: np.vstack(np.array(episode_states)),\n",
    "                                                                 actions: np.vstack(np.array(episode_actions)),\n",
    "                                                                 discounted_episode_rewards_: discounted_episode_rewards \n",
    "                                                                })\n",
    "                \n",
    " \n",
    "                                                                 \n",
    "                # Write TF Summaries\n",
    "                summary = sess.run(write_op, feed_dict={input_: np.vstack(np.array(episode_states)),\n",
    "                                                                 actions: np.vstack(np.array(episode_actions)),\n",
    "                                                                 discounted_episode_rewards_: discounted_episode_rewards,\n",
    "                                                                    mean_reward_: mean_reward\n",
    "                                                                })\n",
    "                \n",
    "               \n",
    "                writer.add_summary(summary, episode)\n",
    "                writer.flush()\n",
    "                \n",
    "            \n",
    "                \n",
    "                # Reset the transition stores\n",
    "                episode_states, episode_actions, episode_rewards = [],[],[]\n",
    "                \n",
    "                break\n",
    "            \n",
    "            state = new_state\n",
    "        \n",
    "        # Save Model\n",
    "        if episode % 100 == 0:\n",
    "            saver.save(sess, \"./models/model.ckpt\")\n",
    "            print(\"Model saved\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./models/model.ckpt\n",
      "****************************************************\n",
      "EPISODE  0\n",
      "Score 250.0\n",
      "****************************************************\n",
      "EPISODE  1\n",
      "Score 286.0\n",
      "****************************************************\n",
      "EPISODE  2\n",
      "Score 204.0\n",
      "****************************************************\n",
      "EPISODE  3\n",
      "Score 191.0\n",
      "****************************************************\n",
      "EPISODE  4\n",
      "Score 363.0\n",
      "****************************************************\n",
      "EPISODE  5\n",
      "Score 216.0\n",
      "****************************************************\n",
      "EPISODE  6\n",
      "Score 205.0\n",
      "****************************************************\n",
      "EPISODE  7\n",
      "Score 271.0\n",
      "****************************************************\n",
      "EPISODE  8\n",
      "Score 175.0\n",
      "****************************************************\n",
      "EPISODE  9\n",
      "Score 128.0\n",
      "Score over time: 228.9\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    env.reset()\n",
    "    rewards = []\n",
    "    \n",
    "    # Load the model\n",
    "    saver.restore(sess, \"./models/model.ckpt\")\n",
    "\n",
    "    for episode in range(10):\n",
    "        state = env.reset()\n",
    "        step = 0\n",
    "        done = False\n",
    "        total_rewards = 0\n",
    "        print(\"****************************************************\")\n",
    "        print(\"EPISODE \", episode)\n",
    "\n",
    "        while True:\n",
    "            \n",
    "\n",
    "            # Choose action a, remember WE'RE NOT IN A DETERMINISTIC ENVIRONMENT, WE'RE OUTPUT PROBABILITIES.\n",
    "            action_probability_distribution = sess.run(action_distribution, feed_dict={input_: state.reshape([1,4])})\n",
    "            #print(action_probability_distribution)\n",
    "            action = np.random.choice(range(action_probability_distribution.shape[1]), p=action_probability_distribution.ravel())  # select action w.r.t the actions prob\n",
    "\n",
    "\n",
    "            new_state, reward, done, info = env.step(action)\n",
    "\n",
    "            total_rewards += reward\n",
    "\n",
    "            if done:\n",
    "                rewards.append(total_rewards)\n",
    "                print (\"Score\", total_rewards)\n",
    "                break\n",
    "            state = new_state\n",
    "    env.close()\n",
    "    print (\"Score over time: \" +  str(sum(rewards)/10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:gameplai]",
   "language": "python",
   "name": "conda-env-gameplai-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}