03C_RL_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Assignment: Reinforcement Learning (RL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short answer\n",
    "\n",
    "1\\. One of the fundamental challenges of reinforcement learning is balancing *exploration* versus *exploitation*. What do these two terms mean, and why do they present a challenge?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2\\. Another fundamental reinforcement learning challenge is what is known as the *credit assignment problem*, especially when rewards are sparse. \n",
    "What do we mean by the phrase, and why does this make learning especially difficult?\n",
    "How does this interact with reward function design, where we have to be careful that our reward captures the true objective?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep SARSA Cart Pole\n",
    "\n",
    "[SARSA (state-action-reward-state-action)](https://en.wikipedia.org/wiki/State–action–reward–state–action) is another Q value algorithm that resembles Q-learning quite closely:\n",
    "\n",
    "Q-learning update rule:\n",
    "\\begin{equation}\n",
    "Q_\\pi (s_t, a_t) \\leftarrow (1 - \\alpha) \\cdot Q_\\pi(s_t, a_t) + \\alpha \\cdot \\big(r_t + \\gamma \\max_a Q_\\pi(s_{t+1}, a)\\big)\n",
    "\\end{equation}\n",
    "\n",
    "SARSA update rule:\n",
    "\\begin{equation}\n",
    "Q_\\pi (s_t, a_t) \\leftarrow (1 - \\alpha) \\cdot Q_\\pi(s_t, a_t) + \\alpha \\cdot \\big(r_t + \\gamma Q_\\pi(s_{t+1}, a_{t+1})\\big)\n",
    "\\end{equation}\n",
    "\n",
    "Unlike Q-learning, which is considered an *off-policy* network, SARSA is an *on-policy* algorithm. \n",
    "When Q-learning calculates the estimated future reward, it must \"guess\" the future, starting with the next action the agent will take. In Q-learning, we assume the agent will take the best possible action: $\\max_a Q_\\pi(s_{t+1}, a)$. SARSA, on the other hand, uses the action that was actually taken next in the episode we are learning from: $Q_\\pi(s_{t+1}, a_{t+1})$. In other words, SARSA learns from the next action he actually took (on policy), as opposed to what the max possible Q value for the next state was (off policy).\n",
    "\n",
    "Build an RL agent that uses SARSA to solve the Cart Pole problem. \n",
    "\n",
    "*Hint: You can and should reuse the Q-Learning agent we went over earlier. In fact, if you know what you're doing, it's possible to finish this assignment in about 30 seconds.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### YOUR CODE HERE ###\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: you should be able to find that SARSA works much better for the demo we went over during lecture.\n",
    "This is not necessarily a general result.\n",
    "Q-learning and SARSA tend to do better on different kinds of problems."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}