assignment2/Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.8928938043362133e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "If we don't divide by `p`, expectation of the dot product of W and X in each layer on test time will be in p times bigger than in train time, so to prevent that we divide.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.302097302967627\n",
      "W1 relative error: 8.20e-04\n",
      "W2 relative error: 9.94e-07\n",
      "W3 relative error: 3.14e-07\n",
      "b1 relative error: 1.93e-04\n",
      "b2 relative error: 1.33e-09\n",
      "b3 relative error: 1.78e-10\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.3030642613822065\n",
      "W1 relative error: 5.09e-06\n",
      "W2 relative error: 1.95e-07\n",
      "W3 relative error: 1.89e-07\n",
      "b1 relative error: 1.57e-07\n",
      "b2 relative error: 5.20e-09\n",
      "b3 relative error: 9.69e-11\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.302532570383488\n",
      "W1 relative error: 5.10e-07\n",
      "W2 relative error: 3.15e-08\n",
      "W3 relative error: 7.36e-08\n",
      "b1 relative error: 1.06e-08\n",
      "b2 relative error: 1.35e-09\n",
      "b3 relative error: 1.16e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.890000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.930000; val_acc: 0.282000\n",
      "(Epoch 12 / 25) train acc: 0.958000; val_acc: 0.300000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.962000; val_acc: 0.317000\n",
      "(Epoch 15 / 25) train acc: 0.964000; val_acc: 0.304000\n",
      "(Epoch 16 / 25) train acc: 0.980000; val_acc: 0.308000\n",
      "(Epoch 17 / 25) train acc: 0.972000; val_acc: 0.322000\n",
      "(Epoch 18 / 25) train acc: 0.992000; val_acc: 0.316000\n",
      "(Epoch 19 / 25) train acc: 0.982000; val_acc: 0.303000\n",
      "(Epoch 20 / 25) train acc: 0.986000; val_acc: 0.303000\n",
      "(Iteration 101 / 125) loss: 0.137000\n",
      "(Epoch 21 / 25) train acc: 0.992000; val_acc: 0.306000\n",
      "(Epoch 22 / 25) train acc: 0.940000; val_acc: 0.308000\n",
      "(Epoch 23 / 25) train acc: 0.968000; val_acc: 0.311000\n",
      "(Epoch 24 / 25) train acc: 0.976000; val_acc: 0.311000\n",
      "(Epoch 25 / 25) train acc: 0.966000; val_acc: 0.308000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318480\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.620000; val_acc: 0.290000\n",
      "(Epoch 7 / 25) train acc: 0.626000; val_acc: 0.298000\n",
      "(Epoch 8 / 25) train acc: 0.680000; val_acc: 0.311000\n",
      "(Epoch 9 / 25) train acc: 0.716000; val_acc: 0.299000\n",
      "(Epoch 10 / 25) train acc: 0.730000; val_acc: 0.299000\n",
      "(Epoch 11 / 25) train acc: 0.752000; val_acc: 0.313000\n",
      "(Epoch 12 / 25) train acc: 0.784000; val_acc: 0.280000\n",
      "(Epoch 13 / 25) train acc: 0.830000; val_acc: 0.312000\n",
      "(Epoch 14 / 25) train acc: 0.810000; val_acc: 0.344000\n",
      "(Epoch 15 / 25) train acc: 0.862000; val_acc: 0.358000\n",
      "(Epoch 16 / 25) train acc: 0.852000; val_acc: 0.301000\n",
      "(Epoch 17 / 25) train acc: 0.846000; val_acc: 0.292000\n",
      "(Epoch 18 / 25) train acc: 0.832000; val_acc: 0.314000\n",
      "(Epoch 19 / 25) train acc: 0.860000; val_acc: 0.329000\n",
      "(Epoch 20 / 25) train acc: 0.836000; val_acc: 0.290000\n",
      "(Iteration 101 / 125) loss: 5.008886\n",
      "(Epoch 21 / 25) train acc: 0.886000; val_acc: 0.316000\n",
      "(Epoch 22 / 25) train acc: 0.870000; val_acc: 0.292000\n",
      "(Epoch 23 / 25) train acc: 0.896000; val_acc: 0.301000\n",
      "(Epoch 24 / 25) train acc: 0.910000; val_acc: 0.319000\n",
      "(Epoch 25 / 25) train acc: 0.918000; val_acc: 0.306000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4UAAAJNCAYAAACGHNDNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzde5hdZX33//fXMMiomHCISDKBBEmDNAnEDiBCAYkQUM5qBA8UqyJWSn+2TQ3P0+bBPLVQQsWHQmtjRcADkFKMQYWo4KF4QCaGJgJGwkEzEyzhkIg6QAjf3x97T9gZZpKZZPZe+/B+Xddce9a91l7ruyebYX/mvtd9R2YiSZIkSWpNLyu6AEmSJElScQyFkiRJktTCDIWSJEmS1MIMhZIkSZLUwgyFkiRJktTCDIWSJEmS1MIMhZKklhcRoyLitxGxT9G1SJJUa+E6hZKkRhMRv63YfAXwLLCpvP3hzPxS7auSJKkxGQolSQ0tIh4BPpiZ397KMTtl5vO1q6p2mvm1SZJqw+GjkqSmExF/HxE3RsT1EfE08N6IODwifhwR6yPi0Yi4IiLaysfvFBEZERPL218s7781Ip6OiB9FxKRBrvWyiLgpIn5dPvd3I+L1FftfERGXR8SvImJDRHw/Il5e3ndUuaYNEbEmIt5Xbr8zIs6pOMcHI+K7/Wr9s4hYDfy83H5lRHRHxG8i4u6IeFPF83eKiL+LiAfL+7siYlxE/FtE/GO/13NrRJy/4/8KkqRGYSiUJDWr04EvA6OBG4Hngb8A9gSOAE4APryV578b+Dtgd+BXwP/dyrFfAyYDrwV+BnyhYt/lwHTgsPK5/hfwQjlkfh34FLAHMANYOYzXdwpwCDCtvH1X+Tq7AzcB/9EXPoE5wDsoveYxwAeBZ4BrgXdHxMsAImIv4GjghmHUIUlqcIZCSVKzujMzb8nMFzKzNzPvzsy7MvP5zHwIWEgpAA3mpszsysyNwJeAgwc6qHz+azLz6cx8BrgI+KOIeGVEjALOAS7IzEczc1Nm3lk+53uB2zJzUbmmxzPznmG8vn/IzKcys7dcxxcy88nyUNJLgVcD+5eP/SDwvzLzgXK995SP/SHQW/FzOAv4dmY+Pow6JEkNzlAoSWpWayo3IuKAiPh6eZjnb4D5lHoNB/Priu9/D7xqoIPKM5deGhEPlc+7urxrT2AvYGfgwQGeOmGQ9qHq//r+JiJ+HhEbgKeAV/Li69vata6jFFApP35hkOMkSU3KUChJalb9Z1L7N0pDO/fPzFcD84AYgeucDbwVOJbSUNW+3rkA/gd4DnjdAM9bM0g7wO8ozara57UDHLP59UXEm4G/BN5OaXjobsBvefH1be1aXwDOiIgZ5WNuGeQ4SVKTMhRKklrFrsAG4HfliWC2dj/hcM/7LPAEpSD3yb4dmbkJuAb4dES8ttyreER5gpsvAidExNvLE8HsGREHlZ96D/D2iGiPiD8A/nQINTwPPA60URrC+sqK/f8O/H1EvC5KDo6I3cs1/rJ8vWuB/ygPgZUktRBDoSSpVfwV8CfA05R6DW8cofN+Hlhb/roX+GG//R8D7geWAU8C/0BpSaiHgZOBj5fbf8qLk8ZcRqkn8DHgakoBcmu+AXwbeAB4BPgN8GjF/gXAYuD28r6FwC4V+68tX9uho5LUglynUJKkFhcRxwKfA/ZLPxhIUsuxp1CSpBYWETtTWqrjswZCSWpNhkJJklpUREyjNFPp7sAVBZcjSSqIw0clSZIkqYXZUyhJkiRJLcxQKEmSJEktbKeiC6iFPffcMydOnFh0GZIkSZJUiGXLlj2emWMH2tcSoXDixIl0dXUVXYYkSZIkFSIifjnYPoePSpIkSVILMxRKkiRJUgszFEqSJElSC6urUBgRV0fEYxHxs0H2R0RcERGrI2JFRLyh1jVKkiRJUjOpq1AIXAOcsJX9JwKTy1/nAv9ag5okSZIkqWnVVSjMzO8DT27lkFOB67Lkx8CYiNi7NtVJkiRJUvNptCUpxgNrKra7y22PFlOOJElSY1q8vIcFS1exdn0v48a0M2fWFE6bMb7ospqSP2vVu0YLhTFAWw54YMS5lIaYss8++1SzJkmSpIayeHkPF968kt6NmwDoWd/LhTevBDCsjDB/1moEdTV8dAi6gQkV2x3A2oEOzMyFmdmZmZ1jx46tSXGSJEmNYMHSVZtDSp/ejZtYsHRVQRXVxuLlPRxxyR1Mmvt1jrjkDhYv76n6NVv1Z63G0mg9hUuA8yPiBuAwYENmOnRUkiRpGNau7x1W+0gqaihlUT12Rf6spaGqq1AYEdcDxwB7RkQ38H+ANoDM/AzwDeCtwGrg98D7i6lUkiQ1k1a752vcmHZ6Bggl48a0V/W6RQ6l3FqPXTWvXdTPWhqOugqFmXnWNvYn8NEalSNJklpAK97zNWfWlC1eM0B72yjmzJpS1esWFcyguB67on7WrarV/sAzUuoqFEqSJNVakUGlqA+wfdeo9bWLHEpZVI9dUT/rVtSKf+AZKYZCSZLU0ooKKkV/gD1txviaf1AucihlkT12Rfysi1bEHzyK/ANPozMUSpJUp1ptGFRRr7eooNKKH2CLDmZgj10ttOqkPo38O9tQKElSHSqyF6mIDzZFvt6igkrRH2CLUHQwa8UeuyK04qQ+Rff87yhDoSRJW1HUX36L+lBV1AebInvNigoqrTorpcGs+bXipD6N3vNvKJQkaRBF/uW3qA9VRX2wKbrXrIig4qyUalatOKlP0b/DdpShUJKkQRT5l9+iPlQV9cGmFXvNih5KKVVL0feOFvHfUKP/DntZ0QVIklSvivzL75xZU2hvG7VFWy0+VA32AabaH2yKer1FO23GeH4w91gevuRt/GDusQZCNYXTZozn4jOmMX5MOwGMH9POxWdMa+r3d6P/DrOnUJKkQRT5l9+iepGK+gu/vWZSc2m1e0cb/XdYZGbRNVRdZ2dndnV1FV2GJKnB9L+nEEoBqdn/4t3I06pLepH/LatSRCzLzM6B9tlTKEnSIBr9L7/bq9X+wi81o0ZfIkG1ZSiUJGkrDEiSGlGjL5Gg2jIUSpIagsOgJGnoGn2JBNWWs49Kkupe3zConvW9JC8Og1q8vKfo0iSpLhU1k7Aak6FQklT3tjYMSpL0Uo2+RIJqy+GjkqS65zAoSRqeVp0oq1ArFsHt82FDN4zugJnzYPrsoqsaEkOhJKnuFbleoCQ1KifKqqEVi+CWC2Bj+f9VG9aUtqEhgqHDRyVJdc9hUJKkunb7/BcDYZ+NvaX2BmBPoSSp7jkMSpI0JEUN4dzQPbz2OlN3oTAiTgD+HzAK+PfMvKTf/n2Bq4GxwJPAezOzMX7akqTt5jAoSdJWFTmEc3RH6XoDtTeAuho+GhGjgKuAE4EDgbMi4sB+h10GXJeZ04H5wMW1rVKSird4eQ9HXHIHk+Z+nSMuucOlGSRJKnII58x50NbvPve29lJ7A6irUAgcCqzOzIcy8zngBuDUfsccCNxe/v47A+yXpKbmmn2SJA2gyCGc02fDyVfA6AlAlB5PvqIhJpmB+hs+Oh6o7HftBg7rd8x/A2+nNMT0dGDXiNgjM5+oTYmSVKytrdnn8EpJUl0o4t6+oodwTp/dMCGwv3rrKYwB2rLf9l8DR0fEcuBooAd4/iUnijg3IroiomvdunUjX6kkFaTINfsctipJ2qa+e/s2rAHyxXv7Viyq7nUbfAhnkeotFHYDEyq2O4C1lQdk5trMPCMzZwD/u9y2of+JMnNhZnZmZufYsWOrWbMk1dRga/NVe80+h61K0nZasQgunwoXjSk9VjscFa2oe/safAhnkept+OjdwOSImESpB/BM4N2VB0TEnsCTmfkCcCGlmUglqWXMmTWFC29eucUQ0lqs2eewVUnaDg2+qPl2Kfrevmb9uVZRXfUUZubzwPnAUuB+YFFm3hsR8yPilPJhxwCrIuIXwF7AJwspVpIKctqM8Vx8xjTGj2kngPFj2rn4jGlVD2ZFDluVpIbV4Iuab5fB7uFrkOUZWlG99RSSmd8AvtGvbV7F9zcBN9W6Lknqb/HynsIWUy9izb5xY9rpGSAAVnvYqiQ1tAZf1Hy7zJy3Ze8oeG9fnaurnkJJahSteH/dnFlTaG8btUVbLYatSlJDa8VeM+/tazh111MoSY2gFe+v63tdRfWOSlJDKrLXrIhlIfp4b19DMRRK0nZo1fvrihi2KkkNrS8Y1TqcteIEN9puhkJJ2g7eXydVQZG9GlI1FdFrtrUJbvzvSv14T6EkbQfvr5NGWFGLXav2Wm3NvqK04gQ32m6GQknaDkUtCyE1rVactr8VGf5rpxUnuNF2c/ioJG0n76+TRlDRvRoOXa0NhzTWjstCaBjsKZQkScUrslfD3qvaKTr8txKXhdAw2FMoSZKKV2SvRpG9V63WQzm6oxy+B2jXyHNZCA2RPYWSJKl4RfZqFNV71Yo9lDPnlcJ+JYc0SoWzp1CSJNWHono1iuq9asX764pas0/SVhkKJUlSaytq6Gqr3l9X5JDGVhuuKw2RoVBSw1u8vIcFS1exdn0v48a0M2fWFGcFVXPwA2xtFNV7VfT9da32/uobrtsX/vuG60Jzv25pCCIzi66h6jo7O7Orq6voMiRVweLlPVx480p6N27a3NbeNso1A9X4+n+AhVLvlbMHNo8i/41b8f11+dRBQvgE+NjPal+PVGMRsSwzOwfa50QzkhragqWrtgiEAL0bN7Fg6aqCKpJGiIu5N78iJ9dpxfdXqw7XlYbA4aOSGtra9b3Dapcahh9gW0NR99e14vur6OG6Uh2zp1BSQxs3pn1Y7VLDKHIxdzW/Vnx/uRyGNChDoaSGNmfWFNrbRm3R1t42ijmzphRUkTRC/ACramrF91eRw3WlOufwUUkNrW8yGWcfVdMpcj23VpuVshW16nqBRS6HIdUxZx+VJEkvasVZKSWpBTTU7KMRcUJErIqI1RExd4D9+0TEdyJieUSsiIi3FlGnJElNqRVnpZSkFldXoTAiRgFXAScCBwJnRcSB/Q77W2BRZs4AzgT+pbZVSpLUxFpxVkpJanF1FQqBQ4HVmflQZj4H3ACc2u+YBF5d/n40sLaG9UmS1NxacVZKSWpx9RYKxwOVC8h0l9sqXQS8NyK6gW8Af16b0iRJagGtOCulJLW4eguFMUBb/5lwzgKuycwO4K3AFyLiJa8jIs6NiK6I6Fq3bl0VSpUkqQk5bb8ktZx6W5KiG5hQsd3BS4eHfgA4ASAzfxQRuwB7Ao9VHpSZC4GFUJp9tFoFS5LUdJy2X5JaSr31FN4NTI6ISRGxM6WJZJb0O+ZXwEyAiHg9sAtgV6AkqTpWLILLp8JFY0qPKxYVXZEkSSOqrnoKM/P5iDgfWAqMAq7OzHsjYj7QlZlLgL8CPhsRH6M0tPScbIXFFiVJtdd/zb4Na0rbYE+aJKlpuHi9JEmDuXxqKQj2N3oCfOxnta9HkqTt1FCL10uSVDdcs0+S1AIMhZIkDcY1+yRJLcBQKEnSYFyzT5LUAgyFkiQNxjX7JEktoK5mH5Ukqe64Zp8kqcnZUyhJkiRJLcxQKElqDC4iL0lSVTh8VJJU/1xEXpKkqrGnUJJU/26f/2Ig7LOxt9QuSZJ2iKFQklT/XERekqSqMRRKkuqfi8hLklQ1hkJJUv1zEXlJkqrGUChJGrqiZgB1EXlJkqrG2UclSUNT9AygLiIvSVJVGAoljYjFy3tYsHQVa9f3Mm5MO3NmTeG0GeOLLksjaWszgBrWJElqWIZCSTts8fIeLrx5Jb0bNwHQs76XC29eCWAwbCbOACpJUlPynkJJO2zB0lWbA2Gf3o2bWLB0VUEVtYAi7u1zBlBJkpqSoVDSDlu7vndY7dpBfff2bVgD5Iv39lU7GDoDqCRJTclQKDWZxct7OOKSO5g09+sccckdLF7eU/VrjhvTPqx27aCt3dtXTc4AKklSU/KeQqmJFHVv35xZU7a4LkB72yjmzJpStWu2tCLv7XMGUEmSmk7d9RRGxAkRsSoiVkfE3AH2Xx4R95S/fhER64uoU6pHRd3bd9qM8Vx8xjTGj2kngPFj2rn4jGlOMlMt3tsnSZJGUF31FEbEKOAq4DigG7g7IpZk5n19x2TmxyqO/3NgRs0LlepUkff2nTZjvCGwVmbO23K9QPDePkmStN3qrafwUGB1Zj6Umc8BNwCnbuX4s4Dra1KZ1AC8t69FeG+fJEkaQXXVUwiMB9ZUbHcDhw10YETsC0wC7qhBXVJD8N6+FuK9fZIkaYTUWyiMAdpykGPPBG7KzE0D7YyIc4FzAfbZZ5+RqU6qc33DNxcsXcXa9b2MG9POnFlTHNYpSZKkQdVbKOwGJlRsdwBrBzn2TOCjg50oMxcCCwE6OzsHC5ZS0/HePkmSJA1Hvd1TeDcwOSImRcTOlILfkv4HRcQUYDfgRzWuT5JetGIRXD4VLhpTeqz24vGSJElVUFc9hZn5fEScDywFRgFXZ+a9ETEf6MrMvoB4FnBDZtoDKKkYKxZtOQPohjWlbfBeP0mS1FCiGrmqHOy+lJlPjfjJt0NnZ2d2dXUVXYakZnL51FIQ7G/0BPjYz2pfjyRJ0lZExLLM7BxoX7WGj76W0hqDi8qL0Q80gYwkNa4N3cNrlyRJqlNVCYWZ+bfAZOBzwDnAAxHxDxHxumpcT5JqbnTH8NolSZLqVNUmminf7/fr8tfzlCaGuSkiLq3WNSWpZmbOg7b2Ldva2kvtkiRJDaQqE81ExAXAnwCPA/8OzMnMjRHxMuAB4G+qcV1Jqpm+yWRun18aMjq6oxQInWRGkiQ1mGrNProncEZm/rKyMTNfiIiTqnRNSaqt6bMNgZIkqeFVa/joN4An+zYiYteIOAwgM++v0jUlSZIkScNUrVD4r8BvK7Z/V26TJEmSJNWRaoXCqFxYPjNfoHpDVSVJkiRJ26laofChiLggItrKX38BPFSla0mSJEmStlO1QuF5wJuAHqAbOAw4t0rXkiRJkiRtp6oM6czMx4Azq3FuSZIkSdLIqdY6hbsAHwD+ENilrz0z/7Qa15MkSZIkbZ9qDR/9AvBaYBbwPaADeLpK15IkSZIkbadqhcL9M/PvgN9l5rXA24BpVbqWpFa3YhFcPhUuGlN6XLGo6IokSZIaRrWWidhYflwfEVOBXwMTq3QtSa1sxSK45QLY2Fva3rCmtA0wfXZxdUmSJDWIavUULoyI3YC/BZYA9wH/WKVrSaoHRfXW3T7/xUDYZ2NvqV2SJEnbNOI9hRHxMuA3mfkU8H1gv5G+hqQ6U2Rv3Ybu4bVLkiRpCyPeU5iZLwDnj/R5JdWxInvrRncMr12SJElbqNbw0W9FxF9HxISI2L3vq0rXklS0InvrZs6DtvYt29raS+2SJEnapmqFwj8FPkpp+Oiy8lfXUJ4YESdExKqIWB0Rcwc5ZnZE3BcR90bEl0esaknbp8jeuumz4eQrYPQEIEqPJ1/hJDOSJElDVJXZRzNz0vY8LyJGAVcBxwHdwN0RsSQz76s4ZjJwIXBEZj4VEa8ZiZqlkbZ4eQ8Llq5i7fpexo1pZ86sKZw2Y3zRZVXHzHlb3lMIte2tmz7bEChJkrSdqhIKI+Lsgdoz87ptPPVQYHVmPlQ+zw3AqZRmL+3zIeCq8kQ2ZOZjO16xNLIWL+/hwptX0rtxEwA963u58OaVAM0ZDPsC2e3zS0NGR3eUAqFBTZIkqe5Va53CQyq+3wWYCfwU2FYoHA+sqdjuBg7rd8wfAETED4BRwEWZedsOVSuNsAVLV20OhH16N25iwdJVzRkKwd46SZKkBlWt4aN/XrkdEaOBLwzhqTHQ6fpt7wRMBo4BOoD/ioipmbm+3zXPBc4F2GeffYZWuDRC1q7vHVa7JEmSVJRqTTTT3+8pBblt6QYmVGx3AGsHOOarmbkxMx8GVg107sxcmJmdmdk5duzY7Sxb2j7jxrQPq12SJEkqSlVCYUTcEhFLyl9foxTcvjqEp94NTI6ISRGxM3AmsKTfMYuBN5evsyel4aQPjVz10o6bM2sK7W2jtmhrbxvFnFlTCqpIkiRJGli17im8rOL754FfZuY2FyzLzOcj4nxgKaX7Ba/OzHsjYj7QlZlLyvuOj4j7gE3AnMx8YuRfgrT9+u4bbJnZRyVJktSwIrP/LXsjcNKIScCjmflMebsd2CszHxnxiw1BZ2dndnUNaZlESZIkSWo6EbEsMzsH2letewr/A3ihYntTuU2SJEmSVEeqFQp3yszn+jbK3+9cpWtJkiRJkrZTtULhuog4pW8jIk4FHq/StSRJkiRJ26laE82cB3wpIq4sb3cDZ1fpWpIqrVgEt8+HDd0wugNmznNReUmSJA2qWovXPwi8MSJeRWkym6ercR1J/axYBLdcABt7S9sb1pS2wWAoSZKkAVVrncJ/iIgxmfnbzHw6InaLiL+vxrUkVbh9/ouBsM/G3lK7JEmSNIBq3VN4Ymau79vIzKeAt1bpWpL6bBhkOdDB2iVJktTyqhUKR0XEy/s2yusUvnwrx0saCaM7htcuSZKklletUPhF4PaI+EBEfAD4FnBtla4lqc/MedDWvmVbW3upXZIkSRpAtSaauTQiVgBvAQK4Ddi3GteSVKFvMhlnH5UkSdIQVWtJCoBfAy8As4GHgf+s4rUk9Zk+2xAoSZKkIRvRUBgRfwCcCZwFPAHcSGlJijeP5HUkSZIkSSNjpHsKfw78F3ByZq4GiIiPjfA1pCFbvLyHBUtXsXZ9L+PGtDNn1hROmzG+6LIkSZKkujHSE828ndKw0e9ExGcjYialewqlmlu8vIcLb15Jz/peEuhZ38uFN69k8fKe2hSwYhFcPhUuGlN6XLGoNteVJEmShmFEQ2FmfiUz3wUcAHwX+BiwV0T8a0QcP5LXkrZlwdJV9G7ctEVb78ZNLFi6qvoXX7EIbrkANqwBsvR4ywUGQ0mSJNWdqixJkZm/y8wvZeZJQAdwDzC3GteSBrN2fe+w2kfU7fNhY7/rbOwttUuSJEl1pFrrFG6WmU9m5r9l5rHVvpZUadyY9mG1j6gN3cNrlyRJkgpS9VAoFWXOrCm0t43aoq29bRRzZk2p/sVHdwyvXZIkSSqIoVBN67QZ47n4jGmMH9NOAOPHtHPxGdNqM/vozHnQ1q9Hsq291C5JkiTVkWouXi8V7rQZ44tZgqJv8fjb55eGjI7uKAVCF5WXJElSnam7UBgRJwD/DxgF/HtmXtJv/znAAqBvXYErM/Pfa1qkGseKRcUFs+mzDYGSJEmqe3UVCiNiFHAVcBzQDdwdEUsy875+h96YmefXvEA1lr5lIfpmAe1bFgIMa5IkSVJZvd1TeCiwOjMfyszngBuAUwuuSY3KZSEkSZKkbaq3UDgeWFOx3V1u6+/tEbEiIm6KiAm1KU0Nx2UhJEmSpG2qt1AYA7Rlv+1bgImZOR34NnDtgCeKODciuiKia926dSNcphqCy0JIkiRJ21RvobAbqOz56wDWVh6QmU9k5rPlzc8CfzTQiTJzYWZ2Zmbn2LFjq1Ks6pzLQkiSJEnbVFcTzQB3A5MjYhKl2UXPBN5deUBE7J2Zj5Y3TwHur22J2h6Ll/ewYOkq1q7vZdyYdubMmlL9pSJcFkKSJEnaproKhZn5fEScDyyltCTF1Zl5b0TMB7oycwlwQUScAjwPPAmcU1jBGpLFy3u48OaV9G7cBEDP+l4uvHklQG2CoSFQkiRJGlRk9r9lr/l0dnZmV1dX0WW0rCMuuYOe9b0vaR8/pp0fzD22gIokSZKk1hIRyzKzc6B99XZPoZrQ2gEC4dbaJUmSJNWOoVBVN25M+7DaJUmSJNWOoVBVN2fWFNrbRm3R1t42ijmzphRUkSRJkqQ+dTXRjJpT32QyNZ99VJIkSdI2GQpVE6fNGG8IlCRJkuqQw0clSZIkqYUZCiVJkiSphRkKJUmSJKmFGQolSZIkqYUZCiVJkiSphRkKJUmSJKmFGQolSZIkqYUZCiVJkiSphRkKJUmSJKmFGQolSZIkqYUZCiVJkiSphe1UdAGqncXLe1iwdBVr1/cybkw7c2ZN4bQZ44suS5IkSVKBDIUtYvHyHi68eSW9GzcB0LO+lwtvXglgMJQkSZJamMNHW8SCpas2B8I+vRs3sWDpqoIqkiRJklQP6i4URsQJEbEqIlZHxNytHPeOiMiI6KxlfY1q7freYbVLkiRJag11FQojYhRwFXAicCBwVkQcOMBxuwIXAHfVtsLGNW5M+7DaR9yKRXD5VLhoTOlxxaLaXFeSJEnSVtVVKAQOBVZn5kOZ+RxwA3DqAMf9X+BS4JlaFtfI5syaQnvbqC3a2ttGMWfWlOpffMUiuOUC2LAGyNLjLRcYDCVJkqQ6UG+hcDywpmK7u9y2WUTMACZk5tdqWVijO23GeC4+Yxrjx7QTwPgx7Vx8xrTaTDJz+3zY2G+Y6sbeUrskSZKkQtXb7KMxQFtu3hnxMuBy4JxtnijiXOBcgH322WeEymtsp80YX8xMoxu6h9cuSZIkqWbqraewG5hQsd0BrK3Y3hWYCnw3Ih4B3ggsGWiymcxcmJmdmdk5duzYKpasbRrdMbx2SZIkSTVTb6HwbmByREyKiJ2BM4ElfTszc0Nm7pmZEzNzIvBj4JTM7CqmXA3JzHnQ1m9Cm7b2UrskSZKkQtVVKMzM54HzgaXA/cCizLw3IuZHxCnFVqftNn02nHwFjJ4AROnx5CtK7ZIkSZIKFZm57aMaXGdnZ3Z12ZnIikWlyV02dJeGbs6cZzCTJEmSWkBELMvMAdd4r7eJZlQtfctC9M0C2rcsBBgMJUmSpBZWV8NHVUUuCyFJkiRpAIbCVuGyEJIkSZIGYChsFS4LIUmSJGkAhsJW4bIQkiRJkgZgKGwVLgshSZIkaQDOPtpKps82BEqSJEnagj2FkiRJktTCDIWSJEmS1MIMhZIkSZLUwgyFkiRJktTCDIWSJEmS1MIMhZIkSZLUwgyFkiRJktTCDIWSJEmS1MIMhZIkSZLUwgyFkiRJktTCDIWSJEmS1MIMhZIkSZLUwgyFkiRJktTC6i4URsQJEbEqIlZHxNwB9p8XESsj4p6IuDMiDiyiTkmSJGPFMD4AACAASURBVElqBnUVCiNiFHAVcCJwIHDWAKHvy5k5LTMPBi4FPlXjMiVJkiSpadRVKAQOBVZn5kOZ+RxwA3Bq5QGZ+ZuKzVcCWcP6JEmSJKmp7FR0Af2MB9ZUbHcDh/U/KCI+CvwlsDNwbG1KkyRJkqTmU289hTFA20t6AjPzqsx8HfBx4G8HPFHEuRHRFRFd69atG+EyJUmSJKk51Fso7AYmVGx3AGu3cvwNwGkD7cjMhZnZmZmdY8eOHcESJUmSJKl51FsovBuYHBGTImJn4ExgSeUBETG5YvNtwAM1rE+SJEmSmkpd3VOYmc9HxPnAUmAUcHVm3hsR84GuzFwCnB8RbwE2Ak8Bf1JcxZIkSZLU2OoqFAJk5jeAb/Rrm1fx/V/UvChJkiRJalL1NnxUkiRJklRDhkJJkiRJamGGQkmSJElqYXV3T6EkSZJay8aNG+nu7uaZZ54puhSp4e2yyy50dHTQ1tY25OcYCguweHkPC5auYu36XsaNaWfOrCmcNmN80WVJkiQVoru7m1133ZWJEycSEUWXIzWszOSJJ56gu7ubSZMmDfl5Dh+tscXLe7jw5pX0rO8lgZ71vVx480oWL+8pujRJkqRCPPPMM+yxxx4GQmkHRQR77LHHsHvdDYU1tmDpKno3btqirXfjJhYsXVVQRZIkScUzEEojY3v+WzIU1tja9b3DapckSVL1/emf/imvec1rmDp16qDHZCYXXHAB+++/P9OnT+enP/3p5n3XXnstkydPZvLkyVx77bXbvN53v/tdTjrppBGpfXtcc801rF27trDrj7TbbruNKVOmsP/++3PJJZcMeMynPvUpDjzwQKZPn87MmTP55S9/uXnfqFGjOPjggzn44IM55ZRTtnm9Rx55ZKvvlWpbvHgx991334idz1BYY+PGtA+rXZIkSdV3zjnncNttt231mFtvvZUHHniABx54gIULF/KRj3wEgCeffJJPfOIT3HXXXfzkJz/hE5/4BE899dR21fH8889v1/OGq5lC4aZNm/joRz/Krbfeyn333cf1118/YGCaMWMGXV1drFixgne84x38zd/8zeZ97e3t3HPPPdxzzz0sWbJku2up1b+fobDBzZk1hfa2UVu0tbeNYs6sKQVVJEmS1FgWL+/hiEvuYNLcr3PEJXeMyNwMRx11FLvvvvtWj/nqV7/K2WefTUTwxje+kfXr1/Poo4+ydOlSjjvuOHbffXd22203jjvuuAED5m233cYBBxzAkUceyc0337y5/aKLLuLcc8/l+OOP5+yzz+aZZ57h/e9/P9OmTWPGjBl85zvfAUpB7tRTT+WEE05gypQpfOITn9h8jk996lNMnTqVqVOn8ulPfxp4aW/WZZddxkUXXcRNN91EV1cX73nPezj44IPp7a3xiLUVi+DyqXDRmNLjikU7dLqf/OQn7L///uy3337svPPOnHnmmXz1q199yXFvfvObecUrXgHAG9/4Rrq7u4d1nWXLlnHQQQdx+OGHc9VVV21uv+aaa3jnO9/JySefzPHHH09mMmfOHKZOncq0adO48cYbgVLv8FFHHcXpp5/OgQceyHnnnccLL7wAwPXXX8+0adOYOnUqH//4xzef+1WvetXm72+66SbOOeccfvjDH7JkyRLmzJnDwQcfzIMPPjis1zEQZx+tsb5ZRp19VJIkafj6Ju3rm6Ohb9I+oOqfp3p6epgwYcLm7Y6ODnp6egZtr/TMM8/woQ99iDvuuIP999+fd73rXVvsX7ZsGXfeeSft7e380z/9EwArV67k5z//Occffzy/+MUvgFIA+tnPfsYrXvEKDjnkEN72trcREXz+85/nrrvuIjM57LDDOProo9ltt90GfB3veMc7uPLKK7nsssvo7OwckZ/NkK1YBLdcABvLQXTDmtI2wPTZ23XKgX7+d91111af87nPfY4TTzxx8/YzzzxDZ2cnO+20E3PnzuW00057yXPe//7388///M8cffTRzJkzZ4t9P/rRj1ixYgW77747//mf/8k999zDf//3f/P4449zyCGHcNRRRwGlf7/77ruPfffdlxNOOIGbb76ZN73pTXz84x9n2bJl7Lbbbhx//PEsXrx4wBoA3vSmN3HKKadw0kkn8Y53vGPIP6etMRQW4LQZ4w2BkiRJ22Frk/ZV+/NVZr6kLSIGba/085//nEmTJjF58mQA3vve97Jw4cLN+0855RTa20u3E9155538+Z//OQAHHHAA++677+ZQeNxxx7HHHnsAcMYZZ3DnnXcSEZx++um88pWv3Nz+X//1X0O6N67mbp//YiDss7G31L6doXAoP/9KX/ziF+nq6uJ73/ve5rZf/epXjBs3joceeohjjz2WadOm8brXvW7z/g0bNrB+/XqOPvpoAN73vvdx6623bt7f11MMpX+/s846i1GjRrHXXntx9NFHc/fdd/PqV7+aQw89lP322w+As846izvvvJO2tjaOOeYYxo4dC8B73vMevv/97w8aCqvB4aNFGOEuc0mSpFZR5KR9HR0drFmzZvN2d3c348aNG7S9v60Flb5ABwOHnMHOMVgoBdhpp502D08Ehr1MQVVsGGTI5mDtQzDUnz/At7/9bT75yU+yZMkSXv7yl29u7zt+v/3245hjjmH58uVbPC8za/7v1//4av77GQprra/LfMMaIF/sMjcYSpIkbVORk/adcsopXHfddWQmP/7xjxk9ejR77703s2bN4pvf/CZPPfUUTz31FN/85jeZNWvWFs894IADePjhhzff/3X99dcPep2jjjqKL33pSwD84he/4Fe/+hVTppTmn/jWt77Fk08+SW9vL4sXL+aII47gqKOOYvHixfz+97/nd7/7HV/5ylf44z/+Y/baay8ee+wxnnjiCZ599lm+9rWvbb7GrrvuytNPPz3SP6JtG90xvPYhOOSQQ3jggQd4+OGHee6557jhhhsG7CVdvnw5H/7wh1myZAmvec1rNrc/9dRTPPvsswA8/vjj/OAHP+DAAw/c4rljxoxh9OjR3HnnnQCb/30GctRRR3HjjTeyadMm1q1bx/e//30OPfRQoDR89OGHH+aFF17gxhtv5Mgjj+Swww7je9/7Ho8//jibNm3i+uuv39wjuddee3H//ffzwgsv8JWvfGXzNUb6389QWGtb6zKXJEnSVlVr0r6zzjqLww8/nFWrVtHR0cHnPvc5AD7zmc/wmc98BoC3vvWt7Lfffuy///586EMf4l/+5V8A2H333fm7v/s7DjnkEA455BDmzZv3kklrdtllFxYuXMjb3vY2jjzySPbdd99Ba/mzP/szNm3axLRp03jXu97FNddcs7lX68gjj+R973sfBx98MG9/+9vp7OzkDW94A+eccw6HHnoohx12GB/84AeZMWMGbW1tzJs3j8MOO4yTTjqJAw44YPM1zjnnHM4777zaTzQzcx609Qvwbe2l9u200047ceWVVzJr1ixe//rXM3v2bP7wD/8QgHnz5m2eTXTOnDn89re/5Z3vfOcWS0/cf//9dHZ2ctBBB/HmN7+ZuXPnviQUAnz+85/nox/9KIcffvjmob4DOf3005k+fToHHXQQxx57LJdeeimvfe1rATj88MOZO3cuU6dOZdKkSZx++unsvffeXHzxxbz5zW/moIMO4g1veAOnnnoqAJdccgknnXQSxx57LHvvvffma5x55pksWLCAGTNmjMhEM7G17spm0dnZmV1dXUWXUXLRGGCgn3nARetrXY0kSVLh7r//fl7/+tcP+fjFy3tactK+a665hq6uLq688sqiS9kxKxaVOkQ2dJd6CGfO2+77CRvJd7/7XS677LItemyrZaD/piJiWWYOOLOQE83U2uiO8tDRAdolSZK0TU7a1+Cmz26JENhIHD5aa1XoMpckSVLzO+eccxq/l7CFHXPMMTXpJdwehsJamz4bTr4CRk8AovR48hX+tUSSJElSIRw+WgS7zCVJkrawrSn/JQ3N9swZU3c9hRFxQkSsiojVETF3gP1/GRH3RcSKiLg9IgafukmSJEl1b5ddduGJJ57Yrg+zkl6UmTzxxBPssssuw3peXfUURsQo4CrgOKAbuDsilmTmfRWHLQc6M/P3EfER4FLgXbWvVpIkSSOho6OD7u5u1q1bV3QpUsPbZZdd6OgY3iSWdRUKgUOB1Zn5EEBE3ACcCmwOhZn5nYrjfwy8t6YVSpIkaUS1tbUxadKkosuQWla9DR8dD1Su19BdbhvMB4Bbq1qRJEmSJDWxeuspHOju4gEHl0fEe4FO4OhB9p8LnAuwzz77jFR9kiRJktRU6q2nsBuYULHdAaztf1BEvAX438ApmfnsQCfKzIWZ2ZmZnWPHjq1KsZIkSZLU6KKeZnmKiJ2AXwAzgR7gbuDdmXlvxTEzgJuAEzLzgSGedx3wy5GveIftCTxedBFqWr6/VE2+v1RNvr9Ubb7HVE31+v7aNzMH7C2rq1AIEBFvBT4NjAKuzsxPRsR8oCszl0TEt4FpwKPlp/wqM08pqNwdEhFdmdlZdB1qTr6/VE2+v1RNvr9Ubb7HVE2N+P6qt3sKycxvAN/o1zav4vu31LwoSZIkSWpS9XZPoSRJkiSphgyFxVpYdAFqar6/VE2+v1RNvr9Ubb7HVE0N9/6qu3sKJUmSJEm1Y0+hJEmSJLUwQ2EBIuKEiFgVEasjYm7R9aj5RMQjEbEyIu6JiK6i61Fji4irI+KxiPhZRdvuEfGtiHig/LhbkTWqcQ3y/rooInrKv8PuKc9MLg1bREyIiO9ExP0RcW9E/EW53d9h2mFbeX813O8wh4/WWESMorQW43FAN6W1GM/KzPsKLUxNJSIeATozsx7XyFGDiYijgN8C12Xm1HLbpcCTmXlJ+Y9bu2Xmx4usU41pkPfXRcBvM/OyImtT44uIvYG9M/OnEbErsAw4DTgHf4dpB23l/TWbBvsdZk9h7R0KrM7MhzLzOeAG4NSCa5KkQWXm94En+zWfClxb/v5aSv8TlIZtkPeXNCIy89HM/Gn5+6eB+4Hx+DtMI2Ar76+GYyisvfHAmortbhr0zaO6lsA3I2JZRJxbdDFqSntl5qNQ+p8i8JqC61HzOT8iVpSHlzq0TzssIiYCM4C78HeYRli/9xc02O8wQ2HtxQBtjuHVSDsiM98AnAh8tDw8S5Iaxb8CrwMOBh4F/qnYctToIuJVwH8C/19m/qboetRcBnh/NdzvMENh7XUDEyq2O4C1BdWiJpWZa8uPjwFfoTRsWRpJ/1O+l6LvnorHCq5HTSQz/yczN2XmC8Bn8XeYdkBEtFH6wP6lzLy53OzvMI2Igd5fjfg7zFBYe3cDkyNiUkTsDJwJLCm4JjWRiHhl+WZnIuKVwPHAz7b+LGnYlgB/Uv7+T4CvFliLmkzfh/Wy0/F3mLZTRATwOeD+zPxUxS5/h2mHDfb+asTfYc4+WoDytLSfBkYBV2fmJwsuSU0kIvaj1DsIsBPwZd9j2hERcT1wDLAn8D/A/wEWA4uAfYBfAe/MTCcL0bAN8v46htKwqwQeAT7cd/+XNBwRcSTwX8BK4IVy8/+idN+Xv8O0Q7by/jqLBvsdZiiUJEmSpBbm8FFJkiRJamGGQkmSJElqYYZCSZIkSWphhkJJkiRJamGGQkmSJElqYYZCSZKGKCI2RcQ9FV9zR/DcEyOi7teykiQ1n52KLkCSpAbSm5kHF12EJEkjyZ5CSZJ2UEQ8EhH/GBE/KX/tX27fNyJuj4gV5cd9yu17RcRXIuK/y19vKp9qVER8NiLujYhvRkR7YS9KktQyDIWSJA1de7/ho++q2PebzDwUuBL4dLntSuC6zJwOfAm4otx+BfC9zDwIeANwb7l9MnBVZv4hsB54e5VfjyRJRGYWXYMkSQ0hIn6bma8aoP0R4NjMfCgi2oBfZ+YeEfE4sHdmbiy3P5qZe0bEOqAjM5+tOMdE4FuZObm8/XGgLTP/vvqvTJLUyuwplCRpZOQg3w92zECerfh+E977L0mqAUOhJEkj410Vjz8qf/9D4Mzy9+8B7ix/fzvwEYCIGBURr65VkZIk9edfICVJGrr2iLinYvu2zOxbluLlEXEXpT+4nlVuuwC4OiLmAOuA95fb/wJYGBEfoNQj+BHg0apXL0nSALynUJKkHVS+p7AzMx8vuhZJkobL4aOSJEmS1MLsKZQkSZKkFmZPoSRJZRExMSIyIrznXpLUMgyFkqSmERFLI2L+AO2nRsSvDXuSJL2UoVCS1EyuAd4XEdGv/X3AlzLz+dqXNDKixP9vS5JGnP9zkSQ1k8XA7sAf9zVExG7AScB15e23RcTyiPhNRKyJiIuGevKImBsRD0bE0xFxX0Sc3m//hyLi/or9byi3T4iImyNiXUQ8ERFXltsviogvVjx/i+GrEfHdiPhkRPwA+D2wX0S8v+IaD0XEh/vVcGpE3FN+fQ9GxAkR8c6IWNbvuL+KiMVDfe2SpOZlKJQkNY3M7AUWAWdXNM8Gfp6Z/13e/l15/xjgbcBHIuK0IV7iQUqBczTwCeCLEbE3QES8E7iofO5XA6cAT0TEKOBrwC+BicB44IZhvKz3AecCu5bP8RilkPtqSuseXl4RPg+lFH7nlF/fUcAjwBJgUkS8vuK87wW+MIw6JElNylAoSWo21wLvjIj28vbZ5TYAMvO7mbkyM1/IzBXA9cDRQzlxZv5HZq4tP/dG4AHg0PLuDwKXZubdWbI6M39Z3j8OmJOZv8vMZzLzzmG8nmsy897MfD4zN2bm1zPzwfI1vgd8kxd7Rj8AXJ2Z3yrX2JOZP8/MZ4EbKQVBIuIPKQXUrw2jDklSkzIUSpKaSjlwrQNOjYj9gEOAL/ftj4jDIuI75aGcG4DzgD2Hcu6IOLs8NHN9RKwHplY8dwKlnsT+JgC/3IH7Gdf0q+HEiPhxRDxZruGtQ6gBSsH43eX7Ld8HLCqHRUlSizMUSpKa0XWUegjfB3wzM/+nYt+XKQ2nnJCZo4HPAP0npnmJiNgX+CxwPrBHZo4Bflbx3DXA6wZ46hpgn0FmPv0d8IqK7dcOcMzmBYUj4uXAfwKXAXuVa/jGEGogM38MPEepV/HdOHRUklRmKJQkNaPrgLcAH6Ji6GjZrsCTmflM+R68dw/xnK+kFNDWAUTE+yn1FPb5d+CvI+KPyjOF7l8Okj8BHgUuiYhXRsQuEXFE+Tn3AEdFxD4RMRq4cBs17Ay8vFzD8xFxInB8xf7PAe+PiJkR8bKIGB8RB1Tsvw64Enh+mENYJUlNzFAoSWo6mfkI8ENKQW5Jv91/BsyPiKeBeZQmphnKOe8D/gn4EfA/wDTgBxX7/wP4JKWeyKcpz4SamZuAk4H9gV8B3cC7ys/5FqV7/VYAy9jGPX6Z+TRwQbnmpygF2iUV+39CefIZYAPwPWDfilN8gVKQtZdQkrRZZOa2j5IkSQ2vPPnOY8AbMvOBouuRJNUHewolSWodHwHuNhBKkioNdNO7JElqMhHxCKUJaYa6JqMkqUU4fFSSJEmSWpjDRyVJkiSphRkKJUmSJKmFtcQ9hXvuuWdOnDix6DIkSZIkqRDLli17PDPHDrSvJULhxIkT6erqKroMSZIkSSpERPxysH0OH5UkSZKkFmYolCRJkqQWZiiUJEmSpBZmKJQkSZKkFmYolCRJkqQWZiiUJEmSpBZmKJQkSfVhxSK4fCpcNKb0uGJR0RVJUktoiXUKJUlSnVuxCG65ADb2lrY3rCltA0yfXVxdktQC7CmUJEnFu33+i4Gwz8beUrskqaoMhZIkqXgbuofXLkkaMYZCSZJUvNEdw2uXJI0YQ6EkSSrezHnQ1r5lW1t7qV2SVFWGQkmSVLzps+HkK2D0BCBKjydf4SQzklQDzj4qSZLqw/TZhkBJKoA9hZIkSZLUwqoaCiPihIhYFRGrI2LuAPvPi4iVEXFPRNwZEQeW2ydGRG+5/Z6I+EzFc/6o/JzVEXFFREQ1X4MkSZIkNbOqhcKIGAVcBZwIHAic1Rf6Knw5M6dl5sHApcCnKvY9mJkHl7/Oq2j/V+BcYHL564RqvQZJkiRJanbV7Ck8FFidmQ9l5nPADcCplQdk5m8qNl8J5NZOGBF7A6/OzB9lZgLXAaeNbNmSJEmS1DqqGQrHA2sqtrvLbVuIiI9GxIOUegovqNg1KSKWR8T3IuKPK85ZuYrtgOeUJEmSJA1NNUPhQPf6vaQnMDOvyszXAR8H/rbc/CiwT2bOAP4S+HJEvHqo5wSIiHMjoisiutatW7ddL0CSJEmSml01Q2E3MKFiuwNYu5Xjb6A8FDQzn83MJ8rfLwMeBP6gfM6OoZwzMxdmZmdmdo4dO3a7X4QkSZIkNbNqhsK7gckRMSkidgbOBJZUHhARkys23wY8UG4fW56ohojYj9KEMg9l5qPA0xHxxvKso2cDX63ia5AkSZKkpla1xesz8/mIOB9YCowCrs7MeyNiPtCVmUuA8yPiLcBG4CngT8pPPwqYHxHPA5uA8zLzyfK+jwDXAO3AreUvSZIkSdJ2iNIkns2ts7Mzu7q6ii5DkqTGsGIR3D4fNnTD6A6YOQ+mzy66KknSDoiIZZnZOdC+qvUUSpKkBrRiEdxyAWzsLW1vWFPaBoOhJDWpat5TKEmSGs3t818MhH029pbaJUlNyVAoSZJetKF7eO2SpIZnKJQkSS8a3TG8dklSwzMUSpKkF82cB23tW7a1tZfaJUlNyVAoSZJeNH02nHwFjJ4AROnx5CucZEaSmpizj0qSpC1Nn20IlKQWYk+hJEmSJLUwQ6EkSZIktTBDoSRJkiS1MEOhJEmSJLUwQ6EkSZIktTBDoSRJkiS1MEOhJEmSJLUwQ6EkSZIktTBDoSRJkiS1MEOhJEmSJLUwQ6EkSZJqZ8UiuHwqXDSm9LhiUdEVSS1vp6ILkCRJUotYsQhuuQA29pa2N6wpbQNMn11cXVKLs6dQkiRJtXH7/BcDYZ+NvaV2SYUxFEqSJKk2NnQPr11STVQ1FEbECRGxKiJWR8TcAfafFxErI+Ke+P/bu/soOev64P/vD8mGbMAmBCJCNkCQ3EFMYtIuoEIDihAUCAExgigiVsTKj1bPHQh3Kw389CYFKh4Kx0pVwBaBgBBDFQLypLEILCYmPEUiKOwGCwECPiwQwuf+Y2aXybJJdjc7e2Vn3q9z5sxcn+thvtfstTPzme9TxJKI2KccPzQiHiyvezAiPlixz93lYy4r395ezXOQJElSPxnZ1Lu4pAFRtaQwIoYAlwEfBvYBTuhI+ip8PzMnZ+ZU4ALg6+X4GuCozJwMfBr4jy77nZiZU8u3Z6t1DpIkSepHh5wDDY0bxhoaS3FJhalmTeF+wKrMfCIzXwOuBY6u3CAzX65Y3A7IcnxpZq4uxx8GhkfEtlUsqyRJkqptymw46hIYOQ6I0v1RlzjIjFSwao4+OhZ4umK5Fdi/60YR8UXgy8Aw4INd1wMfBZZm5qsVsSsiYj3wA+CrmZn9VmpJkiRVz5TZJoHSVqaaNYXRTewtyVtmXpaZ7wTOAv5xgwNEvBv4Z+DzFeETy81K/7p8+1S3Tx5xakS0RETLc88918dTkCRJqp6FS9s4YP6djJ/7Iw6YfycLl7YVXSRJdaiaSWErMK5iuQlYvZFtodS8dFbHQkQ0ATcBJ2XmbzrimdlWvv8D8H1KzVTfIjMvz8zmzGweM2ZMn09CkiSpGhYubePsG1fQtradBNrWtnP2jStMDCUNuGomhQ8AEyJifEQMA44HFlVuEBETKhaPAB4vx0cBPwLOzsyfV2w/NCJ2Kj9uAI4EHqriOUiSJFXFhYtX0r5u/Qax9nXruXDxyoJKJKleVa1PYWa+HhGnA4uBIcB3M/PhiDgPaMnMRcDpEfEhYB3wIqWRRgFOB/YCvhIRXynHDgP+BCwuJ4RDgJ8A/16tc5AkSaqW1WvbexWXpGqp5kAzZOaPgR93iZ1T8fjvNrLfV4GvbuSwf9VvBZQkSSrIrqMaaesmAdx1VGM3W0vqkeUL4I7z4KXW0vyXh5zjwEY9UNXJ6yVJktS9OTMm0tgwZINYY8MQ5syYWFCJpEFu+QK4+Qx46WkgS/c3n1GKa5OqWlMoSZIGn4VL27hw8UpWr21n11GNzJkxkVnTxhZdrJrT8Zr6Wkv95I7zYF2X2vd17aW4tYWbZFIoSZI6dYyI2TEASseImIDJShXMmja2sNfV5H8A2aRxYLzU2ru4Otl8VJIkdXJEzPrgdBgDyCaNA2dkU+/i6mRSKEmSOjkiZn0w+R9Am2rSqP51yDnQ0GWgpobGUlybZFIoSZI6bWzkS0fErC2r17Yzc5slLBl2Bk9s+wmWDDuDmdssMfmvBps0Dpwps+GoS2DkOCBK90ddYlPdHrBPoSRpcLBPzoCYM2PiBn0KwRExa9Gnt7+fM9d9mxHxGgBNsYb5Dd9mdMMw4IhiC1drRjaVm452E1f/mzLbz4Y+sKZQkrT1q9c+OcsXwMWTYN6o0v0AnO+saWM5/9jJjB3VSABjRzVy/rGTHYCkxpzZcF1nQthhRLzGmQ3XFVSiGmaTRg0C1hRKUl9ZczVw6nGY8Y5EuOO8OxJhqPo5FzkipgbGiPbf9yquLdDx/1pnnxdFjW7rqLp9Y1IoSX1R4Bf2ulSPfXLqMRHWwLFJ48CqsyaNRU1t45Q6fWfzUUnqC0eTG1j1OMx4PSbCGjg2aVQVFTW6raPq9p1JoST1hV/YB1Y9foGtx0RYA8dRGlVFRU1t45Q6fWfzUUnqC5teDax67JNzyDkbNlGG2k+EC1SX/ZDqrEljPSrqut51VCNt3SRi1Z7apqjn7TCY30esKZSkvqjHmquiTZkNX3oI5q0t3df6l1lrcgZMRz+ktrXtJG/2Q1q4tK3ookl9VuR1PWfGRBobhmwQG4ipbYp6Xhj87yPWFEpSX9RjzZUGnjU5A2JT/ZAGy6/8UldFXtcdxx/oWrOinrfjOQfz1egJ7gAAIABJREFU+4hJoST1lV/YpZpgPyRVWxHNCou+roua2qao5129tp2Z2yzhzKEL2DXWsDp34oLXZ3Pz2gMHvCx9YVIoSdJWajD3TxlMiu6HpNpW1DQJXtcD69Pb38+Z677NiHgNgKZYw/yGbzO6YRhwRLGF6wH7FEqStBUa7P1T+mLh0jYOmH8n4+f+iAPm3zlg51pkPyTVvqKmSfC6HlhnNlzXmRB2GBGvcWbDdQWVqHesKZQkaSs02Pun9FaRk04X2Q+pHhVZA15PzTi9rgfWiPbf9yq+tTEplCRpK1R0f6CBVnQSXFQ/pHpTZPJfj804va4H0CCfqsrmo5IkbYU29oWxVvsD1VsSXK+KakpZ5HPbjLNODPKpqqqaFEbE4RGxMiJWRcTcbtafFhErImJZRCyJiH0q1p1d3m9lRMzo6TElSaoF9fZFsvAkePkCuHgSzBtVul++YGCet84UmfwX2Yzz/GMnM3ZUIwGMHdXI+cdOrv0avHr7nxrkc8tWrfloRAwBLgMOBVqBByJiUWY+UrHZ9zPz38rbzwS+DhxeTg6PB94N7Ar8JCL+V3mfzR1TkqR+U1T/p3rrDzRnxsQNmvbBACbByxfAzWfAunJy8NLTpWUYNF/oBosim1LajHMA1ev/1CCeqqqafQr3A1Zl5hMAEXEtcDTQmcBl5ssV228HZPnx0cC1mfkq8GRErCofj80dU6p3DmFfJ5YvgDvOg5daS/0VDjlnYD6IinreghTZ/6njOerl/7fQJPiO89788tphXXspXsPXdxGKTP4L/eGh3vg/NehUMykcC1T2tmwF9u+6UUR8EfgyMAz4YMW+v+iyb8enwmaPKW0VCvjyXPQX2MLUWaJS2C+wdfjLb9GDn9SbwpLgl1p7F1efFZn811vte6H8nxp0qpkURjexfEsg8zLgsoj4BPCPwKc3sW93fSDfckyAiDgVOBVgt91262GRpX5S0JfnCxev5ND193DmsAXsGmtYnTtxweuzuXDxsNr90KvDRKWwX2Dr8JdfBz+pE4N81MDBpsga8HqqfS+U/1ODTjUHmmkFxlUsNwGrN7H9tcCszezb42Nm5uWZ2ZyZzWPGjOll0aUttKkvz1XU/PLtzG/4Nk3brGGbgKZt1jC/4ds0v3x7VZ+3UAW91oUq6hfYOvzlt/DBTzQwBvmogdJWx/+pQaeaSeEDwISIGB8RwygNHLOocoOImFCxeATwePnxIuD4iNg2IsYDE4D7e3JMaatQ0Jfns4ddz4h4bYPYiHiNs4ddX9XnLVQdJiob/aW12r/AFvW8Baq3EUDr1iAfNVDa6vg/NehUrfloZr4eEacDi4EhwHcz8+GIOA9oycxFwOkR8SFgHfAipaajlLdbQGkAmdeBL2bmeoDujlmtc5D6rKBmEzuzplfxmlCPTVQOOWfDJrMwML/AFvW8BbIPUh0ZxKMGSlsl/6cGlcjstkteTWlubs6Wlpaii6F60rWfG5S+PFf7V7KLJ20kQRoHX3qoes9bpKJe66I5+qgkSeqFiHgwM5u7XWdSKFVJEV+eTZBMVCRJkrphUmhSqHpigiRJg4fv2aomry9V2FRSWM0pKSQVwTb8kjQ41OOUOho4Xl/qhWqOPipJkqSNqccpdTRwvL7UCyaFkvrH8gWlgW7mjSrdL19QdIkkaetWj1PqFK2ePqu8vtQLJoWStlxHE5WXngbyzSYqtfxhK0lbqg7n/ixUvX1WeX2pF0wKJW05m6hIUu8dck5phOhKNT73Z6Hq7bPK60u9YFIoacvZREWSem/K7NKUQSPHAVG6r/UphIpUb59VXl/qBUcflbTlRjaVm+N0E5ckbZwjRg+cevys8vpSD1lTKGnL2URFkrS187NK2iiTQklbziYqkqStnZ9V0kZFZhZdhqprbm7OlpaWooshSZIkSYWIiAczs7m7ddYUSpIkSVId2+xAMxFxOnB1Zr44AOWRNEgtXNrGhYtXsnptO7uOamTOjInMmja26GKphniNSZJUHT0ZffQdwAMR8Uvgu8DirIc2p5J6bOHSNs6+cQXt69YD0La2nbNvXAHgl3b1C68xSZKqZ7PNRzPzH4EJwHeAk4HHI+L/RsQ7q1w2SYPEhYtXdn5Z79C+bj0XLl5ZUIlUa7zGJEmqnh71KSzXDP6+fHsd2AG4ISIuqGLZJA0Sq9e29you9Vah19jyBXDxJJg3qnS/fEH1n1OSpAG02aQwIs6IiAeBC4CfA5Mz8wvAXwEfrXL5JA0Cu45q7FVc6q3CrrHlC+DmM8oTXmfp/uYzTAwlSTWlJzWFOwHHZuaMzLw+M9cBZOYbwJFVLZ2kQWHOjIk0NgzZINbYMIQ5MyYWVCLVmsKusTvOg3VdaiPXtZfikiTViJ4MNPNj4IWOhYh4G7BPZt6XmY9WrWRSP3C0woHR8Zr6WqtaCrvGXmrtXVySpEFos5PXR8RS4C87RhyNiG2Alsz8ywEoX79w8vr61HW0QijVLJx/7GSTFUk9c/GkctPRLkaOgy89NPDlkSSpj7Z08vqonIKi3Gy0JzWMRMThEbEyIlZFxNxu1n85Ih6JiOURcUdE7F6OfyAillXcXomIWeV1V0bEkxXrpvakLCrWwqVtHDD/TsbP/REHzL+ThUvbqv6cFy5eyaHr72HJsDN4YttPsGTYGRy6/h5HK5TUc4ecAw1d+i02NJbikiTViJ4kd09ExBnAN8vLfws8sbmdImIIcBlwKNBKaa7DRZn5SMVmS4HmzPxzRHyB0mA2H8/Mu4Cp5eOMBlYBt1XsNyczb+hB2bUVKGp+seaXb+f8hm8zIl4DoCnWML/h25z9MsAHq/a80kCwafQAmTK7dH/HeaUmoyObSglhR1ySpBrQk5rC04D3A22Ukrv9gVN7sN9+wKrMfCIzXwOuBY6u3CAz78rMP5cXfwE0dXOc44BbKrbTIFPU/GJnD7u+MyHsMCJe4+xh11f1eaVq6/ihpW1tO8mbP7QMRA18XZoyu9RUdN7a0r0JoSSpxvRk8vpnM/P4zHx7Zu6cmZ/IzGd7cOyxQGVHjNZybGM+C9zSTfx44Jousa+Vm5xeHBHb9qAsKlBR84vtzJpexaXBwoncJUlSf9ps89GIGE4pYXs3MLwjnpmnbG7XbmLdjmoTEZ8EmoGDusR3ASYDiyvCZwO/B4YBlwNnAW8ZGzwiTqVco7nbbrttpqiqpl1HNdLWTQJY7fnFYmRTtwNExMjuKqSlwaPQidwlSVLN6Unz0f8A3gHMAO6h1MTzDz3YrxUYV7HcBKzuulFEfAj4B2BmZr7aZfVs4KaOuREBMvOZLHkVuIJSM9W3yMzLM7M5M5vHjBnTg+KqWgqbX8wBIlSjCpvIXZIk1aSeJIV7ZeZXgD9l5lXAEZRq7zbnAWBCRIyPiGGUmoEuqtwgIqYB36KUEHbXJPUEujQdLdceEhEBzAIcE3wrN2vaWM4/djJjRzUSwNhRjQMzLcSU2XDUJaWh44nS/VGXDFh/oCJGXFV9KOyHFkmSVJN6MvpoRy3d2oiYRKnp5h6b2ykzX4+I0yk1/RwCfDczH46I8yjNc7gIuBDYHri+lOPxVGbOBIiIPSjVNN7T5dBXR8QYSs1Tl1EaCEdbuVnTxhYzMuKU2YUMClHUiKuqD4VN5C5JkmpSTyav/xvgB5RqB6+klMR9JTO/VfXS9RMnr9dAO2D+nd32oxw7qpGfz3U6jFrhtBCSJGmw2NTk9ZusKYyIbYCXM/NF4KfAnlUon1RzHAik9lkbLEmSasUm+xRm5hvA6QNUFqlmOBBI7XNaCEmSVCt6MtDM7RHxvyNiXESM7rhVvWTSIOZAILXP2mBJklQrejLQTMd8hF+siCU2JZU2yoFAal9R829KkiT1t80mhZk5fiAKItWawkZc1YCYM2PiBn0KwdpgSZI0OG02KYyIk7qLZ+b3+r84kjQ4WBssSZJqRU+aj+5b8Xg4cAjwS8CkUFJdszZYkiTVgp40H/3/KpcjYiTwH1UrkSRJkiRpwPRk9NGu/gxM6O+CSJIkSZIGXk/6FN5MabRRKCWR+wALqlkoSdLWaeHSNvtRSpJUY3rSp/CiisevA7/LzNYqlUeStJVauLRtgxFX29a2c/aNKwBMDCVJGsR60nz0KeC+zLwnM38OPB8Re1S1VJKkrc6Fi1duMAUHQPu69Vy4eGVBJZIkSf2hJzWF1wPvr1heX47t2/3m2lrZ7EvSlli9tr1XcUmSNDj0pKZwaGa+1rFQfjysekVSNXQ0+2pb207yZrOvhUvbii6apEFi11GNvYpLkqTBoSdJ4XMRMbNjISKOBtZUr0iqBpt9SdpSc2ZMpLFhyAaxxoYhzJkxsaASSZKk/tCT5qOnAVdHxKXl5VbgpOoVSdVgsy9JW6qjubnN0CVJqi09mbz+N8B7I2J7IDLzD9UvlvrbrqMaaesmAbTZl2qB/WUHzqxpY31tJUmqMZttPhoR/zciRmXmHzPzDxGxQ0R8dSAKp/5TeLOv5Qvg4kkwb1TpfrlTXap/2F9WkiRpy/SkT+GHM3Ntx0Jmvgh8pHpFUjXMmjaW84+dzNhRjQQwdlQj5x87eWB+8V++AG4+A156GsjS/c1nmBiqX9hfVpIkacv0pE/hkIjYNjNfBYiIRmDb6hZL1VBYs687zoN1XZqurmsvxafMHvjyqKbYX1aSJGnL9CQp/E/gjoi4orz8GeCq6hVJNeel1t7FpV6wv6wkSdKW2Wzz0cy8APgq8C5gH+BWYPeeHDwiDo+IlRGxKiLmdrP+yxHxSEQsj4g7ImL3inXrI2JZ+baoIj4+Iu6LiMcj4rqIcM7Erd3Ipt7FpV4ovL+sJEnSINeTPoUAvwfeAD4KHAI8urkdImIIcBnwYUrJ5AkRsU+XzZYCzZk5BbgBuKBiXXtmTi3fZlbE/xm4ODMnAC8Cn+3hOagoh5wDDV1qbRoaS3FpCxXaX1aSJKkGbLT5aET8L+B44ATgeeA6SlNSfKCHx94PWJWZT5SPdy1wNPBIxwaZeVfF9r8APrmpA0ZEAB8EPlEOXQXMA77ZwzKpCB39Bu84r9RkdGRTKSG0P6H6idMkSJIk9d2m+hQ+BvwMOCozVwFExJd6ceyxwNMVy63A/pvY/rPALRXLwyOiBXgdmJ+ZC4EdgbWZ+XrFMf0mOBhMmW0SKEmSJG2FNpUUfpRSTeFdEXErcC0QvTh2d9tmtxtGfBJoBg6qCO+WmasjYk/gzohYAbzci2OeCpwKsNtuu/Wi2JIkSZJUPzbapzAzb8rMjwN7A3cDXwJ2johvRsRhPTh2KzCuYrkJWN11o4j4EPAPwMyOaS/Kz7+6fP9E+fmnAWuAURHRkcx2e8zyfpdnZnNmNo8ZM6YHxZUkSZKk+tOT0Uf/lJlXZ+aRlJKwZcBbRhLtxgPAhPJoocMo1TouqtwgIqYB36KUED5bEd8hIrYtP94JOAB4JDMTuAs4rrzpp4Ef9qAskiRJkqRu9HT0UQAy84XM/FZmfrAH274OnA4spjRa6YLMfDgizouIjtFELwS2B67vMvXEu4CWiPgVpSRwfmZ2DFBzFvDliFhFqY/hd3pzDpIkSZKkN0Wp8q22NTc3Z0tLS9HFkCRJkqRCRMSDmdnc3bpe1RRqkFu+AC6eBPNGle6XLyi6RJIkSZIKtqnRR1VLli+Am8+Ade2l5ZeeLi2DU0VIkiRJdcyawnpxx3lvJoQd1rWX4pIkSZLqlklhvXiptXdxSZIkSXXBpLBejGzqXVySJElSXTAprBeHnAMNjRvGGhpLcUmSJEl1y6SwXkyZDUddAiPHAVG6P+oSB5mRJEmS6pyjj9aTKbNNAiVJkiRtwJpCSZIkSapjJoWSJEmSVMdMCiVJkiSpjpkUSpIkSVIdMymUJEmSpDpmUihJkiRJdcykUJIkSZLqmEmhJEmSJNUxk0JJkiRJqmMmhZIkSZJUx0wKJUmSJKmOmRRKkiRJUh0zKZQkSZKkOlbVpDAiDo+IlRGxKiLmdrP+yxHxSEQsj4g7ImL3cnxqRNwbEQ+X1328Yp8rI+LJiFhWvk2t5jlIkiRJUi2rWlIYEUOAy4APA/sAJ0TEPl02Wwo0Z+YU4AbggnL8z8BJmflu4HDgGxExqmK/OZk5tXxbVq1zkCRJkqRaV82awv2AVZn5RGa+BlwLHF25QWbelZl/Li/+Amgqx3+dmY+XH68GngXGVLGskiRJklSXqpkUjgWerlhuLcc25rPALV2DEbEfMAz4TUX4a+VmpRdHxLb9UVhJkiRJqkfVTAqjm1h2u2HEJ4Fm4MIu8V2A/wA+k5lvlMNnA3sD+wKjgbM2csxTI6IlIlqee+65vp2BJEmSJNW4aiaFrcC4iuUmYHXXjSLiQ8A/ADMz89WK+F8APwL+MTN/0RHPzGey5FXgCkrNVN8iMy/PzObMbB4zxpankiRJktSdaiaFDwATImJ8RAwDjgcWVW4QEdOAb1FKCJ+tiA8DbgK+l5nXd9lnl/J9ALOAh6p4DpIkSZJU04ZW68CZ+XpEnA4sBoYA383MhyPiPKAlMxdRai66PXB9KcfjqcycCcwGpgM7RsTJ5UOeXB5p9OqIGEOpeeoy4LRqnYMkSZIk1brI7LabX01pbm7OlpaWooshSZIkSYWIiAczs7m7dVWdvF6SJEmStHUzKZQkSZKkOmZSKEmSJEl1zKRQkiRJkuqYSaEkSZIk1TGTQkmSJEmqYyaFkiRJklTHTAolSZIkqY6ZFEqSJElSHTMplCRJkqQ6ZlIoSZIkSXXMpFCSJEmS6phJoSRJkiTVMZNCSZIkSapjJoWSJEmSVMdMCiVJkiSpjpkUSpIkSVIdMymUJEmSpDpmUihJkiRJdcykUJIkSZLqmEmhJEmSJNWxqiaFEXF4RKyMiFURMbeb9V+OiEciYnlE3BERu1es+3REPF6+fboi/lcRsaJ8zEsiIqp5DpIkSZJUy6qWFEbEEOAy4MPAPsAJEbFPl82WAs2ZOQW4AbigvO9o4J+A/YH9gH+KiB3K+3wTOBWYUL4dXq1zkCRJkqRaV82awv2AVZn5RGa+BlwLHF25QWbelZl/Li/+AmgqP54B3J6ZL2Tmi8DtwOERsQvwF5l5b2Ym8D1gVhXPQZIkSZJqWjWTwrHA0xXLreXYxnwWuGUz+44tP+7pMSVJkiRJmzC0isfurq9fdrthxCeBZuCgzezbm2OeSqmZKbvtttvmyipJkiRJdamaNYWtwLiK5SZgddeNIuJDwD8AMzPz1c3s28qbTUw3ekyAzLw8M5szs3nMmDF9PglJkiRJqmXVTAofACZExPiIGAYcDyyq3CAipgHfopQQPluxajFwWETsUB5g5jBgcWY+A/whIt5bHnX0JOCHVTwHSZIkSappVWs+mpmvR8TplBK8IcB3M/PhiDgPaMnMRcCFwPbA9eWZJZ7KzJmZ+UJE/P+UEkuA8zLzhfLjLwBXAo2U+iDegiRJkiSpT6I0iGdta25uzpaWlqKLIUmSJEmFiIgHM7O5u3VVnbxekiRJkrR1MymUJEmSpDpmUihJkiRJdcykUJIkSZLqmEmhJEmSJNUxk0JJkiRJqmMmhZIkSZJUx0wKJUmSJKmOmRRKkiRJUh0zKZQkSZKkOmZSKEmSJEl1bGjRBahHC5e2ceHilaxe286uoxqZM2Mis6aNLbpYkiRJhVi3bh2tra288sorRRdFGvSGDx9OU1MTDQ0NPd7HpHCALVzaxtk3rqB93XoA2ta2c/aNKwBMDCVJUl1qbW3lbW97G3vssQcRUXRxpEErM3n++edpbW1l/PjxPd7P5qMD7MLFKzsTwg7t69Zz4eKVBZVIkiSpWK+88go77rijCaG0hSKCHXfcsde17iaFA2z12vZexSVJkuqBCaHUP/ryv2RSOMB2HdXYq7gkSZKq75RTTuHtb387kyZN2ug2mckZZ5zBXnvtxZQpU/jlL3/Zue6qq65iwoQJTJgwgauuumqzz3f33Xdz5JFH9kvZ++LKK69k9erVhT1/f7v11luZOHEie+21F/Pnz+92m69//evss88+TJkyhUMOOYTf/e53neuGDBnC1KlTmTp1KjNnztzs8/32t7/d5LVSbQsXLuSRRx7pt+OZFA6wOTMm0tgwZINYY8MQ5syYWFCJJEmSBpeFS9s4YP6djJ/7Iw6YfycLl7Zt8TFPPvlkbr311k1uc8stt/D444/z+OOPc/nll/OFL3wBgBdeeIFzzz2X++67j/vvv59zzz2XF198sU/leP311/u0X28VmhQuXwAXT4J5o0r3yxds0eHWr1/PF7/4RW655RYeeeQRrrnmmm4TpmnTptHS0sLy5cs57rjjOPPMMzvXNTY2smzZMpYtW8aiRYv6XJaB+vuZFA5ys6aN5fxjJzN2VCMBjB3VyPnHTnaQGUmSpB7oGLSvbW07yZuD9m1pYjh9+nRGjx69yW1++MMfctJJJxERvPe972Xt2rU888wzLF68mEMPPZTRo0ezww47cOihh3abYN56663svffeHHjggdx4442d8Xnz5nHqqady2GGHcdJJJ/HKK6/wmc98hsmTJzNt2jTuuusuoJTIHX300Rx++OFMnDiRc889t/MYX//615k0aRKTJk3iG9/4BvDW2qyLLrqIefPmccMNN9DS0sKJJ57I1KlTaW8fwG5MyxfAzWfAS08DWbq/+YwtSgzvv/9+9tprL/bcc0+GDRvG8ccfzw9/+MO3bPeBD3yAESNGAPDe976X1tbWXj3Pgw8+yHve8x7e9773cdlll3XGr7zySj72sY9x1FFHcdhhh5GZzJkzh0mTJjF58mSuu+46oFQ7PH36dI455hj22WcfTjvtNN544w0ArrnmGiZPnsykSZM466yzOo+9/fbbdz6+4YYbOPnkk/nv//5vFi1axJw5c5g6dSq/+c1venUe3XH00QLMmjbWJFCSJKkPNjVoX7W/X7W1tTFu3LjO5aamJtra2jYar/TKK6/wuc99jjvvvJO99tqLj3/84xusf/DBB1myZAmNjY38y7/8CwArVqzgscce47DDDuPXv/41UEqAHnroIUaMGMG+++7LEUccQURwxRVXcN9995GZ7L///hx00EHssMMO3Z7Hcccdx6WXXspFF11Ec3Nzv7w2PXbHebCuSxK6rr0UnzK7T4fs7vW/7777NrnPd77zHT784Q93Lr/yyis0NzczdOhQ5s6dy6xZs96yz2c+8xn+9V//lYMOOog5c+ZssO7ee+9l+fLljB49mh/84AcsW7aMX/3qV6xZs4Z9992X6dOnA6W/3yOPPMLuu+/O4Ycfzo033sj73/9+zjrrLB588EF22GEHDjvsMBYuXNhtGQDe//73M3PmTI488kiOO+64Hr9Om2JNoSRJkgaNIgfty8y3xCJio/FKjz32GOPHj2fChAlEBJ/85Cc3WD9z5kwaG0tjTCxZsoRPfepTAOy9997svvvunUnhoYceyo477khjYyPHHnssS5YsYcmSJRxzzDFst912bL/99hx77LH87Gc/65dz7ncvbaR2bmPxHujJ61/pP//zP2lpadkgsXvqqadoaWnh+9//Pn//93//ltq3l156ibVr13LQQQcBdP59OnTUFEPp73fCCScwZMgQdt55Zw466CAeeOABAPbbbz/23HNPhgwZwgknnMCSJUt44IEHOPjggxkzZgxDhw7lxBNP5Kc//WnfXow+MiksQj+3o5YkSaoXRQ7a19TUxNNPP9253Nrayq677rrReFebSlS22267zsfdJTkbO8bGklKAoUOHdjZPBHo9TUFVjGzqXbwHevr6A/zkJz/ha1/7GosWLWLbbbftjHdsv+eee3LwwQezdOnSDfbLzAH/+3Xdvpp/v6omhRFxeESsjIhVETG3m/XTI+KXEfF6RBxXEf9ARCyruL0SEbPK666MiCcr1k2t5jn0uyq0o5YkSaoXRQ7aN3PmTL73ve+RmfziF79g5MiR7LLLLsyYMYPbbruNF198kRdffJHbbruNGTNmbLDv3nvvzZNPPtlZA3XNNdds9HmmT5/O1VdfDcCvf/1rnnrqKSZOLJ3f7bffzgsvvEB7ezsLFy7kgAMOYPr06SxcuJA///nP/OlPf+Kmm27ir//6r9l555159tlnef7553n11Vf5r//6r87neNvb3sYf/vCH/n6JNu+Qc6ChSwLf0FiK99G+++7L448/zpNPPslrr73Gtdde2+0IokuXLuXzn/88ixYt4u1vf3tn/MUXX+TVV18FYM2aNfz85z9nn3322WDfUaNGMXLkSJYsWQLQ+ffpzvTp07nuuutYv349zz33HD/96U/Zb7/9gFLz0SeffJI33niD6667jgMPPJD999+fe+65hzVr1rB+/XquueaazhrJnXfemUcffZQ33niDm266qfM5+vvvV7U+hRExBLgMOBRoBR6IiEWZWTlMzlPAycD/rtw3M+8CppaPMxpYBdxWscmczLyhWmWvqiq0o5YkSaoXHf0GL1y8ktVr29l1VCNzZkzc4v6EJ5xwAnfffTdr1qyhqamJc889l89+9rP827/9GwCnnXYaH/nIR/jxj3/MXnvtxYgRI7jiiisAGD16NF/5ylfYd999ATjnnHPeMmjN8OHDufzyyzniiCPYaaedOPDAA3nooYe6Lcvf/u3fctpppzF58mSGDh3KlVde2VmrdeCBB/KpT32KVatW8YlPfKKzT+DJJ5/cmXj8zd/8DdOmTessy/7778/48ePZe++9O5/j5JNP5rTTTqOxsZF77723s+lq1XV8373jvFKT0ZFNpYRwC74HDx06lEsvvZQZM2awfv16TjnlFN797ncDpfNvbm5m5syZzJkzhz/+8Y987GMfA2C33XZj0aJFPProo3z+859nm2224Y033mDu3LlvSQoBrrjiCk455RRGjBjxlqS/0jHHHMO9997Le97zHiKCCy64gHe84x089thjvO9972MczXsUAAAHnElEQVTu3LmsWLGic9CZbbbZhvPPP58PfOADZCYf+chHOProowGYP38+Rx55JOPGjWPSpEn88Y9/BOD444/nc5/7HJdccgk33HAD73znO/v8+gHEpqort+jAEe8D5mXmjPLy2QCZeX43214J/Fd3iV5EnAoclJknbm7bjWlubs6Wlpa+nEb/mzcK6O41D5i3dqBLI0mSVLhHH32Ud73rXUUXY6t35ZVX0tLSwqWXXlp0UdQHd999NxdddNEGNbbV0t3/VEQ8mJndjixUzeajY4GnK5Zby7HeOh7oWr/+tYhYHhEXR8S23e201apCO2pJkiRJ6qtqJoXd9cTsVbVkROwCTAYWV4TPBvYG9gVGA2d1sysRcWpEtEREy3PPPdebp62uKrSjliRJUu07+eSTrSUcxA4++OABqSXsi2omha3AuIrlJmB1L48xG7gpM9d1BDLzmSx5FbgC2K+7HTPz8sxszszmMWPG9PJpq2jKbDjqEhg5DojS/VGX2J9QkiRJUiGqOXn9A8CEiBgPtFFqBvqJXh7jBEo1g50iYpfMfCZK47POArrvobs1mzLbJFCSJKnC5ob8l9QzfRkzpmo1hZn5OnA6paafjwILMvPhiDgvImYCRMS+EdEKfAz4VkQ83LF/ROxBqabxni6HvjoiVgArgJ2Ar1brHCRJklR9w4cP5/nnn+/Tl1lJb8pMnn/+eYYPH96r/ao2+ujWZKsafVSSJEkbWLduHa2trVvH5OrSIDd8+HCamppoaGjYIL6p0Uer2XxUkiRJ2qyGhgbGjx9fdDGkulXNgWYkSZIkSVs5k0JJkiRJqmMmhZIkSZJUx+pioJmIeA74XdHl6MZOwJqiC6Ga5fWlavL6UjV5fanavMZUTVvr9bV7ZnY7gXtdJIVbq4ho2dgIQNKW8vpSNXl9qZq8vlRtXmOqpsF4fdl8VJIkSZLqmEmhJEmSJNUxk8JiXV50AVTTvL5UTV5fqiavL1Wb15iqadBdX/YplCRJkqQ6Zk2hJEmSJNUxk8ICRMThEbEyIlZFxNyiy6PaExG/jYgVEbEsIlqKLo8Gt4j4bkQ8GxEPVcRGR8TtEfF4+X6HIsuowWsj19e8iGgrv4cti4iPFFlGDV4RMS4i7oqIRyPi4Yj4u3Lc9zBtsU1cX4PuPczmowMsIoYAvwYOBVqBB4ATMvORQgummhIRvwWaM3NrnCNHg0xETAf+CHwvMyeVYxcAL2Tm/PKPWztk5llFllOD00aur3nAHzPzoiLLpsEvInYBdsnMX0bE24AHgVnAyfgepi20ietrNoPsPcyawoG3H7AqM5/IzNeAa4GjCy6TJG1UZv4UeKFL+GjgqvLjqyh9CEq9tpHrS+oXmflMZv6y/PgPwKPAWHwPUz/YxPU16JgUDryxwNMVy60M0otHW7UEbouIByPi1KILo5q0c2Y+A6UPReDtBZdHtef0iFhebl5q0z5tsYjYA5gG3IfvYepnXa4vGGTvYSaFAy+6idmGV/3tgMz8S+DDwBfLzbMkabD4JvBOYCrwDPAvxRZHg11EbA/8APj7zHy56PKotnRzfQ269zCTwoHXCoyrWG4CVhdUFtWozFxdvn8WuIlSs2WpP/1PuS9FR5+KZwsuj2pIZv5PZq7PzDeAf8f3MG2BiGig9IX96sy8sRz2PUz9orvrazC+h5kUDrwHgAkRMT4ihgHHA4sKLpNqSERsV+7sTERsBxwGPLTpvaReWwR8uvz408APCyyLakzHl/WyY/A9TH0UEQF8B3g0M79escr3MG2xjV1fg/E9zNFHC1AelvYbwBDgu5n5tYKLpBoSEXtSqh0EGAp832tMWyIirgEOBnYC/gf4J2AhsADYDXgK+FhmOliIem0j19fBlJpdJfBb4PMd/b+k3oiIA4GfASuAN8rh/0Op35fvYdoim7i+TmCQvYeZFEqSJElSHbP5qCRJkiTVMZNCSZIkSapjJoWSJEmSVMdMCiVJkiSpjpkUSpIkSVIdMymUJKmHImJ9RCyruM3tx2PvERFb/VxWkqTaM7ToAkiSNIi0Z+bUogshSVJ/sqZQkqQtFBG/jYh/joj7y7e9yvHdI+KOiFhevt+tHN85Im6KiF+Vb+8vH2pIRPx7RDwcEbdFRGNhJyVJqhsmhZIk9Vxjl+ajH69Y93Jm7gdcCnyjHLsU+F5mTgGuBi4pxy8B7snM9wB/CTxcjk8ALsvMdwNrgY9W+XwkSSIys+gySJI0KETEHzNz+27ivwU+mJlPREQD8PvM3DEi1gC7ZOa6cvyZzNwpIp4DmjLz1Ypj7AHcnpkTystnAQ2Z+dXqn5kkqZ5ZUyhJUv/IjTze2DbdebXi8Xrs+y9JGgAmhZIk9Y+PV9zfW37838Dx5ccnAkvKj+8AvgAQEUMi4i8GqpCSJHXlL5CSJPVcY0Qsq1i+NTM7pqXYNiLuo/SD6wnl2BnAdyNiDvAc8Jly/O+AyyPis5RqBL8APFP10kuS1A37FEqStIXKfQqbM3NN0WWRJKm3bD4qSZIkSXXMmkJJkiRJqmPWFEqSJElSHTMplCRJkqQ6ZlIoSZIkSXXMpFCSJEmS6phJoSRJkiTVMZNCSZIkSapj/w/xEHeZDnyYJAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "As we can see, result on the train data is worse, but on the val data is the same, so we can say that dropout is preventing overfitting.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "If we decided to simplify the model, there is no need to change p, because dropout works the same way regarless the size of hidden layers.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}