03 - Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.107640Z",
     "start_time": "2019-04-12T05:09:51.999265Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.420304Z",
     "start_time": "2019-04-12T05:09:52.109689Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From Wikipedia**\n",
    "\n",
    "_A logistic model is one where the log-odds of the probability of an event is a linear combination of independent or predictor variables._\n",
    "\n",
    "The log-odds or logit function is given by:\n",
    "\n",
    "$$log\\left(\\frac{p}{1 - p}\\right)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $p = P(X)$ be the probability of the event under consideration. \n",
    "\n",
    "By definition, we have\n",
    "\n",
    "$log\\left(\\frac{p}{1 - p}\\right) = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots$\n",
    "\n",
    "$\\Rightarrow \\frac{p}{1 - p} = e^{\\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots}$\n",
    "\n",
    "$\\Rightarrow \\frac{1 - p}{p} = e^{-(\\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots)}$\n",
    "\n",
    "$\\Rightarrow \\frac{1}{p} = 1 + e^{-(\\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots)}$\n",
    "\n",
    "$\\Rightarrow p = \\frac{1}{1 + e^{-(\\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots)}}$\n",
    "\n",
    "$\\Rightarrow p = \\frac{1}{1 + e^{-z}}$\n",
    "\n",
    "where $z = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots$\n",
    "\n",
    "i.e. This is equivalent of saying that probability of the event is the logistic function applied to a linear combination of independent or predictor variables, where the logistic function is given by:\n",
    "\n",
    "$$\\sigma(z) = \\frac{e^z}{1 + e^z} = \\frac{1}{1 + e^{-z}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: By definition, the odds are given by:\n",
    "\n",
    "Odds = $\\frac{p}{1 - p} = e^z$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.429415Z",
     "start_time": "2019-04-12T05:09:52.422815Z"
    }
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1/(1 + np.exp(-z))\n",
    "\n",
    "def logit(p):\n",
    "    return np.log(p/(1 - p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.441124Z",
     "start_time": "2019-04-12T05:09:52.431618Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5, 0.7310585786300049)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(0), sigmoid(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.591852Z",
     "start_time": "2019-04-12T05:09:52.444128Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNX5x/HPQ1YgISyBsIUdQQQECQJuQMUWrUKrVlHRakWsSi2t2p/iWrGWttqqxQ3RUsEKLlTRUhcqwQVBQLawBMMaQAghTFayP78/ZiIjBpMwy83MPO/XKy/nZE7u/Z4kPrmcuxxRVYwxxoSXJk4HMMYY439W3I0xJgxZcTfGmDBkxd0YY8KQFXdjjAlDVtyNMSYMWXE3DSIiD4nIvCDsp4uIFIlIVKD31ZD9Bmv8vvDk7+F0DuMsK+7mWzyFoeajWkSOerWv8fO+OovImyKSKyL5IrJRRK4HUNU9qpqgqlX+3GddfNmviIzyfM+8v4fvBCKn1z7TRWSS9+c8+XcEcr+m8Yt2OoBpXFQ1oea1iOwCJqnqEq/PPeTH3c0F1gNdgTJgANDej9t3wn5V7ex0CGPsyN2cjFgReVlECkVkk4ik1bwhIh09R+OHRGSniNz+PdsZCsxR1WJVrVTVtar6X892uomIiki0p91dRD727HOJiDxdMz3i1fcGEckWkSMi8ksRGSoiG0TEJSIzvTI2EZH7RGS3iOR4xpL0Pftd5tnvh0DyyXzDRGSOiDzi1R4lInu92rtE5E5P3nwRWSAi8V7vjxeRdSJSICLbRWSsiPwBOBeY6flXwkxPXxWRXp7XSZ7xHfKM9z4RaeJ573oR+VREHvN8z3aKyIUnMz7T+FhxNydjHDAfaAksAmqKShPgHdxH452A84GpIvKjE2xnBfC0iEwQkS517PNfwBdAG+Ah4Npa+gwDegNXAk8A9wJjgNOAK0RkpKff9Z6P0UAPIKFmDCfY7xrcRX068PM6cvriCmAs0B0Y6MmIiJwJvAzchft7fh6wS1XvBT4BpnimYqbUss2/A0m4xzkSuA64wev9YUAm7vH9GXhRRMTvIzNBZ8XdnIxPVXWxZ156LnC65/NDgbaq+rCqlnvmfV8AJpxgOz/DXZzuB3Z6jkyHHt/JU/iHAg94tvsp7j8qx5uuqqWq+gFQDLyqqjmqus+zn8GeftcAf1XVHapaBNwDTKg5Wq9lv/erapmqfoz7j9f36ej5l0LNxxV19Pf2lKruV9U8z34GeT5/I/CSqn6oqtWquk9Vt9a1Mc9J4SuBe1S1UFV3AY/z7T+Mu1X1Bc/P8p9AByClAZlNI2XF3ZyMA16vS4B4T2HsynHFDZjGCYqFqh5R1btV9TRPn3XAW7UcOXYE8lS1xOtz2bVs8qDX66O1tGvOJ3QEdnu9txv3+afjc3YEjqhq8XF9v89+VW3p9fFaHf29Hf99rcmbCmxvwHZqJAOxfHesnWrbp9f3NwET8qy4G3/KBnYeV9wSVfWiur5QVXOBx3AX1NbHvf010FpEmnl9LtWHnPtx/yGq0QWo5Nt/DGr220pEmh/X92QUA975G3LiOBvoeYL3vu+xrrlABd8d674G7NuEKCvuxp++AApE5P9EpKmIRIlI/9qmWgBE5E+e96NFJBG4BchS1cPe/VR1N7AaeEhEYkVkBHCJDzlfBX7jOVmaADwKLFDVyhPs9/ee/Z7jw37XAReJSGsRaQ9MbcDXvgjcICLne04GdxKRvp73DuKeT/8Oz1TLa8AfRCRRRLoCvwUa9XX6xj+suBu/8RSTS3DPFe/EfeQ4G/cJvdo0A/4NuIAduI8wx52g7zXACOAw8AiwAPflkyfjJdznCj725CwFfnWCvlfjPumYBzyI+8Tmyai57HMX8AHu/PWiql/gPgn6NyAfWMaxo/Engcs9V7s8VcuX/wr3vxp2AJ/iPkH80skNwYQSscU6TCgSkQXAVlV90OksxjRGduRuQoLnmvWenmmJscB44C2ncxnTWNkdqiZUtAcW4r7OfS9wi6qudTaSMY2XTcsYY0wYsmkZY4wJQ45NyyQnJ2u3bt2c2v1JKy4upnnz5nV3DCORNuZIGy/YmEPJmjVrclW1bV39HCvu3bp1Y/Xq1U7t/qSlp6czatQop2MEVaSNOdLGCzbmUCIidd0lDdi0jDHGhCUr7sYYE4asuBtjTBiy4m6MMWHIirsxxoShOou7iLzkWYos4wTvi4g8JSJZniXCzvB/TGOMMQ1RnyP3ObiX/jqRC3EvbdYbmAw863ssY4wxvqizuHuWFsv7ni7jgZfVbQXQUkQ6+CugMcaYhvPHTUyd+PaSZ3s9n/v6+I4iMhn30T0pKSmkp6f7YffBVVRUFJK5fRFpY4608YKNOZBUlYpqKK2C0kqltFLplNCEqCaBXYfcH8W9toS1Po1MVWcBswDS0tI0FO8OC9W72nwRUWNetYo1W7cy5Je/dDpJUEXUz9ijIWMurajicHE5rpJy8o9WUHC0gvzvfFR+815RWSXFNR/lVVRVf7skfjHtfNq1iA/AqI7xR3Hfy7fXs+yMe41KY0LPXXfR0+WCCCvukUhVOVxUxtf5pexzHSWnsIzcwjIOF5eRW1hOblEZuUVlHC4qp7Cs8oTbiWoitIiPJqlpDElNY2jRNIaOLeNpHhtN87homsdF0Sw2moQ4Tzs2isT4mICPzx/FfREwRUTm416OLF9VvzMlY0xImDmTr1atotZFX01IUVWOlFSwM7eYXbnF7MkrYb/rKPvzj/K1q5TsvBIq3l/yna9r1SyG5IQ42iTE0r9TEskJcbRNjKNN81haNov1KuLugp4QF41IYKdYTkadxV1EXgVGAckishf3OpIxAKr6HLAYuAjIAkpwr/VoTGjq35/i3FynU5gGqKpWdh8uZuuBQrYdLGRXbjE7D5ew81ARBaXHjrhFoF1iHB1bNuXUDi04JaGMYQNOoUNSUzq2jKd9i3haNY8lJio8bv+ps7ir6lV1vK/AbX5LZIyTli+nRUYGRNj8c6goLqtk4758Nu8vYOuBgm8KemlFNeAu4B2TmtI9uTnjBnWkW5vmdE9uTrfk5qS2akZs9LHCnZ6ezqizuzs1lICzZfaM8TZtGj1cLpgyxekkEa+6Wtl+qIi1e1yszXaxLttF5oECas5NtmkeS98OiVwzrCt92ydyaocW9GqXQHxMlLPBGwkr7sZ4e/55MleuZJjTOSKQqpKVU8TnOw6zPOswK3YexlVSAUCL+GhOT23JBaN7MbhLK07r1IJ2iYG92iTUWXE3xlufPhz92q4HCBZXSTnpmYf4aGsOy7cfJreoDIBOLZtywakpnNm9NWd0bUX3Ns1pEuDrwsONFXdjvC1bRtK6dTbnHkDbDxWxZPNB/rclh9W786hWSE6I5ZxeyYzo2YazeiaT2rqZ0zFDnhV3Y7w9+CDdXS6YOtXpJGElO6+ERev38876/Ww9UAjAqR1acNvoXvygbztO79zSjsz9zIq7Md5eeomtK1Yw3OkcYcBVUs5ba/fx9vr9rN3jAmBI11Y8dEk/LjitPZ1aNnU4YXiz4m6Mtx49KN2zx+kUIUtV+WJnHq9+sYfFGQcor6ymb/tE/m9sXy4e2MGmW4LIirsx3pYsodX69Tbn3kBFZZW8vjqbeSt2s/1QMYlx0UwYmsqEoV3o17GF0/EikhV3Y7w98ghdXS644w6nk4SEA/mlzFm+i3+t3E1BaSWDUlvyl8sH8uOBHWgWa+XFSfbdN8bb3Lls+fxzRjido5HblVvMzKVZvL1uH1XVytj+7Zl0bg/O6NLK6WjGw4q7Md5SUynbvt3pFI3WnsMl/P2jr1i4dh8xUcLVZ3bhxnN60KWNzaU3NlbcjfH23nu03rDB5tyPc7CglCeWbOP11Xtp0kS4bkRXbhnV0+4SbcSsuBvjbcYMurhc8LvfOZ2kUSitqOKFj3fwTPp2KquruXpYF24d1Yv2SVbUGzsr7sZ4mz+fzcuXc5bTORymqryz4WtmLN7C/vxSxp7Wnnsu6kvXNs2djmbqyYq7Md7at6e8dWunUzhq+6Eipi3cyMqdeZzWsQWPXzGIET3bOB3LNJAVd2O8vfMObTZujMg59/LKap5btp2ZH2URH9OER386gCuHpgZ8IWcTGFbcjfH2+OOkulwwbZrTSYJqh6uKPzz1CV/lFHHxwA48cEk/O1ka4qy4G+PtjTfY9NlnnO10jiCprKpm5tIsnlpZSvsW8bx0fRo/6JvidCzjB1bcjfGWnExFUpLTKYJiV24xUxesY122ixEdo3h+8nm0iI9xOpbxEyvuxnhbuJDkCFhD9c01e7n/7Qyimwh/v2owiUe2WWEPM+GxzLcx/vLUU3ReuNDpFAFTWlHFPQs3csfr6xnQKYn3f3Mel5ze0elYJgDsyN0Yb2+/zcZPPuFcp3MEQHZeCbe8soaMfQX8cmRP7vzhKURH2fFduLLiboy3pCSqEhKcTuF3H287xK9eXUu1Ki9cl8YF/eykabizP9vGeFuwgLYffeR0Cr+au2I3N8xZRYekeN791TlW2COEFXdjvD37LJ0WLXI6hV9UVSsPLdrE/W9lMPKUtrxxy1n2+IAIYtMyxnhbvJgNH3/MeU7n8FFRWSW/+teXLM08xI3ndGfaRafanaYRxoq7Md6aNaM6PrTvzDxcVMYNc1axaX8Bj/ykPxOHd3U6knGATcsY423ePFI+/NDpFCdtn+soP3v+czIPFDLr2iFW2COYFXdjvM2eTYf//MfpFCclK6eIy59dzqGCMubeOIzzT7UTp5HMpmWM8fbhh6xftoyRTudooIx9+Vz74kqimjRh/s3DOa1jZDxCwZyYFXdjvMXEoNGh9b9Fxr58rpm9koS4aF6ZNIxuyXZFjKnntIyIjBWRTBHJEpG7a3m/i4gsFZG1IrJBRC7yf1RjgmDOHNq/957TKeotY18+E190F/b5k4dbYTffqLO4i0gU8DRwIdAPuEpE+h3X7T7gNVUdDEwAnvF3UGOCIoSK+6b97sLePDaaV28aTmrrZk5HMo1IfY7czwSyVHWHqpYD84Hxx/VRoIXndRKw338RjQmi9HTWPfGE0ynqtPVAAdfMXkmzmChevWk4XdpYYTffJqr6/R1ELgfGquokT/taYJiqTvHq0wH4AGgFNAfGqOqaWrY1GZgMkJKSMmT+/Pn+GkfQFBUVkRCGzx75PpE25sY+3pySah5dWQrAtGHxtGvm+0VvjX3MgRCqYx49evQaVU2rq199zhzVdlvb8X8RrgLmqOrjIjICmCsi/VW1+ltfpDoLmAWQlpamo0Lwmdnp6emEYm5fRNSYX3iBzMxM+jz2mNNJapVTWMqDz30OUdG8dvMITklJ9Mt2I+pn7BHuY65Pcd8LpHq1O/PdaZcbgbEAqvq5iMQDyUCOP0IaEzQLFtDuyBGnU9Qq/2gFP39pFYcKy3hl0jC/FXYTnurz77lVQG8R6S4isbhPmB7/ZKU9wPkAInIqEA8c8mdQY4JiyRLWP/640ym+o7Siikn/XEVWTiHPTRzC4C6tnI5kGrk6i7uqVgJTgPeBLbivitkkIg+LyDhPtzuAm0RkPfAqcL3WNZlvjKmX6mrlt6+tY/XuI/ztykGcd0pbpyOZEFCvuzVUdTGw+LjPPeD1ejNEzILxJpw98wwdt21rVGuo/uWDTBZvPMC0i/py8UBbEs/UT2jdimdMoL3zDm3y8pxO8Y35X+zh2fTtXD2sCzed28PpOCaEWHE3xtt//8vG9HRGOZ0D+Cwrl/veyuDc3sn8ftxpiNjz2E392VMhjWmEsnIK+eW8NfRsm8DT15xBjC1kbRrIfmOM8fbkk3R64w1HI+QfreCml9cQFx3Fi9en0SI+xtE8JjTZtIwx3v73P1odPuzY7qurld8sWEd2XgmvTh5O51b2WAFzcqy4G+Nt0SIyHJxzf2LJNj7amsP08acxtFtrh1KYcGDTMsY0Eu9vOsBTH2XxsyGdbXk84zMr7sZ4e+wxUhcsCPpus3IKueO19QzsnMT0n/S3K2OMz2xaxhhvn39Oi0PBfXJGcVklv5z3JfExTXhu4hDiY6KCun8Tnqy4G+PtzTfZFOQ59/vfzmD7oSLm3TiMji2bBnHPJpzZtIwxDnp9dTYLv9zH7T/ozdm9kp2OY8KIFXdjvM2YQZd//Ssou9p2sJD7385gRI823H5+76Ds00QOm5Yxxtu6dSTkBH4ZgpLySm575UsS4qJ58qpBRDWxE6jGv6y4G+Nt/nw2p6fTLsC7eeDtTWQdKmLuL4bRLjE+wHszkcimZYwJsrfW7uONNXv51Q96c05vm2c3gWHF3Rhv06fT9eWXA7b57LwS7n8rg7Surfi1zbObALJpGWO8ZWbS7ODBgGy6qlq547X1KPC3K22e3QSWFXdjvM2bx5b0dFICsOnnP97OF7vy+OsVp5Pa2h4IZgLLpmWMCYKMffn89YNt/HhgB346uJPTcUwEsCN3Y7w98ADddu3y6xqqR8uruH3+WpIT4viDPTfGBIkVd2O8ZWcT5+dnyzy6eAs7DhXzr0nDaNks1q/bNuZErLgb4+0f/yAzPZ0Oftrcsm2HmLtiNzed252z7PECJohszt2YACkoreDuNzfQu10Cd/ywj9NxTISx4m6Mt3vuofsLL/hlU394dwsHC0r5y89Ot8f4mqCz4m6Mt8OHicnP93kz6Zk5LFidzc0jezIotaUfghnTMDbnboy3WbPYlp5ORx82UVBawT0LN9K7XQJTx9hdqMYZVtyN8bNH3t1MTmEZz00cQly0TccYZ1hxN8bbnXfSMzv7pK9zX5qZw2ur93LrqJ6cbtMxxkE2526Mt6NHaVJWdlJfmn+0gnve3MgpKQn82qZjjMPsyN0Yb08/zVfp6ZzMAwIe/c8WDhWVMes6m44xzqvXkbuIjBWRTBHJEpG7T9DnChHZLCKbRCQ465QZ00is2HGYBauzmXRudwZ2tukY47w6j9xFJAp4GrgA2AusEpFFqrrZq09v4B7gbFU9IiKBXsjGmMCYOpVee/c2aM69rLKKe/+9kc6tmjL1/FMCl82YBqjPkfuZQJaq7lDVcmA+MP64PjcBT6vqEQBVDfwilMY0Es8v28H2Q8VM/0l/msbadIxpHERVv7+DyOXAWFWd5GlfCwxT1Slefd4CtgFnA1HAQ6r6Xi3bmgxMBkhJSRkyf/58f40jaIqKikhISHA6RlBF2pgbMt4DxdXc99lRzmgXxa2DQnct1Ej7GUPojnn06NFrVDWtrn71OaFa2/NJj/+LEA30BkYBnYFPRKS/qrq+9UWqs4BZAGlpaTrKj49VDZb09HRCMbcvIm3M9R2vqjLxxZU0ja1g5i9G0q5F6Bb3SPsZQ/iPuT7TMnuBVK92Z2B/LX3eVtUKVd0JZOIu9saElttuo/cTT9Sr61vr9vFZ1mF+N7ZvSBd2E57qU9xXAb1FpLuIxAITgEXH9XkLGA0gIsnAKcAOfwY1JiiaNqU6Lq7ObkeKy5n+7hYGpbbkmjO7BCGYMQ1T57SMqlaKyBTgfdzz6S+p6iYReRhYraqLPO/9UEQ2A1XAXap6OJDBjQmIxx5je3r6t/6pWpsZ/91K/tEK/njpAJrYQtemEarXTUyquhhYfNznHvB6rcBvPR/GhLWVnmvabx7Zg1M7tHA6jjG1sjtUjfE2eTKn7N9/wuvcyyqrmOa5pv3X59tpJdN4WXE3xlubNlQUF5/w7Vmea9r/ccNQmsXa/z6m8bLfTmO8/fGP7ExPp2stb+3MLebvS7P48cAOjO5jN2Gbxs2eCmlMPagq9721kbioJjx4cT+n4xhTJztyN8bbDTfQ58CB78y511zTPv0n/e2adhMSrLgb4y01lbKqqm99yq5pN6HIirsx3h5+mF3p6XTz+pRd025Ckc25G/M9Vno9p92uaTehxI7cjfE2cSKnHjwIo0bZNe0mpFlxN8Zbnz6UxMYCdk27CW32G2uMt/vvZ3d6OmLXtJsQZ3PuxhzHfU17hl3TbkKaHbkb423CBLKrWvNpzx/bNe0mpIXmkfuoUTBnjvt1RYW7PW+eu11S4m4vWOBu5+e72wsXutu5ue72O++42zU3rLznWRUwO9vdXrLE3d6xw91etgyApnv2uNvLl7vfz8hwt1etcrfXrXO3161zt1etcrczMtzt5cvd7cxMd3vZMnd7h+fx90uWuNvZ2e72e++52wcOuNvvvONu5+a62wsXutv5+e72ggXudkmJuz1vnrtdUeFuz5nz7Rt0XngBxow51n7mGbjwwmPtJ5+k/733Hms/9hhcdtmx9owZMGHCsfb06TBx4rH2Aw/ADTcca99zD0yefKx9551w223H2lOnuj9q3Habu0+NyZPd26hxww3ufdSYONGdocaECe6MNS67zD2GGuPGwZNPftN0ZWTyt9RRx65pHzPG/T2q4eDvHpmZEfe7x7hxx9p+/t3r+eyzzv7uBVhoFndjAmTGiKvIj2lq17SbkFfnAtmBkpaWpqtXr3Zk374I93UXaxMpY/5iZx5XPP85F3WP4Zmbf+h0nKCKlJ+xt1Ads4jUa4FsO3I3BiivrGbavzfSqaKQu1/9k9NxjPGZnVA1Bpj18Xaycor4R5sjVJx6itNxjPGZFXcT8XblFvPUR1n8eEAHRl/zY9LT0+npdChjfGTTMiaieV/T/sAldk27CR925G4i2tvr9vNpVi7Tx59GSot4GDeO/ocPw2efOR3NGJ9YcTcRy1VSzvR3NzMotSVXD/MsrHf++Rz56iuSnY1mjM+suJuINeO/W3EdrWDuTwcQVXNN+69/zb70dOwZkCbU2Zy7iUhf7Mxj/qpsbjynO/062nPaTfixI3cTccoqq7hn4QY6t2rK1DHHHaNfeCED8vJg5UpnwhnjJ1bcTcR5Nn072w8VM6e257RfcgmHt22jjTPRjPEbm5YxESUrp4hnlm5n3OkdGVXbc9pvvZX9P/lJ8IMZ42dW3E3EqK5Wpv17I/ExTbjfntNuwpxNy5iI8fqabL7YmcefLhtA28S42juNGcPpR47AmjXBDWeMn1lxNxHhUGEZf/jPFs7s3por0lJP3PHKK8nJzKRV8KIZExA2LWMiwvR3N1NaUc2jPx2AyPc8p/2mm/j64ouDF8yYAKlXcReRsSKSKSJZInL39/S7XERUROp81rAxwbI0M4dF6/dz6+ie9GqX4HQcY4KizmkZEYkCngYuAPYCq0RkkapuPq5fInA7YBcIm0ajpLyS+/6dQc+2zbllVD2e9ThqFINcrmNL1RkToupz5H4mkKWqO1S1HJgPjK+l33Tgz0CpH/MZ45MnlnzFPtdR/njpQOKio+r+guuv58DYsYEPZkyA1eeEaicg26u9Fxjm3UFEBgOpqvquiNzJCYjIZGAyQEpKCunp6Q0O7LSioqKQzO2LUB3zrvwqZq8o5bzO0ZTs3kD67np8UbduFCUncyAEx+uLUP0Z+yLcx1yf4l7b2advFl4VkSbA34Dr69qQqs4CZoF7DdVQXL8wVNdd9EUojrm8spoZMz+lbaLy9xtHktQ0pn5fWFHBsmXLGBli4/VVKP6MfRXuY65Pcd8LeF871hnY79VOBPoD6Z6rENoDi0RknKqG3grYJiw8vTSLrQcKefHnafUv7AAXXMDpNuduwkB9ivsqoLeIdAf2AROAq2veVNV8OPb4axFJB+60wm6csuXrAp5emsVPBnXk/FNTGvbFkybx9ZYttAxMNGOCps4TqqpaCUwB3ge2AK+p6iYReVhExgU6oDENUVlVzV1vrKdlsxgevOS0hm9g4kQOXnCB/4MZE2T1ukNVVRcDi4/73AMn6DvK91jGnJznP95Bxr4CnrnmDFo1j234BkpKaFJqF3yZ0GePHzBhIyunkCeXfMVFA9pz0YAOJ7eRiy5ioMsFdjmkCXFW3E1YqKpW7npjA83iovj9uP4nv6FbbmHfpk02525CnhV3ExZe+nQna/e4eOLKQSd+4mN9XHklh8L42mcTOezBYSbkbTtYyF8+yGTMqSmMH9TRt43l5xNVVOSfYMY4yI7cTUgrr6zmNwvWkRgXzR8vreOJj/UxfjwDXC6wJ0OaEGfF3YS0p/73FZv2F/D8tUN8m46pcfvt7M3IsDl3E/KsuJuQtWb3EZ5Jz+LyIZ350Wnt/bPRSy8lt3Vr/2zLGAfZnLsJSSXlldzx2jo6JDXlwUv8uB5qbi4x+fn+254xDrEjdxOSHl28hd15Jbx603AS4xvw7Ji6XH45p7lcML62p1obEzqsuJuQszQzh3kr9jDpnO4M79HGvxu/4w6yN260OXcT8qy4m5CSU1jKna+tp09KInf+qI//d3DJJRxOTPT/do0JMivuJmRUVyu/XbCe4vJK5l89nPiYeqys1FAHDhCbl+f/7RoTZFbcTch4/uMdfJqVyx8vHUDvlAAdXU+YQD+XCy69NDDbNyZIrLibkLB2zxEe/yCTHw/owIShqXV/wcm6+272bNhgc+4m5FlxN41eQWkFt89fS0qLeB71x12o32fsWPLi4wO3fWOCxIq7adRUlXv/ncF+Vymv3TyiYUvmnYzsbOJycgK7D2OCwIq7adTmrtjNO+v3c9eP+jCka6vA7/DaaznV5YIrrgj8vowJICvuptH6cs8Rpr+7mfP7tuOWkT2Ds9P77mP3+vU2525CnhV30yjlFpVx67wvaZ8Uz1+vGESTJgGcZ/c2ZgxHou1/CxP67LfYNDpV1crtr67lSEk5b95yFknNAjzP7m3HDuL37w/e/owJECvuptF5/INMlm8/zJ8vH0j/TknB3fkvfkFflwuuvjq4+zXGz6y4m0blvxu/5pn07Vx1ZipXpAXwevYT+f3v2bl2LYODv2dj/MqKu2k0Mvbl89vX1jO4S0sevOQ0Z0KMHEm+qjP7NsaP7HnuplHIKSzlppdX06pZDM9fOyQwz42pj8xMmu7Z48y+jfEjO3I3jiutqGLyy2twlVTw+i9H0C7RwTtEb76ZPi4XXHedcxmM8QMr7sZRqso9CzeyLtvFcxPPCP4J1OM9+ig7vvySM5xNYYzPrLgbRz29NIt/r93HHRecwtj+HZyOA2edRUF5udMpjPGZzbkbx7y+OpvHPtiS5qNYAAANm0lEQVTGTwd3YsoPejkdxy0jg+Y7dzqdwhifWXE3jliamcPdCzdyTq9k/nTZwMA+6bEhpkyh95NPOp3CGJ/ZtIwJug17Xdz2ypf0SUnk2YlnEBvdiI4x/vIXtq9ZwxCncxjjo3r9XyUiY0UkU0SyROTuWt7/rYhsFpENIvI/Eenq/6gmHOw+XMwv5qyidfNY5vxiKInxQXy0QH0MHUph375OpzDGZ3UWdxGJAp4GLgT6AVeJSL/juq0F0lR1IPAG8Gd/BzWhb5/rKFe/sJKqauWfvzjT2UseT2TdOhKyspxOYYzP6nPkfiaQpao7VLUcmA+M9+6gqktVtcTTXAF09m9ME+pyCkq55oUVFJRWMPfGYfRsm+B0pNpNnUqvmTOdTmGMz0TruNVaRC4HxqrqJE/7WmCYqk45Qf+ZwAFVfaSW9yYDkwFSUlKGzJ8/38f4wVdUVERCQiMtTAHi65gLypUZXxwl76hy59B4erV06O7TekjIyqKkpITqgQOdjhJU9nsdOkaPHr1GVdPq6lefE6q1XcZQ618EEZkIpAEja3tfVWcBswDS0tJ01KhR9dh945Kenk4o5vaFL2POL6lgwgsrOFwq/PPGYQzv0ca/4fxt1Cj7GUeIcB9zfaZl9gLej+frDHzngdciMga4FxinqmX+iWdC2eGiMq56YQXbc4qYdV1a4y/sAKtWkbh1q9MpjPFZfYr7KqC3iHQXkVhgArDIu4OIDAaex13YbXVhw8GCUq6ctYIduUXM/nkaI09p63Sk+rnrLno+95zTKYzxWZ3TMqpaKSJTgPeBKOAlVd0kIg8Dq1V1EfAXIAF43XMzyh5VHRfA3KYR23ukhGtmryS3sIw5N5wZGkfsNWbO5KtVqxjqdA5jfFSvm5hUdTGw+LjPPeD1eoyfc5kQteNQEde++AWFpRXMnTSMM7q0cjpSw/TvT3FurtMpjPFZI7o10IS6NbvzuOzZ5RytqOJfNw0PvcIOsHw5LTIynE5hjM+suBu/eC/jAFe/sJKkpjEsvOUs5x/de7KmTaPH7NlOpzDGZ/ZsGeOzOZ/t5PfvbmZQaktmX5dGm4Q4pyOdvOefJ3PlSoY5ncMYH1lxNyetoqqa6e9u5uXPd3NBvxSemjCYprGN9waleunTh6Nff+10CmN8ZtMy5qTkFpVxzeyVvPz5bm46tzvPTRwS+oUdYNkyktatczqFMT6zI3fTYBv35nPz3NUcLi7nyQmDGD+ok9OR/OfBB+nucsHUqU4nMcYnVtxNvakq81dl89CiTSQnxPFmKJ84PZGXXmLrihUMdzqHMT6y4m7qJf9oBdMWbuQ/G7/mnF7JPDFhEMmhfOL0RHr0oHTPHqdTGOMzK+6mTlmuKu576hO+zi/l/8b25ebzetCkSSNZFs/fliyh1fr1EMYPlDKRwYq7OaHyympmLs3i6ZWldGjZlNd/OSI0b0xqiEceoavLBXfc4XQSY3xixd3UKmNfPne+vp6tBwoZ0TGK5246l6SmjWxJvECYO5ctn3/OCKdzGOMjK+7mW0orqpj5URbPLttOm+axzL4ujeicLZFR2AFSUynbvt3pFMb4zIq7+cZHWw/y0KLN7Mkr4bIzOvPAxf1IahZDes4Wp6MFz3vv0XrDBptzNyHPirshO6+E37+zmSVbDtKzbXNemTSMs3slOx3LGTNm0MXlgt/9zukkxvjEinsEc5WU82z6duYs30VUE+GeC/tyw9ndiY2O4BuX589n8/LlnOV0DmN8ZMU9ApVWVDFn+S6eWZpFYVklPx3cibt+1IcOSU2djua89u0pb93a6RTG+MyKewQpraji9dXZPL10OwcKSvlB33b8bmwf+rZv4XS0xuOdd2izcaPNuZuQZ8U9AhSVVfLKit288MlOcovKGNK1FU9MGBRay98Fy+OPk+pywbRpTicxxidW3MPY1/lHeWXFHuau2E3+0QrO7Z3MraMGM7xHazxr3ZrjvfEGmz77jLOdzmGMj6y4hxlV5Yudefzz8128v+kg1apccGoKt47uxaDUlk7Ha/ySk6lICrOHoZmIZMU9TBwuKuPtdft5bXU2Ww8UktQ0hknndGfi8K6ktm7mdLzQsXAhyRkZNuduQp4V9xBWVlnFR1tyePPLvaRnHqKyWhnQKYkZlw5g/KBO4bF4RrA99RSdXS544AGnkxjjEyvuIaa0oopPvsrlvxlfs2TzQQpKK0lpEceN53bnsjM6c0pKotMRQ9vbb7Pxk0841+kcxvjIinsIOFJczsdfHeKDzQdZujWHkvIqWsRHM6ZfCuMHdeKcXslEhesjeIMtKYmqhASnUxjjMyvujVBFVTVr97j4eNshPvnqEBv25aMKyQlx/HRwJ8b2b8/wHm2IiYrgO0kDZcEC2m7aZHPuJuRZcW8EjpZXsS7bxapdeazalcfaPS6KyiqJaiIMTm3Jb8acwrm9kxnYuaUdoQfas8/SyeWChx92OokxPrHiHmRV1crO3CIy9hWwcV8+a3YfIWNfPpXVigj0SUnkJ4M7ck6vZEb0TI6cR+02FosXs+HjjznP6RzG+MiKewDlH60gK6eIrJxCNu8vIGN/AZv3F3C0ogqAuOgmnN65JZPP68HQbq05o0srkppZMXdUs2ZUx8c7ncIYn1lx91F5ZTX7XEfJzithZ26xp5gXkXWoiEOFZd/0ax4bxWkdk5hwZir9OybRv1MSPds2J9rmzRuXefNI2bLF5txNyLPi/j1UlfyjFeQUlpFTUMaBglI++aqcRTnr2Jt3lOwjJRwoKEX12NckxkXTs10CI09pS692CfRqm0Cvdgl0ad0sfBeVDiezZ9PB5YI//MHpJMb4JOKKe2lFFUdKyjlSXIGrpJwjJRXklZTjKi4nt6jMXcgLyzhYUEpOYRnlldXf+noB2icdJrVVM0b0bENqq2aktm5GaqumdEtuTrvEOHtuSyj78EPWL1vGSKdzGOOjehV3ERkLPAlEAbNVdcZx78cBLwNDgMPAlaq6y79R3fJLKjhQUEpRWQVFZVUUlVZSXFZJYZn7v0Wej+KySopK3a8LSis9hbyc0orqE247MT6adolxpLSIJ61rK1JaxNPW0675fNaGLxjzg9GBGJppDGJi0OiIO+YxYajO32IRiQKeBi4A9gKrRGSRqm726nYjcERVe4nIBOBPwJWBCPzKF7v583uZJ3w/LroJCXHRJMRH0zzW/d9OLeM5rWMLWjWLoWWzWFo1iz32unkMrZrF0rJZDHHRdd+uv8umVsLbnDm037rV5txNyKvPIcqZQJaq7gAQkfnAeMC7uI8HHvK8fgOYKSKi6j0b7R8/7JdC19bNaR4XRWJ8NM3j3EW85rXd2GN8MmcO7V0umDGj7r7GNGJSV/0VkcuBsao6ydO+FhimqlO8+mR4+uz1tLd7+uQet63JwGSAlJSUIfPnz/fnWIKiqKiIhAi7PT3Sxhxp4wUbcygZPXr0GlVNq6tffY7ca5uHOP4vQn36oKqzgFkAaWlpOioE/+mbnp5OKOb2RaSNOdLGCzbmcFSfOYy9QKpXuzOw/0R9RCQaSALy/BHQGGNMw9WnuK8CeotIdxGJBSYAi47rswj4uef15cBHgZhvN8YYUz91TsuoaqWITAHex30p5EuquklEHgZWq+oi4EVgrohk4T5inxDI0MYYY75fvS7oVdXFwOLjPveA1+tS4Gf+jWaMMeZk2XWDxhgThqy4G2NMGLLibowxYciKuzHGhKE671AN2I5FDgG7Hdm5b5KB3Dp7hZdIG3OkjRdszKGkq6q2rauTY8U9VInI6vrc+htOIm3MkTZesDGHI5uWMcaYMGTF3RhjwpAV94ab5XQAB0TamCNtvGBjDjs2526MMWHIjtyNMSYMWXE3xpgwZMXdByJyp4ioiCQ7nSWQROQvIrJVRDaIyL9FpKXTmQJFRMaKSKaIZInI3U7nCTQRSRWRpSKyRUQ2icivnc4ULCISJSJrReRdp7MEghX3kyQiqbgXDd/jdJYg+BDor6oDgW3APQ7nCQivxeAvBPoBV4lIP2dTBVwlcIeqngoMB26LgDHX+DWwxekQgWLF/eT9DfgdtSwnGG5U9QNVrfQ0V+BejSscfbMYvKqWAzWLwYctVf1aVb/0vC7EXew6OZsq8ESkM/BjYLbTWQLFivtJEJFxwD5VXe90Fgf8Aviv0yECpBOQ7dXeSwQUuhoi0g0YDKx0NklQPIH74Kza6SCBUq/FOiKRiCwB2tfy1r3ANOCHwU0UWN83XlV929PnXtz/jH8lmNmCqF4LvYcjEUkA3gSmqmqB03kCSUQuBnJUdY2IjHI6T6BYcT8BVR1T2+dFZADQHVgvIuCeovhSRM5U1QNBjOhXJxpvDRH5OXAxcH4Yr49bn8Xgw46IxOAu7K+o6kKn8wTB2cA4EbkIiAdaiMg8VZ3ocC6/spuYfCQiu4A0VQ3Fp8vVi4iMBf4KjFTVQ07nCRQRicZ9wvh8YB/uxeGvVtVNjgYLIHEfofwTyFPVqU7nCTbPkfudqnqx01n8zebcTX3MBBKBD0VknYg853SgQPCcNK5ZDH4L8Fo4F3aPs4FrgR94frbrPEe0JsTZkbsxxoQhO3I3xpgwZMXdGGPCkBV3Y4wJQ1bcjTEmDFlxN8aYMGTF3RhjwpAVd2OMCUP/D8dsG4Go/XSYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116fe0b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 4))\n",
    "plt.plot(np.arange(-5, 5, 0.01), sigmoid(np.arange(-5, 5, 0.01)))\n",
    "plt.grid()\n",
    "plt.hlines([0.5], -5, 5, linestyles='dotted', colors='r')\n",
    "plt.vlines([0], 0, 1, linestyles='dotted', colors='r')\n",
    "_ = plt.title(\"The Sigmoid Function\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:52.722930Z",
     "start_time": "2019-04-12T05:09:52.594700Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/havan/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in log\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VPW9//HXFxKCEAhkIewEAmFVdhdwARR31Kqt3Iq32lqsrVVv1Vat1Wrtrbe1rXrV1tZaW7HCraKCP0RFCLKIskOAAAlLQiA7E0hCyPb9/XFmJKUsITmZMzN5Px+P8zj5Zs6c8/5mJp+cfOcsxlqLiIhEjjZeBxAREXepsIuIRBgVdhGRCKPCLiISYVTYRUQijAq7iEiEUWEX1xljfm6MmeV1jpMxxvQ1xpQbY9p6neVMGGO2GGMmeZ1DQp8Ku5wxf1EMTPXGmCMN2re6vK3XjTFPu7lOa22OtTbWWlvn30a6MebOU2RIMcbY4/q90c1MJ9jmv/XbWjvcWpvektuVyKDCLmfMXxRjrbWxQA4wrcH33vQ6Xwvq0qCfI70OI3IyKuzSUtoZY/5ujDnsH0IYF3jAGNPTGPOOMabIGLPbGHNvUzZgjJlgjFltjCnzzyc0eKy/MeYz//YXGWNeCgwPNdgDjzLG/BK4CHjRvyf+4hlm+Jdhp4br9rfTjTG/MMas8Gf52BiT2GD5C40xK40xPmNMrjHmdmPMTOBW4Mf+TPP9y+4xxlzm/zrGGPOcMWa/f3rOGBPjf2ySMWafMeYBY0yhMeaAMeaOpvyMJTypsEtLuQ6YDXQB5gEvAhhj2gDzgY1AL+BS4H5jzBVnsnJjTDzw/4AXgATgd8D/M8Yk+Bf5B/Cl/7GfA7edaD3W2p8Cy4B7/Hvi95xJjkb6JnAH0A1oBzzo70Nf4EPgf4EkYBSwwVr7J+BN4Nf+TNNOsM6fAuf7nzMSOBd4rMHj3YE4nJ/xd4CXjDFd3e+ahCIVdmkpy621C/zj2G/gFB+A8UCStfYpa221tXYX8Gdg+hmu/xpgp7X2DWttrbX2LSATmOYvmOOBx/3bWI7zx6W5iv171j5jzINn8Ly/Wmt3WGuPAP+HU4zB2StfZK19y1pbY60tsdZuaOQ6bwWestYWWmuLgCf51z9eNf7Ha6y1C4ByYPAZZJYwFuV1AIlY+Q2+rgTa+4cn+gE9jTG+Bo+3xdlrPhM9gb3HfW8vzh5qT6DUWlvZ4LFcoM8ZbuN4idba2iY87/ifRaz/6z5AdhOzHN//vf7vBZQcl7XhdiXCaY9dgi0X2G2t7dJg6mStvfoM17Mf549EQ32BPOAAEG+M6dDgsVMV9eZc4rQCaLid7mfw3Fwg9SSPnS7T8f3v6/+eiAq7BN2XwCFjzE+MMWcZY9oaY0YYY8af4jltjTHtG0ztgAVAmjHmm/4PQW8BhgEfWGv3AmuAnxtj2hljLgBONE4dUAAMaGJ/NgAX+4+NjwMeOYPnvglcZoz5hr8PCcaYwDDN6TK9BTxmjEnyfxj7OBCy5w5IcKmwS1D5x9yn4Ywz7waKgVdxPug7mYeBIw2mxdbaEuBa4AGgBPgxcK21ttj/nFuBC/yPPQ3MAY6eZP3PAzcbYw4aY144w/584l/3JmAt8MEZPDcHuNrfh1KcPxKBzyL+Agzzj+e/d4KnP43zx2sTsBlY5/+eCEY32pDWwBgzB8i01j7hdRaRlqY9dolIxpjxxphUY0wbY8yVwPXAifZ8RSKOjoqRSNUdmItzHPs+4G5r7XpvI4kEh4ZiREQijIZiREQijCdDMYmJiTYlJaVJz62oqKBjx47uBgpx6nProD63Ds3p89q1a4uttUmnW86Twp6SksKaNWua9Nz09HQmTZrkbqAQpz63Dupz69CcPhtjjj/b+oQ0FCMiEmFU2EVEIowKu4hIhFFhFxGJMCrsIiIRRoVdRCTCqLCLiEQYFXYRkQijwi4iEmFU2EUCVq92JpEwp8v2igQ89JAzT0/3NIZIc6mwiwS8+KLXCURcocIuEjBihNcJRFyhMXaRgJUrnUkkzGmPXSTg0UeducbYJcypsIsEvPKK1wkkwhQcquLvn+9h+vi+9InvELTtaihGJGDwYGcSccmXu0t5aUk2vsqaoG5XhV0kYOlSZxJxyfocHzFRbRjSo1NQt6uhGJGAJ55w5hpjF5eszz3IOb3jiG4b3H1oFXaRgNde8zqBRJCjtXVsyTvE7RNTgr5tFXaRgAEDvE4gEWTr/kNU19Uzuk+XoG9bY+wiAYsWOZOIC9bn+AAY3bdr0LetPXaRgKefduaXXeZtDokI63N99IhrT/e49kHftgq7SMAbb3idQCLI+pyDjO4b/GEY0FCMyDF9+jiTSDMVHq5i38EjjPFgGAZcLOzGmLbGmPXGmA/cWqdIUC1c6EwizbThq/F1b/bY3RyKuQ/YBnR2cZ0iwfPMM878yiu9zSFhb32uj+i2huE94zzZviuF3RjTG7gG+CXwIzfWKRJ0s2d7nUAixPqcgwzr0Zn20W092b6x1jZ/Jca8DfwK6AQ8aK299gTLzARmAiQnJ4+d3cRfovLycmJjY5uRNvyoz62D+hwZ6uot3/+0kot6RTFjWMy/Pd6cPk+ePHmttXbcaRe01jZrAq4FXvZ/PQn44HTPGTt2rG2qJUuWNPm54Up9DpJ585zJI3qdI8OWvDLb7ycf2PfW7zvh483pM7DGNqIuuzEUMxG4zhhzNdAe6GyMmWWtneHCukWC57e/debTpnmbQ8Laom0FAIxPifcsQ7MLu7X2EeARAGPMJJyhGBV1CT9vv+11Aglz1lrmrtvH+QPi6dnlLM9y6Dh2kYDERGcSaaK1ew+yp6SSm8b09jSHq2eeWmvTgXQ31ykSNHPnOvMbb/Q2h4Std9blcVZ0W646u4enOXRJAZGAF15w5irs0gRVNXV8sGk/V43oTmyMt6VVhV0k4P33vU4gYeyTrQUcrqrlprHeDsOACrvIMXHenCUokeGddfvoEdee8wckeB1FH56KfGXOHGcSOUOFh6r4bEcRXxvdi7ZtjNdxtMcu8pU//MGZ33KLtzkk7Lyxai/1lpAYhgEVdpFjFizwOoGEoZLyo7y2fDfXntOD1KTQuDyCCrtIQIcOXieQMPTHpdkcqanj/svSvI7yFY2xiwTMmuVMIo1UcKiKv3++l6+N7s3AbqGxtw7aYxc55tVXnfkMXRFDGuelJVnU1Vvuu3SQ11H+hQq7SMAnn3idQMJIbmklb32Zwy3j+9A3IbSG8VTYRQKio71OIGHCWssT87bQto3hnikDvY7zbzTGLhLw+uvOJHIa767PY3FmIQ9dMYQecd5dxfFkVNhFAlTYpREKD1fx5PytjO3XldsnpHgd54Q0FCMSkJ7udQIJcdZafvZeBkdq6vifm84JibNMT0R77CIijfTu+jw+2lLAj6amhdThjcdTYRcJ+POfnUnkBLbsL+PRdzdzbko8d17Y3+s4p6TCLhKgi4DJSRysqOauN9bStUM7Xrp1DFFtQ7t0aoxdJGDRIq8TSAiqravnnrfWUXj4KP+86wKSOsV4Hem0QvvPjoiIh6y1PDl/KyuySnj6hhGM7NPF60iNosIuEvDyy84k4vfbj3fwxqq93HXxAL4xro/XcRpNhV0kYP58ZxIB/vRZNi8uyeI/zu3Dw1cN8TrOGdEYu0jAhx96nUBCxBur9vLfCzK55pwePH3D2RgTmsern4wKu4hIAy+nZ/Hrhdu5dEg3fv+NUSF7EtKpqLCLBDz/vDO/7z5vc4gnrLX8z8Lt/HFpNteN7MlvvzGS6BA/rPFkwjO1SEv49FNnklanuraen7yziT8uzebW8/ry+1tGhW1RB+2xixwzb57XCcQDpRXVfG/WWr7cXcq9UwbyX1PTwm5M/Xgq7CLSau0sOMx3/raG/ENVPD99FNeP6uV1JFeosIsEPPusM3/wQW9zSFC8u34fj87NoGNMFLNnns+Yvl29juQaFXaRgM8/9zqBBEFVTR1Pzt/CW1/mcm7/eP73P0aT3Lm917FcpcIuEvDOO14nkBa2ZX8Z/zVnAzsKyvn+pFR+NDUt5C/o1RTNLuzGmD7A34HuQD3wJ2vt881dr4iIW+rqLX9cms1zi3bQtUM7/vbtc7kkLcnrWC3GjT32WuABa+06Y0wnYK0x5hNr7VYX1i0SPM8848wfftjbHOKqzPxDPPzOZjbk+pwzSa8fQdeO7byO1aKaXdittQeAA/6vDxtjtgG9ABV2CS8bNnidQFxUVVPHi4uz+OPSbDqfFc3z00dx3cieYX8oY2O4OsZujEkBRgNfuLlekaCYPdvrBOKSJZmFPDl/C3tKKrlxTC8eu2YY8RG+l96Qsda6syJjYoGlwC+ttXNP8PhMYCZAcnLy2NlN/CUqLy8nNjZ07zXYEtTn1kF9br7Cynr+sa2aDUV1dO9gmDEshhGJbV1bvxua0+fJkyevtdaOO91yrhR2Y0w08AHwkbX2d6dbfty4cXbNmjVN2lZ6ejqTJk1q0nPDlfocJL/4hTP/2c+Cu10/vc5NV1ZZw4tLdvK3lXuJamu499JBfHtif9pFhd4RL83pszGmUYXdjaNiDPAXYFtjirpIyNq+3esEcoaO1tbx5qocXli8k7IjNdw8pjcPXD6Y7nGRdVz6mXJjjH0icBuw2RgT+PTpUWvtAhfWLRI8s2Z5nUAaqbaunrnr8nhu0Q72l1Vx4cBEHr16KMN6dvY6Wkhw46iY5UDkf8wsIp6rravn/Q37eXFJFruLKxjZpwu/+fpIJg5M9DpaSNGZpyIBjz/uzJ96ytsc8m+qa+t5b0MeLy/JYk9JJUN7dOaV28Zy+bDkVnH44plSYRcJyM31OoEcp7K6ltlf5vLqsl3sL6tieE+noE8dmkybMLyzUbCosIsE/PWvXicQv6LDR/n753uYtWovBytrODclnl/eeDaT0pK0h94IKuwiEjK2HTjEX1fs5r31+6mpr+fSIcncdckAxqfEex0trKiwiwQ88ogz/9WvvM3RytTW1bNoWyGvr9zNql2ltI9uw9fH9eY7F/ZnQFLrOmHLLSrsIgElJV4naFUKD1XxflY1j3y+hANlVfTqchaPXDWEW8b3oUuH1nP6f0tQYRcJ+NOfvE4Q8errLcuzinnryxw+2VpAbb3lokFxPDFtOJcN7RaR10b3ggq7iLS4/b4jvL12H/9cm0tu6RG6dojmjokppJLP9GvO8zpexFFhFwkI3Os0cO9TaZaqmjo+3lrA22v3sWxnEdbCBQMSeOiKIVwxPJmYqLakpxd6HTMiqbCLBBw54nWCsGetZc3eg8xdl8cHm/ZzuKqWnnHt+eGUQXx9bG/6xHfwOmKroMIuEvDSS14nCFtZheXM25DHuxvyyC09wlnRbblqRHduHtub8wck6GSiIFNhF5Em2e87wgeb9vP+hv1s2X+INgYmDkzkR1PTuHxYdzrGqLx4RT95kYD773fmzz3nbY4QVnioigWbD/DBpgOs2XsQgFF9uvD4tcO49pwedOvcui+XGypU2EXklAoOVfHh5gMs2JzP6r2lWAtDunfioSsGc+05PeiX0NHriHIcFXaRAO2pfyW3tJKPtuTzYUY+a/175mnJsdx/aRrXnNOdgd06eZxQTkWFXUSw1rKjoJyPt+SzcEs+W/YfAmBoj878aGoaV43ozqBkFfNwocIuEvCDHzjzVnJ0TF29ZV3OQT7eks/HWwvYW1IJwJi+XXjkqiFcMbw7KYkaZglHKuwiAWed5XWCFldxtJZlO4tYtK2QxZmFlFZUE93WMCE1ke9eNICpw5JJ1gegYU+FXSQgQs843XewksWZhSzaVsiq7BKq6+rp3D6KKUO6cdmwZC5JS6JT+2ivY4qLVNhFIkxtXT3rcnwszixkSWYh2wsOA5CS0IH/vKAflw5NZlxKV6J1wa2IpcIuEjBzpjMPw6s8Fh0+ytIdRaRvL2TZzmLKjtQQ1cYwPiWex64ZypQh3XRt81ZEhV0kICHB6wSNVltXz4ZcH+nbi1i6o4jNeWUAJHWK4fJhyUwZ0o2JgxLprCGWVkmFXSQgxO+cdKDsCJ/tcAr58p3FHKqqpY2BMX278sDUNCYP6cawHp11XRZRYRcJVVU1dXyxu5TPdhTx2Y4idhaWA5DcOYYrR3Tn4rQkLhqYRFwH7ZXLv1JhFwm44w5n/te/erJ5ay3bDhxi2c4ilu0s5ovdpVTX1tMuqg3npsTz9XG9uTgticHJnTBGe+VycirsIgF9+gR9kwWHqli2s5jlO4tYvPUIhz5aBsCgbrHcdn4/LhqUyHn9EzirXdugZ5PwpcIuEvDUUy2+iYqjtXyxu8RfzIu/Gl5J6NiOYQltuGnicC4alET3OJ0kJE2nwi7Sgmrq6tm0z8fynSWsyCpmXc5BaustMVFtOLd/PDeP7c2FgxIZ2r0zn322lEnjgv9fg0QeFXaRgBkznPmsWU1ehbWW7QWHWZFVwsqsYlbtKqGiug5j4Oxecdx50QAuGpTI2H5daR+t4RVpGSrsIgGDBzfpabmllazMLnaKeXYJxeVHAeif2JEbRvdi4sBEJqQm0KVDOzfTipyUCrtIwM9+1qjFSsqPsjK75KtinlPqXBUxMTaGiQMTmJiayISBCfTuqhs3izdcKezGmCuB54G2wKvW2mfcWK9IKDhcVcOXu0tZme2Mk2fmO9de6RQTxXkDErhjYgoTByYyqFusDkOUkNDswm6MaQu8BEwF9gGrjTHzrLVbm7tukaCaPh2AqjfeZN3eg04hzy5m074y6vwfeI5L6cpDVwxmQmoCZ/eKI0oX0pIQ5Ma78lwgy1q7y1pbDcwGrndhvSc06v774fXXnUZNDUyadOzDrspKpz1njtMuK3Pac+c67eJipz1/vtPOz3faCxc67dxcp71okdPetctpL13qtLdvd9orVzrtjAynvXq1096wwWlv2OC0V6922hkZTnvlSqe9fbvTXrrUae/a5bQXLXLaublOe+FCmDSJdqWlTnv+fOfx4mKnPXeu0y5zrhPCnDlOu9IZGmDWLKddU+O0X3/daQf8+c9w2WXH2i+/DFdddaz9/PNw3XXH2s8+CzfddKz9zDNfFUMAfvGLYx9AAjz++LGTfgAeeeTYhbYAHnzw2M0twLmZdOCG0uA89uCDx9ozZzrrCLjjDmcbATNmOBkCpk93MgbcdNO/Xpr3uuvg+ef9V0M8yEsV8XwzaQrnPPkx33z1C/6weAcUFHL3Jan8487z2LjiWd6MyuQHkwcyukcsUZdOcfW9N+r++0PuvUd+vtPWe8/d914Lc2MopheQ26C9Dzjv+IWMMTOBmQDJycmkp6c3aWNn19WRmZlJfno6praWkT4fB7ZtoyA9nTZVVZzj85G3ZQtF6em0LS/nbJ+PfRkZFMfHE11WxnCfj9zNmynp1Il2paUM8/nI2bSJ0vbtiSksZKjPx96NGzkYFUX7/fsZ4vOxe/16yqzlrJwcBvt87Fq3jkPV1XTcvZtBPh/Za9dyuKKC2KwsBvp8ZK1ZQ7nPR6fMTFJ9PnauXk1FcTGdMzIY4POx/YsvOHLgAHEbNtDf5yNz1SqqcnLounEj/Xw+tn3+OUezs4nftIm+Ph8VFRWkp6eTsHkzfXw+tqxYQU1cHIkZGfT2+di8bBl1sbEkbdlCL5+PTZ99Rn379iRv20YPn4+NS5dio6LonplJd5+PDf6ffY/t2+l28CAb/e2eO3aQUFrKZn+7186ddC0pIcPf7pOdTeeiIrb423137SK2sJCt/na/3bvpUFDANn87Zc8eYoqK2O5v98/JIbqsjB3+dmpuLm2OHmWnvz1w3z4AstLTKS8vJy8vj/qYGLL9j6ft309NRQW7/e3B+fkcratjj789tKCAynbt2OtvDysspHzXLnL87eFFRRzKzmbvkiXklVsOxgxgZW5nNj7xIUdqgeHXMKD+MJN7tWFoQjQ3Pfs4lRdP4EBMV6r3HaDqYAl7WvC9l1ZXx8YQe+9tXbmS6vj4FnvvlZeXk56eHlLvPYBBLfTey/W/t5ta/xrLWGubtwJjvg5cYa2909++DTjXWvvDkz1n3Lhxds2aNU3aXnp6OpMa/uVvBdTn5rHWklt6hBXZxazIKubz7BJKKqoB5xrlF6QmMnFgAhcMSCAhNsaVbTaFXufWoTl9NsastdaOO91ybuyx7wManlXRG9jvwnpFmqzwcBUrs44duZLnOwI4F9C6JC2JC1ITmDAwkV5dGtwOL/Cv/jvveJBYxD1uFPbVwCBjTH8gD5gOfNOF9Yo02qGqGr7YVcqKrGJWZhezo8A5Vb9z+yguSE3grksGMCE1kdSkjic/cuWCC4KYWKTlNLuwW2trjTH3AB/hHO74mrV2S7OTiZzC0do61u31sTK7mOVZx45caR/dhvEp8XxtdG8mDkxgeM842jb2+uQNPywTCWOuHMdurV0ALHBjXSInUl9v2ZZ/iBVZxSzPKuHL3SVU1dTTxsDIPl24+5JUJg5MZEy/LsRE6VR9ad105qmErP2+Iyzf6eyRr8gq/uoDz4HdYpk+vi8TUhM4PzXBvdu/BQ6vmzfPnfWJeESFXUJG+dFaVmWXsGxnER9tqiR/4WLAuY/nxWlJTByYyIUDE1vukraXXtoy6xUJMhV28UxdvSUjr4xlO4v4bGcx6/Y6l7RtH92GQXFtuHNSGhcOSgzeHYPuu6/ltyESBCrsElQFh6pY6r+H54qsYg5WOmcmjujVme9efOyStp8vX8akiwZ4nFYkPKmwS4s6WlvHmj0HvyrmgQtoJXWKYcqQZC5Oc4ZXvDwx6CuBU9o//NDbHCLNpMIursstrSR9RxFLtxeyMruEyuo6otsaxqfE8/BVQ7gkLYkh3UPwhszTpnmdQMQVKuzSbNW19azZU8qS7YUs2V5Elv8+nr27nsWNY3oxKa0bF6Qm0DEmxN9u3/++1wlEXBHiv2kSqooOH3UKeWYhy3YWU360lnZt23DegHimj+/DpMHdTn2Wp4i0GBV2aRRrLdsOHObTbQUsyixkY64PgO6d2zNtZE+mDOnGhHDYKz+VwGVkA5fOFQlTYfxbKC2turaeL3aX8MnWAj7dVkie7wjGwDm9u/DA1DSmDO3GsB6dI2ev/JZbvE4g4goVdvkXh6pqSN9exMdb8lm6vYjDR2tpH92GiwYlcd+lg5g8pBtJnULgCJaW8N3vep1AxBUq7EJx+VE+2VrAwox8VmYXU1NnSejYjqvP7sHUYclcOCiR9tG6/opIuFBhb6UOlB1hYUY+H2bks3pPKdZC3/gO3D4hhSuGd2d0366NvypipAjc/KCF724j0tJU2FuRA2VHWLA5nwWbD7B270EA0pJj+eGUQVw5vDtDe4TgseXBdPvtXicQcYUKe4QrPFzFh5vzmb9xP2v8xXxoj848eHkaV47owcBusR4nDCEq7BIhVNgjUNmRGj7KyOf9jXl8nl1CvYUh3Tvx4OVpXH12DwYkqZifUI1z3RqiXboMsIhHVNgjxNHaOtK3F/HuujwWZxZSXVdPSkIH7pk8kGkjezIouZPXEUPf1KnOXGPsEuZU2MOYtZYNuT7eWbeP+RsPUHakhsTYGG49vy83jOrFOb3jWveY+Zm6806vE4i4QoU9DBUequKddXn8c20uu4oqaB/dhiuGd+dro3tx4cBEotq28TpieJoxw+sEIq5QYQ8TtXX1LM4s5P/W5LJkexF19ZbxKV256+IBXH12Dzq5dXu41qyy0pl36OBtDpFmUmEPcXm+I8zdWc1PVi6m4NBRkjrFMPPiAXx9bG99COq2q6925hpjlzCnwh6C6ustn+0sYtaqvSzOLMRamDQ4iV9c35cpQ7ppqKWl3H231wlEXKHCHkIOV9XwzzX7+Pvne9hTUklibDvunpRK//r93HzVuV7Hi3y6CJhECBX2EJBbWslflu/mn2tyqaiuY0zfLvzX1DSuGtGDdlFtSE/P9zpi61BW5szj4rzNIdJMKuweWp9zkD8v28XCjHzaGMO0kT25fUIKI/t08Tpa63T99c5cY+wS5lTYg8xay7KdxbycnsWqXaV0bh/FzItTuX1CCt3j2nsdr3W7916vE4i4QoU9SKy1LNpWyAuf7mRzXhnJnWN47JqhTD+3L7HhfNehSHLjjV4nEHGFKkoLs9byydYCnv90J1v2H6JvfAeeufFsvjamFzFRusZ5SCkuduaJid7mEGkmFfYWtHxnMb/5KJON+8pISejAs18fyQ2jeupwxVB1883OXGPsEuZU2FtARl4Zv/pwGyuySugZ155f33QON47ppYIe6h54wOsEIq5oVmE3xvwGmAZUA9nAHdZanxvBwtF+3xGe/Xg7767Po8tZ0Tx+7TBuPb+vhlzCxbRpXicQcUVz99g/AR6x1tYaY/4HeAT4SfNjhZeqmjpeXbaLF5dkUW9h5sUD+MHkgXTW9VvCS77/fIHu3b3NIdJMzSrs1tqPGzRXATc3L074WZxZwJPzt7K3pJKrz+7OI1cNpU+8LiIVlqZPd+YaY5cw5+YY+7eBOS6uL6QVHq7i5/O2sGBzPqlJHZn1nfO4cJCOpghrDz/sdQIRVxhr7akXMGYRcKL/TX9qrX3fv8xPgXHAjfYkKzTGzARmAiQnJ4+dPXt2kwKXl5cTG+vdVQ2ttXyWV8uczGqq6+H61Giu6h9NVJuWu6GF1332gvrcOqjPZ2by5MlrrbXjTrugtbZZE/At4HOgQ2OfM3bsWNtUS5YsafJzm6ug7Ij91mtf2H4/+cDe8spKm114OCjb9bLPXvGkzzk5zuQRvc6tQ3P6DKyxjaixzT0q5kqcD0svsdZWNmddoW7B5gM8+u5mqmrqeOr64cw4rx9tWnAvXTxw223OXGPsEuaaO8b+IhADfOK/t+Yqa+33mp0qhFTV1PHk/K289WUOI3vH8btbRpGqG1xEpsce8zqBiCuae1TMQLeChKI9xRV8/811bD1wiO9dksoDl6cRrZOMItdll3mdQMQVOvP0JBZnFnDvWxto28bw2u3jmDIk2etI0tJ27XLmAwZ4m0OkmVTYj2Ot5S/Ld/PLBdsY1qMzr9w2lt5ddVx6q/DtbztzjbFLmFNhb6Cmrp7H3s0OSsFuAAAKI0lEQVRgzppcrhzend/dMpIO7fQjajWefNLrBCKuUNXyq6qp4/tvrmNxZiE/mJzKA1MH66iX1uaSS7xOIOIKFXacm0jf+bc1fLmnlKdvGMGM8/t5HUm8sH27Mx882NscIs3U6gu7r7Ka/3ztS7buP8Rzt4zi+lG9vI4kXrnrLmeuMXYJc626sFccreX2v64m88BhXrltLJcO1ZEvrdp//7fXCURc0WoL+9HaOma+sYbNeWW8fOsYFXWBCRO8TiDiilZ5tk1tXT33vrWeFVkl/Pqmc7hiuK6/LUBGhjOJhLlWucf+qw8z+WhLAT+fNoybxvb2Oo6EinvuceYaY5cw1+oK+3vr8/jL8t3cPiGF2yf29zqOhJLf/MbrBCKuaFWFPSOvjJ+8s4nz+sfz02uGeh1HQs348V4nEHFFqxljP1hRzV1vrCW+YzteunWMLuYl/27DBmcSCXOtZo/9Z+9nUHi4ire/N4HE2Biv40gouv9+Z64xdglzraKwL8zI54NNB3jw8jRG9unidRwJVc8953UCEVdEfGE/WFHNY+9lMLxnZ+66JNXrOBLKRo3yOoGIKyK+sD/1wVZ8ldX87dvjNa4up7Z6tTPXh6gS5iK6sKdvL+Td9XncO2Ugw3vGeR1HQt1DDzlzjbFLmIvYwl5fb/n1wu30je/APVMGeR1HwsGLL3qdQMQVEVvYF27JZ+uBQ/z+lpG0i9IQjDTCiBFeJxBxRURWvLp6y+8+2cHAbrFcN1KX4ZVGWrnSmUTCXETusb+/IY+swnJevnUMbXUXJGmsRx915hpjlzAXcYW9pq6e5xbtZFiPzlypqzbKmXjlFa8TiLgi4gr7e+vzyCmt5C/fGqd7lsqZ0S3xJEJE3Bj7nNW5pCZ1ZMqQbl5HkXCzdKkziYS5iNpj31NcwZq9B/nxlYMxRnvrcoaeeMKZa4xdwlxEFfa56/ZhDHxttI6EkSZ47TWvE4i4ImIKe3295Z11eVw4MJEecWd5HUfC0YABXicQcUXEjLF/sbuUPN8RbhqjW91JEy1a5EwiYS5i9tjnrttHbEyUbkwtTff00878ssu8zSHSTBFR2Cura1mw+QDXnNODs9q19TqOhKs33vA6gYgrXBmKMcY8aIyxxphEN9Z3pj7eUkBFdR03ahhGmqNPH2cSCXPNLuzGmD7AVCCn+XGaZnlWMfEd23FuSrxXESQSLFzoTCJhzo2hmN8DPwbed2FdTbI+5yBj+nbRmabSPM8848yvvNLbHCLNZKy1TX+yMdcBl1pr7zPG7AHGWWuLT7LsTGAmQHJy8tjZs2c3aZvl5eXExsZ+1a6osfzg00puGhTNtNR2TVpnqDu+z62BF31uV1oKQHW8N//56XVuHZrT58mTJ6+11o477YLW2lNOwCIg4wTT9cAXQJx/uT1A4unWZ61l7NixtqmWLFnyL+307YW2308+sCt2FjV5naHu+D63Bupz66A+nxlgjW1EjT3tUIy19oTHfhljzgb6Axv9p+/3BtYZY8611uaf9i+KS9bnHMQYOKdPl2BtUiLV/PnOfNo0b3OINFOTx9ittZuBr660dbqhmJayPsfH4OROxMZExJGb4qXf/taZq7BLmAvralhfb9mQ6+Pqs3VSkrjg7be9TiDiCtcKu7U2xa11NdbukgrKjtQwuk/XYG9aIlGiJ6dhiLgurK8Vsz7HB8DovhpfFxfMnetMImEurIdi1uccpFNMFKlJretwKWkhL7zgzG+80dscIs0U5oXdxyidmCRued+zc+xEXBW2QzGV1bVk5h9itA5zFLfExTmTSJgL28K+aV8Z9RZG99UHp+KSOXOcSSTMhe1QTOCD01HaYxe3/OEPzvyWW7zNIdJMYVvYe8S152uje9G1Y2ReH0Y8sGCB1wlEXBG2hf2G0b24QTetFjd16OB1AhFXhO0Yu4jrZs1yJpEwF7Z77CKue/VVZz5jhrc5RJpJhV0k4JNPvE4g4goVdpGA6GivE4i4QmPsIgGvv+5MImFOhV0kQIVdIoSGYkQC0tO9TiDiCu2xi4hEGBV2EZEIo8IuIhJhVNhFRCKMCruISIRRYRcRiTAq7CIiEUaFXUQkwqiwi4hEGGOtDf5GjSkC9jbx6YlAsYtxwoH63Dqoz61Dc/rcz1qbdLqFPCnszWGMWWOtHed1jmBSn1sH9bl1CEafNRQjIhJhVNhFRCJMOBb2P3kdwAPqc+ugPrcOLd7nsBtjFxGRUwvHPXYRETkFFXYRkQgTsoXdGHOlMWa7MSbLGPPwCR6PMcbM8T/+hTEmJfgp3dWIPv/IGLPVGLPJGPOpMaafFznddLo+N1juZmOMNcaE9aFxjemvMeYb/td5izHmH8HO6LZGvK/7GmOWGGPW+9/bV3uR003GmNeMMYXGmIyTPG6MMS/4fyabjDFjXA1grQ25CWgLZAMDgHbARmDYcct8H/ij/+vpwByvcwehz5OBDv6v724NffYv1wn4DFgFjPM6dwu/xoOA9UBXf7ub17mD0Oc/AXf7vx4G7PE6twv9vhgYA2Sc5PGrgQ8BA5wPfOHm9kN1j/1cIMtau8taWw3MBq4/bpnrgb/5v34buNQYY4KY0W2n7bO1dom1ttLfXAX0DnJGtzXmdQb4BfBroCqY4VpAY/r7XeAla+1BAGttYZAzuq0xfbZAZ//XccD+IOZrEdbaz4DSUyxyPfB361gFdDHG9HBr+6Fa2HsBuQ3a+/zfO+Ey1tpaoAxICEq6ltGYPjf0HZy/+OHstH02xowG+lhrPwhmsBbSmNc4DUgzxqwwxqwyxlwZtHQtozF9/jkwwxizD1gA/DA40Tx1pr/vZyTKrRW57ER73scfl9mYZcJJo/tjjJkBjAMuadFELe+UfTbGtAF+D9werEAtrDGvcRTOcMwknP/IlhljRlhrfS2craU0ps//Abxurf2tMeYC4A1/n+tbPp5nWrR+heoe+z6gT4N2b/7937OvljHGROH8C3eqf31CXWP6jDHmMuCnwHXW2qNBytZSTtfnTsAIIN0YswdnLHJeGH+A2tj39fvW2hpr7W5gO06hD1eN6fN3gP8DsNZ+DrTHuVBWJGvU73tThWphXw0MMsb0N8a0w/lwdN5xy8wDvuX/+mZgsfV/KhGmTttn/7DEKzhFPdzHXuE0fbbWlllrE621KdbaFJzPFa6z1q7xJm6zNeZ9/R7Oh+QYYxJxhmZ2BTWluxrT5xzgUgBjzFCcwl4U1JTBNw/4T//RMecDZdbaA66t3etPj0/xqfLVwA6cT9R/6v/eUzi/2OC8+P8EsoAvgQFeZw5CnxcBBcAG/zTP68wt3efjlk0njI+KaeRrbIDfAVuBzcB0rzMHoc/DgBU4R8xsAC73OrMLfX4LOADU4Oydfwf4HvC9Bq/zS/6fyWa339e6pICISIQJ1aEYERFpIhV2EZEIo8IuIhJhVNhFRCKMCruISIRRYRcRiTAq7CIiEeb/A53FYTRf3DGCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116ff92b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 4))\n",
    "plt.plot(np.arange(0, 1, 0.01), logit(np.arange(0, 1, 0.01)))\n",
    "plt.grid()\n",
    "plt.hlines([0], 0, 1, linestyles='dotted', colors='r')\n",
    "plt.vlines([0.5], -5, 5, linestyles='dotted', colors='r')\n",
    "_ = plt.title(\"The Logit Function\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Derivations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train the model, we need to determine the function to be optimized and its gradient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expression for a Single Data Point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For logistic regression, we use the likelihood function:\n",
    "\n",
    "$\\mathcal{L}(\\beta\\:|\\:x) = P(X = x; \\beta)$\n",
    "\n",
    "i.e. the probability of $X$ parameterized by $\\beta$.\n",
    "\n",
    "As we know from above, this probability is given by the logistic function:\n",
    "\n",
    "$P(Y = 1\\:|\\:X = x; \\beta) = \\sigma(\\beta^TX)$  \n",
    "\n",
    "By law of total probability:  \n",
    "$P(Y = 0\\:|\\:X = x; \\beta) = 1 - \\sigma(\\beta^TX)$\n",
    "\n",
    "These two equations can be combined in a convenient way to form a more generalized equation:\n",
    "\n",
    "$$\\mathcal{L}(\\beta) = P(Y = y\\:|\\:X = x; \\beta) = \\sigma(\\beta^TX)^y \\cdot (1 - \\sigma(\\beta^TX))^{(1 - y)}$$\n",
    "\n",
    "Let us break this down:\n",
    "\n",
    "When $y = 1$, the second term's exponent is $0$, and thus the expression evaluates to $\\sigma(\\beta^TX)$  \n",
    "When $y = 0$, the first term's exponent is $0$, and thus the expression evaluates to $1 - \\sigma(\\beta^TX)$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, it is easier to work with the log of the likelihood:\n",
    "\n",
    "$$\\mathcal{LL}(\\beta) = P(Y = y\\:|\\:X = x; \\beta) = y \\log{\\sigma(\\beta^TX)} + (1 - y)\\log{(1 - \\sigma(\\beta^TX))}$$\n",
    "\n",
    "We are able to do this because the point at which the derivative of the likelihood is zero is the same for the log-likelihood as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the entire dataset, the likelihood is given by:\n",
    "\n",
    "$$\\mathcal{L}(\\beta) = \\prod_{i = 1}^{n} P(Y = y^{(i)} | X = x^{(i)} ; \\beta) = \\prod_{i = 1}^{n} \\sigma(\\beta^Tx^{(i)})^{y^{(i)}} \\cdot (1 - \\sigma(\\beta^Tx^{(i)}))^{(1 - y^{(i)})}$$\n",
    "\n",
    "and the log-likelihood is given by:\n",
    "\n",
    "$$\\mathcal{LL}(\\beta) = \\sum_{i = 1}^{n} y^{(i)} \\log{\\sigma(\\beta^Tx^{(i)})} + \\sum_{i = 1}^{n} (1 - y^{(i)})\\cdot\\log{(1 - \\sigma(\\beta^Tx^{(i)}))}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating the Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to determine the gradient w.r.t. $\\beta_j$, the jth entry in the weight vector $\\beta$. Before we start the derivative for the log-likelihood, it is useful to derive it for the sigmoid function, and its functions:\n",
    "\n",
    "1. Derivative of $\\sigma(z)$\n",
    "$$\\frac{d}{dz} \\sigma(z) = \\frac{d}{dz} \\frac{1}{1 + e^{-z}} = \\frac{e^{-z}}{(1 + e^{-z})^2} = \\frac{1}{1 + e^{-z}} \\cdot \\frac{-e^{-z}}{1 + e^{-z}} = \\frac{1}{1 + e^{-z}} \\cdot \\frac{1 + e^{-z} - 1}{1 + e^{-z}} = \\sigma(z) \\cdot (1 - \\sigma(z))$$  \n",
    "\n",
    "2. Derivative of $\\sigma(\\beta^TX)$\n",
    "$$\\frac{\\partial{}}{\\partial{\\beta_j}}\\sigma(\\beta^TX) = \\sigma(\\beta^TX) \\cdot (1 - \\sigma(\\beta^TX)) \\cdot x_j$$\n",
    "\n",
    "3. Derivative of $\\log(\\sigma(\\beta^TX))$\n",
    "$$\\frac{\\partial{}}{\\partial{\\beta_j}}\\log(\\sigma(\\beta^TX)) = \\frac{1}{\\sigma(\\beta^TX)} \\sigma(\\beta^TX) \\cdot (1 - \\sigma(\\beta^TX)) \\cdot x_j = (1 - \\sigma(\\beta^TX)) \\cdot x_j$$\n",
    "\n",
    "4. Derivative of $\\log(1 - \\sigma(\\beta^TX))$\n",
    "$$\\frac{\\partial{}}{\\partial{\\beta_j}}\\log(1 - \\sigma(\\beta^TX)) = \\frac{1}{1 - \\sigma(\\beta^TX)} \\cdot -\\sigma(\\beta^TX) \\cdot (1 - \\sigma(\\beta^TX)) \\cdot x_j = -\\sigma(\\beta^TX) \\cdot x_j$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can derive the expression for the gradient w.r.t the log-likelihood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial{\\mathcal{LL}(\\beta)}}{\\partial{\\beta_j}} = \\frac{\\partial{}}{\\partial{\\beta_j}}y \\log{\\sigma(\\beta^TX)} + \\frac{\\partial{}}{\\partial{\\beta_j}}(1 - y)\\log{(1 - \\sigma(\\beta^TX))}$$\n",
    "\n",
    "$$\\Rightarrow (y \\cdot (1 - \\sigma(\\beta^TX)) \\cdot x_j) + ((1 - y) (-\\sigma(\\beta^TX) \\cdot x_j))$$\n",
    "\n",
    "$$\\Rightarrow x_j \\left(y \\cdot (1 - \\sigma(\\beta^TX)) + (y - 1) \\cdot \\sigma(\\beta^TX)\\right)$$\n",
    "\n",
    "$$\\Rightarrow (y - \\sigma(\\beta^TX)) \\cdot x_j$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entire Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the derivative of the sum is the sum of the derivates, the gradient is simply given by the sum of the above expression over all data points:\n",
    "\n",
    "$$\\frac{\\partial{\\mathcal{LL}(\\beta)}}{\\partial{\\beta_j}} = \\sum_{i = 1}^{n} (y^{(i)} - \\sigma(\\beta^Tx^{(i)})) \\cdot x^{(i)}_j$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:53.593495Z",
     "start_time": "2019-04-12T05:09:52.725609Z"
    }
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.604451Z",
     "start_time": "2019-04-12T05:09:53.596292Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(54074, 16)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_json(\"book_1_001_558.jl\", lines=True)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.640712Z",
     "start_time": "2019-04-12T05:09:54.607125Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asin</th>\n",
       "      <th>author</th>\n",
       "      <th>avg_rating</th>\n",
       "      <th>characters</th>\n",
       "      <th>genres</th>\n",
       "      <th>language</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_ratings</th>\n",
       "      <th>num_reviews</th>\n",
       "      <th>original_publish_year</th>\n",
       "      <th>places</th>\n",
       "      <th>publish_date</th>\n",
       "      <th>rating_histogram</th>\n",
       "      <th>series</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Anne Rice</td>\n",
       "      <td>3.99</td>\n",
       "      <td>[Lestat de Lioncourt, Claudia, Daniel Molloy, ...</td>\n",
       "      <td>[Fiction, Vampires, Horror, Fantasy, Paranormal]</td>\n",
       "      <td>English</td>\n",
       "      <td>342.0</td>\n",
       "      <td>437912.0</td>\n",
       "      <td>9056.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>[New Orleans, Louisiana, Paris]</td>\n",
       "      <td>2004-08-31 00:00:00</td>\n",
       "      <td>{'5': 160553, '4': 152049, '3': 92608, '2': 24...</td>\n",
       "      <td>The Vampire Chronicles #1</td>\n",
       "      <td>Interview with the Vampire</td>\n",
       "      <td>https://www.goodreads.com/book/show/43763.Inte...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>John Irving</td>\n",
       "      <td>4.08</td>\n",
       "      <td>[T.S. Garp, Jenny Fields, Helen Garp, Roberta ...</td>\n",
       "      <td>[Classics, Literature, Fiction, Novels, Contem...</td>\n",
       "      <td>English</td>\n",
       "      <td>610.0</td>\n",
       "      <td>191424.0</td>\n",
       "      <td>5008.0</td>\n",
       "      <td>1978.0</td>\n",
       "      <td>[Vienna]</td>\n",
       "      <td>1999-04-01 00:00:00</td>\n",
       "      <td>{'5': 74265, '4': 72537, '3': 33271, '2': 8126...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>The World According to Garp</td>\n",
       "      <td>https://www.goodreads.com/book/show/7069.The_W...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  asin       author  avg_rating  \\\n",
       "0  NaN    Anne Rice        3.99   \n",
       "1  NaN  John Irving        4.08   \n",
       "\n",
       "                                          characters  \\\n",
       "0  [Lestat de Lioncourt, Claudia, Daniel Molloy, ...   \n",
       "1  [T.S. Garp, Jenny Fields, Helen Garp, Roberta ...   \n",
       "\n",
       "                                              genres language  num_pages  \\\n",
       "0   [Fiction, Vampires, Horror, Fantasy, Paranormal]  English      342.0   \n",
       "1  [Classics, Literature, Fiction, Novels, Contem...  English      610.0   \n",
       "\n",
       "   num_ratings  num_reviews  original_publish_year  \\\n",
       "0     437912.0       9056.0                 1976.0   \n",
       "1     191424.0       5008.0                 1978.0   \n",
       "\n",
       "                            places         publish_date  \\\n",
       "0  [New Orleans, Louisiana, Paris]  2004-08-31 00:00:00   \n",
       "1                         [Vienna]  1999-04-01 00:00:00   \n",
       "\n",
       "                                    rating_histogram  \\\n",
       "0  {'5': 160553, '4': 152049, '3': 92608, '2': 24...   \n",
       "1  {'5': 74265, '4': 72537, '3': 33271, '2': 8126...   \n",
       "\n",
       "                      series                        title  \\\n",
       "0  The Vampire Chronicles #1   Interview with the Vampire   \n",
       "1                        NaN  The World According to Garp   \n",
       "\n",
       "                                                 url  \n",
       "0  https://www.goodreads.com/book/show/43763.Inte...  \n",
       "1  https://www.goodreads.com/book/show/7069.The_W...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.866498Z",
     "start_time": "2019-04-12T05:09:54.643013Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.dropna(subset=['genres'])\n",
    "\n",
    "df['historical'] = df.genres.apply(lambda s: 'Historical' in s)\n",
    "\n",
    "features = ['avg_rating', 'num_pages', 'num_ratings', 'num_reviews', 'original_publish_year']\n",
    "df = df.dropna(subset=features)\n",
    "\n",
    "target = 'historical'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.875729Z",
     "start_time": "2019-04-12T05:09:54.870117Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32576, 17)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.887987Z",
     "start_time": "2019-04-12T05:09:54.878727Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    27129\n",
       "True      5447\n",
       "Name: historical, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.historical.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:54.906126Z",
     "start_time": "2019-04-12T05:09:54.891647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((24432, 5), (8144, 5), (24432,), (8144,))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = df[features], df[target]\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=143)\n",
    "\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.106379Z",
     "start_time": "2019-04-12T05:09:54.908412Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.113799Z",
     "start_time": "2019-04-12T05:09:55.108477Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'avg_rating': -2.7446295061981062e-06,\n",
       " 'num_pages': 0.0010406965374428457,\n",
       " 'num_ratings': -1.1842317541290849e-07,\n",
       " 'num_reviews': 4.491291204147243e-06,\n",
       " 'original_publish_year': -0.0009911150702044862}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(zip(features, model.coef_[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.121726Z",
     "start_time": "2019-04-12T05:09:55.117040Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model.coef_[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.130603Z",
     "start_time": "2019-04-12T05:09:55.123874Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8428290766208252"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation of coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weights learned by the model are interpretable in the way they affect the log-odds of the event. Recall the expression for the model:\n",
    "\n",
    "$$log\\left(\\frac{p}{1 - p}\\right) = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 \\ldots$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intercept"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that, if all feature are 0 (which may not make sense for features such as `original_publish_year`), the log odds of a book being historical are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.138170Z",
     "start_time": "2019-04-12T05:09:55.132776Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-3.81105018e-07])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i.e. the _odds_ of the book being Historical are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.145023Z",
     "start_time": "2019-04-12T05:09:55.140462Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999996188950551"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pow(np.e, model.intercept_[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which gives $p = 0.5$. Indeed, we can validate this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T05:09:55.152184Z",
     "start_time": "2019-04-12T05:09:55.147176Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.5000001, 0.4999999]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_proba([[0,0,0,0,0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interpretation for any arbitrary $\\beta_i$ is: **a unit increase in $x_i$, with all else remaining constant, will result in the log-odds of the event increasing by $\\beta_i$.**\n",
    "\n",
    "In other words, the _odds_ of the event will change by $e^{\\beta_i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our scenario, we see that `num_pages` and `original_publish_year` have the highest magnitudes, and so we can attempt to interpret them first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For `num_pages`, $\\beta_{\\text{num_pages}} ~= 0.0010406$. \n",
    "\n",
    "This means that all else remaining constant, if we increase the number of pages by 1, the log-odds of the book being historical _increase_ by $0.0010406$, i.e. the odds increase by $e^{0.0010406} = 1.001041$. \n",
    "\n",
    "This implies that larger books have a higher probability of being historical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_  _"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, for `original_publish_year`, $\\beta_{\\text{original_publish_year}} = -0.0009911$. \n",
    "\n",
    "This means that for each unit increase in the publish year, the log-odds of the book being historical _decrease_ by $0.0009911$, i.e. the odds decrease by $e^{0.0009911} = 1.00099159$. \n",
    "\n",
    "This implies that older books have a higher probability of being historical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: For models that have interaction variables, the interpretation of these coefficients changes, since we then need to account for a single feature in more than one coefficient."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}