pycon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on with PyData: How to Build a Minimal Recommendation Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Welcome!\n",
    "\n",
    "- About Unata\n",
    "    - What we do and what we use Python for (everything!)\n",
    "- Environment + data files check\n",
    "    - Instructions to set up a local environment and links to download handout and data files: [http://unatainc.github.io/pycon2015/](http://unatainc.github.io/pycon2015/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this tutorial\n",
    "\n",
    "### References\n",
    "Credit where credit is due. Please check the references at the bottom of this document.\n",
    "\n",
    "### Dataset\n",
    "MovieLens from GroupLens Research: [grouplens.org](http://www.grouplens.org/)\n",
    "\n",
    "The MovieLens 1M data set contains 1 million ratings collected from 6000 users\n",
    "on 4000 movies.\n",
    "\n",
    "### What this tutorial is\n",
    "\n",
    "The goal of this tutorial is to provide you with a hands-on overview of two of\n",
    "the main libraries from the scientific and data analysis communities. We're going to\n",
    "use:\n",
    "\n",
    "- numpy -- [numpy.org](http://www.numpy.org)\n",
    "- pandas -- [pandas.pydata.org](http://pandas.pydata.org/)\n",
    "\n",
    "### What this tutorial is not\n",
    "\n",
    "- An exhaustive overview of the recommendation literature\n",
    "- A set of recipes that will win you the next Netflix/Kaggle/etc challenge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Roadmap\n",
    "\n",
    "What exactly are we going to do? Here's a high-level overview:\n",
    "\n",
    "- quick introduction to the recommendation problem (theory)\n",
    "- learn about Pandas Series and DataFrames (practice) \n",
    "- known solutions & challenges to the recommendation problem (theory) \n",
    "- load data, setup evaluation functions, test dummy solution (practice) \n",
    "- minimal reco engine v1.0 (practice) \n",
    "- more formulas! (theory) \n",
    "- pandas aggregation & minimal reco engine v1.1 (practice) \n",
    "- challenge (practice) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Recommendation Problem\n",
    "\n",
    "Recommenders have been around since at least 1992. Today we see different flavours of recommenders, deployed across different verticals: \n",
    "\n",
    "- Amazon\n",
    "- Netflix\n",
    "- Facebook\n",
    "- Last.fm.\n",
    "\n",
    "What exactly do they do?\n",
    "\n",
    "### Definitions from the literature\n",
    "\n",
    "- *In a typical recommender system people provide recommendations as inputs, which\n",
    "the system then aggregates and directs to appropriate recipients.* -- Resnick\n",
    "and Varian, 1997\n",
    "\n",
    "- *Collaborative filtering simply means that people collaborate to help one\n",
    "another perform filtering by recording their reactions to documents they read.*\n",
    "-- Goldberg et al, 1992\n",
    "\n",
    "- *In its most common formulation, the recommendation problem is reduced to the\n",
    "problem of estimating ratings for the items that have not been seen by a\n",
    "user. Intuitively, this estimation is usually based on the ratings given by this\n",
    "user to other items and on some other information [...] Once we can estimate\n",
    "ratings for the yet unrated items, we can recommend to the user the item(s) with\n",
    "the highest estimated rating(s).* -- Adomavicius and Tuzhilin, 2005\n",
    "\n",
    "- *Driven by computer algorithms, recommenders help consumers\n",
    "by selecting products they will probably like and might buy\n",
    "based on their browsing, searches, purchases, and preferences.* -- Konstan and Riedl, 2012"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notation\n",
    "\n",
    "- $U$ is the set of users in our domain. Its size is $|U|$.\n",
    "- $I$ is the set of items in our domain. Its size is $|I|$.\n",
    "- $I(u)$ is the set of items that user $u$ has rated.\n",
    "- $-I(u)$ is the complement of $I(u)$ i.e., the set of items not yet seen by user $u$.\n",
    "- $U(i)$ is the set of users that have rated item $i$.\n",
    "- $-U(i)$ is the complement of $U(i)$.\n",
    "- $S(u,i)$ is a function that measures the utility of item $i$ for user $u$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goal of a recommendation system\n",
    "\n",
    "$\n",
    "i^{*} = argmax_{i \\in -I(u)} S(u,i), \\forall{u \\in U}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem statement\n",
    "\n",
    "The recommendation problem in its most basic form is quite simple to define:\n",
    "\n",
    "```\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| user_id, movie_id | m_1 | m_2 | m_3 | m_4 | m_5 |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_1               | ?   | ?   | 4   | ?   | 1   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_2               | 3   | ?   | ?   | 2   | 2   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_3               | 3   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_4               | ?   | 1   | 2   | 1   | 1   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_5               | ?   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_6               | 2   | ?   | 2   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_7               | ?   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_8               | 3   | 1   | 5   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_9               | ?   | ?   | ?   | ?   | 2   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "```\n",
    "\n",
    "*Given a partially filled matrix of ratings ($|U|x|I|$), estimate the missing values.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pandas: Python Data Analysis Library\n",
    "\n",
    "### What is it?\n",
    "\n",
    "*Python has long been great for data munging and preparation, but less so for\n",
    "data analysis and modeling. pandas helps fill this gap, enabling you to carry\n",
    "out your entire data analysis workflow in Python without having to switch to a\n",
    "more domain specific language like R.*\n",
    "\n",
    "The heart of pandas is the DataFrame object for data manipulation. It features:\n",
    "\n",
    "- a powerful index object\n",
    "- data alignment\n",
    "- handling of missing data\n",
    "- aggregation with groupby\n",
    "- data manipuation via reshape, pivot, slice, merge, join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# set some print options\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(threshold=5)\n",
    "np.set_printoptions(suppress=True)\n",
    "pd.set_option('precision', 3, 'notebook_repr_html', True, )\n",
    "\n",
    "# init random gen\n",
    "np.random.seed(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series: labelled arrays\n",
    "\n",
    "The pandas Series is kind of like an ndarray (used to actually be a subclass of it) that supports more meaninful indices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's look at some creation examples for Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     2.00\n",
       "1     1.00\n",
       "2     5.00\n",
       "3     0.97\n",
       "4     3.00\n",
       "5    10.00\n",
       "6     0.06\n",
       "7     8.00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser = pd.Series([2.0, 1.0, 5.0, 0.97, 3.0, 10.0, 0.06, 8.0])\n",
    "ser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A     2.00\n",
       "B     1.00\n",
       "C     5.00\n",
       "D     0.97\n",
       "E     3.00\n",
       "F    10.00\n",
       "G     0.06\n",
       "H     8.00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = np.array([2.0, 1.0, 5.0, 0.97, 3.0, 10.0, 0.0599, 8.0])\n",
    "labels = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']\n",
    "ser = pd.Series(data=values, index=labels)\n",
    "ser\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "age                                                1\n",
      "gender                                             F\n",
      "genres                                         Drama\n",
      "movie_id                                        1193\n",
      "occupation                                        10\n",
      "rating                                             5\n",
      "timestamp                                  978300760\n",
      "title         One Flew Over the Cuckoo's Nest (1975)\n",
      "user_id                                            1\n",
      "zip                                            48067\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "movie_rating = {\n",
    "    'age': 1,\n",
    "    'gender': 'F',\n",
    "    'genres': 'Drama',\n",
    "    'movie_id': 1193,\n",
    "    'occupation': 10,\n",
    "    'rating': 5,\n",
    "    'timestamp': 978300760,\n",
    "    'title': \"One Flew Over the Cuckoo's Nest (1975)\",\n",
    "    'user_id': 1,\n",
    "    'zip': '48067'\n",
    "    }\n",
    "ser = pd.Series(movie_rating)\n",
    "print ser\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'age', u'gender', u'genres', u'movie_id', u'occupation', u'rating', u'timestamp', u'title', u'user_id', u'zip'], dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 'F', 'Drama', ..., \"One Flew Over the Cuckoo's Nest (1975)\", 1,\n",
       "       '48067'], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Series indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.loc['gender']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender        F\n",
       "zip       48067\n",
       "dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.loc[['gender', 'zip']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, ..., False, False, False], dtype=bool)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "booleans = np.array([False, False, False, False, False, True, False, False, False, False])\n",
    "booleans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "rating    5\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.loc[booleans]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.iloc[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender        F\n",
       "genres    Drama\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.iloc[[1,2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.ix['gender']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.ix[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser['gender']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender        F\n",
       "genres    Drama\n",
       "dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser[[1,2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Operations between Series with different index objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A     6\n",
      "B   NaN\n",
      "C     9\n",
      "G   NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "ser_1 = pd.Series(data=[1,3,4], index=['A', 'B', 'C'])\n",
    "ser_2 = pd.Series(data=[5,5,5], index=['A', 'G', 'C'])\n",
    "print ser_1 + ser_2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatic upcasting when performing operations between Series with different dtypes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A     6\n",
       "B   NaN\n",
       "C     9\n",
       "G   NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser_1 = pd.Series(data=[1,3,4], index=['A', 'B', 'C'], dtype='int')\n",
    "ser_2 = pd.Series(data=[5,5,5], index=['A', 'G', 'C'], dtype='float')\n",
    "ser_1 + ser_2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame\n",
    "\n",
    "The DataFrame is the 2-dimensional version of a Series.\n",
    "\n",
    "#### Let's look at some creation examples for DataFrame\n",
    "\n",
    "You can think of it as a spreadsheet whose columns are Series objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45.00</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.00</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col_1  col_2\n",
       "0   0.12    0.9\n",
       "1   7.00    9.0\n",
       "2  45.00   34.0\n",
       "3  10.00   11.0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# build from a dict of equal-length lists or ndarrays\n",
    "pd.DataFrame({'col_1': [0.12, 7, 45, 10], 'col_2': [0.9, 9, 34, 11]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explicitly set the column names and index values as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45.00</td>\n",
       "      <td>34.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.00</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col_1  col_2 col_3\n",
       "0   0.12    0.9   NaN\n",
       "1   7.00    9.0   NaN\n",
       "2  45.00   34.0   NaN\n",
       "3  10.00   11.0   NaN"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data={'col_1': [0.12, 7, 45, 10], 'col_2': [0.9, 9, 34, 11]},\n",
    "             columns=['col_1', 'col_2', 'col_3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>obs1</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs2</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs3</th>\n",
       "      <td>45.00</td>\n",
       "      <td>34.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs4</th>\n",
       "      <td>10.00</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col_1  col_2 col_3\n",
       "obs1   0.12    0.9   NaN\n",
       "obs2   7.00    9.0   NaN\n",
       "obs3  45.00   34.0   NaN\n",
       "obs4  10.00   11.0   NaN"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data={'col_1': [0.12, 7, 45, 10], 'col_2': [0.9, 9, 34, 11]},\n",
    "             columns=['col_1', 'col_2', 'col_3'],\n",
    "             index=['obs1', 'obs2', 'obs3', 'obs4'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also think of it as a dictionary of Series objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rating_events</th>\n",
       "      <th>r_1</th>\n",
       "      <th>r_2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>Drama</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_id</th>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "rating_events        r_1        r_2\n",
       "rating_data                        \n",
       "gender                 F          F\n",
       "genres             Drama      Drama\n",
       "movie_id            1193       1193\n",
       "rating                 5          5\n",
       "timestamp      978300760  978300760\n",
       "user_id                1          1"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_rating = {\n",
    "    'gender': 'F',\n",
    "    'genres': 'Drama',\n",
    "    'movie_id': 1193,\n",
    "    'rating': 5,\n",
    "    'timestamp': 978300760,\n",
    "    'user_id': 1,\n",
    "    }\n",
    "ser_1 = pd.Series(movie_rating)\n",
    "ser_2 = pd.Series(movie_rating)\n",
    "df = pd.DataFrame({'r_1': ser_1, 'r_2': ser_2})\n",
    "df.columns.name = 'rating_events'\n",
    "df.index.name = 'rating_data'\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rating_data</th>\n",
       "      <th>gender</th>\n",
       "      <th>genres</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_events</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>r_1</th>\n",
       "      <td>F</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r_2</th>\n",
       "      <td>F</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "rating_data   gender genres movie_id rating  timestamp user_id\n",
       "rating_events                                                 \n",
       "r_1                F  Drama     1193      5  978300760       1\n",
       "r_2                F  Drama     1193      5  978300760       1"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.T\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'gender', u'genres', u'movie_id', u'rating', u'timestamp', u'user_id'], dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'r_1', u'r_2'], dtype='object')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['F', 'Drama', 1193, 5, 978300760, 1],\n",
       "       ['F', 'Drama', 1193, 5, 978300760, 1]], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding/Deleting entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r_1</th>\n",
       "      <th>r_2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_id</th>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   r_1        r_2\n",
       "rating_data                      \n",
       "gender               F          F\n",
       "movie_id          1193       1193\n",
       "rating               5          5\n",
       "timestamp    978300760  978300760\n",
       "user_id              1          1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'r_1': ser_1, 'r_2': ser_2})\n",
    "df.drop('genres', axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r_2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_data</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_id</th>\n",
       "      <td>1193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   r_2\n",
       "rating_data           \n",
       "gender               F\n",
       "genres           Drama\n",
       "movie_id          1193\n",
       "rating               5\n",
       "timestamp    978300760\n",
       "user_id              1"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop('r_1', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add a new column using dictionary notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r_1</th>\n",
       "      <th>r_2</th>\n",
       "      <th>r_3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>Drama</td>\n",
       "      <td>Drama</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_id</th>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   r_1        r_2        r_3\n",
       "rating_data                                 \n",
       "gender               F          F          F\n",
       "genres           Drama      Drama      Drama\n",
       "movie_id          1193       1193       1193\n",
       "rating               5          5          5\n",
       "timestamp    978300760  978300760  978300760\n",
       "user_id              1          1          1"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# careful with the order here\n",
    "df['r_3'] = ['F', 'Drama', 1193, 5, 978300760, 1]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r_1</th>\n",
       "      <th>r_2</th>\n",
       "      <th>r_3</th>\n",
       "      <th>r_4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "      <td>F</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>Drama</td>\n",
       "      <td>Drama</td>\n",
       "      <td>Drama</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_id</th>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "      <td>1193</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "      <td>978300760</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   r_1        r_2        r_3  r_4\n",
       "rating_data                                      \n",
       "gender               F          F          F    M\n",
       "genres           Drama      Drama      Drama  NaN\n",
       "movie_id          1193       1193       1193  NaN\n",
       "rating               5          5          5  NaN\n",
       "timestamp    978300760  978300760  978300760  NaN\n",
       "user_id              1          1          1  NaN"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['r_4'] = pd.Series({'gender': 'M'})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--> Go to \"Pandas questions: Series and DataFrames\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DataFrame indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can index into a column using it's label, or with dot notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>obs1</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs2</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs3</th>\n",
       "      <td>45.00</td>\n",
       "      <td>34.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs4</th>\n",
       "      <td>10.00</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col_1  col_2 col_3\n",
       "obs1   0.12    0.9   NaN\n",
       "obs2   7.00    9.0   NaN\n",
       "obs3  45.00   34.0   NaN\n",
       "obs4  10.00   11.0   NaN"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(data={'col_1': [0.12, 7, 45, 10], 'col_2': [0.9, 9, 34, 11]},\n",
    "                  columns=['col_1', 'col_2', 'col_3'],\n",
    "                  index=['obs1', 'obs2', 'obs3', 'obs4'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "obs1     0.12\n",
       "obs2     7.00\n",
       "obs3    45.00\n",
       "obs4    10.00\n",
       "Name: col_1, dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['col_1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "obs1     0.12\n",
       "obs2     7.00\n",
       "obs3    45.00\n",
       "obs4    10.00\n",
       "Name: col_1, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.col_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use multiple columns to select a subset of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>obs1</th>\n",
       "      <td>0.9</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>7.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs3</th>\n",
       "      <td>34.0</td>\n",
       "      <td>45.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs4</th>\n",
       "      <td>11.0</td>\n",
       "      <td>10.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col_2  col_1\n",
       "obs1    0.9   0.12\n",
       "obs2    9.0   7.00\n",
       "obs3   34.0  45.00\n",
       "obs4   11.0  10.00"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['col_2', 'col_1']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrame has similar .loc and .iloc methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['obs1', 'col_1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The .ix method gives you the most flexibility to index into certain rows, or\n",
    "even rows and columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "col_1     45\n",
       "col_2     34\n",
       "col_3    NaN\n",
       "Name: obs3, dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix['obs3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "col_1    0.12\n",
       "col_2     0.9\n",
       "col_3     NaN\n",
       "Name: obs1, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>obs1</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs2</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col_1  col_2 col_3\n",
       "obs1   0.12    0.9   NaN\n",
       "obs2   7.00    9.0   NaN"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "obs1    0.9\n",
       "obs2    9.0\n",
       "Name: col_2, dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[:2, 'col_2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>obs1</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obs2</th>\n",
       "      <td>7.00</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col_1  col_2\n",
       "obs1   0.12    0.9\n",
       "obs2   7.00    9.0"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[:2, ['col_1', 'col_2']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--> Go to \"Pandas questions: Indexing\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MovieLens dataset: loading and first look\n",
    "\n",
    "Loading of the MovieLens dataset is based on the intro chapter of 'Python\n",
    "for Data Analysis\".\n",
    "\n",
    "The MovieLens data is spread across three files. We'll load each file using the `pd.read_table` function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/virtualenv/pycon2015/lib/python2.7/site-packages/pandas/io/parsers.py:644: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'.\n",
      "  ParserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp\n",
       "0        1      1193       5  978300760\n",
       "1        1       661       3  978302109\n",
       "2        1       914       3  978301968\n",
       "3        1      3408       4  978300275\n",
       "4        1      2355       5  978824291"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users = pd.read_table('data/ml-1m/users.dat',\n",
    "                      sep='::', header=None, \n",
    "                      names=['user_id', 'gender', 'age', 'occupation', 'zip'])\n",
    "\n",
    "ratings = pd.read_table('data/ml-1m/ratings.dat',\n",
    "                        sep='::', header=None, \n",
    "                        names=['user_id', 'movie_id', 'rating', 'timestamp'])\n",
    "\n",
    "movies = pd.read_table('data/ml-1m/movies.dat',\n",
    "                       sep='::', header=None, \n",
    "                       names=['movie_id', 'title', 'genres'])\n",
    "\n",
    "# show how one of them looks\n",
    "ratings.head(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `pd.merge` we get it all into  one big DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978298413</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978220179</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>12</td>\n",
       "      <td>32793</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978199279</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>22903</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978158471</td>\n",
       "      <td>M</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>95350</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp gender  age  occupation    zip  \\\n",
       "0        1      1193       5  978300760      F    1          10  48067   \n",
       "1        2      1193       5  978298413      M   56          16  70072   \n",
       "2       12      1193       4  978220179      M   25          12  32793   \n",
       "3       15      1193       4  978199279      M   25           7  22903   \n",
       "4       17      1193       5  978158471      M   50           1  95350   \n",
       "\n",
       "                                    title genres  \n",
       "0  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "1  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "2  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "3  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "4  One Flew Over the Cuckoo's Nest (1975)  Drama  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens = pd.merge(pd.merge(ratings, users), movies)\n",
    "movielens.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge prep: evaluation mechanism\n",
    "\n",
    "Before we start building our minimal reco engine we need a basic mechanism to evaluate the performance of our engine. For that we will:\n",
    "\n",
    "- split the data into train and test sets\n",
    "- introduce a performance criterion\n",
    "- write an `evaluate` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: split ratings into train and test sets\n",
    "\n",
    "This subsection will generate training and testing sets for evaluation. You do\n",
    "not need to understand every single line of code, just the general gist:\n",
    "\n",
    "- take a smaller sample from the full 1M dataset for speed reasons;\n",
    "- make sure that we have at least 2 ratings per user in that subset;\n",
    "- split the result into training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 10)\n",
      "3698\n",
      "2275\n"
     ]
    }
   ],
   "source": [
    "# let's work with a smaller subset for speed reasons\n",
    "movielens = movielens.ix[np.random.choice(movielens.index, size=10000, replace=False)]\n",
    "print movielens.shape\n",
    "print movielens.user_id.nunique()\n",
    "print movielens.movie_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8442, 10)\n"
     ]
    }
   ],
   "source": [
    "user_ids_larger_1 = pd.value_counts(movielens.user_id, sort=False) > 1\n",
    "user_ids_larger_1 = user_ids_larger_1[user_ids_larger_1].index\n",
    "\n",
    "movielens = movielens.select(lambda l: movielens.loc[l, 'user_id'] in user_ids_larger_1)\n",
    "print movielens.shape\n",
    "assert np.all(movielens.user_id.value_counts() > 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now generate train and test subsets by marking 20% of each users's ratings, using groupby and apply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8442, 11)\n",
      "(5801, 11)\n",
      "(2641, 11)\n"
     ]
    }
   ],
   "source": [
    "def assign_to_set(df):\n",
    "    sampled_ids = np.random.choice(df.index,\n",
    "                                   size=np.int64(np.ceil(df.index.size * 0.2)),\n",
    "                                   replace=False)\n",
    "    df.ix[sampled_ids, 'for_testing'] = True\n",
    "    return df\n",
    "\n",
    "movielens['for_testing'] = False\n",
    "grouped = movielens.groupby('user_id', group_keys=False).apply(assign_to_set)\n",
    "movielens_train = movielens[grouped.for_testing == False]\n",
    "movielens_test = movielens[grouped.for_testing == True]\n",
    "print movielens.shape\n",
    "print movielens_train.shape\n",
    "print movielens_test.shape\n",
    "assert len(movielens_train.index & movielens_test.index) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store these two sets in text files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "movielens_train.to_csv('data/my_generated_movielens_train.csv')\n",
    "movielens_test.to_csv('data/my_generated_movielens_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: performance criterion\n",
    "\n",
    "Performance evaluation of recommendation systems is an entire topic all in\n",
    "itself. Some of the options include:\n",
    "\n",
    "- RMSE: $\\sqrt{\\frac{\\sum(\\hat y - y)^2}{n}}$\n",
    "- Precision / Recall / F-scores\n",
    "- ROC curves\n",
    "- Cost curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compute_rmse(y_pred, y_true):\n",
    "    \"\"\" Compute Root Mean Squared Error. \"\"\"\n",
    "    \n",
    "    return np.sqrt(np.mean(np.power(y_pred - y_true, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: the 'evaluate' method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def evaluate(estimate_f):\n",
    "    \"\"\" RMSE-based predictive performance evaluation with pandas. \"\"\"\n",
    "    \n",
    "    ids_to_estimate = zip(movielens_test.user_id, movielens_test.movie_id)\n",
    "    estimated = np.array([estimate_f(u,i) for (u,i) in ids_to_estimate])\n",
    "    real = movielens_test.rating.values\n",
    "    return compute_rmse(estimated, real)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_estimate_function(user_id, movie_id):\n",
    "    return 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for my estimate function: 1.25716424791\n"
     ]
    }
   ],
   "source": [
    "print 'RMSE for my estimate function: %s' % evaluate(my_estimate_function)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--> Go to \"Mini Challenge prep: data loading & evaluation functions\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Well-known Solutions to the Recommendation Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content-based filtering\n",
    "\n",
    "*Recommend based on the user's rating history.* \n",
    "\n",
    "Generic expression (notice how this is kind of a 'row-based' approach):\n",
    "\n",
    "$$ \n",
    "\\newcommand{\\aggr}{\\mathop{\\rm aggr}\\nolimits}\n",
    "r_{u,i} = \\aggr_{i' \\in I(u)} [r_{u,i'}]\n",
    "$$\n",
    "\n",
    "\n",
    "A simple example using the mean as an aggregation function:\n",
    "\n",
    "$$ \n",
    "r_{u,i} = \\bar r_u = \\frac{\\sum_{i' \\in I(u)} r_{u,i'}}{|I(u)|} \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative filtering\n",
    "\n",
    "*Recommend based on other user's rating histories.* \n",
    "\n",
    "Generic expression (notice how this is kind of a 'col-based' approach):\n",
    "\n",
    "$$ \n",
    "\\newcommand{\\aggr}{\\mathop{\\rm aggr}\\nolimits}\n",
    "r_{u,i} = \\aggr_{u' \\in U(i)} [r_{u',i}] \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple example using the mean as an aggregation function:\n",
    "\n",
    "$$ \n",
    "r_{u,i} = \\bar r_i = \\frac{\\sum_{u' \\in U(i)} r_{u',i}}{|U(i)|} \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hybrid solutions\n",
    "\n",
    "The literature has lots of examples of systems that try to combine the strengths\n",
    "of the two main approaches. This can be done in a number of ways:\n",
    "\n",
    "- Combine the predictions of a content-based system and a collaborative system.\n",
    "- Incorporate content-based techniques into a collaborative approach.\n",
    "- Incorporarte collaborative techniques into a content-based approach.\n",
    "- Unifying model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenges\n",
    "\n",
    "#### Availability of item metadata\n",
    "\n",
    "Content-based techniques are limited by the amount of metadata that is available\n",
    "to describe an item. There are domains in which feature extraction methods are\n",
    "expensive or time consuming, e.g., processing multimedia data such as graphics,\n",
    "audio/video streams. In the context of grocery items for example, it's often the\n",
    "case that item information is only partial or completely missing. Examples\n",
    "include:\n",
    "\n",
    "- Ingredients\n",
    "- Nutrition facts\n",
    "- Brand\n",
    "- Description\n",
    "- County of origin\n",
    "\n",
    "#### New user problem\n",
    "\n",
    "A user has to have rated a sufficient number of items before a recommender\n",
    "system can have a good idea of what their preferences are. In a content-based\n",
    "system, the aggregation function needs ratings to aggregate.\n",
    "\n",
    "#### New item problem\n",
    "\n",
    "Collaborative filters rely on an item being rated by many users to compute\n",
    "aggregates of those ratings. Think of this as the exact counterpart of the new\n",
    "user problem for content-based systems.\n",
    "\n",
    "#### Data sparsity\n",
    "\n",
    "When looking at the more general versions of content-based and collaborative\n",
    "systems, the success of the recommender system depends on the availability of a\n",
    "critical mass of user/item iteractions. We get a first glance at the data\n",
    "sparsity problem by quantifying the ratio of existing ratings vs $|U|x|I|$. A\n",
    "highly sparse matrix of interactions makes it difficult to compute similarities\n",
    "between users and items. As an example, for a user whose tastes are unusual\n",
    "compared to the rest of the population, there will not be any other users who\n",
    "are particularly similar, leading to poor recommendations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.0: simple mean ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content-based filtering using mean ratings\n",
    "\n",
    "With this table-like representation of the ratings data, a basic content-based\n",
    "filter becomes a one-liner function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for estimate1: 1.26844670375\n"
     ]
    }
   ],
   "source": [
    "def content_mean(user_id, movie_id):\n",
    "    \"\"\" Simple content-filtering based on mean ratings. \"\"\"\n",
    "    \n",
    "    user_condition = movielens_train.user_id == user_id\n",
    "    return movielens_train.loc[user_condition, 'rating'].mean()\n",
    "\n",
    "print 'RMSE for estimate1: %s' % evaluate(content_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--> Go to \"Reco systems questions: Minimal reco engine v1.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More formulas!\n",
    "\n",
    "Here are some basic ways in which we can generalize the simple mean-based algorithms we discussed before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generalizations of the aggregation function for content-based filtering: incorporating similarities\n",
    "\n",
    "Possibly incorporating metadata about items, which makes the term 'content' make more sense now.\n",
    "\n",
    "$$ r_{u,i} = k \\sum_{i' \\in I(u)} sim(i, i') \\; r_{u,i'} $$\n",
    "\n",
    "$$ r_{u,i} = \\bar r_u + k \\sum_{i' \\in I(u)} sim(i, i') \\; (r_{u,i'} - \\bar r_u) $$\n",
    "\n",
    "Here $k$ is a normalizing factor,\n",
    "\n",
    "$$ k = \\frac{1}{\\sum_{i' \\in I(u)} |sim(i,i')|} $$\n",
    "\n",
    "and $\\bar r_u$ is the average rating of user u:\n",
    "\n",
    "$$ \\bar r_u = \\frac{\\sum_{i \\in I(u)} r_{u,i}}{|I(u)|} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generalizations of the aggregation function for collaborative filtering: incorporating similarities\n",
    "\n",
    "Possibly incorporating metadata about users.\n",
    "\n",
    "$$ r_{u,i} = k \\sum_{u' \\in U(i)} sim(u, u') \\; r_{u',i} $$\n",
    "\n",
    "$$ r_{u,i} = \\bar r_u + k \\sum_{u' \\in U(i)} sim(u, u') \\; (r_{u',i} - \\bar r_u) $$\n",
    "\n",
    "Here $k$ is a normalizing factor,\n",
    "\n",
    "$$ k = \\frac{1}{\\sum_{u' \\in U(i)} |sim(u,u')|} $$\n",
    "\n",
    "and $\\bar r_u$ is the average rating of user u:\n",
    "\n",
    "$$ \\bar r_u = \\frac{\\sum_{i \\in I(u)} r_{u,i}}{|I(u)|} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregation in pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groupby\n",
    "\n",
    "The idea of groupby is that of *split-apply-combine*:\n",
    "\n",
    "- split data in an object according to a given key;\n",
    "- apply a function to each subset;\n",
    "- combine results into a new object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender\n",
       "F    3.59\n",
       "M    3.51\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.groupby('gender')['rating'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender  age\n",
       "F       1      3.61\n",
       "        18     3.43\n",
       "        25     3.61\n",
       "        35     3.65\n",
       "        45     3.54\n",
       "        50     3.67\n",
       "        56     4.07\n",
       "M       1      3.38\n",
       "        18     3.44\n",
       "        25     3.43\n",
       "        35     3.61\n",
       "        45     3.55\n",
       "        50     3.76\n",
       "        56     3.70\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.groupby(['gender', 'age'])['rating'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pivoting\n",
    "\n",
    "Let's start with a simple pivoting example that does not involve any\n",
    "aggregation. We can extract a ratings matrix as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>movie_id</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>14</th>\n",
       "      <th>...</th>\n",
       "      <th>3930</th>\n",
       "      <th>3932</th>\n",
       "      <th>3935</th>\n",
       "      <th>3938</th>\n",
       "      <th>3943</th>\n",
       "      <th>3945</th>\n",
       "      <th>3948</th>\n",
       "      <th>3949</th>\n",
       "      <th>3950</th>\n",
       "      <th>3952</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 1934 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "movie_id  1     2     3     4     5     6     7     10    11    14    ...   \\\n",
       "user_id                                                               ...    \n",
       "8          NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "10         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "11         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "\n",
       "movie_id  3930  3932  3935  3938  3943  3945  3948  3949  3950  3952  \n",
       "user_id                                                               \n",
       "8          NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "10         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "11         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "\n",
       "[3 rows x 1934 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform the ratings frame into a ratings matrix\n",
    "ratings_mtx_df = movielens_train.pivot_table(values='rating',\n",
    "                                             index='user_id',\n",
    "                                             columns='movie_id')\n",
    "ratings_mtx_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>movie_id</th>\n",
       "      <th>1196</th>\n",
       "      <th>1197</th>\n",
       "      <th>1198</th>\n",
       "      <th>1199</th>\n",
       "      <th>1200</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "movie_id  1196  1197  1198  1199  1200\n",
       "user_id                               \n",
       "11         NaN   NaN   NaN   NaN   NaN\n",
       "13           5   NaN   NaN   NaN   NaN\n",
       "15         NaN   NaN   NaN   NaN   NaN"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# grab another subsquare of the ratings matrix to actually diplay some real entries!\n",
    "ratings_mtx_df.loc[11:16, 1196:1200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The more interesting case with `pivot_table` is as an interface to\n",
    "`groupby`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.61</td>\n",
       "      <td>3.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.43</td>\n",
       "      <td>3.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3.61</td>\n",
       "      <td>3.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3.65</td>\n",
       "      <td>3.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3.54</td>\n",
       "      <td>3.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>3.67</td>\n",
       "      <td>3.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>4.07</td>\n",
       "      <td>3.70</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gender     F     M\n",
       "age               \n",
       "1       3.61  3.38\n",
       "18      3.43  3.44\n",
       "25      3.61  3.43\n",
       "35      3.65  3.61\n",
       "45      3.54  3.55\n",
       "50      3.67  3.76\n",
       "56      4.07  3.70"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.pivot_table(values='rating', index='age', columns='gender', aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass in a list of functions, such as `[np.mean, np.std]`, to compute mean ratings and a measure of disagreement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">mean</th>\n",
       "      <th colspan=\"2\" halign=\"left\">std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.61</td>\n",
       "      <td>3.38</td>\n",
       "      <td>1.28</td>\n",
       "      <td>1.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.43</td>\n",
       "      <td>3.44</td>\n",
       "      <td>1.20</td>\n",
       "      <td>1.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3.61</td>\n",
       "      <td>3.43</td>\n",
       "      <td>1.08</td>\n",
       "      <td>1.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3.65</td>\n",
       "      <td>3.61</td>\n",
       "      <td>1.02</td>\n",
       "      <td>1.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3.54</td>\n",
       "      <td>3.55</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>3.67</td>\n",
       "      <td>3.76</td>\n",
       "      <td>1.14</td>\n",
       "      <td>1.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>4.07</td>\n",
       "      <td>3.70</td>\n",
       "      <td>0.96</td>\n",
       "      <td>1.08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mean         std      \n",
       "gender     F     M     F     M\n",
       "age                           \n",
       "1       3.61  3.38  1.28  1.30\n",
       "18      3.43  3.44  1.20  1.15\n",
       "25      3.61  3.43  1.08  1.14\n",
       "35      3.65  3.61  1.02  1.04\n",
       "45      3.54  3.55  1.00  1.06\n",
       "50      3.67  3.76  1.14  1.04\n",
       "56      4.07  3.70  0.96  1.08"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.pivot_table(values='rating', index='age', columns='gender', aggfunc=[np.mean, np.std])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.1: implicit sim functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to need a user index from the users portion of the dataset. This will allow us to retrieve information given a specific user_id in a more convenient way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        gender  age  occupation    zip\n",
       "user_id                               \n",
       "1            F    1          10  48067\n",
       "2            M   56          16  70072\n",
       "3            M   25          15  55117\n",
       "4            M   45           7  02460\n",
       "5            M   25          20  55455"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_info = users.set_index('user_id')\n",
    "user_info.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this in hand, we can now ask what the gender of a particular user_id is like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'M'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_id = 3\n",
    "user_info.loc[user_id, 'gender']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative-based filtering using implicit sim functions\n",
    "\n",
    "Using the pandas aggregation framework we will build a collaborative filter that estimates ratings using an implicit `sim(u,u')` function to compare different users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for collab_gender: 1.1953061054\n"
     ]
    }
   ],
   "source": [
    "def collab_gender(user_id, movie_id):\n",
    "    \"\"\" Collaborative filtering using an implicit sim(u,u') based on gender. \"\"\"\n",
    "    \n",
    "    user_condition = movielens_train.user_id != user_id\n",
    "    movie_condition = movielens_train.movie_id == movie_id\n",
    "    ratings_by_others = movielens_train.loc[user_condition & movie_condition]\n",
    "    if ratings_by_others.empty: \n",
    "        return 3.0\n",
    "    \n",
    "    means_by_gender = ratings_by_others.pivot_table('rating', index='movie_id', columns='gender')\n",
    "    user_gender = user_info.ix[user_id, 'gender']\n",
    "    if user_gender in means_by_gender.columns: \n",
    "        return means_by_gender.ix[movie_id, user_gender]\n",
    "    else:\n",
    "        return means_by_gender.ix[movie_id].mean()\n",
    "\n",
    "print 'RMSE for collab_gender: %s' % evaluate(collab_gender)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point it seems worthwhile to write a `learn` function to pre-compute whatever datastructures we need at estimation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabGenderReco: 1.1953061054\n"
     ]
    }
   ],
   "source": [
    "class CollabGenderReco:\n",
    "    \"\"\" Collaborative filtering using an implicit sim(u,u'). \"\"\"\n",
    "\n",
    "    def learn(self):\n",
    "        \"\"\" Prepare datastructures for estimation. \"\"\"\n",
    "        \n",
    "        self.means_by_gender = movielens_train.pivot_table('rating', index='movie_id', columns='gender')\n",
    "\n",
    "    def estimate(self, user_id, movie_id):\n",
    "        \"\"\" Mean ratings by other users of the same gender. \"\"\"\n",
    "        \n",
    "        if movie_id not in self.means_by_gender.index: \n",
    "            return 3.0\n",
    "        \n",
    "        user_gender = user_info.ix[user_id, 'gender']\n",
    "        if ~np.isnan(self.means_by_gender.ix[movie_id, user_gender]):\n",
    "            return self.means_by_gender.ix[movie_id, user_gender]\n",
    "        else:\n",
    "            return self.means_by_gender.ix[movie_id].mean()\n",
    "\n",
    "reco = CollabGenderReco()\n",
    "reco.learn()\n",
    "print 'RMSE for CollabGenderReco: %s' % evaluate(reco.estimate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Break!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini-Challenge!\n",
    "\n",
    "- Not a real challenge\n",
    "- Focus on understanding the different versions of our minimal reco\n",
    "- Try to mix and match some of the ideas presented to come up with a minimal reco of your own\n",
    "- Evaluate it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mini-Challenge: first round\n",
    "Implement an `estimate` function of your own using other similarity notions, eg.:\n",
    "\n",
    "- collaborative filter based on age similarities\n",
    "- collaborative filter based on zip code similarities\n",
    "- collaborative filter based on occupation similarities\n",
    "- content filter based on movie genre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.2: custom similarity functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A few similarity functions\n",
    "\n",
    "These were all written to operate on two pandas Series, each one representing the rating history of two different users. You can also apply them to any two feature vectors that describe users or items. In all cases, the higher the return value, the more similar two Series are. You might need to add checks for edge cases, such as divisions by zero, etc.\n",
    "\n",
    "- Euclidean 'similarity'\n",
    "\n",
    "$$ sim(x,y) = \\frac{1}{1 + \\sqrt{\\sum (x - y)^2}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def euclidean(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their euclidean 'similarity'.\"\"\"\n",
    "    diff = s1 - s2\n",
    "    return 1 / (1 + np.sqrt(np.sum(diff ** 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Cosine similarity\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x . y)}{\\sqrt{(x . x) (y . y)}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cosine(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their cosine similarity.\"\"\"\n",
    "    return np.sum(s1 * s2) / np.sqrt(np.sum(s1 ** 2) * np.sum(s2 ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pearson correlation\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x - \\bar x).(y - \\bar y)}{\\sqrt{(x - \\bar x).(x - \\bar x) * (y - \\bar y)(y - \\bar y)}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def pearson(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return a pearson correlation.\"\"\"\n",
    "    s1_c = s1 - s1.mean()\n",
    "    s2_c = s2 - s2.mean()\n",
    "    return np.sum(s1_c * s2_c) / np.sqrt(np.sum(s1_c ** 2) * np.sum(s2_c ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Jaccard similarity\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x . y)}{(x . x) + (y . y) - (x . y)} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def jaccard(s1, s2):\n",
    "    dotp = np.sum(s1 * s2)\n",
    "    return dotp / (np.sum(s1 ** 2) + np.sum(s2 ** 2) - dotp)\n",
    "\n",
    "def binjaccard(s1, s2):\n",
    "    dotp = (s1.index & s2.index).size\n",
    "    return dotp / (s1.sum() + s2.sum() - dotp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative-based filtering using custom sim functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabPearsonReco: 1.08629365955\n"
     ]
    }
   ],
   "source": [
    "class CollabPearsonReco:\n",
    "    \"\"\" Collaborative filtering using a custom sim(u,u'). \"\"\"\n",
    "\n",
    "    def learn(self):\n",
    "        \"\"\" Prepare datastructures for estimation. \"\"\"\n",
    "        \n",
    "        self.all_user_profiles = movielens.pivot_table('rating', index='movie_id', columns='user_id')\n",
    "\n",
    "    def estimate(self, user_id, movie_id):\n",
    "        \"\"\" Ratings weighted by correlation similarity. \"\"\"\n",
    "        \n",
    "        user_condition = movielens_train.user_id != user_id\n",
    "        movie_condition = movielens_train.movie_id == movie_id\n",
    "        ratings_by_others = movielens_train.loc[user_condition & movie_condition]\n",
    "        if ratings_by_others.empty: \n",
    "            return 3.0\n",
    "        \n",
    "        ratings_by_others.set_index('user_id', inplace=True)\n",
    "        their_ids = ratings_by_others.index\n",
    "        their_ratings = ratings_by_others.rating\n",
    "        their_profiles = self.all_user_profiles[their_ids]\n",
    "        user_profile = self.all_user_profiles[user_id]\n",
    "        sims = their_profiles.apply(lambda profile: pearson(profile, user_profile), axis=0)\n",
    "        ratings_sims = pd.DataFrame({'sim': sims, 'rating': their_ratings})\n",
    "        ratings_sims = ratings_sims[ratings_sims.sim > 0]\n",
    "        if ratings_sims.empty:\n",
    "            return their_ratings.mean()\n",
    "        else:\n",
    "            return np.average(ratings_sims.rating, weights=ratings_sims.sim)\n",
    "        \n",
    "reco = CollabPearsonReco()\n",
    "reco.learn()\n",
    "print 'RMSE for CollabPearsonReco: %s' % evaluate(reco.estimate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mini-Challenge: second round\n",
    "Implement an `estimate` function of your own using other custom similarity notions, eg.:\n",
    "\n",
    "- euclidean\n",
    "- cosine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References and further reading\n",
    "\n",
    "- Goldberg, D., D. Nichols, B. M. Oki, and D. Terry. “Using Collaborative Filtering to Weave an Information Tapestry.” Communications of the ACM 35, no. 12 (1992): 61–70.\n",
    "- Resnick, Paul, and Hal R. Varian. “Recommender Systems.” Commun. ACM 40, no. 3 (March 1997): 56–58. doi:10.1145/245108.245121.\n",
    "- Adomavicius, Gediminas, and Alexander Tuzhilin. “Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions.” IEEE Transactions on Knowledge and Data Engineering 17, no. 6 (2005): 734–749. doi:http://doi.ieeecomputersociety.org/10.1109/TKDE.2005.99.\n",
    "- Adomavicius, Gediminas, Ramesh Sankaranarayanan, Shahana Sen, and Alexander Tuzhilin. “Incorporating Contextual Information in Recommender Systems Using a Multidimensional Approach.” ACM Trans. Inf. Syst. 23, no. 1 (2005): 103–145. doi:10.1145/1055709.1055714.\n",
    "- Koren, Y., R. Bell, and C. Volinsky. “Matrix Factorization Techniques for Recommender Systems.” Computer 42, no. 8 (2009): 30–37.\n",
    "- William Wesley McKinney. Python for Data Analysis. O’Reilly, 2012.\n",
    "- Toby Segaran. Programming Collective Intelligence. O’Reilly, 2007.\n",
    "- Zhou, Tao, Zoltan Kuscsik, Jian-Guo Liu, Matus Medo, Joseph R Wakeling, and Yi-Cheng Zhang. “Solving the Apparent Diversity-accuracy Dilemma of Recommender Systems.” arXiv:0808.2670 (August 19, 2008). doi:10.1073/pnas.1000488107.\n",
    "- Shani, G., D. Heckerman, and R. I Brafman. “An MDP-based Recommender System.” Journal of Machine Learning Research 6, no. 2 (2006): 1265.\n",
    "- Joseph A. Konstan, John Riedl. \"Deconstructing Recommender Systems.\" IEEE Spectrum, October 2012."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}