Recommender_System_CI&T_Desktop_7_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, I read the following kernel:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.kaggle.com/gspmoreira/recommender-systems-in-python-101#Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main RecSys techniques are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collaborative Filtering: This method makes automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on a set of items, A is more likely to have B's opinion for a given item than that of a randomly chosen person."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Content-Based Filtering: This method uses only information about the description and attributes of the items users has previously consumed to model user's preferences. In other words, these algorithms try to recommend items that are similar to those that a user liked in the past (or is examining in the present). In particular, various candidate items are compared with items previously rated by the user and the best-matching items are recommended."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import packages and load data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy\n",
    "import pandas as pd\n",
    "import math\n",
    "import random\n",
    "import sklearn\n",
    "from nltk.corpus import stopwords\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from scipy.sparse.linalg import svds\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of simplicity, we only consider here the \"CONTENT SHARED\" event type, assuming (naively) that all articles were available during the whole one year period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>eventType</th>\n",
       "      <th>contentId</th>\n",
       "      <th>authorPersonId</th>\n",
       "      <th>authorSessionId</th>\n",
       "      <th>authorUserAgent</th>\n",
       "      <th>authorRegion</th>\n",
       "      <th>authorCountry</th>\n",
       "      <th>contentType</th>\n",
       "      <th>url</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1459193988</td>\n",
       "      <td>CONTENT SHARED</td>\n",
       "      <td>-4110354420726924665</td>\n",
       "      <td>4340306774493623681</td>\n",
       "      <td>8940341205206233829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML</td>\n",
       "      <td>http://www.nytimes.com/2016/03/28/business/dea...</td>\n",
       "      <td>Ethereum, a Virtual Currency, Enables Transact...</td>\n",
       "      <td>All of this work is still very early. The firs...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1459194146</td>\n",
       "      <td>CONTENT SHARED</td>\n",
       "      <td>-7292285110016212249</td>\n",
       "      <td>4340306774493623681</td>\n",
       "      <td>8940341205206233829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML</td>\n",
       "      <td>http://cointelegraph.com/news/bitcoin-future-w...</td>\n",
       "      <td>Bitcoin Future: When GBPcoin of Branson Wins O...</td>\n",
       "      <td>The alarm clock wakes me at 8:00 with stream o...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1459194474</td>\n",
       "      <td>CONTENT SHARED</td>\n",
       "      <td>-6151852268067518688</td>\n",
       "      <td>3891637997717104548</td>\n",
       "      <td>-1457532940883382585</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML</td>\n",
       "      <td>https://cloudplatform.googleblog.com/2016/03/G...</td>\n",
       "      <td>Google Data Center 360° Tour</td>\n",
       "      <td>We're excited to share the Google Data Center ...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1459194497</td>\n",
       "      <td>CONTENT SHARED</td>\n",
       "      <td>2448026894306402386</td>\n",
       "      <td>4340306774493623681</td>\n",
       "      <td>8940341205206233829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML</td>\n",
       "      <td>https://bitcoinmagazine.com/articles/ibm-wants...</td>\n",
       "      <td>IBM Wants to \"Evolve the Internet\" With Blockc...</td>\n",
       "      <td>The Aite Group projects the blockchain market ...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1459194522</td>\n",
       "      <td>CONTENT SHARED</td>\n",
       "      <td>-2826566343807132236</td>\n",
       "      <td>4340306774493623681</td>\n",
       "      <td>8940341205206233829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML</td>\n",
       "      <td>http://www.coindesk.com/ieee-blockchain-oxford...</td>\n",
       "      <td>IEEE to Talk Blockchain at Cloud Computing Oxf...</td>\n",
       "      <td>One of the largest and oldest organizations fo...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    timestamp       eventType            contentId       authorPersonId  \\\n",
       "1  1459193988  CONTENT SHARED -4110354420726924665  4340306774493623681   \n",
       "2  1459194146  CONTENT SHARED -7292285110016212249  4340306774493623681   \n",
       "3  1459194474  CONTENT SHARED -6151852268067518688  3891637997717104548   \n",
       "4  1459194497  CONTENT SHARED  2448026894306402386  4340306774493623681   \n",
       "5  1459194522  CONTENT SHARED -2826566343807132236  4340306774493623681   \n",
       "\n",
       "       authorSessionId authorUserAgent authorRegion authorCountry contentType  \\\n",
       "1  8940341205206233829             NaN          NaN           NaN        HTML   \n",
       "2  8940341205206233829             NaN          NaN           NaN        HTML   \n",
       "3 -1457532940883382585             NaN          NaN           NaN        HTML   \n",
       "4  8940341205206233829             NaN          NaN           NaN        HTML   \n",
       "5  8940341205206233829             NaN          NaN           NaN        HTML   \n",
       "\n",
       "                                                 url  \\\n",
       "1  http://www.nytimes.com/2016/03/28/business/dea...   \n",
       "2  http://cointelegraph.com/news/bitcoin-future-w...   \n",
       "3  https://cloudplatform.googleblog.com/2016/03/G...   \n",
       "4  https://bitcoinmagazine.com/articles/ibm-wants...   \n",
       "5  http://www.coindesk.com/ieee-blockchain-oxford...   \n",
       "\n",
       "                                               title  \\\n",
       "1  Ethereum, a Virtual Currency, Enables Transact...   \n",
       "2  Bitcoin Future: When GBPcoin of Branson Wins O...   \n",
       "3                       Google Data Center 360° Tour   \n",
       "4  IBM Wants to \"Evolve the Internet\" With Blockc...   \n",
       "5  IEEE to Talk Blockchain at Cloud Computing Oxf...   \n",
       "\n",
       "                                                text lang  \n",
       "1  All of this work is still very early. The firs...   en  \n",
       "2  The alarm clock wakes me at 8:00 with stream o...   en  \n",
       "3  We're excited to share the Google Data Center ...   en  \n",
       "4  The Aite Group projects the blockchain market ...   en  \n",
       "5  One of the largest and oldest organizations fo...   en  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_df = pd.read_csv('/Users/rileychiu/Desktop/deskdrop/shared_articles.csv')\n",
    "articles_df = articles_df[articles_df['eventType'] == 'CONTENT SHARED']\n",
    "articles_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>eventType</th>\n",
       "      <th>contentId</th>\n",
       "      <th>personId</th>\n",
       "      <th>sessionId</th>\n",
       "      <th>userAgent</th>\n",
       "      <th>userRegion</th>\n",
       "      <th>userCountry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1465413032</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>-3499919498720038879</td>\n",
       "      <td>-8845298781299428018</td>\n",
       "      <td>1264196770339959068</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1465412560</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>8890720798209849691</td>\n",
       "      <td>-1032019229384696495</td>\n",
       "      <td>3621737643587579081</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2...</td>\n",
       "      <td>NY</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1465416190</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>310515487419366995</td>\n",
       "      <td>-1130272294246983140</td>\n",
       "      <td>2631864456530402479</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1465413895</td>\n",
       "      <td>FOLLOW</td>\n",
       "      <td>310515487419366995</td>\n",
       "      <td>344280948527967603</td>\n",
       "      <td>-3167637573980064150</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1465412290</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>-7820640624231356730</td>\n",
       "      <td>-445337111692715325</td>\n",
       "      <td>5611481178424124714</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1465413742</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>310515487419366995</td>\n",
       "      <td>-8763398617720485024</td>\n",
       "      <td>1395789369402380392</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebK...</td>\n",
       "      <td>MG</td>\n",
       "      <td>BR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1465415950</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>-8864073373672512525</td>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>1143207167886864524</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1465415066</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>-1492913151930215984</td>\n",
       "      <td>4254153380739593270</td>\n",
       "      <td>8743229464706506141</td>\n",
       "      <td>Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53...</td>\n",
       "      <td>SP</td>\n",
       "      <td>BR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1465413762</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>310515487419366995</td>\n",
       "      <td>344280948527967603</td>\n",
       "      <td>-3167637573980064150</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1465413771</td>\n",
       "      <td>VIEW</td>\n",
       "      <td>3064370296170038610</td>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>1143207167886864524</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    timestamp eventType            contentId             personId  \\\n",
       "0  1465413032      VIEW -3499919498720038879 -8845298781299428018   \n",
       "1  1465412560      VIEW  8890720798209849691 -1032019229384696495   \n",
       "2  1465416190      VIEW   310515487419366995 -1130272294246983140   \n",
       "3  1465413895    FOLLOW   310515487419366995   344280948527967603   \n",
       "4  1465412290      VIEW -7820640624231356730  -445337111692715325   \n",
       "5  1465413742      VIEW   310515487419366995 -8763398617720485024   \n",
       "6  1465415950      VIEW -8864073373672512525  3609194402293569455   \n",
       "7  1465415066      VIEW -1492913151930215984  4254153380739593270   \n",
       "8  1465413762      VIEW   310515487419366995   344280948527967603   \n",
       "9  1465413771      VIEW  3064370296170038610  3609194402293569455   \n",
       "\n",
       "             sessionId                                          userAgent  \\\n",
       "0  1264196770339959068                                                NaN   \n",
       "1  3621737643587579081  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2...   \n",
       "2  2631864456530402479                                                NaN   \n",
       "3 -3167637573980064150                                                NaN   \n",
       "4  5611481178424124714                                                NaN   \n",
       "5  1395789369402380392  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebK...   \n",
       "6  1143207167886864524                                                NaN   \n",
       "7  8743229464706506141  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53...   \n",
       "8 -3167637573980064150                                                NaN   \n",
       "9  1143207167886864524                                                NaN   \n",
       "\n",
       "  userRegion userCountry  \n",
       "0        NaN         NaN  \n",
       "1         NY          US  \n",
       "2        NaN         NaN  \n",
       "3        NaN         NaN  \n",
       "4        NaN         NaN  \n",
       "5         MG          BR  \n",
       "6        NaN         NaN  \n",
       "7         SP          BR  \n",
       "8        NaN         NaN  \n",
       "9        NaN         NaN  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interactions_df = pd.read_csv('/Users/rileychiu/Desktop/deskdrop/users_interactions.csv')\n",
    "interactions_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As there are different interactions types, we associate them with a weight or strength, assuming that, for example, a comment in an article indicates a higher interest of the user on the item than a like, or than a simple view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "event_type_strength = {\n",
    "   'VIEW': 1.0,\n",
    "   'LIKE': 2.0, \n",
    "   'BOOKMARK': 2.5, \n",
    "   'FOLLOW': 3.0,\n",
    "   'COMMENT CREATED': 4.0,  \n",
    "}\n",
    "\n",
    "interactions_df['eventStrength'] = interactions_df['eventType'].apply(lambda x: event_type_strength[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recommender systems have a problem known as user cold-start, in which is hard do provide personalized recommendations for users with none or a very few number of consumed items, due to the lack of information to model their preferences.\n",
    "For this reason, we are keeping in the dataset only users with at leas 10 interactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# users: 1895\n",
      "# users with at least 10 interactions: 949\n"
     ]
    }
   ],
   "source": [
    "users_interactions_count_df = interactions_df.groupby('personId').size()\n",
    "print('# users: %d' % len(users_interactions_count_df))\n",
    "users_with_enough_interactions_df = users_interactions_count_df[users_interactions_count_df >= 10].reset_index()[['personId']]\n",
    "print('# users with at least 10 interactions: %d' % len(users_with_enough_interactions_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of interactions: 72312\n",
      "# of interactions from users with at least 10 interactions: 68524\n"
     ]
    }
   ],
   "source": [
    "print('# of interactions: %d' % len(interactions_df))\n",
    "interactions_from_selected_users_df = interactions_df.merge(users_with_enough_interactions_df, \n",
    "               how = 'right',\n",
    "               left_on = 'personId',\n",
    "               right_on = 'personId')\n",
    "print('# of interactions from users with at least 10 interactions: %d' % len(interactions_from_selected_users_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of unique user/item interactions: 37926\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>personId</th>\n",
       "      <th>contentId</th>\n",
       "      <th>eventStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-8949113594875411859</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-8377626164558006982</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-8208801367848627943</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-8187220755213888616</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-7423191370472335463</td>\n",
       "      <td>3.169925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-7331393944609614247</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-6872546942144599345</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-6728844082024523434</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-6590819806697898649</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-9223121837663643404</td>\n",
       "      <td>-6558712014192834002</td>\n",
       "      <td>1.584963</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              personId            contentId  eventStrength\n",
       "0 -9223121837663643404 -8949113594875411859       1.000000\n",
       "1 -9223121837663643404 -8377626164558006982       1.000000\n",
       "2 -9223121837663643404 -8208801367848627943       1.000000\n",
       "3 -9223121837663643404 -8187220755213888616       1.000000\n",
       "4 -9223121837663643404 -7423191370472335463       3.169925\n",
       "5 -9223121837663643404 -7331393944609614247       1.000000\n",
       "6 -9223121837663643404 -6872546942144599345       1.000000\n",
       "7 -9223121837663643404 -6728844082024523434       1.000000\n",
       "8 -9223121837663643404 -6590819806697898649       1.000000\n",
       "9 -9223121837663643404 -6558712014192834002       1.584963"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def smooth_user_preference(x):\n",
    "    return math.log(1+x, 2)\n",
    "    \n",
    "interactions_full_df = interactions_from_selected_users_df \\\n",
    "                    .groupby(['personId', 'contentId'])['eventStrength'].sum() \\\n",
    "                    .apply(smooth_user_preference).reset_index()\n",
    "print('# of unique user/item interactions: %d' % len(interactions_full_df))\n",
    "interactions_full_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Recommender Systems, there are a set metrics commonly used for evaluation. We chose to work with Top-N accuracy metrics, which evaluates the accuracy of the top recommendations provided to a user, comparing to the items the user has actually interacted in test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# interactions on Train set: 30340\n",
      "# interactions on Test set: 7586\n"
     ]
    }
   ],
   "source": [
    "interactions_train_df, interactions_test_df = train_test_split(interactions_full_df,\n",
    "                                   stratify=interactions_full_df['personId'], \n",
    "                                   test_size=0.20,\n",
    "                                   random_state=42)\n",
    "\n",
    "print('# interactions on Train set: %d' % len(interactions_train_df))\n",
    "print('# interactions on Test set: %d' % len(interactions_test_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Indexing by personId to speed up the searches during evaluation\n",
    "interactions_full_indexed_df = interactions_full_df.set_index('personId')\n",
    "interactions_train_indexed_df = interactions_train_df.set_index('personId')\n",
    "interactions_test_indexed_df = interactions_test_df.set_index('personId')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Recommender Systems, there are a set metrics commonly used for evaluation. We chose to work with Top-N accuracy metrics, which evaluates the accuracy of the top recommendations provided to a user, comparing to the items the user has actually interacted in test set.\n",
    "This evaluation method works as follows:\n",
    "\n",
    "For each user:\n",
    "\n",
    "For each item the user has interacted in test set:\n",
    "\n",
    "Sample 100 other items the user has never interacted.\n",
    "Ps. Here we naively assume those non interacted items are not relevant to the user, which might not be true, as the user may simply not be aware of those not interacted items. But let's keep this assumption.\n",
    "\n",
    "Ask the recommender model to produce a ranked list of recommended items, from a set composed one interacted item and the 100 non-interacted (\"non-relevant!) items\n",
    "\n",
    "Compute the Top-N accuracy metrics for this user and interacted item from the recommendations ranked list\n",
    "\n",
    "Aggregate the global Top-N accuracy metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Top-N accuracy metric choosen was Recall@N which evaluates whether the interacted item is among the top N items (hit) in the ranked list of 101 recommendations for a user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall=(recommend intersect interacted)/interacted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall at k is the proportion of relevant items found in the top-k recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we computed recall at 10 and found it is 40% in our top-10 recommendation system. This means that 40% of the total number of the relevant items appear in the top-k results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_items_interacted(person_id, interactions_df):\n",
    "    # Get the user's data and merge in the movie information.\n",
    "    interacted_items = interactions_df.loc[person_id]['contentId']\n",
    "    return set(interacted_items if type(interacted_items) == pd.Series else [interacted_items])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Top-N accuracy metrics consts\n",
    "EVAL_RANDOM_SAMPLE_NON_INTERACTED_ITEMS = 100\n",
    "\n",
    "class ModelEvaluator:\n",
    "\n",
    "\n",
    "    def get_not_interacted_items_sample(self, person_id, sample_size, seed=42):\n",
    "        interacted_items = get_items_interacted(person_id, interactions_full_indexed_df)\n",
    "        all_items = set(articles_df['contentId'])\n",
    "        non_interacted_items = all_items - interacted_items\n",
    "\n",
    "        random.seed(seed)\n",
    "        non_interacted_items_sample = random.sample(non_interacted_items, sample_size)\n",
    "        return set(non_interacted_items_sample)\n",
    "    def _verify_hit_top_n(self, item_id, recommended_items, topn):        \n",
    "            try:\n",
    "                index = next(i for i, c in enumerate(recommended_items) if c == item_id)\n",
    "            except:\n",
    "                index = -1\n",
    "            hit = int(index in range(0, topn))\n",
    "            return hit, index\n",
    "    def evaluate_model_for_user(self, model, person_id):\n",
    "        #Getting the items in test set\n",
    "        interacted_values_testset = interactions_test_indexed_df.loc[person_id]\n",
    "        if type(interacted_values_testset['contentId']) == pd.Series:\n",
    "            person_interacted_items_testset = set(interacted_values_testset['contentId'])\n",
    "        else:\n",
    "            person_interacted_items_testset = set([int(interacted_values_testset['contentId'])])  \n",
    "        interacted_items_count_testset = len(person_interacted_items_testset) \n",
    "\n",
    "        #Getting a ranked recommendation list from a model for a given user\n",
    "        person_recs_df = model.recommend_items(person_id, \n",
    "                                               items_to_ignore=get_items_interacted(person_id, \n",
    "                                                                                    interactions_train_indexed_df), \n",
    "                                               topn=10000000000)\n",
    "\n",
    "        hits_at_5_count = 0\n",
    "        hits_at_10_count = 0\n",
    "        #For each item the user has interacted in test set\n",
    "        for item_id in person_interacted_items_testset:\n",
    "            #Getting a random sample (100) items the user has not interacted \n",
    "            #(to represent items that are assumed to be no relevant to the user)\n",
    "            non_interacted_items_sample = self.get_not_interacted_items_sample(person_id, \n",
    "                                                                          sample_size=EVAL_RANDOM_SAMPLE_NON_INTERACTED_ITEMS, \n",
    "                                                                          seed=item_id%(2**32))\n",
    "\n",
    "            #Combining the current interacted item with the 100 random items\n",
    "            items_to_filter_recs = non_interacted_items_sample.union(set([item_id]))\n",
    "\n",
    "            #Filtering only recommendations that are either the interacted item or from a random sample of 100 non-interacted items\n",
    "            valid_recs_df = person_recs_df[person_recs_df['contentId'].isin(items_to_filter_recs)]                    \n",
    "            valid_recs = valid_recs_df['contentId'].values\n",
    "            #Verifying if the current interacted item is among the Top-N recommended items\n",
    "            hit_at_5, index_at_5 = self._verify_hit_top_n(item_id, valid_recs, 5)\n",
    "            hits_at_5_count += hit_at_5\n",
    "            hit_at_10, index_at_10 = self._verify_hit_top_n(item_id, valid_recs, 10)\n",
    "            hits_at_10_count += hit_at_10\n",
    "\n",
    "        #Recall is the rate of the interacted items that are ranked among the Top-N recommended items, \n",
    "        #when mixed with a set of non-relevant items\n",
    "        recall_at_5 = hits_at_5_count / float(interacted_items_count_testset)\n",
    "        recall_at_10 = hits_at_10_count / float(interacted_items_count_testset)\n",
    "        precision_at_5 = hits_at_5_count/101\n",
    "        precision_at_10 = hits_at_10_count/101\n",
    "\n",
    "        person_metrics = {'hits@5_count':hits_at_5_count, \n",
    "                          'hits@10_count':hits_at_10_count, \n",
    "                          'interacted_count': interacted_items_count_testset,\n",
    "                          'recall@5': recall_at_5,\n",
    "                          'recall@10': recall_at_10,\n",
    "                          'precision@5': precision_at_5,\n",
    "                          'precision@10':precision_at_10}\n",
    "        return person_metrics\n",
    "    def evaluate_model(self, model):\n",
    "        #print('Running evaluation for users')\n",
    "        people_metrics = []\n",
    "        for idx, person_id in enumerate(list(interactions_test_indexed_df.index.unique().values)):\n",
    "            #if idx % 100 == 0 and idx > 0:\n",
    "            #    print('%d users processed' % idx)\n",
    "            person_metrics = self.evaluate_model_for_user(model, person_id)  \n",
    "            person_metrics['_person_id'] = person_id\n",
    "            people_metrics.append(person_metrics)\n",
    "        print('%d users processed' % idx)\n",
    "\n",
    "        detailed_results_df = pd.DataFrame(people_metrics) \\\n",
    "                            .sort_values('interacted_count', ascending=False)\n",
    "        \n",
    "        global_recall_at_5 = detailed_results_df['hits@5_count'].sum() / float(detailed_results_df['interacted_count'].sum())\n",
    "        global_recall_at_10 = detailed_results_df['hits@10_count'].sum() / float(detailed_results_df['interacted_count'].sum())\n",
    "        global_precision_at_5 = detailed_results_df['hits@5_count'].sum() / (101*len(interactions_test_indexed_df.index.unique()))\n",
    "        global_precision_at_10 = detailed_results_df['hits@10_count'].sum() / (101*len(interactions_test_indexed_df.index.unique()))\n",
    "        \n",
    "        \n",
    "        global_metrics = {'modelName': model.get_model_name(),\n",
    "                          'recall@5': global_recall_at_5,\n",
    "                          'recall@10': global_recall_at_10,\n",
    "                          'precision@5': global_precision_at_5,\n",
    "                          'precision@10': global_precision_at_10}    \n",
    "        return global_metrics, detailed_results_df\n",
    "    \n",
    "model_evaluator = ModelEvaluator() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Popularity model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>contentId</th>\n",
       "      <th>eventStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-4029704725707465084</td>\n",
       "      <td>285.863434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-133139342397538859</td>\n",
       "      <td>214.269680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-6783772548752091658</td>\n",
       "      <td>212.007270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-8208801367848627943</td>\n",
       "      <td>190.107608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-6843047699859121724</td>\n",
       "      <td>188.410170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8224860111193157980</td>\n",
       "      <td>187.445559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2581138407738454418</td>\n",
       "      <td>174.282876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-2358756719610361882</td>\n",
       "      <td>173.941026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7507067965574797372</td>\n",
       "      <td>172.509040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3367026768872537336</td>\n",
       "      <td>162.939312</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             contentId  eventStrength\n",
       "0 -4029704725707465084     285.863434\n",
       "1  -133139342397538859     214.269680\n",
       "2 -6783772548752091658     212.007270\n",
       "3 -8208801367848627943     190.107608\n",
       "4 -6843047699859121724     188.410170\n",
       "5  8224860111193157980     187.445559\n",
       "6  2581138407738454418     174.282876\n",
       "7 -2358756719610361882     173.941026\n",
       "8  7507067965574797372     172.509040\n",
       "9  3367026768872537336     162.939312"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Computes the most popular items\n",
    "item_popularity_df = interactions_full_df.groupby('contentId')['eventStrength'].sum().sort_values(ascending=False).reset_index()\n",
    "item_popularity_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PopularityRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Popularity'\n",
    "    \n",
    "    def __init__(self, popularity_df, items_df=None):\n",
    "        self.popularity_df = popularity_df\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "        \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        # Recommend the more popular items that the user hasn't seen yet.\n",
    "        recommendations_df = self.popularity_df[~self.popularity_df['contentId'].isin(items_to_ignore)] \\\n",
    "                               .sort_values('eventStrength', ascending = False) \\\n",
    "                               .head(topn)\n",
    "\n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'contentId', \n",
    "                                                          right_on = 'contentId')[['eventStrength', 'contentId', 'title', 'url', 'lang']]\n",
    "        return recommendations_df\n",
    "    \n",
    "popularity_model = PopularityRecommender(item_popularity_df, articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Popularity recommendation model...\n",
      "947 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Popularity', 'recall@5': 0.24677036646453995, 'recall@10': 0.38083311363037176, 'precision@5': 0.019551322220829677, 'precision@10': 0.030172954004261187}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_person_id</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>precision@10</th>\n",
       "      <th>precision@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>recall@5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>45</td>\n",
       "      <td>26</td>\n",
       "      <td>192</td>\n",
       "      <td>0.445545</td>\n",
       "      <td>0.257426</td>\n",
       "      <td>0.234375</td>\n",
       "      <td>0.135417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2626634673110551643</td>\n",
       "      <td>30</td>\n",
       "      <td>14</td>\n",
       "      <td>134</td>\n",
       "      <td>0.297030</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>0.223881</td>\n",
       "      <td>0.104478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>-1032019229384696495</td>\n",
       "      <td>28</td>\n",
       "      <td>12</td>\n",
       "      <td>130</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>0.118812</td>\n",
       "      <td>0.215385</td>\n",
       "      <td>0.092308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>-1443636648652872475</td>\n",
       "      <td>17</td>\n",
       "      <td>6</td>\n",
       "      <td>117</td>\n",
       "      <td>0.168317</td>\n",
       "      <td>0.059406</td>\n",
       "      <td>0.145299</td>\n",
       "      <td>0.051282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>-2979881261169775358</td>\n",
       "      <td>31</td>\n",
       "      <td>23</td>\n",
       "      <td>87</td>\n",
       "      <td>0.306931</td>\n",
       "      <td>0.227723</td>\n",
       "      <td>0.356322</td>\n",
       "      <td>0.264368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>-3596626804281480007</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>80</td>\n",
       "      <td>0.178218</td>\n",
       "      <td>0.128713</td>\n",
       "      <td>0.225000</td>\n",
       "      <td>0.162500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>1116121227607581999</td>\n",
       "      <td>18</td>\n",
       "      <td>8</td>\n",
       "      <td>73</td>\n",
       "      <td>0.178218</td>\n",
       "      <td>0.079208</td>\n",
       "      <td>0.246575</td>\n",
       "      <td>0.109589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>692689608292948411</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>69</td>\n",
       "      <td>0.178218</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>0.260870</td>\n",
       "      <td>0.202899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>3636910968448833585</td>\n",
       "      <td>32</td>\n",
       "      <td>21</td>\n",
       "      <td>68</td>\n",
       "      <td>0.316832</td>\n",
       "      <td>0.207921</td>\n",
       "      <td>0.470588</td>\n",
       "      <td>0.308824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>-9016528795238256703</td>\n",
       "      <td>26</td>\n",
       "      <td>15</td>\n",
       "      <td>68</td>\n",
       "      <td>0.257426</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.382353</td>\n",
       "      <td>0.220588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              _person_id  hits@10_count  hits@5_count  interacted_count  \\\n",
       "3    3609194402293569455             45            26               192   \n",
       "5   -2626634673110551643             30            14               134   \n",
       "37  -1032019229384696495             28            12               130   \n",
       "89  -1443636648652872475             17             6               117   \n",
       "200 -2979881261169775358             31            23                87   \n",
       "121 -3596626804281480007             18            13                80   \n",
       "127  1116121227607581999             18             8                73   \n",
       "80    692689608292948411             18            14                69   \n",
       "116  3636910968448833585             32            21                68   \n",
       "104 -9016528795238256703             26            15                68   \n",
       "\n",
       "     precision@10  precision@5  recall@10  recall@5  \n",
       "3        0.445545     0.257426   0.234375  0.135417  \n",
       "5        0.297030     0.138614   0.223881  0.104478  \n",
       "37       0.277228     0.118812   0.215385  0.092308  \n",
       "89       0.168317     0.059406   0.145299  0.051282  \n",
       "200      0.306931     0.227723   0.356322  0.264368  \n",
       "121      0.178218     0.128713   0.225000  0.162500  \n",
       "127      0.178218     0.079208   0.246575  0.109589  \n",
       "80       0.178218     0.138614   0.260870  0.202899  \n",
       "116      0.316832     0.207921   0.470588  0.308824  \n",
       "104      0.257426     0.148515   0.382353  0.220588  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Popularity recommendation model...')\n",
    "pop_global_metrics, pop_detailed_results_df = model_evaluator.evaluate_model(popularity_model)\n",
    "print('\\nGlobal metrics:\\n%s' % pop_global_metrics)\n",
    "pop_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Content-Based Filtering model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3047x5000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 638928 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Ignoring stopwords (words with no semantics) from English and Portuguese (as we have a corpus with mixed languages)\n",
    "stopwords_list = stopwords.words('english') + stopwords.words('portuguese')\n",
    "\n",
    "#Trains a model whose vectors size is 5000, composed by the main unigrams and bigrams found in the corpus, ignoring stopwords\n",
    "vectorizer = TfidfVectorizer(analyzer='word',\n",
    "                     ngram_range=(1, 2),\n",
    "                     min_df=0.003,\n",
    "                     max_df=0.5,\n",
    "                     max_features=5000,\n",
    "                     stop_words=stopwords_list)\n",
    "\n",
    "item_ids = articles_df['contentId'].tolist()\n",
    "tfidf_matrix = vectorizer.fit_transform(articles_df['title'] + \"\" + articles_df['text'])\n",
    "tfidf_feature_names = vectorizer.get_feature_names()\n",
    "tfidf_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_item_profile(item_id):\n",
    "    idx = item_ids.index(item_id)\n",
    "    item_profile = tfidf_matrix[idx:idx+1]\n",
    "    return item_profile\n",
    "\n",
    "def get_item_profiles(ids):\n",
    "    item_profiles_list = [get_item_profile(x) for x in ids]\n",
    "    item_profiles = scipy.sparse.vstack(item_profiles_list)\n",
    "    return item_profiles\n",
    "\n",
    "def build_users_profile(person_id, interactions_indexed_df):\n",
    "    interactions_person_df = interactions_indexed_df.loc[person_id]\n",
    "    user_item_profiles = get_item_profiles(interactions_person_df['contentId'])\n",
    "    \n",
    "    user_item_strengths = np.array(interactions_person_df['eventStrength']).reshape(-1,1)\n",
    "    #Weighted average of item profiles by the interactions strength\n",
    "    user_item_strengths_weighted_avg = np.sum(user_item_profiles.multiply(user_item_strengths), axis=0) / np.sum(user_item_strengths)\n",
    "    user_profile_norm = sklearn.preprocessing.normalize(user_item_strengths_weighted_avg)\n",
    "    return user_profile_norm\n",
    "\n",
    "def build_users_profiles(): \n",
    "    interactions_indexed_df = interactions_full_df[interactions_full_df['contentId'] \\\n",
    "                                                   .isin(articles_df['contentId'])].set_index('personId')\n",
    "    user_profiles = {}\n",
    "    for person_id in interactions_indexed_df.index.unique():\n",
    "        user_profiles[person_id] = build_users_profile(person_id, interactions_indexed_df)\n",
    "    return user_profiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "949"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_profiles = build_users_profiles()\n",
    "len(user_profiles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 5000)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>token</th>\n",
       "      <th>relevance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>learning</td>\n",
       "      <td>0.305655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>machine learning</td>\n",
       "      <td>0.255557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>machine</td>\n",
       "      <td>0.246095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>google</td>\n",
       "      <td>0.208590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>data</td>\n",
       "      <td>0.172509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ai</td>\n",
       "      <td>0.136818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>algorithms</td>\n",
       "      <td>0.102396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>graph</td>\n",
       "      <td>0.098438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>like</td>\n",
       "      <td>0.096970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>language</td>\n",
       "      <td>0.083993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>people</td>\n",
       "      <td>0.077122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>use</td>\n",
       "      <td>0.073203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>models</td>\n",
       "      <td>0.073168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>deep</td>\n",
       "      <td>0.072377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>deep learning</td>\n",
       "      <td>0.071892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>research</td>\n",
       "      <td>0.071503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>company</td>\n",
       "      <td>0.071293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>intelligence</td>\n",
       "      <td>0.070650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>human</td>\n",
       "      <td>0.070126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>artificial</td>\n",
       "      <td>0.068062</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               token  relevance\n",
       "0           learning   0.305655\n",
       "1   machine learning   0.255557\n",
       "2            machine   0.246095\n",
       "3             google   0.208590\n",
       "4               data   0.172509\n",
       "5                 ai   0.136818\n",
       "6         algorithms   0.102396\n",
       "7              graph   0.098438\n",
       "8               like   0.096970\n",
       "9           language   0.083993\n",
       "10            people   0.077122\n",
       "11               use   0.073203\n",
       "12            models   0.073168\n",
       "13              deep   0.072377\n",
       "14     deep learning   0.071892\n",
       "15          research   0.071503\n",
       "16           company   0.071293\n",
       "17      intelligence   0.070650\n",
       "18             human   0.070126\n",
       "19        artificial   0.068062"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myprofile = user_profiles[-1479311724257856983]\n",
    "print(myprofile.shape)\n",
    "pd.DataFrame(sorted(zip(tfidf_feature_names, \n",
    "                        user_profiles[-1479311724257856983].flatten().tolist()), key=lambda x: -x[1])[:20],\n",
    "             columns=['token', 'relevance'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ContentBasedRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Content-Based'\n",
    "    \n",
    "    def __init__(self, items_df=None):\n",
    "        self.item_ids = item_ids\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "        \n",
    "    def _get_similar_items_to_user_profile(self, person_id, topn=1000):\n",
    "        #Computes the cosine similarity between the user profile and all item profiles\n",
    "        cosine_similarities = cosine_similarity(user_profiles[person_id], tfidf_matrix)\n",
    "        #Gets the top similar items\n",
    "        similar_indices = cosine_similarities.argsort().flatten()[-topn:]\n",
    "        #Sort the similar items by similarity\n",
    "        similar_items = sorted([(item_ids[i], cosine_similarities[0,i]) for i in similar_indices], key=lambda x: -x[1])\n",
    "        return similar_items\n",
    "    \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        similar_items = self._get_similar_items_to_user_profile(user_id)\n",
    "        #Ignores items the user has already interacted\n",
    "        similar_items_filtered = list(filter(lambda x: x[0] not in items_to_ignore, similar_items))\n",
    "        \n",
    "        recommendations_df = pd.DataFrame(similar_items_filtered, columns=['contentId', 'recStrength']) \\\n",
    "                                    .head(topn)\n",
    "\n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'contentId', \n",
    "                                                          right_on = 'contentId')[['recStrength', 'contentId', 'title', 'url', 'lang']]\n",
    "\n",
    "\n",
    "        return recommendations_df\n",
    "    \n",
    "content_based_recommender_model = ContentBasedRecommender(articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Content-Based Filtering model...\n",
      "947 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Content-Based', 'recall@5': 0.43000263643553915, 'recall@10': 0.5346691273398365, 'precision@5': 0.03406859673309103, 'precision@10': 0.042361198145130965}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_person_id</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>precision@10</th>\n",
       "      <th>precision@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>recall@5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>192</td>\n",
       "      <td>0.247525</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.130208</td>\n",
       "      <td>0.078125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2626634673110551643</td>\n",
       "      <td>30</td>\n",
       "      <td>13</td>\n",
       "      <td>134</td>\n",
       "      <td>0.297030</td>\n",
       "      <td>0.128713</td>\n",
       "      <td>0.223881</td>\n",
       "      <td>0.097015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>-1032019229384696495</td>\n",
       "      <td>36</td>\n",
       "      <td>20</td>\n",
       "      <td>130</td>\n",
       "      <td>0.356436</td>\n",
       "      <td>0.198020</td>\n",
       "      <td>0.276923</td>\n",
       "      <td>0.153846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>-1443636648652872475</td>\n",
       "      <td>57</td>\n",
       "      <td>41</td>\n",
       "      <td>117</td>\n",
       "      <td>0.564356</td>\n",
       "      <td>0.405941</td>\n",
       "      <td>0.487179</td>\n",
       "      <td>0.350427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>-2979881261169775358</td>\n",
       "      <td>17</td>\n",
       "      <td>11</td>\n",
       "      <td>87</td>\n",
       "      <td>0.168317</td>\n",
       "      <td>0.108911</td>\n",
       "      <td>0.195402</td>\n",
       "      <td>0.126437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>-3596626804281480007</td>\n",
       "      <td>27</td>\n",
       "      <td>19</td>\n",
       "      <td>80</td>\n",
       "      <td>0.267327</td>\n",
       "      <td>0.188119</td>\n",
       "      <td>0.337500</td>\n",
       "      <td>0.237500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>1116121227607581999</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>73</td>\n",
       "      <td>0.108911</td>\n",
       "      <td>0.029703</td>\n",
       "      <td>0.150685</td>\n",
       "      <td>0.041096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>692689608292948411</td>\n",
       "      <td>23</td>\n",
       "      <td>17</td>\n",
       "      <td>69</td>\n",
       "      <td>0.227723</td>\n",
       "      <td>0.168317</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.246377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>3636910968448833585</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>68</td>\n",
       "      <td>0.089109</td>\n",
       "      <td>0.039604</td>\n",
       "      <td>0.132353</td>\n",
       "      <td>0.058824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>-9016528795238256703</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>68</td>\n",
       "      <td>0.168317</td>\n",
       "      <td>0.099010</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.147059</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              _person_id  hits@10_count  hits@5_count  interacted_count  \\\n",
       "3    3609194402293569455             25            15               192   \n",
       "5   -2626634673110551643             30            13               134   \n",
       "37  -1032019229384696495             36            20               130   \n",
       "89  -1443636648652872475             57            41               117   \n",
       "200 -2979881261169775358             17            11                87   \n",
       "121 -3596626804281480007             27            19                80   \n",
       "127  1116121227607581999             11             3                73   \n",
       "80    692689608292948411             23            17                69   \n",
       "116  3636910968448833585              9             4                68   \n",
       "104 -9016528795238256703             17            10                68   \n",
       "\n",
       "     precision@10  precision@5  recall@10  recall@5  \n",
       "3        0.247525     0.148515   0.130208  0.078125  \n",
       "5        0.297030     0.128713   0.223881  0.097015  \n",
       "37       0.356436     0.198020   0.276923  0.153846  \n",
       "89       0.564356     0.405941   0.487179  0.350427  \n",
       "200      0.168317     0.108911   0.195402  0.126437  \n",
       "121      0.267327     0.188119   0.337500  0.237500  \n",
       "127      0.108911     0.029703   0.150685  0.041096  \n",
       "80       0.227723     0.168317   0.333333  0.246377  \n",
       "116      0.089109     0.039604   0.132353  0.058824  \n",
       "104      0.168317     0.099010   0.250000  0.147059  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Content-Based Filtering model...')\n",
    "cb_global_metrics, cb_detailed_results_df = model_evaluator.evaluate_model(content_based_recommender_model)\n",
    "print('\\nGlobal metrics:\\n%s' % cb_global_metrics)\n",
    "cb_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaborative Filtering model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>contentId</th>\n",
       "      <th>-9222795471790223670</th>\n",
       "      <th>-9216926795620865886</th>\n",
       "      <th>-9194572880052200111</th>\n",
       "      <th>-9192549002213406534</th>\n",
       "      <th>-9190737901804729417</th>\n",
       "      <th>-9189659052158407108</th>\n",
       "      <th>-9184137057748005562</th>\n",
       "      <th>-9176143510534135851</th>\n",
       "      <th>-9172673334835262304</th>\n",
       "      <th>-9171475473795142532</th>\n",
       "      <th>...</th>\n",
       "      <th>9191014301634017491</th>\n",
       "      <th>9207286802575546269</th>\n",
       "      <th>9208127165664287660</th>\n",
       "      <th>9209629151177723638</th>\n",
       "      <th>9209886322932807692</th>\n",
       "      <th>9213260650272029784</th>\n",
       "      <th>9215261273565326920</th>\n",
       "      <th>9217155070834564627</th>\n",
       "      <th>9220445660318725468</th>\n",
       "      <th>9222265156747237864</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>personId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>-9223121837663643404</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9207251133131336884</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9199575329909162940</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9196668942822132778</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9188188261933657343</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9172914609055320039</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9156344805277471150</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9120685872592674274</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9109785559521267180</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9060214117327732109</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 2933 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "contentId             -9222795471790223670  -9216926795620865886  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   2.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId             -9194572880052200111  -9192549002213406534  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId             -9190737901804729417  -9189659052158407108  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId             -9184137057748005562  -9176143510534135851  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId             -9172673334835262304  -9171475473795142532  ...  \\\n",
       "personId                                                          ...   \n",
       "-9223121837663643404                   0.0                   0.0  ...   \n",
       "-9207251133131336884                   0.0                   0.0  ...   \n",
       "-9199575329909162940                   0.0                   0.0  ...   \n",
       "-9196668942822132778                   0.0                   0.0  ...   \n",
       "-9188188261933657343                   0.0                   0.0  ...   \n",
       "-9172914609055320039                   0.0                   0.0  ...   \n",
       "-9156344805277471150                   0.0                   0.0  ...   \n",
       "-9120685872592674274                   0.0                   0.0  ...   \n",
       "-9109785559521267180                   0.0                   0.0  ...   \n",
       "-9060214117327732109                   0.0                   0.0  ...   \n",
       "\n",
       "contentId              9191014301634017491   9207286802575546269  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId              9208127165664287660   9209629151177723638  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId              9209886322932807692   9213260650272029784  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId              9215261273565326920   9217155070834564627  \\\n",
       "personId                                                           \n",
       "-9223121837663643404                   0.0                   0.0   \n",
       "-9207251133131336884                   0.0                   0.0   \n",
       "-9199575329909162940                   0.0                   0.0   \n",
       "-9196668942822132778                   0.0                   0.0   \n",
       "-9188188261933657343                   0.0                   0.0   \n",
       "-9172914609055320039                   0.0                   0.0   \n",
       "-9156344805277471150                   0.0                   0.0   \n",
       "-9120685872592674274                   0.0                   0.0   \n",
       "-9109785559521267180                   0.0                   0.0   \n",
       "-9060214117327732109                   0.0                   0.0   \n",
       "\n",
       "contentId              9220445660318725468   9222265156747237864  \n",
       "personId                                                          \n",
       "-9223121837663643404                   0.0                   0.0  \n",
       "-9207251133131336884                   0.0                   0.0  \n",
       "-9199575329909162940                   0.0                   0.0  \n",
       "-9196668942822132778                   0.0                   0.0  \n",
       "-9188188261933657343                   0.0                   0.0  \n",
       "-9172914609055320039                   0.0                   0.0  \n",
       "-9156344805277471150                   0.0                   0.0  \n",
       "-9120685872592674274                   0.0                   0.0  \n",
       "-9109785559521267180                   0.0                   0.0  \n",
       "-9060214117327732109                   0.0                   0.0  \n",
       "\n",
       "[10 rows x 2933 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Creating a sparse pivot table with users in rows and items in columns\n",
    "users_items_pivot_matrix_df = interactions_train_df.pivot(index='personId', \n",
    "                                                          columns='contentId', \n",
    "                                                          values='eventStrength').fillna(0)\n",
    "\n",
    "users_items_pivot_matrix_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 2., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_items_pivot_matrix = users_items_pivot_matrix_df.as_matrix()\n",
    "users_items_pivot_matrix[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-9223121837663643404,\n",
       " -9207251133131336884,\n",
       " -9199575329909162940,\n",
       " -9196668942822132778,\n",
       " -9188188261933657343,\n",
       " -9172914609055320039,\n",
       " -9156344805277471150,\n",
       " -9120685872592674274,\n",
       " -9109785559521267180,\n",
       " -9060214117327732109]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_ids = list(users_items_pivot_matrix_df.index)\n",
    "users_ids[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The number of factors to factor the user-item matrix.\n",
    "NUMBER_OF_FACTORS_MF = 15\n",
    "#Performs matrix factorization of the original user item matrix\n",
    "U, sigma, Vt = svds(users_items_pivot_matrix, k = NUMBER_OF_FACTORS_MF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(949, 15)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15, 2933)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Vt.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15, 15)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma = np.diag(sigma)\n",
    "sigma.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.04239031,  0.0008807 ,  0.01225532, ..., -0.01590902,\n",
       "        -0.00310311, -0.00271164],\n",
       "       [ 0.00718358,  0.00748234, -0.01501065, ..., -0.01689828,\n",
       "        -0.00369596,  0.01090595],\n",
       "       [ 0.02172282,  0.00027781,  0.00585874, ...,  0.00053862,\n",
       "        -0.00375026,  0.00300554],\n",
       "       ...,\n",
       "       [-0.02749938,  0.00613331, -0.0301241 , ..., -0.03862674,\n",
       "        -0.00188554,  0.01523204],\n",
       "       [-0.02651425,  0.00084792,  0.02506398, ..., -0.00776269,\n",
       "         0.00516115, -0.01689742],\n",
       "       [-0.02111173, -0.00109385,  0.15840137, ..., -0.0558295 ,\n",
       "         0.03173557,  0.02190565]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) \n",
    "all_user_predicted_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>-9223121837663643404</th>\n",
       "      <th>-9207251133131336884</th>\n",
       "      <th>-9199575329909162940</th>\n",
       "      <th>-9196668942822132778</th>\n",
       "      <th>-9188188261933657343</th>\n",
       "      <th>-9172914609055320039</th>\n",
       "      <th>-9156344805277471150</th>\n",
       "      <th>-9120685872592674274</th>\n",
       "      <th>-9109785559521267180</th>\n",
       "      <th>-9060214117327732109</th>\n",
       "      <th>...</th>\n",
       "      <th>9102085903669288476</th>\n",
       "      <th>9105269044962898535</th>\n",
       "      <th>9109075639526981934</th>\n",
       "      <th>9135582630122950040</th>\n",
       "      <th>9137372837662939523</th>\n",
       "      <th>9148269800512008413</th>\n",
       "      <th>9187866633451383747</th>\n",
       "      <th>9191849144618614467</th>\n",
       "      <th>9199170757466086545</th>\n",
       "      <th>9210530975708218054</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>contentId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>-9222795471790223670</th>\n",
       "      <td>0.042390</td>\n",
       "      <td>0.007184</td>\n",
       "      <td>0.021723</td>\n",
       "      <td>-0.010762</td>\n",
       "      <td>0.001325</td>\n",
       "      <td>0.011603</td>\n",
       "      <td>0.051796</td>\n",
       "      <td>-0.015273</td>\n",
       "      <td>0.004398</td>\n",
       "      <td>-0.001038</td>\n",
       "      <td>...</td>\n",
       "      <td>0.044255</td>\n",
       "      <td>-0.004637</td>\n",
       "      <td>-0.028743</td>\n",
       "      <td>0.003642</td>\n",
       "      <td>0.005009</td>\n",
       "      <td>-0.001472</td>\n",
       "      <td>-0.027778</td>\n",
       "      <td>-0.027499</td>\n",
       "      <td>-0.026514</td>\n",
       "      <td>-0.021112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9216926795620865886</th>\n",
       "      <td>0.000881</td>\n",
       "      <td>0.007482</td>\n",
       "      <td>0.000278</td>\n",
       "      <td>-0.000505</td>\n",
       "      <td>0.000553</td>\n",
       "      <td>0.000355</td>\n",
       "      <td>0.000883</td>\n",
       "      <td>-0.000650</td>\n",
       "      <td>0.003010</td>\n",
       "      <td>0.000097</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.000298</td>\n",
       "      <td>0.001903</td>\n",
       "      <td>0.013378</td>\n",
       "      <td>0.000167</td>\n",
       "      <td>0.000696</td>\n",
       "      <td>0.008339</td>\n",
       "      <td>0.000899</td>\n",
       "      <td>0.006133</td>\n",
       "      <td>0.000848</td>\n",
       "      <td>-0.001094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9194572880052200111</th>\n",
       "      <td>0.012255</td>\n",
       "      <td>-0.015011</td>\n",
       "      <td>0.005859</td>\n",
       "      <td>0.028180</td>\n",
       "      <td>0.000821</td>\n",
       "      <td>0.028659</td>\n",
       "      <td>-0.015218</td>\n",
       "      <td>-0.017560</td>\n",
       "      <td>0.015087</td>\n",
       "      <td>-0.006885</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.006691</td>\n",
       "      <td>0.004277</td>\n",
       "      <td>0.036837</td>\n",
       "      <td>0.004882</td>\n",
       "      <td>0.024364</td>\n",
       "      <td>0.147745</td>\n",
       "      <td>0.017941</td>\n",
       "      <td>-0.030124</td>\n",
       "      <td>0.025064</td>\n",
       "      <td>0.158401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9192549002213406534</th>\n",
       "      <td>0.020014</td>\n",
       "      <td>-0.004314</td>\n",
       "      <td>-0.007215</td>\n",
       "      <td>0.023871</td>\n",
       "      <td>-0.000737</td>\n",
       "      <td>0.014883</td>\n",
       "      <td>0.040538</td>\n",
       "      <td>0.009572</td>\n",
       "      <td>-0.002032</td>\n",
       "      <td>-0.013772</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.004392</td>\n",
       "      <td>-0.006671</td>\n",
       "      <td>0.174971</td>\n",
       "      <td>0.002558</td>\n",
       "      <td>-0.000142</td>\n",
       "      <td>0.043061</td>\n",
       "      <td>0.014262</td>\n",
       "      <td>-0.003465</td>\n",
       "      <td>0.030838</td>\n",
       "      <td>0.102175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9190737901804729417</th>\n",
       "      <td>0.002898</td>\n",
       "      <td>0.005858</td>\n",
       "      <td>0.000592</td>\n",
       "      <td>-0.000404</td>\n",
       "      <td>0.000235</td>\n",
       "      <td>0.000644</td>\n",
       "      <td>0.000260</td>\n",
       "      <td>-0.000211</td>\n",
       "      <td>0.005017</td>\n",
       "      <td>-0.000280</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001010</td>\n",
       "      <td>0.001272</td>\n",
       "      <td>0.014423</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.001243</td>\n",
       "      <td>0.001038</td>\n",
       "      <td>0.006060</td>\n",
       "      <td>0.001626</td>\n",
       "      <td>-0.002482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9189659052158407108</th>\n",
       "      <td>0.025061</td>\n",
       "      <td>0.003186</td>\n",
       "      <td>-0.002692</td>\n",
       "      <td>0.005258</td>\n",
       "      <td>0.001853</td>\n",
       "      <td>0.000855</td>\n",
       "      <td>0.018287</td>\n",
       "      <td>0.028799</td>\n",
       "      <td>0.003002</td>\n",
       "      <td>0.003587</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.001910</td>\n",
       "      <td>0.016023</td>\n",
       "      <td>0.050675</td>\n",
       "      <td>0.000177</td>\n",
       "      <td>0.000173</td>\n",
       "      <td>0.095995</td>\n",
       "      <td>0.008093</td>\n",
       "      <td>0.012076</td>\n",
       "      <td>0.013978</td>\n",
       "      <td>0.038864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9184137057748005562</th>\n",
       "      <td>0.001760</td>\n",
       "      <td>0.003935</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>-0.000263</td>\n",
       "      <td>0.000330</td>\n",
       "      <td>0.000632</td>\n",
       "      <td>-0.000184</td>\n",
       "      <td>0.000512</td>\n",
       "      <td>0.003389</td>\n",
       "      <td>0.000526</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.000284</td>\n",
       "      <td>0.000878</td>\n",
       "      <td>0.010947</td>\n",
       "      <td>-0.000223</td>\n",
       "      <td>-0.000589</td>\n",
       "      <td>-0.003847</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.004369</td>\n",
       "      <td>0.001487</td>\n",
       "      <td>-0.000878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9176143510534135851</th>\n",
       "      <td>0.031526</td>\n",
       "      <td>0.019330</td>\n",
       "      <td>0.000523</td>\n",
       "      <td>-0.006001</td>\n",
       "      <td>-0.001163</td>\n",
       "      <td>0.004800</td>\n",
       "      <td>0.017766</td>\n",
       "      <td>0.084287</td>\n",
       "      <td>0.040455</td>\n",
       "      <td>0.023444</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.010236</td>\n",
       "      <td>0.107691</td>\n",
       "      <td>-0.006547</td>\n",
       "      <td>-0.004320</td>\n",
       "      <td>-0.008847</td>\n",
       "      <td>-0.000208</td>\n",
       "      <td>0.056074</td>\n",
       "      <td>0.016221</td>\n",
       "      <td>-0.012184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9172673334835262304</th>\n",
       "      <td>0.000305</td>\n",
       "      <td>0.002076</td>\n",
       "      <td>-0.000402</td>\n",
       "      <td>0.000506</td>\n",
       "      <td>-0.000003</td>\n",
       "      <td>0.000731</td>\n",
       "      <td>0.003484</td>\n",
       "      <td>-0.005040</td>\n",
       "      <td>0.001426</td>\n",
       "      <td>-0.002147</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000230</td>\n",
       "      <td>0.002130</td>\n",
       "      <td>0.006407</td>\n",
       "      <td>-0.000095</td>\n",
       "      <td>-0.000096</td>\n",
       "      <td>0.000024</td>\n",
       "      <td>0.002031</td>\n",
       "      <td>-0.000522</td>\n",
       "      <td>0.002588</td>\n",
       "      <td>-0.000774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9171475473795142532</th>\n",
       "      <td>0.017546</td>\n",
       "      <td>-0.007473</td>\n",
       "      <td>0.001472</td>\n",
       "      <td>0.001906</td>\n",
       "      <td>-0.000547</td>\n",
       "      <td>0.005783</td>\n",
       "      <td>-0.003028</td>\n",
       "      <td>-0.011424</td>\n",
       "      <td>0.014401</td>\n",
       "      <td>-0.002513</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001684</td>\n",
       "      <td>0.006940</td>\n",
       "      <td>-0.006419</td>\n",
       "      <td>0.000325</td>\n",
       "      <td>0.000214</td>\n",
       "      <td>0.004157</td>\n",
       "      <td>0.005238</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>0.007035</td>\n",
       "      <td>-0.002244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 949 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      -9223121837663643404  -9207251133131336884  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.042390              0.007184   \n",
       "-9216926795620865886              0.000881              0.007482   \n",
       "-9194572880052200111              0.012255             -0.015011   \n",
       "-9192549002213406534              0.020014             -0.004314   \n",
       "-9190737901804729417              0.002898              0.005858   \n",
       "-9189659052158407108              0.025061              0.003186   \n",
       "-9184137057748005562              0.001760              0.003935   \n",
       "-9176143510534135851              0.031526              0.019330   \n",
       "-9172673334835262304              0.000305              0.002076   \n",
       "-9171475473795142532              0.017546             -0.007473   \n",
       "\n",
       "                      -9199575329909162940  -9196668942822132778  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.021723             -0.010762   \n",
       "-9216926795620865886              0.000278             -0.000505   \n",
       "-9194572880052200111              0.005859              0.028180   \n",
       "-9192549002213406534             -0.007215              0.023871   \n",
       "-9190737901804729417              0.000592             -0.000404   \n",
       "-9189659052158407108             -0.002692              0.005258   \n",
       "-9184137057748005562              0.000089             -0.000263   \n",
       "-9176143510534135851              0.000523             -0.006001   \n",
       "-9172673334835262304             -0.000402              0.000506   \n",
       "-9171475473795142532              0.001472              0.001906   \n",
       "\n",
       "                      -9188188261933657343  -9172914609055320039  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.001325              0.011603   \n",
       "-9216926795620865886              0.000553              0.000355   \n",
       "-9194572880052200111              0.000821              0.028659   \n",
       "-9192549002213406534             -0.000737              0.014883   \n",
       "-9190737901804729417              0.000235              0.000644   \n",
       "-9189659052158407108              0.001853              0.000855   \n",
       "-9184137057748005562              0.000330              0.000632   \n",
       "-9176143510534135851             -0.001163              0.004800   \n",
       "-9172673334835262304             -0.000003              0.000731   \n",
       "-9171475473795142532             -0.000547              0.005783   \n",
       "\n",
       "                      -9156344805277471150  -9120685872592674274  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.051796             -0.015273   \n",
       "-9216926795620865886              0.000883             -0.000650   \n",
       "-9194572880052200111             -0.015218             -0.017560   \n",
       "-9192549002213406534              0.040538              0.009572   \n",
       "-9190737901804729417              0.000260             -0.000211   \n",
       "-9189659052158407108              0.018287              0.028799   \n",
       "-9184137057748005562             -0.000184              0.000512   \n",
       "-9176143510534135851              0.017766              0.084287   \n",
       "-9172673334835262304              0.003484             -0.005040   \n",
       "-9171475473795142532             -0.003028             -0.011424   \n",
       "\n",
       "                      -9109785559521267180  -9060214117327732109  ...  \\\n",
       "contentId                                                         ...   \n",
       "-9222795471790223670              0.004398             -0.001038  ...   \n",
       "-9216926795620865886              0.003010              0.000097  ...   \n",
       "-9194572880052200111              0.015087             -0.006885  ...   \n",
       "-9192549002213406534             -0.002032             -0.013772  ...   \n",
       "-9190737901804729417              0.005017             -0.000280  ...   \n",
       "-9189659052158407108              0.003002              0.003587  ...   \n",
       "-9184137057748005562              0.003389              0.000526  ...   \n",
       "-9176143510534135851              0.040455              0.023444  ...   \n",
       "-9172673334835262304              0.001426             -0.002147  ...   \n",
       "-9171475473795142532              0.014401             -0.002513  ...   \n",
       "\n",
       "                       9102085903669288476   9105269044962898535  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.044255             -0.004637   \n",
       "-9216926795620865886             -0.000298              0.001903   \n",
       "-9194572880052200111             -0.006691              0.004277   \n",
       "-9192549002213406534             -0.004392             -0.006671   \n",
       "-9190737901804729417              0.001010              0.001272   \n",
       "-9189659052158407108             -0.001910              0.016023   \n",
       "-9184137057748005562             -0.000284              0.000878   \n",
       "-9176143510534135851              0.000655              0.010236   \n",
       "-9172673334835262304              0.000230              0.002130   \n",
       "-9171475473795142532              0.001684              0.006940   \n",
       "\n",
       "                       9109075639526981934   9135582630122950040  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670             -0.028743              0.003642   \n",
       "-9216926795620865886              0.013378              0.000167   \n",
       "-9194572880052200111              0.036837              0.004882   \n",
       "-9192549002213406534              0.174971              0.002558   \n",
       "-9190737901804729417              0.014423              0.000139   \n",
       "-9189659052158407108              0.050675              0.000177   \n",
       "-9184137057748005562              0.010947             -0.000223   \n",
       "-9176143510534135851              0.107691             -0.006547   \n",
       "-9172673334835262304              0.006407             -0.000095   \n",
       "-9171475473795142532             -0.006419              0.000325   \n",
       "\n",
       "                       9137372837662939523   9148269800512008413  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670              0.005009             -0.001472   \n",
       "-9216926795620865886              0.000696              0.008339   \n",
       "-9194572880052200111              0.024364              0.147745   \n",
       "-9192549002213406534             -0.000142              0.043061   \n",
       "-9190737901804729417              0.000048              0.001243   \n",
       "-9189659052158407108              0.000173              0.095995   \n",
       "-9184137057748005562             -0.000589             -0.003847   \n",
       "-9176143510534135851             -0.004320             -0.008847   \n",
       "-9172673334835262304             -0.000096              0.000024   \n",
       "-9171475473795142532              0.000214              0.004157   \n",
       "\n",
       "                       9187866633451383747   9191849144618614467  \\\n",
       "contentId                                                          \n",
       "-9222795471790223670             -0.027778             -0.027499   \n",
       "-9216926795620865886              0.000899              0.006133   \n",
       "-9194572880052200111              0.017941             -0.030124   \n",
       "-9192549002213406534              0.014262             -0.003465   \n",
       "-9190737901804729417              0.001038              0.006060   \n",
       "-9189659052158407108              0.008093              0.012076   \n",
       "-9184137057748005562              0.000340              0.004369   \n",
       "-9176143510534135851             -0.000208              0.056074   \n",
       "-9172673334835262304              0.002031             -0.000522   \n",
       "-9171475473795142532              0.005238             -0.007142   \n",
       "\n",
       "                       9199170757466086545   9210530975708218054  \n",
       "contentId                                                         \n",
       "-9222795471790223670             -0.026514             -0.021112  \n",
       "-9216926795620865886              0.000848             -0.001094  \n",
       "-9194572880052200111              0.025064              0.158401  \n",
       "-9192549002213406534              0.030838              0.102175  \n",
       "-9190737901804729417              0.001626             -0.002482  \n",
       "-9189659052158407108              0.013978              0.038864  \n",
       "-9184137057748005562              0.001487             -0.000878  \n",
       "-9176143510534135851              0.016221             -0.012184  \n",
       "-9172673334835262304              0.002588             -0.000774  \n",
       "-9171475473795142532              0.007035             -0.002244  \n",
       "\n",
       "[10 rows x 949 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Converting the reconstructed matrix back to a Pandas dataframe\n",
    "cf_preds_df = pd.DataFrame(all_user_predicted_ratings, columns = users_items_pivot_matrix_df.columns, index=users_ids).transpose()\n",
    "cf_preds_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "949"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cf_preds_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CFRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Collaborative Filtering'\n",
    "    \n",
    "    def __init__(self, cf_predictions_df, items_df=None):\n",
    "        self.cf_predictions_df = cf_predictions_df\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "        \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        # Get and sort the user's predictions\n",
    "        sorted_user_predictions = self.cf_predictions_df[user_id].sort_values(ascending=False) \\\n",
    "                                    .reset_index().rename(columns={user_id: 'recStrength'})\n",
    "\n",
    "        # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
    "        recommendations_df = sorted_user_predictions[~sorted_user_predictions['contentId'].isin(items_to_ignore)] \\\n",
    "                               .sort_values('recStrength', ascending = False) \\\n",
    "                               .head(topn)\n",
    "\n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'contentId', \n",
    "                                                          right_on = 'contentId')[['recStrength', 'contentId', 'title', 'url', 'lang']]\n",
    "\n",
    "\n",
    "        return recommendations_df\n",
    "    \n",
    "cf_recommender_model = CFRecommender(cf_preds_df, articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Collaborative Filtering (SVD Matrix Factorization) model...\n",
      "947 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Collaborative Filtering', 'recall@5': 0.3377273925652518, 'recall@10': 0.471658317954126, 'precision@5': 0.026757739065045746, 'precision@10': 0.03736892676609433}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_person_id</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>precision@10</th>\n",
       "      <th>precision@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>recall@5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>47</td>\n",
       "      <td>20</td>\n",
       "      <td>192</td>\n",
       "      <td>0.465347</td>\n",
       "      <td>0.198020</td>\n",
       "      <td>0.244792</td>\n",
       "      <td>0.104167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2626634673110551643</td>\n",
       "      <td>42</td>\n",
       "      <td>28</td>\n",
       "      <td>134</td>\n",
       "      <td>0.415842</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>0.313433</td>\n",
       "      <td>0.208955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>-1032019229384696495</td>\n",
       "      <td>32</td>\n",
       "      <td>21</td>\n",
       "      <td>130</td>\n",
       "      <td>0.316832</td>\n",
       "      <td>0.207921</td>\n",
       "      <td>0.246154</td>\n",
       "      <td>0.161538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>-1443636648652872475</td>\n",
       "      <td>49</td>\n",
       "      <td>35</td>\n",
       "      <td>117</td>\n",
       "      <td>0.485149</td>\n",
       "      <td>0.346535</td>\n",
       "      <td>0.418803</td>\n",
       "      <td>0.299145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>-2979881261169775358</td>\n",
       "      <td>38</td>\n",
       "      <td>24</td>\n",
       "      <td>87</td>\n",
       "      <td>0.376238</td>\n",
       "      <td>0.237624</td>\n",
       "      <td>0.436782</td>\n",
       "      <td>0.275862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>-3596626804281480007</td>\n",
       "      <td>31</td>\n",
       "      <td>23</td>\n",
       "      <td>80</td>\n",
       "      <td>0.306931</td>\n",
       "      <td>0.227723</td>\n",
       "      <td>0.387500</td>\n",
       "      <td>0.287500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>1116121227607581999</td>\n",
       "      <td>37</td>\n",
       "      <td>22</td>\n",
       "      <td>73</td>\n",
       "      <td>0.366337</td>\n",
       "      <td>0.217822</td>\n",
       "      <td>0.506849</td>\n",
       "      <td>0.301370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>692689608292948411</td>\n",
       "      <td>31</td>\n",
       "      <td>28</td>\n",
       "      <td>69</td>\n",
       "      <td>0.306931</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>0.449275</td>\n",
       "      <td>0.405797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>3636910968448833585</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "      <td>68</td>\n",
       "      <td>0.227723</td>\n",
       "      <td>0.188119</td>\n",
       "      <td>0.338235</td>\n",
       "      <td>0.279412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>-9016528795238256703</td>\n",
       "      <td>29</td>\n",
       "      <td>16</td>\n",
       "      <td>68</td>\n",
       "      <td>0.287129</td>\n",
       "      <td>0.158416</td>\n",
       "      <td>0.426471</td>\n",
       "      <td>0.235294</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              _person_id  hits@10_count  hits@5_count  interacted_count  \\\n",
       "3    3609194402293569455             47            20               192   \n",
       "5   -2626634673110551643             42            28               134   \n",
       "37  -1032019229384696495             32            21               130   \n",
       "89  -1443636648652872475             49            35               117   \n",
       "200 -2979881261169775358             38            24                87   \n",
       "121 -3596626804281480007             31            23                80   \n",
       "127  1116121227607581999             37            22                73   \n",
       "80    692689608292948411             31            28                69   \n",
       "116  3636910968448833585             23            19                68   \n",
       "104 -9016528795238256703             29            16                68   \n",
       "\n",
       "     precision@10  precision@5  recall@10  recall@5  \n",
       "3        0.465347     0.198020   0.244792  0.104167  \n",
       "5        0.415842     0.277228   0.313433  0.208955  \n",
       "37       0.316832     0.207921   0.246154  0.161538  \n",
       "89       0.485149     0.346535   0.418803  0.299145  \n",
       "200      0.376238     0.237624   0.436782  0.275862  \n",
       "121      0.306931     0.227723   0.387500  0.287500  \n",
       "127      0.366337     0.217822   0.506849  0.301370  \n",
       "80       0.306931     0.277228   0.449275  0.405797  \n",
       "116      0.227723     0.188119   0.338235  0.279412  \n",
       "104      0.287129     0.158416   0.426471  0.235294  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Collaborative Filtering (SVD Matrix Factorization) model...')\n",
    "cf_global_metrics, cf_detailed_results_df = model_evaluator.evaluate_model(cf_recommender_model)\n",
    "print('\\nGlobal metrics:\\n%s' % cf_global_metrics)\n",
    "cf_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hybrid Recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, hybrid methods have performed better than individual approaches in many studies and have being extensively used by researchers and practioners.\n",
    "Let's build a simple hybridization method, by only add the CF score with the Content-Based score, and ranking by resulting score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HybridRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Hybrid'\n",
    "    \n",
    "    def __init__(self, cb_rec_model, cf_rec_model, items_df):\n",
    "        self.cb_rec_model = cb_rec_model\n",
    "        self.cf_rec_model = cf_rec_model\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "    \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        #Getting the top-1000 Content-based filtering recommendations\n",
    "        cb_recs_df = self.cb_rec_model.recommend_items(user_id, items_to_ignore=items_to_ignore, verbose=verbose,\n",
    "                                                           topn=1000).rename(columns={'recStrength': 'recStrengthCB'})\n",
    "        \n",
    "        #Getting the top-1000 Collaborative filtering recommendations\n",
    "        cf_recs_df = self.cf_rec_model.recommend_items(user_id, items_to_ignore=items_to_ignore, verbose=verbose, \n",
    "                                                           topn=1000).rename(columns={'recStrength': 'recStrengthCF'})\n",
    "        \n",
    "        #Combining the results by contentId\n",
    "        recs_df = cb_recs_df.merge(cf_recs_df,\n",
    "                                   how = 'inner', \n",
    "                                   left_on = 'contentId', \n",
    "                                   right_on = 'contentId')\n",
    "        \n",
    "        #Computing a hybrid recommendation score based on CF and CB scores\n",
    "        recs_df['recStrengthHybrid'] = (recs_df['recStrengthCB'] + recs_df['recStrengthCF'])/2\n",
    "        \n",
    "        #Sorting recommendations by hybrid score\n",
    "        recommendations_df = recs_df.sort_values('recStrengthHybrid', ascending=False).head(topn)\n",
    "        \n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'contentId', \n",
    "                                                          right_on = 'contentId')[['recStrengthHybrid', 'contentId', 'title', 'url', 'lang']]\n",
    "\n",
    "\n",
    "        return recommendations_df\n",
    "    \n",
    "hybrid_recommender_model = HybridRecommender(content_based_recommender_model, cf_recommender_model, articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Hybrid model...\n",
      "947 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Hybrid', 'recall@5': 0.4831268125494332, 'recall@10': 0.5705246506722911, 'precision@5': 0.03827756193340853, 'precision@10': 0.04520198855328571}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_person_id</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>precision@10</th>\n",
       "      <th>precision@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>recall@5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3609194402293569455</td>\n",
       "      <td>37</td>\n",
       "      <td>25</td>\n",
       "      <td>192</td>\n",
       "      <td>0.366337</td>\n",
       "      <td>0.247525</td>\n",
       "      <td>0.192708</td>\n",
       "      <td>0.130208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2626634673110551643</td>\n",
       "      <td>42</td>\n",
       "      <td>23</td>\n",
       "      <td>134</td>\n",
       "      <td>0.415842</td>\n",
       "      <td>0.227723</td>\n",
       "      <td>0.313433</td>\n",
       "      <td>0.171642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>-1032019229384696495</td>\n",
       "      <td>32</td>\n",
       "      <td>28</td>\n",
       "      <td>130</td>\n",
       "      <td>0.316832</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>0.246154</td>\n",
       "      <td>0.215385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>-1443636648652872475</td>\n",
       "      <td>49</td>\n",
       "      <td>43</td>\n",
       "      <td>117</td>\n",
       "      <td>0.485149</td>\n",
       "      <td>0.425743</td>\n",
       "      <td>0.418803</td>\n",
       "      <td>0.367521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>-2979881261169775358</td>\n",
       "      <td>32</td>\n",
       "      <td>21</td>\n",
       "      <td>87</td>\n",
       "      <td>0.316832</td>\n",
       "      <td>0.207921</td>\n",
       "      <td>0.367816</td>\n",
       "      <td>0.241379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>-3596626804281480007</td>\n",
       "      <td>33</td>\n",
       "      <td>25</td>\n",
       "      <td>80</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>0.247525</td>\n",
       "      <td>0.412500</td>\n",
       "      <td>0.312500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>1116121227607581999</td>\n",
       "      <td>24</td>\n",
       "      <td>17</td>\n",
       "      <td>73</td>\n",
       "      <td>0.237624</td>\n",
       "      <td>0.168317</td>\n",
       "      <td>0.328767</td>\n",
       "      <td>0.232877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>692689608292948411</td>\n",
       "      <td>30</td>\n",
       "      <td>22</td>\n",
       "      <td>69</td>\n",
       "      <td>0.297030</td>\n",
       "      <td>0.217822</td>\n",
       "      <td>0.434783</td>\n",
       "      <td>0.318841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>3636910968448833585</td>\n",
       "      <td>21</td>\n",
       "      <td>13</td>\n",
       "      <td>68</td>\n",
       "      <td>0.207921</td>\n",
       "      <td>0.128713</td>\n",
       "      <td>0.308824</td>\n",
       "      <td>0.191176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>-9016528795238256703</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>68</td>\n",
       "      <td>0.247525</td>\n",
       "      <td>0.198020</td>\n",
       "      <td>0.367647</td>\n",
       "      <td>0.294118</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              _person_id  hits@10_count  hits@5_count  interacted_count  \\\n",
       "3    3609194402293569455             37            25               192   \n",
       "5   -2626634673110551643             42            23               134   \n",
       "37  -1032019229384696495             32            28               130   \n",
       "89  -1443636648652872475             49            43               117   \n",
       "200 -2979881261169775358             32            21                87   \n",
       "121 -3596626804281480007             33            25                80   \n",
       "127  1116121227607581999             24            17                73   \n",
       "80    692689608292948411             30            22                69   \n",
       "116  3636910968448833585             21            13                68   \n",
       "104 -9016528795238256703             25            20                68   \n",
       "\n",
       "     precision@10  precision@5  recall@10  recall@5  \n",
       "3        0.366337     0.247525   0.192708  0.130208  \n",
       "5        0.415842     0.227723   0.313433  0.171642  \n",
       "37       0.316832     0.277228   0.246154  0.215385  \n",
       "89       0.485149     0.425743   0.418803  0.367521  \n",
       "200      0.316832     0.207921   0.367816  0.241379  \n",
       "121      0.326733     0.247525   0.412500  0.312500  \n",
       "127      0.237624     0.168317   0.328767  0.232877  \n",
       "80       0.297030     0.217822   0.434783  0.318841  \n",
       "116      0.207921     0.128713   0.308824  0.191176  \n",
       "104      0.247525     0.198020   0.367647  0.294118  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Hybrid model...')\n",
    "hybrid_global_metrics, hybrid_detailed_results_df = model_evaluator.evaluate_model(hybrid_recommender_model)\n",
    "print('\\nGlobal metrics:\\n%s' % hybrid_global_metrics)\n",
    "hybrid_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing the methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision@10</th>\n",
       "      <th>precision@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>recall@5</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>modelName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Popularity</th>\n",
       "      <td>0.030173</td>\n",
       "      <td>0.019551</td>\n",
       "      <td>0.380833</td>\n",
       "      <td>0.246770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Collaborative Filtering</th>\n",
       "      <td>0.037369</td>\n",
       "      <td>0.026758</td>\n",
       "      <td>0.471658</td>\n",
       "      <td>0.337727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Content-Based</th>\n",
       "      <td>0.042361</td>\n",
       "      <td>0.034069</td>\n",
       "      <td>0.534669</td>\n",
       "      <td>0.430003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hybrid</th>\n",
       "      <td>0.045202</td>\n",
       "      <td>0.038278</td>\n",
       "      <td>0.570525</td>\n",
       "      <td>0.483127</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         precision@10  precision@5  recall@10  recall@5\n",
       "modelName                                                              \n",
       "Popularity                   0.030173     0.019551   0.380833  0.246770\n",
       "Collaborative Filtering      0.037369     0.026758   0.471658  0.337727\n",
       "Content-Based                0.042361     0.034069   0.534669  0.430003\n",
       "Hybrid                       0.045202     0.038278   0.570525  0.483127"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "global_metrics_df = pd.DataFrame([pop_global_metrics, cf_global_metrics, cb_global_metrics, hybrid_global_metrics]) \\\n",
    "                        .set_index('modelName')\n",
    "global_metrics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our simple hybrid approach surpasses Content-Based filtering with its combination with Collaborative Filtering. Now we have a Recall@5 of 48% and Recall@10 of 57%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAIPCAYAAADguTYvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XtUV1X+//HXlouoeMFIK8DQwQtyEfSDVohhllomZekEmnkZl9WY5tjN+X3LrHFGK0eN0OW370zpzKToeAmb0iZn5KtWXqBQEXTQwIDM1MoLqMCH8/tD/XwlMVARjvB8rOWazzl7f/Z+HwbX8tU+Zx9jWZYAAAAAAPbUqK4LAAAAAABcGqENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYmHtdTezr62sFBgbW1fQAAAAAUKfS09OPWJZ1Y1X96iy0BQYGKi0tra6mBwAAAIA6ZYw5UJ1+3B4JAAAAADZGaAMAAAAAGyO0AQAAAICN1dkzbQAAAMD1oLS0VAUFBTp9+nRdl4LrlJeXl/z9/eXh4XFF3ye0AQCABm/dunV6+umn5XQ6NW7cOE2dOrVC+6JFi/Tcc8/Jz89PkvTUU09p3Lhx2rBhg37zm9+4+u3Zs0fJycl68MEHlZSUpHnz5mn//v06fPiwfH19a/WaUHMKCgrUvHlzBQYGyhhT1+XgOmNZlo4ePaqCggK1b9/+isYgtAEAgAbN6XRqwoQJ+uSTT+Tv76+oqCjFxcWpa9euFfo98sgjSkpKqnCub9++ysjIkCR9//33CgoKUv/+/SVJ0dHRuv/++xUbG1sr14Fr5/Tp0wQ2XDFjjG644QYdPnz4isfgmTYAANCgbdu2TUFBQerQoYM8PT0VHx+vlJSUyx5nxYoVuvfee9W0aVNJUmRkpHgnbf1BYMPVuNrfH0IbAABo0AoLCxUQEOA69vf3V2Fh4UX9Vq5cqfDwcA0dOlT5+fkXtScnJyshIeGa1gqgYSK0AQAAVGHw4MHKy8vTzp07dc8992jUqFEV2g8ePKhdu3ZpwIABdVQhcG0EBgbqyJEj1e5jjNEzzzzjaps9e7amT59+LUtsEAhtAACgQfPz86uwclZQUODacOS8G264QY0bN5YkjRs3Tunp6RXaly9friFDhlzxznBAfdG4cWOtWrWqyqCHy0NoAwAADVpUVJRycnKUm5urkpISJScnKy4urkKfgwcPuj6vWbNGwcHBFdqXLl3KrZGwjby8PHXp0kWjR49Wp06dNGLECK1fv17R0dHq2LGjtm3bpu+//14PPvigwsPDddttt2nnzp2SpKNHj6p///4KCQnRuHHjZFmWa9y//e1v6tmzpyIiIvT444/L6XReNLe7u7vGjx+vuXPnXtT2wQcfqFevXoqMjNTdd9+tQ4cOSZKmT5+uUaNGKSYmRrfeeqtWrVql559/XmFhYRo4cKBKS0slSenp6brzzjvVo0cPDRgwoMLfy/qO0AYAABo0d3d3JSUlacCAAQoODtYvf/lLhYSEaNq0aVqzZo0kKTExUSEhIerWrZsSExO1aNEi1/fz8vKUn5+vO++8s8K4iYmJ8vf3V0FBgcLDwzVu3LjavCw0cPv27dMzzzyjPXv2aM+ePVqyZIk2b96s2bNn6w9/+INefvllRUZGaufOnfrDH/6gxx57TJL0yiuvqHfv3tq9e7eGDBmir7/+WpKUnZ2tZcuW6dNPP1VGRobc3Nz03nvvVTr3hAkT9N577+nYsWMVzvfu3VtbtmzRl19+qfj4eL3++uuutv379+vf//631qxZo0cffVR9+/bVrl271KRJE3344YcqLS3VxIkTtWLFCqWnp2vs2LH6r//6r2v007MftvwHAAAN3n333af77ruvwrlXX33V9XnmzJmaOXNmpd8NDAysdOOSSZMmadKkSTVbKFBN7du3V1hYmCQpJCRE/fr1kzFGYWFhysvL04EDB7Ry5UpJ0l133aWjR4/q+PHj2rhxo1atWiVJGjRokHx8fCRJ//rXv5Senq6oqChJ0qlTp9SmTZtK527RooUee+wxJSYmqkmTJq7zBQUFeuSRR3Tw4EGVlJRUeGfZvffeKw8PD4WFhcnpdGrgwIGS5Kp37969yszM1D333CPp7Ks6br755pr8kdkaoQ0AAACoZ84/gylJjRo1ch03atRIZWVll/38pWVZGjVq1CX/48VPTZ48Wd27d9eYMWNc5yZOnKgpU6YoLi5OqampFTYoubA+Dw8P1xb55+u1LEshISH6/PPPL6vu+oLbIwEAAIAGJiYmxnV7Y2pqqnx9fdWiRQv16dNHS5YskSStXbtWP/zwgySpX79+WrFihb777jtJZ18mf+DAgUuO37p1a/3yl7/Un//8Z9e5Y8eOuTb5Wbx48WXV27lzZx0+fNgV2kpLS7V79+7LGuN6RmgDAAAAGpjp06crPT1d4eHhmjp1qitEvfzyy9q4caNCQkK0atUqtWvXTpLUtWtXzZgxQ/3791d4eLjuueeeKjcCeeaZZyrsIjl9+nQNGzZMPXr0kK+v72XV6+npqRUrVuiFF15Qt27dFBERoc8+++wyr/r6ZS7cEaY2ORwOKy0trU7mBgAAAKorOzv7oh1DgctV2e+RMSbdsixHVd9lpQ0AAAAAbIzQBgAAAAA2xu6RAACgQcvuUnu3vQXvya61uQDUH6y0AQAAAICNEdoAAAAAwMYIbQAAAABgYzzTBgAAAFyGwKkf1uh4ebMGVdnHzc1NYWFhKisrU3BwsBYvXqymTZvWWA2LFi1SWlqakpKSqv2dtLQ0/eUvf1FiYqJSU1Pl6empO+64o8Zqwv9hpQ0AAACwuSZNmigjI0OZmZny9PTUwoUL67SesrIyORwOJSYmSpJSU1Mb1MuuaxuhDQAAALiOxMTEaN++fZKkOXPmKDQ0VKGhoZo3b54kKS8vT126dNGIESMUHBysoUOHqri4WJIUGBioI0eOSDq7UhYbG3vR+B988IF69eqlyMhI3X333Tp06JAkafr06Ro5cqSio6M1cuRIpaam6v7771deXp4WLlyouXPnKiIiQps2bVL79u1VWloqSTp+/HiFY1w+QhsAAABwnSgrK9PatWsVFham9PR0vfvuu9q6dau2bNmi//mf/9GXX34pSdq7d69+/etfKzs7Wy1atNCCBQuqPUfv3r21ZcsWffnll4qPj9frr7/uasvKytL69eu1dOlS17nAwEA98cQT+s1vfqOMjAzFxMQoNjZWH3549jbS5ORkPfTQQ/Lw8Kihn0LDQ2gDAAAAbO7UqVOKiIiQw+FQu3bt9Ktf/UqbN2/WkCFD1KxZM3l7e+uhhx7Spk2bJEkBAQGKjo6WJD366KPavHlztecqKCjQgAEDFBYWpjfeeEO7d+92tcXFxalJkyZVjjFu3Di9++67kqR3331XY8aMuZzLxU+wEQkAAABgc+efaasuY0ylx+7u7iovL5cknT59utLvTpw4UVOmTFFcXJxSU1M1ffp0V1uzZs2qNX90dLTy8vKUmpoqp9Op0NDQateOi7HSBgAAAFyHYmJi9P7776u4uFhFRUVavXq1YmJiJElff/21Pv/8c0nSkiVL1Lt3b0lnb2VMT0+XJK1cubLScY8dOyY/Pz9J0uLFi6tVS/PmzXXixIkK5x577DENHz6cVbYawEobAACwrXXr1unpp5+W0+nUuHHjNHXq1ArtixYt0nPPPef6B+ZTTz2lcePG6cCBAxoyZIjKy8tVWlqqiRMn6oknnpAkxcbG6uDBg65bvJLKynSDO/8kQvVVZ4v+2tC9e3eNHj1aPXv2lHT2lsTIyEjl5eWpc+fOmj9/vsaOHauuXbvqySeflCS9/PLL+tWvfqWXXnqp0k1IpLMbjgwbNkw+Pj666667lJubW2UtgwcP1tChQ5WSkqK33npLMTExGjFihF588UUlJCTU2DU3VMayrDqZ2OFwWGlpaXUyNwAAsD+n06lOnTrpk08+kb+/v6KiorR06VJ17drV1edS75YqKSmRZVlq3LixTp48qdDQUH322We65ZZbFBsbq9mzZ8vhcEiSsrsE19o1Be/JrrW5UHOys7MVHFx7vydXKy8vT/fff78yMzPrtI4VK1YoJSVFf/3rX+u0Druo7PfIGJNuWZajqu/yn5UAAIAtbdu2TUFBQerQoYMkKT4+XikpKRVC26V4enq6Pp85c8b1DA+A2jFx4kStXbtWH330UV2XUi/wTBsAALClwsJCBQQEuI79/f1VWFh4Ub+VK1cqPDxcQ4cOVX5+vut8fn6+wsPDFRAQoBdeeEG33HKLq23MmDGKiIjQ7373O9XVXUfAtRIYGFjnq2xvvfWW9u3bp06dOtVpHfUFoQ0AAFy3Bg8erLy8PO3cuVP33HOPRo0a5WoLCAjQzp07tW/fPi1evNj1guD33ntPu3bt0qZNm7Rp0yatOX68rsoHgGohtAEAAFvy8/OrsHJWUFDg2nDkvBtuuEGNGzeWdHYThvO74l3olltuUWhoqOv9VefHaN68uYYPH65dp09dq0sAgBpBaAMAALYUFRWlnJwc5ebmqqSkRMnJyYqLi6vQ5+DBg67Pa9ascT3kX1BQoFOnzoaxH374QZs3b1bnzp1VVlamI0eOSJJKS0v1j3/8Q0HnQh8A2BUbkQAAAFtyd3dXUlKSBgwYIKfTqbFjxyokJETTpk2Tw+FQXFycEhMTtWbNGrm7u6t169ZatGiRpLO7tD3zzDMyxsiyLD377LMKCwtTUVGRBgwYoNLSUjmdTt19990a1rJV3V4oAFSBLf8BAECDxpb/qMpFW7VPb1mzE0w/VmWXb7/9VpMnT9b27dvVqlUrtW3bVvPmzbvkRh8Xbvufmpqq2bNn6x//+Mclx7/U6zNqwrx58zR+/Hg1bdpUknTfffdpyZIlatXq6v6DSWBgoJo3by43NzdJ0oIFCxQYGKhJkyZpxYoVFa47NTVVnp6euuOOOy5rjrS0NP3lL39RYmLiVdUqseU/AAAAUG9ZlqUhQ4Zo1KhRSk5OliTt2LFDhw4dssXujJZlybIsNWpU+ZNX8+bN06OPPuoKbTX5GoANGzbI19e3wrkVK1Zc1C81NVXe3t6XFdrKysrkcDhc73SsSzzTBgAAANjYhg0b5OHhoSeeeMJ1rlu3boqJiZFlWXruuecUGhqqsLAwLVu27GfH2rZtm26//XZFRkbqjjvu0N69e11t+fn5io2NVceOHfXKK6+4zs+ZM0ehoaEKDQ3VvHnzJJ1dyevcubMee+wxhYaGKj8/X08++aQcDodCQkL08ssvS5ISExP1zTffqG/fvurbt6+ksytkR44c0dSpUzV//nzXPNOnT9fs2bMlSW+88YaioqIUHh7uGqs68vLyFBoaetG5hQsXau7cuYqIiNCmTZt0+PBhPfzww4qKilJUVJQ+/fRTVw0jR45UdHS0Ro4cqdTUVN1///2utrFjxyo2NlYdOnSosPr2u9/9Tp07d1bv3r2VkJDguo6awkobAAAAYGOZmZnq0aNHpW2rVq1SRkaGduzYoSNHjigqKkp9+vS55FhdunTRpk2b5O7urvXr1+v//b//p5UrV0o6G+gyMzPVtGlTRUVFadCgQTLG6N1339XWrVtlWZZ69eqlO++8Uz4+PsrJydHixYt12223SZJ+//vfq3Xr1nI6nerXr5927typSZMmac6cOZWuiD3yyCOaPHmyJkyYIElavny5Pv74Y/3zn/9UTk6Otm3bJsuyFBcXp40bN1Z6XX379pWbm5saN26srVu3VnrNgYGBeuKJJ+Tt7a1nn31WkjR8+HD95je/Ue/evfX1119rwIABys4+e/tyVlaWNm/erCZNmig1NbXCWHv27NGGDRt04sQJde7cWU8++aQyMjK0cuVK7dixQ6Wlperevfsl//+6UoQ2AAAA4Dq1efNmJSQkyM3NTW3bttWdd96p7du3Kzw8vNL+x44d06hRo5STkyNjjEpLS11t99xzj2644QZJ0kMPPaTNmzfLGKMhQ4aoWbNmrvObNm1SXFycbr31Vldgk86GrrfffltlZWU6ePCgsrKyLlmHJEVGRuq7777TN998o8OHD8vHx0cBAQF688039c9//lORkZGSpJMnTyonJ6fS0FZZGKyO9evXKysry3V8/PhxnTx5UpIUFxenJk2aVPq9QYMGqXHjxmrcuLHatGmjQ4cO6dNPP9UDDzwgLy8veXl5afDgwZddT1UIbQAAAICNhYSEVPqc1pV46aWX1LdvX61evVp5eXmKjY11tRljKvT96fFPnQ9ykpSbm6vZs2dr+/bt8vHx0ejRo3X69Okq6xk2bJhWrFihb7/9Vo888oiks8/I/fa3v9Xjjz9+GVd2ecrLy7VlyxZ5eXld1Hbhdf1U4wteEeLm5qaysrJrUt9P8UwbAAAAYGN33XWXzpw5o7ffftt1bufOndq0aZNiYmK0bNkyOZ1OHT58WBs3blTPnj0vOdaxY8dcL5g//4qM8z755BN9//33OnXqlN5//31FR0crJiZG77//voqLi1VUVKTVq1crJibmonGPHz+uZs2aqWXLljp06JDWrl3ramvevLlOnDhRaT2PPPKIkpOTtWLFCg0bNkySNGDAAL3zzjuula/CwkJ999131fthXcJPa+jfv7/eeust13FGRsYVjx0dHa0PPvhAp0+f1smTJ392l84rxUobAACwpbDFYbUyz/JamQX1SjW26K9JxhitXr1akydP1muvvSYvLy8FBgZq3rx56t27tz7//HN169ZNxhi9/vrruummm5SXl1fpWM8//7xGjRqlGTNmaNCgQRXaevbsqYcfflgFBQV69NFHXbsmjh492hUEx40bp8jIyIvG79atmyIjI9WlSxcFBAQoOjra1TZ+/HgNHDhQt9xyizZs2FDheyEhITpx4oT8/Px08803SzobqLKzs3X77bdLkry9vfW3v/1Nbdq0ueKf4eDBgzV06FClpKTorbfeUmJioiZMmKDw8HCVlZWpT58+Wrhw4RWNHRUVpbi4OIWHh6tt27YKCwtTy5Y1+1oI3tMGAABsqdZC28zaub1J4j1t16vK3q8FXOjkyZPy9vZWcXGx+vTpo7ffflvdu3ev0If3tAEAAABAHRk/fryysrJ0+vRpjRo16qLAdrUIbQAAAABwFZYsWXJNx2cjEgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2xkYkAAAAwGWo6ddR7Bq1q8o+3377rSZPnqzt27erVatWatu2rebNm6dOnTpd1lzz5s3T+PHj1bRp0yuqNTU1VZ6enrrjjjsqbQ8MDFTz5s3l5uYmp9OpGTNm6IEHHriiuarD29vb9RLu+oyVNgAAAMDGLMvSkCFDFBsbq/379ys9PV0zZ87UoUOHLnusefPmqbi4+IprSU1N1WefffazfTZs2KCMjAytWLFCkyZNuuK58H8IbQAAAICNbdiwQR4eHnriiSdc57p166bevXvrueeeU2hoqMLCwrRs2TJJZ4NVbGyshg4dqi5dumjEiBGyLEuJiYn65ptv1LdvX/Xt21eS9M9//lO33367unfvrmHDhrlWrQIDA/Xyyy+re/fuCgsL0549e5SXl6eFCxdq7ty5ioiI0KZNm3627uPHj8vHx8d1/OCDD6pHjx4KCQnR22+/LUlyOp0aPXq06xrmzp0rSdq/f78GDhyoHj16KCYmRnv27JEk5ebm6vbbb1dYWJhefPHFGvoJ2x+3RwIAAAA2lpmZqR49elx0ftWqVcrIyNCOHTt05MgRRUVFqU+fPpKkL7/8Urt379Ytt9yi6Ohoffrpp5o0aZLmzJmjDRs2yNfXV0eOHNGMGTO0fv16NWvWTK+99prmzJmjadOmSZJ8fX31xRdfaMGCBZo9e7b+9Kc/6YknnpC3t7eeffbZS9bbt29fWZalr776SsuXL3edf+edd9S6dWudOnVKUVFRevjhh5WXl6fCwkJlZmZKkn788UdJZ19WvXDhQnXs2FFbt27Vr3/9a/373//W008/rSeffFKPPfaY5s+fX2M/Y7sjtAEAAADXoc2bNyshIUFubm5q27at7rzzTm3fvl0tWrRQz5495e/vL0mKiIhQXl6eevfuXeH7W7ZsUVZWlqKjoyVJJSUluv32213tDz30kCSpR48eWrVqVbXrOh8K9+/fr379+ik2Nlbe3t5KTEzU6tWrJUn5+fnKyclR586d9dVXX2nixIkaNGiQ+vfvr5MnT+qzzz7TsGHDXGOeOXNGkvTpp59q5cqVkqSRI0fqhRdeuNwf23WJ0AYAAADYWEhIiFasWHFZ32ncuLHrs5ubm8rKyi7qY1mW7rnnHi1duvRnx7jU951Op2sFMC4uTq+++mqF9l/84hdq27atsrKyVFxcrPXr1+vzzz9X06ZNFRsbq9OnT8vHx0c7duzQxx9/rIULF2r58uWaN2+eWrVqpYyMjErrMsZU74dQj/BMGwAAAGBjd911l86cOeN6DkySdu7cqVatWmnZsmVyOp06fPiwNm7cqJ49e/7sWM2bN9eJEyckSbfddps+/fRT7du3T5JUVFSk//znP9X+vpubmzIyMpSRkXFRYJOk7777Trm5ubr11lt17Ngx+fj4qGnTptqzZ4+2bNkiSTpy5IjKy8v18MMPa8aMGfriiy/UokULtW/fXn//+98lnQ2XO3bskCRFR0crOTlZkvTee+9V+bOrL1hpAwAAAC5Ddbbor0nGGK1evVqTJ0/Wa6+9Ji8vLwUGBmrevHk6efKkunXrJmOMXn/9dd10002uTTsqM378eA0cOFC33HKLNmzYoEWLFikhIcF1++GMGTN+9jUCgwcP1tChQ5WSkqK33npLMTExF/Xp27ev3NzcVFpaqlmzZqlt27YaOHCgFi5cqODgYHXu3Fm33XabJKmwsFBjxoxReXm5JGnmzJmSzgayJ598UjNmzFBpaani4+PVrVs3vfnmmxo+fLhee+21a/oqAbsxlmXVycQOh8NKS0urk7kBAID91fS7sC5l+cyLb/u6VoL3ZNfaXKg52dnZCg4OrusycJ2r7PfIGJNuWZajqu9yeyQAAAAA2BihDQAAAABsjNAGAAAAADZGaAMAAAAAGyO0AQAAAICNVSu0GWMGGmP2GmP2GWOmVtI+2hhz2BiTce7PuJovFQAAAAAanirf02aMcZM0X9I9kgokbTfGrLEsK+snXZdZlvXUNagRAAAAsI3sLjW7/X91XgXh7e2tkydPuo4XLVqktLQ0JSUlXfI7sbGxmj17thyOn99RPi0tTX/5y1+UmJh4UVtgYKDS0tLk6+v7s2McO3ZM+fn5sixLvr6+uvnmmyvt98MPP2j//v0KDg5Ws2bNVF5ergMHDqi4uFiWZemGG27QzTffrPLycu3du1fl5eWyLEs+Pj7y8/P72Rrqs+q8XLunpH2WZX0lScaYZEkPSPppaAMAAABwHSkrK5PD4agy2P0cy7L09ddfq1OnTvLw8FB2drZatWqlJk2aVOjndDp16NAhNWvWzHXuhx9+UHl5uUJCQuR0OrV79261bt1anp6e6tSpk9zc3FwBrmXLlvL29r7iOq9n1bk90k9S/gXHBefO/dTDxpidxpgVxpiAygYyxow3xqQZY9IOHz58BeUCAAAAOO/EiRNq3769SktLJUnHjx+vcPzXv/5VERERCg0N1bZt2yRJ06dP18iRIxUdHa2RI0cqNTVV999/vyTp6NGj6t+/v0JCQjRu3DhZllVlDUVFRWrcuLEaN26sRo0aqXXr1vrxxx8v6ldYWKibbrpJjRpVjCDnV9Msy5IxRm5ubq7/leRqa8hqaiOSDyQFWpYVLukTSYsr62RZ1tuWZTksy3LceOONNTQ1AAAAUL+dOnVKERERrj/Tpk2TJDVv3lyxsbH68MMPJUnJycl66KGH5OHhIUkqLi5WRkaGFixYoLFjx7rGy8rK0vr167V06dIK87zyyivq3bu3du/erSFDhujrr7+usraSkhJ5enq6jj09PV2h8byioiKVlJSoVatWFc77+PioUaNG2rFjh3bu3KmbbrpJ7u5nbwa0LEu7d+/Wjh071KJFiwa7yiZVL7QVSrpw5cz/3DkXy7KOWpZ15tzhnyT1qJnyAAAAADRp0kQZGRmuP6+++qqrbdy4cXr33XclSe+++67GjBnjaktISJAk9enTR8ePH3etgMXFxV10+6Ikbdy4UY8++qgkadCgQfLx8bnq2i3LUkFBgQICLr4Zr7i4WMYYhYeHKywsTIcOHdKZM2djhTFGISEhCg8PV1FRkU6dOnXVtVyvqhPatkvqaIxpb4zxlBQvac2FHYwxFz5pGCep6qcpAQAAAFy16Oho5eXlKTU1VU6nU6Ghoa42Y0yFvuePL3yu7Gp5enqqpKTEdVxSUuJa6ZPOPst26tQp7d27Vzt37tTJkye1b98+FRUV6ejRo2rZsqUaNWokDw8PeXt7q6ioqML47u7uat68uY4dO1ZjNV9vqgxtlmWVSXpK0sc6G8aWW5a12xjzqjEm7ly3ScaY3caYHZImSRp9rQoGAAAAUNFjjz2m4cOHV1hlk6Rly5ZJkjZv3qyWLVuqZcuWPztOnz59tGTJEknS2rVr9cMPP1Q5d7NmzXTmzBmdOXNG5eXl+v777yvcBunu7q6IiAiFh4crPDxc3t7eCgoKUrNmzeTp6anjx49LOhvuTp48KS8vL5WWlqqsrEzS2Wfejh8/Li8vr+r/QOqZ6uweKcuyPpL00U/OTbvg828l/bZmSwMAAADspzpb9Ne2ESNG6MUXX3TdDnmel5eXIiMjVVpaqnfeeafKcV5++WUlJCQoJCREd9xxh9q1a1fld4wxateunf7zn/9Iknx9fdWkSRMVFhaqWbNmFz3HdqE2bdooLy9PmZmZru82bdpUxcXFysvLc21A4uPj87Pj1HemrnZicTgcVlpaWp3MDQAArty6dev09NNPy+l0aty4cZo6dWql/VauXKmhQ4dq+/btcjgceu+99/TGG2+42nfu3KkvvvhCnTp10rBhw7R//365ublp8ODBmjVrlsJGFdjmAAAgAElEQVQWh9XK9SyfWVYr80j2/Mc+qpadna3g4Jp9N1tNW7FihVJSUvTXv/61rkvBJVT2e2SMSbcsq8r3LVRrpQ0AAEA6e/vShAkT9Mknn8jf319RUVGKi4tT165dK/Q7ceKE3nzzTfXq1ct1bsSIERoxYoQkadeuXXrwwQcVERGh4uJiPfvss+rbt69KSkrUr18/rV27tlavC7ieTZw4UWvXrtVHH31UdWdcl2pqy38AANAAbNu2TUFBQerQoYM8PT0VHx+vlJSUi/q99NJLeuGFFy75DMrSpUsVHx8vSWratKn69u0r6eyGBt27d1dBQcG1uwignnnrrbe0b98+derUqa5LwTVCaAMAANVWWFhYYdtuf39/FRZWeBOQvvjiC+Xn52vQoEGXHGfZsmUXPXsjST/++KM++OAD9evXr+aKBmpAQ3+5M67O1f7+ENoAAECNKS8v15QpU/THP/7xkn22bt2qpk2bVtiWXJLKysqUkJCgSZMmqUOHDte6VKDavLy8dPToUYIbrohlWTp69OhV7X7JM20AAKDa/Pz8lJ+f7zouKCiQn5+f6/jEiRPKzMxUbGysJOnbb79VXFyc1qxZI4fj7LP2ycnJla6yjR8/Xh07dtTkyZOv7UUAl8nf318FBQU6fPhwXZeC65SXl5f8/f2v+PuENgAAUG1RUVHKyclRbm6u/Pz8lJyc7HqnkyS1bNlSR44ccR3HxsZq9uzZrsBWXl6u5cuXa9OmTRXGffHFF3Xs2DH96U9/qp0LAS6Dh4eH2rdvX9dl2Fp2l9rZXbOh7sBKaAMAANXm7u6upKQkDRgwQE6nU2PHjlVISIimTZsmh8OhuLi4n/3+xo0bFRAQUOH2x4KCAv3+979Xly5d1L17d0nSU089JXlc00sBgOsG72kDAAC2xHvagOsHK21XprrvaWMjEgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABtjy38AAHB5presnXnat6udeQDA5lhpAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAADUU+vWrVPnzp0VFBSkWbNmXbLfypUrZYxRWlqaJGnbtm2KiIhQRESEunXrptWrV1/2mKg57nVdAAAAAICa53Q6NWHCBH3yySfy9/dXVFSU4uLi1LVr1wr9Tpw4oTfffFO9evVynQsNDVVaWprc3d118OBBdevWTYMHD5YxptIxTW1fXAPDShsAAABQD23btk1BQUHq0KGDPD09FR8fr5SUlIv6vfTSS3rhhRfk5eXlOte0aVO5u59d3zl9+rSMMZc1JmoWoQ0AAACohwoLCxUQEOA69vf3V2FhYYU+X3zxhfLz8zVo0KCLvr9161aFhIQoLCxMCxculLu7e7XGRM0jtAEAAAANUHl5uaZMmaI//vGPlbb36tVLu3fv1vbt2zVz5kydPn26livEeYQ2AAAAoB7y8/NTfn6+67igoEB+fn6u4xMnTigzM1OxsbEKDAzUli1bFBcX59qM5Lzg4GB5e3srMzOzyjFxbRDaAAAAgHooKipKOTk5ys3NVUlJiZKTkxUXF+dqb9mypY4cOaK8vDzl5eXptttu05o1a+RwOJSbm6uysjJJ0oEDB7Rnzx4FBgZWOSauDXaPBAAAAOohd3d3JSUlacCAAXI6nRo7dqxCQkI0bdo0ORyOnw1bmzdv1qxZs+Th4aFGjRppwYIF8vX1laRKx8yurYtqoIxlWXUyscPhsH669AoAAK4D01vWyjRh7dvVyjzLZ5bVyjySFLyHf9qifsruElwr89S3v0PGmHTLshxV9eP2SAAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjfGeNgAAAKAeClscVmtzLa+1mRomVtoAAAAAwMYIbQAAAABgY4Q2AAAAALAxQhsAAAAA2BihDQAAAABsjNAGAAAAADZGaAMAAAAAGyO0AQAAAICNEdoAAAAAwMYIbQAAAABgY4Q2AAAAALAxQhsAAAAA2BihDQAAAABsjNAGAAAAADZGaAMAAAAAGyO0AQAAAICNEdoAAAAAwMaqFdqMMQONMXuNMfuMMVN/pt/DxhjLGOOouRIBAAAAoOGqMrQZY9wkzZd0r6SukhKMMV0r6ddc0tOSttZ0kQAAAADQUFVnpa2npH2WZX1lWVaJpGRJD1TS73eSXpN0ugbrAwAAAIAGrTqhzU9S/gXHBefOuRhjuksKsCzrw58byBgz3hiTZoxJO3z48GUXCwAALm3dunXq3LmzgoKCNGvWrIvaFy5cqLCwMEVERKh3797KysqSJJWWlmrUqFEKCwtTcHCwZs6c6frO2LFj1aZNG4WGhtbadQAAKrrqjUiMMY0kzZH0TFV9Lct627Ish2VZjhtvvPFqpwYAAOc4nU5NmDBBa9euVVZWlpYuXeoKZecNHz5cu3btUkZGhp5//nlNmTJFkvT3v/9dZ86c0a5du5Senq7//u//Vl5eniRp9OjRWrduXW1fDgDgAtUJbYWSAi449j937rzmkkIlpRpj8iTdJmkNm5EAAFB7tm3bpqCgIHXo0EGenp6Kj49XSkpKhT4tWrRwfS4qKpIxRpJkjFFRUZHKysp06tQpeXp6uvr26dNHrVu3rr0LAQBcxL0afbZL6miMaa+zYS1e0vDzjZZlHZPke/7YGJMq6VnLstJqtlQAAHAphYWFCgj4v//G6u/vr61bL94bbP78+ZozZ45KSkr073//W5I0dOhQpaSk6Oabb1ZxcbHmzp1LUAMAG6lypc2yrDJJT0n6WFK2pOWWZe02xrxqjIm71gUCAICaM2HCBO3fv1+vvfaaZsyYIensKp2bm5u++eYb5ebm6o9//KO++uqrOq4UAHBedVbaZFnWR5I++sm5aZfoG3v1ZQEAgMvh5+en/Pz/2zesoKBAfn5+l+wfHx+vJ598UpK0ZMkSDRw4UB4eHmrTpo2io6OVlpamDh06XPO6AQBVu+qNSAAAQN2LiopSTk6OcnNzVVJSouTkZMXFVbwhJicnx/X5ww8/VMeOHSVJ7dq1c90qWVRUpC1btqhLly61VzwA4GdVa6UNAADYm7u7u5KSkjRgwAA5nU6NHTtWISEhmjZtmhwOh+Li4pSUlKT169fLw8NDPj4+Wrx4saSzt0yOGTNGISEhsixLY8aMUXh4uCQpISFBqampOnLkiPz9/fXKK6/oV3V5oQDQABnLsupkYofDYaWlsVcJAADXnekta2WasPbtamWe5TPLamUeSQrek11rcwFhi8Nqba7a+ntU3/4OGWPSLcuqctd9bo8EAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMbb8BwCgngic+mGtzJPnVSvTAADOYaUNAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAqGXr1q1T586dFRQUpFmzZl3UvnDhQoWFhSkiIkK9e/dWVlaWJGnbtm2KiIhQRESEunXrptWrV7u+M3fuXIWEhCg0NFQJCQkqLymvtevBtUVoAwAAAGqR0+nUhAkTtHbtWmVlZWnp0qWuUHbe8OHDtWvXLmVkZOj555/XlClTJEmhoaFKS0tTRkaG1q1bp8cff1xlZWUqLCxUYmKi0tLSlJmZKafTqWNbj9XF5eEaILQBAAAAtWjbtm0KCgpShw4d5Onpqfj4eKWkpFTo06JFC9fnoqIiGWMkSU2bNpW7u7sk6fTp067zklRWVqZTp06prKxMxcXFcvdxr4WrQW0gtAEAAAC1qLCwUAEBAa5jf39/FRYWXtRv/vz5+sUvfqHnn39eiYmJrvNbt25VSEiIwsLCtHDhQrm7u8vPz0/PPvus2rVrp5tvvlktW7ZU89DmtXI9uPYIbQAAAIANTZgwQfv379drr72mGTNmuM736tVLu3fv1vbt2zVz5kydPn1aP/zwg1JSUpSbm6tvvvlGRUVF+vGzH+uwetQkQhsAAABQi/z8/JSfn+86LigokJ+f3yX7x8fH6/3337/ofHBwsLy9vZWZman169erffv2uvHGG+Xh4aGHHnpIxfuKr0n9qH2ENgAAAKAWRUVFKScnR7m5uSopKVFycrLi4uIq9MnJyXF9/vDDD9WxY0dJUm5ursrKyiRJBw4c0J49exQYGKh27dppy5YtKi4ulmVZ+te//qXGNzeuvYvCNcXTiQAAAEAtcnd3V1JSkgYMGCCn06mxY8cqJCRE06ZNk8PhUFxcnJKSkrR+/Xp5eHjIx8dHixcvliRt3rxZs2bNkoeHhxo1aqQFCxbI19dXvr6+Gjp0qLp37y53d3dFRkbKJ9anjq8UNcVYllUnEzscDistLa1O5gYAoD4KnPphrcyT5zW8VuYJa9+uVuZZPrOsVuaRpOA92bU2FxC2OKzW5qqtv0f17e+QMSbdsixHVf24PRIAAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGO8pw0AAACoTdNb1s48tfTaDFx7rLQBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI0R2gAAAADAxghtAAAAAGBjhDYAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANlat0GaMGWiM2WuM2WeMmVpJ+xPGmF3GmAxjzGZjTNeaLxUAAAAAGp4qQ5sxxk3SfEn3SuoqKaGSULbEsqwwy7IiJL0uaU6NVwoAAAAADVB1Vtp6StpnWdZXlmWVSEqW9MCFHSzLOn7BYTNJVs2VCAAAAAANl3s1+vhJyr/guEBSr592MsZMkDRFkqekuyobyBgzXtJ4SWrXrt3l1goAAAAADU6NbURiWdZ8y7J+IekFSS9eos/blmU5LMty3HjjjTU1NQAAAADUW9UJbYWSAi449j937lKSJT14NUUBAAAAAM6qTmjbLqmjMaa9McZTUrykNRd2MMZ0vOBwkKScmisRAAAAABquKp9psyyrzBjzlKSPJblJeseyrN3GmFclpVmWtUbSU8aYuyWVSvpB0qhrWTQAAAAANBTV2YhElmV9JOmjn5ybdsHnp2u4LgAAAACAanAjEgAAAABAzSO0AQAAAICNEdoAAAAAwMYIbQAAAABgY4Q2AAAAALAxQhsAAAAA2BihDQAAAABsjNAGAAAAADZGaAMAAAAAGyO0AQAAAICNEdoAAAAAwMYIbQAAAABgY4Q2AAAAALAxQhsAAAAA2BihDQAAAABsjNAGAAAAADZGaAMAAAAAGyO0AQAAAICNEdoAAACAc9atW6fOnTsrKChIs2bNuqh9zpw56tq1q8LDw9WvXz8dOHCgQvvx48fl7++vp556SpJ04sQJRUREuP74+vpq8rrTtXItqD8IbQAAAIAkp9OpCRMmaO3atcrKytLSpUuVlZVVoU9kZKTS0tK0c+dODR06VM8//3yF9pdeekl9+vRxHTdv3lwZGRmuP7feeqseCnavletB/UFoAwAAACRt27ZNQUFB6tChgzw9PRUfH6+UlJQKffr27aumTZtKkm677TYVFBS42tLT03Xo0CH179+/0vH/85//6LvvvlNMO7drdxGolwhtAAAAgKTCwkIFBAS4jv39/VVYWHjJ/n/+85917733SpLKy8v1zDPPaPbs2Zfsn5ycrEceeUTGmJorGg0CoQ0AAAC4TH/729+Ulpam5557TpK0YMEC3XffffL397/kd5KTk5WQkFBbJaIe4YZaAAAAQJKfn5/y8/NdxwUFBfLz87uo3/r16/X73/9e//u//6vGjRtLkj7//HNt2rRJCxYs0MmTJ1VSUiJvb2/XZiY7duxQWVmZevToIX1QO9eD+oPQBgAAAEiKiopSTk6OcnNz5efnp+TkZC1ZsqRCny+//FKPP/641q1bpzZt2rjOv/fee67PixYtUlpaWoXdJ5cuXcoqG64YoQ0AAACQ5O7urqSkJA0YMEBOp1Njx45VSEiIpk2bJofDobi4OD333HM6efKkhg0bJklq166d1qxZU+XYy5cv10cffXStLwH1FKENAAAAOOe+++7TfffdV+Hcq6++6vq8fv36KscYPXq0Ro8eXeHcV199VSP1oWFiIxIAAAAAsDFCGwAAAADYGKENAAAAAGyM0AYAAAAANkZoAwAAAAAbI7QBAAAAgI2x5T8AAAAgKXDqh7UyT55XrUyDeoSVNgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABurVmgzxgw0xuw1xuwzxkytpH2KMSbLGLPTGPMvY8ytNV8qAAAAADQ8VYY2Y4ybpPmS7pXUVVKCMabrT7p9KclhWVa4pBWSXq/pQgEAAACgIarOSltPSfssy/rKsqwSScmSHriwg2VZGyzLKj53uEWSf82WCQAAAAANU3VCm5+k/AuOC86du5RfSVp7NUUBAAAAAM5yr8nBjDGPSnJIuvMS7eMljZekdu3a1eTUAAAAAFAvVWelrVBSwAXH/ufOVWCMuVvSf0mKsyzrTGUDWZb1tmVZDsuyHDfeeOOV1AsAAAAADUp1Qtt2SR2NMe2NMZ6S4iWtubCDMSZS0n/rbGD7rubLBAAAAICGqcrQZllWmaSnJH0sKVvScsuydhtjXjXGxJ3r9oYkb0l/N8ZkGGPWXGI4AAAAAMBlqNYzbZZlfSTpo5+cm3bB57truC4AAAAAgKr5cm0AAAAAQN0gtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2RmgDAAAAABsjtAEAAACAjRHaAAAAAMDGCG0AAAAAYGOENgAAAACwMUIbAAAAANgYoQ0AAAAAbIzQBgAAAAA2Vq3QZowZaIzZa4zZZ4yZWkl7H2PMF8aYMmPM0JovEwAAAAAapipDmzHGTdJ8SfdK6iopwRjT9SfdvpY0WtKSmi4QAAAAABoy92r06Slpn2VZX0mSMSZZ0gOSss53sCwr71xb+TWoEQAAAAAarOrcHuknKf+C44Jz5y6bMWa8MSbNGJN2+PDhKxkCAAAAABqUWt2IxLKsty3LcliW5bjxxhtrc2oAAAAAuC5VJ7QVSgq44Nj/3DkAAAAAwDVWndC2XVJHY0x7Y4ynpHhJa65tWQAAAAAAqRqhzbKsMklPSfpYUrak5ZZl7Tb/v737j5a6rvc9/nwr/kLUUsKjoCFSyq9AQLHUMk+paOJFMbTwR9Kp1c1r6u2ElqvQdZeaqWmhJy1Pxx9HUesUmMCRMFtqh5+KheCvI5iQV29iCogi9L5/zOztns0GtnvP3t8ZeD7WmsV8f8zMe0ZnZr/n8/m+vhGXR8QogIg4NCKWA6cBN0fEUx1ZtCRJkiRtK1qTHklmTgOmNVv33SbX51GaNilJkiRJqqJODSKRJEmSJL0/Nm2SJEmSVMNs2iRJkiSphtm0SZIkSVINs2mTJEmSpBpm0yZJkiRJNcymTZIkSZJqmE2bJEmSJNUwmzZJkiRJqmE2bZIkSZJUw2zaJEmSJKmG2bRJkiRJUg2zaZMkSZKkGmbTJkmSJEk1zKZNkiRJkmqYTZskSZIk1TCbNkmSJEmqYTZtkiRJklTDbNokSZIkqYbZtEmSJElSDbNpU+FmzJjBQQcdRN++fbnqqqs22v7OO+8wduxY+vbty4gRI1i2bFnF9j//+c9069aNa665pnFd7969GTRoEEOGDGH48OEd/RQkSZKkDmPTpkJt2LCBr3/960yfPp3Fixdz9913s3jx4op9br31Vj74wQ/y/PPPc+GFFzJhwoSK7RdddBEjR47c6L5/97vfsXDhQubPn9+hz0GSJEnqSDZt2qRqj4C99NJLfPrTn6Z///4MGDCAG264gblz59K3b1/69OnDjjvuyOmnn86UKVMq7mfKlCmcffbZAIwZM4ZZs2aRmQD8+te/5oADDmDAgAEd8ApIkiRJxbNpU4s6YgSsS5cuXHvttSxevJjZs2dz4403Mnv2bPbbb7/GfXr16sWKFSsq7mfFihWN+3Tp0oU99tiD1157jdWrV/P973+f733vexvVHxEce+yxDBs2jFtuuaXdr4ckSZJUlC5FF6Da1HQEDGgcAevfv3/jPlOmTGHixIlAaQTsvPPOIzOJiMYRsF133bVx/3322Yd99tkHgN12241+/fqxcuXKNtc4ceJELrzwQrp167bRtkcffZSePXvy6quv8tnPfpaDDz6YT37yk21+LEmSJKkojrTVobZOW5w7dy5DhgxhyJAhDB48mF/96lcAPPPMM43rhwwZwu67787NN9/cISNgDZYtW8YTTzzBpz71KV566aXG9cuXL6dnz54V+/bs2bNxn/Xr1/PGG2+w1157MWfOHL71rW/Ru3dvrr/+eq644gomTZrUeBuAHj16MHr0aObOnduq11aqV9X+XAD44Q9/yIABAxg4cCBnnHEGb7/9dmc9HUmS1IRNW51pz7TFgQMHMn/+fBYuXMiMGTP46le/yvr16znooINYuHAhCxcuZMGCBXTt2pURI0a0ucbNjYABrF69mlNPPZXrr7+eo48+mueee46lS5eybt06Jk+ezKhRoyr2HzVqFLfddhsAv/jFLzjmmGOICB555BGWLVvGsmXLuOCCC/j2t7/Neeedx5o1a1i1ahUAa9as4cEHH2TgwIFtfj5SreuIz4UVK1bwox/9iPnz57No0SI2bNjA5MmTi3h6kiRt82za6kx7gju6du1Kly6lGbFvv/02EbHR/c+aNYsDDzyQIUOGdMgI2Lvvvsupp57KF7/4RU455RS6dOnCpEmTOO644+jXrx+f//znGTBgAN/97neZOnUqAOPHj+e1116jb9++XHfddS2OIjT1yiuvcOSRRzJ48GAOO+wwTjzxRI4//vj38zJLdaWjPhfWr1/P2rVrWb9+PW+99Rb77rtv5z0pSZLUyGPa6kzTKYlQmrY4Z86cTe7TdNpi9+7dmTNnDueeey4vvvgid9xxR+Mfaw0mT57MGWecwaGHHto4AtazZ08mT57MXXfdVbFvwwjYxz/+8Y1GwBpMnDiRbt26NR7vNn78ePr168dFF13UuM8JJ5zACSecUHHfl19+eeP1nXfemfvuu2+zr0vDsXUAffr04cknn9zs/lJnmTFjBt/4xjfYsGEDX/7yl7n44osrtr/zzjucddZZLFiwgL322ot77rmH3r17M3fuXL7yla8AkJlMnDiR0aNHN95uw4YNDB8+nJ49e3LOOedU/XOhZ8+efPOb32T//fdnl1124dhjj+XYY4/tqJdJkiRthiNt25gRI0bw1FNPMW/ePK688sqKY1TWrVvH1KlTOe200zpkBOyxxx7jjjvu4KGHHmo8hmbatGkd+nylInXEtMUGN9xwA/369atKnS19Lrz++utMmTKFpUuX8pe//IU1a9Zw5513VuXxJEnS++NIW51pOiURNj9tsVevXhXTFpvq168f3bp1Y9GiRQwfPhyA6dOnM3ToUPbee2+g+iNgRx55ZOP51aRtQXtSWLt27dq4T/Npi8uXL+eBBx7gO9/5Dtddd12HfC4sXbqUAw44gA996EMAnHLKKfzhD39g3Lhx1XlxJElSqznSVkUdkd7W/D6bTlt8v8EdS5cubfyl/sUXX+Tpp5+md+/ejbe7++67OeOMM6r8qkjbrpamM7c2hRVgzpw5DBgwgEGDBvGTn/ykcTrzBRdcwNVXX81225U+wjvic2H//fdn9uzZvPXWW2Qms2bNqtrIniRJen8caauShmlQM2fOpFevXhx66KGMGjWq4kU9G/sAABMfSURBVBf1ptOgJk+ezIQJE7jnnnsap0F16dKFl19+mcGDB3PSSScRES3eZ8O0xQ0bNnDuuec2TlscPnw4o0aNYvz48Zx55pn07duXPffcszHx7dFHH+Wqq65ihx12YLvttuOmm26ie/fuQCllcebMmdx8882FvH6SNtYwbXHJkiWcffbZjBw5kt/+9rf06NGDYcOG8fDDDwNUTGeu1udC9+7dGTNmDEOHDqVLly4ccsghjcfYSZKkzmXTViUdMQ1qU/d5ySWXtGna4plnnsmZZ57ZYv277rpr46/70taureEgM2fO5OKLL2bdunXsuOOO/OAHP+CYY45h1apVHHXUUY23X758OePGjWPs2LFVn7b42GOPMXXqVKZNm8bbb7/Nm2++ybhx47jzzjur/rlw2WWXcdlll23h1ZQkSR3Npq1KOiLVsTX3uTVYcnDnTLnq9/SSTnkc1bb2jIp3796d+++/n3333ZdFixZx3HHHsWLFCnbbbTcWLlzYePthw4ZxyimntCuFdenSpey333506dKlYtrilVdeyZVXXgnAww8/zDXXXGNAiCRJWzmPaasRm0t1lFQ97Tmn2SGHHNJ4rrIBAwawdu1a3nnnnYrbPvvss7z66qscddRR7UphffTRRxk8eDBDhgxh9OjRFdOZJUnStsWRtirpiPS21txnRxp026BOeZx7O+VRpJL2joo3+OUvf8nQoUPZaaedKm47efJkxo4d2zjNua0prJubttjg6KOP5uijj97sPpIkqf450lYlHZHe1pr7lNT5nnrqKSZMmNBicE/DCeolSZKqxaatSjpiGtSm7lPaWrX1tBkzZ85k2LBhDBo0iGHDhvHQQw813mbBggUMGjSIvn37cv7557Pvvvu2elQc2GhUfPny5YwePZrbb7+dAw88sOJ2Tz75JOvXr2fYsGFVeT0kSZLA6ZFV1RHToFq6T2lr1BEBIQBf+9rX+OlPf8qIESM44YQTWLlyZZvDQf72t79x4oknctVVV3HEEUds9ByKOtehYT6SJG3dbNrq0cQ9OudxDti/cx5Hon2nzTjkkEMa92kaELJy5UrefPNNDj/8cADOOuss7r///jaf02zSpEk8//zzXH755Y0/yDz44IP06NEDgHvvvZdp06Z1+GslSZK2LTZtkmpCRwSErFixgl69elXc54oVK9o8Kn7ppZdy6aWXbvI5vPDCC618tpIkSa1n0yZpq9EQEPLggw8WXQpgAqskSaoOg0gk1YT3c9oMaF1ASM+ePVm+fPlm71OSJKnW2bRJqgntOW3GpgJC9tlnH3bffXdmz55NZnL77bdz8sknd+rzkiRJai+nR1ZJ74sf6LTHWrZzpz2U1GmanuKimgEhN910E+eccw5r165l5MiRjBw5ssinKUmS9L7ZtEmqGR0REDJ8+HAWLVpU3UIlSZI6kU2bpG2Pp82QJEl1xGPaJEmSJKmG2bRJkiRJUg2zaZMkSZKkGmbTJkmSJEk1zKZNkiRJkmqY6ZGSakZnne/Qcx1KkqR64kibJEmSJNUwmzZJkiRJqmE2bZIkSZJUw2zaJEmSJKmG2bRJkiRJUg2zaZMkSZKkGmbTJkmSJEk1zKZNkiRJkmqYTZskSZIk1TCbNkmSJEmqYa1q2iLi+Ih4JiKej4iLW9i+U0TcU94+JyJ6V7tQSZIkSdoWbbFpi4jtgRuBkUB/4IyI6N9st/HA65nZF/gh8P1qFypJkiRJ26LWjLQdBjyfmS9k5jpgMnBys31OBm4rX/8F8I8REdUrU5IkSZK2TV1asU9P4KUmy8uBEZvaJzPXR8QbwF7AX5vuFBFfAb5SXlwdEc+0pehtXed1w4u60+y/YUdoPmzbYfwdQWW+h9rI95DKfA+1g+8jsfW9h8Dvonb4cGt2ak3TVjWZeQtwS2c+ptouIuZn5vCi65Dqle8hqX18D0nt43to69Ga6ZErgP2aLPcqr2txn4joAuwBvFaNAiVJkiRpW9aapm0e8JGIOCAidgROB6Y222cqcHb5+hjgoczM6pUpSZIkSdumLU6PLB+jdh7wn8D2wL9m5lMRcTkwPzOnArcCd0TE88BKSo2d6p9TWaX28T0ktY/vIal9fA9tJcIBMUmSJEmqXa06ubYkSZIkqRg2bZIkSZJUw2zaJEmSJKmG2bRJkiRJUg3r1JNrq7ZFRACHAT3Lq1YAcz19g7RlEXF8Zs4oX98DuA44FFgEXJiZrxRZn1QvIuJg4GQqv4umZuaS4qqS6ldE7JmZK4uuQ+3jSJsAiIhjgeeAicAJ5ctlwHPlbZI274om168FXgZOonSuy5sLqUiqMxExAZgMBDC3fAng7oi4uMjapHoQEZc2ud4/Ip4FFkTEsogYUWBpaicj/wVARCwBRmbmsmbrDwCmZWa/QgqT6kREPJ6ZQ8vXF2bmkCbbKpYltaz8B+aAzHy32fodgacy8yPFVCbVh2bfRQ8AkzJzekQcBlyfmZ8otkK1ldMj1aALsLyF9SuAHTq5Fqke9YiIiyiNCuweEdFkarGzGqTW+TuwL/Bis/X7lLdJar19M3M6QGbOjYhdii5IbWfTpgb/CsyLiMnAS+V1+wGnA7cWVpVUP34K7Fa+fhvQHfh/EfEPwMLCqpLqywXArIh4jve+i/YH+gLnFVaVVD/6RMRUSj8g9oqIrpn5VnmbP8LXMadHqlFE9KPlg78XF1eVJGlbEhHbsXEo1rzM3FBcVVJ9iIhPNVu1IDNXR8TewJjMvLGIutR+Nm2SVEURsQ+lEeo+wKvAPZn5bLFVSfUvIrpl5uqi65CkInichbYoIqYXXYNUDyLifODfgP8GbgR+D1wdEZ8tjx5IajtnfUitFBFjIuLXETErIh6IiC8VXZPax2PaBEBEDN3UJsDUO2kLIuJE4HDgeOA0StO7AKYDlwAHRMRfMvM3BZUo1bxymE+Lm4BunVmLVI/KPxBOBpYAZ2fmG+X01QkRcQFwH/ByZhrsU2ds2tRgHqVRgWhh2wc6uRapHp0PnJOZGRHDKQUnPAh8FpgD/AdwF2DTJm3aFcAPgPUtbHO0Wtqy84AnMvPKiLg+InYvr98O6A+8AuwNXF9UgWobmzY1WAJ8NTOfa74hIl5qYX9JlXpk5svl658Ajio3cDcDj2TmJeUDwSVt2uPArzNzQfMNEfHlAuqR6s1Y4DPl669TOn3GdOA44AXgV8DD2LTVHZs2NZjIpn/F/F+dWIdUr1ZHRPfM/CvwBvC5iPhPSl+UqyJiV8AQBWnzvgS8toltwzuzEKlO7ZaZa8vXP5eZh5avPx0R8zLz8ojYqaji1HamR0pSFUTEeGBAZl4UEd2BbwMHA08DVwFfBV7PzEkFlilJ2oqVz9E2ITOXRMQkYB0wg9IPiLtQmoL8s8w8vsAy1QY2bapgXLnUNhERwL8Dy4ArGqLJI6IrMAEYSOkcOX7oSlsQEWOAcZROWP828IvM/HmxVUm1LyI+DfwzcGJ51Um89wPiNOBu4NbMnFFMhWormzY1KseVn0gpqvxZoAdwUXl5lklD0pZFxNnAWcD2wN+BpJTk9TMbNmnzmiXfXdc0+Q5Yhcl30hZFxNeAMZRG1WYDGyilG38HmJ6Z1xVYntrIpk1AY1z5F8uX04Cdy5t2oXRQ62TAuHJJUocp/3i4a0PyHdA8+e6HwN6ZaYiCtBkR0ZfSMaKDKf14uAi4PTOXFFqY2symTQCUAxPOycyXI+JqKuPKnwWuBe7KzGMLLFOqeeUDvE8FetMk7CkzLy+qJqleRMRjwGcyc21EfA94k/eS7/YArgYezszDCyxTkjqd6ZFqYFy5VB1TKKVHLgDeKbgWqd6YfCe1Q0TcT2lkrUWZOaoTy1EV2bSpgXHlUnX0MpVLarNlEdGvPIVrTkRcx3vJd/MiohelkwNLatk1RRegjuH0SAHGlUvVEhG3AD/OzD8VXYtUb0y+k6SW2bQJMK5cqpaIWEzpmNCllKZHBpCZ+bFCC5PqhMl3UttFxJ/Y/PRIv4vqlE2bKhhXLrVPRHy4pfWZ+WJn1yLVK5PvpLbZ1HdQA7+L6pdNmyRVWUQMBo4qLz6SmU8WWY8kSapvNm2qYFy51D4R8Q3gn4D/KK8aDdySmT8uriqpPph8J1VHRBwO/BjoB+xIaQbVmszcfbM3VM2yaVOFiJjBe3HlGxrWZ+a1hRUl1ZGI+CPw8cxcU17eFfgvjyOQtiwiPrW57Zn5+86qRapnETEfOB24DxhO6dCXj2bmJYUWpjYz8l/NGVcutU/Q5AeP8vUoqBaprtiUSdWTmc9HxPaZuQH4eUQ8Adi01SmbNjX3h4gYZFy51GY/p3R+qV+Vl/8HcGuB9Uh1w+Q7qWreiogdgYURcTXwMrBdwTWpHZweqQrGlUvtFxFDgSPLi49k5hNF1iPVC5PvpOoov5deoXQ824XAHsBNmfl8oYWpzWzaVMG4cqltImL3zHwzIvZsaXtmruzsmiRJ26by8dRrM/Pv5eXtgZ0y861iK1NbOUyqCuXm7APASeXLB2zYpFa5q/zvAmB+k0vDsqRWiojDI2JeRKyOiHURsSEi3iy6LqmOzAK6NlneBfhtQbWoCjymTRVaiCu/MyKMK5e2IDM/V/73gKJrkbYCk2gh+a7QiqT6snNmrm5YyMzVEdF1czdQbXOkTc2NB0Zk5ncz87vA4ZSaOEmtEBFHlKelEBHjIuK6iNi/6LqkelM+9mb7zNyQmT8HTDaWWm9N+fhqACJiGLC2wHrUTo60qTnjyqX2+RdgcEQMBv438DPgDmCz55+SVMHkO6l9LgDui4i/UPo77h+AscWWpPawaVNzxpVL7bM+MzMiTgYmZeatETG+6KKkOnMmpSbtPErJd/sBpxZakVRHMnNeRBwMHFRe9UxmvltkTWof0yO1EePKpbaLiN8DM4AvAZ8EXgWezMxBhRYm1RGT76T2KR+/dhHw4cz8p4j4CHBQZv6m4NLURk41EFCKKy//uyewDLizfHlxUxHmklo0ltI5Dsdn5v8FegE/KLYkqe6YfCe1z8+BdcDHy8srgP9TXDlqL0faBEBE/CYzPxcRS4Gm/1M0nFy7T0GlSZK2MRGxMDOHbGmdpJZFxPzMHB4RT2TmIeV1T2bm4KJrU9t4TJsA48ql9oqIRzPzyIhYRcs/fOxeUGlSPVoTEUMz83Ew+U5qg3URsQvl76OIOJDSLBDVKUfaVCEijgAWZuaaiBgHDAWuz8w/F1yaJGkbERGHApOBiuS7zFxQaGFSHYiIoBTmMx7oDzwIHAGck5kPF1ia2sGmTRUi4o/AYOBjwL9Riiv/fGYaVy61QkQcDjyVmavKy7sB/TNzTrGVSfUlInbA5DupTSLiT8DRlM63G8DszPxroUWpXQwiUXPrs9TJN8SV3wjsVnBNUj35F2B1k+U15XWSWqmcfDcB+EZmLgJ6R8TnCi5LqiePA30y84HM/I0NW/2zaVNzqyLiEmAc8EBEbAfsUHBNUj2JbDKFoRxZ7vHD0vtj8p3UPiOA/4qI/46IP0bEn8qzqVSn/ENCzY0FvkA5rjwi9se4cun9eCEizue90bX/CbxQYD1SPTowM8dGxBkAmflW+TgdSa1zXNEFqLo8pk2SqigiegA/Ao6hlNo1C7ggM18ttDCpjkTEH4B/BB7LzKHl5Lu7M/OwgkuTpELYtAkwrlySVBtMvpOkjdm0SVIVRcRHKU2N3DszB0bEx4BRmenxOFIrmXwnSZUMIlGFiDi8HFHesLxbRIwosiapzvwUuAR4FyAz/wicXmhFUv0x+U6SmrBpU3PGlUvt0zUz5zZbt76QSqT6ZfKdJDVheqSa2yiuPCL8/0Rqvb+WQxMSICLGAC8XW5JUd0y+k6Qm/GNczRlXLrXP14FbgIMjYgWwFPhisSVJ9SUzXyy6BkmqJQaRqIJx5VLblU9GPyYz742IXYHtMnNV0XVJkqT6ZtMmSVUUEfMzc3jRdUiSpK2HQSSqEBEfjYhZEbGovPyxiLi06LqkOvLbiPhmROwXEXs2XIouSpIk1S9H2lQhIn4P/DNwc2YeUl63KDMHFluZVB8iYimVJ6gHIDP7FFCOJEnaChhEoua6ZubciGi6zrhyqfX6UwrwOZJS8/YI8JNCK5IkSXXNpk3NGVcutc9twJuUAn0AvlBe9/nCKpIkSXXN6ZGqEBF9KMWVfwJ4nXJcufHLUutExOLM7L+ldZIkSa3lSJsalePKh2fmZ4wrl9rs8Yg4PDNnA0TECGB+wTVJkqQ65kibKhhXLrVPRCwBDgL+XF61P/AMpWNDMzM/VlRtkiSpPtm0qUJEXAX8FbgHWNOwPjNXFlaUVEci4sOb2+5UY0mS9H7ZtKmCceWSJElSbbFpU4WI2IUW4sozc22hhUmSJEnbKJs2VYiIeynFlf97edUXgD0y07hySZIkqQA2bapgXLkkSZJUW7YrugDVnMcj4vCGBePKJUmSpGI50qYKxpVLkiRJtcWmTRWMK5ckSZJqi02bJEmSJNUwj2mTJEmSpBpm0yZJkiRJNcymTZIkSZJqmE2bJEmSJNWw/w/a+OztbA+L/QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "ax = global_metrics_df.transpose().plot(kind='bar', figsize=(15,8))\n",
    "for p in ax.patches:\n",
    "    ax.annotate(\"%.3f\" % p.get_height(), (p.get_x() + p.get_width() / 2., p.get_height()), ha='center', va='center', xytext=(0, 10), textcoords='offset points')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we've explored and compared the main Recommender Systems techniques on CI&T Deskdrop dataset. It could be observed that for articles recommendation, content-based filtering and a hybrid method performed better than Collaborative Filtering alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}