3_Extracting the Data & Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os, glob, string, re, datetime\n",
    "from bs4 import BeautifulSoup\n",
    "from nltk.corpus import stopwords\n",
    "import pandas as pd\n",
    "from collections import Counter\n",
    "from tqdm import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting the Data\n",
    "Know that we have learned that we will not be able to process the data using a Spark pipeline, we will implement here a Python pipeline to read the files following the method that was implemented in the [Filtering the data notebook](https://github.com/ThomVett/Project_ADA/blob/master/1_Exploring%20the%20Data.ipynb).\n",
    "\n",
    "## 1 - Extracting the paths\n",
    "\n",
    "We first have to get all the paths containing the XML files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "GDL = 'GDL'\n",
    "JDG = 'JDG'\n",
    "listFiles = []\n",
    "\n",
    "#Go through all the files of both journals\n",
    "for root, dirs, files in os.walk(GDL): \n",
    "    for file in files:\n",
    "        if file.endswith('.xml'):\n",
    "             listFiles.append(os.path.join(root, file))\n",
    "\n",
    "for root, dirs, files in os.walk(JDG): \n",
    "    for file in files:\n",
    "        if file.endswith('.xml'):\n",
    "             listFiles.append(os.path.join(root, file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 2 - Extracting the word count\n",
    "Know that we have the paths for all text files, we can read the text and then clean it.\n",
    "\n",
    "First we need the updated stop word list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "stop_words = list(stopwords.words('french'))\n",
    "stop_words.extend(['.', ',', '\"', \"'\", '?', '!', ':', ';', '(', ')', '[', ']',\n",
    "                   '{', '}', '<', '>', '...', '-', '•', '/', '%', '«', '»', 'le',\n",
    "                  'les','..','©','plus','*','``','av','p.','fr','cette','a.','v.','b.','d.',\n",
    "                   'c.','e.','f.','i.','g.','h.','i.','j.','l.','m.','n.','o.','q.','r.','q.',\n",
    "                   's.','t.','u.','v.','w.','x.','z.','y.',\"d'un\",\"d'une\",'dm',\"c'est\",\n",
    "                   \"jusqu'au\",'entre','comme','si','di','„','&',\"qu'il\",'_','fd',\"n'a\",'alors',\n",
    "                   \"s'est\",\"n'est\",'cs','dès','où',\"jusqu'à\",'déjà',\"''\",'|','£','®','+','-«',\n",
    "                   '--','.-',\"la\",'---','-le','-n','î','â'])\n",
    "\n",
    "stop_words.extend(['','-.','.,','(,','»,',').','-.','av.','».',\n",
    "                      '....','..','...','.....','()','(r).','—',')-«',\n",
    "                      '\".','-,','(-.',')-.','(©','™','--','---','—•','•-',\n",
    "                  '•','••','•••','•••','—la','ï',\"'la\",'•—',\n",
    "                  '——','û','ê','ë','ù',\"'la\",'•—','——','—le',\"'''\",\"''\",\n",
    "                  \"'\",\"'le\",\"'la\",\"'i\",'-on',\"•'\",\"'-\",\"-'\",\"a'\",\"'de\",\"'la\",\n",
    "                  \"'â\",\"aa\",\"'-\",\"-de\",\"-et\",'-on','-les',',,,', '——•','•-•',\n",
    "                  '„„','•—•','ils','ont','tous','tout','aussi','ans',\n",
    "                  'on','quelque','dont','ainsi','après','aprè',\n",
    "                  'depuis','deux','dont','leurs','leur','une',\n",
    "                  'fait','faite','non','oui','dit','en',' ',\n",
    "                  '  ','   ','    ','     ','      ','       ','        '])\n",
    "stop_words.extend(list(string.ascii_lowercase))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And preparing the dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "listPunct = list(string.punctuation)\n",
    "listPunct.remove(\"'\")\n",
    "listPunct.remove(\"-\")\n",
    "#we do not want to stick together all the words that\n",
    "#have an apostrophe (l'armée should not become larmée)\n",
    "stopDict = dict(zip(listPunct,['']*len(listPunct)))\n",
    "#We want to break apart words with \"'\" and also replace some\n",
    "#random characters that we saw appeared in the text\n",
    "stopDict[\"'\"] = ' '\n",
    "stopDict[\"•\"] = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need a constant which tells us how many of the most frequent words we take."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numMostFreq = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can go through the list to extract the data in each file. We also set a counter to see how long it takes.\n",
    "\n",
    "We first extract the data in 43 different dataframes, otherwise the algorithm's complexity is too high and the process is endless.\n",
    "\n",
    "We also chose to extract 3000 + i most frequent words in the loop as we do not want to miss too many words during the later years, where the size of the dataset increases a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "total = len(listFiles)\n",
    "allText = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "4335it [11:33:20, 22.15s/it]\n"
     ]
    }
   ],
   "source": [
    "for idx, file in tqdm(enumerate(listFiles)):   \n",
    "    #find the date at which the article is published.\n",
    "    date = pd.to_datetime(file[4:11])\n",
    "    #Open the XML\n",
    "    f = open(file,'r')\n",
    "    soupArticle = BeautifulSoup(f,'lxml')\n",
    "    f.close()\n",
    "    \n",
    "    #Extracting the entire text from the XML\n",
    "    Article = []\n",
    "    for ft in soupArticle.find_all('full_text'):\n",
    "        #Here we lower each words and we remove the numbers from the text\n",
    "        Article.append(re.sub(r'\\d+', '', ft.text).lower())\n",
    "\n",
    "    textArticle = ' '.join(Article)  \n",
    "    \n",
    "    #Cleaning and counting\n",
    "    for k,v in stopDict.items():\n",
    "        textArticle = textArticle.replace(k,v)\n",
    "\n",
    "    dictCounter = [wordsDict for wordsDict in textArticle.split(' ') if wordsDict not in stop_words]\n",
    "    countDict = Counter(dictCounter).most_common(numMostFreq+idx)\n",
    "    \n",
    "    #Transofming to a dataframe\n",
    "    df = pd.DataFrame(countDict).transpose()\n",
    "    df.columns = df.loc[0]\n",
    "    df.drop(0,inplace=True)\n",
    "    df.index = [date]\n",
    "    allText.append(df)\n",
    "    \n",
    "    if idx > 0:\n",
    "        if idx%50 == 0:\n",
    "            dfAllText = pd.concat(allText,axis=0)\n",
    "            dfAllText.to_csv('4_Data/Part_'+str(idx)+'.csv')\n",
    "            del allText\n",
    "            allText = []\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dfAllText.to_csv('4_Data/Part_end.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all the separate dataframes we can extract them one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "csvFiles = []\n",
    "\n",
    "for root, dirs, files in os.walk('4_Data'): \n",
    "    for file in files:\n",
    "        if file.endswith('.csv'):\n",
    "             csvFiles.append(pd.read_csv(os.path.join(root, file),\n",
    "                                         index_col=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can concatenate all of the separate DataFrames!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dfFinal = pd.concat(csvFiles,axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we look at the results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words :  275577\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>----</th>\n",
       "      <th>-----</th>\n",
       "      <th>------</th>\n",
       "      <th>-------</th>\n",
       "      <th>--------</th>\n",
       "      <th>---------</th>\n",
       "      <th>----------</th>\n",
       "      <th>-----------</th>\n",
       "      <th>------------</th>\n",
       "      <th>-------------</th>\n",
       "      <th>...</th>\n",
       "      <th>„„«</th>\n",
       "      <th>„„»</th>\n",
       "      <th>„„„</th>\n",
       "      <th>„„„„</th>\n",
       "      <th>€</th>\n",
       "      <th>™-</th>\n",
       "      <th>™»</th>\n",
       "      <th>™—</th>\n",
       "      <th>™™</th>\n",
       "      <th>™™»</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1806-05-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1806-06-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1806-07-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1806-08-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1806-09-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 275577 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            ----  -----  ------  -------  --------  ---------  ----------  \\\n",
       "1806-05-01   NaN    NaN     NaN      NaN       NaN        NaN         NaN   \n",
       "1806-06-01   NaN    NaN     NaN      NaN       NaN        NaN         NaN   \n",
       "1806-07-01   NaN    NaN     NaN      NaN       NaN        NaN         NaN   \n",
       "1806-08-01   NaN    NaN     NaN      NaN       NaN        NaN         NaN   \n",
       "1806-09-01   NaN    NaN     NaN      NaN       NaN        NaN         NaN   \n",
       "\n",
       "            -----------  ------------  ------------- ...   „„«  „„»  „„„  \\\n",
       "1806-05-01          NaN           NaN            NaN ...   NaN  NaN  NaN   \n",
       "1806-06-01          NaN           NaN            NaN ...   NaN  NaN  NaN   \n",
       "1806-07-01          NaN           NaN            NaN ...   NaN  NaN  NaN   \n",
       "1806-08-01          NaN           NaN            NaN ...   NaN  NaN  NaN   \n",
       "1806-09-01          NaN           NaN            NaN ...   NaN  NaN  NaN   \n",
       "\n",
       "            „„„„    €  ™-  ™»  ™—  ™™  ™™»  \n",
       "1806-05-01   NaN  4.0 NaN NaN NaN NaN  NaN  \n",
       "1806-06-01   NaN  NaN NaN NaN NaN NaN  NaN  \n",
       "1806-07-01   NaN  NaN NaN NaN NaN NaN  NaN  \n",
       "1806-08-01   NaN  NaN NaN NaN NaN NaN  NaN  \n",
       "1806-09-01   NaN  NaN NaN NaN NaN NaN  NaN  \n",
       "\n",
       "[5 rows x 275577 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Number of words : ',dfFinal.shape[1])\n",
    "origShape = dfFinal.shape \n",
    "dfFinal.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a lot of words, as we saw before, we still have a lot of unwanted words and random punctuations signs that we were not able to remove during the cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['----', '-----', '------', '-------', '--------', '---------',\n",
       "       '----------', '-----------', '------------', '-------------',\n",
       "       ...\n",
       "       '„„«', '„„»', '„„„', '„„„„', '€', '™-', '™»', '™—', '™™', '™™»'],\n",
       "      dtype='object', length=275577)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfFinal.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First Cleaning Steps\n",
    "\n",
    "We will begin by performing a little bit of rough cleaning before going into custom NLP cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sumWords = dfFinal.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we decide to remove all the words that are only present once in the dataset, as they shurely belong to either random characters or, misspells, and they do not contain a lot of information about the rest of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words that will be removed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index(['--j-«-—', '--mettre', '--pasteur', '--présente', '--quant',\n",
       "       '--quelques', '--r', '--tejournaldesdébatsfaittopsles', '--ëphémérides',\n",
       "       '--—les', '--——-v', '--™', '-a-genevjï', '-a-js-hs', '-accepter',\n",
       "       '-accepté', '-accumulés', '-achats', '-actuel', '-actuellement'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Words that will be removed')\n",
    "dfFinal.columns[(sumWords<2).values][20:40]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before the removal of absent words:  (4351, 275577)\n",
      "Number of columns that will be removed:  62657\n",
      "After the cleaning:  (4351, 212920)\n"
     ]
    }
   ],
   "source": [
    "print('Before the removal of absent words: ', dfFinal.shape)\n",
    "print('Number of columns that will be removed: ',(sumWords<2).sum())\n",
    "dfFinal.drop(dfFinal.columns[(sumWords<2).values],axis=1,inplace=True)\n",
    "print('After the cleaning: ', dfFinal.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we also see that we have two indexes per month, therefore we can groupby. That is due to the fact that we had one month per newspaper and we had 2 newspapers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dfFinal = dfFinal.fillna(0)\n",
    "year = dfFinal.groupby(by='index',level=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "And aggregate per year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2351, 212920)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yearData = year.agg(sum)\n",
    "yearData.head()\n",
    "yearData.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting the index in datetime to be able to plot dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yearData.index = pd.to_datetime(yearData.index)\n",
    "yearIndex = yearData.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sumYear = yearData.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the evolution of the total number of words over the years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1ed68ea90>]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjQAAAFoCAYAAABAL2ESAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xt8XVWd///X3klIW9omaekNYQaZabm0gKQgtIByFUcE\nSoGlXGaAItdBRgGB8QLfGXUERHG8lAqClQGUBQWKMP5AwBGlQaAplRahlYsK9JKWtKX0lmav3x97\nn/Tk5Jyc+2Un7+fj0UeTrL32/iTtST5Zl8/ynHOIiIiIxJlf7QBEREREiqWERkRERGJPCY2IiIjE\nnhIaERERiT0lNCIiIhJ7SmhEREQk9pTQiIiISOwpoREREZHYU0IjIiIisVefz8XGmF2B7wNHAZsA\nC/y7tXabMea/gc8DDvCivz9vrZ0d9T0WuAXYE2gDLrDWvpl07y8AVwEjgPuBy6y1W6K2RmA2MDN6\n7nestd9N6rsHcDswDXgL+KK19tdJ7f0+W0REROIt3xGaecAQ4DDgs8CJwNejtn2Aa4AJwPjo7zsB\njDG7Aw8BdwAHAWuAhxM3NcacClwHXAAcDRwK3JT03JuBVuBI4FLgemPMzKT2h4F3ganA3cBDxpjd\ncnm2iIiIxF/OIzTGmL2AjwLjrLVroo9dB3ybMJHZB7jJWrs6TffPAS9Ya78X9TsPWGmM+Zi19hng\ncuAWa+2vovaLgCeMMVcTJl3nA8dbaxcDi40xNwGXAQ8aY44mHHk5NBrRucEYcwwwC/hPwiSpv2eL\niIhIzOUzQrMS+GQimYl4QJMxZgTwIWBZhr6HAj3Jg7V2M9AOTDPG+MDBwO+Srn8O2Ak4IPpTTzhV\nlPB74JDo7UOA9sT0VFL7tKT2tM/O8vmKiIhITOQ8QmOtXQ8kr0vxCEdJniQcnXHAV40x/wSsBb5r\nrb0runwC4ZRQslXAbkAz4TRWT7u1ttsYszZqd8Aaa+32lL5DjDGjs9w727NFRERkAMhrUXCKbwMf\nIRxdOQgIgFcIFw0fCdxmjFlvrZ0PDAO2pvTfCjRGbfTT7mdoI6l/pr7k0J6L0cDxhAuOt/R/qYiI\niCQZAuwBPE444FEWBSU0xpgbCde9GGvtK8ArxphHrLXrokuWGGMmAZcA8wmTgNQEohHoZEeCkK59\nUxRjujai9i3AqAx9yfLsXB0P3JPH9SIiItLbWcC95bp53gmNMeYHwEXAWdbant1CSclMwp8It3cD\nvEO48ynZeGARYba2JXp/WfSMOsJRkRWEIzS7GGN8a22Q1HeztXadMeYdYN80916Rw7Nz9RbAhg0b\n6O7uzqNbddXV1TFy5MjYxQ3xjT2ucUN8Y49r3BDf2OMaN8Q39rjGDdDQ0MDw4cMh+llaLvnWobke\nuBD4jLX2oaSP/wcw3Vp7XNLlBwKvRm8/BxyedP2wqP06a60zxrwQtScW704HtgGLCRcedxEuLF4Q\ntR8BvJB072uMMY3W2sTU0uHsWGSc6dnX5/GpbwHYunUrXV1deXSrroaGBiB+cUN8Y49r3BDf2OMa\nN8Q39rjGDfGNPa5xpyjrko18tm3vA3wV+C9ggTFmXFLzL4FrjTFXENZ4OR44m3AtDYT1aK6KtmE/\nSphMvJG0bXo2MMcYs5RwAe9s4Lakwnp3Re2zCBfzXgmcE/X9LfA3YK4x5uvASYTres7t59mvW2t/\nm+vnLiIiIrUtn23bJ0XXf5Uw6XiXcFrnXWvti8BpwL8ALxPufjrDWvs8gLX2L4RVfmcBzxPubJqR\nuLG19j7gW8CPCRcNtRHWtkm4AlgIPA38APhatNiYaBrqZMJppBeBM4EZ1tq3+3n2KXl83iIiIlLj\nPOdctWOIg1ZgYUdHR6yG+hoaGhgzZgxxixviG3tc44b4xh7XuCG+scc1bohv7HGNG2Do0KG0tLRA\nWM2/vVzP0eGUIiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMSe\nEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMSe\nEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMSe\nEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMSe\nEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMSe\nEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7CmhERERkdhTQiMiIiKxp4RGREREYk8JjYiIiMRe\nfbUDEBERkdrlnIPlS3Hr3sNrHgUTJ+N5XrXD6kMJjYiIiKTl2tsIHvgpdKwM3wcYMx7/tPPwWqdV\nNbZUmnISERGRPlx7G8GcG3uSmR4dKwnm3Ihrb6tOYBnkNUJjjNkV+D5wFLAJsMC/W2u3GWP2AG4H\npgFvAV+01v46qe+xwC3AnkAbcIG19s2k9i8AVwEjgPuBy6y1W6K2RmA2MDN67nestd9N6lvUs0VE\nRGQH51w4MuOCDBcEBPPm4h94aM1MP+U7QjMPGAIcBnwWOBH4etQ2H3gXmArcDTxkjNkNwBizO/AQ\ncAdwELAGeDhxU2PMqcB1wAXA0cChwE1Jz70ZaAWOBC4FrjfGzExqf7jQZ4uIiEiK5Uv7jsykWr0C\nlr9SmXhykHNCY4zZC/gocK619lVr7bOESciZxpijgA8DF1lrX7PW3kA4EjIr6n4B8IK19nvW2j8B\n5wF7GGM+FrVfDtxirf2VtXYhcBFwvjFmiDFmGHA+cLm1drG1dj5hsnNZFNfRhCMvhT5bREREkrh1\n7+V23frcrquEfEZoVgKftNauSfl4E+GISntiiijye8IpIIBDgGcSDdbazUA7MM0Y4wMHA79L6vsc\nsBNwQPSnnjBJSb73IUn3LujZ2T9lERGRwcdrHpXbdU25XVcJOSc01tr1KetSPMJRkqeACYRTPslW\nAbtFb/fX3kw4jdXTbq3tBtZG7ROANdba7Sl9hxhjRhf5bBEREUk1cTKMGd//NWMnwMR9KxNPDorZ\ntv1t4EDC0ZUrgK0p7VuBxujtYf20D0t6P127n6GNpP6FPjsv9fXx2uWeiDducUN8Y49r3BDf2OMa\nN8Q39rjGDfGNvRpxd3/2c3T98L8yLAz2aPjM+dTttFPW+9TV1ZU+uDQK+soYY24kXPdirLWvGGO2\nAKnjTo2EO5IAttA3gWgEOqM2MrRvimJM10bUXsyz89LS0pJvl5oQ17ghvrHHNW6Ib+xxjRviG3tc\n44b4xl7RuD85g00jm+icfQNB59peTV5zCyNHNjFszJjKxZNF3gmNMeYHhIt2z7LWJnYLvQOkjjuN\nB1YktaeOXY0HFhFOLW2J3l8WPaMOGB3194FdjDG+tTZI6rvZWrvOGFPMs/PS2dnJ9u3bs19YI+rr\n62lpaYld3BDf2OMaN8Q39rjGDfGNPa5xQ3xjr1bc3RvWE3T2Xfjr1r3H2m9ezYbPf4W6qYf1e4/G\nxkZGjhxZrhB75FuH5nrgQuAz1tqHkpqeA64xxjRaaxPTO4ezY6Hvc9H7ifsMI5yuus5a64wxL0Tt\nicW704FtwGLAA7oIFx4viNqPAF4o8tnX5/O5A2zfvp2urq58u1VdXOOG+MYe17ghvrHHNW6Ib+xx\njRviG3sl43bOEfzsh0T1gdNdQdddP6J7v4P7rUVTqWmynJ9ijNkH+CrwX8ACY8y4pObfAn8D5hpj\nvg6cRLi25tyo/U7gKmPM1cCjhMnEG9baRAIzG5hjjFlKuIB3NnBbUmG9u6L2WYSLea8Ezini2a9b\na3+b6+cuIiIy2LhlS2DDuv4vWt+JW7YUb68plQmqH/ls2z4puv6rhEnHu4TTOu9GU0EzCKdyXgTO\nBGZYa98GsNb+hbDK7yzgecKdTTMSN7bW3gd8C/gx8DjhFu1rkp59BbAQeBr4AfC1qB4N0bNPzvPZ\np+TxeYuIiAw67jf/m9t1r71c5khy4zmXaShJkrQCCzs6OmI1RNnQ0MCYMWOIW9wQ39jjGjfEN/a4\nxg3xjT2ucUN8Y6903M45gn87EzZ/kPVa78Qz8E86I2P70KFDE4uZpxLWgSsLHU4pIiIivS1fmlMy\nA8De+5U3lhwpoREREZFecj36AICNG8oXSB6U0IiIiEgvuR59AODm/YxaWL6ihEZERER6y+Xog4Qa\nOXVbCY2IiIj04nke/mnnEZaCy64WTt1WQiMiIiJ9eK3T8E4+M7dra+DUbSU0IiIikpZ3gonNqdtK\naERERCStnqknL0O64Pn4p57b79EHlaKERkRERDLyWqfhfWIG1NX1bqirw/vEDLzWadUJLIUSGhER\nEcnItbfhnngYurt7N3R34554GNfeVp3AUiihERERkbSccwT33AouyHBBQDBvrurQiIiISO1yj9ns\nJ27XSB2a+moHICIiIrXHOYd7+tGcrg1eXYy3bm1YYXji5KosElZCIyIiIn0tXwrvr8/t2l/+Agc4\ngDHj8U87r+KLhTXlJCIiIn3kdUBlso6VBHNurPhiYSU0IiIi0kc+B1T2UYXFwkpoREREpK98DqhM\np8KLhZXQiIiISB+e5+G1Ti/qHpU8tFIJjYiIiPTRU1CvCJU8tFIJjYiIiPTinCN44KeZC+rlosKH\nViqhERERkd6WL4WOlYX3r8KhlapDIyIiIr0UvGUbYOyEMJmpcB0aJTQiIiLSi9c8inw3XHsnnoG3\n9/4wcV9VChYREZEakNiyneu009gJeCd+tiqJTILW0IiIiEgvnufBzHNyvLjy62XSUUIjIiIifXi5\n1pA57JiKr5dJRwmNiIiI9OFWr8jtwr+8jlu2pKLHHKSjhEZERET68MZOyO3Cv71B8O0vE3zlooof\nSJlMCY2IiIj0ddQJ4OWRJnSsJLj1hqolNUpoREREpA/f92G/qXn2cgT3zqnK9JMSGhEREUnLP/6U\n/Dut78Q9dn/pg8lCCY2IiIikl6hHkyf3xEMVH6VRQiMiIiJpeZ6Hf9p5+a2lAdj8AW750vIElYES\nGhEREcnIa52Gf/E14enZ+Xj15fIElIESGhEREckuCPK6vNLLgnWWk4iIiGTk2tsI5twILr+EBr+y\nYyYaoREREZG0nHMED/w0/2QGoO3pii4MVkIjIiIi6S1fmvuJ26lWr4Dlr5Q2nn4ooREREZG03Loc\nD6jM1D/XAy5LQAmNiIiIpOU1jyquf1Nx/fOhhEZERETSmzgZdhlXWN+xE2DivqWNpx9KaERERCQt\nz/Ng+jGF9T1wWti/QpTQiIiISEZe0F1QP7eoTbucREREpDYUnJJol5OIiIjUCm/v/QvuG3SuKWEk\n/VNCIyIiIhl5k6bAyObCOr+5rLTB9EMJjYiIiKTlnMMtWxLudirsBqUNqB86y0lERET6cO1tBPfc\nChvWFXwPb9yuJYyof0poREREpBfX3kZw6w0UdWa278ORnypZTFkfV7EniYiISM1zzhHcfydFJTMA\nUw/Hr+CJ2wWP0BhjGoEXgX+11j4Tfey/gc8TfhW86O/PW2tnR+3HArcAewJtwAXW2jeT7vkF4Cpg\nBHA/cJm1dkvS82YDM4FNwHestd9N6rsHcDswDXgL+KK19tdJ7f0+W0RERAgPpFyzqrh77DQE/4Ir\nSxNPjgpKnaLk4udAak3jfYBrgAnA+OjvO6M+uwMPAXcABwFrgIeT7nkqcB1wAXA0cChwU9K9bwZa\ngSOBS4HrjTEzk9ofBt4FpgJ3Aw8ZY3bL5dkiIiISKvZASgDv/C9UtEowFDBCY4zZB7g3Q/M+wE3W\n2tVp2j4HvGCt/V50n/OAlcaYj0UjPJcDt1hrfxW1XwQ8YYy5mjDxOh843lq7GFhsjLkJuAx40Bhz\nNOHIy6HRiM4NxphjgFnAfxImSf09W0RERAgPpCxqsmnPvfFbp5cqnJwVMkLzceApwqmdnvTLGDMC\n+BCQadP5oUBP8mCt3Qy0A9OMMT5wMPC7pOufA3YCDoj+1BNOFSX8HjgkevsQoD0xPZXUPi2pPe2z\ns362IiIig0kxB1ICfPSI0sWSh7xHaKy1cxJvG2OSm/YhXDPzVWPMPwFrge9aa++K2icQTgklWwXs\nBjQDQ5LbrbXdxpi1UbsD1lhrt6f0HWKMGZ3l3tmeLSIiIhHP8/BPn1XwLidvRFPpg8pBKbdt7w0E\nwCvA9wnXutxmjFlvrZ0PDAO2pvTZCjRGbfTT7mdoI6l/pr7k0J6T+vp47XJPxBu3uCG+scc1bohv\n7HGNG+Ibe1zjhvjGXvG4D/kY3fV1dP3sh3nXoWnYZRx+Q0PP+3V1daWOLq2SfWWstXcZYx6x1iY+\n8yXGmEnAJcB8YAt9E4hGoDNqI0P7pijOdG1E7VuAURn6kuXZOWtpacnn8poR17ghvrHHNW6Ib+xx\njRviG3tc44b4xl7RuD85gy0f2p2Oay/KuYvfMpqxhx1Z8QXBUOLCeknJTMKfgKOit98h3PmUbDyw\niHB6akv0/jIAY0wdMBpYQThCs4sxxrfWBkl9N1tr1xlj3qHvjqvxUd9sz85ZZ2cn27dvz35hjaiv\nr6elpSV2cUN8Y49r3BDf2OMaN8Q39rjGDfGNvVpxb3/rjbyuDzrXsurx+dRNPaznY42NjYwcObLU\nofVRsoTGGPMfwHRr7XFJHz4QeDV6+zng8KTrh0Xt11lrnTHmhag9sXh3OrANWEy4+LiLcGHxgqj9\nCOCFpHtfY4xptNYmppYOZ8ci40zPvj6fz3H79u10dXXl06UmxDVuiG/scY0b4ht7XOOG+MYe17gh\nvrFXMm7X3kbw89vz7td13x1073dwzyhNpabJSvmUXwLXGmOuIKzxcjxwNuFaGgjr0VwVbcN+lDCZ\neCNp2/RsYI4xZinhAt7ZwG1JhfXuitpnES7mvRI4J+r7W+BvwFxjzNeBkwh3TZ3bz7Nft9b+toSf\nv4iIyIDg2tsI5twILsh+carVK2D5KzCpwAMtC1RsTeKe5c/W2heB04B/AV4mrBFzhrX2+aj9L4RV\nfmcBzxPubJqR1P8+4FvAj4HHCbdoX5P0rCuAhcDTwA+Ar0WLjYmmoU4mnEZ6ETgTmGGtfbufZ59S\n5OcuIiIy4DjnCB74aWHJTCR46bkSRpQbz1XwaO8YawUWdnR0xGqIsqGhgTFjxhC3uCG+scc1bohv\n7HGNG+Ibe1zjhvjGXsm43bIlBN/+cnE3aR6Nf9OdeJ7H0KFDE4uZpxLWgCsLHU4pIiIiPUpx9AHr\n1obTThWkhEZERER6eM2pVVAK49aXIDHKgxIaERER2WHiZGgqPqnxSnCPfCihERERkR6e5+GdcWFx\nNxk7ASamlocrLyU0IiIi0os/dTp85JDsF2bgTTum4tWCldCIiIhIL669DRa/kP3CTMZNKF0wOVJC\nIyIiIj1KUYem0utnQAmNiIiIJFu+FDpWFt6/qaXi62dACY2IiIgkKboOzYZ1sKjylYKV0IiIiEiP\nouvQOEcwby6VPolACY2IiIjsMHEyjBlf3D0SB1RWkBIaERER6eF5Hv5p54FXXIqgSsEiIiJSVV7r\nNPyLrwkL5BV6D1UKFhERkWrzWqfhf2MO3mcvyL+zKgWLiIhIrfA8D0Y05dnJxz/13IpXCq6v6NNE\nREQkVrzmUeS8X2nshDCZaZ1WzpDSUkIjIiIiaTnnwu3XI5vD+jKZjGjCu/gavImTKz4yk6CERkRE\nRPpw7W3hEQjZqgZ7Pv7Zl+JNmlKZwDJQQiMiIiK9uPY2gjk3Zj/PqYpTTKmU0IiIiEiPnA6nHNmM\nd9HVVZ1iSqVdTiIiIrJDLodTbliHh1czyQwooREREZEkuR5OWelKwNkooREREZEeuR5OWelKwNko\noREREZEdcjmcsgqVgLNRQiMiIiI9sh5OWaVKwNkooREREZFeMh5OOXYC/sXX1MQ27VTati0iIiJ9\neK3T8A88FJYvxa3vDNfMTNy35kZmEpTQiIiISFqe58GkKdRmCtObppxEREQk9pTQiIiISOwpoRER\nEZHYU0IjIiIisaeERkRERGJPCY2IiIjEnhIaERERiT0lNCIiIhJ7SmhEREQk9pTQiIiISOwpoRER\nEZHYU0IjIiIisaeERkRERGJPCY2IiIjEnhIaERERiT0lNCIiIhJ7SmhEREQk9pTQiIiISOwpoRER\nEZHYU0IjIiIisaeERkRERGJPCY2IiIjEXn2hHY0xjcCLwL9aa5+JPrYHcDswDXgL+KK19tdJfY4F\nbgH2BNqAC6y1bya1fwG4ChgB3A9cZq3dkvS82cBMYBPwHWvtd5P6FvVsERGRwcY5B8uX4ta9h9c8\nCiZOxvO8aodVkIJGaKLk4ufAvilNDwPvAlOBu4GHjDG7RX12Bx4C7gAOAtZE1yfueSpwHXABcDRw\nKHBT0r1vBlqBI4FLgeuNMTNL8WwREZHBxrW3EXzlIoJvfxl3+80E3/4ywVcuwrW3VTu0guSd0Bhj\n9gGeAz6c8vGjCUc/LrLWvmatvYFwJGRWdMkFwAvW2u9Za/8EnAfsYYz5WNR+OXCLtfZX1tqFwEXA\n+caYIcaYYcD5wOXW2sXW2vmEyc5lJXq2iIjIoOHa2wjm3AgdK3s3dKwkmHMjrr0N5xxu2RKC55/B\nLVsSjubUsEJGaD4OPEU4tZM8LnUI0J6YIor8Prou0f5MosFauxloB6YZY3zgYOB3SX2fA3YCDoj+\n1BMmKcn3PqTYZ+f0GYuIiAwQzjmCe24FF2S4ICC4d07sRm/yTmistXOstVelJA8AEwinfJKtAnbL\nob0ZGJLcbq3tBtZG7ROANdba7Sl9hxhjRhf5bBERkUHDPWZhw7r+L1rf2e/oTS0qeFFwGsOArSkf\n2wo05tA+LOn9dO1+hjaS+hf67JzV15fyy1V+iXjjFjfEN/a4xg3xjT2ucUN8Y49r3BDf2EsVt3OO\nrU8UsYTUBbgHf0bDR4/IefFwXV1d4c/LQyn/RbcAo1I+1ki4IynRnppANAKdURsZ2jcRxpmujai9\nmGfnrKWlJZ/La0Zc44b4xh7XuCG+scc1bohv7HGNG+Ibe7Fxb3l5IR2bPyjqHm7VuzStfofGKQcW\ndZ9SK2VC8w59dz2NB1YktY9P076IcGppS/T+MgBjTB0wOurvA7sYY3xrbZDUd7O1dp0xpphn56yz\ns5Pt27dnv7BG1NfX09LSEru4Ib6xxzVuiG/scY0b4ht7XOOG+MZeqri3zvufksTT+dbr1I3LbdVG\nY2MjI0eOLMlz+1PKhOY54BpjTKO1NjG9czg7Fvo+F70PQLRz6UDgOmutM8a8ELUnFu9OB7YBiwkX\nH3cRbuVeELUfAbxQ5LOvz+cT3L59O11dXfl0qQlxjRviG3tc44b4xh7XuCG+scc1bohv7MXE7ZzD\nLcnr9/jMcbz7NkGOcVRqeq+UT/kt8DdgrjHm68BJhDuXzo3a7wSuMsZcDTxKmEy8kSjKR1g0b44x\nZinhAt7ZwG1JhfXuitpnES7mvRI4p4hnv26t/W0JP38REZHatXwpFDndlODansKdcHpNFeEr9uiD\nnk3p0VTQyYRTOS8CZwIzrLVvR+1/IazyOwt4nnBn04yk/vcB3wJ+DDxOuEX7mqRnXQEsBJ4GfgB8\nLapHU+izTynycxcREYkNt+690t1s9QpY/krp7lcCXq0XyqkRrcDCjo6OWA1RNjQ0MGbMGOIWN8Q3\n9rjGDfGNPa5xQ3xjj2vcEN/YSxG3W7aE4NtfLllM3oVfwj/4iKzXDR06NLGYeSphDbiy0OGUIiIi\ng8HEydCUuiG4cF4J71UKSmhEREQGAc/z8M64sDQ3G9kME1M3F1eXEhoREZFBwp86HcZ9qOj7eEd9\nuqYWBENpdzmJiIhIjUocNsmaVcXdqKkF74TTSxNUCSmhERERGeBcexvBAz/tez5T3jz8My+uudEZ\nUEIjIiIyoLn2NoI5N2Y+XTtXdXV4x56M1zqtNIGVmNbQiIiIDFDOuXBkpthkBqC7G/fEwzV72rYS\nGhERkYFq+dISTDMlcQHBvLnUYg07JTQiIiIDVEmrAyfUYJVgUEIjIiIyYHnN5Sl+59aXIVEqkhIa\nERGRAcr9475A6Xck1VqVYFBCIyIiMmB5f36FpHOkS2PshJqrEgxKaERERAascqyh8aYdU5N1aJTQ\niIiIDFTlmBoaN6H09ywBFdYTEREZgFx7G87eUfL71uL6GVBCIyIiMuCUrDpwqhpdPwOachIRERlQ\nSlodOIV/6rk1uX4GlNCIiIgMLKWuDhzxTj67Zs9xAiU0IiIiA4ZzjmDpotLfeEQT3gmnl/6+JaQ1\nNCIiIgOAa28Lp5rKMDrDB+/jFrXhtU4v/b1LRCM0IiIieXLO4ZYtIXj+GdyyJVU/rDFcBHxDeZIZ\ngCDAzbmxZk/aBo3QiIiI5CV1JMQBjBmPf9p5ZVtj4pxjy5J2ut98HTeiCSZO7lmc65wjuOdWKHdS\n5RzBvLn4Bx5akwuDldCIiIjkKON26I6VBLfegH/JtSVPalx7G9vmzaVj9YodH2wahXfGhfhTp+OW\nLYUN60r6zIwSJ21PmlyZ5+VBU04iIiI5yL4d2hHcO6ek00+JBMolJzMA69/DzbmB7vt/ivvNYyV7\nXi6CRc9V9Hm5UkIjIiKSi1y2Q6/vxD12f0kel1M9mScegoXPluR5OXvuN1VfM5SOEhoREZEc5HrQ\no/vNo6X5gV+mejJF27ghnHaqMUpoREREIpl2LwVBgHv91dxusmFd0T/wnXO4Py0u6h7l5NaX/hTv\nYmlRsIiICJl3L/Ghv4c/vghBd+73Wv9euOto+VLcuvfwmkf12pmUTxy1qBYPqFRCIyIig15/u5cK\nSixWrSD4ykW9k6OknUl5x1FLmkfX5AGVmnISEZFBrRyHObpH7umbCEU7k4IH5qad2irnoZIldfDh\nqkMjIiJSc8qx+LafRcHu8QdxbU/31I5JTG1504+p6WmmBP8jh1Y7hLQ0QiMiIoNarruXSiq1EF7H\nStz8eyqcoWo7AAAgAElEQVQfR76aWmpyugmU0IiIyGCXWrRusBkyNPdrN6wDFdYTERGpLa69DTf/\n3mqHUT1jJ8Dhx+V+fXSekwrriYiI1IieRbjU3g/nSvFPPRf/wDzXxCTOc0qjmqeQa1GwiIgMTrVa\nibdCvJPPxmudFiYdu4yDNaty7uvWv0fqPqdMdXy6L7wSph9Vsrgz0QiNiIgMSrV6yGLFjJsAgOd5\neIcdm1fX1MJ6PfVzUhPEjpVs/8VPigozV0poRERk0Ole+Cw8+Ui1w6iq5KTEjRmfe8exE3rtdKqV\n+jlKaEREZFBxztFVoVGDmpWSlHgbN+TWz/PwTz23d2G9Gpm60xoaEREZ0FzSmUrBLmPZurKpJn4A\nV43n901Kho/Mre9+B+G1Tuv1oarU8UlDCY2IiAxYrr2N4P47exa8bgPWNLVUN6hqGjshTGZSkhKv\nZXRue73++gbOuV7JkNc8qib2iSmhERGR2HNpTrZm0XMEt95A6rZst76zOkFWmXfy2XgnnJ72HCb3\n/gbw/OzrYNatDbdsT5rcu28NUEIjIiI1L13CkvjBnHa78PCRsG0bg7nGTA/fx7voavzW9Kd8u/Y2\n3I9vynlRb/KWbeccbt7c0sRZJCU0IiKDQH8JQa3LVN/EP+08gHC7cOoP41wXuQ4GF34pczJTwA6l\nXlu2a2RBMCihEREZ8LoXLoD/+RF88D4QJQRNLXhnXIQ/Nf0PulrRU98k9Qdux8pwOmlkU9W3C9c6\nf0Rz5sZ8E5LULds1siAYtG1bRGRA6/7RN2HODT3JTI/1nbg5NxA8MLcqceUi++iB63tqtfTh1mdO\nOvJKSNLtjkopsFdNGqERERmguu//Kbz0h36vcY8/SLDnpIxTEj3XRWf0uFf/CHV1bJn2cdy43UsS\nZ8bpsBqazoiz1Kq+vdpy3aHUPBr/jAv77I6qpTVKSmhERGIuXULgnIMnHsqt/53/TdDVhdcyuieZ\nSL6nW/0u/PoR2LQRgG6gY/690NSCf+bFaX7I5RF7P+tjXIYDECUPKVNEfUycDGPG9584No/Gu/EO\nPD/NpE4N7RhTQiMiEmOZEgImTcn9Jls3437ynbDvsOGwz0fgTy/1JDAZre8kmHMD/sXXFpTUZF0f\nU0O//cdSuqq+fS7x8E87L/2/A4TTTGdcmD6ZoXZq0IDW0IiIxFZ/BwLy7JOF3XTTRlj4++zJTE8Q\n4ToX5/L7sZbT+hgpnOfBMSfC8BFZ/2281mn4F18TjuYkGzsB/+Jr+k9WEyM8NaCkIzTGmBnAg4T/\nE73o73nWWmOM2QO4HZgGvAV80Vr766S+xwK3AHsCbcAF1to3k9q/AFwFjADuBy6z1m6J2hqB2cBM\nYBPwHWvtd5P69vtsEZG4qZUDAYFwROXmr+KddAbepMzbwZOnsXh/vdbHlJNz8OQjBE8+0jOF119i\n4rVOwz/w0PDfZ31nuO5m4r5Zt/ZnHeGpoFKP0OwLPAKMj/5MAD4Xtc0H3gWmAncDDxljdgMwxuwO\nPATcARwErAEeTtzUGHMqcB1wAXA0cChwU9JzbwZagSOBS4HrjTEzk9ofzvRsEZFSSCyaDZ5/Jlw8\nm+eIRd7PW1ZjC2aXvYy7+csEV52Da2/r0+za2wi+chHBt7+Mu/1m3C9ur0KQg1THSoI5N6b9d0nm\neR7epCn4Bx/Rb2Lap19ihGdEUymiLVip19DsAyyx1nYkf9AYczTwYeCQaFTlBmPMMcAs4D8JE5UX\nrLXfi64/D1hpjPmYtfYZ4HLgFmvtr6L2i4AnjDFXEyZl5wPHW2sXA4uNMTcBlwEPRs/eEzg0w7NF\nRPLWa7Rh9Qrcgqd6r2MZ0YR39KfxTjAlL2Dn2ttwd88u6T1LZsO6PutqMq6VkcpxAcG8ufgHHlqW\ngope6zS8nUfgbv5yye+dq3KM0CxL8/FDgPbEFFHk94RTQIn2ZxIN1trNQDswzRjjAwcDv0vq+xyw\nE3BA9KeecJoq+d6H5PhsEZG89BltmH9P39GS99fj5t9D8IWzCBYuKNmzg/YFBHNuCKdsapVzBPPm\nhqNWtTQ1NtitXhGew1QG4Yikg6E7l+X+uSj1CM1ewCeNMV8B6gjXulxHOPX0bsq1q4DEtE9/7c3A\nkOR2a223MWZt1O6ANdba7Sl9hxhjRufwbBGRnOU92rBpY1jA7viZ+KedW9Szuxc+C7d9O1wfUet6\nfni62poaG+SSz2Eq2T1TdtpVS8kSGmPM3wFDgc3A6YRTTN+PPjYM2JrSZSvQGL3dX/uwpPfTtfsZ\n2kjq39+zc1ZfH69d7ol44xY3xDf2uMYN8Y29HHEHQUDw1C9xq97BG/ch/GNOxPd9nHNsnTe3oNEG\n9/iDeBP3pf6gw3o+1l/szjmC116me+lLuLWrcW8uhxV/Lfhzqgb//XV4gMZmakfD6LH4DQ0lu1/3\nwmfpqpHpxJJ9B7DW/tUYM9pam6hD/UdjTB3hItyfAi0pXRoJdyQBbKFvgtEIdEZtZGjfRPg5pGsj\nat8CpJZJTH52zlpaUj+FeIhr3BDf2OMaN8Q39lLF3Xnn99n48D3Q3b3jg7+4neEzzsIftjNbV68o\n+N7dc79Py2670ziltdc6htTYNy34DZ2zbyDoXFvws2pB4ztvse3N5dUOQyL1u+7O2MOOLNkaGucc\nKx+YWxPJDJR4yikpmUn4E+F00UrCBcPJxgOJ7wzvRO+nti8C1hImJeOJ1udEidLoqL8P7GKM8a21\nQVLfzdbadcaYdwjX9mR6ds46OzvZvn179gtrRH19PS0tLbGLG+Ibe1zjhvjG3l/cPeX6O9fitYzG\nmzSl32/mXfYOuv/3gb4N3d1snHdX0bG6jRvouPYiaB5F/dmXMOTQI/vE3r3wWbp+8E0GQh2WD355\nX7VDKExTS01VwM1o5xG9z+hqagnPtko3Jen5eKeew5o1a0r2+OC1l9m+4u2S3a9YpZxy+gRwL7Bb\n0gLcAwm3YP8OuMoY02itTUz/HM6Ohb7PRe8n7jUs6nudtdYZY16I2hMLh6cD24DFhPVuugi3cidW\n3h0BvJB072v6eXbOtm/fTldXV77dqi6ucUN8Y49r3BDf2Ldt20bXEw/jVq/AGzsB1zwa5s3tPa/f\nNArv4CPwDjxkx3lBkSAIcP9fbkcFFG3de2z/4TfZ9PqrtFz27z1fc+ccwc9vZyAkM3HmHXkC7pF7\na2bkIaOjTsDfZ/9edWNY9BzBvLnhGqaEsRPwTz2XYP+PEpTwtR2sWV2ye5VCKUdoFhBO4/zEGPOf\nwD8Q1oq5kTAR+Rsw1xjzdeAkwp1L50Z97yRMeK4GHgWuB96ItmxDWDRvjjFmKeEC39nAbUmF9e6K\n2mcRLva9Ejgn6vvbLM8WkZjrvPP7bHvobgjCH0AZ04H17+GenI97cj4MGQZTp+PtcwA0j8ItbIOg\nO1PPsgh+NY/OYTvDiWeEH1i+FNasqmgMksa4sEJun8Sgxnj77B+OOiZ/sMACeQU9v4aOPYASbtu2\n1m4EjgfGEI6O3A7MsdZ+J5oKOolwqudF4ExghrX27ajvXwir/M4Cnifc2TQj6d73Ad8Cfgw8TrhF\n+5qkx18BLASeBn4AfM1aOz/qGwAnZ3q2iMRbl70jnA4K8vxtessmePbJ8Ayjm78Cv3m0PAFmsXHe\nXWx/8VmA2K+ZGSi8plFhsbhvzMH/0n/Bpz9T7ZD6amrBmzg5bVOhBfLyVkPHHgB45a5mOUC0Ags7\nOjpiNRTf0NDAmDFjiFvcEN/Y4xo3xDP2IAhwl8zMP5mpNY1D4chPwhPza3+aY6AbOwH/G3N6JQHd\n8++BR2toPZDnFXwgaKm59rasB4k2/MNejP/+PRBW628vVyw6nFJEYss9/Wj8kxmArZvh8YeUzFSb\n56c9nTrn8Y1hw4t8fvYneWN3rZlkBqIKwceeVO0wgNIX1hMRKateRw48+otqhyMx5A9vwg3bGbc6\nqeZqtHA2baKw9wG5jdC0HgrPPpV/4cOmFvwzLwbou25nl3Gw137UDR3G8H/ci80fPZLu7squ9crG\nO/CQcF1alSmhEZGalZy8eM2jCN7fAL+4Dda9V+3QJMaGHfdpuk46i65XXspp4aw3aTJuZHO4JToT\n34ffP5l/MEd/Gv+zF/Q8O3lBL6tW4BY8Cc8+STew/knwHrwbL1PiVS0TJ0O2r08FKKERkZoUtC/A\n/XxH8qLVflIqQw/5ONujhbO5TCd5noc37Wjc4w9mvqiQqc+6OrzPfK5XIuV5HkyaAu1tBGm2jrvV\nK3BzbsS/+JraSWoWPVf1ZAaU0IhIjQmCAHfLdfDqH6sdigxQW5csgvG753y9a2/DPfFwyePwjj0Z\n3++7lDXrgZ5lPjk7H845gvvvrGoMCUpoRKSqeqr5vvpHeOUleOO1aockyT5yCLz7197rOlpG0zhx\nX7Y+n3d90pqw8TFLwzEn5nRtSU4L9/3eIzh1dWEyk+mw0uVLsx/0mDj8c1L6rdsVU0O1k5TQiEjV\nuPY2grtnw/vrqx3KwHb4cTBlKjz4sz4VZL2Z5+AWLoCFv8/4Q7dnLVO03qRh3wNoeHJ+7SU0zaNy\nWl8VdK7FLVsKe+6V/Z65JBfZzPoi3sYNuI6VeGMnwJGfSjsyk+ByXCNWjpOz85VrrJWghEZEqiKs\nX/GtaocxsEW7ZxJrLVzrtPQVZKdOJwiugN88lvaHbmJdR/IPz1KcbVVSno/32Qtx99yaU4Ls1uVW\nxLAUP7D9ll3wDvl4ztfnWoHXa0o9d7nyaqlasBIaEak451w4MiOl4Xlw3Ay8/Q/CvfYyHh7svR9e\nynlV6RKTBN/3IddpmGVLYMvmEgWfo51H4P/LZUCarc1JW66DFX/Dzb8n6+285tE5PbboH9hjJ4Rn\nLOXBvb8B/Lr+j+Io4L5lMXEyjGiqiVFWJTQi0q/kNS4ewN4H4CXm7ZO2VKce9tjvPZctqYlvgDWh\nrh66izjdfNIUvCu/sWMKY6/9ShNXP1w1jmg486Kekab+ziryTjC43zzW766b+l13D/8P53KqfKK8\nfyHTThkK9fXHtbfhfnxT/2t2CrhvWdVIHEpoRAQIE5ctS9rpfvN13Igm3D/uC/97P+6Jh2DzpvAa\ngEfvww0ZBjvt1PNDw0F4kvUZF+JPnZ79WdrB1GP4jDPZ+ODdBS869U8+C6+f9Rhl0Vz5qQ4/aUSl\nv5Emz/Pwz7qEYM6N6b+mnk/TeZfzQY4/hD3Pwz/tvH7v531iBm5RW8ZRo1zltAC5rg7vwi/Vzpbt\n5UtrYss2KKERibXU0RO31/7hN/v16UdNEos7g8614SLFnUfAxg3w5z/R/afFdGzamNuDt2wK/yRb\n/x5uzg10H/wxvI98FJpawoKpr6UZ2Vm2pASfffz5HzuellmXs3XXv6PrJ9/tSRxzVivTDuWW5+fp\ntU5Lf1r22Ak0fOZ8hk0/ig86Okpyv0TS4k49p/gTrnNZgNzdjTe8Kb/7lpFb9Idqh9BDCY1IjUit\nipttCse1txHcc2vvUZJH7+uZ73cAQ3eG407G//RnYNFz4W9/0TfMsi3ke+EZ3AvP9I41EZvvg+cX\nN8UykESjDnVTD6Nr1UqwP8m9bzWnHSq5s6XAz9NrnZZ2aqpup50KCyPD/XqmuvoZNcpVnHY3QfQL\n1YKnqh1GDyU0IjXAtbf1TTbGjMc/7by0Q8u5nHALwOYP4JF7Cf73ftheAydoBwGgAxgTev2Q3v3D\neXQMpzmqNe3gteS2oLZoBUzbJCtFklHO+6VyI5tzu25EbteVm1u2FHId1a0AJTQiVeba29LPz3es\nJEhT4nxHZc48xlhqIZkZiIYNh/r6gtcQ+Pse0PO2t/693P9FXYB74mHcnntVJanxJk3BH7crwap3\ns19cyP1PPANv7/0Lm7aJs7f/ktt1774Fe5d/8XdWry6udgS9KKERqaKCSpzXUGXOWEit0loqno9/\nzuch+TDBDetxv7gtt/47j8CfNGXH+xs35Pf8CpW/D4IgrE+zekVYn+aoE/A8j5bPfZG13/xS/jes\nq4P+ToseOwHvxM8OrkQmoWNF9msAVhdZ6K9E8j1UvNyU0IhUUwElzmupMmet804+Gz51Gt6fX8k/\n4Ug4/pTw8L3+drBE0xDOOdyT83NKOL1/+dfeC7aHj8wvLih7+fvggbm4X8/vqYfiAO67g22fPIUx\nl/07Gz7/Vbrm3ARd2/q/UfMovNNn4TWPxm1cj5uTYVtyrW1HrjBv7ITcCuqNnVD2WHIyfES1I+hF\nCY1IFRWyCLCWKnPWrP4Sjqceyb2myAEHU3faebhTz81pB4vnefinz8q6vsk7fiZ+a+/t7X7L6IJW\nF5VrgWjwwNz0p0u7gOBX8+joXEvdxdfQ/aP7CW6/GV7IcAyC5+OfsaOGjAe4LDuGBq2jTgB7Z/8F\n9erq4MhPVS6mfngjmmrqe5ESGpEqyrfEuXMu3A2Vbdh+sBrZjHfR1X0q5Cb01BTJZUE14H9iZk+/\nXBeDeq3T8C+5luDeObC+s3fjzsPx/vmy9LV6Jk6Gkc15r8cpR/n7IAjCkZl+bHnu/6ifehjeRw6h\n7sIvERx0GO7nt0PykQIZkpRsO4YGK9/3CY47OX0iGak7/pRwGrUGeC2jldCISCSXKqRRDQ7X3kZg\n74C1qysXXy2pr4cxE2DF39K3ez7+WZfgJa9LSXdZ6zS8i6/B3f7trGs5Cq3xkviB7ZYt6fcogmSu\nvS3/xcXlqkPzm8f6HyWIbL/rh/gHfDRMFFun4w7McFZUGuXeMRRX/mnnEkA4dZn8/7OujuEzzmL7\niWfQ1VUji/yLqaJcBkpoRKptj0mZvyFEawpY9NzgPchx6M54nzgF74TT8TwvTOxSpivqd90d79Rz\nCPb/aE639KdOJ/C+hJtzY/qVjSVYy+F5Ht5e++V0FEH3wmdxc27I8wHlW2/ict299P76Xmt4lKSU\nhn/auQQz/6XXYaENx51My7hxdORRELDcslZRrjAlNCJVEAQB7q4fwh/+r//zZI47GbfzcNz3/6Ni\nsdUE34dPnY63zwF9D1hMma5oGD2WsYcdyZo1awjy+M3Vb52Ou/jaqq/lcM7R9fPb8+tUQ+tNaqXI\n20CTelioXyPTTKkyVlFOaB5N3WfOr0gsSmhEKizjYst0npwfnqU02Bx9InUnn5WxOXkkwG9oKHiU\nohbWcmxduiivbfjeGRfiRVuny2bI0NzjKcMaHomX5NdRsO492LAePtgQTbXuT90+H6lIHEpoRCoo\nr2QGylM/JQb8Aw+t2LOqPU3SvSaPKYShw8qfzEDvhb39GTJ0cJwlJVklXkd+exvB07+EjpXRkSe/\noOvgw+H/fa/sMSihEamQIAhwjw/C0ZZ8DZYDFyPdGzqzX5QwYbfKjB6NHpvTZf5+Bw36nUmyQ6aq\n5+69yqz7qc1JOZEByD39KGU8EnJg8LxBV1itLsfzewAYWZnpHW/vA7JfBNQde2L2i2RQyFr1vAI0\nQiMVle+J0gPKsiXVjqDmeSedVRMLXSupbpfcRkMAGLZz+QJJ4k2aHB6U2M82cr9lF/xJUwj6W9Qu\ng0cuVc/LTAmNVEy+J0rHQWqC5vbtZ/Hbtq2VCwxg/G7hluRV7xR/r6E7w5SpsLS9fKfrNo3CO+H0\n8ty7hjVOPhCaWvoW4Utn+tHlD4hoO+5ZlxDMuSHDtnaPlkuv4YPB8suIZFULR7IooZGKyPdE6ThI\nl6BtHdHE2mlH0vXhvQjWd+KGDYe3loc/FCr9gj/qn6g7+kS6Fy6Auf8NWzbnf4/UGjDO4R6zuN88\nVvAJ05l4Z144aEbrEolw9/vr2frhf6Du7Evp/tE3++80bOfeh1mWWbgdN/229obPnM+w6UfxQQ3V\nRJHqqoUjWZTQSNkVdKJ0jcuYoL2/nk1P9F8yviI8D+/IEwComzod1zqNYP498JjNrX9DA96nPtOT\nyOy4rYf36c/gTjC9tjq7jetx837W+wffmPF404+BcbvCqhW4R+7N+H8g3dlGA1X3wgVw1w9h00YC\noANg1BjY9e/g3b9m7Oefc3nFXx+ZtrXX7bRTReOQGKiBqsFKaKT8CjhRupY55wjuubUmKmNm9IkZ\nvQpxeZ6X30Fye0zC/7TJ2Jy61dmDrGXv3a679/1tv3k03mcvSH+20QDU/aNvwkt/6NvQ3y6QKhfR\nq/a2domHWqgarIRGys4tSvMNPN11Mak46h6zJZ9uKak9JlJ32nl9PuyGj8z9HmPG5f3YbD/4aqGI\nXTV13//T9MlMFt7Mc2I3HSuDU6aqwd6oMRV5vhIaKSvnHO6F3+V0bRwqjjrn8iuMV2nHzaDOzErb\n5LeMJuffm6YfU7KQkg3W3/aDIIACKz67u36Aa502aBI/ibd0v7g07D+1Is9WQiPltXwprM9hMWzz\n6FgUU3PLlhS2uLbcDj8W758v6/+8l4mTYZdx2cvsDxte0cWng0FYg6hAmz4gWLaEuhwOuRSpBX2m\npCuUjKuwnpRVzlv5mlti8Ruoe+WlaofQh3fxtdSdc3nWw+s8z8M/fRZkGR9pOP8Lsfi3iJU/PFNc\n/wVPlyYOkQFMIzRSVjlv5fvL6wRBULUTZRPbaIPOtXgbN8DwkXgto3H/uC/en18h6Fwb1gmppemm\noTvjn3t5XusrvNZp+JdcS3DvnL51T5paGH3Zl/lg4pS8Tq2W/jnn4J23irvJlk0liUVkIFNCI+U1\ncTLsPBw+yFKMzTnc/z0GR1e+lHragn9p/q6IYcOzF64bNhyOOxn/BFPQSEpijtstW4J77eXoRNz9\n2GnfjzBs7FjVFim15Uuha1tx92geXZpYRAYwJTRSVp7nwZgJ8MHy7Be/uqTiCU1YTyZDNdRKG7Yz\n3i134/35Fdz6ThjZAkS7v97fACNG4kVrjYqdEvI8D2+v/SBpXYammcqjJBVU99y7+HuIDHBKaKT8\ndh6e23XvvIVzriw/WIMggN88hlu9Am/sBDjqBDzPi+rJ1EAyQ1Q4zfcH5S6ggawUFVT9Fo3QiGSj\nhEbKb4+JsHRR9utWryC46hz8sy6BQz5WsscHD8zF/Xo+BN1ANIVk78S1Tq+NejIjW/DPuli1Rgaq\niZNhRBO8v76w/jHZAShSbUpopOT6nKi99wG5l9zfsI5gzo1019fBJ2f0uW/w6h/De/31DfD9sLLw\nhVdTV5/+v3LwwNz0dWOCbngxt/o4ZfXRj+F/7kpN9wxgnufhHf1p3Px7Cut/xgX6/yGSAyU0UlJp\nF9jWN+R5k4Dt9k7c8Sf3fChoX4C7/TuwPWX3zaLn4JKZdEN4Fo7vw257wG4fhpcXwmt/LOKzKaNB\nVvJ/sPNOMIUd6Hn8KYPmjCuRYimhkZLJeGBjahKSy71Wvcu2pS/BuN3ofuCn8HgOVVYTB/u9/Rbw\nf3k/sxK8407G+8ihg6rkv0Q1gM66JL9zbj5xStojLEQkPSU0UhLlOLBxy+Ln2bLkx/DHF0t2z0oY\ncfbFvP/E/D4nT/unnad1MoNYpnNu+mgcgnfeFzR6J5InJTSSl57dQqveBc+DPfcKFzz+/LaSL7Dd\ncO/tJb1fRQzbmabPns+2Y06i65WXBuUhjJJZ8jk3wXtroL0Nf/W7BF3b4B/2gcOOwZ80Rf9XRAqg\nhEaycs6FRdgevAveeK13YzFn1AxADed/MVwE6nl42n4taSTOuakDGo44jjFjxtDR0UGXqjOLFEUJ\nTRkl7/ZxI5vh7b9Ax446KNUq858P195GcPfswrecDhYjmvDPvpS6qYdVOxIRkUFJCU0eur9xJd1/\nfiV8p6ERJh8IF6XfMhy0L8D9z2zYuKFPmwP4xe3hzhzPh5FN8IlT8I49qapnGQWvvQwLnoLNm8Pa\nF+s7YdGCqsQTG5Om4J18Jt7EyZomEBGpIiU0+XDdO97u2govRVuG95hI3Ve+09OUsfZJ2nsGYeJw\n/524+++ke8+98Gaegzep9w/I5MMT2bAOXn8NliyErZvDC7y6cMtyXR0MGQojR9I9vIkVWzazvXEI\nrFkFW7bAti3h2pfAhdf7XrgLafv2UnyFBh1vr/3wJ02pdhgiIoOeEppSeGs53d+8krqvfIdg4YLc\nk5l03ngNd/OX8y+V7rqhuxu6u8KkZUN4krLSlDLbe7/s14iISNkNmoTGGNMIzAZmApuA71hrv1uy\nB7y1nO6uLrjzeyW7pVRZfUP/NXSaWvAmTq5cPCIiklHtr0otnZuBVuBI4FLgemPMzJI+4ZbrwtER\nib+PHIJ/wVXh9Fw6nod/5sVaNyMiUiMGxQiNMWYYcD5wvLV2MbDYGHMTcBlQxPxQihV/LdmtpEom\nt8Lnv0ZdXR0A/sXX9i2ENnYC/qnnqkieiEgNGRQJDXAA4efalvSx3wNfLulTuktXJVey846fiVv4\nbLjgueeDdTBqFxgyBLZsgvc3wLatO9rr6mDKwbB9G+zUCKNGw4f3xm8Znbb4XXIhNBXJExGpXYMl\noZkArLHWJq+RXQUMMcaMttauLclTxoyHv75eklsNOA2NMKUV3vkrrH6nuHuNbMY/6xK81mm4U8/p\nN9noqQVURDKSKISmFEZEpHYNloRmGLA15WOJ9xtz6D8EoH63PTJe4H3o7/E+9PcEDYPlS5qF74cn\nSu83lbo998L/+3/E8zyCIKDrPy4v/LZHfYr6j/9T76TkgIP775StvYQSU1WNjY3Up6lPVMviGntc\n44b4xh7XuCG+scc1boCGhobEm0PK+Zx4fVUKt4W+iUvi/U059N8DYJerv5n9yrMvyiOsQer791Q7\ngrIbOXJktUMoWFxjj2vcEN/Y4xo3xDf2uMYd2QMoW7XWwZLQvAPsYozxrbWJhS7jgc3W2lxOVHwc\nOAt4izA5EhERkdwMIUxmHi/nQwZLQvMS0AUcyo7s8AjghRz7rwXuLUNcIiIig0HZz9HxnMu7Jm0s\nGWNuBQ4DZgG7AXOBc6y186sZl4iIiBRvsIzQAFxBWCn4aWA98DUlMyIiIgPDoBmhERERkYFrMB19\nIP1huH0AAAeNSURBVCIiIgOUEhoRERGJPSU0IiIiEntKaERERCT2lNCIiIhI7A2mbdsDgjGmEXgR\n+Fdr7TPRx44AbgH2BpYBX7LWPpWm71nA56y1R6Xc72bAAA54GPiitXZzLcRujLkC+DdgF+AZ4PPW\n2j9XMvYC414M7BfF5UV/72etfaWWv+bGmDeBv09zq+ustd+o8a/5PwNfITyM9kngUmvtqqT7VfNr\nPhX4AeH/iZejZ/8hTd+qvUYLibsWXp9FxF7112i+cdfK67OQ2KP2sr5GNUITI9E/+M+BfZM+NgZ4\nhLCS8RTgfmC+MWbXlL5HAT8m/I+S7P8RVk3+JHBC9PZ/1ULs0Tf3rwIXAvsTVmz+ZSVjLzBuH5gY\nxTOB8JiNCcCrlYq70NiBg6J4E38+D6wjLERZkdgL/JofD9wJ/DdwMPAB8Kuk25Y97iyxPwksBqYC\nFvi1MWa3lL5Ve40WEnctvD6LiL3qr9EC/69U/fVZaOyVeI1qhCYmjDH7kP74hcOALmvtd6P3v2WM\nuZLwmIcHo77XA9cS/lab6p+A26y1i6JrbyX8BlULsY8ErrbWPh7d50bgJWPMLtbaNeWOvYi4Pww0\nAC9Ya7el6V+zX3Nr7dqke4wErgOusNa+XYnYi/iaXwbcba29NbrPhcDbxpjjrLW/LnfcWWI/B1hD\n+NuoA5YZYz4BXEL422pVX6NFxF3V12eRse9JFV+jhcZd7ddnMbFTgdeoRmji4+PAU8A0wuHRhLXA\naGPMKQDGmBnAcMLhvoRjgOOIEpwUa4HTjDHNxpgWYCbQXguxW2tvtdb+JGprInxBLIm+WVYi9kK/\n5vsCf8vwjbIScRcTe7IvAe9aa+dWMPZ84/5j1L4n0DO0ba3dAvw5uk8l4u4v9g8DC6Nv8gl/TIoN\nqvsaLSjuGnh9Fhw7sA/VfY0W838loRqvTyg89rK/RjVCExPW2jmJt40xyR//nTFmNvCAMSYgTFLP\ns9YuT7rmY1G/Y9Lc+kuE30QTmf8fgRNrJfaoz3nAHYQnnR9fqdiLiHsfoMsY80vCIeLXCNd7JA5D\njcPXfCjhD6gLUm5da1/zP0eXrAI+lNTXi97fpRJx9xd7FNv+KZf/XVJsVX2NFhN31Kcqr08oKvaq\nvkZL8DWvyusTioq97K9RjdDEnDFmOGHmex3hvOQ3gR8YYybleIuJwF+AI4FPEB7zfkvpI+0rj9h/\nDRwI3A48YoxJLIqrSuw5xL030AzcRjiM+grwlDEm8WKOw9f8s8D79B0xqNWv+X3AJcaYQ40x9YRD\n3GOBnaoZd2QecIgx5nPGmLpoLcFJSbFlU63Yc427pl6fkWyx1+prNNeveU29PiPZYi/7a1QJTfxd\nA2Ct/aa19iVr7XWEw3r/lq2jMWYE8BPgSmvt76IdI+cDs4wx48oZdCSn2K21b1trF1tr/w34G3BO\nlWPPFvfngH+w1v4yar8UeBP457h8zYFTgfustUHiAzX+Nb+dcJHi74BNhNN+/wtsqPbX3Fq7lPA3\n6e8SjmJ8A/gRsCFb32rGnmvcNfj6zCX2mnyN5vF/pdZen7nEXvbXqBKa+GslXFWebBHpt/al2hsY\nxo51CIm+PrB7SaLrX7+xG2OOTDNy8CfCIcpqxt5v3NbawFq7MaX9VcLh1Zr+mgMYY3Yi/C3p4ZTr\nav1r/nlgBDDWWnsmsCvwFtX/mmOt/RnQBOxm7f/fzt2DVhFEYRh+U9naia0gKCjEwl5QhIAgWBwI\ngtglXVIIVoLYaWMhJCAiJCDC6SQIaiGKlYFgrBIs1MYLYmGVIFjE4qxxuSa7c9HLzMD3VPnZuXwM\nc5Kzf+Onmx9/ThiaNXtX7oLrE+jOXnKN9q2VQusTSJrzsdaonqGp34DWq3ONY8TZRspYmvHrzdfH\niddGU8b/q77s14lLkLOw+6rlJHEZckA8kJYje2duM3sJvHL3W833E8S95XuZc0PaejlJ/G1Y3WMs\nlDnn88ABd78N/DCzw8RtkKv8OUPMMudmdgaYcfdp4GuzHqaAxYTh2ea8I/dCc0ip9dmbvdQaTVwr\nJdZnypyPvUbV0NTvAfDGzOaIfTouEg/mTfYNdPcvZvYcuG9ms0Q3vAg89tbrgWPUl30BcDN7DawB\n14j7qsvuvm1mzzJl78u9Atwws3fEw4bzxFnLkrtvZczdlf1U65gTwEd3/9kemHm99M35J+Chmb0F\nvhH7uay4+wZA5nX+AbhgZjPAC+Lhx4PAUt/AzHO+X+7l5vel1mdK9lJrNGWtlFifXdl/z/nYa1S3\nnOq0+1qcxy6Ml4gu9z1wGZhy9829h/5lmrjM95T4R7HKf963YEhydndfIfYwuElcfjwCnHf37QzZ\nR8l9F7hDnO2tE2caZ919K0Pu1OwbreMPAd/3+axS5/wJMeePiB1rN4ErmXIPZx8Qu5/ONRmOAuda\n67hPrjnvzF1YfY6avaQaHXWtlFKfMNqcj71GJ3Z2hjelFBEREamLrtCIiIhI9dTQiIiISPXU0IiI\niEj11NCIiIhI9dTQiIiISPXU0IiIiEj11NCIiIhI9dTQiIiISPXU0IiIiEj11NCIiIhI9dTQiIiI\nSPV+AckeVnvZ99srAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f567cda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot_date(yearIndex,sumYear.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that it increases a lot over time! \n",
    "\n",
    "\n",
    "Now let's look at one word that we suspect having a periodicity over one year!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1022c6dd8>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgUAAAFoCAYAAADHHogUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xm4HVWd6P1vnXPg2GmBnGBCgkEBBYHYNqMQQGyUVrvf\nVwTEpeJ7r4AivLbd/WorXIer/YpNExzAqY0DQ3sdV6uAjfe9NK39MhmaIYiCCIhAAyEhkBC1ISEn\np+4fVTtsds5Qtedd5/t5Hp6cvVdV7fWj9q761VqrViVpmiJJkjTU6wpIkqT+YFIgSZIAkwJJkpQz\nKZAkSYBJgSRJypkUSJIkwKRAkiTlTAokSRJgUiBJknIjZRYOIbwduBhIgaTu34kY40gIYQ/gK8BS\n4H7gvTHGq+rWPwY4H9gTWAGcFmO8rw1xSJKkFpVtKfgOsBBYlP/7QuDXwAV5+WXAKuAg4BvApSGE\nxQAhhN2AS4ELgYOBx/LlJUlSH0haefZBCOGDwCnAEuAVZCf5BTHGjXn5VcC1McaPhxA+DhwZY3xV\nXvYHwGrg9THGa1oLQ5IktarpMQUhhDHgTOCsGONm4FBgZS0hyF1H1pVAXr715B9jfApYWVcuSZJ6\nqJWBhu8GHo4xXpq/XkTWdVBvDbC4YLkkSeqhUgMNG7wDOLfu9RxgU8Mym4DRguUz2Rl4LdkAxo3T\nLypJkuo8B9gduBJ4fKqFmkoKQgiHAM8Hvlv39kZgXsOio8CTdeWNCcAosL7gx74W+Ga5mkqSpDpv\nA741VWGzLQWvBa6JMW6oe+9hYL+G5RYCj9SVL5yk/NaCn3k/wG9/+1u2bNlSqrL9aHh4mB133LEy\n8TQyvsFX9RirHh9UP0bjK78t8nPpVJpNCg4Frm947wbgrBDCaIyx1k1wJHBtXfmRtYVDCHOAA4CP\nFfzMjQCbNm1i8+bNTVa7f2y33XZAdeJpZHyDr+oxVj0+qH6Mxld+W8zQ/d5sUvBS4H80vHc18CBw\nSQjhbOBY4BDg5Lz8IuD9IYQzgSvIkoF7Y4xXN1kHSZLURs3efbCAhrEAMcYJ4A1kXQI3AycBx8UY\nH8rLHwBOAE4FbgTmAsc3+fmSJKnNmmopiDH+4RTv/wY4epr1rgT2aeYzJUlSZ/lAJEmSBJgUSJKk\nnEmBJEkCTAokSVKulWmOJUnquDRN4Z47SJ9YRzJ3Huy1hCRJel2tSjIpkCT1rXTlCia+dzGsXZ29\nBpi/kKETT4FDj+pp3arI7gNJUl9KV65gYvmyrQnBVmtXM7F8GVtuaZxYV60yKZAk9Z00TbMWgnRi\nigUmGI8XZV0LahuTAklS/7nnjm1bCBqka1bx9B0/61KFZgeTAklS30mfWFdouS3r1na4JrOLSYEk\nqe8kc+cVWm543vwO12R2MSmQJPWfvZbA/IXTLpLssivbL9m/SxWaHUwKJEl9J0mS7LbDZIrTVDLE\nSDjV+QrazKRAktSXkgOXMnTGWbBg0bMLFixi6IyzGD7oiN5UrMKcvEiS1LeSA5cydMBh2YyGG9aT\n7DQP9trPFoIOMSmQJPW1JElg75diGtB5JgWSJPXYZM936AWTAkmSemiq5ztsecs74XXHdbUuJgWS\nJPXI1uc7NE7nvHY1m79wDk/uuBPs9dKu1ce7DyRJ6oEiz3fYcPHnuvp8B5MCSZJ6ocDzHcZXPUh6\n9x1dqpBJgSRJPVH0+Q7pE493uCbPMCmQJKkHij7fIZm7c4dr8gyTAkmSeqHA8x1Gdt2NZO/u3Z5o\nUiBJUg8Ueb7DTqf8VVdnbzQpkCSpR6Z7vsN27/kQcw4/uqv1cZ4CSZJ6aKrnOwxvv33X62JSIElS\nj/XL8x3sPpAkSYBJgSRJypkUSJIkwKRAkiTlSg00DCFsD5wPvBXYBFwUY/xwXrY78FVgKXA/8N4Y\n41V16x6Tr7snsAI4LcZ4X+shSNLUJntOfTfv+5YGSdmWgs8Brwb+FDgJOC2EcFpedjmwCjgI+AZw\naQhhMUAIYTfgUuBC4GDgMeCylmsvSdNIV65g4sOnM/HJD5F+9VNMfPJDTHz4dNKVK3pdNakvFU4K\nQghjwKnAO2OMt8QY/w34FHBoCOFoYA/g9BjjXTHGc8laA07NVz8NuCnGeEGM8U7gFGD3EMJR7QxG\nkmq2Pqe+8Sl0a1czsXwZW265vjcVk/pYmZaCI4EnYozX1d6IMZ4XY3wncBiwMsa4sW7568i6EgAO\nBa6pW+8pYGVduSS1TZHn1I/Hi7r6nHppEJQZU7AncH8I4b8AHwK2By4G/g5YRNZ1UG8NsDj/e6Zy\nqW9N1ietPlfgOfXpmlU8fcfPYBcPQ1JNmaTgucDewLuAk8lO9F8GngTmkA08rLcJGM3/nqlc6kvp\nyhXZFWd+gkkB5i9ky1veCa87rqd109SKPqd+y7q1JgVSnTJJwTiwA/DWGONDACGEFwLvBv4FaHzg\n8yhZwgCwkW0TgFFgfekKj1RjZuZaHFWJp1EV4ttyy/VsXr5s2ybotavZ/IVzeHLHnRjZd//eVK4L\nBnkfTjxvAU8XWG543vyBjK+oQd6HRRhf+W3NuFyJbT4CbKwlBLm7yLoAHgYa21QX5uuQlzc+NHoh\ncGuJzwdgbGys7Cp9rWrxNBrU+NI0ZfX3Lpm2T3rDxZ9j4Vd+UPnb2wZxH6bPO5rVixYz/shDUy4z\nsutubL9kf0Yrvv9gMPdhGcbXPmWSghuA54QQXhxj/HX+3n5kcxLcAHwwhDAaY6x1ExwJXFu37pG1\nDYUQ5gAHAB8rW+H169czPj5edrW+MzIywtjYWGXiaTTo8U3c9YtpTygA46se5PEbriV98b5dqlV3\nDfo+TE48Gb5wzuSJXTLEyJtOJUmSgY2viEHfhzMxvvLbmnG5ohuMMd4dQvgRcEkI4d1kYwrOAj5O\ndmfBg3nZ2cCxwCFkYw8ALgLeH0I4E7iCLBm4N8Z4deGIcuPj42zevLnsan2ravE0GtT4Jh57tNBy\n448/ysQLX9zh2vTWoO5DXvZyhs44i4nvXwKPPvLM+wsWMfTGk+GAw4ABjq+EqsdofO1TtqPibcDn\nyVoAngQ+F2P8IkAI4ViyyYluBn4NHFfraogxPhBCOAH4LPBR4Hrg+LZEIHVAMnceRW5WS+Y2DqVR\nP5nqOfVV7/KRmlUqKYgx/o7s6v/kScp+Axw9zbpXAvuUq57UI3stgfkLp72tbWTX3Uj2XgIVbLas\nkn55Tr00CHwgkjSJJEkYOvEUSKb4iSRD7HTKX3nFKalSTAqkKSQHLmXojLNgwaJnFyxYxHbv+RBz\nDp+yYUySBlI1b+6U2mSqPunh7bfvddUkqe1MCqQZ2Cctabaw+0CSJAEmBZIkKWdSIEmSAJMCSZKU\nMymQJEmASYEkScqZFEiSJMCkQJIk5UwKJEkS4IyGkjTw0jTNpuJ+Yh3J3Hmw1xIf1qWmmBRI0gBL\nV65g4nsXb33Mdwowf2H2lM9Dj+pp3TR47D6QpAGVrlzBxPJlWxOCrdauZmL5Mrbccn1vKqaBZVIg\nSQMoTdOshSCdmGKBCcbjRVnXglSQSYEkDaJ77ti2haBBumYVT9/xsy5VSFVgUiBJAyh9Yl2h5bas\nW9vhmqhKTAokaQAlc+cVWm543vwO10RVYlIgSYNoryUwf+G0iyS77Mr2S/bvUoVUBSYFkjSAkiTJ\nbjtMpjiMJ0OMhFOdr0ClmBSoJWmakt59OxM3XkN69+2OdJa6KDlwKUNnnAULFj27YMEihs44i+GD\njuhNxTSwnLxITXPSFKn3kgOXMnTAYdmMhhvWk+w0D/bazxYCNcWkQE3ZOmlK4z3StUlTRobhdcf1\npnLSLJMkCez9UkwD1Cq7D1Sak6ZIUjWZFKg8J02RpEoyKVBpTpoiSdVkUqDSnDRFkqrJpEDlOWmK\nJFWSSYFKc9IUSaomk4IOq+rkPk6aIknV4zwFHVT1yX2cNEWSqqV0UhBCOA74Adk5Lsn//X6MMYQQ\ndge+CiwF7gfeG2O8qm7dY4DzgT2BFcBpMcb7WoyhL82WyX2cNEWSqqOZ7oP9gB8CC/P/FgHvzMsu\nB1YBBwHfAC4NISwGCCHsBlwKXAgcDDwGXNZK5fuVk/tIkgZRM90H+wK3xxifdRN6COFVwB7AoTHG\njcC5IYRXA6cCHwdOA26KMV6QL38KsDqEcFSM8ZpWgug7ZSb32WVxlyolSdL0mm0puHuS9w8FVuYJ\nQc11ZF0JtfKtJ/8Y41PAyrryynByH0nSIGqmpeAlwOtCCB8GhoF/Aj5K1o2wqmHZNUDtUnim8spI\n5s6jSMeAk/tIkvpJqaQghPAC4A+Ap4A3kXUXfC5/bw6wqWGVTcBo/vdM5cUqPNL/N0yk++3P0wsW\nkT76yJTL1Cb3GXniiS7WrHtq+2kQ9lczqh4fVD/GqscH1Y/R+Mpva8blymw0xvgfIYSdY4y1M9nP\nQwjDZIMKLwbGGlYZBZ7M/97ItgnAKLC+TB3Gxho/oj89edr7ePzvz4KJSQYbDg0x753vJUmSgYmn\nWcY3+KoeY9Xjg+rHaHztUzr9qEsIau4EngOsJhuEWG8hULtcfjh/3Vh+a5nPX79+PePj42VW6Y29\nXsp2f/Gh7C6DNc/0miS77MpIOJVN++7PHAYonpJGRkYYGxszvgFW9RirHh9UP0bjK7+tGZcrs9EQ\nwmuAbwGL6wYUHkB2e+G1wPtDCKMxxlo3wZH5+wA35K9r25qTr/uxMnUYHx9n8+bNZVbpnZe9nOSP\nDiFpmNxnIkm27uCBiqcJxjf4qh5j1eOD6sdofO1TtqXgp2TdAV8LIXwceBFwHrCM7M6CB4FLQghn\nA8cChwAn5+teRJY0nAlcQZYM3BtjvLrVIPqZk/tIkgZFqVsSY4y/B14LzAduIpu9cHmM8dMxxgmy\nRGAhcDNwEnBcjPGhfN0HgBPI5i24EZgLHN+mOCRJUouaGVNwJ1liMFnZb4Cjp1n3SmCfsp8pSZI6\nz6ckSpIkwKRAkiTlTAokSRJgUiBJknImBZIkCWjugUiSKiRNU7jnDtIn1pHMnQd7Lcnm15A065gU\nSLNYunIFE9+7GNauzl4DzF/I0ImnwKFH9bRukrrP7gNplkpXrmBi+bKtCcFWa1czsXwZW265vjcV\nk9QzJgXSLJSmadZCkE7yFE+AdCJ7mFeadrdiknrKpECaje65Y9sWggbpmlU8fcfPulQhSf3ApECa\nhdIn1hVabsu6tR2uiaR+YlIgzULJ3HmFlhueN7/DNZHUT0wKpNloryUwf+G0iyS77Mr2S/bvUoUk\n9QOTAmkWSpIku+0wmeIQkAwxEk51vgJpljEpkGap5MClDJ1xFixY9OyCBYsYOuMshg86ojcVk9Qz\nTl4kzWLJgUsZOuCwbEbDDetJdpoHe+1nC4E0S5kUSLNckiSw90sxDZBk94EkSQJMCiRJUs6kQJIk\nASYFkiQpZ1IgSZIAkwJJkpQzKZAkSYBJgSRJypkUSJIkwKRAkiTlTAokSRJgUiBJknImBZIkCTAp\nkCRJOZMCSZIEwEizK4YQfgSsiTGemr/eHfgqsBS4H3hvjPGquuWPAc4H9gRWAKfFGO9ruuaSJKmt\nmmopCCG8BfizhrcvA1YBBwHfAC4NISzOl98NuBS4EDgYeCxfXpIk9YnSSUEIYQw4D7ix7r1XkbUA\nnB5jvCvGeC5Za8Cp+SKnATfFGC+IMd4JnALsHkI4qtUAJElSezTTUvAp4OvAnXXvHQqsjDFurHvv\nOrKuhFr5NbWCGONTwMq6ckmS1GOlkoK8ReAVwNkNRYvIug7qrQEWFyyXJEk9VnigYQhhFFgOvDvG\nuCmEUF88B9jUsMomYLRgeWEjI02PjewrtTiqEk8j4xt8VY+x6vFB9WM0vvLbmnG5Etv8W7JxAf86\nSdlGYF7De6PAk3XljQnAKLC+xOcDMDY2VnaVvla1eBoZ3+CreoxVjw+qH6PxtU+ZpODNwC4hhN/l\nr0cBQggnAucA+zUsvxB4JP/74fx1Y/mtpWoLrF+/nvHx8bKr9Z2RkRHGxsYqE08j4xt8VY+x6vFB\n9WM0vvLbmnG5Ett8JbBd3evzgBQ4E9gd+G8hhNEYY62b4Ejg2vzvG/LXAIQQ5gAHAB8r8fkAjI+P\ns3nz5rKr9a2qxdPI+AZf1WOsenxQ/RiNr30KJwUxxgfrX+ctBmmM8b4QwgPAg8AlIYSzgWOBQ4CT\n88UvAt4fQjgTuIIsGbg3xnh16yFIkqR2aMs0xzHGCeANZF0CNwMnAcfFGB/Kyx8ATiCbt+BGYC5w\nfDs+W5IktUfTQxpjjKc0vP4NcPQ0y18J7NPs50mSpM7ygUiSJAkwKZAkSTmTAkmSBJgUSJKknEmB\nJEkCTAokSVLOpECSJAEmBZIkKWdSIEmSAJMCSZKUMymQJEmASYEkScqZFEiSJMCkQJIk5UwKJEkS\nYFIgSZJyJgWSJAkwKZAkSTmTAkmSBJgUSJKknEmBJEkCTAokSVLOpECSJAEmBZIkKWdSIEmSAJMC\nSZKUMymQJEmASYEkScqZFEiSJMCkQJIk5UwKJEkSYFIgSZJyI2VXCCG8CPgicATwOPCFGOOn8rLd\nga8CS4H7gffGGK+qW/cY4HxgT2AFcFqM8b7WQpAkSe1QqqUghJAAPwLWAPsDZwAfCSG8JV/kcmAV\ncBDwDeDSEMLifN3dgEuBC4GDgceAy9oQgyRJaoOy3Qe7ALcC744x3htj/F/Aj4EjQwhHA3sAp8cY\n74oxnkvWGnBqvu5pwE0xxgtijHcCpwC7hxCOakskkiSpJaW6D2KMq4G31l6HEI4AXgG8GzgMWBlj\n3Fi3ynVkXQkAhwLX1G3rqRDCyrz8GiRJUk81PdAwhHA/2cl8BfADYBFZ10G9NcDi/O+ZyiVJUg+V\nHmhY5wRgIfAlssGDc4BNDctsAkbzv2cqL2RkpJUq949aHFWJp5HxDb6qx1j1+KD6MRpf+W3NuFyz\nHxBjXAkQQngf8E2yAYRjDYuNAk/mf29k2wRgFFhf5nPHxho/YrBVLZ5Gxjf4qh5j1eOD6sdofO1T\nKikIISwAlsYYL697+5fA9sAjwL4NqyzM3wd4OH/dWH5rmTqsX7+e8fHxMqv0pZGREcbGxioTTyPj\nG3xVj7Hq8UH1YzS+8tuacbmS290D+EEIYXGMsXayPxh4lGxQ4QdCCKMxxlo3wZHAtfnfN+SvAQgh\nzAEOAD5WpgLj4+Ns3ry5ZLX7V9XiaWR8g6/qMVY9Pqh+jMbXPmWTgpuAm4GL8m6DPYDzgE+QDTp8\nELgkhHA2cCxwCHByvu5FwPtDCGcCV5AlA/fGGK9uNQhJktS6UncfxBgngDcA/wn8FPgKcEGM8Qt5\n2bFkXQI3AycBx8UYH8rXfYBscOKpwI3AXOD4NsUhSZJaVHqgYT5XwYlTlP0GOHqada8E9in7mZIk\nqfN8IJIkSQJMCiRJUs6kQJIkASYFkiQpZ1IgSZIAkwJJkpQzKZAkSYBJgSRJypkUSJIkwKRAkiTl\nTAokSRJgUiBJknImBZIkCTApkCRJOZMCSZIEmBRIkqScSYEkSQJMCiRJUs6kQJIkASYFkiQpZ1Ig\nSZIAkwJJkpQzKZAkSYBJgSRJypkUSJIkwKRAkiTlTAokSRJgUiBJknImBZIkCTApkCRJOZMCSZIE\nmBRIkqTcSJmFQwi7Ap8DjgaeBCLwwRjj0yGE3YGvAkuB+4H3xhivqlv3GOB8YE9gBXBajPG+NsQg\nSZLaoGxLwfeB5wBHAG8BXg+cnZddDqwCDgK+AVwaQlgMEELYDbgUuBA4GHgMuKzVykuSpPYpnBSE\nEF4CvBw4Ocb4qxjj9cBHgZNCCEcDewCnxxjvijGeS9YacGq++mnATTHGC2KMdwKnALuHEI5qZzCS\nJKl5ZVoKVgOvizE+1vD+TsBhwMoY48a6968j60oAOBS4plYQY3wKWFlXLkmSeqzwmIIY4wagfoxA\nArwH+DGwiKzroN4aYHH+90zlkiSpx1q5++CTwAHAh4E5wKaG8k3AaP73TOWSJKnHSt19UBNCWAb8\nFRBijL8MIWwE5jUsNkp2hwLARrZNAEaB9WU/e2SkqSr3nVocVYmnkfENvqrHWPX4oPoxGl/5bc24\nXNkNhxA+D5wOvC3GWLuD4GFgv4ZFFwKP1JUvnKT81rKfPzY2VnaVvla1eBoZ3+CreoxVjw+qH6Px\ntU/ZeQo+BrwLeHOM8dK6ohuAs0IIozHGWjfBkcC1deVH1m1nDlnXw8fKVnj9+vWMj4+XXa3vjIyM\nMDY2Vpl4Ghnf4Kt6jFWPD6ofo/GV39aMyxXdYAhhX+AjwDnAT0MIu9QVXw08CFwSQjgbOBY4BDg5\nL78IeH8I4UzgCrJk4N4Y49VFP79mfHyczZs3l12tb1UtnkbGN/iqHmPV44Pqx2h87VNmoOGx+fIf\nIbuTYBVZ98CqGOMEcBxZl8DNwEnAcTHGhwBijA8AJ5DNW3AjMBc4vk0xSJKkNihzS+IyYNk05feS\nTX88VfmVwD6laidJkrrGByJJkiTApECSJOVMCiRJEmBSIEmSciYFkiQJMCmQJEk5kwJJkgSYFEiS\npJxJgSRJAkwKJElSzqRAkiQBJgWSJClnUiBJkgCTAkmSlDMpkCRJgEmBJEnKmRRIkiTApECSJOVG\nel0BSZLqpWkK99xB+sQ6krnzYK8lJEnS62rNCiYFkqS+ka5cwcT3Loa1q7PXAPMXMnTiKSQHLu1p\n3WYDuw8kSX0hXbmCieXLtiYEW61dzcTyZaQrV/SmYrOISYEkqefSNM1aCNKJKRaYYOL7l2RdC+oY\nkwJJUu/dc8e2LQSNHn0E7vlld+ozS5kUSJJ6Ln1iXbHlNhRbTs1xoKFUUpqmTNz1C5785WYmhrcj\n3eMljoyWWpTMnUeRjoFkp3kdr8tsZlIglVAbGf302tU8XnvTkdFS6/ZaAvMXTt+FsGAR7LVf9+o0\nC9l9IBXkyGipc5IkYejEUyCZ4rSUDDH0xpNtleswkwKpAEdGS52Rpinp3bczceM18NwdSM44M2sR\nqLdgEUNnnGVrXBfYfSAVUWZk9N5LulMnacBNNVFR8saTSXbYkXTD+mwMwV772ULQJbYUSAU4Mlpq\nr+m649Ivnwe//x1Dh7yCZG+nOO4mkwKpgGRusRHPjoyWZmZ3XP8yKZCKqI2Mno4jo6VinKiob5kU\nSAU4MlpqH7vj+lfTAw1DCKPAzcBfxBivyd/bHfgqsBS4H3hvjPGqunWOAc4H9gRWAKfFGO9rtg5S\nNyUHLmXojLOY+P4l2VVMzYJFWULgyGipkHZMVOQkYp3RVFKQJwTfBhrbSi8DbgMOAo4HLg0h7BNj\nfCiEsBtwKfDfgSuBj+XL/3GTdZe6LjlwKUMHHMbIfXexw8Q4vxvajvE99vZgJJXR4kRFTiLWOaW7\nD0II+wI3AHs0vP8qshaA02OMd8UYzyVrDTg1X+Q04KYY4wUxxjuBU4DdQwhHtRKA1G1JkjD0kj9i\nzlGvYeglLzUhkEpqpTvOScQ6q5kxBa8EfkzWRVC/xw4FVsYYN9a9d12+XK38mlpBjPEpYGVduSRp\nlqh1x5WZqMi7FjqvdPdBjHF57e8QQn3RImBVw+JrgMUFyyVJs0itO4577ig2UdEsmkSsV2Mm2jmj\n4RxgU8N7m4DRguWFjIxUYxLGWhxViaeR8Q2+qsdY9fhggGJcckChxbb8bgNTtBE8y/DvNzC83Xat\n1amHttxyPePfvZCnH31k65iJZMEiht/8DoYPOqKpbRb9DrTzm7IRaBwqOgo8WVfemACMAuvLfMjY\n2FhTletXVYunkfENvqrHWPX4oDoxbtzjRawtsNzY7i9idP78jtenE5786b/x+BfPgYlnpz/po4+w\n+YvnsOMHlzHn8KM79vntTAoeZtu7ERYCj9SVN87+shC4tcyHrF+/nvHx8aYq2E9GRkYYGxurTDyN\njG/wVT3GqscH1YsxXbCYZMEi0vpbghsku+zKhgXPJ1lbJH3oL2ma8vRXP7NNQrDVxATrvnY+v39x\n+amfa9+FGZcrtdXp3QCcFUIYjTHWugmOBK6tKz+ytnAIYQ5wANmtiYWNj4+zefPmNlS3P1QtnkbG\nN5i29mdu2czTw9sxXuF7wN2HgyV548mky5dNPtgwGSI54e0DmwCld98+bcIDkK5ZxeZf3kbSoTET\n7UwKrgYeBC4JIZwNHAscApycl18EvD+EcCZwBVkycG+M8eo21kFSi7wHfPBVeR9WeRKxMjM9diq9\na3Wa4633fcQYJ4A3kHUJ3AycBBwXY3woL38AOIFs3oIbgblkExxJ6hPeAz74ZsM+TA5cytAnlrP9\nB89j57POYfsPfpKhTywf6IQA+uPBay21FMQYhxte/waYcgREjPFKYJ9WPlNSZxS9B3zogMMq0Qxd\nRbNpH26dRGz+fP5z7VqSKnQBtTjTYzv4QCRJGZ9cN/jchwOtHx68ZlKgtts6wOnqK5m46xfOLjYg\nfHLd4HMfDr5mZnpspz6f0UKDpsoDnKquHU+uU2+5D6uhlw9es6VAbTMbBjhVWq0/czod7s9Ui9yH\nldGrB6+ZFPRAFZvXfVDJ4OuH/sxuqeJvEGbXPlRn2H3QZZVtXp9FDyqpsirfA15T2d9gbjbsQ3WO\nSUEXbW1eb7yazpvXuzGIpFP6YdINtUcv+zM7rcq/wXpV3ofqLJOCLqn6/cMOcKqWKt4DXvXfYKMq\n7kN1nmMKuqXq9w87wEn9ruq/QakNTAq6pOr3D1dlgFOapqR3387EjddkDyepyAA0Vf83KLWD3Qdd\nMhua1wd9gFNtAFrtajKFSg1Am+1mw29QapVJQbf0wZzW3TCoA5xmywC0WW2W/AalVth90CVVaV4v\noleTbjTLORZmh9n0G5SaZVLQRb2e01pTcADarOFvUJqe3QddNqjN61XmHAuzi79BaWomBT3g/cP9\nxQFos4+/QU0nTVO45w7SJ9aRzJ0Hey2ZNUmjSYHkADRJudl+F5JjCjTrOQBNmt1q85Ns+e6FTCw/\nd1Y/6dWWAonBn2NBz5jNTb8qr7FlYOoFqzUN9lRMCqRcbQAa99xBumF9NoZgr/0G5gDgydCmX5Uz\n5fwkU+kTPgCeAAASi0lEQVTxk1678Rs3KWgjD8qDL0kS2PulA3eXgSdDJ6BSMbXj9MT6x2G6+Umm\nWr/kXUiTnReA0ueKbv3GTQraxIOyesWT4ex6AqIXH80r3FUwjTJ3IU16Xthxblb42yeeeW+Gc8WW\nW67v2m/cpKANZtNB2QNSf5lNJ8PJbL3qu/O24hNQ9ajptx28+Ghe6a6CyZS4C2nKz8uTgWeZ5lyR\npinj372wa79xk4IWzaaDsgekPlRmNsYBPhlOppmrvkGegGo2XXy024zH6SJK3IXU1OdNca7YdMet\npPWDnyfTxt+4tyS2apZMkbv1gDSLb9XpR7P1ccBTfh9nMKgTUPl8jhYVOU5Pp+w02M1+3iTnionH\n1xZatV2/cVsKWjSIU+SW7QKYTa0hg2Y2zsbY9FXfIE9ANYtbhNqh6HH62RKSPz2WZP/DSt+F1Nzn\n5es2nCuGdp5faL12/cZNClo0aAflproAKnpAGuTxEc8aQT13Hkx3EBrkk+FkmrkKG/AJqAbx4qOf\nFD1Ob9Xi/CSlP69+3YZzxeiSA0gWLJq+C6GNv3GTglYN0BS5Zfskt554bvlpse134YDUyom8fl0e\nfYT0pz8eyPER5frSE5KlryK96VqYO4/0xfuR/PqXPU2EWk3GSl+F9WACqnYlnFu388iDhZbvl4uP\nvlPkOD13HsmbTiWZu3Pr85MU+bzJTHKuSJKEkTe/g81fOGfy1rE2J7wDlxSkD/yadNEL+ibjr02R\nO+Wo1i5foUx1MCrUBfDtL5NsfppkbGfS3/+W9HuXlPpST3dAaqxXMyenqVo5kjeeTLLDjtNuq9CJ\ntIUBW91qdSg1gnqnMUhT0su/ma0LMDRMOrHlmdddToTaMVi1cOvc699Kss/LCh/g23Yib9OA3NID\nKfvk4qOfPGufHv5q0h9+e+rj9FtPb9vvYMbzwqQrTX2uGD7oCLZMMeNqcsLb4bk7MHHjNU0fW+sN\nXFIw8fUvMPHbDdv8wHrZFNzqFLnN1n2bE23Dibz+YMRzd5j54PLEOtKvfbq5Zq9pDkiTHiRLnpym\na+VIl5+7tc6TbavUibSJ8RHduiujUF96frXDo6tJf/itbZfN/59v1cWR620bPV+wdS55/Vta24fP\n24XkiGOybZVJXKeK8UvnkhxzLMkBh848jqfs7XNNXnwMchfaTCZNqnacC0kCG9Y/816HWpKmPC/s\nNJb9W7IOk824mv5+A+n3LiEtcGzl0KMK1XvgkgJgm4NIP9wq1+wUuVtuuZ6J73ytdN0njXky+f+r\n5NWvLx9UUdMckKacdKPEyan0wLK6bXHAYeUHpZUYH9GNSUVK3Yv/xDrYaR7pZd8oHnMXBoq2c7Bq\nu1vnptyHj615divLTInrjN/TlPRfLyf918un3Vbp73uTFx/TXUT0exfaTKafIyAhecPbYJddS01l\n3kwCNdV5AWhqOvX6GVfTlStIl59X+Nia7jgX5r96xs8YzKQAth5EEtLJ/8f04N7dslPkPvnTf5u8\nn2iGupe+ikgnSG++tmCtSprmgDTjpBvbrDDFiaGZgWW178cfFmghmWz1AuMjujGpSFP34t/18/Ix\nd3qgaJsHq7brAValvqMzHVPKfE+n21bB7ZTpHpn04qNsvQZEoeRsxU8Y+sTyrrQGTnleaGE69dKJ\nYzrBxE/+GY7qs6QghDAK/ANwAvAk8OkY42ea3uCjj5B++ysdPSh3SpqmbLjos6Xr3vTtWE+sm3mU\nehmvfj1DBx4+7QGp0KQbjSY5MTR9e8+jj8Cvft7UqkUGbHV6UpFmZ2BL0mkO+tN9XgcHinZi9Hw7\nHmBV+js6zTGl9Pd0qt940e0sWkxS4Hs15cVHyXoNjDYnoH05aVQzF0rris130O2Wgk8BBwJ/AuwO\nfD2EcH+M8QdNb3GmH9Cjj2SDS/Z92aQPophsUEbjMkXXKzWC+u7bGX/kodJ1T39yRfOTcOy2R9aP\n1YYJToYOPHzGA1LRSTe2We+W6xkifeb/++82NLUdgPThB8qvVHDAVtH4avGU+a6ld99O+s0vlU/+\nFiyCfV4GV3yn3HrMnAgVGSzaGM/WZQqOnmfVg6R3317uN9fCFVdT39GpjilFY5xpWwW/70US1xkv\nPgrUq53HyMnWa2fikaYp6Z23FVu2YGtgmQHa3YgRWrhQKqBrSUEIYQ7wDuC1McbbgNtCCOcB7wGa\nTwoKSK/4DukV35n8QRSNgzImW6bIeiX74tL1j7dU96b84pbs3+Fh2LJl+mWnU/CkWXTSjW385Aom\nfnJFe2K+teRMiyX6owvHV4unie9aKXnd2XsJadnboWbYp4UGi87wOymi1G+1Df3fzX5H2/m7bGpb\nBX+DhS4+ZqpXu46RTTwIqFR9S3a1Fbp9s8gVef0A7Q7HWNPKPAgz6eY0x39MloTUH6WvAw7tWg1+\n+8S2P7jGA9ZkyxRZr+R0v8nYzoWWm7YOzdqyBUjgmGOzLoUySpw0a5NuNK2dMRdRcirT0vE1+10r\noq7utYF4JAV/3jPs060D8RoPjkXiKZEQlN5WG6bY7qvvaNFtlZmDv+DFx7TadYycbJk2TZM+5Xd0\nKkWTqrJX5B2M8Vlqd+B0QDeTgkXAYzHG8br31gDPCSGUPEP2qRLzjyd7v5SRRYu7UKmppPDzm0je\n+q7iJ4+SJ83apBuFt99t8xeSvOFtJO/6AEMf+Pts4FGJbL4f4kte/9ZJ614biEfjCW94+NmvZ9in\npQeLdluLc/73wz4spexvsOzFRy+0uA9Lf0dLJFVJ2YumqbT52RSlE/8SujmmYA6wqeG92uvRAus/\nB2Bk8e5trFJnbPfoKoZ2f/G0ywwPD7PdX36YDRd9rqcH3O2e/0LSDy1jy79cTlo3ECWZN5+hPz2W\noTnPJf3db0l22InkhS8q3Dc2nJ985hz+Kp7ecadtts/QcPNXkbXP+PM3wQ47MXFVw7ZnWu+Vf87Q\nnnuXimebbcwUXxck8+azXThl6hiOeBXp4UdnE37l+5AX7An/cW+hfTo8PMzT9/6KkR12hB12bHv9\nh/74UCZu+/e2bKvIb65RN/bh0JIDmfjlrW0ZxzP8529i+OVHlfrODv3RQQwdehRbHlvT8ud3WrP7\nsMx3NJk3n+HXvIHhffcvtP30ZQez+ZAj2/a9KBtj7Ts6OjrKyEjD6fqIV7Fl3rzCx9a6c+dzpvvM\npFtP1QohnAh8Lsa4a917+wB3ADvHGGdqNzsJ+GYHqyhJUtW9DfjWVIXdbCl4GHheCGEoxli7NF4I\nPFUgIQC4kiyY+4GNnamiJEmV9Byyu/6unG6hbiYFPwM2A4cBtSfsvAK4qeD6jzNNdiNJkqY149Pt\nutZ9ABBC+BJwBHAqsBi4BHh7jPHyrlVCkiRNqtuTF72PbEbDnwAbgP9uQiBJUn/oakuBJEnqXwNy\nc64kSeo0kwJJkgSYFEiSpJxJgSRJAkwKJElSrtu3JFZaCGEUuBn4ixjjNfl7BwGfB/4I+AXw3hjj\nv9etcwpwJtm8DbcDfxNj/Gnd9j4FBLKncF6Wr/9U14Kq04H4RoC/A/4vsu/i14H/FmNs7aEILWgy\nxvcBfw08D7gG+MsY46/ryv8f4P3ADsA/Ae+JMfZkVs5OxFe33FeBB2OMH+94IFNod3whhO3JvqNv\nAf4Q+P/z8oe7FVOjDsQ4B/gscDyQAN8D3hdj/M+uBVWnw9/RDwDvjjHu0fFAptGBfTgXWEd2nqg9\nHOOxGOOCsnWzpaBN8p38bWC/uvfmA/8K3AYcBETgqhDC4rz8dcAXgP+X7NHSVwH/M4RQeybm35LN\n+vg64P/I/z6nC+Fso0PxnQ38F+AU4LXAq4HPdCOeyTQZ49uAjwDvAl5GNvPmP9et/0bgo8BpwKvI\nZvQ8rwvhbKMT8dVt50zgHR0OYVodiu/jwBuAtwKHA9sBP+h0LFPpUIyfBQ4EjiH7Db4c+HSnY5lM\nh7+jewIfIztx9kyHYtwPeIzs0QG1/2Z+NvQkTAraIISwL3AD0Jh9vp1sR707xnh3jPEC4Drg/64r\nvzjG+J0Y429ijB8FVpMlAAB/BnwlxnhrjPEW4EtkP9qu6mB87yZrGfiXGOPPgDOAM/Irl65qIcYd\ngTNjjFfGGO8FlgF7hxCel5f/FXB+jPH/y/fh6cA7QgjTPqms3ToVXwhhhxDC98hag/6jC6FMqoP7\n7+3Ah2KM18UYf0WW3B0SQnhRh0PaRgdj3EjWevWz/Hd4EXBkh8PZRgfjq/kSsLJjARTQwRj3Be6O\nMa6NMT6a//dYM3W0+6A9Xgn8mCyTe7Lu/T2AW2KM9Znpz4Haw9CXAb+bZHs75f8+DpwYQvgWWZPQ\nCfTmS932+PLMeAfgxoZ1twMOJmse66amYowxfqn2ZghhJ+A9wO0xxsdCCEPAIWRXJzU3ANuTtZy0\n57nBxbQ9vrr1tye70vzHjtV+Zp3YfwnZQ9jqf3O1ptmd6L6O7MMY41/Wle9O9kTaf+tMCNPq1HeU\nEMJ/Bf4AuJBn/x67rVMx7gfc3Y4KmhS0QYxxee3vEEJ90Rqypp56LyDrEyLPyrfKm9v3IvvSAHyA\nrKny8fz1z4HXt6veRXUovnVkD8h6PvCrunWprd9NzcZYt84pZAecjWRdIQBzyZ5Mtqruc7aEEB4n\nG2PRtaSgQ/ERY/w5cOwk2+2qTsSXH6B/0rDuXwNryX6LXdWpfVhXfgnwX4H7yLr2uqpT8eUXIOfy\nTNdIz3RwH+4LbBdC+HeyY+q1ZGMSVpeto0lBZ30f+EgI4Z3AxWR9dscCDzUumDdHXgx8I8Z4W/72\nXsADZP3u25P1z59P1q/UD1qKL4TwA+DvQwjHA78HPkmWKGzfneoXUjTGq4ADyB729cMQwv5AbcDk\npoZlNwGjHatxOU3HF2N8oKs1bU7b4gshvAH4G+BdMcbxjte8uHbFeC7Zs2mWAf+LrPWnH7Qa32eA\ni2KMd4YQepoUTKPVGPcBHiVLWoeAvweuCCEc0tD6MCPHFHRQjPEOsj7Iz5Bldp8Avgj8tn65EMLe\nZFck95Cf8EMIOwBfIxutf22M8cdkA7lODSHs0rUgptFKfLm/zJd9MP/vemB94/q9VDTGGONDMcbb\nYox/TRbL2/PlYdsEYJRnNx32TIvx9b12xRdCOA74LvDZGOPF3ah7Ue2KMcb4qxjjjcCbgZeFEI7q\nRv1n0kp8IYTXkDXB11o+EvpQG/bhfsArY4w3xOzurhOB/YFDy9bFpKDDYoz/SNb/uDjGeEj+9v21\n8hDCEuBqskFafx5jrF1V7gPM4dnNlLeS7bPdOlztwlqIjxjjYzHGY4B5wAKylpAF9ev3g+liDCH8\nSZ701LuTrNnvcbIfeO1uC0IIw8DOwCMdrnZhLcQ3EFqNL4TwFrLR4MtjjO/vfI3LazbGEMJICOGE\nEMJz67b1KNl3t2/2cQv78C1kXXWPhRB+RzbY8IUhhN+GEI7oSuULauV7GmPc2HBsXUu2D59fth52\nH3RQCOFPgNNjjG8F1uQDl/6M7ItJyG7NuxK4C/iz+Oz5B2r90PsBtb75fclup7mv87WfWYvxEUL4\nOvA/YoxX5a/fBKyJMf6ye1FMb5oY/yFf5CyyLp4z8uWHyDL082OMaQjhJrKR3LWBk4cDT5PdetRz\nLcR3QfdrW16r8YUQXk02f8bn+jUhaOU7SnY8+UfgnWQtIYQQXkCWuN7ZvSim1uI+/Ceyq+6aN5K1\nUL6SurE+vdbKPsxblR8Ajo8xXp2XP58sYfgVJZkUdNbdwP8ZQjgd+BeygYNzeWaU9qfJrvzfCewY\nQtgxf//3McaHQwhXAl8JIZyRL/cl4NsxxsfpD63E959kmewnQgirgPlkE3f8XRfrX8RUMX49L/8H\nIIYQrgZuIZuk6DkN5ctDCHeQHYT+gew2055MXjSJZuPr5Z0GZTQdX96qcxHZhEWfbOi2Wxdj3Nyd\nEGbU9Hc0H/j6ZeCcEMJDZC1bnwcuizH2RVJAC/swxvgk2a1+AIQQHgXGY4x9cWFVp5V9+GQI4Rqy\nBOFdwARZQvQ/826JUuw+aL+tgzpijKuAQDb44+dkAwePyb+oAMcBu5BdSa+q++9v8vKT8vV+BPyQ\n7Pa9Xg8ybGd8HyG7GrmW7Mv/6Rjj57sQw0wKxxhj/Geye4n/lqx7Z0/gNXXl3yUb9PNlslaTFWRZ\nfy+1Lb6ptttj7YrvYLKm51fzzHf3kfzf2m23vdLOffghsoFu/0R2Z9CdwMndCGIanfqO9pN2xvh2\nsltnf0Q2fus3ZDPFlpakab/8jiVJUi/ZUiBJkgCTAkmSlDMpkCRJgEmBJEnKmRRIkiTApECSJOVM\nCiRJEmBSIEmSciYFkiQJMCmQJEk5kwJJkgTA/wYUrSItMgDzzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1eda90240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot_date(yearIndex[1500:1600],yearData['noël'].values[1500:1600])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed we see that it appears each year around december which is nice to see!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Cleaning Steps\n",
    "\n",
    "We will now clean the data by doing the next two major cleaning steps:\n",
    "\n",
    "1. Removal of words that are unimportant for the data (not present enough, only two letter words)\n",
    "2. Doing some custom NLTK processing to group together french words that have the same meaning. \n",
    "\n",
    "### Removal of 2 letter words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape before the removal : (2351, 212920)\n",
      "We will remove  1248  columns\n",
      "Shape after removal :  (2351, 211672)\n"
     ]
    }
   ],
   "source": [
    "print('Shape before the removal :', yearData.shape)\n",
    "colLength = yearData.columns.map(lambda x: len(x))\n",
    "print('We will remove ', len(colLength[colLength<3]),' columns')\n",
    "yearData = yearData.drop(yearData.columns[colLength<3],axis=1)\n",
    "print('Shape after removal : ',yearData.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the minimum number of times one word appears each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1798-02-01     1.0\n",
       "1798-03-01     2.0\n",
       "1798-04-01     2.0\n",
       "1798-05-01     2.0\n",
       "1798-06-01     2.0\n",
       "1798-07-01     2.0\n",
       "1798-08-01     3.0\n",
       "1798-09-01     2.0\n",
       "1798-10-01     3.0\n",
       "1798-11-01     3.0\n",
       "1798-12-01     3.0\n",
       "1799-01-01     2.0\n",
       "1799-02-01     2.0\n",
       "1799-03-01     3.0\n",
       "1799-04-01     2.0\n",
       "1799-05-01     3.0\n",
       "1799-06-01     2.0\n",
       "1799-07-01     2.0\n",
       "1799-08-01     3.0\n",
       "1799-09-01     2.0\n",
       "1799-10-01     3.0\n",
       "1799-11-01     3.0\n",
       "1799-12-01     3.0\n",
       "1804-01-01     1.0\n",
       "1804-02-01     1.0\n",
       "1804-03-01     1.0\n",
       "1804-04-01     2.0\n",
       "1804-05-01     3.0\n",
       "1804-06-01     3.0\n",
       "1804-07-01     3.0\n",
       "              ... \n",
       "1995-07-01    15.0\n",
       "1995-08-01    16.0\n",
       "1995-11-01    18.0\n",
       "1995-12-01    27.0\n",
       "1996-01-01    29.0\n",
       "1996-02-01    19.0\n",
       "1996-03-01    15.0\n",
       "1996-04-01    28.0\n",
       "1996-05-01    19.0\n",
       "1996-06-01    31.0\n",
       "1996-07-01    22.0\n",
       "1996-08-01    28.0\n",
       "1996-09-01    30.0\n",
       "1996-10-01    18.0\n",
       "1996-11-01    11.0\n",
       "1996-12-01    16.0\n",
       "1997-01-01    23.0\n",
       "1997-02-01    25.0\n",
       "1997-03-01    26.0\n",
       "1997-04-01    27.0\n",
       "1997-05-01    26.0\n",
       "1997-06-01    27.0\n",
       "1997-07-01    25.0\n",
       "1997-08-01    24.0\n",
       "1997-09-01    28.0\n",
       "1997-10-01    29.0\n",
       "1997-11-01    29.0\n",
       "1997-12-01    27.0\n",
       "1998-01-01    26.0\n",
       "1998-02-01    27.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yearData[yearData>0].min(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that by taking more and more words over the years we were able to capture a very large part of the long tail distribution!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The columns that begin or end by '-'\n",
    "\n",
    "Because we wanted to keep words such as week-end, we have a certain number of columns that start with '-' and a word, we will remove the dash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "colName = yearData.columns.map(lambda x: x[1:] if str(x).startswith('-') else x)\n",
    "yearData.columns = colName"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the ones that end with a dash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "endNames = yearData.columns.map(lambda x: x[1:] if str(x).endswith('-') else x)\n",
    "yearData.columns = endNames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we groupby, so that their count is associated with the word itself as they are the same (-manger and manger) should be put together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape Before Groupby : (2351, 211672)\n",
      "Shape Before Groupby : (2351, 211672)\n"
     ]
    }
   ],
   "source": [
    "print('Shape Before Groupby :', yearData.shape)\n",
    "dfFinal1 = yearData.groupby(by=yearData.columns,axis=1,level=0).agg(sum)\n",
    "print('Shape Before Groupby :', yearData.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape After Groupby:  (2351, 209401)\n"
     ]
    }
   ],
   "source": [
    "print('Shape After Groupby: ',dfFinal1.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with adverbs\n",
    "\n",
    "We will group words with theit adverb (word and word + ement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "adverbs = dfFinal1.columns.map(lambda x: x[:-5] if str(x).endswith('ement') else x)\n",
    "dfFinal1.columns = adverbs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another groupby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape Before Groupby : (2351, 209401)\n",
      "Shape Before Groupby : (2351, 209043)\n"
     ]
    }
   ],
   "source": [
    "print('Shape Before Groupby :', dfFinal1.shape)\n",
    "dfFinal2 = dfFinal1.groupby(by=dfFinal1.columns,axis=1,level=0).agg(sum)\n",
    "print('Shape Before Groupby :', dfFinal2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Merging singular and plural words\n",
    "\n",
    "As we were not satisfied by the NLTK work on the french language, we will implement our own small set of rules to merge similar words together. \n",
    "\n",
    "We start were merging singular and plural words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wordCountMontClean2 = dfFinal2.copy()\n",
    "#Getting the words\n",
    "columnNames = wordCountMontClean2.columns.map(lambda x: str(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look up a few of the column titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['abattoir', 'abattoirs', 'abattre', 'abattu', 'abattucci',\n",
       "       'abattue', 'abattues', 'abattus', 'abatucci', 'abauzit', 'abayc',\n",
       "       'abaye', 'abayes', 'abazzia', 'abb', 'abb-sécheron', 'abba',\n",
       "       'abba-garima', 'abbadia', 'abbadie'], dtype=object)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columnNames[340:360]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the plural of a word comes normally just after it's singular, we therefore do not have to check all of the other words for plurals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Singular-Plural word pairs :  15708\n",
      "['abkhaze', 'able', 'aboi', 'abolie', 'abolitioniste', 'abolitionniste', 'abominable']\n",
      "['abkhazes', 'ables', 'abois', 'abolies', 'abolitionistes', 'abolitionnistes', 'abominables']\n"
     ]
    }
   ],
   "source": [
    "sing = []\n",
    "plural = []\n",
    "for i in range(len(columnNames)):\n",
    "    col = columnNames[i]\n",
    "    colPlur = col+'s'\n",
    "    colPlurX = col+'x'\n",
    "    if (i+5) > (len(columnNames) - 1):\n",
    "        lastEl =  len(columnNames) - 1\n",
    "    else:\n",
    "        lastEl = i+5\n",
    "    for plur in columnNames[i:lastEl]:\n",
    "        if (colPlur == plur) or (colPlurX == plur):\n",
    "            sing.append(col)\n",
    "            plural.append(plur)\n",
    "\n",
    "print('Number of Singular-Plural word pairs : ', len(dict(zip(sing,plural))))\n",
    "#We have to remove the jouis and jouiss pairs because, jouis will already be removed with the joui pair\n",
    "#pluralWords.pop(1329)\n",
    "singPlurDict = dict(zip(plural,sing))\n",
    "print(sing[20:27])\n",
    "print(plural[20:27])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see just above an example of the words that are grouped together.\n",
    "\n",
    "Now that we have that we can replace the column names and then groupby so that we merge and add the word counts for the two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wordCountMontClean2.rename(columns = singPlurDict,inplace=True)\n",
    "cleanData = wordCountMontClean2.groupby(by=wordCountMontClean2.columns,axis=1,level=0).agg(sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we look at the masculine and feminine version of words and of the verbs and their infinitif. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "newCol = cleanData.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of fem singular, verbs word pairs :  9240\n",
      "['able', 'aboli', 'aboli', 'abondant', 'abonde', 'abonnem', 'abonné']\n",
      "['abler', 'abolie', 'abolir', 'abondante', 'abonder', 'abonneme', 'abonnée']\n"
     ]
    }
   ],
   "source": [
    "fem = []\n",
    "masc = []\n",
    "\n",
    "for i in range(len(newCol)):\n",
    "    col = newCol[i]\n",
    "    colfem= col+'e'\n",
    "    infVerb = col+'r'\n",
    "    if (i+5) > (len(newCol) - 1):\n",
    "        lastEl =  len(newCol) - 1\n",
    "    else:\n",
    "        lastEl = i+5\n",
    "    for plur in newCol[i:lastEl]:\n",
    "        if (colfem == plur) or (infVerb == plur):\n",
    "            masc.append(col)\n",
    "            fem.append(plur)\n",
    "\n",
    "dictFM = dict(zip(masc,fem))\n",
    "print('Number of fem singular, verbs word pairs : ', len(dictFM))\n",
    "print(masc[20:27])\n",
    "print(fem[20:27])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see just above an example of the words that are grouped together.\n",
    "\n",
    "And we groupby again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData.rename(columns = dictFM,inplace=True)\n",
    "cleanData1 = cleanData.groupby(by=cleanData.columns,axis=1,level=0).agg(sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the verbs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of fem singular, verbs word pairs :  963\n",
      "['abuse', 'abusiv', 'abyssin', 'accabl', 'accable', 'accablé', 'accam']\n",
      "['abuser', 'abusive', 'abyssine', 'accable', 'accabler', 'accablée', 'accame']\n"
     ]
    }
   ],
   "source": [
    "verb = []\n",
    "conj = []\n",
    "\n",
    "for i in range(len(cleanData1.columns)):\n",
    "    col = cleanData1.columns[i]\n",
    "    infVerb = col[:-1]+'er'\n",
    "    if (i+5) > (len(cleanData1.columns) - 1):\n",
    "        lastEl =  len(cleanData1.columns) - 1\n",
    "    else:\n",
    "        lastEl = i+5\n",
    "    for plur in cleanData1.columns[i:lastEl]:\n",
    "        if (infVerb == plur):\n",
    "            verb.append(col)\n",
    "            conj.append(plur)\n",
    "\n",
    "dictVC = dict(zip(verb,conj))\n",
    "print('Number of fem singular, verbs word pairs : ', len(dictVC))\n",
    "print(masc[50:57])\n",
    "print(fem[50:57])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we see just above an example of the words that are grouped together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData1.rename(columns = dictVC,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to replace \"participe passé\" feminine to the infinitif verb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['abaissée', 'abandonnée', 'abfentée', 'abforbée', 'abonnée', 'abordée',\n",
      "       'abreuvée', 'abritée', 'abrogée', 'abrégée',\n",
      "       ...\n",
      "       'évadée', 'évaluée', 'éveillée', 'évitée', 'évoluée', 'évoquée',\n",
      "       'ôméée', 'ôuiéée', '—méditerranée', '—sodée'],\n",
      "      dtype='object', length=1953)\n"
     ]
    }
   ],
   "source": [
    "print(cleanData1.columns[np.chararray.endswith(cleanData1.columns.values.astype(str),'ée')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData1.columns = cleanData1.columns.map(lambda x: x[:-2]+'er' if str(x).endswith('ée') else x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can groupby again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2351, 182601)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData2 = cleanData1.groupby(by=cleanData1.columns,axis=1,level=0).agg(sum)\n",
    "cleanData2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to group together verbs and their conjugaisons. Here we implemenent a custom set of rules that should put together verbs conjugation that were not taken care of by the previous steps. \n",
    "\n",
    "We will count all different conjugations of one words as the count of the verb and not separate entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "colClean2 = cleanData2.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have identified:  4779 Verbs to put together\n",
      "['abriter', 'abroger', 'abréger', 'abréger', 'absenter', 'absenter', 'absorber']\n",
      "['abritant', 'abrogeant', 'abrégerai', 'abrégerait', 'absentent', 'absentera', 'absorbait']\n"
     ]
    }
   ],
   "source": [
    "verbList = []\n",
    "verbFamily = []\n",
    "for i in range(len(colClean2)):\n",
    "    if colClean2[i].endswith('er'):\n",
    "        wordClean = colClean2[i]\n",
    "        word = colClean2[i][:-2]\n",
    "        for j in range(i-4,i+4):\n",
    "            if j == i:\n",
    "                pass\n",
    "            \n",
    "            otherWord = colClean2[j]\n",
    "            if ((otherWord[:len(word)] == word) and ((otherWord[-2:] == 'en') \n",
    "                | (otherWord[-3:] == 'ent')  | (otherWord[-3:] == 'era')  \n",
    "                | (otherWord[-3:] == 'ait')  | (otherWord[-3:] == 'ant') \n",
    "                | (otherWord[-2:] == 'és') | (otherWord[-4:] == 'eant') \n",
    "                | (otherWord[-2:] == 'es') | (otherWord[-2:] == 'ai'))):\n",
    "                verbList.append(wordClean)\n",
    "                verbFamily.append(otherWord)\n",
    "\n",
    "print('We have identified: ', len(verbList), 'Verbs to put together')\n",
    "dictVerbFamily = dict(zip(verbFamily,verbList))\n",
    "print(verbList[20:27])\n",
    "print(verbFamily[20:27])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see above a few examples of which verbs were put together.\n",
    "\n",
    "And we can put these columns together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData2.rename(columns = dictVerbFamily,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another GroupBy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2351, 177907)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanDataVerb = cleanData2.groupby(by=cleanData2.columns,axis=1,level=0).agg(sum)\n",
    "cleanDataVerb.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our dataset is still very large, but those operations manage to reduce its size without loosing information because we grouped together words that meant the same. \n",
    "\n",
    "We gained here some information because perhaps each separate entity does not hold information and would be deleted due to not being present enough but together they form an interesting time serie to analyze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique words before grouping:  212920\n",
      "Number of unique words after grouping:  177907\n"
     ]
    }
   ],
   "source": [
    "print('Number of unique words before grouping: ', dfFinal.shape[1])\n",
    "print('Number of unique words after grouping: ', cleanDataVerb.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have tried not to loose too much of information by trying to merge words that have the same meaning we can start doing some more direct actions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with missing data\n",
    "\n",
    "One of the problem that we have for our dataset is that there are some months were a word count is missing (for some words). When importing the data we had to choose a frequency limit, and we did not save words that appeared below this limit. Due to the long tail distribution of the data words that have a fequency near the limit can be above it in one month and below it in the other month, therefore we will have a lot of missing values for this word. A part of this problem was solved by grouping the data as we did before.\n",
    "\n",
    "For furutre predictions and visualization we are more interested in the general shape of the time series, and not the few months in between where the word is missing, therefore it could be interesting to interpolate the missing data.\n",
    "\n",
    "But there are also words that are only in present only in a few months and not more. We do not need these words as they are not interesting to predict and do not bring an interesting visualizations.\n",
    "\n",
    "Due to the fact that we have made some aggregations in the previous steps, we assume that removing words that do not occur very often will not loose information about the other words. We choose to remove words that only appear in 10 different months or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData3 = cleanDataVerb.copy()\n",
    "numMonth = cleanData3.astype(bool).sum(axis=0).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at which words are in this situations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['--------------', '---------------', '----------------',\n",
       "       '-----------------', '------------------', '-------------------',\n",
       "       '--------------------------------------------------------------------------',\n",
       "       '---------------------------------------------------------------------------',\n",
       "       '------har', '---le', '--am', '--beecham', '--bowater', '--br',\n",
       "       '--broken', '--can', '--clôture', '--coca', '--courtaulds', '--de',\n",
       "       '--dimanche', '--dôme', '--e', '--east', '--free', '--g', '--ii',\n",
       "       '--imp', '--kloof', '--l', '--m', '--marks', '--n-', '--plessey',\n",
       "       '--pots', '--rank', '--roan', '--rolls', '--s', '--sel', '--slater',\n",
       "       '--st', '--tampax', '--tesco', '--tube', '--union', '--unit',\n",
       "       '--vickers', '--volume'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.columns[numMonth<10][1:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['actionprime', 'actionsdejouissanceducanaldu', 'actionsjura-simplon',\n",
       "       'actionsou', 'actious', 'actium', 'activite', 'actlvlte', 'acto',\n",
       "       'acton', 'actoteatro', 'actouest-suiss', 'acts', 'actu', 'actual',\n",
       "       'actualisation', 'actualite', 'actuariels', 'actue', 'actueles',\n",
       "       'actuellede', 'actère', 'acu', 'acuminariatique', 'acun', 'acunto',\n",
       "       'acupuncture', 'acv', 'acy', 'acéré', 'acétate', 'acétique',\n",
       "       'acétylcholine', 'acùt', 'acût', 'ad', 'ad-', 'ad-astra', 'ad-hoc',\n",
       "       'ad-intérim', 'ada-agamus', 'adage', 'adalberl', 'adalberto',\n",
       "       'adalgisa', 'adalr', 'adam-benjamin', 'adam-louis', 'adamec',\n",
       "       'adamello'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.columns[numMonth<10][1000:1050]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['acliard', 'aclif', 'aclivité', 'acloque', 'aclrice', 'acluclle',\n",
       "       'acluelle', 'acmvveveyn', 'acmvveveyp', 'acné', 'acolade', 'acoliar',\n",
       "       'acollas', 'acolytes', 'acom', 'acommission', 'acomp', 'acon', 'acon-',\n",
       "       'aconit', 'aconitine', 'aconnex', 'aconte-', 'acores', 'acoust',\n",
       "       'acoûté', 'acquadro', 'acquainted', 'acquapendente', 'acquarelle',\n",
       "       'acqueducs', 'acquerra', 'acquerraient', 'acquerrait', 'acquerront',\n",
       "       'acques-', 'acqui', 'acquiesc', 'acquiescer', 'acquiescerons',\n",
       "       'acquiescé', 'acquiesça', 'acquife', 'acquill', 'acquiller', 'acquin',\n",
       "       'acquired', 'acquisilion', 'acquiter', 'acquitta'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.columns[numMonth<10][900:950]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that these are not very important words, and as they do not have many datapoints we can easily remove them. They are ea"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before word aggregation :  (4351, 212920)\n",
      "Before the removal of absent words:  (2351, 177907)\n",
      "After the cleaning:  (2351, 43243)\n"
     ]
    }
   ],
   "source": [
    "print('Before word aggregation : ', dfFinal.shape)\n",
    "print('Before the removal of absent words: ', cleanData3.shape)\n",
    "cleanData3.drop(cleanData3.columns[numMonth < 10].values,axis=1,inplace=True)\n",
    "print('After the cleaning: ', cleanData3.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also remove words that are only present less than ten times in the entire dataset, as they do not represent a very strong signal and will not be very useful for visualization or prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sumWords = cleanData3.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before the removal of absent words:  (2351, 43243)\n",
      "Number of columns that will be removed:  0\n",
      "After the cleaning:  (2351, 43176)\n"
     ]
    }
   ],
   "source": [
    "print('Before the removal of absent words: ', cleanData3.shape)\n",
    "print('Number of columns that will be removed: ',(sumWords<10).sum())\n",
    "cleanData3.drop(cleanData3.columns[(sumWords<15).values],axis=1,inplace=True)\n",
    "print('After the cleaning: ', cleanData3.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>-</th>\n",
       "      <th>--</th>\n",
       "      <th>---</th>\n",
       "      <th>----</th>\n",
       "      <th>-----</th>\n",
       "      <th>------</th>\n",
       "      <th>-------</th>\n",
       "      <th>--------</th>\n",
       "      <th>---------</th>\n",
       "      <th>...</th>\n",
       "      <th>—turin</th>\n",
       "      <th>—une</th>\n",
       "      <th>—union</th>\n",
       "      <th>—vingt-quatrième</th>\n",
       "      <th>—vingt-troisième</th>\n",
       "      <th>—voici</th>\n",
       "      <th>—vous</th>\n",
       "      <th>———</th>\n",
       "      <th>————</th>\n",
       "      <th>„„„</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1798-02-01</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-03-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-04-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-05-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-06-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43176 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   -   --  ---  ----  -----  ------  -------  --------  \\\n",
       "1798-02-01  1.0  0.0  0.0  0.0   0.0    0.0     0.0      0.0       0.0   \n",
       "1798-03-01  0.0  0.0  0.0  0.0   0.0    0.0     0.0      0.0       0.0   \n",
       "1798-04-01  0.0  0.0  0.0  0.0   0.0    0.0     0.0      0.0       0.0   \n",
       "1798-05-01  0.0  0.0  5.0  0.0   0.0    0.0     0.0      0.0       0.0   \n",
       "1798-06-01  0.0  0.0  0.0  0.0   0.0    0.0     0.0      0.0       0.0   \n",
       "\n",
       "            --------- ...   —turin  —une  —union  —vingt-quatrième  \\\n",
       "1798-02-01        0.0 ...      0.0   0.0     0.0               0.0   \n",
       "1798-03-01        0.0 ...      0.0   0.0     0.0               0.0   \n",
       "1798-04-01        0.0 ...      0.0   0.0     0.0               0.0   \n",
       "1798-05-01        0.0 ...      0.0   0.0     0.0               0.0   \n",
       "1798-06-01        0.0 ...      0.0   0.0     0.0               0.0   \n",
       "\n",
       "            —vingt-troisième  —voici  —vous  ———  ————  „„„  \n",
       "1798-02-01               0.0     0.0    0.0  0.0   0.0  0.0  \n",
       "1798-03-01               0.0     0.0    0.0  0.0   0.0  0.0  \n",
       "1798-04-01               0.0     0.0    0.0  0.0   0.0  0.0  \n",
       "1798-05-01               0.0     0.0    0.0  0.0   0.0  0.0  \n",
       "1798-06-01               0.0     0.0    0.0  0.0   0.0  0.0  \n",
       "\n",
       "[5 rows x 43176 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are still random signes that we were not able to remove we will do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['', '-', '--', '---', '----', '-----', '------', '-------', '--------',\n",
       "       '---------', '----------', '-----------', '------------', '--i',\n",
       "       '-arbr', '-c', '-d', '-genève', '-i', '-le', '-s', '-s-', '-tarifs',\n",
       "       '-téléphone', '-—', 'a-', 'a-b', 'a-bu', 'a-ce', 'a-k', 'a-l-il',\n",
       "       'a-l-on', 'a-t', 'a-t-elle', 'a-t-il', 'a-t-on', 'aa-', 'aaa', 'aac',\n",
       "       'aach', 'aacom', 'aal', 'aaland', 'aandoz', 'aap', 'aar', 'aar-tessin'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.columns[0:47]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData3.drop(cleanData3.columns[0:47],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also have the bizarre dash that we can quickly take care of again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['—-a', '—-la', '—-le', '—-on', '—-—', '—adopté', '—alu', '—am',\n",
       "       '—annecy', '—au',\n",
       "       ...\n",
       "       '—trois', '—turin', '—une', '—union', '—vingt-quatrième',\n",
       "       '—vingt-troisième', '—voici', '—vous', '———', '————'],\n",
       "      dtype='object', length=105)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData3.columns[np.chararray.startswith(cleanData3.columns.values.astype(str),'—')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData3.columns = \\\n",
    "cleanData3.columns.map(lambda x: x[1:] if x.startswith('—') else x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before:  (2351, 43129)\n",
      "Before:  (2351, 43051)\n"
     ]
    }
   ],
   "source": [
    "print('Before: ',cleanData3.shape)\n",
    "cleanData4 = cleanData3.groupby(by=cleanData3.columns,axis=1,level=0).agg(sum)\n",
    "print('Before: ',cleanData4.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>-a</th>\n",
       "      <th>-la</th>\n",
       "      <th>-le</th>\n",
       "      <th>-on</th>\n",
       "      <th>-—</th>\n",
       "      <th>aarau</th>\n",
       "      <th>aarberg</th>\n",
       "      <th>aarbourg</th>\n",
       "      <th>aarg</th>\n",
       "      <th>aarhus</th>\n",
       "      <th>...</th>\n",
       "      <th>œil</th>\n",
       "      <th>œillet</th>\n",
       "      <th>œquo</th>\n",
       "      <th>œrlikon</th>\n",
       "      <th>œuf</th>\n",
       "      <th>œur-</th>\n",
       "      <th>œuvre</th>\n",
       "      <th>——</th>\n",
       "      <th>———</th>\n",
       "      <th>„„„</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1798-02-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-03-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-04-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-05-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1798-06-01</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43051 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             -a  -la  -le  -on   -—  aarau  aarberg  aarbourg  aarg  aarhus  \\\n",
       "1798-02-01  0.0  0.0  0.0  0.0  1.0    0.0      0.0       0.0   0.0     0.0   \n",
       "1798-03-01  0.0  0.0  0.0  0.0  0.0    0.0      0.0       0.0   0.0     0.0   \n",
       "1798-04-01  0.0  0.0  3.0  0.0  0.0    0.0      0.0       0.0   0.0     0.0   \n",
       "1798-05-01  0.0  0.0  0.0  0.0  0.0    0.0      0.0       0.0   0.0     0.0   \n",
       "1798-06-01  0.0  0.0  0.0  0.0  0.0    0.0      0.0       0.0   0.0     0.0   \n",
       "\n",
       "           ...   œil  œillet  œquo  œrlikon  œuf  œur-  œuvre   ——  ———  „„„  \n",
       "1798-02-01 ...   0.0     0.0   0.0      0.0  0.0   0.0    0.0  0.0  0.0  0.0  \n",
       "1798-03-01 ...   0.0     0.0   0.0      0.0  0.0   0.0    0.0  0.0  0.0  0.0  \n",
       "1798-04-01 ...   2.0     0.0   0.0      0.0  0.0   0.0    0.0  0.0  0.0  0.0  \n",
       "1798-05-01 ...   0.0     0.0   0.0      0.0  0.0   0.0    0.0  0.0  0.0  0.0  \n",
       "1798-06-01 ...   0.0     0.0   0.0      0.0  0.0   0.0    0.0  0.0  0.0  0.0  \n",
       "\n",
       "[5 rows x 43051 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanData4.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the dataset is a bit more cleaned up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleanData4.to_csv('3kplusWordMonth.csv')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}