MIDS-W261-2015-HWK-Week05-Anderson-Draft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## W261 Machine Learning at Scale\n",
    "\n",
    "Names Safyre Anderson, Howard Wen , Vamsi Sakhamuri\n",
    "\n",
    "Emails safyre@berkeley.edu, howard.wen1@gmail.com, vamsi@ischool.berkeley.edu\n",
    "\n",
    "Time of Initial Submission: February 18th, 2016 8am PST\n",
    "\n",
    "Section W261-3, Spring 2016\n",
    "\n",
    "Week 5 Homework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HW 5.0\n",
    "*What is a data warehouse? What is a Star schema? When is it used?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. *Data Warehouse:*\n",
    "\n",
    "https://en.wikipedia.org/wiki/Data_warehouse. A data warehouse is an enterprise-level repository for historical and current datasets. Typically, a business has many sources of data that need to be processed for a variety of applications: e.g. reporting, business analyics, and/ or predictive analytics.  For example, in mobile advertising data pipelines, an ad request from an app on a customer's mobile device will require a decision that involves streaming usage data in addition to historical data. The historical data would have been used to determine which types of ads are relevant to the end user, while the streaming data used to generate a prediction. Data collected from these different sources and various purposes need to be passed through some form of transformation and storage in order to be processed in subsequent parts of the pipeline. The gathering of data, transforming them and storing them is known as ETL or extract, transform, and load. The loading piece of ETL refers to storing processed data into the data warehouse--typically, a relational database, but can also include semi-stuctured and unstructured data storage.\n",
    "\n",
    "\n",
    "Two traditional examples of data warehousing strategies are OLAP (online analytical processing) and OLTP (online transactional processing).\n",
    "\n",
    "2. *Star Schema:*\n",
    "\n",
    "The star schema is a framework for organizing data in a relational database where a central fact table acts as a means to connect data between different dimensional tables. The fact table will contain foreign keys for each of the branching tables. Additionally, each of the branching tables focusses on data related to one dimension that an analysis could splice on. For example an ecommerce schema might have a transactions-based fact table that contains foreign keys for a customers table, and a payment table. The transactions table will be able to join with both the customers and payment tables directly, but the payment table and customer table would need to join with each other indirectly through the transactions table since they would not (in this example) contain each others foreign keys. \n",
    "\n",
    "3. *When is it used?:*\n",
    "\n",
    "The star schema is useful for highly structured data that may have numerous dimensions that could be used to break down the data during analyses. The main reason this is useful in most reporting applications is because it makes querying data much more efficient. Instead of having one large table with all the data and many rows, joining a smaller table with a larger one can shrink the size of the data significantly before it is extracted.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HW 5.1\n",
    "1. *In the database world What is 3NF? Does machine learning use data in 3NF? If so why? In what form does ML consume data?*\n",
    "\n",
    "3NF stands for \"Third normal form\". This refers to how data are organized with their respective tables and keys: Attributes in a table should provide a fact about the table's key and nothing else (\"Nothing but the key\" https://en.wikipedia.org/wiki/Third_normal_form). This structure allows a database to avoid duplicating data as much as possible and therefore save storage space. Machine learning would not be able to utilize data in this form very well--features would be scattered across different sources so the classifier would not be able to train on features in different tables. ML classifiers need to have all the data they depend on in one location (distributed or not) and therefore need the data to be denormalized.\n",
    "\n",
    "2. *Why would one use log files that are denormalized?*\n",
    "\n",
    "Log files are normally stored as semi-structured or unstructured data. They may have some regular format, but for the most part, log files contain plain English that would need to be parsed during analysis. If log files were placed into a structure and normalized, closely related pieces of information that could be useful would be separated from each other--for instance, an error message with a time stamp. Furthermore, a denormalized log file is more usable because log files are all text. Normalized data typically contains data that can be strongly typed, but if a log file is more intuitively parsed as a document then normalization may just overcomplicate things.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### HW 5.2\n",
    "*Using MRJob, implement a hashside join (memory-backed map-side) for left, \n",
    "right and inner joins. Run your code on the  data used in HW 4.4: (Recall HW 4.4: Find the most frequent visitor of each page using mrjob and the output of 4.2  (i.e., transfromed log file). In this output please include the webpage URL, webpageID and Visitor ID.)*\n",
    ":\n",
    "\n",
    "Justify which table you chose as the Left table in this hashside join.\n",
    "\n",
    "Please report the number of rows resulting from:\n",
    "\n",
    "(1) Left joining Table Left with Table Right\n",
    "(2) Right joining Table Left with Table Right\n",
    "(3) Inner joining Table Left with Table Right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-02-15 14:26:38--  http://archive.ics.uci.edu/ml/machine-learning-databases/anonymous/anonymous-msweb.data\n",
      "Resolving archive.ics.uci.edu... 128.195.10.249\n",
      "Connecting to archive.ics.uci.edu|128.195.10.249|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1423098 (1.4M) [text/plain]\n",
      "Saving to: 'anonymous-msweb.data'\n",
      "\n",
      "anonymous-msweb.dat 100%[=====================>]   1.36M  1.91MB/s   in 0.7s   \n",
      "\n",
      "2016-02-15 14:26:39 (1.91 MB/s) - 'anonymous-msweb.data' saved [1423098/1423098]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## first get 4.2 data and preprocess it:\n",
    "!wget -O anonymous-msweb.data http://archive.ics.uci.edu/ml/machine-learning-databases/anonymous/anonymous-msweb.data\n",
    "\n",
    "## python code to reformat the data:\n",
    "# check format of whole file\n",
    "#!cat anonymous-msweb.data\n",
    "write_file_loc = \"msweb_processed.txt\"\n",
    "writefile = open(write_file_loc, 'wb')\n",
    "\n",
    "with open(\"anonymous-msweb.data\", 'rU') as f:\n",
    "    for line in f.readlines():\n",
    "        line = line.strip().split(',')\n",
    "        \n",
    "        # skip all the header rows\n",
    "        if line[0] not in ['C', 'V']:\n",
    "            continue\n",
    "            \n",
    "        if line[0] == 'C':\n",
    "            user_id = line[2]\n",
    "        \n",
    "        elif line[0] == 'V':\n",
    "            writefile.write('V,{},1,C,{}\\n'.format(line[1], user_id))\n",
    "        \n",
    "        else:\n",
    "            continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "V,1000,1,C,10001\r\n",
      "V,1001,1,C,10001\r\n",
      "V,1002,1,C,10001\r\n",
      "V,1001,1,C,10002\r\n",
      "V,1003,1,C,10002\r\n",
      "V,1001,1,C,10003\r\n",
      "V,1003,1,C,10003\r\n",
      "V,1004,1,C,10003\r\n",
      "V,1005,1,C,10004\r\n",
      "V,1006,1,C,10005\r\n"
     ]
    }
   ],
   "source": [
    "# preprocessed file is:\n",
    "!head -n10 msweb_processed.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I,4,\"www.microsoft.com\",\"created by getlog.pl\"\r\n",
      "T,1,\"VRoot\",0,0,\"VRoot\"\r\n",
      "N,0,\"0\"\r\n",
      "N,1,\"1\"\r\n",
      "T,2,\"Hide1\",0,0,\"Hide\"\r\n",
      "N,0,\"0\"\r\n",
      "N,1,\"1\"\r\n",
      "A,1287,1,\"International AutoRoute\",\"/autoroute\"\r\n",
      "A,1288,1,\"library\",\"/library\"\r\n",
      "A,1289,1,\"Master Chef Product Information\",\"/masterchef\"\r\n",
      "A,1297,1,\"Central America\",\"/centroam\"\r\n",
      "A,1215,1,\"For Developers Only Info\",\"/developer\"\r\n",
      "A,1279,1,\"Multimedia Golf\",\"/msgolf\"\r\n",
      "A,1239,1,\"Microsoft Consulting\",\"/msconsult\"\r\n",
      "A,1282,1,\"home\",\"/home\"\r\n"
     ]
    }
   ],
   "source": [
    "!head -n15 anonymous-msweb.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-02-15 15:20:24--  http://archive.ics.uci.edu/ml/machine-learning-databases/anonymous/anonymous-msweb.data\n",
      "Resolving archive.ics.uci.edu... 128.195.10.249\n",
      "Connecting to archive.ics.uci.edu|128.195.10.249|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1423098 (1.4M) [text/plain]\n",
      "Saving to: 'anonymous-msweb.data'\n",
      "\n",
      "anonymous-msweb.dat 100%[=====================>]   1.36M  1.88MB/s   in 0.7s   \n",
      "\n",
      "2016-02-15 15:20:25 (1.88 MB/s) - 'anonymous-msweb.data' saved [1423098/1423098]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## now process a new set of data with just URL and webpageID for merging\n",
    "## first get 4.2 data and preprocess it:\n",
    "!wget -O anonymous-msweb.data http://archive.ics.uci.edu/ml/machine-learning-databases/anonymous/anonymous-msweb.data\n",
    "\n",
    "## python code to reformat the data:\n",
    "# check format of whole file\n",
    "#!cat anonymous-msweb.data\n",
    "write_file_loc = \"msweb_dim_url.txt\"\n",
    "writefile = open(write_file_loc, 'wb')\n",
    "\n",
    "with open(\"anonymous-msweb.data\", 'rU') as f:\n",
    "    for line in f.readlines():\n",
    "        line = line.strip().split(',')\n",
    "        \n",
    "        # skip all the header rows\n",
    "        if line[0] == 'A':\n",
    "            # attach the relative url to the home url. Remove \"\"'s\n",
    "            full_url = \"http://www.microsoft.com\"+line[4].strip('\"')\n",
    "            writefile.write('{url_id}\\t{url}\\n'.format(url_id=line[1],url=full_url))\n",
    "        \n",
    "        #elif line[0] == 'V':\n",
    "        #    writefile.write('V,{},1,C,{}\\n'.format(line[1], user_id))\n",
    "        \n",
    "        #else:\n",
    "        #    continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1287\thttp://www.microsoft.com/autoroute\r\n",
      "1288\thttp://www.microsoft.com/library\r\n",
      "1289\thttp://www.microsoft.com/masterchef\r\n",
      "1297\thttp://www.microsoft.com/centroam\r\n",
      "1215\thttp://www.microsoft.com/developer\r\n",
      "1279\thttp://www.microsoft.com/msgolf\r\n",
      "1239\thttp://www.microsoft.com/msconsult\r\n",
      "1282\thttp://www.microsoft.com/home\r\n",
      "1251\thttp://www.microsoft.com/referencesupport\r\n",
      "1121\thttp://www.microsoft.com/magazine\r\n"
     ]
    }
   ],
   "source": [
    "!head -n10 msweb_dim_url.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     304 msweb_dim_url.txt\n",
      "   98654 msweb_processed.txt\n"
     ]
    }
   ],
   "source": [
    "!wc -l msweb_dim_url.txt\n",
    "!wc -l msweb_processed.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting MR_HW52_joins.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile MR_HW52_joins.py\n",
    "#!/usr/bin/env python\n",
    "# find the most frequent visitors per url\n",
    "from mrjob.job import MRJob\n",
    "from mrjob.step import MRStep\n",
    "import csv\n",
    "from heapq import heappush, heappop,nlargest\n",
    "\n",
    "class MapSideJoins(MRJob):\n",
    "    def steps(self):\n",
    "        return [MRStep(\n",
    "            mapper_init = self.mapper_init,\n",
    "            mapper = self.mapper,\n",
    "            mapper_final = self.mapper_final\n",
    "            )]\n",
    "        \n",
    "    def mapper_init(self):\n",
    "        # set variables\n",
    "        # inspired by example on lecture slide 121\n",
    "        self.URLS = {}\n",
    "        \n",
    "        # for right outer join\n",
    "        self.URLS_ON_RIGHT = {}\n",
    "        \n",
    "        # initialize row counts\n",
    "        self.left = 0\n",
    "        self.right = 0 \n",
    "        self.inner = 0\n",
    "        \n",
    "        # read in the page_url table\n",
    "        # formatted like : 1287 http://www.microsoft.com/autoroute\n",
    "        f = open(\"msweb_dim_url.txt\", 'rU')\n",
    "        for line in f:\n",
    "            line = line.strip()\n",
    "            data = line.split()\n",
    "            #store page-id as key in URLS dictionary\n",
    "            self.URLS[data[0]] = data[1]\n",
    "                \n",
    "    def mapper(self, _, line):\n",
    "        \"\"\"stream in output from 4.2 (msweb_processed.txt):\n",
    "        \n",
    "        V,1000,1,C,10001\n",
    "        \n",
    "        and match with keys in self.URLS\"\"\"\n",
    "        self.increment_counter('group', 'Number_mapper_calls',1)\n",
    "        line = line.strip().split(',')\n",
    "        key = line[1]\n",
    "        \n",
    "        if key in self.URLS.keys():\n",
    "            ## outputs are : JOIN_TYPE, URL_ID, URL, CUSTOMER_ID\n",
    "            # map inner join\n",
    "            self.inner += 1\n",
    "            yield None, (\"inner\", key, self.URLS[key], line[4])\n",
    "            \n",
    "            # map visits left join urls on url_id\n",
    "            # this code assumes each customer has a visit\n",
    "            # if there are visits to no URLS, we will catch it in the second mapper\n",
    "            # although this probably isn't the case...\n",
    "            self.left += 1\n",
    "            yield None, (\"left\", key, self.URLS[key], line[4])\n",
    "            \n",
    "            # map visits right join urls on url_id\n",
    "            # keep track of url_ids that have not been visited\n",
    "            self.right += 1\n",
    "            yield None, (\"right\", key, self.URLS[key], line[4])\n",
    "            self.URLS_ON_RIGHT[key] = self.URLS[key]\n",
    "        \n",
    "        #right join will also include sites that have not been visited\n",
    "        # no customer_id\n",
    "        else:\n",
    "            self.right += 1\n",
    "            yield None, (\"right\", key, self.URLS[key], \"NA\")\n",
    "            \n",
    "    def mapper_final(self):\n",
    "        for key in self.URLS.keys():\n",
    "            # in a left join, if a key exists on the left table, but not the right table\n",
    "            # then we need a row with data from the left table and null values from the missing\n",
    "            # data on the right\n",
    "            if key not in self.URLS_ON_RIGHT.keys():\n",
    "                self.left += 1\n",
    "                yield None, (\"left\", key, \"NA\",  self.URLS[key])\n",
    "        \n",
    "        ## final outputs:\n",
    "        yield None, ()\n",
    "        yield None, (\"left-joined rows: \" + str(self.left))\n",
    "        yield None, (\"right-joined rows: \" + str(self.right))\n",
    "        yield None, (\"inner-joined rows: \" + str(self.inner))\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    MapSideJoins.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no configs found; falling back on auto-configuration\n",
      "no configs found; falling back on auto-configuration\n",
      "creating tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239\n",
      "\n",
      "PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols\n",
      "\n",
      "writing to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239/step-0-mapper_part-00000\n",
      "Counters from step 1:\n",
      "  group:\n",
      "    Number_mapper_calls: 98654\n",
      "Moving /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239/step-0-mapper_part-00000 -> /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239/output/part-00000\n",
      "Streaming final output from /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239/output\n",
      "removing tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160218.162259.447239\n"
     ]
    }
   ],
   "source": [
    "!rm hw52_out.txt\n",
    "!./MR_HW52_joins.py msweb_processed.txt \\\n",
    "    --file msweb_dim_url.txt > hw52_out.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "null\t[\"left\", \"1291\", \"NA\", \"http://www.microsoft.com/news\"]\r\n",
      "null\t[\"left\", \"1297\", \"NA\", \"http://www.microsoft.com/centroam\"]\r\n",
      "null\t[\"left\", \"1294\", \"NA\", \"http://www.microsoft.com/bookshelf\"]\r\n",
      "null\t[\"left\", \"1287\", \"NA\", \"http://www.microsoft.com/autoroute\"]\r\n",
      "null\t[\"left\", \"1289\", \"NA\", \"http://www.microsoft.com/masterchef\"]\r\n",
      "null\t[\"left\", \"1288\", \"NA\", \"http://www.microsoft.com/library\"]\r\n",
      "null\t[]\r\n",
      "null\t\"left-joined rows: 98663\"\r\n",
      "null\t\"right-joined rows: 98654\"\r\n",
      "null\t\"inner-joined rows: 98654\"\r\n"
     ]
    }
   ],
   "source": [
    "!tail -n10 hw52_out.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got unexpected keyword arguments: ssh_tunnel\n",
      "using existing scratch bucket mrjob-62fd57571f35a64e\n",
      "using s3://mrjob-62fd57571f35a64e/tmp/ as our scratch dir on S3\n",
      "creating tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160219.153400.354127\n",
      "writing master bootstrap script to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW52_joins.Safyre.20160219.153400.354127/b.py\n",
      "Copying non-input files into s3://mrjob-62fd57571f35a64e/tmp/MR_HW52_joins.Safyre.20160219.153400.354127/files/\n",
      "Waiting 5.0s for S3 eventual consistency\n",
      "Creating Elastic MapReduce job flow\n",
      "Job flow created with ID: j-PBLI033NYNV6\n",
      "Created new job flow j-PBLI033NYNV6\n",
      "Job launched 30.5s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 61.4s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 91.9s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 122.5s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 153.0s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 184.0s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 214.5s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 245.1s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 275.6s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 306.5s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 337.0s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 368.4s ago, status STARTING: Provisioning Amazon EC2 capacity\n",
      "Job launched 399.7s ago, status STARTING: Configuring cluster software\n",
      "Job launched 430.6s ago, status STARTING: Configuring cluster software\n",
      "Job launched 461.1s ago, status STARTING: Configuring cluster software\n",
      "Job launched 492.0s ago, status STARTING: Configuring cluster software\n",
      "Job launched 522.6s ago, status BOOTSTRAPPING: Running bootstrap actions\n",
      "Job launched 553.2s ago, status BOOTSTRAPPING: Running bootstrap actions\n",
      "Job launched 583.7s ago, status RUNNING: Running step\n",
      "Job launched 614.3s ago, status RUNNING: Running step\n",
      "Job launched 644.8s ago, status RUNNING: Running step (MR_HW52_joins.Safyre.20160219.153400.354127: Step 1 of 1)\n",
      "Opening ssh tunnel to Hadoop job tracker\n",
      "Connect to job tracker at: http://localhost:40328/jobtracker.jsp\n",
      "Job launched 677.1s ago, status RUNNING: Running step (MR_HW52_joins.Safyre.20160219.153400.354127: Step 1 of 1)\n",
      "Unable to load progress from job tracker\n",
      "Job launched 707.7s ago, status RUNNING: Running step (MR_HW52_joins.Safyre.20160219.153400.354127: Step 1 of 1)\n",
      "Job completed.\n",
      "Running time was 100.0s (not counting time spent waiting for the EC2 instances)\n",
      "Fetching counters from S3...\n",
      "Waiting 5.0s for S3 eventual consistency\n",
      "Counters from step 1:\n",
      "  (no counters found)\n",
      "Killing our SSH tunnel (pid 4419)\n",
      "Terminating job flow: j-PBLI033NYNV6\n"
     ]
    }
   ],
   "source": [
    "!chmod +x MR_HW52_joins.py\n",
    "!python MR_HW52_joins.py -r emr s3://w261-safyre-hw52-input/msweb_processed.txt \\\n",
    "    --conf-path /Users/Safyre/cc-mrjob/mrjob.conf\\\n",
    "    --file s3://w261-safyre-hw52-input/msweb_dim_url.txt \\\n",
    "    --output-dir= s3://w261-safyre-hw52-output/output/out/ \\\n",
    "    --no-output \\\n",
    "    --cleanup=NONE\\\n",
    "    --no-strict-protocol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using existing scratch bucket mrjob-62fd57571f35a64e\n",
      "using s3://mrjob-62fd57571f35a64e/tmp/ as our scratch dir on S3\n",
      "Creating persistent job flow to run several jobs in...\n",
      "creating tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/no_script.Safyre.20160219.061229.360977\n",
      "writing master bootstrap script to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/no_script.Safyre.20160219.061229.360977/b.py\n",
      "Copying non-input files into s3://mrjob-62fd57571f35a64e/tmp/no_script.Safyre.20160219.061229.360977/files/\n",
      "Waiting 5.0s for S3 eventual consistency\n",
      "Creating Elastic MapReduce job flow\n",
      "Job flow created with ID: j-4B9KNCB48TTT\n",
      "j-4B9KNCB48TTT\n"
     ]
    }
   ],
   "source": [
    "#persistent job flow\n",
    "!python -m mrjob.tools.emr.create_job_flow --conf-path /Users/Safyre/cc-mrjob/mrjob.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HW 5.3  EDA of Google n-grams dataset\n",
    "A large subset of the Google n-grams dataset\n",
    "\n",
    "https://aws.amazon.com/datasets/google-books-ngrams/\n",
    "\n",
    "which we have placed in a bucket/folder on Dropbox on s3:\n",
    "\n",
    "   https://www.dropbox.com/sh/tmqpc4o0xswhkvz/AACUifrl6wrMrlK6a3X3lZ9Ea?dl=0 \n",
    "\n",
    "   `s3://filtered-5grams/`\n",
    "\n",
    "In particular, this bucket contains (~200) files (10Meg each) in the format:\n",
    "\n",
    "\t(ngram) \\t (count) \\t (pages_count) \\t (books_count)\n",
    "\n",
    "For HW 5.3-5.5, for the Google n-grams dataset unit test and regression test your code using the \n",
    "first 10 lines of the following file:\n",
    "\n",
    "googlebooks-eng-all-5gram-20090715-0-filtered.txt\n",
    "\n",
    "Once you are happy with your test results proceed to generating  your results on the Google n-grams dataset. \n",
    "\n",
    "Do some EDA on this dataset using mrjob, e.g., \n",
    "\n",
    "- Longest 5-gram (number of characters)\n",
    "- Top 10 most frequent words (please use the count information), i.e., unigrams\n",
    "- 20 Most/Least densely appearing words (count/pages_count) sorted in decreasing order of relative frequency \n",
    "- Distribution of 5-gram sizes (character length).  E.g., count (using the count field) up how many times a 5-gram of 50 characters shows up. Plot the data graphically using a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-02-15 23:42:40--  https://www.dropbox.com/sh/tmqpc4o0xswhkvz/AACr50woxiBWoaiiLmnwduX8a/googlebooks-eng-all-5gram-20090715-0-filtered.txt?dl=0\n",
      "Resolving www.dropbox.com... 108.160.172.206, 108.160.172.238\n",
      "Connecting to www.dropbox.com|108.160.172.206|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://dl.dropboxusercontent.com/content_link/aIZVHfoBfzACzWxyagqbBmxeGTl601USgFACattc1MpkSXDxZ9gk5hMyZTQFqWBn/file [following]\n",
      "--2016-02-15 23:42:41--  https://dl.dropboxusercontent.com/content_link/aIZVHfoBfzACzWxyagqbBmxeGTl601USgFACattc1MpkSXDxZ9gk5hMyZTQFqWBn/file\n",
      "Resolving dl.dropboxusercontent.com... 199.47.217.69\n",
      "Connecting to dl.dropboxusercontent.com|199.47.217.69|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 11444614 (11M) [text/plain]\n",
      "Saving to: 'google-5grams-test.txt'\n",
      "\n",
      "google-5grams-test. 100%[=====================>]  10.91M  1.03MB/s   in 8.3s   \n",
      "\n",
      "2016-02-15 23:42:50 (1.31 MB/s) - 'google-5grams-test.txt' saved [11444614/11444614]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## download dev data\n",
    "!wget -O google-5grams-test.txt \"https://www.dropbox.com/sh/tmqpc4o0xswhkvz/AACr50woxiBWoaiiLmnwduX8a/googlebooks-eng-all-5gram-20090715-0-filtered.txt?dl=0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A BILL FOR ESTABLISHING RELIGIOUS\t59\t59\t54\r\n",
      "A Biography of General George\t92\t90\t74\r\n",
      "A Case Study in Government\t102\t102\t78\r\n",
      "A Case Study of Female\t447\t447\t327\r\n",
      "A Case Study of Limited\t55\t55\t43\r\n",
      "A Child's Christmas in Wales\t1099\t1061\t866\r\n",
      "A Circumstantial Narrative of the\t62\t62\t50\r\n",
      "A City by the Sea\t62\t60\t49\r\n",
      "A Collection of Fairy Tales\t123\t117\t80\r\n",
      "A Collection of Forms of\t116\t103\t82\r\n"
     ]
    }
   ],
   "source": [
    "#ngram \\t count \\t page_count \\t book_count\n",
    "!head -n10 google-5grams-test.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11444614"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## upload test set to s3 bucket\n",
    "import boto\n",
    "from boto.s3.connection import S3Connection\n",
    "from boto.s3.key import Key\n",
    "\n",
    "# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are environment variables\n",
    "# S3Connection argument can be empty\n",
    "conn = S3Connection()\n",
    "bucket = conn.create_bucket('w261-safyre-hw53-input')  # sub-datasets bucket already exists\n",
    "myBucket = conn.get_bucket('w261-safyre-hw53-input')\n",
    "\n",
    "k = Key(myBucket)\n",
    "k.key = 'google-5grams-test.txt'\n",
    "k.set_contents_from_filename('google-5grams-test.txt')\n",
    "#googlebooks-eng-all-5gram-20090715-0-filtered.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting MR_HW53.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile MR_HW53.py\n",
    "#!/usr/bin/env python\n",
    "# mapper for 5.3 word count\n",
    "# find the most frequent visitors per url\n",
    "from mrjob.job import MRJob\n",
    "from mrjob.step import MRStep\n",
    "import csv\n",
    "from heapq import heappush, heappop,nlargest, nsmallest\n",
    "import re\n",
    "\n",
    "class WordCount(MRJob):\n",
    "    def steps(self):\n",
    "        return [MRStep(\n",
    "            mapper_init = self.mapper_init,\n",
    "            mapper = self.mapper,\n",
    "            combiner = self.combiner,\n",
    "            reducer = self.reducer,\n",
    "            reducer_final = self.reducer_final\n",
    "            )]\n",
    "        \n",
    "    def mapper_init(self):\n",
    "        self.counts = {}\n",
    "        self.pages = {}\n",
    "        #self.lengths = [] # heap queue\n",
    "        self.unigram_counts = []\n",
    "        self.density_counts = []\n",
    "        #self.counts_final = {}\n",
    "        #self.density_final = {}\n",
    "    \n",
    "    def mapper(self, _, line):\n",
    "        line.strip()\n",
    "        [ngram, gram_count, pages,books] = re.split(\"\\t\",line)\n",
    "        gram_count = int(gram_count)\n",
    "        pages = int(pages)\n",
    "        words = re.split(\" \",ngram)\n",
    "\n",
    "        \n",
    "        # emit words, counts and pages (for density calcs)\n",
    "        for word in words:\n",
    "            self.counts.setdefault(word,0)\n",
    "            self.pages.setdefault(word,0)\n",
    "            self.counts[word] += gram_count\n",
    "            self.pages[word] += pages\n",
    "            \n",
    "        for word in self.counts.keys():\n",
    "            yield word, (self.counts[word], self.pages[word], ngram)\n",
    "\n",
    "\n",
    "    def combiner(self, word, counts_pages):\n",
    "        sum_counts = sum_pages = num = 0\n",
    "        for count, pages, ngram in counts_pages:\n",
    "            sum_counts += count\n",
    "            sum_pages += pages\n",
    "        yield word, (sum_counts, sum_pages, ngram)\n",
    "    \n",
    "    def reducer(self, word, sums):\n",
    "        sum_counts = sum_pages = 0\n",
    "        self.counts_final = {}\n",
    "        self.density_final = {}\n",
    "        self.lengths = []\n",
    "        for count, pages, ngram in sums:\n",
    "            self.counts_final.setdefault(word,0)\n",
    "            self.density_final.setdefault(word,0)\n",
    "            sum_counts += count\n",
    "            sum_pages += pages\n",
    "            self.counts_final[word] = sum_counts\n",
    "            self.density_final[word] = float(sum_counts)/sum_pages\n",
    "            heappush(self.lengths, (len(ngram), ngram))\n",
    "        #yield word, (sum_counts, sum_pages)\n",
    "    '''\n",
    "    def reducer_final(self, word, sums):\n",
    "        density = 0\n",
    "        for count_sums, page_sums in sums:\n",
    "            density = float(count_sums, page_sums)\n",
    "        yield word, (count_sums, density)\n",
    "     '''\n",
    "    def reducer_final(self):\n",
    "        unigram_counts = []\n",
    "        density_counts = []\n",
    "        #for word, counts, density in word_count_density:\n",
    "        \n",
    "        for word in self.counts_final.keys():\n",
    "            heappush(unigram_counts, (self.counts_final[word], word))\n",
    "            heappush(density_counts, (self.density_final[word], word))\n",
    "        top10_counts = nlargest(10, unigram_counts)\n",
    "        top10_density = nlargest(10, density_counts)\n",
    "        bottom10_density = nsmallest(10, density_counts)\n",
    "        \n",
    "        yield None, top10_counts # top 10 most freq words\n",
    "        yield None, top10_density # top 10 most dense words\n",
    "        yield None, bottom10_density # bottom 10 dense words\n",
    "        yield None, nlargest(1, self.lengths) # max ngram\n",
    "        \n",
    "        for i in range(len(self.lengths)):\n",
    "            # pop out each item for histogram, \n",
    "            # is sorted already in asc order\n",
    "            yield heappop(self.lengths) \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    WordCount.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#test with a small file\n",
    "#!head -n100 google-5grams-test.txt > test_input52.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no configs found; falling back on auto-configuration\n",
      "no configs found; falling back on auto-configuration\n",
      "creating tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699\n",
      "\n",
      "PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols\n",
      "\n",
      "writing to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/step-0-mapper_part-00000\n",
      "Counters from step 1:\n",
      "  (no counters found)\n",
      "writing to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/step-0-mapper-sorted\n",
      "> sort /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/step-0-mapper_part-00000\n",
      "writing to /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/step-0-reducer_part-00000\n",
      "Counters from step 1:\n",
      "  (no counters found)\n",
      "Moving /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/step-0-reducer_part-00000 -> /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/output/part-00000\n",
      "Streaming final output from /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699/output\n",
      "removing tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53.Safyre.20160219.161734.789699\n"
     ]
    }
   ],
   "source": [
    "!chmod +x MR_HW53.py\n",
    "!python MR_HW53.py test_input52.txt > hw53b_out.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "null\t[[676, \"where\"]]\r\n",
      "null\t[[1.0, \"where\"]]\r\n",
      "null\t[[1.0, \"where\"]]\r\n",
      "null\t[[27, \"A branch was established at\"]]\r\n",
      "27\t\"A branch was established at\"\r\n"
     ]
    }
   ],
   "source": [
    "!cat hw53b_out.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got unexpected keyword arguments: ssh_tunnel\n",
      "using existing scratch bucket mrjob-62fd57571f35a64e\n",
      "using s3://mrjob-62fd57571f35a64e/tmp/ as our scratch dir on S3\n",
      "creating tmp directory /var/folders/jz/dhc2gzfj2091nhfmrbvv3cjh0000gn/T/MR_HW53bc.Safyre.20160219.064554.578414\n",
      "Copying non-input files into s3://mrjob-62fd57571f35a64e/tmp/MR_HW53bc.Safyre.20160219.064554.578414/files/\n",
      "Adding our job to existing job flow j-4B9KNCB48TTT\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!chmod +x MR_HW53bc.py\n",
    "!python MR_HW53bc.py -r emr \\\n",
    "    --emr-job-flow-id=j-4B9KNCB48TTT \\\n",
    "    --conf-path /Users/Safyre/cc-mrjob/mrjob.conf\\\n",
    "    --file s3://w261-safyre-hw53-input/google-5grams-test.txt \\\n",
    "    --output-dir= s3://w261-safyre-hw53-output/output/out/ \\\n",
    "    --no-output \\\n",
    "    --cleanup=NONE\\\n",
    "    --no-strict-protocol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!tail -n10 hw53b_out.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HW 5.4  Synonym detection over 2Gig of Data\n",
    "\n",
    "*For the remainder of this assignment you will work with two datasets:*\n",
    "\n",
    "#### 1: unit/systems test data set: SYSTEMS TEST DATASET\n",
    "*Three terms, A,B,C and their corresponding strip-docs of co-occurring terms\n",
    "\n",
    "DocA {X:20, Y:30, Z:5}\n",
    "DocB {X:100, Y:20}\n",
    "DocC {M:5, N:20, Z:5}*\n",
    "\n",
    "\n",
    "#### 2: A large subset of the Google n-grams dataset as was described above\n",
    "\n",
    "*For each HW 5.4 -5.5.1 Please unit test and system test your code with respect \n",
    "to SYSTEMS TEST DATASET and show the results. \n",
    "Please compute the expected answer by hand and show your hand calculations for the \n",
    "SYSTEMS TEST DATASET. Then show the results you get with you system.*\n",
    "\n",
    "*In this part of the assignment we will focus on developing methods\n",
    "for detecting synonyms, using the Google 5-grams dataset. To accomplish\n",
    "this you must script two main tasks using MRJob:*\n",
    "\n",
    "(*1) Build stripes for the most frequent 10,000 words using cooccurence informationa based on\n",
    "the words ranked from 9001,-10,000 as a basis/vocabulary (drop stopword-like terms),\n",
    "and output to a file in your bucket on s3 (bigram analysis, though the words are non-contiguous).*\n",
    "\n",
    "\n",
    "*(2) Using two (symmetric) comparison methods of your choice \n",
    "(e.g., correlations, distances, similarities), pairwise compare \n",
    "all stripes (vectors), and output to a file in your bucket on s3.*\n",
    "\n",
    "*==Design notes for (1)==\n",
    "For this task you will be able to modify the pattern we used in HW 3.2\n",
    "(feel free to use the solution as reference). To total the word counts \n",
    "across the 5-grams, output the support from the mappers using the total \n",
    "order inversion pattern:*\n",
    "\n",
    "*<*word,count>*\n",
    "\n",
    "*to ensure that the support arrives before the cooccurrences.*\n",
    "\n",
    "*In addition to ensuring the determination of the total word counts,\n",
    "the mapper must also output co-occurrence counts for the pairs of\n",
    "words inside of each 5-gram. Treat these words as a basket,\n",
    "as we have in HW 3, but count all stripes or pairs in both orders,\n",
    "i.e., count both orderings: (word1,word2), and (word2,word1), to preserve\n",
    "symmetry in our output for (2).*\n",
    "\n",
    "*==Design notes for (2)==\n",
    "For this task you will have to determine a method of comparison.\n",
    "Here are a few that you might consider:*\n",
    "\n",
    "- Jaccard\n",
    "- Cosine similarity\n",
    "- Spearman correlation\n",
    "- Euclidean distance\n",
    "- Taxicab (Manhattan) distance\n",
    "- Shortest path graph distance (a graph, because our data is symmetric!)\n",
    "- Pearson correlation\n",
    "- Kendall correlation\n",
    "...\n",
    "\n",
    "*However, be cautioned that some comparison methods are more difficult to\n",
    "parallelize than others, and do not perform more associations than is necessary, \n",
    "since your choice of association will be symmetric.*\n",
    "\n",
    "*Please use the inverted index (discussed in live session #5) based pattern to compute the pairwise (term-by-term) similarity matrix.*\n",
    "\n",
    "*Please report the size of the cluster used and the amount of time it takes to run for the index construction task and for the synonym calculation task. How many pairs need to be processed (HINT: use the posting list length to calculate directly)? Report your  Cluster configuration!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing HW_54.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile HW_54.py\n",
    "#!/usr/bin/python\n",
    "from mrjob.job import MRJob\n",
    "from mrjob.step import MRStep\n",
    "from mrjob.protocol import RawValueProtocol\n",
    "import re    \n",
    "import operator\n",
    "from collections import Counter, OrderedDict\n",
    " \n",
    "class BigramCooccurrence(MRJob):\n",
    "    \n",
    "    OUTPUT_PROTOCOL = RawValueProtocol\n",
    "    \n",
    "    URLs = {}\n",
    "\n",
    "    def steps(self):\n",
    "        return [MRStep(\n",
    "                mapper = self.mapper,\n",
    "                combiner = self.combiner,\n",
    "                reducer_init = self.reducer_init,\n",
    "                reducer = self.reducer\n",
    "                )]\n",
    "    \n",
    "    def mapper_1(self, _, line):\n",
    "        \n",
    "        # emit stripes with coocurrence line count\n",
    "        for line in sys.stdin:\n",
    "            line=line.strip()\n",
    "            co_words=line.split() \n",
    "            \n",
    "            # prevent double counts\n",
    "            co_words.sort()\n",
    "            for i, first_word in enumerate(co_words):\n",
    "                stripe={}\n",
    "                for following_word in co_words[i+1:]:\n",
    "                    stripe.setdefault(following_word, 1)\n",
    "                    stripe[following_word] += 1\n",
    "\n",
    "                yield first_word, stripe\n",
    "        \n",
    "    def combiner_1(self,first_word,stripe):\n",
    "        combined_coorrences=Counter({}) #Counters make tracking individual product counts easier\n",
    "        for line in sys.stdin:\n",
    "\n",
    "        cooccurrences=Counter(eval(stripe)) \n",
    "        \n",
    "        '''\n",
    "        allVisits = {}\n",
    "        for visit in visits:\n",
    "            for custID in visit.keys():\n",
    "                allVisits.setdefault(custID,0)\n",
    "                allVisits[custID] += visit[custID]\n",
    "        yield pageID,allVisits\n",
    "        \n",
    "    def reducer_init(self):\n",
    "        with open(\"anonymous-msweb-URLs.data\", \"r\") as IF:\n",
    "            for line in IF:\n",
    "                line = line.strip()\n",
    "                data = re.split(\",\",line)\n",
    "                URL = data[4]\n",
    "                pageID = data[1]\n",
    "                self.URLs[pageID] = URL\n",
    "\n",
    "    def reducer(self,pageID,visits):\n",
    "        allVisits = {}\n",
    "        for visit in visits:\n",
    "            for custID in visit.keys():\n",
    "                allVisits.setdefault(custID,0)\n",
    "                allVisits[custID] += visit[custID]\n",
    "        custID = max(allVisits.iteritems(), key=operator.itemgetter(1))[0]\n",
    "        yield None,self.URLs[pageID]+\",\"+pageID+\",\"+custID+\",\"+str(allVisits[custID])\n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    igramCooccurrence.run()\n",
    "    '''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}