learningPandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Pandas Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <br>PART ONE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pandas is one of the most commonly used Python packages/libraries for data science.<br><br>\n",
    "- Pandas is Python's answer for making two dimensional tables (ala Excel and SQL).<br><br>\n",
    "- Pandas calls a table a \"DataFrame\".<br><br>\n",
    "- Pandas DataFrames are used by Python's other packages for statistical analysis, data manipulation, and data visualization.<br><br>\n",
    "- Pandas DataFrames can be exported as .csv and other files.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About our practice data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be working with two datasets today.\n",
    "\n",
    "One records the length, width, and color of flower petals. \n",
    "\n",
    "The second dataset is more complicated. It is from UNICEF, and can be found here:\n",
    "https://data.unicef.org/resources/dataset/symptoms-pneumonia-careseeking-2016/\n",
    "\n",
    "This data set was collected in many countries over many years. Researchers visited the countries and collected data on two questions: was your young child (under 5 years old) sick in the past two weeks with symptoms of pneumonia? and did you take your sick child to a healthcare facility? The numbers reported in this dataset represent the percentage of sick children who went to a healthcare facility.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because pandas is one of the most commonly used Python packages, it often gets imported as a shortened version of it's actual name. This makes it quicker to type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## <br>Loading data as a Pandas DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open the file, define it as a Pandas DataFrame, assign it to a variable, and close the file in one line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading .csv files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df = pd.read_csv(\"sampleDataset.csv\")\n",
    "# Tip:'df' is often used to name DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading other file types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can still use `pd.read_csv()` to open other file types, as long as the data is organized in rows, with some delimiter separating the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df = pd.read_csv(\"sampleDataset.txt\", delimiter = \"\\t\")  # tab delimited"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing your DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>You can first check the length of your DataFrame before you view the whole DataFrame. The length is the number of rows.  If it's very large, you may not want to view the entire DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(flower_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To view the entire DataFrame, simply call the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To view only the top 5 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>What does the following code do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier, we loaded a second DataFrame called `pneu_care_df`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How long is the `pneu_care_df` DataFrame? (how many rows does it have?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If the DataFrame is longer than 20 lines, view the top 5 lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### View a sample of 10 lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Renaming columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Renaming columns is a common task when working with DataFrames. As you probably saw when you looked at the top of `pneu_care_df`, the column names don't quite match up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a list of the new column names that we would like to use. I've put the list together for you. To rename columns, give the columns attribute of the data frame a new list of names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_column_names = [\"Countries and areas\", \"Year\", \"Short Source\", \"Long Source\",\n",
    "                    \"National\", \"Male\", \"Female\", \"Urban\", \"Rural\", \"Poorest\", \n",
    "                    \"Second\", \"Middle\", \"Fourth\", \"Richest\"]\n",
    "\n",
    "pneu_care_df.columns = new_column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's check to see if it worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\"Countries and areas\" is a long column name. Let's shorten it to \"Countries\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the rename function. We need to pass the function a dictionary of the old name to be replaced as the key and the new name as the value.\n",
    "\n",
    "Tip: `inplace = True` has to be used because we want to rename the column within the same data frame.  Without this option, the `rename` function will return a copy of `pneu_care_df` instead of changing `pneu_care_df` directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.rename(columns = {\"Countries and areas\": \"Countries\"}, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Check the top of the DataFrame again to be sure it worked as expected. You know the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping rows and columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Dropping rows and columns is another common operation when cleaning your data. You may have noticed that row 0 in the `pneu_care_df` DataFrame is messy and not part of our usable data.<br><br> When we made our DataFrames, a number was added at the beginning of each row. This is called the index. We can reference the index to drop a row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.drop(0, inplace = True)\n",
    "pneu_care_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Now, let's learn to drop a column. First let's view the `flower_df` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>With the included row indices, we do not need the plant_id column, so let's [`drop`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html) it. We have to add one more argument `axis` so that the function knows we are removing a column instead of a row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.drop(\"plant_id\", axis = 1, inplace = True)\n",
    "flower_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The column names are a little long for the flower_df DataFrame. Make a list of the column names without the word \"petal\" included, and rename the columns.\n",
    "\n",
    "*Note: you'll need to get the answer to this question right for code later in this notebook to work, so check the answers if you missed this exercise.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the DataFrame to see if it worked as expected.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write code to remove the Short Source and Long Source columns from the `pneu_care_df` DataFrame. Add another line of code in the same cell to view only the top 5 rows to check your result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reformatting columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Often we need to change the type of the data in a particular row."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `pneu_care_df`, the Year column data is currently a float (2007.0). We want to change the column data to an integer (2007)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df[\"Year\"] = pneu_care_df.Year.astype(int)\n",
    "pneu_care_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait! That worked, but why? What did we just do? We haven't learned that yet!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsampling the DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's go back to the `flower_df` DataFrame. To reference only one column, you can use the column name:\n",
    "\n",
    "*Note: if you get a `Key Error` here, make sure you did the first part of Practice Exercise 2 above correctly.  Remember: you can check `flower_df.columns` to see the current column names.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df[\"color\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>If you want to reference multiple columns of the data, you can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df[[\"length\", \"width\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>We use two sets of brackets because one set is telling Pandas that you are referencing a \"key\" (i.e. a column) and the second set is telling Pandas that it is a list of multiple keys."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>You may have noticed that the formatting was different between the two outputs. The second method returned a more attractive table, called a DataFrame object. The first method returned an object that Pandas calls a Series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try using double brackets with only one object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df[[\"color\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>There is a second way to reference a single column. This way also returns a Series object, and will allow you to apply different statistical functions to a particular column of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.color"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Now let's view only the rows of data for flowers with blue petals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.color == \"blue\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>While that was the intuitive way to do it, it didn't give us what we really wanted. It will be important to know the distinction between the line of code above and the line of code below. If you are referencing a value in a column to compute a statistical analysis, the line of code above may be what the computer needs. If you want to see all the rows that have a particular value in a particular column, you will need to put that line of code inside the framework of the entire DataFrame, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df[flower_df.color == \"blue\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>You can also string multiple functions in a row. What do you think this will do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df[[\"length\", \"width\"]][flower_df.color == \"blue\"].sample(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's return to the larger DataFrame, `pneu_care_df`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First, you need to remember the column labels (names). Write code to view the top few rows of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write code to view only the countries, year, and national columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### View the countries and national columns for the year 2003."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As an extra challenge, write code to view the countries and national columns of 5 rows that have a National rate higher than 60."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <br> PART TWO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas data aggregation, missing data, grouping, and simple plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part two of the pandas tutorial! <br>To summarize part one, we wrote code to:\n",
    "- Load data in as a DataFrame\n",
    "- View the DataFrame in different ways\n",
    "- Rename the column headers\n",
    "- Drop a row or column\n",
    "- Subset the data by column name or data point attribute\n",
    "- And, briefly, change the formatting of a single column\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data aggregation means taking many data points and reducing them to one number, whether it's a count, sum, mean, or other single statistic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions all work the same way, as functions that get added on to the end of our DataFrame object:\n",
    "- [`.count()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.count.html)\n",
    "- [`.sum()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sum.html)\n",
    "- [`.mean()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html)\n",
    "- [`.median()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.median.html)\n",
    "- [`.min()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.min.html)\n",
    "- [`.max()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.max.html)\n",
    "- [`.unique()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.unique.html)\n",
    "- [`.nunique()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.nunique.html)\n",
    "- [`.std()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.std.html)   #Standard error\n",
    "- [`.var()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.var.html)   #Variance\n",
    "- And more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try some of these on our `flower_df` DataFrame and see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Of course, we usually want to use these functions on a single column. Because we are referencing a single column and not multiple columns, we can stack the column name and the desired function onto the end of out DataFrame. Let's try it with `flower_df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.length.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.width.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.color.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.color.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Now let's try it on our `pneu_care_df` DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.National.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.Richest.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>What happened??!! Any ideas?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's look at that column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df[[\"Richest\"]].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with missing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Pandas replaces missing fields with \"NaN\". Let's see what data type the column is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df[\"Richest\"].dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas turns all data in columns with missing data into the generic data type \"objects\", and pandas doesn't perform statistical calculations on \"objects\" (though some functions, like \"count\", do work with objects). Luckily, \"NaN\" data can also take the data type \"float\", a type that can be aggregated by statistical calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's change the data type of the \"Urban\" column, and then try calculating the mean again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df[\"Richest\"] = pneu_care_df.Richest.astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.Richest.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What is the earliest year that data was collected in the pneumonia care study?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How many unique countries are represented in the pneumonia care study?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean of the Poorest column in the `pneu_care_df` DataFrame. (Hint: This may take a couple lines of code.)\n",
    "\n",
    "*Note: you'll need to get the answer to this question right for code later in this notebook to work, so check the answers if you missed this exercise.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, you will want to calculate the statistics for a particular subgroup of a data column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's say we are comparing the sizes of blue and purple flowers, which is noted in the \"color\" column of the `flower_df`. This code will tell you the mean for every column in the DataFrame except the column that you are using to group your data (i.e. \"color\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.groupby(\"color\").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you only want to see the mean for one column in the DataFrame, you can use the subsampling techniques we learned in part one of this lesson. The subsampling code can come after or before the \"mean\" function, but both must come after the \"groupby\" function. In other words, first we group all the data by color, then we do calculations and subsampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To view the data as a DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.groupby(\"color\").mean()[[\"width\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To view the data as a Series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_df.groupby(\"color\").mean().width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's look at the mean values for each year of the pneumonia care study:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.groupby(\"Year\").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the \"Countries\" column is not represented. Why do you think this is? Let's try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneu_care_df.groupby(\"Year\").mean().Countries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why do you think we get this error?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For each year of the pneumonia care study, how many unique countries were studied?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For each country, what is the most recent year that the study was conducted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other python packages can help you make beautiful visualizations of your data. With Pandas, you can make several simple plots, including histograms, box and whisker plots, bar graphs, scatter plots, and pie charts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with the flower data set, making a simple scatter plot, with petal width on the x axis and petal length on the y axis. We use the \"plot\" function. At the least, we need to include three arguments: the kind of plot to make, the data to use for the x axis, and the data to use for the y axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_scatter = flower_df.plot(kind = \"scatter\", x = \"width\", y = \"length\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>The kind of plot to make can also be called as a function of plot, instead of as an argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_scatter = flower_df.plot.scatter(x = \"width\", y = \"length\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>It looks like there isn't a very strong correlation between petal length and petal width. Next, we might want to see if the relationship between length and width is different for the two different colors of flowers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To label the data points based on the color of the flowers, we subsample the data as two groups: blue flowers and purple flowers. Then we plot both of these datasets to the same scatter plot. We first assign one dataset to one set of axes. Then we can plot the second dataset, calling the first set as the argument \"ax\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blue_axes = flower_df[flower_df.color == \"blue\"].plot(kind = \"scatter\", \n",
    "                                                     x = \"width\", \n",
    "                                                     y = \"length\", \n",
    "                                                     color = 'blue', \n",
    "                                                     label = 'blue flowers')\n",
    "flower_scatter_colors = flower_df[flower_df.color == \"purple\"].plot(kind = \"scatter\", \n",
    "                                                                    x = 'width', \n",
    "                                                                    y = 'length', \n",
    "                                                                    color = 'purple', \n",
    "                                                                    label = 'purple flowers', \n",
    "                                                                    ax = blue_axes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Now we can see a pattern emerging: purple flowers are longer than blue flowers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's make a bar graph of the mean petal lengths for the two groups. First we group by color, then subsample only the length column, then calculate the means, and finally plot the means. I also added an argument for \"title\" to this plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_bar = flower_df.groupby(\"color\")[\"length\"].mean().plot(kind = 'bar', \n",
    "                                                              title = \"Petal length\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>This bar graph shows that the means are different, but it doesn't mean much without error bars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flower_bar_error = flower_df.groupby(\"color\")[\"length\"].mean().plot(kind = 'bar', \n",
    "                                                                    yerr = flower_df.groupby(\"color\")[\"length\"].std(), \n",
    "                                                                    title = \"Petal length\")         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice exercise 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the `pneu_care_df` DataFrame, let's explore the question of wealth as it relates to healthcare access for children with pneumonia symptoms around the world. <br><br>Make a bar graph that shows the means of the Richest and Poorest columns of data. Include a title for your graph. This dataset, if you remember, is using percentages, and not counts; omit the error bars. <br><br>This exercise requires some thinking and piecing together of the previous few examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here's another way to think about this question (this will be challenging, so try a few ideas and then we will discuss):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write code to count how many countries have a higher rate of care for the Richest column than the Poorest column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write code to count how many countries have a higher rate of care for the Poorest column than the Richest column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write code to count how many countries have an equal rate of care for the Richest column and the Poorest column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If we have time, we will walk through how to turn these counts into a pie chart!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Great work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize part two, we wrote code to:\n",
    "- Run data aggregation functions on a dataframe\n",
    "- Run data aggregation functions on single columns\n",
    "- Run data aggregation functions on subgroups of columns\n",
    "- Change the type of a column to a float to ignore missing data\n",
    "- Make scatter and bar plots with labeled axes and error bars\n",
    "- Make color-coded scatter plots and pie charts"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}