dataframes.Rmd

---
title: "Data Frames"
author: Christina Maimone
output:
  html_document:
    df_print: paged
    code_download: TRUE
    toc: TRUE
    toc_float: TRUE
editor_options:
  chunk_output_type: console
---

# Introduction

This workshop covers some of the basics of working with data frames.  We're focused here on using the built-in, base R functions and features to work with data frames.  It's good to understand these basics so that you can read R code that others write, and you can perform basic functions without other packages.  However, in my daily work, I use the [tidyverse](https://github.com/nuitrcs/r-tidyverse) packages to work with data frames.

# Getting Data

## Reading in a Data File

Most of the time, you'll read in data from a file.  For example:

```{r}
evp <- read.csv("data/ev_police_jan.csv")
evp
```

As with other variables, when we type the name of the data frame, it will print the contents of the variable to the console.  


Note: Evanston police data comes from the [Stanford Open Policing Project](https://openpolicing.stanford.edu/data/).  This is a small subset of data: just January 2017.

TIPS: if you get an error message that a file can't be found when you're trying to import it: 

1) Check the spelling of the filename for typos

2) Check your working directory (`getwd()`) and make sure the path to the file is correct and completely specified given what your working directory is.

3) Make sure the file is actually in the folder you think it is.  I recommend copying or moving any downloaded files into the project/directory for this workshop so you know you really have the file.  I've seen some problems on Windows computers in particular before where a .zip file isn't really unzipped - it's just letting you see inside without actually expanding the contents and creating real files.


## From a Package

Today, we're going to work with data on penguins from https://github.com/allisonhorst/palmerpenguins.  It's been included in an R package.  

```{r, installpackages, eval=FALSE}
install.packages("palmerpenguins")
```

```{r}
library(palmerpenguins)
```

There's a data frame in the package called penguins.  Like with functions in a package, we can use it once we've loaded the package with the library command above.

```{r}
penguins
```



## Manual Creation

It's also possible to create a data frame from scratch.  You will not do this often.
```{r, manualdf}
x <- data.frame(month=month.name,
                index=1:12,
                days=c(31,28,31,30,31,30,31,31,30,31,30,31))
x
```

# Data Shape and Names

We're going to work with the penguins data frame

```{r, loaddata}
penguins
```

What is `penguins`? `class()` tells us the type of the object -- what's stored in the variable.
```{r}
class(penguins)
```

"tbl_df" is a tibble data frame.  These behave a little bit differently from normal data frames.  You'll see tibbles instead of data frames within the tidyverse set of packages (and those packages that work within that framework).

The biggest difference is that tibbles give you a tibble back when subsetting with [], while data frames sometimes give you a vector.

View the first few rows
```{r}
head(penguins)
```

Or use the viewer:
```{r, eval=FALSE}
View(penguins)
```

Clicking on the name of the data frame in the environment tab will also open the viewer.

Dimensions of the data frame
```{r}
dim(penguins)
```

Rows is the first number, columns second.  

What will the length of a data frame be?  
```{r}
length(penguins)
```

\# of columns.  This is because it's technically a list of vectors (lists are a different type).  Don't use length with a data frame; use `ncol()` instead:
```{r}
ncol(penguins)
```


```{r}
nrow(penguins)
```

What are the variable names?  The columns
```{r}
names(penguins)
```

Rows have names too
```{r}
rownames(penguins)
```

These were generated by default.  It's best to ignore row names.  If there's unique information in the row names, make sure it gets put in an actual column in the data frame instead.

What are the column types?
```{r}
str(penguins)
```

Some variables are factors.  These are categorical variables with a limited set of values.  The different values are called levels.   

If you read data in from a data frame, these would be read in as character data by default (AKA strings, text).  Because this data frame was in a package, the columns had already been converted to factors.  

Quick summary of every column
```{r}
summary(penguins)
```


## EXERCISE

Run the code that creates the `x` data frame.

Using the `x` data frame, write commands to figure out:

* How many rows and columns?
* Names of the variables?

```{r}
x <- data.frame(month=month.name,
                index=1:12,
                days=c(31,28,31,30,31,30,31,31,30,31,30,31))
```


# Indexing Basics

Indexing works like it does for vectors, but there are two dimensions: rows and columns.  Rows come first, then columns.

Select first row
```{r}
penguins[1,]
```


Select first two rows
```{r}
penguins[1:2,]
```


Select first column
```{r}
penguins[,1]
```

This gave us a tibble/data.frame back, but with `x`:
```{r}
x[,1]
```

we get a vector back.  This is one difference between tibbles and regular data frames.

We can select rows and columns at the same time:

```{r}
penguins[1:2, 4:5]
```

If we want rows or columns that aren't next to each other, you can use a vector.

```{r}
x[c(1, 3), ]
```


## EXERCISE

Using `x` created above: select rows 2 through 5, and columns 1 through 2 from `x`
```{r}

```

# Names

Reference columns by name with `$` notation (no quotes on names)
```{r}
names(penguins)
```

```{r}
penguins$species
```

Note that the `$` notation got us a vector back.

```{r}
penguins$bill_length_mm
```

Use names in `[]`: put them in quotes
```{r}
penguins[,"species"]
```

Since we used `[]`, we get a tibble/data.frame back this time
```{r}
penguins[,"bill_length_mm"]
```

Multiple columns by name, need to use a vector of names:
```{r}
penguins[,c("island", "species")]
```


## EXERCISE

Using the `x` data frame we created above:

* Select the days column using `$` notation
* Select the month and days columns by name using `[]`

```{r}

```



# Boolean/Conditional Selection

If we have a boolean vector (`TRUE` and `FALSE` values) that is the same length as the number of rows or columns, we can use it to select from the data frame as we did with vectors.
```{r}
penguins$bill_length_mm < 34
```

Select the rows where bill length is less than 34:

```{r}
penguins[penguins$bill_length_mm < 34,]
```


Note the rows of `NA`.

Remember: the expression inside [] needs to be a complete expression.  So we must use both the data frame name and the column name.  

If we don't want the missing rows included:

```{r}
sum(is.na(penguins$bill_length_mm))  # how many missing?
penguins[is.na(penguins$bill_length_mm), ]  # which rows have missing values

# select rows where bill_length_mm is not missing and < 34
penguins[!is.na(penguins$bill_length_mm) & penguins$bill_length_mm < 34,]
```


If I forget the `,` in the `[]`:
```{r, eval=FALSE}
penguins[penguins$bill_length_mm < 34]
```

It tries to index the columns instead, and our vector is too long.

Multiple conditions
```{r}
penguins[penguins$bill_length_mm < 34 & penguins$bill_depth_mm < 16,]
```

```{r}
penguins[penguins$bill_length_mm < 34 | penguins$bill_length_mm > 58,]
```

## EXERCISE

Using `x` created above: select rows from `x` with 31 days.  Use `==` for testing for equality.

```{r}

```



# Renaming Columns

Normally we don't need to do this first, but I want to keep the current names so I can reset the names later

```{r}
oldnames <- names(penguins)  # save so we can reset later
```

Just like we use `names()` to get the names, use the same function to assign the names.  This is different syntax than we see other places in R; `names()` is a special type of function called a replacement function.  

I can change the name of the first variable with:

```{r}
names(penguins)[1]
names(penguins)[1] <- "boo"  # change the name of the first column
names(penguins)
```

```{r}
names(penguins) <- c("a", "b", "c", "d", "e", "f", "g", "h")  # change all of the column names
head(penguins)
```

Put the old names back
```{r}
names(penguins) <- oldnames
head(penguins)  # check
```


# Working with Variables

We can work with individual columns as a vector by themselves:

```{r}
max(penguins$bill_length_mm)
```

```{r}
max(penguins$bill_length_mm, na.rm=TRUE)
```

Now to get the observations (rows) for the penguins with that max value:

```{r}
penguins[penguins$bill_length_mm == 59.6,]
```

But, we don't want to have to look up the value first, and then type it in.  We can put the expression that got us that max value directly in the code:

```{r}
penguins[penguins$bill_length_mm == max(penguins$bill_length_mm, na.rm=TRUE),]
```

We could also break it out into steps and save the max value in a variable we can use:

```{r}
max_bill_length <- max(penguins$bill_length_mm, na.rm=TRUE)
penguins[penguins$bill_length_mm == max_bill_length,]
```

Why?  What if something in the data changes?  Then we'd have the wrong value written in (hard coded) into our code.  We want to avoid "hard coding" specific values that are derived from data into our scripts where we can.  

## EXERCISE

Using `x` created above (data frame of months): select the rows from `x` where `days` is at its minimum value.


```{r}

```

You can also use `which.min()` or `which.max()`  to get the index location of the first value with the minimum or maximum value respectively (if there's more than one you only get one):

```{r}
x[which.min(x$days), ]
```



# Making new variables

We can add a new variable to the data frame by naming it with the `$` notation, and assigning a value to it:.  For example, make a variable that has bill length in **centimeters (cm)** instead of **millimeters (mm)**

```{r}
penguins$bill_length_cm <- penguins$bill_length_mm / 10  # make new variable: note CM instead of MM in the name
names(penguins)  # check to see that it was added
penguins[, c("bill_length_cm", "bill_length_mm")]  # select the two vars to view them
```




## EXERCISE

Using `x` created above: make a new variable as part of `x`, called `weeks`, that is the number of days divided by 7.

```{r}


```



# Missing values

Remember: `NA` denotes a missing value.  

Any missing values?  They will show up in summary() output:
```{r}
summary(penguins)
```

Look at the rows where `body_mass_g` is missing:
```{r}
is.na(penguins$body_mass_g)
penguins[is.na(penguins$body_mass_g),]
```

Remove rows where `body_mass_g` is missing by selecting rows where the value is not missing:
```{r}
penguins[!is.na(penguins$body_mass_g),]
penguins <- penguins[!is.na(penguins$body_mass_g),]
```

The function `complete.cases()` can be used to find which rows have no missing values at all:


## EXERCISE

Run the code to create a y data frame.  Then remove the rows where the height variable has a missing value.

```{r}
y <- data.frame(age = c(40, 38, 44, 41, 68, 6, 35, 29),
                height = c(NA, 65, 68, 70, 63, 50, NA, 73))
```


# Replacing Values

```{r}
table(penguins$species)
```

`species` is a factor (a special type for categorical variables); let's un-factor it first so we can change values -- this is an extra step related to it being a factor.  We'll convert it to be data of type character, which is just free text.

```{r}
penguins$species <- as.character(penguins$species)
```

Here are the general steps we'd do if we didn't have a factor

```{r}
penguins$species[penguins$species == "Gentoo"]  # which observations do we want to replace?

penguins$species[penguins$species == "Gentoo"] <- "gentoo"  # set the value

table(penguins$species)
```


## EXERCISE

Using `x` created above: replace the `month` value "December" with the value "Dec" instead

```{r}

```


# Practice

Here are some practice exercises for working with data frames.  They all use the policing data here: 

```{r}
evp <- read.csv("data/ev_police_jan.csv")
```

If you mess up the data frame, you can always read it back in from the file.  

## EXERCISE 1

What are the variable names, and how many observations are there?

```{r}

```




## EXERCISE 2

Are there any missing values in the data?

```{r}

```



## EXERCISE 3

What is the most common location?  You may want to use the `table()` function.  

Hint: you can use the `sort()` function in combination with the `table()` function: `sort(table(x))`

```{r}

```


## EXERCISE 4

How many stops found contraband?

Hint (for one approach to this): if you `sum()` a boolean variable, TRUE = 1, FALSE = 0, so it counts the TRUE values.  Remember to deal with missing values with `sum()`: `sum(x, na.rm=TRUE)`

```{r}

```


## EXERCISE 5

Rename the column "raw_row_number" to "id"


```{r}

```


## EXERCISE 6

Subset the data frame to just have rows where subject_sex is "female".

```{r}

```


## EXERCISE 7

Make a new column in the data frame called evanston that is TRUE if the location is 60201 or 60202 and FALSE otherwise.

```{r}

```


## EXERCISE 8

This exercise is more challenging than the others.

What is the *make* of the oldest vehicles in the data set.  

Hint: there are two observations in the data that both have the minimum vehicle_year.  

```{r}

```


## EXERCISE 9

This exercise is more challenging than the others.

What percentage of men in the dataset drive a Toyota ("TOYT")?  How does this compare to the percentage of women who drive a Toyota?  

What percentage of Toyota drivers are male?  What percentage of drivers in the data set are male?

```{r}

```