saveObjects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <br>json and pickle files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we're talking about saving Python objects like dictionaries, lists and strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't already found a need to save Python objects, you will."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your Jupyter notebook is getting really long and slow, save that dictionary that you carefully built, shut down your kernel, and continue your analysis in another notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My general advice is, at a minimum, to be sure to keep data collection, cleaning, analysis, and visualization in different notebooks. It is also often good to separate different analyses, especially if they produce large objects that are saved in memory, as this will slow down your notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Saving and loading objects using built-in Python functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might already save and load dictionaries and lists with your own code by **writing** text files, and then **reading** and parsing text files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speed_dict = {\"Peregrine falcon\": \"242 mph\", \n",
    "              \"Golden eagle\": \"150–200 mph\", \n",
    "              \"White-throated needletail swift\": \"105 mph\", \n",
    "              \"Eurasian hobby\": \"100 mph\", \n",
    "              \"Mexican free-tailed bat\": \"100 mph\", \n",
    "              \"Frigatebird\": \"95 mph\", \n",
    "              \"Rock dove (pigeon)\": \"92.5 mph\", \n",
    "              \"Spur-winged goose\": \"88 mph\", \n",
    "              \"Gyrfalcon\": \"80 mph\", \n",
    "              \"Grey-headed albatross\": \"79 mph\", \n",
    "              \"Cheetah\": \"68.0–75.0 mph\", \n",
    "              \"Sailfish\": \"67.85 mph\", \n",
    "              \"Anna's hummingbird\": \"61.06 mph\", \n",
    "              \"Swordfish\": \"60 mph\", \n",
    "              \"Pronghorn\": \"55.0 mph\", \n",
    "              \"Springbok\": \"55 mph\", \n",
    "              \"Quarter Horse\": \"55.0 mph\", \n",
    "              \"Blue wildebeest\": \"50.0 mph\", \n",
    "              \"Lion\": \"50.0 mph\", \n",
    "              \"Blackbuck\": \"50 mph\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>We can open a new file, loop through the keys and values in our dictionary, and write them to the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimals.txt\", \"w\") as f:\n",
    "    for k, v in speed_dict.items():\n",
    "        f.write(k + \"\\t\" + v + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>Then we can read in the file and change its contents into a Python object. I often use a list comprehension or dictionary comprehension to change .txt files into a list or dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimals.txt\", \"r\") as f:\n",
    "    txt_speed_dict = {line.split(\"\\t\")[0]: line.rstrip(\"\\n\").split(\"\\t\")[1] for line in f}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt_speed_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>**Do you have to write your own code to save and then recreate these objects? Is there something faster than looping through all the items?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Serializing Python objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of converting a Python object into a format that can be stored is called **serialization**. The process of reconstructing that data is called **deserialization**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will cover two different file formats for serialization: pickle and json. There are now many other formats for serializing data, but these are two of the most recognizable formats, and they have very simple commands. <br><br>First let's go over the differences between the two and why you might want to use one or the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <br><br>pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- commonly ends in .pkl or .pickle\n",
    "- stores objects in binary format\n",
    "- can be used to save any Python object, including your object classes and functions\n",
    "- pickle objects can only be opened in Python\n",
    "- is binary, so is not human readable - you can't open the files and read the data\n",
    "- WARNING: can be used to store malicious code!!! So never ever open pickle files you receive from someone else. Only open your own pickle files, and only if no one else has had access to them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <br><br>json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ends in .json\n",
    "- can be used to store strings, integers, floats, lists, dictionaries, tuples, and booleans. Cannot store classes or functions\n",
    "- can be opened in other languages\n",
    "- is human readable - you can look at the actual file and read your data\n",
    "- cannot be used to store malicious code\n",
    "- usually a little faster to write and read than pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Let's practice with pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speed_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>First, we import the pickle module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To write our dictionary to a pickle file, we open the filename in write mode using a with statement, but we have to open in **write binary mode**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After our with statement, we **dump** the data into the pickle file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimals.pkl\", \"wb\") as f:\n",
    "    pickle.dump(speed_dict, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To load a pickle file, we use the same syntax, except we open the file in **read binary mode** and we **load** the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimals.pkl\", \"rb\") as f:\n",
    "    pkl_speed_dict = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pkl_speed_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Pickle Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speed_list = [\"Peregrine falcon\", \n",
    "              \"Golden eagle\", \n",
    "              \"White-throated needletail swift\", \n",
    "              \"Eurasian hobby\", \n",
    "              \"Mexican free-tailed bat\", \n",
    "              \"Frigatebird\", \n",
    "              \"Rock dove (pigeon)\", \n",
    "              \"Spur-winged goose\", \n",
    "              \"Gyrfalcon\", \n",
    "              \"Grey-headed albatross\", \n",
    "              \"Cheetah\", \n",
    "              \"Sailfish\", \n",
    "              \"Anna's hummingbird\", \n",
    "              \"Swordfish\", \n",
    "              \"Pronghorn\", \n",
    "              \"Springbok\", \n",
    "              \"Quarter Horse\", \n",
    "              \"Blue wildebeest\", \n",
    "              \"Lion\", \n",
    "              \"Blackbuck\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code above to save `speed_list`. Write code to save the speed_list as a pickle file called `speedyAnimals.pkl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now write code to open the pickle file you just created as a list called `pickle_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pickle_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>Notice that with both the dictionary and the list, pickle automatically maintained the type of Python object without us needing to specify what type of object it was. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Let's practice with json."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>With json, we can open in regular write and read modes. We still use dump and load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimal.json\", \"w\") as f:\n",
    "    json.dump(speed_dict, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"fastestAnimal.json\", \"r\") as f:\n",
    "    json_speed_dict = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_speed_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>Json Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "falcon = \"Peregrine falcons are among the world's most common birds of prey and live on all continents except Antarctica. They prefer wide-open spaces, and thrive near coasts where shorebirds are common, but they can be found everywhere from tundra to deserts. Peregrines are even known to live on bridges and skyscrapers in major cities.\"\n",
    "print(falcon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code above to store the string `falcon`. Write code to save the string as a json file called `falcon_info.json`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write code to open up the file you just created. Save it as a string `json_string`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <br><br>BONUS: Saving pandas dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always save a pandas dataframe as a csv, which is a type of plain text file that is human readable. You can also save a dataframe as a pickle or json file, which are faster to write and faster to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame.from_dict(speed_dict, orient=\"index\", columns=[\"speed\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Now that we have a dataframe, we can save it as a csv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"speeds.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Or we can save it as a pickle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle(\"speeds.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>To confirm that it worked, let's load our pickled dataframe back in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.read_pickle(\"speeds.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>You can also pickle a dataframe object the same way we did earlier, by opening a new file in write binary mode and dumping the dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Let's save our dataframe with json. You **cannot** save a dataframe as a json file using the open/dump method, but you can use this pandas function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_json(\"speeds.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3 = pd.read_json(\"speeds.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>There are other formats you can use to serialize pandas dataframes that are safer and faster than pickle, including *feather* and *parquet*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Also, keep in mind that json and pickle are used for storing and loading your Python objects faster, but they are not used for compression. If smaller file size is your goal, you will want to use a compression package, with or without serialization, like gzip or brotli."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}