python_series_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of contents:\n",
    " * [Part 1](#Introduction-to-Python---Part-1)\n",
    " * [Part 2](#Introduction-to-Python---Part-2)\n",
    " * [Part 3](#Introduction-to-Python---Part-3)\n",
    " * [Part 4](#Introduction-to-Python---Part-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Python - Part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<figure class=\"half\" style=\"display:flex\">\n",
    "    <img style=\"width:40%\" src=\"monty_python.jpg\">\n",
    "    <img style=\"width:40%\" src=\"learning_python.jpeg\">\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Python is called an interpreted language but is compiled to bytecode. The .py source code is first compiled to byte code as .pyc. This byte code can be interpreted (official CPython), or JIT compiled (PyPy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Jupyter notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Useful to write both code and text in the same place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Double click on a cell, choose in the top menu either \"code\" or \"Markdown\" for text.\n",
    "After writing something in the cell, just press `shift` + `Enter` to view the result (or click on the \"Run\" button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### In addition of headings (#, ##, ###, ...) with markdown you can write bullet lists (\\*)\n",
    "* Use #, ##, ###, etc. on the first line for the size of your headings\n",
    "* Use single star (\\*) for each entry of your bullet list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Headings are obviously optional. In addition of bullet lists, you can create numbered lists using the format \"number\" \"fullpoint(.)\" \"space\":\n",
    "\n",
    "1. Markdown allows you to write simple text\n",
    "2. But also LaTeX formulas, by simply putting it between dollar signs: $\\sum_{n=1}^{\\infty} \\frac{1}{2^n} = 1$\n",
    "3. You can escape special markdown characters with a backslash (\\\\): \\*\n",
    "4. You can even quote commands by using apostrophes (\\`), as in: use `jupyter notebook` to launch Jupyter notebook from the terminal (after installing python and Jupyter)\n",
    "5. You can put words in *italics* by writing them in between stars (\\*) or in **bold** using two stars (\\*\\*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Now let's do some python code. Comments can be made with a hash sign (#).\n",
    "2 + 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "While the result of executing a cell (`shift` + `Enter`) in *Markdown* is **formatting that cell**, for *code* (like `python` code) the result is **running that code and displaying the output (if any)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The jupyter notebook can be in either of two modes:\n",
    "\n",
    "* **edit mode** when you are editing a cell (i.e. typing Markdown or code): **the edited cell is green**\n",
    "* **command mode** when you can can execute commands from the keyboard (e.g. adding or deleting a cell): **the cell marker becomes blue**  \n",
    "\n",
    "To go from edit mode to command mode, press the `Esc` key. To go from command mode to edit mode, press the `Enter` key. This is useful to avoid using the mouse and thus saving time.\n",
    "\n",
    "Press the keyboard button on the menu at the top to see all keyboard shortcuts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Exercise\n",
    "1. Create two new cells clicking on `+` in the top left corner, once for each cell\n",
    "2. Make the first cell a Markdown cell, by clicking on it and selecting `Markdown` in the menu (top center)\n",
    "3. Edit it by double-clicking it, then write a bullet list of items\n",
    "4. Make the second cell a code cell, by clicking on it and selecting `code` in the menu (top center)\n",
    "5. Edit it by double-clicking it, then write an arithmetic operation\n",
    "6. Make sure to execute both cells (if not already done) by selecting it and pressing the \"Run\" button (top left)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Exercise (part 1)\n",
    "Do the same as above but without using the mouse except for the initial step `0`.\n",
    "\n",
    "0. Click on the last cell you created and make sure the cell marker is blue (otherwise press the `Esc` key)\n",
    "1. Create two new cells by pressing twice on `b` on the keyboard\n",
    "2. Navigate to the first cell you created using the up / down arrows on your keyboard\n",
    "3. While in command mode (make sure the cell marker is blue otherwise press the `Esc` key), press the `m` key to make the cell a Markdown cell\n",
    "4. Switch to edit mode by pressing the `Enter` key, then write a numbered list of items\n",
    "5. Execute the cell by pressing `Shift` + `Enter`: since the cell is a markdown cell this will format the cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Exercise (part 2)\n",
    "6. Executing the previous cell will bring you to the next cell. While in command mode (make sure the cell marker is blue otherwise press the `Esc` key), press the `y` key to make the cell a code cell\n",
    "7. Switch to edit mode by pressing the `Enter` key, then write an arithmetic operation\n",
    "8. Execute the cell by pressing `Shift` + `Enter`: since the cell is a code cell this run the code inside the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python variables and datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Variables allow for generalization and reusability of programming instructions. A variable is a reference to an object. The object can be:\n",
    "\n",
    "* A sequence of characters, i.e. a \"string\":\n",
    "  * `my_var = \"Felix\"`\n",
    "  * `my_var = \"My cat is called Felix\"`\n",
    "  * `my_var = \"/home/felix/how_to_manipulate_humans.txt\"`\n",
    "* A number (integer, float)\n",
    "  * `my_var = 3.14`\n",
    "* A list of objects:\n",
    "  * `my_vars = [\"felix\", \"/home/felix/file.txt\", 3.14]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Variable names\n",
    "\n",
    "* can only contain letters, digits, and underscore `_` (no spaces!)\n",
    "* cannot start with a digit\n",
    "* are case sensitive (`fruit`, `FRUIT` and `Fruit` are three different variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's look at some code. In python spaces can be used on each side of equal sign, and the content of the variable is simply obtained by indicating the variable name (no need for `$` like in bash)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# a refers to an integer (int)\n",
    "a = 2\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Functions (to be discussed later) are like commands in bash, but take arguments inside parentheses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# simple_pi refers to a float\n",
    "simple_pi = 3.14\n",
    "print(simple_pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Compare with the bash version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "pi=3.14\n",
    "echo ${pi}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As in bash, strings (i.e. sequence of characters) are indicated inside quotes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# favorite_fruit refers to a string (str)\n",
    "favorite_fruit = \"kiwi\"\n",
    "print(favorite_fruit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Python can define a boolean value. A boolean is a variable which can only have one of two values: `True` or `False`. For convenience `0` is considered `False` while anything different from `0` is considered `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# like_kiwi refers to a boolean\n",
    "like_kiwi = True\n",
    "print(like_kiwi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can check the type of a variable with `type`.  \n",
    "Note that in Python arguments to a function are separated by `,` and spaces can be used before (not recommended) or after (recommended) the comma. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(type(a), type(simple_pi), type(favorite_fruit), type(like_kiwi), sep=' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print?\n",
    "# Output: print(value, ..., sep=' ', end='\\n', file=sys.stdout, flush=False) [can also use \"Shift\"+\"Tab\" for help]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Simple datatypes are **immutable** (you cannot change them)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Setting a variable to a new immutable object (e.g. integer) will refer to a new object entirely\n",
    "a = 3\n",
    "a = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"immutvar_newval_up.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Consider a variable 'a' referring to an immutable object (e.g. an int).\n",
    "# Setting a variable 'b' to 'a' can also be thought as creating a new object, with the same value but independent\n",
    "a = 3\n",
    "b = a \n",
    "print('a is', a)\n",
    "print('b is', b)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<tr><td><img src='immutvar_by_val.png'></td><td><img src='immutvar_by_val_wall.png'></td></tr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Since they are completely independent, changing one will not affect the other\n",
    "a = 3\n",
    "b = a\n",
    "a = 4\n",
    "print('a is', a)\n",
    "print('b is', b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"immutvar_by_val_newval.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Strings, similarly to list (cf next section), are 0-indexed sequences\n",
    "atom_name = 'helium'\n",
    "print('letter at index 0 is', atom_name[0])\n",
    "print('letter at index 4 is', atom_name[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=str_indexing_only.svg>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Do not forget that strings are immutable (you cannot change them)\n",
    "atom_name[1] = 'i'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# However you can do many things with strings, for example you can concatenate them\n",
    "first_name = 'John'\n",
    "last_name = 'Doe'\n",
    "full_name = 'Mr' + ' ' + first_name + ' ' + last_name\n",
    "print(full_name) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You can slice them, i.e. extract several sequential characters at a time\n",
    "# <sub_string> = <main_string>[start:stop] where stop is the index AFTER the last wanted character\n",
    "red_fruit = 'strawberry'\n",
    "sub_string = red_fruit[2:5]\n",
    "print('sub_string is:', sub_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You can ommit the starting index if you want to start from the beginning\n",
    "# Similarly, you can ommit the stop index if you want to stop at the end\n",
    "red_fruit = 'strawberry'\n",
    "sub_string = red_fruit[5:]\n",
    "print('sub_string is:', sub_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python data structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Data structures prove often essential to write any kind of code, to facilitate reasoning / processing (hence shopping list and calories table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits = ['kiwi', 'mango']\n",
    "print('fav_fruits:', fav_fruits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits_cals = [42, 60]\n",
    "print('fav_fruits_cals:', fav_fruits_cals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To use functions on objects such as lists, you can use either:\n",
    "\n",
    "* **general functions** with `<function_name>(<object_name>)` such as `len(fav_fruits)` (to compute length with `len`), OR \n",
    "* **specific object functions**, called **methods**, with  `<object_name>.<method_name>` such as `fav_fruits.extend(['apple', 'pear'])` (to append a list of new elements to the original list with `extend`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "len(fav_fruits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To see the list of all methods applying to an object, use general function `dir`. You can also use \"TAB\" after typing a dot ('.') after your variable name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dir(fav_fruits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fav_food = fav_fruits\n",
    "fav_food.extend(['4cheese_pizza', 'banana_split'])\n",
    "print('fav_food:', fav_food)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_food_cals = fav_fruits_cals\n",
    "fav_food_cals.extend([1200, 900])\n",
    "print('fav_food_cals:', fav_food_cals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "total_fruit_diet_cals = sum(fav_fruits_cals)\n",
    "print('total_fruit_diet_cals:', total_fruit_diet_cals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"mutvar_error_article.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## WARNING: Lists are mutable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"mutvar_by_ref.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print('fav_fruits:', fav_fruits)\n",
    "print('fav_fruits_cals:', fav_fruits_cals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits = ['kiwi', 'mango']\n",
    "fav_fruits_cals = [42, 60]\n",
    "# Copy then extend fav_food and fav_food_cals\n",
    "fav_food = fav_fruits.copy() # copy\n",
    "fav_food.extend(['4cheese_pizza', 'banana_split'])\n",
    "fav_food_cals = fav_fruits_cals.copy() # copy\n",
    "fav_food_cals.extend([1200, 900])\n",
    "print('fav_fruits: ', fav_fruits)\n",
    "print('fav_fruits_cals ', fav_fruits_cals)\n",
    "total_fruit_diet_cals = sum(fav_fruits_cals)\n",
    "print('total_fruit_diet_cals:', total_fruit_diet_cals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"mutvar_error_articleerratum.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Typically there are no problems such as above if you do not initialize a list with another\n",
    "fav_fruits = ['kiwi', 'mango']\n",
    "fav_food = fav_fruits + ['4cheese_pizza', 'banana_split']\n",
    "print('fav_fruits: ', fav_fruits)\n",
    "print('fav_food: ', fav_food)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Python overloads traditional operators in context you may not have think of\n",
    "print(['we', 'will'] * 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(['we', 'will'] * 2 + ['rock', 'you'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "food_cals = {\n",
    "          'kiwi': 42,\n",
    "          'mango': 60,\n",
    "          '4cheese_pizza': 1200,\n",
    "          'banana_split': 900\n",
    "          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "food_cals['mango']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "food_cals.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "food_cals.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "food_cals[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## WARNING: Dictionaries are mutable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Tuples: like lists but immutable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_experiment_params = (42, 'method3', 3.14, 'paramx', 'paramy')\n",
    "office_lat_long = (46.2221685, 6.1482567)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dir(office_lat_long)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Unpack tuple (number of variables on the left should be equal to len(tuple))\n",
    "office_lat, office_long = office_lat_long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "##### Formatting strings so that to use the content of variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# PREFERRED: f-strings (NOTE: you can see a \"f\" just before the string)\n",
    "print(f'My office latitude is {office_lat} and its longitude is {office_long}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# NOT preferred\n",
    "print('My office latitude is {} and its longitude is {}'.format(office_lat, office_long))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# NOT preferred\n",
    "print('My office latitude is {lat} and its longitude is {long}'.format(lat=office_lat, long=office_long))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "1. Create a variable named `my_fruit` referring to the string 'pineapple'\n",
    "2. Using indices, slice (i.e. extract) 'pine' from your variable\n",
    "3. Use either `dir` or \"TAB\" (after writing '.' after your variable name) to display all methods applicable to your variable `my_fruit`\n",
    "4. Use the list above to find a method to test if what is in your variable `my_fruit` ends with `apple`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "my_fruit = \"pineapple\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pine\n"
     ]
    }
   ],
   "source": [
    "print(my_fruit[0:4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__add__',\n",
       " '__class__',\n",
       " '__contains__',\n",
       " '__delattr__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__getitem__',\n",
       " '__getnewargs__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__iter__',\n",
       " '__le__',\n",
       " '__len__',\n",
       " '__lt__',\n",
       " '__mod__',\n",
       " '__mul__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__rmod__',\n",
       " '__rmul__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " 'capitalize',\n",
       " 'casefold',\n",
       " 'center',\n",
       " 'count',\n",
       " 'encode',\n",
       " 'endswith',\n",
       " 'expandtabs',\n",
       " 'find',\n",
       " 'format',\n",
       " 'format_map',\n",
       " 'index',\n",
       " 'isalnum',\n",
       " 'isalpha',\n",
       " 'isascii',\n",
       " 'isdecimal',\n",
       " 'isdigit',\n",
       " 'isidentifier',\n",
       " 'islower',\n",
       " 'isnumeric',\n",
       " 'isprintable',\n",
       " 'isspace',\n",
       " 'istitle',\n",
       " 'isupper',\n",
       " 'join',\n",
       " 'ljust',\n",
       " 'lower',\n",
       " 'lstrip',\n",
       " 'maketrans',\n",
       " 'partition',\n",
       " 'replace',\n",
       " 'rfind',\n",
       " 'rindex',\n",
       " 'rjust',\n",
       " 'rpartition',\n",
       " 'rsplit',\n",
       " 'rstrip',\n",
       " 'split',\n",
       " 'splitlines',\n",
       " 'startswith',\n",
       " 'strip',\n",
       " 'swapcase',\n",
       " 'title',\n",
       " 'translate',\n",
       " 'upper',\n",
       " 'zfill']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(my_fruit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_fruit.endswith(\"apple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "1. Create a dictionary `wish_list` representing a list of items you'd like with item names as keys and item prices as values\n",
    "2. Using your dictionary `wish_list`, print the list of item names\n",
    "3. Using your dictionary `wish_list`, print the total cost of your wish list (hint: use the function `sum`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "wish_list = {\n",
    "    \"hawaii_tickets\": 1200,\n",
    "    \"diving_lessons\": 120,\n",
    "    \"good_books\": 12\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['hawaii_tickets', 'diving_lessons', 'good_books'])\n"
     ]
    }
   ],
   "source": [
    "print(wish_list.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1332\n"
     ]
    }
   ],
   "source": [
    "total_cost = sum(wish_list.values())\n",
    "print(total_cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Control structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Most programs need to do repeated actions, and perform different actions according to different conditions. That is why control structures such as \"for loop\" and \"if else statements\" are essential."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### For loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A for loop will iterate for each item in a sequence. A typical example is iterating over a list of integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "seq = [0, 1, 2, 3]\n",
    "for i in seq:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In Python, a built-in function `range` is typically used to generate this sequence. Only indicating the integer `n` to stop at (exclusive) is enough, such as in `range(n)`. Both starting and stopping integers could be used, as in `range(k, n)`. And in the latter case, a `step` can even be set, as in `range(k, n, step)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n = 3\n",
    "for i in range(n): # same as: for i in range(0, n)\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n = 3\n",
    "for i in range(1, n+1): # remember that stopping integer is exclusive\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n = 3\n",
    "for i in range(0, n, 2): # the index increase by 2 at each iteration\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The \"for block\" is indicated by an \"indentation\" (typically 4 spaces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n = 3\n",
    "for i in range(n):\n",
    "    print(i)\n",
    "    print('launch!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n = 3\n",
    "for i in range(n):\n",
    "    print(i)\n",
    "print('launch!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The sequence does not have to be a sequence of integers, it can be any \"iterable\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fav_fruits = ['kiwi', 'mango', 'papaya']\n",
    "for fruit in fav_fruits:\n",
    "    print(fruit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Python allows for concise and elegant syntax and this shows when extracting both the iterating index and the corresponding item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Other programming language\n",
    "n = len(fav_fruits)\n",
    "for i in range(n):\n",
    "    fruit = fav_fruits[i]\n",
    "    print(f'Rank {i+1}: {fruit}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Python uses the built-in function \"enumerate\" to extract both index and item at the same time\n",
    "for i, fruit in enumerate(fav_fruits):\n",
    "    print(f'Rank {i+1}: {fruit}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This shows as well when iterating over dictionary items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "food_cals = {\n",
    "          'kiwi': 42,\n",
    "          'mango': 60,\n",
    "          '4cheese_pizza': 1200,\n",
    "          'banana_split': 900\n",
    "          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Other programming language\n",
    "all_food_names = food_cals.keys()\n",
    "for food_name in all_food_names:\n",
    "    n_cals = food_cals[food_name]\n",
    "    print(f'Food item {food_name} has {n_cals} calories')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Python extract directly the (key, val) pairs with the \"items\" method function\n",
    "for food_name, n_cals in food_cals.items():\n",
    "    print(f'Food item {food_name} has {n_cals} calories')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## If - else statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "An \"if - else\" statement allows to execute commands only if a logical expression is True. The logical expression is called \"Boolean expression\" and is always either True or False. If False the block of commands is not executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Exam for which grade < 5 is fail, grade = 5 requires to resit exam, and grade > 5 is a pass\n",
    "grade = 7\n",
    "if grade < 5:\n",
    "    print(\"Grade too low\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "An `else` statement is executed if the `if` statement is false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grade = 7\n",
    "if grade < 5:\n",
    "    print(\"Grade too low\")\n",
    "else:\n",
    "    print(\"Grade high enough\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One or more `elif` statements can be added to add tests. Tests are always evaluated only once, in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grade = 7\n",
    "if grade < 5:\n",
    "    print(\"Grade too low\")\n",
    "elif  grade == 5:\n",
    "    print(\"Please resit exam\")\n",
    "elif grade < 8:\n",
    "    print(\"Grade high enough\")\n",
    "elif grade <= 10:\n",
    "    print(\"Excellent grade\")\n",
    "else:\n",
    "    print(\"Your grade is greater than 10!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Boolean expressions can be combined with boolean operators: `and`, `or` or `not`. Below is an example with a dictionary having as values other dictionaries (values can be any objects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "shopping_list = {\n",
    "              'kiwi': {'cals': 42, 'price': 2},\n",
    "              'mango': {'cals': 60, 'price': 5},\n",
    "              '4cheese-pizza': {'cals': 1200, 'price': 18},\n",
    "              'banana split': {'cals': 900, 'price': 9}\n",
    "              }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for food_name, food_info in shopping_list.items():\n",
    "    if (not food_info['cals'] < 500) and (food_info['price'] < 15):\n",
    "        print(f'Dinner tonight could be {food_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "1. Create a variable 'my_dict' referring to a dictionary (any content is fine)\n",
    "2. Loop on all methods available for this dictionary, and only print the ones NOT starting with an underscore (`_`)\n",
    "  1. Create a variables `methods` listing all methods applying to your dictionary (hint: use the `dir` function)\n",
    "  2. Create a variable `proper_methods` set to an empty list (you will fill it next)\n",
    "  3. Create a for loop on all the items in the `methods` variable and append it to the list `proper_methods` if it does not start with an underscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {\n",
    "    \"item1\": \"label1\",\n",
    "    \"item2\": \"label2\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "methods = dir(my_dict)\n",
    "proper_methods = []\n",
    "for method in methods:\n",
    "    if not method.startswith(\"__\"):\n",
    "        proper_methods.append(method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['clear', 'copy', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']\n"
     ]
    }
   ],
   "source": [
    "print(proper_methods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Python - Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A function allows you to group *a logical unit set of commands*, which has a specific aim, into **a single entity**. This way you can reuse that function every time you want to achieve that aim. \n",
    "\n",
    "The aim can be very simple such has finding the maximum of a list (function `max`), its length (function `len`) or more complex (finding each unique element and count how many times it occurs )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade):\n",
    "    if grade < 5:\n",
    "        return('Grade too low')\n",
    "    else:\n",
    "        return('Grade high enough')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A function can have optional arguments, always indicated after the compulsory arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade, mode='normal'):\n",
    "    if grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade > 5:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To call the function, use its name, with arguments inside parentheses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "comment_grade(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In many cases, the user is interested in a return value that he will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "comment = comment_grade(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A function can also have no arguments at all. And return nothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def say_hello():\n",
    "    print('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "say_hello()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "greeting = say_hello()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(greeting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A special Python keyword: None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`None` is a case-sensitive keyword to state that a variable does not have any value (like \"null\" in other languages). `None` is not the same as `0`, `False`, or an empty string. `None` has a datatype of its own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = None\n",
    "print(type(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A function always return something. So if nothing is explicitely returned by the function, the return value is `None`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Function docstring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When you write a function, you need to document it. This will be used to automatically generate help and documentation, and also help anyone (including yourself at a later date) better understanding what it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade, mode = 'normal'):\n",
    "    ''' Provide a feedback according to the grade value\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        grade : int\n",
    "            The grade obtained by the student (out of 10)\n",
    "        mode : str\n",
    "            The feedback mode, either \"normal\" (default) or \"positive_reinforcement\"\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        comment : str\n",
    "            The grade feedback\n",
    "    '''\n",
    "    if grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade > 5:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Best practice is now to use Python \"type hints\" for the function arguments and return values. You can remove this information from the documentation since it is redundant. Type hints provide useful information when using IDEs (linting) and allows to check code logic via third party libraries (e.g. `mypy`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade: int, mode: str = 'normal') -> str :\n",
    "    ''' Provide a feedback according to the grade value\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        grade\n",
    "            The grade obtained by the student (out of 10)\n",
    "        mode\n",
    "            The feedback mode, either \"normal\" (default) or \"positive_reinforcement\"\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        comment\n",
    "            The grade feedback\n",
    "    '''\n",
    "    if grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade > 5:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "student_results = {\n",
    "               'John': 3,\n",
    "               'Mary': 9,\n",
    "               'Peter': 5\n",
    "               }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for s_name, s_grade in student_results.items():\n",
    "    s_feedback = comment_grade(s_grade, mode='positive_reinforcement')\n",
    "    print(f'Feedback for {s_name}: {s_feedback}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Proper Python code development: use an IDE (e.g. Visual Studio Code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Jupyter notebooks do not allow to easily debug, reuse and version-control your code**. As such it is much better practice to develop functions inside an Integrate Development Environment (IDE) to write the main functions there. You can then import them inside your Jupyter Notebook (and reuse them in other notebooks, in other projects, and with other people with e.g. GitHub)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Functions can be (and often are) imported from modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from random import randrange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "randrange?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "max_grade = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "random_grade = randrange(0, max_grade+1)\n",
    "print(random_grade)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "random_grade = random.randrange(0, max_grade+1)\n",
    "print(random_grade)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Remember `PATH` in the Linux lecture ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sys.path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "os.__file__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A module is a python file (ending in `.py`), usually containing python functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To import functions, simply use `import` followed by the name of your python file without the `.py` extension, e.g. `import grading` if your python file is called `grading.py`.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (using module in Jupyter notebook)\n",
    "1. Open a terminal in VS Code (`Terminal` --> `New Terminal`) and create the directory `autograder` in the `python_lecture` directory\n",
    "2. Create a new file in VS Code (`File` --> `New File`) and copy the definition of the function `comment_grade`\n",
    "3. Save it in the directory `autograder` with the name `grading.py` (`File` --> `Save As`, then search the `autograder` directory you created, clicking on `..` (parent directory) if required)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "4. Go back into the Jupyter notebook and try to import your `grading` module so that to call the function `comment_grade` within your notebook. Does it work ?\n",
    "5. Add the path to your `autograder` directory by appending it to the `sys.path` list.  \n",
    "Remember that:\n",
    "    * you can append an element `my_el` to a list `my_list` with `my_list.append(my_el)`\n",
    "    * a path is a string (i.e. a sequence of characters) and so should be put in between quotes\n",
    "6. Try again to import your `grading` module\n",
    "7. Print the help of your function (`help(fun)`, or `fun?`, or `Shift + Tab` after having written the function name)\n",
    "8. Call the function with a grade of your choice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step1` in Github repo)\n",
    "1. To do in your own environment\n",
    "```bash\n",
    "cd\n",
    "cd python_lecture\n",
    "mkdir autograder\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. and 3. To do in your own environment, cf `grading.py` in `solutions_part2/step1` on Github"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. It does not work, python cannot find the module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. `sys.path.append(\"/home/brainhacker/python_lecture/autograder\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. `import grading` should work now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Self explanatory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. `grading.comment_grade(8)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (using module in VS Code)\n",
    "1. Create a new file in VS Code (`File` --> `New File`) and copy:\n",
    "    * the definition of the dictionary `student_results`\n",
    "    * the code applying the function `comment_grade` on each item of that dictionary\n",
    "2. Make sure your code will use the function `comment_grade` of the module `grading`:\n",
    "    * Import the module `grading`\n",
    "        * choose between `import grading` or `from grading import comment_grade` and adapt the code if required\n",
    "3. Save the script in the directory `autograder` with the name `exam1.py` (`File` --> `Save As`, then search the `autograder` directory you created, clicking on `..` (parent directory) if required)\n",
    "\n",
    "Note: your file `exam1.py` will find the module `grading` because `grading.py` is in the same directory (more generally the directory in which resides your script is always on `sys.path`, contrarily to `bash` and `$PATH`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "4. In the terminal type `python` and then the path to your `exam1.py` file for python to run your code (note: this is the same as clicking on the `run` icon (green triangle) on the top right)\n",
    "    * e.g. `python /home/brainhacker/python_lecture/autograder/exam1.py`\n",
    "5. Can you see something strange in the outputs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step1` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1., 2. and 3. cf `exam1.py` in `solutions_part2/step1` in Github repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Run `python /home/brainhacker/python_lecture/autograder/exam1.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. The strange thing is that the output for Peter is `None`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example of debugging (`Run` --> `Start debugging`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Exercise (using module in VS Code)\n",
    "1. Correct the bug and check it works\n",
    "2. Save the file\n",
    "3. Import `exam1` inside the Jupyter notebook. What do you notice ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step2` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. and 2. (cf file `grading.py` in solutions_part2/step2 in Github repo)\n",
    "    1. Use the VS Code debugging feature (click next to a line number so that to set a red \"Breakpoint\") at the right place, and try to see why the feedback for Peter is None\n",
    "    2. Change `grade > 5` to `grade >= 5` in the code of `grading.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Your python file can be used in two ways:\n",
    "* Called on the command line with `python` (e.g. `python exam1.py`)\n",
    "* Imported in another file or in a Jupyter notebook to use all the functions defined inside\n",
    "\n",
    "Careful: when you import a file, all the code inside it will be executed !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Python use the keywork `__name__` to understand how your file was used:\n",
    "* If it was called from the command line, then `__name__` will be automatically set to `\"__main__\"`\n",
    "* If it was imported, then `__name__` will be automatically set to your file name (e.g. `\"exam1\"`)  \n",
    "\n",
    "To have a part of your file automatically run on the command line, use:  \n",
    "```\n",
    "if __name__ == \"__main__\":\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (using module in VS Code)\n",
    "1. Modify `exam1` so that it only executes the code when it is run from the command line\n",
    "2. Test by saving it and running it from the command line\n",
    "3. Test by importing in the Jupyter notebook and observing nothing happened\n",
    "    * Tip: use the following to re-import a module which has been modified:  \n",
    "```\n",
    "import importlib\n",
    "importlib.reload(my_module)```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step2` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Cf `exam1.py` in `solutions_part2/step2` in Github repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Run `python /home/brainhacker/python_lecture/autograder/exam1.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.\n",
    "```python\n",
    "import importlib\n",
    "importlib.reload(exam1)\n",
    "```\n",
    "Nothing should happen when running the two lines above (your latest `exam1.py` and `grading.py` should be in the same directory than the one you added to `sys.path`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exceptions and assert statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Errors are essential to be able to understand why something went wrong. You can throw yourself so called \"Exceptions\" in your code should something unexpected occurs, such as another module or a user not using your function as intended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade: int, mode: str = 'normal') -> str :\n",
    "    ''' Provide a feedback according to the grade value\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        grade\n",
    "            The grade obtained by the student (out of 10)\n",
    "        mode\n",
    "            The feedback mode, either \"normal\" (default) or \"positive_reinforcement\"\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        comment\n",
    "            The grade feedback\n",
    "    '''\n",
    "    if grade >= 0 and grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade >= 5 and grade <= 10:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')\n",
    "        else:\n",
    "            raise ValueError('The mode should be \"normal\" or \"positive_reinforcement\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "comment_grade(7, mode=\"tough\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For a list of Python Exceptions, please see [here](https://docs.python.org/3/library/exceptions.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Assert statements are particularly useful to test the internal logic of your code, to check that impossible events indeed never happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade: int, mode: str = 'normal') -> str :\n",
    "    ''' Provide a feedback according to the grade value\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        grade\n",
    "            The grade obtained by the student (out of 10)\n",
    "        mode\n",
    "            The feedback mode, either \"normal\" (default) or \"positive_reinforcement\"\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        comment\n",
    "            The grade feedback\n",
    "    '''\n",
    "    if grade >= 0 and grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade > 5 and grade <= 10:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')\n",
    "        else:\n",
    "            raise ValueError('The mode should be \"normal\" or \"positive_reinforcement\"')\n",
    "    else:\n",
    "        assert (grade < 0 or grade > 10), 'INTERNAL BUG: grade is not less than 0 or greater than 10'\n",
    "        raise ValueError('EXTERNAL ERROR: The grade entered should be between 0 and 10')\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "comment_grade(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "It is convenient to define test functions (starting with `test_`) to run a group of assert statements. The `pytest` package is helpful to automatically run this kind of test functions (cf later section)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def test_comments():\n",
    "    assert comment_grade(0, mode='normal') == 'Grade too low'\n",
    "    assert comment_grade(2, mode='normal') == 'Grade too low'\n",
    "    assert comment_grade(5, mode='normal') == 'Grade high enough'\n",
    "    assert comment_grade(5, mode='positive_reinforcement') == 'Well done, keep going!'\n",
    "    assert comment_grade(10, mode='normal') == 'Grade high enough'\n",
    "    assert comment_grade(10, mode='positive_reinforcement') == 'Well done, keep going!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "test_comments()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`pytest` can be used not only to run tests from functions, but also to check results of Numpy docstring examples. In this case, use the `--doctest-modules` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def comment_grade(grade: int, mode: str = 'normal') -> str :\n",
    "    ''' Provide a feedback according to the grade value\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        grade\n",
    "            The grade obtained by the student (out of 10)\n",
    "        mode\n",
    "            The feedback mode, either \"normal\" (default) or \"positive_reinforcement\"\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        comment\n",
    "            The grade feedback\n",
    "\n",
    "        Examples\n",
    "        --------\n",
    "        >>> comment_grade(6)\n",
    "        'Grade high enough'\n",
    "\n",
    "    '''\n",
    "    if grade >= 0 and grade < 5:\n",
    "        return('Grade too low')\n",
    "    elif grade >= 5 and grade <= 10:\n",
    "        if mode == 'normal':\n",
    "            return('Grade high enough')\n",
    "        elif mode == 'positive_reinforcement':\n",
    "            return('Well done, keep going!')\n",
    "        else:\n",
    "            raise ValueError('The mode should be \"normal\" or \"positive_reinforcement\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (in VS Code)\n",
    "We will test the use of `pytest` on the previous bug we discovered.\n",
    "1. Modify the `comment_grade` function in the `grading` module by reintroducing the bug we discovered\n",
    "2. Include a series of tests by copy-pasting the definition of the `test_comments` function above at the end of the `grading.py` file\n",
    "3. Save the file and run pytest on your module with `pytest <path to grading.py>`\n",
    "4. Correct the bug, save the file, and rerun pytest again\n",
    "Note: by default `pytest` run all the functions starting with `test`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step3` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. and 2. Cf `grading.py` in `solutions_part2/step3` in Github repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Run `pytest /home/brainhacker/python_lecture/autograder/grading.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Replace `grade > 5` by `grade >= 5`, then run `pytest /home/brainhacker/python_lecture/autograder/grading.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (in VS Code)\n",
    "We will test the use of `pytest` to test documentation example\n",
    "1. Modify the `comment_grade` function in the `grading` module by adding the following example in the docstring:\n",
    "```\n",
    "        Examples\n",
    "        --------\n",
    "        >>> comment_grade(6)\n",
    "        'Grade high enough!'\n",
    "```\n",
    "2. Save the file and run pytest on the `grading` module with the option indicating you want to also test examples in your docstring\n",
    "\n",
    "TIP: as indicated previously, you need to use the `--doctest-modules` option\n",
    "\n",
    "3. Did you get an error ? If yes can you modify your example and rerun `pytest` so that not to get any error ?\n",
    "\n",
    "BONUS:\n",
    "\n",
    "4. In Jupyter, use the `importlib` module to reimport your `grading` module, and then display the help of the `comment_grade` function to check that your example appears"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part2/step4` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. and 2. cf `solutions_part2/step4` in Github repo, and run:\n",
    "`pytest --doctest-modules /home/brainhacker/python_lecture/autograder/grading.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Replace `'Grade high enough!'` with `'Grade high enough'` in Examples docstring (i.e. remove the exclamation mark so that the output is exactly as expected)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Python - Part 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Object oriented programming (OOP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Python allows you to implement object oriented programming (OOP). While we can use only a set of data structures and functions to describe the problem we are trying to solve, it can be more natural to describe conceptual entities which have properties (e.g. a student has a first name, last name, average exam grade), and ways of interacting with them (e.g. updating average grade after new exam, providing full name, ...).  \n",
    "\n",
    "The properties and ways of interacting are called *data attributes* (often just ***attributes***) and *method attributes* (often just ***methods***) respectively.\n",
    "\n",
    "This approach tends to better match our mental representation of the world, and thus to help thinking more clearly about the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The new conceptual type we define is called a class. Creating an object of this class, (e.g. a student with first name `John`, surname `Doe` and average grade `8`) is also called *creating an instance* of that class, or *instantiating that class*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's see an example of class definition. Creating an object often requires providing essential attributes (e.g. name and first name for a student). In Python, this is done with the special function `__init__`, and the special keyword `self`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "### Skeleton version without type hints and numpydoc\n",
    "class Student:\n",
    "    \n",
    "    def __init__(self, first_name, last_name, avg_grade=None):\n",
    "        self.first_name = first_name\n",
    "        self.last_name = last_name\n",
    "        self.avg_grade = avg_grade\n",
    "        \n",
    "    def get_fullname(self):\n",
    "        return self.first_name + ' ' + self.last_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "### After adding type hints\n",
    "class Student:\n",
    "    \n",
    "    def __init__(self, first_name: str, last_name: str, avg_grade: float = None) -> None:\n",
    "        self.first_name = first_name\n",
    "        self.last_name = last_name\n",
    "        self.avg_grade = avg_grade\n",
    "        \n",
    "    def get_fullname(self) -> str:\n",
    "        return self.first_name + ' ' + self.last_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "### After adding type hints and numpydoc\n",
    "class Student:\n",
    "    \"\"\"\n",
    "    Named student with yearly grade average.\n",
    "\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "    first_name : str\n",
    "        Student first name\n",
    "    last_name : str\n",
    "        Student last name\n",
    "    avg_grade : float\n",
    "        Average grade (default None)    \n",
    "\n",
    "    Methods\n",
    "    -------\n",
    "    get_fullname()\n",
    "        Get the full student name\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, first_name: str, last_name: str, avg_grade: float = None) -> None:\n",
    "        self.first_name = first_name\n",
    "        self.last_name = last_name\n",
    "        self.avg_grade = avg_grade\n",
    "        \n",
    "    def get_fullname(self) -> str:\n",
    "        return self.first_name + ' ' + self.last_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "student1 = Student('Jane', 'Smith')\n",
    "student2 = Student('John', 'Doe', 8.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(f'Student 1 full name is {student1.get_fullname()}, and is average grade {student1.avg_grade}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(f'Student 2 full name is {student2.get_fullname()}, and is average grade {student2.avg_grade}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "type(student1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_str = 'This is a string'\n",
    "type(my_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_int = 2\n",
    "type(my_int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "dir(student1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "dir(my_int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How to use specific groups of packages (at given versions) for each project ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np.sqrt(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"conda.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To create an environment and activate it:\n",
    "\n",
    "* `conda create --name datascience38 python=3.8`\n",
    "* `conda activate datascience38`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To install packages in an environment (once activated):\n",
    "\n",
    "* `conda install numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can use conda to switch between environments:\n",
    "\n",
    "* `conda activate proj27`\n",
    "* `conda deactivate`\n",
    "* `conda activate proj38`\n",
    "* etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To list packages in current environment\n",
    "\n",
    "* `conda list`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To remove a package, e.g. `numpy`\n",
    "\n",
    "* `conda remove numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To list environments:\n",
    "\n",
    "* `conda info --envs`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To remove an environment:\n",
    "\n",
    "* `conda remove -n datascience38 --all`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Demonstration when calling python interpreter from command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (using terminal in VS Code)\n",
    "We will prepare a conda environment for our next project on the International Space Station (ISS). We will use `python 3.8` in the environment, so we will call it `iss38`.\n",
    "\n",
    "1. Make sure you have a terminal or start one (`Terminal` --> `New Terminal`)\n",
    "2. Create a new environment called `iss38` using `python 3.8`\n",
    "3. Activate that environment\n",
    "4. List existing packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "5. Install the packages `requests`, `reverse-geocode`, `pylint` and `pytest`\n",
    "6. List again existing packages to check they were properly installed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. and 2. `conda create -n iss38 python=3.8`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. `conda activate iss38`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. `conda list`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. `conda install requests reverse-geocode pylint pytest`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. `conda list`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ISS and astronauts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Jupyter notebook CELL 1 for exercise\n",
    "import requests\n",
    "from typing import List\n",
    "# Get the names of astronauts on board the ISS \n",
    "astros_json = requests.get('http://api.open-notify.org/astros.json').json()\n",
    "astro_names = []\n",
    "for astro_info in astros_json['people']:\n",
    "    astro_names.append(astro_info['name'])\n",
    "# Note: equivalent to astro_names = [astro_info['name'] for astro_info in astros_json['people']]\n",
    "print(astro_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Jupyter notebook CELL 2 for exercise\n",
    "import time\n",
    "from datetime import datetime\n",
    "import reverse_geocode\n",
    "\n",
    "n_trials = 3\n",
    "trial_pause_seconds = 5.0\n",
    "for i_trial in range(n_trials):\n",
    "    # Get date and time\n",
    "    date_time_now = datetime.now()\n",
    "    time_str = date_time_now.strftime(\"%d-%b-%Y %H:%M:%S\")\n",
    "    # ISS position\n",
    "    issnow_json = requests.get('http://api.open-notify.org/iss-now.json').json()\n",
    "    if issnow_json['message']  == 'success':\n",
    "        iss_latlong = (issnow_json['iss_position']['latitude'], \n",
    "                       issnow_json['iss_position']['longitude'])\n",
    "        # The search method expects a list and returns a list.\n",
    "        # Hence we need to coerce a single value latlong to a list with [latlong]\n",
    "        # Afterwards we need to extract the result, itself a single-item list, with result_list[0]\n",
    "        iss_pos_list = reverse_geocode.search([iss_latlong])\n",
    "        iss_pos = iss_pos_list[0]\n",
    "        iss_info = f'At {time_str}, the ISS was above {iss_pos[\"city\"]} in {iss_pos[\"country\"]}'\n",
    "        print(iss_info)\n",
    "        time.sleep(trial_pause_seconds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (part 1)\n",
    "Convert the code above into separate functions\n",
    "\n",
    "1. Using the terminal in VS Code create a new directory in `python_lecture` called `space` \n",
    "2. Create a new file in VS Code (File --> New File) and save it in the directory `space` with the name `iss.py`\n",
    "3. Change your current VS Code environment to `iss38`\n",
    "    * Click on the current Python environmnent in the bottom left, then choose the `iss38` environment in the menu appearing on the top"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "4. Use the code in the two Jupyter notebook cells above to create two functions in the same `iss.py` module (save that file early to benefit from VS Code syntax highligting and other features)\n",
    "    1. Copy the import statements found in either cell at the top of your `iss.py` file (all modules imported are by convention at the top of a python file)\n",
    "    2. Create two functions, each corresponding to a cell\n",
    "        1. A function `get_astro_names` corresponding to the first cell: the function has no arguments and **returns** the `astro_names` list\n",
    "        *Hint*: we are interested in a returned object !\n",
    "        2. A function `get_iss_positions` corresponding to the second cell: the function has the number of trials and the trial pause as arguments, and **returns** a list of the `iss_info` strings (generated for all trials).  \n",
    "        *Hint*: the function parameters should not be assigned again in the function body\n",
    "        *Hint*: for the returned list, you need to create an empty list, and then append the `iss_info` string to that list at each iteration\n",
    "**EXTRA BONUS if you have time to create a Numpydoc docstring with correct hint types**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "5. Raise a `requests.exceptions.RequestException` if the API message is not a success\n",
    "6. Create two test function with one assert statement each:\n",
    "    1. One function checking that the `get_astro_names` function called always return a list of length between 2 and 13\n",
    "    *Hint 1*: assign to a variable the object returned by `get_astro_names` and write assertions on that variable\n",
    "    *Hint 2*: the length of a list `my_list` is obtained with `len(my_list)`\n",
    "    *Hint 3*: comparison operator for \"greater than\" and \"less than\" are `>=` and `<=` respectively\n",
    "    2. One function checking that the `get_iss_positions`function called with three trials lasting 0.5 seconds returns a list of length 3.\n",
    "    *Hint*: assign to a variable the object returned by `get_iss_positions` called with the appropriate parameters, and write assertions on that variable\n",
    "8. Run pytest (`pytest iss.py`) and check that all tests run correctly\n",
    "    1. Make sure you have a terminal or start one (`Terminal` --> `New Terminal`)\n",
    "    2. Make sure you are in the `space` directory (use `ls` and `cd` to navigate the filesystem)\n",
    "    3. Make sure you are in the `iss38` conda environment (otherwise run `conda activate iss38`)\n",
    "    3. Type `pytest iss.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise (part 2)\n",
    "1. Go inside the Jupyter notebook and add the path to your `space` directory to `sys.path` (make sure you run `import sys` first inside your Jupyter notebook)\n",
    "2. Import your `iss.py` module inside the Jupyter notebook (the module name is your filename without the .py extension, i.e. `iss`)\n",
    "3. Call the first function to print the list of astronauts\n",
    "4. Call the second function and assign the output to a list variable. Print the list from this variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "5. Go back to VS Code, and add two lines of code to `iss.py` so that it calls one example of each of your functions\n",
    "6. Test that:\n",
    "    1. You can run the file on the terminal and it will print the desired output\n",
    "    2. Importing the same file as a module in Jupyter Notebook will unfortunately also print the ouput  \n",
    "\n",
    "    Tip: use the following to re-import a module which has been modified (replace `my_module` with the name of your module, `iss` for example!)  \n",
    "```\n",
    "import importlib\n",
    "importlib.reload(my_module)```  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "7. Go back to VS Code and modify your code so that it the examples are called only if the python file is run on the command line (hint: use `if __name__ == '__main__'`)\n",
    "8. Test that:\n",
    "    1. You can run the file on the terminal and it will print the desired output\n",
    "    2. Importing the same file as a module in Jupyter Notebook will not print anything  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution (cf `solutions_part3` in Github repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cf file `iss.py` in `solutions_part3/space` in GitHub repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Numerical analysis and plotting with numpy and matplotlib "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Numpy is an essential package for scientific computing in python. At its core are n-dimensional arrays, called  ndarrays. \n",
    "\n",
    "ndarrays can be considered as \"homogeneous\" (same-type elements) nested lists having fixed size in a given dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_1d_list = [2, 3, 5]\n",
    "a_1d = np.array(a_1d_list)\n",
    "a_1d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "b_1d = np.array([4, 6, 7])\n",
    "b_1d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Operations are very similar to Matlab (for those familier with it), but **careful with indices and slicing**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = np.linspace(0, 2*np.pi, 100)\n",
    "y = np.sin(x)\n",
    "y[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similar functions to Matlab but different \"defaults\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d = np.array([[2, 3, 5], \n",
    "                 [8, 12, 16], \n",
    "                 [18, 22, 26],\n",
    "                 [27, 31, 40]] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(a_2d.shape)\n",
    "print(np.median(a_2d))\n",
    "print(np.median(a_2d, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can make up arrays quickly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "zeros_3by2 = np.zeros((3, 2))        \n",
    "ones_4by3 = np.ones((4, 3))\n",
    "eight_3by2 = np.full((3, 2), 8)  \n",
    "diag_of_5 = np.eye(5)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can index them the usual way as for lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d.dot(eight_3by2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "print(a_2d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(a_2d[1:4, :2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Careful of changes of dimensions. Use slicing to keep dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d_row2 = a_2d[1, :2] \n",
    "print(f'Result of shape {a_2d_row2.shape}: {a_2d_row2}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d_submat = a_2d[1:2, :2]\n",
    "print(f'Result of shape {a_2d_submat.shape}: {a_2d_submat}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can create boolean arrays for masking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ix_gt5_lt20 = np.logical_and(a_2d>5, a_2d<20)\n",
    "a_2d[ix_gt5_lt20] = 0\n",
    "a_2d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Same as before in one line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d[np.logical_and(a_2d>5, a_2d<20)] = 0\n",
    "a_2d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And do typical operations. But careful of difference with Matlab: in Python `*` is element-wise multiplication. You need to use `dot` method for matrix dot product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d * ones_4by3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d.dot(eight_3by2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Use `T` method for transpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_2d.dot(ones_4by3.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `matplotlib` is a core Python plotting library, working particularly well with `numpy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Plot points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a_1d = np.array([2, 3, 5])\n",
    "b_1d = np.array([4, 6, 7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Line plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.plot(a_1d, b_1d); # Use a semi-colon at the end to avoid extra information from matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Scatter plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.scatter(a_1d, b_1d)\n",
    "plt.xlim((0, 6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Bar plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.bar(a_1d, b_1d)\n",
    "plt.xlim((0, 6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Overlays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.plot(a_1d, b_1d)\n",
    "plt.scatter(a_1d, b_1d, c='r')\n",
    "plt.bar(a_1d, b_1d, color='g')\n",
    "plt.xlim((0, 6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "random.seed(13)\n",
    "rand6_1000 = random.choices(range(6), weights=[0.1, 0.35, 0.05, 0.1, 0.05, 0.35], k=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.hist(rand6_1000, bins=np.arange(-0.5, 6)) # [-0.5, 0.5], [0.5, 1.5], ..., [4.5, 5.5] \n",
    "plt.xlim((-0.5, 6));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "rand_gauss = np.random.normal(3, 0.7, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,2))\n",
    "plt.hist(rand_gauss)\n",
    "plt.xlim((0, 6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Plot functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = np.linspace(0, 2*np.pi, 100)\n",
    "y_sin = np.sin(x)\n",
    "plt.figure(figsize=(6,2))\n",
    "plt.plot(x, y_sin);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "... on the same figure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y_cos = np.cos(x)\n",
    "# Plot the points using matplotlib\n",
    "plt.figure(figsize=(6,2))\n",
    "plt.plot(x, y_sin)\n",
    "plt.plot(x, y_cos)\n",
    "plt.xlabel('My x axis label')\n",
    "plt.ylabel('My y axis label')\n",
    "plt.title('sin and cos from 0 to $2\\pi$')\n",
    "plt.legend(['Sine', 'Cosine']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Subplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's plot a total of 2 figures: 1 column and 2 rows\n",
    "plt.figure(figsize=(8, 4))\n",
    "n_plots = 2\n",
    "n_cols = 1\n",
    "# Plot first figure\n",
    "plt.subplot(n_plots, n_cols, 1)\n",
    "plt.plot(x, y_cos)\n",
    "plt.ylabel('Cos values')\n",
    "plt.title('sin and cos from 0 to $2\\pi$')\n",
    "# Plot second figure\n",
    "plt.subplot(n_plots, n_cols, 2)\n",
    "plt.plot(x, y_sin)\n",
    "plt.xlabel('0 to 2 $\\pi$ values')\n",
    "plt.ylabel('Sin values')\n",
    "plt.tight_layout() # Magic command to optimize the overall plot display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### 2D plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(a_2d)\n",
    "# Making sure x and y ticks are integers\n",
    "plt.xticks(np.arange(0, a_2d.shape[1]))\n",
    "plt.yticks(np.arange(0, a_2d.shape[0]))\n",
    "# Adding a colorbar\n",
    "plt.colorbar()\n",
    "plt.title('visualizing a_2d');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Python - Part 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Example of manipulating data with numpy and matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Datasets from [Software Carpentry workshop](https://swcarpentry.github.io/python-novice-inflammation/) of patients who are given a daily treatment for arthritis and their inflammation is recorded over a period of 40 days. Each file includes data for a group of patients, with one patient per line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Warning** This dataset is different from the one we studied during the linux lecture. It is not split by institutes (`AL`, `MN`, `TX`), and it has the same number of patients in each file (i.e. all files have the same number of lines) except for the small sample files having `small` in their names.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"inflammation_data.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Let's have a look at the data in a terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The data is at `/home/brainhacker/python_lecture/inflammation_data`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can use `cd`, `ls`, `cat`, `more`, `head`, `tail`, `wc`, etc. to have an idea about the data. To list files we can also use the fact that bash interprets wildcard characters (e.g. `*`, `?`) in a string to expand all matching file names: e.g. `ls *.csv`. This is called globbing or filename expansion (cf linux lecture). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading a file in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "home_data_path = os.path.expanduser('~') \n",
    "data_path = os.path.join(home_data_path, 'python_lecture', 'inflammation_data')\n",
    "first_file = os.path.join(data_path, 'inflammation-01.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The un-pythonic way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_handler = open(first_file, 'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_content = file_handler.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_handler.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "print(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The pythonic way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open(first_file, 'r') as file_handler:\n",
    "    file_content = file_handler.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Looping on a file handler means looping over the lines in that file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open(first_file, 'r') as file_handler:\n",
    "    for file_line in file_handler:\n",
    "        print(f\"Line: {file_line}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Most text files you'll analyze are structured with a special character to separate field values:\n",
    "* a comma: Comma Separated Value (CSV) file\n",
    "* a tab: Tab Separated Value (TSV) file\n",
    "* a space\n",
    "* other"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Dedicated libraries can be used to extract data from these files. `pandas` is a common package to use for this, with its `read_csv` function. However we are going to look first at the special case of purely numerical data, using the `numpy` package and its `loadtxt` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np.loadtxt?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data = np.loadtxt(fname=first_file, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading several files to examine data across dataset\n",
    "Daily inflammation measure in patients who have been given a new treatment for arthritis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Get list of data files\n",
    "data_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "path_with_wildcards = os.path.join(data_path, '*.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "patient_files_all = sorted(glob.glob(path_with_wildcards))\n",
    "patient_files_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Let's exclude files with `small` in their names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Not most pythonic way but ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "patient_files = []\n",
    "for f in patient_files_all:\n",
    "    if not 'small' in f:\n",
    "        patient_files.append(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "patient_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "More pythonic: list comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "patient_files = [f for f in patient_files_all if not 'small' in f]\n",
    "patient_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Let's plot data of 1 and then N subjects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Plotting for one subject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data = np.loadtxt(fname=patient_files[0], delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "first_file_data[0, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 4))\n",
    "plt.plot(first_file_data[0, :]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Plotting for N subjects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n_subjects = 3\n",
    "plt.figure(figsize=(12, 4))\n",
    "for i_subject in range(n_subjects):\n",
    "    plt.plot(first_file_data[i_subject, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Even better with a legend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "n_subjects = 3\n",
    "plt.figure(figsize=(12, 4))\n",
    "for i_subject in range(n_subjects):\n",
    "    plt.plot(first_file_data[i_subject, :], label=f'subject {i_subject}')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Get an overview of the whole sample using `imshow` for 2D data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,6))\n",
    "plt.imshow(first_file_data)\n",
    "plt.colorbar();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<figure class=\"half\" style=\"display:flex\">\n",
    "    <img style=\"width:40%\" src=\"inflammation_data.png\">\n",
    "    <img style=\"width:40%\" src=\"axes.jpg\">\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can use functions of the `numpy` package to describe basic aspect of our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,3))\n",
    "plt.plot(np.median(first_file_data, axis=1), label = 'median')\n",
    "plt.title('Median of inflammation score across ... for each ...');\n",
    "plt.xlabel('...');\n",
    "plt.ylabel('...');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "1. Substitute the `...` above with the right words between `subject(s)` and `day(s)`\n",
    "2. Create a new figure, and plot on that figure the minimum, median and maximum inflammation score across patients for each day (i.e. day is on the x-axis)  \n",
    "Hint: use `np.min` and `np.max`\n",
    "3. Add an appropriate label for each plot, and display the legend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,3))\n",
    "plt.plot(np.median(first_file_data, axis=1), label = 'median')\n",
    "plt.title('Median of inflammation score across days for each patient');\n",
    "plt.xlabel('Patient');\n",
    "plt.ylabel('Median inflammation score');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. and 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,3))\n",
    "plt.plot(np.median(first_file_data, axis=0), label = 'min')\n",
    "plt.plot(np.median(first_file_data, axis=0), label = 'median')\n",
    "plt.plot(np.median(first_file_data, axis=0), label = 'max')\n",
    "plt.title('Median of inflammation score across patients for each day');\n",
    "plt.xlabel('Day');\n",
    "plt.ylabel('Median inflammation score');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For a 2D array, the boxplot function will produce a boxplot of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4))\n",
    "plt.boxplot(first_file_data);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Read all files in one big array, but check they all have the same size before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ref_shape = first_file_data.shape\n",
    "for p_file in patient_files:\n",
    "    if np.loadtxt(fname=p_file, delimiter=',').shape != ref_shape:\n",
    "        print(f\"File {p_file} has a shape different from the reference shape {ref_shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ref_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Preallocate zero array for all data\n",
    "sample_size, n_days = ref_shape\n",
    "n_patients_total = sample_size * len(patient_files)\n",
    "all_data = np.zeros((n_patients_total, n_days))\n",
    "# Fill array file by file\n",
    "for i_file, p_file in enumerate(patient_files):\n",
    "    start_ix = i_file * sample_size\n",
    "    stop_ix = (i_file+1) * sample_size\n",
    "    all_data[start_ix:stop_ix, :] = np.loadtxt(fname=p_file, delimiter=',')\n",
    "print(all_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 68))\n",
    "plt.imshow(all_data);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Get indices of each sample starting point\n",
    "sample_ixs = np.arange(0, all_data.shape[0], sample_size)\n",
    "sample_ixs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Plot the whole data with markers at sample starting points\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.plot(np.mean(all_data, axis=1)) # mean across days\n",
    "for sample_ix in sample_ixs:\n",
    "    plt.axvline(sample_ix, color='r', alpha=0.2)\n",
    "plt.xlabel('Subject')\n",
    "plt.ylabel('Mean inflammation score across days')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Indices matching a condition can be found with `np.where`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "healthy_ixs = np.where(all_data.sum(axis=1)==0)\n",
    "healthy_ixs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data analysis with pandas (and seaborn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `pandas` package is useful to deal with dataframes. A dataFrame is like a spreadsheet: it has rows for each observation, and labeled columns of potentially different types (`str`, `int`, ...) for the different properties measured. \n",
    "\n",
    "It is particularly convenient for reading csv or spreadsheet-like data, with `read_csv`, and for describing the resulting data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's look at a NASA meteorite dataset:\n",
    "* name: the name of the meteorite (typically a location, often modified with a number, year, composition, etc)\n",
    "* id: a unique identifier for the meteorite\n",
    "* nametype: one of: \n",
    "    * valid: a typical meteorite \n",
    "    * relict: a meteorite that has been highly degraded by weather on Earth\n",
    "* recclass: the class of the meteorite; one of a large number of classes based on physical, chemical, and other characteristics (see the Wikipedia article on meteorite classification for a primer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* mass: the mass of the meteorite, in grams\n",
    "* fall: whether the meteorite was seen falling, or was discovered after its impact; one of: \n",
    "    * Fell: the meteorite's fall was observed \n",
    "    * Found: the meteorite's fall was not observed\n",
    "* year: the year the meteorite fell, or the year it was found (depending on the value of fell)\n",
    "* reclat: the latitude of the meteorite's landing\n",
    "* reclong: the longitude of the meteorite's landing\n",
    "* GeoLocation: a parentheses-enclose, comma-separated tuple that combines reclat and reclong"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pd.read_csv?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_path = os.path.join(home_data_path, 'python_lecture', 'meteorite_data')\n",
    "met_file = os.path.join(met_data_path, 'meteorite-landings.csv')\n",
    "met_data_withna = pd.read_csv(met_file)\n",
    "met_data_withna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The row labels are automatically generated and initially correspond to the number of lines, but that can change as you manipulate the dataframe ! This is call the dataframe index and should not be used to estimate the total number of rows for example !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.head(5) # try with tail to show last 25 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pandas dataframes offer many convenient methods. When can leverage the fact that a method returns a dataframe to chain method calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.isna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Selecting data within a dataframe "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Select columns by indicating the list of required columns names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna[['name', 'mass']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*Warning*: a single column is a different object, called a `Series`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna['mass']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To keep the output a dataframe in case of a single column, use a list with a single column name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna[['mass']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To filter according to row or/and column counts, use the `.iloc` method (integer location)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.iloc[5:10, 0:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To filter according to row or/and labels, use the `.loc` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.loc[:, [\"name\", \"recclass\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna.loc[2:6, [\"name\", \"recclass\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Rows can be filtered according to a statement to be tested and which is either True or False. This is called a boolean expression and will result in a boolean mask, i.e. an array or Series of boolean values of the same length as the number of rows of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# A boolean expression will provide boolean values (True, False) for each row  \n",
    "met_data_withna['year']>1500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna[met_data_withna['year']>1500] # can use .loc too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna[met_data_withna['year']>1500].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Boolean masks can be combined with `np.logical_and` and `np.logical_or`, or more simply `&` and `|` but don't forget parentheses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "valid_year_mask = np.logical_and(met_data_withna['year']>1500, met_data_withna['year']<=2016)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "valid_year_mask = (met_data_withna['year']>1500) & (met_data_withna['year']<=2016)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's count the number of observartion with a valid year (`False` count as `0`, and `True` as `1`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "valid_year_mask.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Exercise\n",
    "* Create a boolean mask Series called `valid_long_mask` to select only longitudinal values (`reclong`) which make sense (i.e. between -180 and 180)\n",
    "* Apply this mask to the main dataframe (`met_data_withna`)\n",
    "* Create a new dataframe called `met_data_clean` which include only observation with valid years and valid longitudinal data\n",
    "* Find the total number of valid observations (note: you can use sum on a Series, but it would not make sense to use it on our whole dataframe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_long_mask = (met_data_withna['reclong']>=-180) & (met_data_withna['reclong']<=180)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_withna[valid_long_mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data_mask = valid_year_mask & valid_long_mask\n",
    "#alternative: clean_data_mask = np.logical_and(valid_year_mask, valid_long_mask)\n",
    "met_data_clean = met_data_withna[clean_data_mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data_mask.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You may often have to face NaN (not a number) values. You can identify, count and remove them with different `pandas` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data_clean.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data = met_data_clean.dropna()\n",
    "met_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dataframes have convenient methods to generate plot of their data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data.hist(figsize=(12,6));\n",
    "plt.tight_layout() # magical function to avoid overlap of subplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data.boxplot(figsize=(12,3));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### More control and more visually enhanced plots are provided with the `seaborn` package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set_style(\"darkgrid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for metric in ['mass', 'year', 'reclat', 'reclong']:\n",
    "    plt.figure(figsize=(6,1))\n",
    "    sns.boxplot(x=met_data[metric])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Examine distribution across years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,4))\n",
    "sns.histplot(met_data['year'], binwidth=10, stat=\"density\", kde=True, kde_kws={\"bw_adjust\":2});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Examine distribution according to latitude and longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,4))\n",
    "sns.kdeplot(x=met_data['reclong'], y=met_data['reclat'], shade=True, thresh=0.4)\n",
    "plt.xlim(-180,180)\n",
    "plt.ylim(-90,90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "os.environ['PROJ_LIB'] = '' # To avoid Basemap bug\n",
    "from mpl_toolkits.basemap import Basemap\n",
    "plt.figure(figsize=(12,6))\n",
    "met_map = Basemap(projection='cyl')\n",
    "met_map.drawmapboundary(fill_color='w')\n",
    "met_map.drawcoastlines(linewidth=0.5)\n",
    "met_map.drawmeridians(range(0, 360, 20),linewidth=0.1)\n",
    "met_map.drawparallels([-66.56083,-23.5,0.0,23.5,66.56083], linewidth=0.6)\n",
    "met_map.scatter(met_data['reclong'], met_data['reclat'], marker='.',alpha=0.25,c='green',edgecolor='None')\n",
    "plt.title('Map of all valid impacts', fontsize=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "sns.scatterplot(x='year', y='reclat', data=met_data, alpha=0.4)\n",
    "plt.xlim(1800,2010)\n",
    "plt.ylim(-90,90)\n",
    "plt.ylabel('Latitude')\n",
    "plt.xlabel('Year')\n",
    "plt.title('Meteorite recorded latitude vs year');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Count categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_data['nametype'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,4))\n",
    "sns.countplot(y=\"nametype\", data=met_data);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data['fall'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,4))\n",
    "sns.countplot(y=\"fall\", data=met_data);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "met_data['recclass'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "sns.set_palette('deep')\n",
    "plt.figure(figsize=(8,4))\n",
    "count_met_cat = met_data['recclass'].value_counts()\n",
    "count_met_cat.nlargest(20).plot(kind='bar');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The dataframe returned by value_counts() has the value names as index. Let's use that.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "top_met_cat = list(count_met_cat.nlargest(8).index)\n",
    "top_met_cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's create a boolean mask to select only rows corresponding to these meteorite category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "met_topcat_mask = met_data['recclass'].isin(top_met_cat)\n",
    "met_data_topcat = met_data[met_topcat_mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "sns.violinplot(x=\"mass\", y=\"recclass\", kind=\"box\", data=met_data_topcat)\n",
    "plt.xlim((-1000, 50000))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:shared-ds38]",
   "language": "python",
   "name": "conda-env-shared-ds38-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}