numpy_exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# NumPy Exercises\n",
    "----\n",
    "\n",
    "Several of these exercise where taken from https://github.com/rougier/numpy-100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import\n",
    "\n",
    "Import numpy once at the beginning; also `matplotlib` as there are some exercises with plotting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# static graphs\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Creating Arrays\n",
    "\n",
    "A number of quick array creation exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a null vector (all 0s) of size 10, except make the 6th value equal to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a vector (single dimension array) with values ranging from 5 to 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Divide each element in the array you created above by 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an array of size 10 with random normal variables; hint: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.normal.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Get Information about an Array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function that accepts as an argument a numpy array - which is provided - and prints the array dimentions, the shape and the size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myarray = np.array([[0.1,0.2,0.3],[0.,0.3,0.05],[0.,0.4,0.01]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then reshape the array to a 2D: 1x9 and 1D of 9 elements. Assign two different variables (a and b) to hold the new reshaped arrays, print the results with your function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Linear Algebra:* Calculate the determinant of `myarray` and print the value.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Matrices\n",
    "\n",
    "Create a numpy array of 100 elements with values from 0.0 to 0.2 and reshape to a square 10x10 matrix. Hint: use the `arange` function\n",
    "\n",
    "*Linear Algebra*: Create an identity matrix of size 10x10 and verify that the matrix multiplication with the previously defined matrix results in the same matrix. Use both the `dot` and `matmul` functions for numpy arrays to check to make sure multiplying by the identify matrix gave you back the same original matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Find min and max\n",
    "\n",
    "Create a 10x10 array with random values and find the minimum and maximum values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Filling Matrices\n",
    "\n",
    "Create a 5x5 matrix with values 1,2,3,4 just below the diagonal.  Hint: look for a parameter to set with the `diag` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Adding a Border\n",
    "\n",
    "Create a 8x8 2D array of 1s, then add a border of 0s around it on all sides.  Hunt: look for a function that will help you do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Changing Array Values\n",
    "\n",
    "Create a 1D array with values 0 to 10.  Negate (make negative) all elements which are between 3 and 8, in place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Dates\n",
    "\n",
    "NumPy can handle dates.  Get the date for today, yesterday, and tomorrow.  Hint: https://docs.scipy.org/doc/numpy/reference/arrays.datetime.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Normalizing Values\n",
    "\n",
    "Create a 10x10 array of random values.  Normalize the values by row by subtracting the row mean and dividing by the row standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: OLS Regression\n",
    "\n",
    "*You may be able to do parts of this without knowing linear algebra, but this is fundamentally about linear algebra operations.*\n",
    "\n",
    "Calculate a ordinary least squares regression with linear algebra, then use NumPy's function to check.\n",
    "\n",
    "Create the data:\n",
    "\n",
    "1) Create a 1D array, `y` of 100 random values.\n",
    "\n",
    "2) Create a 2D array, `X`, that is 100x3, initially filled with 1s.\n",
    "\n",
    "3) Set the 2nd column of `X` to be `2 * y + 0.1 * random noise`\n",
    "\n",
    "4) Set the 3rd column of `X` to be  `-3 * y + 0.2 * random noise`\n",
    "\n",
    "Compute the regression coefficients ($\\beta$).  Remember: $\\hat\\beta = (X^T X)^{-1} X^T y$\n",
    "\n",
    "Compare the $\\beta$ you computed to the output of `numpy.linalg.lstsq`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenging Exercise: 2D Random Walk\n",
    "\n",
    "Create a random walk in 2D with equal probability with steps of 1 and -1 along each direction.  The objective is to determine at what point the distance traveled exceeds a threshold  (\"first crossing distance\").\n",
    "\n",
    "Suggested steps: \n",
    "* Define number of steps of 1000\n",
    "* Create an integer random array of draws between 0 and 1 of shape 100x100 and name it `draws`\n",
    "* Use the `where` function to set the values of the array to 1 if the original array value is > 0 otherwise -1 : Name that new array: `steps`\n",
    "* Set the (0,0) element of `draws` to 0 \n",
    "* Calculate the cumulative sum of steps along the 0-axis (across rows) and the cumulative sum along the 1-axis (across columns). Each operation will result to an array of the same shape as steps\n",
    "* Calculate the 2D array of walks by adding the squares of the two cumulative sums and then take the square root (square each matrix, add them, take the square root of the result). Name the 2D array of walks as `walk`\n",
    "* Verify that the shape of the `walk` array is the same as the `draws` array \n",
    "[This is verification is just a check because we need to create a 2D array of walk. If we forget to specify the axis in the cumulative sum, the output would be a 1D array. ]\n",
    "\n",
    "Once we have the walk array populated, we set up a critical value in the distance traveled. We then seek to identify the cells above that critical value and select the first occurance where the condition is satisfied. \n",
    "\n",
    "To do that, we find the list of indices along the rows (axis=0) where the condition: walk >= 10 (first crossing distance) is first satisfied using `argmax(axis=0)` method on the booleen array `walk >=10`. The `.argmax(axis=0)` method will return the index of the first occurence.\n",
    "\n",
    "Name the output `ii`\n",
    "\n",
    "Repeat for the list of indices along the columns (axis=1) and name output `jj`\n",
    "\n",
    "The value of the first crossing distance is equal `walk[ii[0],jj[0]]`\n",
    "\n",
    "You may contour plot the walk 2D array by importing the `matplotlib` package and use [`countourf`](https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.contourf.html) routine. Do not specify any axes, so you would be contour plotting the values of the array against the indices of the array elements.\n",
    "\n",
    "Set colourMap = plt.cm.coolwarm for the color table. When you run: \n",
    "plt.show(), the contour plot will show a 2D map of the walk array values, where blue cells have low walk value, and red high walk values. The `ii` index runs from the point of origin upwards and the `jj` index runs from the point of origin rightwards. Therefore you would be able to visually see which red dot roughly corresponds to the location of first crossing distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Working with Matrices\n",
    "\n",
    "Define two square matrices of 10x10. One with values from 0.0 to 1 (step is 0.01) and another with values from 1 to 101 (step is 1) Hint: You can use the `matrix` routine to convert the array to a matrix after you construct a `numpy` array of 100 elements with `arange` and `reshape`. \n",
    "\n",
    "a) Write a function that accepts as input an array (2D) and calculates the sum of the elements in the diagonal. Verify the results by comparing with the output from the trace attribute of a matrix. Hint: `np.trace(*square matrix*)` -> outputs the sum of all of the elements in the diagonal. Check your function on the two matrices you created.\n",
    "\n",
    "b) Write a function that accepts as an input an array (2D) calculates the average values of all the elements. You can use the sum attribute and divide by the total number of elements. Hint: `np.sum(*matrix*)` -> outputs the sum of all the elements in the matrix and `np.size(*matrix*)` -> prints the total number of elements.  Check this function on the two matrices you created.\n",
    "\n",
    "c) From the output of second function, calculate how many elements are above or below/equal the average value. You can use the `count_nonzero` attribute as: `np.count_nonzero(*matrix* > *average*)` to calculate the number of elements greater than the average value. To find the number of elements that are less or equal to the average value you can subtract from the total number of elements (100) the number of elements that have values greater than the average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Fit a Polynomial\n",
    "\n",
    "Load file [`input.dat`](https://github.com/nuitrcs/pythonworkshops/raw/master/dataanalysis/numpy/input.dat) using the `loadtxt` routine to a variable with the name `data`. It will contain a two column tabulated data of x and y. Use the `polyfit` routine to fit the data to a 3rd degree polynomial and plot the data and the fit using 200 points (hint: use `linspace`). Calculate and print the roots of the fit. \n",
    "\n",
    "Note: we'll cover `matplotlib` in another section; for now, use the example from the presentation/lecture materials as a guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Least-squares fit\n",
    "\n",
    "For this exercise please refer to the example from the presentation labeled `Least Squares with Numpy` and your work in `Exercise: OLS Regression` in this notebook.\n",
    "\n",
    "Write a python code that, for 10 x values $ 0.2 \\le x_i \\le 2\\pi $ produces y values randomly perturbed around the curve $y_{i}= 0.5cos(x_{i})+3sin(5x_{i})$ by adding to the value of the function 0.05 * a draw from a random normal distribution.\n",
    "\n",
    "We seek a fit through this random set of data. We know that the fit will  have the form of $y=c_{1}*cos(x_{i})+c_{2}*sin(5x_{i})$. The objective is to calculate the constants. To do that, we need to construct the following 10x2 array:\n",
    "\n",
    "`A = [ [cos(x[0]),sin(x[0])] `\n",
    "\n",
    "      [cos(x[1]),sin(x[1])]\n",
    "      \n",
    "      ..................\n",
    "      \n",
    "      [cos(x[10]),sin(x[10]) ] \n",
    "      \n",
    "\n",
    "Then fit the data using numpy's least squares submodule `linalg.lstsq`. The lstsq routine accepts two arguments: Array A and the 1D array with contains the perturbed values, e.g. zi. It will return 4 outputs. You can name them: c, resid, rank, sigma. For example the call to the lstsq routine may have the form:\n",
    "\n",
    "c, resid, rank, sigma = np.linalg.lstsq(A, zi)\n",
    "\n",
    "Array c is a 2 element array that contains the constants we seek, $c_{1}$ and $c_{2}$. You can then create a large number of points between x[0] and x[10] and plot the fit against those values For example:\n",
    "\n",
    "`x2 = np.arange(x[0],x[10],0.01)\n",
    "y2 = c[0]*np.cos(x2) + c[1]*np.sin(5*x2)`\n",
    "\n",
    "Note: we'll cover `matplotlib` in another section; for now, use the example from the presentation/lecture materials as a guide or skip the plotting part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Explore Normal Distribution\n",
    "\n",
    "Use the `random.randn` routine from numpy to create a standard normal distribution array of 4000 values.\n",
    "\n",
    "Use the `histogram` routine from numpy with number of bins equal to 100. \n",
    "\n",
    "The `histogram` routine will return a tuple of two arrays. The first tuple element is the histogram values (100 values) and second is the bin edges of each interval - 101 points. \n",
    "\n",
    "Calculate a numpy array of the middle points for each bin. Plot the `histogram` values against the array of middle points. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}