Day_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "emerging-andrews",
   "metadata": {},
   "source": [
    "# Machine Learning Basics with Scikit-learn: Day 4\n",
    "\n",
    "## Data pre-processing\n",
    "\n",
    "One of the main tasks in machine learning is pre-processing the data. This means transforming the raw data in an efficient format. Some common issues that raw data have are:\n",
    "* Presence of outliers\n",
    "* Missing data\n",
    "* Data with different scales\n",
    "* Data without a normal distribution\n",
    "* Redundant features\n",
    "* Noisy features\n",
    "* Data operationalization\n",
    "\n",
    "The major steps in data pre-processing are:\n",
    "1. **Data Cleaning**: Data can have many irrelevant and missing parts. To handle this part, data cleaning is done. It involves handling of missing data, noisy data etc. \n",
    "2. **Data Transformation**: Data and features' scales need to be changed in appropiate forms that respect the statistical principles. \n",
    "3. **Data Reduction**: Reducing the number of features and observations can increase the storage efficiency and reduce data storage and analysis costs. Also, prevent bias issues. \n",
    "\n",
    "The `sklearn` library provides several utility functions and transformer classes to change raw data into a representation that is more suitable for the downstream estimators.\n",
    "\n",
    "Today, we will review three main practices that are core in data-preprocessing:\n",
    "* Data standarization and normalization\n",
    "* Feature selection\n",
    "* Imputing data \n",
    "\n",
    "Let's start loading the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fifty-diabetes",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import f_regression\n",
    "from sklearn.experimental import enable_iterative_imputer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.impute import IterativeImputer\n",
    "from sklearn.impute import KNNImputer\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.datasets import fetch_california_housing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinct-victory",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "major-offense",
   "metadata": {},
   "source": [
    "For today's session, we will use the Boston houses' prices for demo purposes. You will use the California dataset for the exercises:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td style=\"text-align:center; width:50%\">\n",
    "            <h3>Boston house prices dataset</h3>\n",
    "            <img src=\"https://www.bostonmagazine.com/wp-content/uploads/sites/2/2020/03/back-bay-brownstone.jpg\" width=\"400px\">\n",
    "            <ul class=\"simple\">\n",
    "                <li><p>CRIM     per capita crime rate by town</p></li>\n",
    "                <li><p>ZN       proportion of residential land zoned for lots over 25,000 sq.ft.</p></li>\n",
    "                <li><p>INDUS    proportion of non-retail business acres per town</p></li>\n",
    "                <li><p>CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)</p></li>\n",
    "                <li><p>NOX      nitric oxides concentration (parts per 10 million)</p></li>\n",
    "                <li><p>RM       average number of rooms per dwelling</p></li>\n",
    "                <li><p>AGE      proportion of owner-occupied units built prior to 1940</p></li>\n",
    "                <li><p>DIS      weighted distances to five Boston employment centres</p></li>\n",
    "                <li><p>RAD      index of accessibility to radial highways</p></li>\n",
    "                <li><p>TAX      full-value property-tax rate per \\$10,000</p></li>\n",
    "                <li><p>PTRATIO  pupil-teacher ratio by town</p></li>\n",
    "                <li><p>B        1000(Bk - 0.63)^2 where Bk is the proportion of black people by town</p></li>\n",
    "                <li><p>LSTAT    \\% lower status of the population</p></li>\n",
    "                <li><p>MEDV     Median value of owner-occupied homes in $1000’s</p></li>\n",
    "            </ul>\n",
    "        </td>\n",
    "        <td style=\"text-align:center\">\n",
    "            <h3>California dataset </h3>\n",
    "            <img src=\"https://cdn.vox-cdn.com/thumbor/xIC4EBWBo2QPsmB__soEniUe2ZU=/0x0:8736x5824/1820x1213/filters:focal(3670x2214:5066x3610):format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/56696503/shutterstock_154454792.0.0.jpeg\" width=\"400px\">\n",
    "            <p>This dataset was derived from the 1990 U.S. census, using one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).</p>\n",
    "            <ul class=\"simple\">\n",
    "                <li>MedInc median income in block</li>\n",
    "                <li>HouseAge median house age in block</li>\n",
    "                <li>AveRooms average number of rooms</li>\n",
    "                <li>AveBedrms average number of bedrooms</li>\n",
    "                <li>Population block population</li>\n",
    "                <li>AveOccup average house occupancy</li>\n",
    "                <li>Latitude house block latitude</li>\n",
    "                <li>Longitude house block longitude</li>\n",
    "            </ul>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "packed-deployment",
   "metadata": {},
   "source": [
    "### Loading the Boston dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "solid-korean",
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_dataset = load_boston()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "likely-advertiser",
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_df = pd.DataFrame(boston_dataset.data, columns = boston_dataset.feature_names)\n",
    "boston_df['PRICE'] = boston_dataset['target']\n",
    "boston_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adaptive-howard",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "Load the California dataset and create a DataFreme. This dataset contains 20,640 samples and 9 features.\n",
    "\n",
    "Remember that the scikit-learn raw datasets come with the following objects:\n",
    "* `data`\n",
    "* `target`\n",
    "* `feature_names`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "photographic-journalist",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data from scikit-learn\n",
    "california_dataset = fetch_california_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ignored-gibraltar",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "#california_df = ....\n",
    "#california_df['MedInc'] = ...\n",
    "#california_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "signal-extraction",
   "metadata": {},
   "source": [
    "## Data descriptive analysis\n",
    "The first step is usually comprehend and analyze the data. This can be done by checking their descriptive statistics, distributions, and correlations.\n",
    "\n",
    "We run the command `describe` to see the statistical information of this dataset: count, mean, standard deviation, minimum value, quartiles, and maximum value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "patient-college",
   "metadata": {},
   "outputs": [],
   "source": [
    "round(boston_df.describe(),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "negative-diving",
   "metadata": {},
   "source": [
    "We can also see each feature's distribtuion visually by calculating its **histogram.** This plot shows how many observations with a specific value `x` are present in the dataset. In statistical analysis, data should follow a normal distribution to be appropiaterly analyzed. \n",
    "\n",
    "Let's start checking the Boston dataset distribution. We will check first how the target (i.e., PRICE) is distributed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "published-extraction",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(10,8))\n",
    "plt.hist(boston_df['PRICE'], edgecolor = 'black', bins = int(180/5))\n",
    "plt.title(\"Histogram of Boston houses' prices\")\n",
    "plt.xlabel('Price')\n",
    "plt.ylabel('Frequency')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "narrow-triangle",
   "metadata": {},
   "source": [
    "Now, let's see how the other variables are distributed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "clinical-seminar",
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_df.hist(figsize=(15,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "contained-reservoir",
   "metadata": {},
   "source": [
    "Finally, we will check the correlations among the variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beneficial-festival",
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_df.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liquid-underwear",
   "metadata": {},
   "source": [
    "### Excerise 2\n",
    "Check the descriptive statistics and distributions of the California dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "focal-orange",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the descriptives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "little-black",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the target value ('MedInc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vietnamese-pottery",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of all features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "straight-indian",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the correlations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "differential-import",
   "metadata": {},
   "source": [
    "## Standarization: Z-score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "perceived-selection",
   "metadata": {},
   "source": [
    "In general, learning algorithms benefit from standardization of the data set. If some outliers are present in the set, robust scalers or transformers are more appropriate. \n",
    "\n",
    "Standardization of datasets is a common requirement for many machine learning estimators implemented in scikit-learn. They might behave badly if the individual features do not more or less look like standard normally distributed data: **zero mean and unit variance**.\n",
    "\n",
    "In practice, we often ignore the shape of the distribution and just **transform the data to center it by removing the mean value of each feature, then scale it by dividing non-constant features by their standard deviation**.\n",
    "\n",
    "<img class=\"XqHOTb IGEbUc\" alt=\"Z = \\frac{x - \\mu}{\\sigma}\" src=\"https://www.gstatic.com/education/formulas2/355397047/en/z_score.svg\" role=\"img\" data-atf=\"0\" data-frt=\"0\">\n",
    "\n",
    "where *Z* is the standard score, *x* is the observed value, *u* is the mean of the sample, and is *s* the standard deviation of the sample. \n",
    "\n",
    "For instance, many elements used in the objective function of a learning algorithm assume that all features are centered around zero and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.\n",
    "\n",
    "The preprocessing module provides the **StandardScaler** utility class, which is a quick and easy way to perform the following operation on an array-like dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "british-brush",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the scaler\n",
    "scaler = preprocessing.StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "laden-orientation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit and transform the data\n",
    "boston_df_scaled = scaler.fit_transform(boston_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "atmospheric-decrease",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the data into a pandas Dataframe\n",
    "boston_df_scaled = pd.DataFrame(boston_df_scaled, columns = boston_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "second-wisconsin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the descriptives. We round the results to 2 decimals. \n",
    "round(boston_df_scaled.describe(),2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "interracial-constitutional",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We plot the features' distribution.\n",
    "boston_df_scaled.hist(figsize=(15,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "human-party",
   "metadata": {},
   "source": [
    "### Exercise 3:\n",
    "Use the `StandardScaler` to transform the California dataset. Then check that the means are equal to zero and standard deviation are equal to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eleven-darwin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the scaler\n",
    "# scaler = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pressed-bride",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit and transform the data\n",
    "# california_df_scaled = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "animated-interaction",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the data into a pandas Dataframe\n",
    "# california_df_scaled = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "brazilian-sampling",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the descriptives. We round the results to 2 decimals. \n",
    "# ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "little-cleveland",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We plot the features' distribution.\n",
    "# ...."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driving-replica",
   "metadata": {},
   "source": [
    "### Summary\n",
    "* Z-score normalization handles outlier well but does not produce normalized data with the exact same scale.\n",
    "* This method rescales data to have a mean equal to 0 and a standar deviation of 1. \n",
    "\n",
    "<img class=\"img-responsive img-thumbnail\" src=\"https://www.simplypsychology.org/standardizing.svg\" alt=\"Standard Normal Distribution (SND)\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alive-seafood",
   "metadata": {},
   "source": [
    "## Min-max normalization: Scaling features to a range\n",
    "An alternative standardization is scaling features to lie between a given minimum and maximum value, often between zero and one, or so that the maximum absolute value of each feature is scaled to unit size. This can be achieved using *MinMaxScaler* or *MaxAbsScaler*, respectively.\n",
    "\n",
    "The formula is:\n",
    "<img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/0222c9472478eec2857b8bcbfa4148ece4a11b84\">\n",
    "\n",
    "Here is an example to scale a toy data matrix to the [0, 1] range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "formed-husband",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the scaler\n",
    "min_max_scaler = preprocessing.MinMaxScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "muslim-success",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit and transform the data\n",
    "boston_df_scaled_minmax = min_max_scaler.fit_transform(boston_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "focal-violin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the data into a pandas Dataframe\n",
    "boston_df_scaled_minmax = pd.DataFrame(boston_df_scaled_minmax, columns = boston_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "strong-monte",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the descriptives. We round the results to 2 decimals. \n",
    "round(boston_df_scaled_minmax.describe(),2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incorrect-search",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "boston_df_scaled_minmax.hist(figsize=(15,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "endless-while",
   "metadata": {},
   "source": [
    "### Exercise 4:\n",
    "Use the `MinMaxScaler` to scale the California dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "occupational-nickel",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the scaler\n",
    "#min_max_scaler = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "assumed-maine",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit and transform the data\n",
    "#california_df_scaled_minmax = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "christian-baltimore",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the data into a pandas Dataframe\n",
    "#california_df_scaled_minmax = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "delayed-prompt",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the descriptives. We round the results to 2 decimals. \n",
    "#..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dying-banana",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# We plot the features' distribution.\n",
    "# ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "champion-complex",
   "metadata": {},
   "source": [
    "### Summary:\n",
    "* Min-max normalization method guarantees all features will have the exact same scale but does not handle outliers well. \n",
    "* This transformation rescales the dataset to another range, preserving the original scale distances and standard deviation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "remarkable-thing",
   "metadata": {},
   "source": [
    "## Feature selection\n",
    "\n",
    "Feature selection is the process of reducing the number of input variables when developing a predictive model. It is desirable to reduce the number of input variables to both reduce the computational cost of modeling, reducing the amount of data collected, and, in some cases, to improve the performance of the model.\n",
    "\n",
    "For this exercise, we will split the dataset in two parts:\n",
    "* the predictive features `X`\n",
    "* the target `y`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coupled-leather",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_boston = boston_df.iloc[:,:-1]\n",
    "y_boston = boston_df[\"PRICE\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "documented-passenger",
   "metadata": {},
   "source": [
    "We check the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "known-sample",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_boston"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mobile-permit",
   "metadata": {},
   "source": [
    "We do the same for the California datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "continuous-bosnia",
   "metadata": {},
   "outputs": [],
   "source": [
    "#X_california = california_df.iloc[:,1:]\n",
    "#y_california = california_df[\"MedInc\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "planned-shower",
   "metadata": {},
   "outputs": [],
   "source": [
    "#X_california"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adapted-feelings",
   "metadata": {},
   "source": [
    "### Removing features with low variance\n",
    "\n",
    "VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hindu-union",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the selector `VarianceThreshold`\n",
    "selector = VarianceThreshold(threshold=(.8 * (1 - .8)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "blind-employee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the selector \n",
    "selector.fit(X_boston)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "given-bishop",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create the dataframe with the features that explain at least 16% of the data variance\n",
    "boston_df_selected_variance = X_boston[X_boston.columns[selector.get_support(indices=True)]]\n",
    "boston_df_selected_variance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "persistent-magnet",
   "metadata": {},
   "source": [
    "#### Exercise 5:\n",
    "Create the selector `VarianceThreshold` with the California dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "historical-ecuador",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the selector\n",
    "# selector = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "controversial-transcription",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the selector using X_california\n",
    "# ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "infectious-isolation",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create the dataframe using the california dataframe\n",
    "#california_df_selected_variance = X_california[X_california.columns[selector.get_support(indices=True)]]\n",
    "#california_df_selected_variance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coastal-jerusalem",
   "metadata": {},
   "source": [
    "### Univariate feature selection¶\n",
    "Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method.\n",
    "\n",
    "We will check [SelectKBest](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest), which removes all but the *k* highest scoring features.\n",
    "\n",
    "How does this method select the *k* features? Depending on the type of problem (i.e., classification or regression), it runs F-tests to see what variables explain data's variance the most. These objects take as input a scoring function that returns univariate scores and p-values:\n",
    "\n",
    "* For regression: `f_regression`, `mutual_info_regression`\n",
    "* For classification: `chi2`, `f_classif`, `mutual_info_classif`\n",
    "\n",
    "The methods based on F-test estimate the degree of linear dependency between two random variables. On the other hand, mutual information methods can capture any kind of statistical dependency, but being nonparametric, they require more samples for accurate estimation.\n",
    "\n",
    "Since this dataset is usually tested with regressions, we will use the `f_regression` function to select the best *k* features in the model. This function tests the individual effect of each feature in a regression model. [More details here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "seeing-facial",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the SelectKBest object \n",
    "selector = SelectKBest(f_regression, k=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "royal-david",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "selector.fit(X_boston, y_boston)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worth-winning",
   "metadata": {},
   "source": [
    "We will see the F-scores of adding each feature to a regression model (which has `PRICE` as the target value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "discrete-teaching",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the F-scores \n",
    "selector.scores_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-attitude",
   "metadata": {},
   "source": [
    "And these are the p-values of these F-tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adolescent-spice",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the p-values\n",
    "selector.pvalues_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loaded-bloom",
   "metadata": {},
   "source": [
    "We select the features with the highest F-scores (smallest p-values). In this case, the 6th variable (RM) and the last variable (LSTAT) are the two features that explain the variance the most."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "terminal-blocking",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create the dataframe with the two best features\n",
    "boston_df_selected_kbest = X_boston[X_boston.columns[selector.get_support(indices=True)]]\n",
    "boston_df_selected_kbest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "floppy-italic",
   "metadata": {},
   "source": [
    "#### Exercise 6\n",
    "Create the selector `SelectKBest` with the California dataset (`X_california`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surgical-johns",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the selector SelectKBest with the California data\n",
    "#selector = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "juvenile-dover",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model. Remember to use `X_california` and `y_california`\n",
    "#...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "liable-specification",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create and print the selected variables\n",
    "#california_df_selected_k_best = X_california[X_california.columns[selector.get_support(indices=True)]]\n",
    "#california_df_selected_k_best"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dying-tracy",
   "metadata": {},
   "source": [
    "## Missing data\n",
    "\n",
    "For various reasons, many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. Such datasets however are incompatible with scikit-learn estimators which assume that all values in an array are numerical, and that all have and hold meaning. \n",
    "\n",
    "A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. However, this comes at the price of losing data which may be valuable (even though incomplete). A better strategy is to **impute the missing values**, i.e., to infer them from the known part of the data. \n",
    "\n",
    "For this session, we will create an example where the Boston dataset will contain missing information (`NaN`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "historic-heritage",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe with missing data. It will remove 10% of the observations\n",
    "X_boston_missing_data = X_boston.mask(np.random.random(X_boston.shape) < .1)\n",
    "X_boston_missing_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "modern-posting",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how many missing values each feature has. \n",
    "X_boston_missing_data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solar-walnut",
   "metadata": {},
   "source": [
    "We create the California dataset with missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "seasonal-shooting",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe with missing data\n",
    "#X_california_missing_data = X_california.mask(np.random.random(X_california.shape) < .1)\n",
    "#X_california_missing_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moral-edinburgh",
   "metadata": {},
   "source": [
    "### Univariate feature imputation\n",
    "One type of imputation algorithm is univariate, which imputes values in the i-th feature dimension using only non-missing values in that feature dimension.\n",
    "\n",
    "The `SimpleImputer` class provides basic strategies for imputing missing values. Missing values can be imputed with a provided constant value, or using the statistics (mean, median or most frequent) of each column in which the missing values are located. This class also allows for different missing values encodings.\n",
    "\n",
    "The strategies are:\n",
    "* “mean”: replace missing values using the mean along each column. Can only be used with numeric data.\n",
    "* “median”: replace missing values using the median along each column. Can only be used with numeric data.\n",
    "* “most_frequent”: replace missing using the most frequent value along each column. Can be used with strings or numeric data. If there is more than one such value, only the smallest is returned.\n",
    "* “constant”: replace missing values with fill_value. Can be used with strings or numeric data.\n",
    "\n",
    "The following snippet demonstrates how to replace missing values, encoded as np.nan, using the mean value of the columns (axis 0) that contain the missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aerial-bosnia",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the SimpleImputer class\n",
    "imp = SimpleImputer(missing_values=np.nan, strategy='mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "western-security",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and Transform the Missing data to Imputed data. \n",
    "X_boston_imputed = imp.fit_transform(X_boston_missing_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dutch-wholesale",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "X_boston_imputed = pd.DataFrame(X_boston_imputed, columns = X_boston.columns)\n",
    "X_boston_imputed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "computational-spokesman",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that the features do not have any missing data\n",
    "X_boston_imputed.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chronic-executive",
   "metadata": {},
   "source": [
    "#### Exercise 7: \n",
    "Create an imputer with the California dataset (`X_california`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reduced-concrete",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the SimpleImputer class\n",
    "# imp = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "primary-geography",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and Transform the Missing data to Imputed data. Remember to use `X_california_missing_data`\n",
    "# X_california_imputed = ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "according-pricing",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "#X_boston_imputed = pd.DataFrame(X_boston_imputed, columns = X_boston.columns)\n",
    "#X_boston_imputed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "musical-young",
   "metadata": {},
   "source": [
    "### Multivariate feature imputation\n",
    "Multivariate imputation algorithms use the entire set of available feature dimensions to estimate the missing values. Scikit-learn offers the `IterativeImputer` class, which models each feature with missing values as a function of other features, and uses that estimate for imputation. It does so in an iterated round-robin fashion: \n",
    "* At each step, a feature column is designated as output y and the other feature columns are treated as inputs X. A regressor is fit on (X, y) for known y. \n",
    "* Then, the regressor is used to predict the missing values of y.\n",
    "* This is done for each feature in an iterative fashion, and then is repeated for max_iter imputation rounds. The results of the final imputation round are returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "champion-response",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the IterativeImputer class\n",
    "imp = IterativeImputer(max_iter=100, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coordinated-verse",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and Transform the Missing data to Imputed data\n",
    "X_boston_imputed = imp.fit_transform(X_boston_missing_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "loving-output",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "X_boston_imputed = pd.DataFrame(X_boston_imputed, columns = X_boston.columns)\n",
    "X_boston_imputed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "latest-endorsement",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that there are no missing values\n",
    "X_boston_imputed.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "familiar-vanilla",
   "metadata": {},
   "source": [
    "#### Exercise 8: \n",
    "Create an `IterativeImputer` with the California dataset (`X_california`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "satisfied-mixer",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the IterativeImputer class\n",
    "#imp = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tropical-military",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and transform the data using the imputer. Remember to use X_california_missing_data\n",
    "#X_california_imputed = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "forbidden-communications",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "#X_california_imputed = pd.DataFrame(X_california_imputed, columns = X_california.columns)\n",
    "#X_california_imputed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "substantial-concept",
   "metadata": {},
   "source": [
    "### Nearest neighbors imputation\n",
    "\n",
    "The `KNNImputer` class provides imputation for filling in missing values using the k-Nearest Neighbors approach. Each missing feature is imputed using values from `n_neighbors` nearest neighbors that have a value for the feature. The feature of the neighbors are averaged uniformly or weighted by distance to each neighbor. If a sample has more than one feature missing, then the neighbors for that sample can be different depending on the particular feature being imputed. \n",
    "Some rules:\n",
    "* When the number of available neighbors is less than `n_neighbors` and there are no defined distances to the training set, the training set average for that feature is used during imputation. \n",
    "* If there is at least one neighbor with a defined distance, the weighted or unweighted average of the remaining neighbors will be used during imputation.\n",
    "* If a feature is always missing in training, it is removed during transform. \n",
    "\n",
    "For more information, check the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "happy-portuguese",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the KNNImputer\n",
    "imp = KNNImputer(n_neighbors=3, weights=\"uniform\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "second-latitude",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and Transform the Missing data to Imputed data\n",
    "X_boston_imputed = imp.fit_transform(X_boston_missing_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "productive-receptor",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "X_boston_imputed = pd.DataFrame(X_boston_imputed, columns = X_boston.columns)\n",
    "X_boston_imputed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sharing-nitrogen",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We check that there are no NaN values\n",
    "X_boston_imputed.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coastal-vulnerability",
   "metadata": {},
   "source": [
    "#### Exercise 9: \n",
    "Create an `KNNImputer` with the California dataset (`X_california`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "native-transcript",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the KNNImputer class\n",
    "# imp = ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reasonable-member",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and transform the data using the KNNImputer. Remember to use X_california_missing_data\n",
    "# X_california_imputed = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "joined-calculation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataframe\n",
    "# X_california_imputed = pd.DataFrame(X_california_imputed, columns = X_california.columns)\n",
    "# X_california_imputed"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}