Day_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "continuing-sperm",
   "metadata": {},
   "source": [
    "# Machine Learning Basics with Scikit-learn: Day 5\n",
    "In our last session, we will build a model to predict the chronic kidney disease. The data was taken over a 2-month period in India with 25 features ( eg, red blood cell count, white blood cell count, etc). The target is the 'classification', which is either 'ckd' or 'notckd' - ckd=chronic kidney disease. We will use machine learning techniques to predict if a patient is suffering from a chronic kidney disease or not. This dataset has misssing data, features in different scales, and 25 features. For more information click [here](https://archive.ics.uci.edu/ml/datasets/Chronic_Kidney_Disease). \n",
    "\n",
    "We will perform the following actions:\n",
    "1. Import the dataset\n",
    "2. Analyze the dataset\n",
    "3. Imput missing values\n",
    "4. Scale the data\n",
    "5. Create training and testing datasets\n",
    "6. Train a neural network with the training dataset\n",
    "7. Evaluate the model with the testing dataset\n",
    "\n",
    "Let's start loading the functions and libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "czech-headline",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import f_classif\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "from sklearn.impute import SimpleImputer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesser-temple",
   "metadata": {},
   "source": [
    "# 1. Import the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "honey-burner",
   "metadata": {},
   "source": [
    "We import the dataset. This time, we will use a CSV file which contains the required data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "comparative-sender",
   "metadata": {},
   "outputs": [],
   "source": [
    "kidney_df = pd.read_csv(\"https://raw.githubusercontent.com/dgzara/nuit_machine_learning_workshop/main/kidney_disease.csv\", index_col=0)\n",
    "kidney_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flying-popularity",
   "metadata": {},
   "source": [
    "The target column was coded using strings (`notckd` and `ckd`). We can replace them with binary values (0 and 1).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "trained-occasions",
   "metadata": {},
   "outputs": [],
   "source": [
    "kidney_df['classification'] = kidney_df['classification'].str.replace('notckd', '0')\n",
    "kidney_df['classification'] = kidney_df['classification'].str.replace('ckd', '1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "varying-spending",
   "metadata": {},
   "outputs": [],
   "source": [
    "kidney_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "welsh-medicine",
   "metadata": {},
   "source": [
    "We split the dataset into `X` and `y` variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "transparent-interaction",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = kidney_df.drop(['classification'], axis=1)\n",
    "y = kidney_df['classification'].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fifty-morning",
   "metadata": {},
   "source": [
    "## 2. Analyze the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gorgeous-sociology",
   "metadata": {},
   "source": [
    "According to the dataset's documentation, the features available are:\n",
    "1. age - age\n",
    "2. bp - blood pressure\n",
    "3. sg - specific gravity\n",
    "4. al - albumin\n",
    "5. su - sugar\n",
    "6. rbc - red blood cells\n",
    "7. pc - pus cell\n",
    "8. pcc - pus cell clumps\n",
    "9. ba - bacteria\n",
    "10. bgr - blood glucose random\n",
    "11. bu - blood urea\n",
    "12. sc - serum creatinine\n",
    "13. sod - sodium\n",
    "14. pot - potassium\n",
    "15. hemo - hemoglobin\n",
    "16. pcv - packed cell volume\n",
    "17. wc - white blood cell count\n",
    "18. rc - red blood cell count\n",
    "19. htn - hypertension\n",
    "20. dm - diabetes mellitus\n",
    "21. cad - coronary artery disease\n",
    "22. appet - appetite\n",
    "23. pe - pedal edema\n",
    "24. ane - anemia\n",
    "\n",
    "We check the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "expired-evolution",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "overall-radius",
   "metadata": {},
   "source": [
    "Are the columns numeric or categorical?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "excellent-boring",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sorted-stroke",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How much data is missing? \n",
    "X.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "excited-spelling",
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are the main descriptive statistics of this dataset?\n",
    "X.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "urban-speaker",
   "metadata": {},
   "source": [
    "The first finding is that some columns have many missing observations. For example, potassium (`pot`) has 88 missing values. \n",
    "\n",
    "How is the numeric data distributed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "rural-paraguay",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.hist(figsize=(15,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worth-bryan",
   "metadata": {},
   "source": [
    "We are going to select the numeric columns and categorical columns separately for our transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "russian-prospect",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_columns = X.select_dtypes(include=np.number).columns.tolist()\n",
    "numeric_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sustainable-summer",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_columns = list(set(X.columns) - set(numeric_columns))\n",
    "categorical_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "greater-egypt",
   "metadata": {},
   "source": [
    "## 3. Imput missing values\n",
    "\n",
    "We will use the `SimpleImputer` class, which provides basic strategies for imputing missing values. Missing values can be imputed with a provided constant value, or using the statistics (mean, median or most frequent) of each column in which the missing values are located. This class also allows for different missing values encodings.\n",
    "* For numeric attributes, we can use `mean` to replace `NaN` values.\n",
    "* For categorical attributes, we can use `most_frequent` to replace `NaN` values.\n",
    "\n",
    "Let's start imputing the missing numerical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "involved-significance",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will copy the X dataframe to X_imp and keep the original dataframe X separated\n",
    "X_imp = X.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dangerous-technique",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the SimpleImputer object\n",
    "imp = SimpleImputer(missing_values=np.nan, strategy='mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "presidential-aviation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the imputer to fit and transform the numeric data. Replace the \n",
    "X_imp[numeric_columns] = imp.fit_transform(X_imp[numeric_columns])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "auburn-jurisdiction",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_imp[numeric_columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "instrumental-dimension",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how many missing values each feature has. \n",
    "X_imp.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sensitive-olympus",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Create an `SimpleImputer` object and impute the missing categorical data. Use `most_frequent` as the strategy. Do not forget to use `X_imp` for this operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "swedish-report",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the imputer using `most_frequent` strategy\n",
    "# imp = ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "infrared-melissa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Impute the missing categorical values of X_imp\n",
    "# X_imp[categorical_columns] = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "interesting-hopkins",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the results\n",
    "# X_imp[categorical_columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "private-least",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how many missing values each feature has. All values should be zero now. \n",
    "# X_imp.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "positive-albany",
   "metadata": {},
   "source": [
    "## 4. Scale the data\n",
    "Yesterday we discussed the importance of scaling numeric data to (a) handle outliers and (b) compare the effect of increasing one unit of each feature on the dependent variable. We will standarize the numeric features present in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "useful-desktop",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We copy the imputed data to a new dataframe.\n",
    "X_scaled = X_imp.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "settled-brisbane",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Create the scaler using `preprocessing.StandardScaler()` and then replace the numeric columns of `X_scaled` with the scaler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thermal-corruption",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create the scaler using preprocessing.StandardScaler() function\n",
    "# scaler = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "assured-sodium",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use the scaler to fit and transform only the numeric features (`X_imp[numeric_columns]`)\n",
    "# X_scaled[numeric_columns] = ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acquired-oliver",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We print the descriptive statistics\n",
    "# round(X_scaled.describe(),2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "interracial-stadium",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We plot the distributions and check the scales\n",
    "# X_scaled.hist(figsize=(15,10))\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "norman-medication",
   "metadata": {},
   "source": [
    "### Handling categorical data\n",
    "These are transformers that are not intended to be used on features, only on supervised learning targets. For example, categorical data is not mathematically represented in the dataset. \n",
    "\n",
    "To handle this situation, scikit-learn provides the function `LabelEncoder`, which is a utility class to help normalize labels from categorical features. It will replace the classes with integers that represent each class. In R, this is also known as *factors.* `LabelEncoder` can be used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "liked-classic",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create the `LabelEncoder`\n",
    "le = preprocessing.LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "retained-arthritis",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a toy example. First, we will encode the classes and detect the unique classes\n",
    "le.fit([\"low\", \"medium\", \"high\", \"medium\"])\n",
    "list(le.classes_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cardiac-homeless",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Based on this Encoder, we took the vector of categorical data and transform it to numeric data\n",
    "le.transform([\"low\", \"low\", \"high\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tough-authentication",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can decode a vector of numeric values to obtain the original classes\n",
    "list(le.inverse_transform([1, 1, 0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prostate-yorkshire",
   "metadata": {},
   "source": [
    "Now, let's get back to our dataset. We need to transform the categorical features and encode them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "following-immigration",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confidential-terry",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Since we have 13 categorical features, we will create a Python loop and save the encoders in a dictionary. You will have to create inside of the loop the Encoder and transform the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fitting-remains",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dictionary of encoders\n",
    "encoders = {}\n",
    "\n",
    "# Run the loop\n",
    "for column in categorical_columns:\n",
    "    # Create the LabelEncoder\n",
    "    # le = ....\n",
    "    \n",
    "    # Train the LabelEncoder using the X[column] \n",
    "    # \n",
    "    \n",
    "    # Replace the X_scale[column] with the transformed data\n",
    "    # X_scaled[column] = ...\n",
    "    \n",
    "    # Append the Encoder to the dictionary\n",
    "    # encoders[column] = le"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "potential-projector",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the dictionary of encoders\n",
    "# encoders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "express-mouth",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# We check the columns, we will see that now all columns are numeric.\n",
    "# X_scaled.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "union-reserve",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can print the final dataframe\n",
    "# X_scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hungarian-pharmacy",
   "metadata": {},
   "source": [
    "## 5. Create training and testing datasets\n",
    "We split our dataset into two portions: training dataset and testing dataset. We will use the former to train the predictive model, and the latter to evaluate the model's accuracy. To split the data, we use `train_test_split` function provided by scikit-learn library. There are four parameters of this function\n",
    "* The imput data (X_scaled)\n",
    "* The target vector (y)\n",
    "* `test_size`: The dataset's percentage that will be used for testing. We will use `0.20`.\n",
    "* `random_state`: Random seed. We can keep it fixed so we can reproduce the random numbers across multiple calls. We will use [42](https://hitchhikers.fandom.com/wiki/42). \n",
    "\n",
    "### Excercise\n",
    "Create the training and testing datasets. Remember that this method returns 4 variables (`X_train`, `X_test`, `y_train`, `y_test`). You can check the function's [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to check how the parameters are set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "premier-tonight",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create here the training and testing datasets using train_test_split\n",
    "# X_train, X_test, y_train, y_test = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "color-malpractice",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We print the training dataset. We will have 320 observations.\n",
    "# X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "labeled-warner",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the testing dataset. We will have 80 observations\n",
    "# X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "muslim-inventory",
   "metadata": {},
   "source": [
    "## 6. Train a neural network with the training dataset\n",
    "\n",
    "For this exercise, we will create a neural network called Multi-layer perceptron (MLP). The model learns from the dataset through three types of layers:\n",
    "* *The input layer*: The input layer receives the features to be processed.\n",
    "* *The output layer*: The prediction and classification is performed by the output layer.\n",
    "* *The hidden layers*: An arbitrary number of hidden layers that are placed in between the input and output layer perform the computation. Each layer has nodes (also known as neurons) that transforms the values from the previous layer into an activation funcition. The output layer receives the activated values from the last hidden layer and transforms them into output values.\n",
    "\n",
    "<img src=\"https://static.javatpoint.com/tutorial/tensorflow/images/multi-layer-perceptron-in-tensorflow.png\" alt=\"drawing\" width=\"400\"/>\n",
    "\n",
    "Multi-layer perceptrons are networks of nodes that pass a linear combination of their inputs from one layer to another. As they do this, the nodes decide how to modify their inputs, utilizing a given activation function. The activation function of a neuron is the key here. By selecting non-linear activation functions, the neural network can embed non-linearity in its operation:\n",
    "\n",
    "<img src=\"https://www.baeldung.com/wp-content/uploads/sites/4/2020/06/Copy-of-Copy-of-Blank-Diagram1.svg\" alt=\"drawing\" width=\"400\"/>\n",
    "\n",
    "Neural networks are good for non-linear classes of problems. The first advantage of neural networks is, therefore, their flexibility in addressing problems with non-linear shapes. MLPs are implemented in diverse fields such as speech recognition, image recognition, and machine translation software. From artificial neural networks, **deep learning** has emerged as a sub-field of machine learning.\n",
    "\n",
    "Let's create a simple perceptron with 5 nodes in two layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "proud-scroll",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(hidden_layer_sizes=(5, 2), random_state=1,max_iter=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "future-buying",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Use the `fit` function to train the model. Remember that it has two parameters: `X_train` and `y_train`.\n",
    "Then, use the `predict` to predict the values using `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stock-tsunami",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "protecting-worry",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the predict function to test the model\n",
    "# y_predicted = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dirty-chile",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a dataframe with the real target and the predicted target\n",
    "# y_train_df = pd.DataFrame({'y_real': y_test, 'y_predicted': y_predicted})\n",
    "# y_train_df['difference'] = y_train_df.y_real == y_train_df.y_predicted \n",
    "# y_train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advance-glucose",
   "metadata": {},
   "source": [
    "## 7. Evaluate the model with the testing dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "understood-consciousness",
   "metadata": {},
   "source": [
    "In binary classiifcation problems, there are two main metrics:\n",
    "* Precision: What proportion of positive identifications was actually correct?\n",
    "* Recall: What proportion of actual positives was identified correctly?\n",
    "\n",
    "Precision and recall are performance metrics that apply to data retrieved from a classification process. We can see the diferences here:\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/700px-Precisionrecall.svg.png\" width=\"400px\">\n",
    "\n",
    "\n",
    "To check the mathematical formulas, check [here](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stock-drama",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We a text report showing the main classification metrics.\n",
    "# print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "junior-tribune",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can print these numbers as the confusion matrix.\n",
    "# plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "married-chester",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Create a new model using 10 layers with 10 nodes each layer. Would the performance improve? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "equipped-leave",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coordinate-training",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model, fit, and predict.\n",
    "#clf = ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "champion-engagement",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "australian-liberal",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "# plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conditional-victory",
   "metadata": {},
   "source": [
    "## What if..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interracial-circulation",
   "metadata": {},
   "source": [
    "### Using the raw data, removing the NaNs and only numerical values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hearing-tours",
   "metadata": {},
   "outputs": [],
   "source": [
    "kidney_no_na = kidney_df.dropna()\n",
    "X_no_na = kidney_no_na.drop(['classification'], axis=1)\n",
    "y_no_na = kidney_no_na['classification'].astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "considerable-transmission",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_no_na"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "proud-hormone",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_no_na[numeric_columns], y_no_na, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "virtual-breathing",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1,max_iter=1000)\n",
    "clf.fit(X_train, y_train)\n",
    "y_predicted = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "developing-testimony",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial-works",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "short-variable",
   "metadata": {},
   "source": [
    "### No categorical attributes and no rescale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "billion-manhattan",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_imp[numeric_columns], y, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "extreme-smell",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1,max_iter=1000)\n",
    "clf.fit(X_train, y_train)\n",
    "y_predicted = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "macro-institution",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thousand-expansion",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hundred-promise",
   "metadata": {},
   "source": [
    "### Scaled, but no categorical attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "anonymous-placement",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled[numeric_columns], y, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "informed-subdivision",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(hidden_layer_sizes=(5, 2), random_state=1, max_iter=1000)\n",
    "clf.fit(X_train, y_train)\n",
    "y_predicted = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "numerical-garden",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hispanic-danish",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strong-north",
   "metadata": {},
   "source": [
    "### Only categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "czech-landing",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled[categorical_columns], y, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sacred-absolute",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1,max_iter=2000)\n",
    "clf.fit(X_train, y_train)\n",
    "y_predicted = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "internal-legislation",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "centered-warning",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion_matrix(clf, X_test, y_test, cmap=plt.cm.Blues)  \n",
    "plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cultural-nursery",
   "metadata": {},
   "source": [
    "## More to learn...\n",
    "* How do we estimate the correct parameters to use?: [Hyperparameter optimization](https://scikit-learn.org/stable/modules/grid_search.html)\n",
    "* How do evaluate the model? [Metrics and scoring](https://scikit-learn.org/stable/modules/model_evaluation.html)\n",
    "* How can I train a model using different partitions of my dataset? [Corss-validation](https://scikit-learn.org/stable/modules/cross_validation.html)\n",
    "* Neural Networks: [TensorFlow](https://www.tensorflow.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "absolute-ottawa",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}