1-DataPrep/data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Workshop\n",
    "### _**Data Preparation**_\n",
    "\n",
    "---\n",
    "In this part of the workshop we will prepare the data to later train our churn model.\n",
    "\n",
    "---\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Background](#Background) - Getting the rawata prepared in the previous lab.\n",
    "2. [Prepare](#Prepare) - Prepare the data with [Amazon SageMaker Data Wrangler](https://aws.amazon.com/sagemaker/data-wrangler/)\n",
    "    * [Creating features](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-transform.html)\n",
    "    * [Creating analysis](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-analyses.html)\n",
    "    * [Analyzing the data and features](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-data-bias.html)\n",
    "3. [Submiting the data to Feature Store](#FeatureStore) - Store the features created in [Amazon SageMaker Feature Store](https://aws.amazon.com/sagemaker/feature-store/)\n",
    "  \n",
    "---\n",
    "\n",
    "## Background\n",
    "\n",
    "In the previous [Introduction lab](../0-Introduction/introduction.ipynb) we created a S3 bucket and uploaded the raw data to it.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "Get variables from previous configuration notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r bucket\n",
    "%store -r region\n",
    "%store -r prefix\n",
    "%store -r s3uri_raw\n",
    "%store -r docker_image_name\n",
    "%store -r framework_version\n",
    "bucket, prefix, s3uri_raw, region, docker_image_name, framework_version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import the libraries for this lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "#bucket = sess.default_bucket()\n",
    "#prefix = \"sagemaker/DEMO-xgboost-churn\"\n",
    "\n",
    "# Define IAM role\n",
    "import boto3\n",
    "import re\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import json\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from IPython.display import display as dis\n",
    "from time import strftime, gmtime\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.serializers import CSVSerializer\n",
    "from IPython import display "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Data on DataWrangler\n",
    "\n",
    "To start, we will create a new flow and import the raw data to perform analysis and transformations on it. On the left menu, click on \"Components and Registries\", select \"Data Wrangler\" on the dropdown, and click \"New Flow\". This process is shown on the image below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/010-new_flow.png\" width=\"30%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As soon as we click on \"New Flow\", Data Wrangler will be on a loading state. After a couple minutes you should be able to import the raw data on it. While we wait, we can rename our flow by right clicking on the flow tab and choosing \"Rename Data Wrangler Flow...\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/020-load_new_flow.png\" width=\"100%\" />\n",
    "\n",
    "Let's call the file `churn.flow`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After Data Wrangler finishes loading, we may proceed importing our data. We'll be importing our data from Amazon S3. The following images guide us through the process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/030-importing_from_s3.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the search box or the table in the bottom, locate the S3 Bucket where our dataset is stored.\n",
    "\n",
    "To remember the bucket and prefix run cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3uri_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/040-choosing_bucket.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose the \"churn.csv\" file, disable the \"Enable sampling\" checkbox, and click \"Import\". Feel free to check the data preview on the \"Preview\" session on the bottom!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/050-choose_csv_preview.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis on Data Wrangler\n",
    "\n",
    "Next, we are going to analyze our data by understanding feature distribution and how each of them impacts our target column. Data Wrangler allow us to perform such analysis inside user interface, let's start creating these analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start our analysis, we'll create a summary of our imported data. The summary can be rertieved by adding a new analysis on Data Wrangler. The following images show the step by step to create our table summary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/060-add_first_analysis.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/070-table_summary.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As soon as you click on Preview or entering in your saved analysis, you'll see the table summary as shown in the image below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/080-table_summary_preview.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By modern standards, it’s a relatively small dataset, with only 3,333 records, where each record uses 21 attributes to describe the profile of a customer of an unknown US mobile operator. The attributes are:\n",
    "\n",
    "- `State`: the US state in which the customer resides, indicated by a two-letter abbreviation; for example, OH or NJ\n",
    "- `Account Length`: the number of days that this account has been active\n",
    "- `Area Code`: the three-digit area code of the corresponding customer’s phone number\n",
    "- `Phone`: the remaining seven-digit phone number\n",
    "- `Int’l Plan`: whether the customer has an international calling plan: yes/no\n",
    "- `VMail Plan`: whether the customer has a voice mail feature: yes/no\n",
    "- `VMail Message`: presumably the average number of voice mail messages per month\n",
    "- `Day Mins`: the total number of calling minutes used during the day\n",
    "- `Day Calls`: the total number of calls placed during the day\n",
    "- `Day Charge`: the billed cost of daytime calls\n",
    "- `Eve Mins, Eve Calls, Eve Charge`: the billed cost for calls placed during the evening\n",
    "- `Night Mins`, `Night Calls`, `Night Charge`: the billed cost for calls placed during nighttime\n",
    "- `Intl Mins`, `Intl Calls`, `Intl Charge`: the billed cost for international calls\n",
    "- `CustServ Calls`: the number of calls placed to Customer Service\n",
    "- `Churn?`: whether the customer left the service: true/false\n",
    "\n",
    "The last attribute, `Churn?`, is known as the target attribute–the attribute that we want the ML model to predict.  Because the target attribute is binary, our model will be performing binary prediction, also known as binary classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing our data analysis, we can leverage different Analysis type on Data Wrangler. Next, we can use Histogram to understand our feature distribution and how it impacts our target value. Going back to the \"Analyze\" tab, we can add a new Analysis. Then, we can choose Histogram as the \"Analysis type\" and select a feature to explore its distribution. On the following images we've chosen `Account Length` as feature and colored it by our target variable `Churn?`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/090-analyze_tab_create_new_analysis.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/100-account_length_histogram.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we are able to check the distribution of our feature and how its distribution relates to our target value. Feel free to create new histograms for any other feature!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our last analysis, we'll leverage the \"Quick Model\" analysis provided by Data Wrangler. This analysis trains a Random Forest algorithm on its own and calculates a feature importance score for each feature on our dataset. You can learn more about the \"Quick Model\" analysis on this [page](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-analyses.html#data-wrangler-quick-model) of the Amazon SageMaker Data Wrangler documentation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/110-quick_model_analysis.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The higher the score, the more important the feature is. Therefore, feature `Day Mins` is the most important feature on our dataset according to the \"Quick Model\" analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Transforms on Data Wrangler\n",
    "\n",
    "Go back to the DAG view by clicking on `Back to data flow` tab (on the top  left).\n",
    "\n",
    "Follow the instructions on the image below:  \n",
    "1 - Click on the plus (+) button  \n",
    "2 - Click Edit data types  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/120-edit_data_types.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 - Find the column you want to change  \n",
    "4 - Select the desired type from the dropdown (change **Area Code** to String)   \n",
    "5 - Click preview  \n",
    "6 - Click Add"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/130-area_code_to_object.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you finish, click \"Back to data flow\" on the top right corner\n",
    "\n",
    "### Now lets drop the Phone column by adding a Transform  \n",
    "1 - Click the plus (+)  \n",
    "2 - Add Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/140-add_transform.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 - Click on manage columns  \n",
    "4 - Select the `Phone` column from the dropdown (as shown in step 2)  \n",
    "5 - Click on preview  \n",
    "6 - Click add"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/150-steps_drop_phone_col.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now lets Drop a few more columns  \n",
    "I'll be dropping the first (Day Charge) as an example, just repeat the steps in the image below for the following columns:\n",
    "* \"Day Charge\"\n",
    "* \"Eve Charge\"\n",
    "* \"Night Charge\"\n",
    "* \"Intl Charge\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/160-adding_new_transform.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/170-drop_column_pt2.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now lets do OneHot Encoding using a custom transform\n",
    "You can copy the code for the custom transform here:\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "model_data = pd.get_dummies(df)\n",
    "\n",
    "df = pd.concat(\n",
    "  [model_data[\"Churn?_True.\"],\n",
    "   model_data.drop( [\"Churn?_False.\", \"Churn?_True.\"], axis=1)],\n",
    "  axis=1\n",
    ").rename( \n",
    "  columns = {\n",
    "    \"Churn?_True.\": \"Churn\"\n",
    "  }\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/180-custom_transform.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exporting Transformed data on Data Wrangler\n",
    "\n",
    "After performing the transformations needed on our dataset, we'll export the transformed data to our S3 bucket. We are able to do so inside Data Wrangler UI by following the steps highlighted on the images below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/190-select_transforms_to_export.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/200-exporting_to_s3.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After selecting the `Save to S3` option, a new notebook will be displayed similar to the one presented on the image below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/210-export_to_s3_notebook.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can proceed to `Run all cells`, as descrribed in the notebook. The processing job will start and it will take a few minutes to finish. Upon completion, we'll see a similar output on the cell presented on the following image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/219-dw-notebook-run-all.png\" width=\"100%\" />\n",
    "\n",
    "In the end of that notebook check that Processing Job is running (with the Data Wrangler Docker image):\n",
    "\n",
    "<img src=\"media/220-processing_job_finished.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the left menu, click on \"Components and Registries\", select \"Experiments and trial\" on the dropdown.\n",
    "\n",
    "Select and double-click \"Unassigned trial components\":\n",
    "\n",
    "<img src=\"media/225-open-experiments-processing.png\" width=\"50%\"/>\n",
    "\n",
    "Select your processing job trial and open (right click and select \"Open in trial details\"):\n",
    "\n",
    "<img src=\"media/225-find-processing-info.png\" width=\"60%\"/>\n",
    "\n",
    "Go to the `Artifacts` tab, and **copy** the S3 URI of the output:\n",
    "\n",
    "<img src=\"media/225-find-processing-output.png\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Paste your S3 URI below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3uri_processed = \"s3://sagemaker-us-east-1-686948287393/export-flow-31-14-44-25-b1725f5a/output\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3uri_processed_file = sagemaker.s3.S3Downloader.list(s3uri_processed)[0]\n",
    "s3uri_processed_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to check it in S3 console run the cell bellow and click the link:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "from sagemaker.s3 import parse_s3_url\n",
    "\n",
    "out_bucket, out_prefix = parse_s3_url(s3uri_processed_file)\n",
    "out_path = os.path.dirname(out_prefix)\n",
    "out_file = os.path.basename(out_prefix)\n",
    "\n",
    "s3_url_placeholder = \"https://s3.console.aws.amazon.com/s3/buckets/{}?&prefix={}/\"\n",
    "display(HTML(f\"<a href={s3_url_placeholder.format(out_bucket, out_path)}>Go to S3 console and check output of Data Wrangler</a>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the S3 console you should see:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/230-download_transformed_data_s3.png\" width=\"100%\" />\n",
    "\n",
    "(If you want to download the data to you computer follow the steps in the image above)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's download the data to Studio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.download_data(\".\", \n",
    "                   out_bucket, \n",
    "                   key_prefix=out_prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click the refresh button on Studio. You should see something like:\n",
    "\n",
    "<img src=\"media/230-download_transformed_data_s3_local.png\" width=\"50%\" />\n",
    "\n",
    "(The CSV file is downloaded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_data = pd.read_csv(out_file)\n",
    "model_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we should see the transformed data with `Churn` in the first columns, the one-hot-encoded columns and so on.\n",
    "\n",
    "Finally, let's break the data into **train, validation and test sets:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, validation_data, test_data = np.split(\n",
    "    model_data.sample(frac=1, random_state=1729),\n",
    "    [int(0.7 * len(model_data)), int(0.9 * len(model_data))],\n",
    ")\n",
    "train_data.shape, validation_data.shape, test_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create CSV files for the 3 datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file_name = \"train.csv\"\n",
    "validation_file_name = \"validation.csv\"\n",
    "test_file_name = \"test.csv\"\n",
    "\n",
    "train_data.to_csv(train_file_name , header=False, index=False)\n",
    "validation_data.to_csv(validation_file_name, header=False, index=False)\n",
    "test_data.to_csv(test_file_name, header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we'll upload these files to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the URLs of the uploaded files, so they can be reviewed or used elsewhere\n",
    "train_dir = f\"{prefix}/data/train\"\n",
    "val_dir = f\"{prefix}/data/validation\"\n",
    "test_dir = f\"{prefix}/data/test\"\n",
    "\n",
    "s3uri_train = sagemaker.s3.S3Uploader.upload(train_file_name, f's3://{bucket}/{train_dir}')\n",
    "s3uri_validation = sagemaker.s3.S3Uploader.upload(validation_file_name, f's3://{bucket}/{val_dir}')\n",
    "s3uri_test = sagemaker.s3.S3Uploader.upload(test_file_name, f's3://{bucket}/{test_dir}')\n",
    "s3uri_train, s3uri_validation, s3uri_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the S3 URIs for the 3 datasets for later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store s3uri_train\n",
    "%store s3uri_validation\n",
    "%store s3uri_test"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}