Use XGBoost to classify tumors.ipynb

{
    "nbformat_minor": 2, 
    "cells": [
        {
            "source": "<div style=\"background:#F5F7FA; height:100px; padding: 2em; font-size:14px;\">\n<span style=\"font-size:18px;color:#152935;\">Want to do more?</span><span style=\"border: 1px solid #3d70b2;padding: 15px;float:right;margin-right:40px; color:#3d70b2; \"><a href=\"https://ibm.co/wsnotebooks\" target=\"_blank\" style=\"color: #3d70b2;text-decoration: none;\">Sign Up</a></span><br>\n<span style=\"color:#5A6872;\"> Try out this notebook with your free trial of IBM Watson Studio.</span>\n</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Use XGBoost to classify tumors with IBM Watson Machine Learning</b></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr>\n   <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://raw.githubusercontent.com/pmservice/wml-sample-notebooks/master/images/cancer_banner-06.png\" alt=\"Icon\" width=\"700\"> </th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to get data from the IBM Watson Studio Community, create a predictive model, and start scoring new data. This notebook introduces commands for getting data and for basic data cleaning and exploration, model training, model persistance to Watson Machine Learning repository, model deployment, and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.5, XGBoost, and scikit-learn.\n\nYou will use a publicly available data set, the Breast Cancer Wisconsin (Diagnostic) Data Set, to train an XGBoost Model to classify breast cancer tumors (as benign or malignant) from 569 diagnostic images based on measurements such as radius, texture, perimeter and area. XGBoost is short for \u201cE**x**treme **G**radient **Boost**ing\u201d.\n\nThe XGBoost classifier makes its predictions based on the majority vote from collection of models which are a set of classification trees. It uses the combination of weak learners to create a single strong learner. It\u2019s a sequential training process, whereby new learners focus on the misclassified examples of previous learners.\n\n\n## Learning goals\n\nYou will learn how to:\n\n-  Load a CSV file into numpy array\n-  Explore data\n-  Prepare data for training and evaluation\n-  Create an XGBoost machine learning model\n-  Train and evaluate a model\n-  Use cross-validation to optimize model's hyperparameters\n-  Persist a model in Watson Machine Learning repository\n-  Deploy a model for online scoring\n-  Score sample data\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Set up the environment](#setup)\n2.\t[Load and explore the data](#load)\n3.\t[Create the XGBoost model](#model)\n4.\t[Persist model](#persistence)\n5.\t[Deploy and score in a Cloud](#scoring)\n6.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Set up the environment\n\nBefore you use the sample code in this notebook, you have to perform the following setup tasks:\n\n- Create a [Watson Machine Learning (WML) Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered and information about how to create the instance is [here](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html))\n-  Download **Breast Cancer Wisconsin (Diagnostic) Data Set** dataset from Watson Studio [Community](https://dataplatform.ibm.com/community?context=analytics).\n\n**Note:** We provide the code to download data set, see [step 2](#load).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 2. Load and explore the data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will load the data as a numpy array and perform a basic exploration.\n\nTo load the data as a numpy array, user `wget` to download the data, then use the `genfromtxt` method to read the data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**: First, you need to install the required packages. You can do this by running the following code. Run it only one time.<BR><BR>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!pip install wget --upgrade"
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "BreastCancerWisconsinDataSet.csv\n"
                }
            ], 
            "source": "# Get the data.\nimport wget, os\n\nWisconsinDataSet = 'BreastCancerWisconsinDataSet.csv' \nif not os.path.isfile(WisconsinDataSet):\n    link_to_data = 'https://apsportal.ibm.com/exchange-api/v1/entries/c173693bf48aeb22e41bbe2b41d79c1f/data?accessKey=941eec501eadcdceb5abd25cf7c029d5'\n    WisconsinDataSet = wget.download(link_to_data)\n\nprint(WisconsinDataSet)"
        }, 
        {
            "source": "The csv file **BreastCancerWisconsinDataSet.csv** is downloaded. Run the code in the next cells to load the file to the numpy array.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Note:** Update `numpy` to ensure you have the latest version.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting numpy\n  Downloading numpy-1.14.2-cp35-cp35m-manylinux1_x86_64.whl (12.1MB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12.1MB 75kB/s  eta 0:00:01\n\u001b[?25hInstalling collected packages: numpy\n  Found existing installation: numpy 1.13.3\n    Uninstalling numpy-1.13.3:\n      Successfully uninstalled numpy-1.13.3\nSuccessfully installed numpy-1.14.2\n"
                }
            ], 
            "source": "# Run this code to upgrade numpy.\n!pip install numpy --upgrade"
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "(842302, 'M', 17.99, 10.38, 122.8, 1001., 0.1184, 0.2776, 0.3001, 0.1471, 0.2419, 0.07871, 1.095, 0.9053, 8.589, 153.4, 0.006399, 0.04904, 0.05373, 0.01587, 0.03003, 0.006193, 25.38, 17.33, 184.6, 2019., 0.1622, 0.6656, 0.7119, 0.2654, 0.4601, 0.1189)\n"
                }
            ], 
            "source": "import numpy as np\n\nnp_data = np.genfromtxt(WisconsinDataSet, delimiter=',', names=True, dtype=None, encoding='utf-8')\nprint(np_data[0])"
        }, 
        {
            "source": "Run the code in the next cell to view the feature names and data storage types.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[('id', '<i8'), ('diagnosis', '<U1'), ('radius_mean', '<f8'), ('texture_mean', '<f8'), ('perimeter_mean', '<f8'), ('area_mean', '<f8'), ('smoothness_mean', '<f8'), ('compactness_mean', '<f8'), ('concavity_mean', '<f8'), ('concave_points_mean', '<f8'), ('symmetry_mean', '<f8'), ('fractal_dimension_mean', '<f8'), ('radius_se', '<f8'), ('texture_se', '<f8'), ('perimeter_se', '<f8'), ('area_se', '<f8'), ('smoothness_se', '<f8'), ('compactness_se', '<f8'), ('concavity_se', '<f8'), ('concave_points_se', '<f8'), ('symmetry_se', '<f8'), ('fractal_dimension_se', '<f8'), ('radius_worst', '<f8'), ('texture_worst', '<f8'), ('perimeter_worst', '<f8'), ('area_worst', '<f8'), ('smoothness_worst', '<f8'), ('compactness_worst', '<f8'), ('concavity_worst', '<f8'), ('concave_points_worst', '<f8'), ('symmetry_worst', '<f8'), ('fractal_dimension_worst', '<f8')]\n"
                }
            ], 
            "source": "# Display the feature names and data storage types.\nprint(np_data.dtype)"
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of rows: 569\nNumber of columns: 32\n"
                }
            ], 
            "source": "# Display the number of records and features.\nprint('Number of rows: {}'.format(np_data.size))\nprint('Number of columns: {}'.format(len(np_data[0])))"
        }, 
        {
            "source": "You can see that the data set has 569 records and 32 features.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 3. Create an XGBoost model\n\nIn this section you will learn how to train and test an XGBoost model.\n\n- [3.1. Prepare the data](#prepare)\n- [3.2. Create the XGBoost model](#create)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1. Prepare data<a id=\"prepare\"></a>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now, you can prepare your data for model building. You will use the `diagnosis` column as your target variable so you must remove it from the set of predictors. You must also remove the `id` variable.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "y = 1.0*(np_data['diagnosis'] == 'M')\nX = np.array([list(r)[2:] for r in np_data])"
        }, 
        {
            "source": "Split the data set into: \n- Train data set\n- Test data set", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Split the data set and create two data sets.\nfrom sklearn.cross_validation import train_test_split \n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.35, random_state=143)"
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of training records: 369\nNumber of testing records : 200\n"
                }
            ], 
            "source": "# List the number of records in each data set.\nprint(\"Number of training records: \" + str(X_train.shape[0]))\nprint(\"Number of testing records : \" + str(X_test.shape[0]))"
        }, 
        {
            "source": "The data has been successfully split into two data sets:\n- The train data set, which is the largest group, will be used for training\n- The test data set will be used for model evaluation and is used to test the assumptions of the model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2. Create the XGBoost model<a id=\"create\"></a>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Start by importing the necessary libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n  DeprecationWarning)\n"
                }
            ], 
            "source": "# Import the libraries you need to create the XGBoost model.\nfrom xgboost.sklearn import XGBClassifier\nfrom sklearn.grid_search import GridSearchCV\nfrom sklearn.metrics import accuracy_score"
        }, 
        {
            "source": "#### 3.2.1. Create an XGBoost classifier", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you create an XGBoost classifier with default hyperparameter values and you will call it *xgb_model*. \n\n**Note** The next sections show you how to improve this base model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Create the XGB classifier, xgb_model.\nxgb_model = XGBClassifier()"
        }, 
        {
            "source": "Display the default parameters for *xgb_model*.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'learning_rate': 0.1, 'min_child_weight': 1, 'silent': 1, 'reg_lambda': 1, 'seed': 0, 'subsample': 1, 'scale_pos_weight': 1, 'n_estimators': 100, 'base_score': 0.5, 'reg_alpha': 0, 'missing': None, 'max_delta_step': 0, 'gamma': 0, 'colsample_bylevel': 1, 'objective': 'binary:logistic', 'colsample_bytree': 1, 'max_depth': 3}\n"
                }
            ], 
            "source": "# List the default parameters.\nprint(xgb_model.get_xgb_params())"
        }, 
        {
            "source": "Now that your XGBoost classifier, *xgb_model*, is set up, you can train it by invoking the fit method. You will also evaluate *xgb_model* while the train and test data are being trained.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[0]\tvalidation_0-error:0.02168\tvalidation_1-error:0.05\n[1]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[2]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[3]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[4]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[5]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[6]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[7]\tvalidation_0-error:0.01626\tvalidation_1-error:0.05\n[8]\tvalidation_0-error:0.01355\tvalidation_1-error:0.05\n[9]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[10]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[11]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[12]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[13]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[14]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[15]\tvalidation_0-error:0.00813\tvalidation_1-error:0.05\n[16]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[17]\tvalidation_0-error:0.01084\tvalidation_1-error:0.04\n[18]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[19]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[20]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[21]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[22]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[23]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[24]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[25]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[26]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[27]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[28]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[29]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[30]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[31]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[32]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[33]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[34]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[35]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[36]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[37]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[38]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[39]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[40]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[41]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[42]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[43]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[44]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[45]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[46]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[47]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[48]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[49]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[50]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[51]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[52]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[53]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[54]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[55]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[56]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04\n[57]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04\n[58]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[59]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[60]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05\n[61]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[62]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[63]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05\n[64]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[65]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[66]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[67]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[68]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[69]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[70]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[71]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[72]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[73]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[74]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[75]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[76]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[77]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[78]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[79]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[80]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[81]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[82]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[83]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[84]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[85]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[86]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[87]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[88]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[89]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[90]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[91]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[92]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[93]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[94]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[95]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[96]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[97]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[98]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[99]\tvalidation_0-error:0\tvalidation_1-error:0.055\n"
                }, 
                {
                    "execution_count": 14, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=100, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "# Train and evaluate.\nxgb_model.fit(X_train, y_train, eval_metric=['error'], eval_set=[((X_train, y_train)),(X_test, y_test)])"
        }, 
        {
            "source": "**Note:** You can also use a pandas dataFrame instead of the numpy array.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Plot the model performance evaluated during the training process to assess model overfitting.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Import the library\nfrom matplotlib import pyplot\n\n%matplotlib inline"
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 16, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Text(0.5,0,'Number of iterations')"
                    }, 
                    "output_type": "execute_result"
                }, 
                {
                    "output_type": "display_data", 
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAGDCAYAAAAGSkjRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8XOV97/HPT5vlFdsjsXlfZMBsBoSJA8GEEAI0hQRIArdJWJK6uUlaunBb57bNQpqGNEsbAg2XBhNoUgiXtLnkFi4QgmUCBWz2xUiyDcbCBo/kFcu2PNLv/nHO2OPxSDOSZtGZ+b5fL70055xnzjwzDP7qOc9znsfcHRERERn5qkpdAREREcmNQltERCQiFNoiIiIRodAWERGJCIW2iIhIRCi0RUREIkKhLSIiEhEKbZGIMLPlZrbVzEaVui4iUhoKbZEIMLOZwAcABy4u4uvWFOu1RCQ7hbZINHwWeAr4KXBVcqeZjTaz75vZejPbbma/M7PR4bGzzOxJM9tmZhvM7Opw/3Iz+3zKOa42s9+lbLuZfcnM2oH2cN8Pw3PsMLNnzewDKeWrzex/mtlaM9sZHp9mZreY2fdT34SZ/drM/rQQH5BIJVBoi0TDZ4Gfhz8fMbMjwv3fA04D3g9MBv4S6DOz6cCDwI+ARmAB8MIgXu9jwBnA/HB7ZXiOycC/Af/bzOrDY38OXAlcBEwArgW6gTuBK82sCsDMGoAPAXcP5o2LyAEKbZERzszOAmYA97r7s8Ba4L+FYXgtcJ27v+3uve7+pLvvBf4A+I273+3u+9y9y90HE9rfdvct7r4bwN1/Fp4j4e7fB0YBx4RlPw/8jbu3euDFsOwzwHaCoAa4Alju7u8O8yMRqVgKbZGR7yrgYXfvDLf/LdzXANQThHi6af3sz9WG1A0z+wszWx1egt8GHBa+frbXuhP4dPj408C/DqNOIhVPg0xERrCwf/qTQLWZvRPuHgVMBI4C9gBzgBfTnroBWNjPaXcBY1K2j8xQZv/yf2H/9V8RtJhfdfc+M9sKWMprzQFeyXCenwGvmNnJwHHAr/qpk4jkQC1tkZHtY0AvQd/ygvDnOOBxgn7uZcAPzOzocEDYovCWsJ8D55nZJ82sxsxiZrYgPOcLwKVmNsbM5gKfy1KH8UACiAM1ZvZVgr7rpJ8A3zSzJgucZGYxAHfvIOgP/1fgl8nL7SIyNAptkZHtKuAOd3/L3d9J/gA3E/RbLwVeJgjGLcB3gCp3f4tgYNhfhPtfAE4Oz/mPQA/wLsHl659nqcNDBIPa2oD1BK371MvnPwDuBR4GdgC3A6NTjt8JnIgujYsMm7l79lIiIkNkZmcTXCaf6e59pa6PSJSppS0iBWNmtcB1wE8U2CLDp9AWkYIws+OAbQQD5v6pxNURKQu6PC4iIhIRammLiIhEhEJbREQkIkbc5CoNDQ0+c+bMUldDRESkaJ599tlOd2/MVm7EhfbMmTNZtWpVqashIiJSNGa2PpdyujwuIiISEQptERGRiFBoi4iIRMSI69POZN++fXR0dLBnz55SV6Vo6uvrmTp1KrW1taWuioiIjBCRCO2Ojg7Gjx/PzJkzMbPsT4g4d6erq4uOjg5mzZpV6uqIiMgIEYnL43v27CEWi1VEYAOYGbFYrKKuLIiISHaRCG2gYgI7qdLer4iIZBeZ0C6lrq4uFixYwIIFCzjyyCOZMmXK/u2enp6cznHNNdfQ2tpa4JqKiEg5i0SfdqnFYjFeeOEFAL7+9a8zbtw4rr/++oPKuDvuTlVV5r+D7rjjjoLXU0REylvWlraZLTOzzWb2Sj/HzcxuMrM1ZvaSmZ2acuwqM2sPf67KZ8VHgjVr1nDCCSfwhS98gVNPPZVNmzaxZMkSmpubOf7447nhhhv2lz3rrLN44YUXSCQSTJw4kaVLl3LyySezaNEiNm/eXMJ3ISIiUZFLS/unwM3AXf0cvxBoCn/OAH4MnGFmk4GvAc2AA8+a2f3uvnU4Ff7Gr1/ltY07hnOKQ8w/egJf+/3jh/Tc1157jTvuuINbb70VgBtvvJHJkyeTSCT44Ac/yOWXX878+fMPes727dtZvHgxN954I3/+53/OsmXLWLp06bDfh4iIlLesoe3uK8xs5gBFLgHu8mBh7qfMbKKZHQWcAzzi7lsAzOwR4ALg7uFWeiSZM2cOp59++v7tu+++m9tvv51EIsHGjRt57bXXDgnt0aNHc+GFFwJw2mmn8fjjjxe1ziKVbsuuHl7s2FbqakiEnXD0YTSOH1X0181Hn/YUYEPKdke4r7/9hzCzJcASgOnTpw/4YkNtERfK2LFj9z9ub2/nhz/8Ic888wwTJ07k05/+dMbbturq6vY/rq6uJpFIFKWuIhL4y/te4jer3y11NSTCbvvMaZx//JFFf918hHame5N8gP2H7nS/DbgNoLm5OWOZKNixYwfjx49nwoQJbNq0iYceeogLLrig1NUSkRR7E708saaTSxYczdXvn1nq6khEzW4YV5LXzUdodwDTUranAhvD/eek7V+eh9cbsU499VTmz5/PCSecwOzZsznzzDNLXSURSbPqza3s3tfLJQuO5pTpk0pdHZFBsaArOkuhoE/7/7r7CRmO/R7wZeAigoFoN7n7wnAg2rNAcjT5c8BpyT7u/jQ3N3v6etqrV6/muOOOy1rPclOp71ukkP7+gdX89Ik3eeFrH2ZMne56lZHBzJ519+Zs5bJ+Y83sboIWc4OZdRCMCK8FcPdbgQcIAnsN0A1cEx7bYmbfBFaGp7ohW2CLiBRaS2uc02dNUmBLJOUyevzKLMcd+FI/x5YBy4ZWNRGR/Nq0fTet7+7kstOOLXVVRIZE05iKSMVY0RYHYPG8w0tcE5GhUWiLSMVY0dbJkRPqmXdEaUb+igyXQltEKkKit4/H2+MsnteoVfQkshTaIlIRXuzYxo49Cc6e11jqqogMmUI7B/lYmhNg2bJlvPPOOwWsqYj0p6U1TpXBWXMbSl0VkSHTPQ85yGVpzlwsW7aMU089lSOPLP7UdyKVrqUtzinTJ3HYmNpSV0VkyNTSHqY777yThQsXsmDBAr74xS/S19dHIpHgM5/5DCeeeCInnHACN910E7/4xS944YUX+NSnPjXoFrqIDM+WXT289PZ2FuvSuERc9FraDy6Fd17O7zmPPBEuvHHQT3vllVf4j//4D5588klqampYsmQJ99xzD3PmzKGzs5OXXw7quW3bNiZOnMiPfvQjbr75ZhYsWJDf+ovIgB5vj+OOQlsiL3qhPYL85je/YeXKlTQ3BzPP7d69m2nTpvGRj3yE1tZWrrvuOi666CLOP//8EtdUZORYvWkHN/92Db19xVsbqPXdnUweW8eJUw4r2muKFEL0QnsILeJCcXeuvfZavvnNbx5y7KWXXuLBBx/kpptu4pe//CW33XZbCWooMvLc+eSbPPLau8xqGJu9cJ7UVVfx+Q/MoqpKt3pJtEUvtEeQ8847j8svv5zrrruOhoYGurq62LVrF6NHj6a+vp5PfOITzJo1iy984QsAjB8/np07d5a41iKl4+60tMX50HGH8+NPn1bq6ohEjkJ7GE488US+9rWvcd5559HX10dtbS233nor1dXVfO5zn8PdMTO+853vAHDNNdfw+c9/ntGjR/PMM89QV1dX4ncgUlztm99j0/Y9XPch9S2LDEVOS3MWk5bmPKBS37eUr588vo6/+8/VPLn0XI6eOLrU1REZMXJdmlO3fIlI0bS0xWk6fJwCW2SIFNoiUhTdPQmeXrdFt12JDINCW0SK4ul1W+jp7WPxMQptkaGKTGiPtL73Qqu09yvlr6UtTn1tFafPnFzqqohEViRCu76+nq6urooJMnenq6uL+vr6UldFJG9a2uIsmh2jvra61FURiaxI3PI1depUOjo6iMfjpa5K0dTX1zN16tRSV0MkL9Z37eKNzl1ctWhGqasiEmmRCO3a2lpmzZpV6mqIyBCtaAv+4NZa1iLDE4nL4yISbS1tcaZNHl3UqUtFypFCW0QKqifRx5Nru1g8rxEzzf0tMhyRuDwu0efu7NidGLDMqNqqAQcp7e7ppSfRl++qSYGtWr+F7p5eFs87vNRVEYk8hbYUxd/952pu/90bA5aZUF/D4391LoeNrj3k2Ovv7OCjN/2ORBGXc5T8qa02Fs2JlboaIpGn0JaCc3f+86VNLJg2kYtPPjpjmc0793Jry1qeWNPJRScedcjxh155l153/vqi46jW8oqRM+fwcYwbpX9uRIZL/xdJwbVvfo93duzhT89r4oqF0zOWSfT28fOn17OiLZ4xtFe0xzlp6kT+8OzZha6uiMiIpYFoUnAtrdlv96mpruKsuQ20tMUPmURne/c+nn9rq+asFpGKp9CWgmtpizPviOwrOy2e18im7Xto3/zeQft/t6aTPkehLSIVT6EtBdXdk+CZN3Jb2SnZEk+2zJNa2jYzob6Gk6ceVpA6iohEhUJbCuqpdV3Byk453O5z9MTRNB0+jhXtB0Lb3Wlpi/OBpkZqqvV1FZHKpn8FpaBaWuOMrq2meeaknMovntfI0+u20N0T3NPd+u5O3t2xV5fGRURQaEuBrWjv5H2zJ+e8stPiYxrp6e3j6XVbgNwGsYmIVAqFthRMcmWnwbSST585mfraKlrCBSZa2uIce+R4jjxMy5SKiCi0pWCSKzstPib36Svra6tZNDtGS1ucXXsTrHpTt3qJiCQptKVgWtriTJ88hpmxMYN63tnzGnmjcxf3rtpAT2+fLo2LiIQU2lIQexO9Q17ZKdmy/uGj7YMaxCYiUu4U2lIQz765NVzZafCt5FkNY5k2eTTbuvfx/jkxRtXkNohNRKTcae7xMrR5xx5ef2dnzpeVE7193L1yA917B146czCeXNs15JWdzIzF8xr52VNvsfgYXRoXEUlSaJehHzzSxr2rNvDs33yYSWPrspZ/9PXN/O2vXsl7PT5y/BGMHeLKTr9/0tH8+sVNfOi4I/JcKxGR6FJolxl3Z3lrnD6Hx9d09rsUZqqWtjjjRtXw5FfOpSaPy16OzvHe7EzOmB3jxa+dn7e6iIiUA4V2mUkugwnBLVfZQtvdWdEW5/1zYkyory1GFUVEZIg0EK3MJGcQO33mpIzLXKZb17mLjq271XcsIhIBCu0y09IW55gjxvPJ5mnEd+5l9aadA5dPThPapNAWERnpFNplZP8ymMc07r/VKjkdaH9a2uLMaRzLtMmDmwBFRESKT6FdRg4sg9nI4RPqOe6oCbS0be63/J59vTy1rkszjomIREROoW1mF5hZq5mtMbOlGY7PMLNHzewlM1tuZlNTjn3HzF4Jfz6Vz8rLwdKXwVw8r5FVb27lvX7uv376jS3sTfRpbm8RkYjIGtpmVg3cAlwIzAeuNLP5acW+B9zl7icBNwDfDp/7e8CpwALgDOB/mNmE/FVfUq1o72RRygxii+c1kuhz/mttV+bybXFG1VTxvtmDnwBFRESKL5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePpRyfD7S4e8LddwEvAhcMv9qSLtMymKfNmMTYuup+L5G3tMU5Y3Ys57WuRUSktHIJ7SnAhpTtjnBfqheBy8LHHwfGm1ks3H+hmY0xswbgg8C09BcwsyVmtsrMVsXjAw+cksz2L4OZEtp1NVUsmtPA8tZDb/3q2NrNms3v6dK4iEiE5BLamabISr/593pgsZk9DywG3gYS7v4w8ADwJHA38F/AIR2s7n6buze7e3Njo0JkKFra4syIjWFmw9iD9i8+ppGOrbt5o3PXQftXtHUGx+c1FK2OIiIyPLmEdgcHt46nAhtTC7j7Rne/1N1PAf463Lc9/P0td1/g7h8m+AOgPS81l/16En08ubYr473Wi5sy3/rV0raZKRNHM6dxXFHqKCIiw5dLaK8EmsxslpnVAVcA96cWMLMGM0ue6yvAsnB/dXiZHDM7CTgJeDhflZfAqvVb+l0Gc3psDLMbxu6/fA6wr7ePJ9cEt3oNdq1rEREpnaxzj7t7wsy+DDwEVAPL3P1VM7sBWOXu9wPnAN82MwdWAF8Kn14LPB4Gww7g0+6ev/UfI2jX3gRX3/EMXbt68nbOHbv3DbgM5tnzGvnZU+s59/vLAUj0Ojv3JtSfLSISMTktGOLuDxD0Tafu+2rK4/uA+zI8bw/BCHIJPd7eyco3t3LOMY2MG+KylZmcOn1Sv8tgfmbRDLZ195DoOzAU4aymBs7RfOMiIpGiVb6KLLkM5m2faaaupjgT0s1pHMc/XXFKUV5LREQKR9OYFlHqMpjFCmwRESkfSo4iWhvfxdvbtAymiIgMjUK7iJK3XWkZTBERGQqFdhG1tMWZrWUwRURkiBTaRbJnXy9Pr+vSbVYiIjJkCu0i0TKYIiIyXArtImlp1TKYIiIyPArtImlp26xlMEVEZFgU2kWwYUs3a+O7OLtJK2qJiMjQKbSLYEV7cKuXpg0VEZHhUGgXwYq2uJbBFBGRYVNoF9i+3j6e0DKYIiKSBwrtAntu/Vbe25tg8Tz1Z4uIyPAotAuspS1OTZXx/rkKbRERGR6FdoG1tMU5dfokJtTXlroqIiIScQrtAorv3MurG3doVS8REckLhXYBPR7e6qWpS0VEJB8U2gXU0hanYVwd84+aUOqqiIhIGVBoF0hvn7OiLc7ZTY1UVelWLxERGT6FdoG88vZ2tnbvU3+2iIjkjUK7QFa0xTGDs3Srl4iI5IlCu0Ba2uKcNOUwYuNGlboqIiJSJhTaBbC9ex/PvbWVszVqXERE8kihXQBPrO2kz3Wrl4iI5JdCuwBaWuOMr69hwbSJpa6KiIiUEYV2nrk7LW1xPtDUQE21Pl4REckfpUqetW9+j3d27NGlcRERyTuFdp61tAZTl2oQmoiI5JtCO89a2uIcc8R4jjpsdKmrIiIiZUahnUfdPQmeeWMLZ8/ThCoiIpJ/Cu08empdFz29fSyed3ipqyIiImVIoZ1HLa1xRtdW0zxzUqmrIiIiZUihnUctbXEWzYlRX1td6qqIiEgZUmjnyfquXbzZ1a1bvUREpGAU2nmyok23eomISGEptPOkpS3O9MljmBkbU+qqiIhImVJo58HeRC9Pru1i8bxGzKzU1RERkTKl0M6DZ9/cSndPr/qzRUSkoBTaedDSFqe22lg0J1bqqoiISBlTaOdBS1uc5hmTGTuqptRVERGRMlZxKePu9PZ53s737s69vP7OTpZeeGzezikiIpJJRYW2u3PxzU/w8tvb837us5vUny0iIoVVUaG9Nr6Ll9/ezu+ddBTHHjE+b+c9fMIojjsqf+cTERHJpKJCuyWcAGXpBccybbLupxYRkWjJaSCamV1gZq1mtsbMlmY4PsPMHjWzl8xsuZlNTTn2D2b2qpmtNrObrIQ3Mre0xZndOFaBLSIikZQ1tM2sGrgFuBCYD1xpZvPTin0PuMvdTwJuAL4dPvf9wJnAScAJwOnA4rzVfhD27Ovl6XVdupdaREQiK5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePpRx3oB6oA0YBtcC7w630UDz9xhb2JvoU2iIiElm5hPYUYEPKdke4L9WLwGXh448D480s5u7/RRDim8Kfh9x9dfoLmNkSM1tlZqvi8fhg30NOWlrjjKqp4n2zNQGKiIhEUy6hnakPOv1G5+uBxWb2PMHl77eBhJnNBY4DphIE/blmdvYhJ3O/zd2b3b25sbEwLeGWts2cMVtrXYuISHTlEtodwLSU7anAxtQC7r7R3S9191OAvw73bSdodT/l7u+5+3vAg8D78lLzQdiwpZu18V2c3dRQ7JcWERHJm1xCeyXQZGazzKwOuAK4P7WAmTWYWfJcXwGWhY/fImiB15hZLUEr/JDL44W2oj245H7OMerPFhGR6Moa2u6eAL4MPEQQuPe6+6tmdoOZXRwWOwdoNbM24AjgW+H++4C1wMsE/d4vuvuv8/sWslvRFmfKxNHMaRxX7JcWERHJm5wmV3H3B4AH0vZ9NeXxfQQBnf68XuCPhlnHYdnX28cTa7r4/ZOP1lrXIiISaWW/ytdz67fy3t6EbvUSEZHIK/vQbmmLU11lvH+ubvUSEZFoq4jQPm36JCbU15a6KiIiIsNS1qG9vXsfr7+zk8UaNS4iImWgrFf5OmxMLc/+zXl4+lQwIiIiEVTWoQ0wcUxdqasgIiKSF2V9eVxERKScKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGRU2ib2QVm1mpma8xsaYbjM8zsUTN7ycyWm9nUcP8HzeyFlJ89ZvaxfL8JERGRSpA1tM2sGrgFuBCYD1xpZvPTin0PuMvdTwJuAL4N4O6PufsCd18AnAt0Aw/nsf4iIiIVI5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePZTgOcDnwoLt3D7WyIiIilSyX0J4CbEjZ7gj3pXoRuCx8/HFgvJnF0spcAdyd6QXMbImZrTKzVfF4PIcqiYiIVJ5cQtsy7PO07euBxWb2PLAYeBtI7D+B2VHAicBDmV7A3W9z92Z3b25sbMyp4iIiIpWmJocyHcC0lO2pwMbUAu6+EbgUwMzGAZe5+/aUIp8E/sPd9w2vuiIiIpUrl5b2SqDJzGaZWR3BZe77UwuYWYOZJc/1FWBZ2jmupJ9L4yIiIpKbrKHt7gngywSXtlcD97r7q2Z2g5ldHBY7B2g1szbgCOBbyeeb2UyClnpLXmsuIiJSYcw9vXu6tJqbm33VqlWlroaIiEjRmNmz7t6crZxmRBMREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiMgptM3sAjNrNbM1ZrY0w/EZZvaomb1kZsvNbGrKselm9rCZrTaz18xsZv6qLyIiUjmyhraZVQO3ABcC84ErzWx+WrHvAXe5+0nADcC3U47dBXzX3Y8DFgKb81FxERGRSlOTQ5mFwBp3XwdgZvcAlwCvpZSZD/xZ+Pgx4Fdh2flAjbs/AuDu7+Wp3jIc78Xh0W9AYm+pazLyzFgEzdcW5tzu8Lt/hPmXQGxO5jLP3glv/q4wr19MM8+C067KfCzeBr/7AfT1Htg3ZjKc/3dQXVuc+olEVC6hPQXYkLLdAZyRVuZF4DLgh8DHgfFmFgPmAdvM7N+BWcBvgKXu3pv6ZDNbAiwBmD59+hDehgzKusfg+X+Fw6ZBVS5fgQqxewu0PwSnXQNm+T//9o7gj6W9O+C8r2cu8+g3oDcRhFhUdW+Btb/tP7Rf/Dd48R6YNDPYTuyBnZvgxE/A1OaiVVMkinL5FzvTv16etn09cLOZXQ2sAN4GEuH5PwCcArwF/AK4Grj9oJO53wbcBtDc3Jx+bsm37q7g9x+tiHY45Nt//TM89JXg8xnbkP/zd60Jfne2Zz7evSV47Q9/E878k/y/frE8cRM88reweyuMnnTo8c52aGiCL68MtuNtcMvpweej0BYZUC4D0TqAaSnbU4GNqQXcfaO7X+rupwB/He7bHj73eXdf5+4Jgsvmp+al5jJ03VvAqqB+YqlrMrI0NAW/+wvV4UqGdvL3IcfXHlyPqNr/Ofb3PtdALOU9TpoJVl24z12kjOQS2iuBJjObZWZ1wBXA/akFzKzBzJLn+gqwLOW5k8ysMdw+l4P7wqUUuruCFlCV7vg7SGxu8LurQOGRDKUt6w7uz01Kvm4s4qGdrH+mP076eoP33zD3wL6auiC4C/W5i5SRrP9qhy3kLwMPAauBe939VTO7wcwuDoudA7SaWRtwBPCt8Lm9BJfOHzWzlwkutf9L3t+FDM7uLTBal8UPMXE6VI8qYEs7PG9vD2x769Djne3BGINJMwrz+sUyaUbwPjKF8La3gvef/odJQ1P/LXMR2S+nUUju/gDwQNq+r6Y8vg+4r5/nPgKcNIw6Sr51d8GYWKlrMfJUVcPk2f1fvh6uzjVw2HTY/lbwGpNnHXy8qx0mzYr+COrq2uB9ZPrjJ/nZpncBxObCuuXQ16crQCID0P8dlah7qwag9adhbmFa2vt2w/YNcMyFwXam1+hcE/3+7KSGpsx//HT20wXQ0BSMIt++4dDniMh+Cu1K1N2l0O5PbC5sfQN69+X3vF1rAYdpC4MBgOmXjpN9vbG5GZ8eObE5wXtO77vvag/ef/r3r9DjCUTKhEK70rirT3sgsSboS8DW9fk9bzKMGprC/tu0cNr2FvTuLZ+WdqwpeD/pLefk7V7p98HHsow4FxFAoV159nUHlyHV0s4sGZr5bvElwyg2Nwio9EvHye2ojxxP6u+2r/TbvZLGHQ6jJqilLZKFQrvSdG8JfmsgWmbJy7T57tfuaocJU6BubHDpeOcm2LvzwPHOlJZ4OYhl+ONn787gfTdk6AIwCz573astMiCFdqVJzoamy+OZjZkc/EGT95Z2+4E/CBoy3Mfc1Q71h5XPH1NjG4L3kxrC2a4m9Dd4TUT2U2hXmt1qaWcVazowO1k+uAfnS4Z1pv7b5GXjQswluPr2AAAT50lEQVR5Xgpm4eeYEtqd/dzulRRrgh1vQ8+uwtdPJKIU2pVm/+VxtbT7le/bvnbFYe/2A2E9eTZghwZauVwaT0qfMKWrPZg+d/LsfsonR5Dn8Q8mkTKj0K406tPOLtYEuzbDnu35Od/+/uowlGrrg9nXkvv3vgc7N5bP7V5JsbnB+9obrsjb2R6875pR/ZQv0CBAkTKi0K40yT5tLRbSv2wLXgxWpjnFG1IuHfc3S1jUpffdd7UPPDo+Ngcw3fYlMgCFdqXZvSUI7Gqto92vfLf4Otuhpj5Yvzz1NbrWBtN2ltvtXkmpC4f09R3cr59J7ejgM1JLW6RfCu1K071F/dnZ5HupyK41MHnOwXNqN8wN7pnfuTF8Heu/rzeqkn33ne3B+9zXnb0LoFDTyIqUCYV2pdFiIdnV1AUrVeWzpR2bc/C+/SPI24PXmTg96OsuJ7X1MDFsOe+fczxLaMfmBn/kuBe+fiIRpNCuNJrCNDexPC0VmeiBrW8eelk4tb83ObVnOYqFU7bm2m8fa4Ke92DnO4Wvm0gEKbQrTfcWtbRz0dAEW8I+5+HY+iZ476H91eOPgrpx0NkW9PWWW392UkPYd9/ZFrzf8UdlKa+FQ0QGotCuNOrTzk1sbn6WikxdKCSVWXDJ/I3HYd+uzFN7loPY3OD9vfF48H6zTR6T2m0gIodQaFeSfXuCf0AV2tnla+GQgfpyY00QX33gcTlKfo7x1bm9xwlToGa0pjMV6YdCu5IkpzBVn3Z2+VoqsqsdxjbC6Az3xTc0ZX5cTtLvTc+mqkoLh4gMQKFdSZITq6ilnd3+pSKHG9oD9FcnW9+59PVG1YSjoXZs8DjXGd8a5qqlLdIPhXYl0RSmuUsuFZmPy+P99VfvX0Akh77eqEr23UPuVxNiTbBtPST2Fq5eIhGlabEqiZblHJyGJnj9P+Hu/za053sfdHdmb2mXa392UkMTvPPSIFraTcFnd/cVQf82BPfOn/93cNjUzM/57bfg3VcP3nfa1TDv/CFXW2QkUmhXEi3LOTjHXwqbV8O2t4Z+jimnwdzzMh+rGwsLl/R/vFyc9KngD8W6sbmVn3kWTDsD3osH294Hm1+FGWfCwj88tHzPLljxD0EXw5iGYN+WddC7V6EtZUehXUmSl8dHTyptPaLimAuCn0K66LuFPf9IMO8jwU+uJhwNn3v4wLY7fHtq//3cyaU8L7gRjv9Y8Pi+a6Fj1dDqKzKCqU+7knRvCQZX1dSVuiYiuUuOL+hvRHmme+FjTcEVkn17Cl8/kSJSaFeS7i61siWaUpcyTde5hkMWXGloAjy4TC5SRhTalWS3pjCViIo1wbYNsG/3oce62oOFSWpHp5TXdKhSnhTalURTmEpUxebQb8u5s/3QkenJ28w0SYuUGYV2JdGynBJVDf3MSe4eDFBLv21u1PhgNLkmaZEyo9CuJLu36h5tiab+LnfvfCdYyjPTxC2aDlXKkEK7UiR6YO8OtbQlmurGBouJpM8F3zXAgizJwWvuha+fSJEotCvF7q3B7zEaPS4RlWla2c5+lj6F4JL5nu2wq7PwdRMpEoV2pdi/WIha2hJRDU1BSzu15dy1BmrHwPijM5dPlhEpEwrtSqFlOSXqYk2wdzvsih/Y19kejBSvyvBPmW77kjKk0K4UWpZToi65Wlrq4LKu9v4XXJk4HapHaTCalBWFdqXQspwSdclwTracE3uDqUr7W/KzqjqYJU2Xx6WMKLQrhZbllKg7bNrBLect64IVwAZa2rRBt31JeVFoV4rdW6F2LNTWl7omIkNTVRX0XydbzskwTs5+lklsLmx9A3r3Fb5+IkWg0K4U3V3qz5boS50wZaB7tPeXb4K+BGxdX/i6iRSBQrtSaN5xKQcNTbD1zWCyoM41MO5IqJ8wcHnQCHIpGwrtStHdpf5sib5YE3gvbFsfXCbvbxDa/vIZRpyLRJhCu1JoWU4pB6kLh3RlWN0r3ZjJwfdeLW0pEwrtSqHL41IOkiG94elgcGW2ljYErfOutYWtl0iRKLQrQW8C9mxTS1uib/REGNsIrQ8G2wPd7pWk276kjCi0K8GebcFv9WlLOYg1QWdr8Lghy+XxZPldm4PFQ0QiTqFdCTSFqZSTZFBX18HEGTmUT/aDa2Y0ib6cQtvMLjCzVjNbY2ZLMxyfYWaPmtlLZrbczKamHOs1sxfCn/vzWXnJ0f4pTBXaUgaSl8Qnzw6mKs21vAajSRmoyVbAzKqBW4APAx3ASjO7391fSyn2PeAud7/TzM4Fvg18Jjy2290X5LneMhhallPKSbLlnG3keNKkmWDV6teWspA1tIGFwBp3XwdgZvcAlwCpoT0f+LPw8WPAr/JZySHr64OffbzUtSi9ne8Gv9WnLeUgGda5hnZNHUyaAc//DN5eVbh6SW7GNsIl/xz8d8mmrxce/CtYuAQa5xW+bhGQS2hPATakbHcAZ6SVeRG4DPgh8HFgvJnF3L0LqDezVUACuNHdDwl0M1sCLAGYPn36oN/EgPbtzu/5oqh+Ahz/cZhwdKlrIjJ8k2fDqZ+FEy7N/Tmn/yG89iv9e1Bqe7bDuuWw6Etw9CnZy3e2wcp/gbENcM4hPbMVKZfQtgz7PG37euBmM7saWAG8TRDSANPdfaOZzQZ+a2Yvu/tBN026+23AbQDNzc3p5x66qir43MN5O52IjABV1XDxjwb3nEVfDH6ktN59DX68KBgUmFNotx/8W3IK7Q5gWsr2VGBjagF33whcCmBm44DL3H17yjHcfZ2ZLQdOATTTgYhIpZk8G7DcBwV2pS0OIzmNHl8JNJnZLDOrA64ADhoFbmYNZpY811eAZeH+SWY2KlkGOJOD+8JFRKRS1NbDxOm5t5yTt+l1rQXP30XYKMsa2u6eAL4MPASsBu5191fN7AYzuzgsdg7QamZtwBHAt8L9xwGrzOxFggFqN6aNOhcRkUrS0DT4lnbPe7BzU+HqFCG5XB7H3R8AHkjb99WUx/cB92V43pPAicOso4iIlItYE6x/Mri7p2qAdqN70CKfPAe2rA0eazCtZkQTEZEiapgL+7ph58aBy3V3BVMwH3NhsK1+bUChLSIixRRLWV51IMnjs86G2rGahjak0BYRkeJJzmjXlSWEky3rhiaIzVFLO6TQFhGR4hl/FNSNy62lnVwUpqFJ92qHFNoiIlI8Zrm1nLvWHFgUJtYE296CfXuKU8cRTKEtIiLFFWvK3kfd2X5gfvmGJsBhy7qCV22kU2iLiEhxNTTB9g39zwXfm4Ctbxy6olu2fvAKoNAWEZHiis1lwJbztvXQlzgw0jw2J/itwWgKbRERKbKGLLd9dbYfXG7U+GAAm277UmiLiEiR7b/c3U9oJ/enrpkem6uWNgptEREptrqxMGFK/y3nznYYE4Mxkw/sS972VeELhyi0RUSk+AZqOXetOdCfvb98UzCtaXdX4es2gim0RUSk+BrC274ytZw724M5ytPLJ49VMIW2iIgUX6wJ9m6HXfGD9+/ZDrs2Z2hpZ+kHrxAKbRERKb5kSzq95Zzs546ltbQnTofqUWppl7oCIiJSgfprOXel3e6VVFUdTGta4ROsKLRFRKT4DpuWueXc2Q5WDZNmHfqchrlqaZe6AiIiUoGqqsOFQ9Jazl3tMGkG1NQd+pxYUzC9ae++4tRxBFJoi4hIacQytJw7M9zulVq+LwFb1xe+biOUQltEREqjoQm2vgmJnmC7rw+2rD20Pzu1PFT0CPKaUldAREQqVKwJvBduPw+q64JWdGLPoSPH95cP9z/4V/D494tXz0zO+wbMPLPoL6vQFhGR0phzLhz7UdjXfWDfMRdB04czlx8zGc7479DZWpz6DaSqNPGp0BYRkdIYfwRc8fPBPefCGwtTl4hQn7aIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIS5e6nrcBAziwPr83zaBqAzz+esRPoc80OfY37oc8wPfY75MdzPcYa7N2YrNOJCuxDMbJW7N5e6HlGnzzE/9Dnmhz7H/NDnmB/F+hx1eVxERCQiFNoiIiIRUSmhfVupK1Am9Dnmhz7H/NDnmB/6HPOjKJ9jRfRpi4iIlINKaWmLiIhEXlmHtpldYGatZrbGzJaWuj5RYWbTzOwxM1ttZq+a2XXh/slm9oiZtYe/J5W6rlFgZtVm9ryZ/d9we5aZPR1+jr8ws7pS13GkM7OJZnafmb0efi8X6fs4eGb2Z+H/06+Y2d1mVq/vY3ZmtszMNpvZKyn7Mn7/LHBTmDsvmdmp+axL2Ya2mVUDtwAXAvOBK81sfmlrFRkJ4C/c/TjgfcCXws9uKfCouzcBj4bbkt11wOqU7e8A/xh+jluBz5WkVtHyQ+D/ufuxwMkEn6e+j4NgZlOAPwGa3f0EoBq4An0fc/FT4IK0ff19/y4EmsKfJcCP81mRsg1tYCGwxt3XuXsPcA9wSYnrFAnuvsndnwsf7yT4B3IKwed3Z1jsTuBjpalhdJjZVOD3gJ+E2wacC9wXFtHnmIWZTQDOBm4HcPced9+Gvo9DUQOMNrMaYAywCX0fs3L3FcCWtN39ff8uAe7ywFPARDM7Kl91KefQngJsSNnuCPfJIJjZTOAU4GngCHffBEGwA4eXrmaR8U/AXwJ94XYM2ObuiXBb38vsZgNx4I6wm+EnZjYWfR8Hxd3fBr4HvEUQ1tuBZ9H3caj6+/4VNHvKObQtwz4NlR8EMxsH/BL4U3ffUer6RI2ZfRTY7O7Ppu7OUFTfy4HVAKcCP3b3U4Bd6FL4oIV9rpcAs4CjgbEEl3LT6fs4PAX9f7ycQ7sDmJayPRXYWKK6RI6Z1RIE9s/d/d/D3e8mL/OEvzeXqn4RcSZwsZm9SdA9cy5By3tieHkS9L3MRQfQ4e5Ph9v3EYS4vo+Dcx7whrvH3X0f8O/A+9H3caj6+/4VNHvKObRXAk3hyMg6ggEX95e4TpEQ9rveDqx29x+kHLofuCp8fBXwf4pdtyhx96+4+1R3n0nw/futu/8B8BhweVhMn2MW7v4OsMHMjgl3fQh4DX0fB+st4H1mNib8fzz5Oer7ODT9ff/uBz4bjiJ/H7A9eRk9H8p6chUzu4igZVMNLHP3b5W4SpFgZmcBjwMvc6Av9n8S9GvfC0wn+AfgE+6ePjhDMjCzc4Dr3f2jZjaboOU9GXge+LS77y1l/UY6M1tAMJivDlgHXEPQ6ND3cRDM7BvApwjuEHke+DxBf6u+jwMws7uBcwhW8noX+BrwKzJ8/8I/iG4mGG3eDVzj7qvyVpdyDm0REZFyUs6Xx0VERMqKQltERCQiFNoiIiIRodAWERGJCIW2iIhIRCi0RYbJzNzMvp+yfb2ZfT1P5/6pmV2eveSwX+cT4epZj6XtP9rM7gsfLwhvo8zXa040sy9mei0RyUyhLTJ8e4FLzayh1BVJFa50l6vPAV909w+m7nT3je6e/KNhATCo0E6ZaSuTicD+0E57LRHJQKEtMnwJ4Dbgz9IPpLeUzey98Pc5ZtZiZveaWZuZ3Whmf2Bmz5jZy2Y2J+U055nZ42G5j4bPrzaz75rZynDN3j9KOe9jZvZvBJPjpNfnyvD8r5jZd8J9XwXOAm41s++mlZ8Zlq0DbgA+ZWYvmNmnzGxsuM7wynAhj0vC51xtZv/bzH4NPGxm48zsUTN7Lnzt5Gp7NwJzwvN9N/la4TnqzeyOsPzzZvbBlHP/u5n9PwvWMf6HlM/jp2FdXzazQ/5biJSDgf4KFpHc3QK8lAyRHJ0MHEew5N864CfuvtDMrgP+GPjTsNxMYDEwB3jMzOYCnyWYHvF0MxsFPGFmD4flFwInuPsbqS9mZkcTrJ18GsG6yQ+b2cfc/QYzO5dgxraMMze5e08Y7s3u/uXwfH9PMDXrtWY2EXjGzH4TPmURcFI4Q1QN8HF33xFejXjKzO4nWPTjBHdfEJ5vZspLfil83RPN7NiwrvPCYwsIVp7bC7Sa2Y8IVliaEq4TTVgfkbKjlrZIHoSroN0F/MkgnrYyXLt8L7AWSIbuywRBnXSvu/e5eztBuB8LnE8wv/ELBNPLxoCmsPwz6YEdOh1YHi4YkQB+TrBO9VCdDywN67AcqCeY0hHgkZQpRQ34ezN7CfgNwbSZR2Q591nAvwK4++vAeiAZ2o+6+3Z330Mwd/YMgs9ltpn9yMwuALQqnZQltbRF8uefgOeAO1L2JQj/OA7nJK5LOZY6v3NfynYfB/+/mT7XsBME4R+7+0OpB8I5znf1U79MSwYOhwGXuXtrWh3OSKvDHwCNwGnuvs+CVc/qczh3f1I/t16gxt23mtnJwEcIWumfBK7N6V2IRIha2iJ5ErYs7yUY1JX0JsHlaAjWMq4dwqk/YWZVYT/3bKAVeAj47xYsoYqZzTOzsVnO8zSw2MwawkFqVwItg6jHTmB8yvZDwB+Hf4xgZqf087zDCNYV3xf2Tc/o53ypVhCEPeFl8ekE7zuj8LJ7lbv/EvhbgqU7RcqOQlskv75PsBJQ0r8QBOUzQHoLNFetBOH6IPCF8LLwTwguDT8XDt76X2S5chYuD/gVgqUYXwSec/fBLMP4GDA/ORAN+CbBHyEvhXX4Zj/P+znQbGarCIL49bA+XQR98a+kD4AD/hmoNrOXgV8AV2dZeWoKsDy8VP/T8H2KlB2t8iUiIhIRammLiIhEhEJbREQkIhTaIiIiEaHQFhERiQiFtoiISEQotEVERCJCoS0iIhIRCm0REZGI+P/rnJtklTcHgAAAAABJRU5ErkJggg==\n", 
                        "text/plain": "<matplotlib.figure.Figure at 0x2af3dcf932e8>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "# Plot and display the performance evaluation\nxgb_eval = xgb_model.evals_result()\neval_steps = range(len(xgb_eval['validation_0']['error']))\n\nfig, ax = pyplot.subplots(1, 1, sharex=True, figsize=(8, 6))\n\nax.plot(eval_steps, [1-x for x in xgb_eval['validation_0']['error']], label='Train')\nax.plot(eval_steps, [1-x for x in xgb_eval['validation_1']['error']], label='Test')\nax.legend()\nax.set_title('Accuracy')\nax.set_xlabel('Number of iterations')"
        }, 
        {
            "source": "You can see that there is model overfitting, and there is a decrease in model accuracy after about 60 iterations \n\nSelect the trained model obtained after 30 iterations.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 17, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "# Select trained model.\nn_trees = 30\ny_pred = xgb_model.predict(X_test, ntree_limit= n_trees)"
        }, 
        {
            "execution_count": 18, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 96.0%\n"
                }
            ], 
            "source": "# Check the accuracy of the trained model.\naccuracy = accuracy_score(y_test, y_pred)\n\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "**Note:** You will use the accuracy value obtained on the test data to compare the accuracy of the model with default parameters to the accuracy of the model with tuned parameters.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### 3.2.2. Use grid search and cross-validation to tune the model ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can use grid search and cross-validation to tune your model to achieve better accuracy.\n\nXGBoost has an extensive catalog of hyperparameters which provides great flexibility to shape an algorithm\u2019s desired behavior. Here you will the optimize the model tuning which adds an L1 penalty (`reg_alpha`).\n\nUse a 5-fold cross-validation because your training data set is small.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In the cell below, create the XGBoost pipeline and set up the parameter grid for the search.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 19, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Create XGBoost pipeline, set up parameter grid.\nxgb_model_gs = XGBClassifier()\nparameters = {'reg_alpha': [0.0, 1.0], 'reg_lambda': [0.0, 1.0], 'n_estimators': [n_trees], 'seed': [1337]}"
        }, 
        {
            "source": "Use ``GridSearchCV`` to search for the best parameters over the parameters values that were specified in the previous section.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 21, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Fitting 5 folds for each of 4 candidates, totalling 20 fits\n"
                }, 
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed: 15.9min finished\n"
                }, 
                {
                    "execution_count": 21, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "GridSearchCV(cv=5, error_score='raise',\n       estimator=XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=100, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1),\n       fit_params={}, iid=True, n_jobs=1,\n       param_grid={'reg_alpha': [0.0, 1.0], 'reg_lambda': [0.0, 1.0], 'n_estimators': [30], 'seed': [1337]},\n       pre_dispatch='2*n_jobs', refit=True, scoring='accuracy', verbose=1)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "# Search for the best parameters.\nclf = GridSearchCV(xgb_model_gs, parameters, scoring='accuracy', cv=5, verbose=1, n_jobs=-1, refit=True)\nclf.fit(X_train, y_train)"
        }, 
        {
            "source": "From the grid scores, you can see the performance result of all parameter combinations including the best parameter combination based on model performance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 22, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 22, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "[mean: 0.94580, std: 0.01698, params: {'n_estimators': 30, 'reg_alpha': 0.0, 'reg_lambda': 0.0, 'seed': 1337},\n mean: 0.94851, std: 0.02311, params: {'n_estimators': 30, 'reg_alpha': 0.0, 'reg_lambda': 1.0, 'seed': 1337},\n mean: 0.95122, std: 0.02637, params: {'n_estimators': 30, 'reg_alpha': 1.0, 'reg_lambda': 0.0, 'seed': 1337},\n mean: 0.95122, std: 0.02637, params: {'n_estimators': 30, 'reg_alpha': 1.0, 'reg_lambda': 1.0, 'seed': 1337}]"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "# View the performance result.\nclf.grid_scores_"
        }, 
        {
            "source": "Display the accuracy estimated using cross-validation and the hyperparameter values for the best model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 23, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Best score: 95.1%\nBest parameter set: {'n_estimators': 30, 'reg_alpha': 1.0, 'reg_lambda': 0.0, 'seed': 1337}\n"
                }
            ], 
            "source": "print(\"Best score: %.1f%%\" % (clf.best_score_*100))\nprint(\"Best parameter set: %s\" % (clf.best_params_))"
        }, 
        {
            "source": "Display the accuracy of best parameter combination on the test set.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 24, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 96.0%\n"
                }, 
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "y_pred = clf.best_estimator_.predict(X_test, ntree_limit= n_trees)\n\naccuracy = accuracy_score(y_test, y_pred)\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "The accuracy on test set is about the same for tuned model as it is for the trained model that has default hyperparameters values, even though the selected hyperparameters are different to the default parameters.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### 3.2.3. Model with pipeline data preprocessing", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Here you learn how to use the XGBoost model within the scikit-learn pipeline. \n\nLet's start by importing the required objects.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 25, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn.pipeline import Pipeline\nfrom sklearn.decomposition import PCA"
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "pca = PCA(n_components=10)\nxgb_model_pca = XGBClassifier(n_estimators=n_trees)\npipeline = Pipeline(steps=[('pca', pca), ('xgb', xgb_model_pca)])"
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 27, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Pipeline(memory=None,\n     steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=10, random_state=None,\n  svd_solver='auto', tol=0.0, whiten=False)), ('xgb', XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=30, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1))])"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "pipeline.fit(X_train, y_train)"
        }, 
        {
            "source": "Now you are ready to evaluate accuracy of the model trained on the reduced set of features.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 95.5%\n"
                }, 
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "y_pred = pipeline.predict(X_test)\naccuracy = accuracy_score(y_test, y_pred)\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "You can see that this model has a similar accuracy to the model trained using default hyperparameter values.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Let's see how you can save your XGBoost pipeline using the WML service instance and deploy it for online scoring.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 4. Persist model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you learn how to use the Python client libraries to store your XGBoost model in the WML repository.\n\nFirst, you must import client libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting watson-machine-learning-client\n  Downloading watson_machine_learning_client-1.0.53-py3-none-any.whl (561kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 563kB 1.7MB/s eta 0:00:01\n\u001b[?25hRequirement not upgraded as not directly required: requests in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: tabulate in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: lomond in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: pandas in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: urllib3 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: tqdm in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: certifi in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement not upgraded as not directly required: idna<2.7,>=2.5 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement not upgraded as not directly required: six>=1.10.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement not upgraded as not directly required: python-dateutil>=2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement not upgraded as not directly required: pytz>=2011k in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement not upgraded as not directly required: numpy>=1.9.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nInstalling collected packages: watson-machine-learning-client\n  Found existing installation: watson-machine-learning-client 1.0.12\n    Uninstalling watson-machine-learning-client-1.0.12:\n      Successfully uninstalled watson-machine-learning-client-1.0.12\nSuccessfully installed watson-machine-learning-client-1.0.53\n"
                }
            ], 
            "source": "!pip install watson-machine-learning-client --upgrade"
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "source": "Authenticate to the Watson Machine Learning service on IBM Cloud.\n\n**Tip**: Authentication information (your credentials) can be found in the [Service Credentials](https://console.bluemix.net/docs/services/service_credentials.html#service_credentials) tab of the service instance that you created on IBM Cloud. <BR>If you cannot see the **instance_id** field in **Service Credentials**, click **New credential (+)** to generate new authentication information. \n\n**Action**: Enter your Watson Machine Learning service instance credentials here.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "#wml_credentials = {\n # \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  #\"access_key\": \"***\",\n  #\"username\": \"***\",\n  #\"password\": \"***\",\n  #\"instance_id\": \"***\"\n#}"
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "### 4.1. Save the XGBoost model to the WML Repository\n\nSave the model artifact as *XGBoost model for breast cancer* to your WML instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_props = {client.repository.ModelMetaNames.NAME: \"XGBoost model for breast cancer\"}\nmodel_details = client.repository.store_model(pipeline, model_props)"
        }, 
        {
            "source": "Get the saved model metadata from WML.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "# 5. Deploy and score in the IBM Cloud\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to use WML to create online scoring and score a new data record.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can list all stored models using the  `list_models` method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  -------------------------------  ------------------------  -----------------\nGUID                                  NAME                             CREATED                   FRAMEWORK\n6b76f1d7-aff2-445d-b816-0c4c573fbf7d  Product Line Prediction          2018-02-18T19:51:57.776Z  mllib-2.0\n4b7c9d27-6ca3-4b89-b726-9dd32ccb3958  Tumor_Type_Detection_v1_pipe     2018-03-18T23:37:40.678Z  scikit-learn-0.19\n7c60098c-5c16-47f8-89e6-f72a629a9bff  My cool mnist model              2018-03-19T09:42:42.204Z  tensorflow-1.5\n5999a2d9-e22c-424e-839f-d1f9c37ba8c0  XGBoost model for breast cancer  2018-03-29T14:00:39.597Z  scikit-learn-0.19\nbc1104f5-9bba-4bf4-a8cc-34eff218ae4a  XGBoost model for breast cancer  2018-04-03T10:05:17.385Z  scikit-learn-0.19\n5eb68ad4-1f39-4b1f-9687-0a071907aed0  XGBoost model for breast cancer  2018-04-05T11:33:31.059Z  scikit-learn-0.19\n950423e2-f164-4094-89af-44f005acaf77  XGBoost model for breast cancer  2018-04-05T12:27:33.111Z  scikit-learn-0.19\n------------------------------------  -------------------------------  ------------------------  -----------------\n"
                }
            ], 
            "source": "# Display a list of all the models.\nclient.repository.list_models()"
        }, 
        {
            "source": "You need the model uid to create the deployment. You can extract the model uid from the saved model details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "950423e2-f164-4094-89af-44f005acaf77\n"
                }
            ], 
            "source": "# Extract the uid.\nmodel_uid = client.repository.get_model_uid(model_details)\nprint(model_uid)"
        }, 
        {
            "source": "Use this modul_uid in the next section to create the deployment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.1. Create the model deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you can create a deployment, *Predict breast cancer*.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: 'd8143f44-220c-4a17-a1b1-0d3f61d5d013' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='d8143f44-220c-4a17-a1b1-0d3f61d5d013'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "# Create the deployment.\ndeployment_details = client.deployments.create(model_uid, 'Predict breast cancer')"
        }, 
        {
            "source": "Get a list of all deployments.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ----------------------  ------  --------------  ------------------------  -----------------\nGUID                                  NAME                    TYPE    STATE           CREATED                   FRAMEWORK\n206ce253-eaaa-40f0-9aac-bc0d25e3fe88  online                  online  DEPLOY_SUCCESS  2018-02-18T19:52:22.583Z  mllib-2.0\n40f24853-f991-4aa1-9936-830f197aa758  Mnist model deployment  online  DEPLOY_SUCCESS  2018-03-19T09:42:58.735Z  tensorflow-1.5\nd8143f44-220c-4a17-a1b1-0d3f61d5d013  Predict breast cancer   online  DEPLOY_SUCCESS  2018-04-05T12:27:51.539Z  scikit-learn-0.19\n------------------------------------  ----------------------  ------  --------------  ------------------------  -----------------\n"
                }
            ], 
            "source": "# List the deployments.\nclient.deployments.list()"
        }, 
        {
            "source": "The *Predict breast cancer model* has been successfully deployed.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.2. Perform prediction", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now, extract the url endpoint, *scoring_url*, which will be used to send scoring requests.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/950423e2-f164-4094-89af-44f005acaf77/deployments/d8143f44-220c-4a17-a1b1-0d3f61d5d013/online\n"
                }
            ], 
            "source": "# Extract endpoint url and display it.\nscoring_url = client.deployments.get_scoring_url(deployment_details)\nprint(scoring_url)"
        }, 
        {
            "source": "Prepare the scoring payload with the values to score.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'values': [[12.23, 19.56, 78.54, 461.0, 0.09586, 0.08087, 0.04187, 0.04107, 0.1979, 0.06013, 0.3534, 1.326, 2.308, 27.24, 0.007514, 0.01779, 0.01401, 0.0114, 0.01503, 0.003338, 14.44, 28.36, 92.15, 638.4, 0.1429, 0.2042, 0.1377, 0.108, 0.2668, 0.08174]]}\n"
                }
            ], 
            "source": "# Prepare scoring payload.\npayload_scoring = {\"values\": [X_test[0].tolist()]}\nprint(payload_scoring)"
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'fields': ['prediction', 'probability'], 'values': [[0.0, [0.9505169987678528, 0.04948299005627632]]]}\n"
                }
            ], 
            "source": "# Perform prediction and display the result.\nresponse_scoring = client.deployments.score(scoring_url, payload_scoring)\nprint(response_scoring)"
        }, 
        {
            "source": "**Result**: The patient record is classified as a benign tumor.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 6. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You successfully completed this notebook! \n\nYou learned how to use XGBoost machine learning as well as Watson Machine Learning to create and deploy a model. \n\nCheck out our [Online Documentation](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html) for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Data citations\n\nLichman, M. (2013). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Author\n\n**Wojciech Sobala** is a Data Scientist at IBM.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017, 2018 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}