From XGBoost model to scoring.ipynb

{
    "nbformat_minor": 2, 
    "cells": [
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Breast cancer detection with XGBoost and IBM Watson Machine Learning</b></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr>\n   <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://raw.githubusercontent.com/pmservice/wml-sample-notebooks/master/images/cancer_banner-06.png\" alt=\"Icon\" width=\"700\"> </th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to get data from the IBM Watson Data Platform Community, create a predictive model, and start scoring new data. This notebook introduces commands for getting and exploration data, model training. We show how to model persist model to repository, deployment and score using Watson Machine Learning service.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.5, XGBoost and scikit-learn.\n\nYou will use a publicly available data set, Breast Cancer Wisconsin (Diagnostic) Data Set to train XGBoost Model, which is short for \u201cExtreme Gradient Boosting\u201d, to classify breast cancer tumors (as benign or malignant) from 569 diagnostic images based on measurements such as radius, texture, perimeter and area.\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Load a CSV file into numpy array.\n-  Explore data.\n-  Prepare data for training and evaluation.\n-  Create an XGBoost machine learning model.\n-  Train and evaluate a model.\n-  Use cross-validation to optimize model's hyperparameters.\n-  Persist a model in Watson Machine Learning repository.\n-  Deploy a model for online scoring.\n-  Score sample data.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Setup](#setup)\n2.\t[Load and explore data](#load)\n3.\t[Create XGBoost model](#model)\n4.\t[Persist model](#persistence)\n5.\t[Deploy and score in a Cloud](#scoring)\n6.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Setup\n\nBefore you use the sample code in this notebook, you have to perform the following setup tasks:\n\n-  Create a [Watson Machine Learning Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered). \n-  Download **Breast Cancer Wisconsin (Diagnostic) Data Set** dataset from IBM Watson Data Platform. We provide code to download dataset below.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Note:** The Breast Cancer Wisconsin (Diagnostic) Data Set is a freely available data set on the Watson Data Platform Community [page](https://dataplatform.ibm.com/community?context=analytics).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 2. Load and explore data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will load the data as numpy array and perform a basic exploration.\n\nLoad the data to the numpy array by using `wget` to download the data to gpfs and then read data using `genfromtxt` method. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**: First, you need to install required packages. You can do it by running the following code. Run it only one time.<BR><BR>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement already up-to-date: wget in /home/dsxuser/.local/lib/python3.5/site-packages\r\n"
                }
            ], 
            "source": "!pip install wget --user --upgrade"
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "BreastCancerWisconsinDataSet.csv\n"
                }
            ], 
            "source": "import wget, os\n\nWisconsinDataSet = 'BreastCancerWisconsinDataSet.csv' \nif not os.path.isfile(WisconsinDataSet):\n    link_to_data = 'https://apsportal.ibm.com/exchange-api/v1/entries/c173693bf48aeb22e41bbe2b41d79c1f/data?accessKey=941eec501eadcdceb5abd25cf7c029d5'\n    WisconsinDataSet = wget.download(link_to_data)\n\nprint(WisconsinDataSet)"
        }, 
        {
            "source": "The csv file **BreastCancerWisconsinDataSet.csv** is availble on gpfs now. Load the file to numpy array using code below.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting numpy\n  Downloading numpy-1.14.2-cp35-cp35m-manylinux1_x86_64.whl (12.1MB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12.1MB 70kB/s  eta 0:00:01\n\u001b[?25hInstalling collected packages: numpy\nSuccessfully installed numpy-1.14.2\n"
                }
            ], 
            "source": "!pip install numpy --user --upgrade"
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import numpy as np\n\nnp_data = np.genfromtxt(WisconsinDataSet, delimiter=',', names=True, dtype=None, encoding='utf-8')"
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 5, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "array([(  842302, 'M', 17.99, 10.38, 122.8, 1001., 0.1184 , 0.2776 , 0.3001, 0.1471 , 0.2419, 0.07871, 1.095 , 0.9053, 8.589, 153.4 , 0.006399, 0.04904, 0.05373, 0.01587, 0.03003, 0.006193, 25.38, 17.33, 184.6, 2019., 0.1622, 0.6656, 0.7119, 0.2654, 0.4601, 0.1189 ),\n       (  842517, 'M', 20.57, 17.77, 132.9, 1326., 0.08474, 0.07864, 0.0869, 0.07017, 0.1812, 0.05667, 0.5435, 0.7339, 3.398,  74.08, 0.005225, 0.01308, 0.0186 , 0.0134 , 0.01389, 0.003532, 24.99, 23.41, 158.8, 1956., 0.1238, 0.1866, 0.2416, 0.186 , 0.275 , 0.08902),\n       (84300903, 'M', 19.69, 21.25, 130. , 1203., 0.1096 , 0.1599 , 0.1974, 0.1279 , 0.2069, 0.05999, 0.7456, 0.7869, 4.585,  94.03, 0.00615 , 0.04006, 0.03832, 0.02058, 0.0225 , 0.004571, 23.57, 25.53, 152.5, 1709., 0.1444, 0.4245, 0.4504, 0.243 , 0.3613, 0.08758)],\n      dtype=[('id', '<i8'), ('diagnosis', '<U1'), ('radius_mean', '<f8'), ('texture_mean', '<f8'), ('perimeter_mean', '<f8'), ('area_mean', '<f8'), ('smoothness_mean', '<f8'), ('compactness_mean', '<f8'), ('concavity_mean', '<f8'), ('concave_points_mean', '<f8'), ('symmetry_mean', '<f8'), ('fractal_dimension_mean', '<f8'), ('radius_se', '<f8'), ('texture_se', '<f8'), ('perimeter_se', '<f8'), ('area_se', '<f8'), ('smoothness_se', '<f8'), ('compactness_se', '<f8'), ('concavity_se', '<f8'), ('concave_points_se', '<f8'), ('symmetry_se', '<f8'), ('fractal_dimension_se', '<f8'), ('radius_worst', '<f8'), ('texture_worst', '<f8'), ('perimeter_worst', '<f8'), ('area_worst', '<f8'), ('smoothness_worst', '<f8'), ('compactness_worst', '<f8'), ('concavity_worst', '<f8'), ('concave_points_worst', '<f8'), ('symmetry_worst', '<f8'), ('fractal_dimension_worst', '<f8')])"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "np_data[:3]"
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of rows: 569\nNumber of columns: 32\n"
                }
            ], 
            "source": "print('Number of rows: {}'.format(np_data.size))\nprint('Number of columns: {}'.format(len(np_data[0])))"
        }, 
        {
            "source": "We can see that dataset has 569 records and 32 features.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "We can generate basic information about features in our dataset.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": " 0. id: int64\n 1. diagnosis: <U1\n 2. radius_mean: float64\n 3. texture_mean: float64\n 4. perimeter_mean: float64\n 5. area_mean: float64\n 6. smoothness_mean: float64\n 7. compactness_mean: float64\n 8. concavity_mean: float64\n 9. concave_points_mean: float64\n10. symmetry_mean: float64\n11. fractal_dimension_mean: float64\n12. radius_se: float64\n13. texture_se: float64\n14. perimeter_se: float64\n15. area_se: float64\n16. smoothness_se: float64\n17. compactness_se: float64\n18. concavity_se: float64\n19. concave_points_se: float64\n20. symmetry_se: float64\n21. fractal_dimension_se: float64\n22. radius_worst: float64\n23. texture_worst: float64\n24. perimeter_worst: float64\n25. area_worst: float64\n26. smoothness_worst: float64\n27. compactness_worst: float64\n28. concavity_worst: float64\n29. concave_points_worst: float64\n30. symmetry_worst: float64\n31. fractal_dimension_worst: float64\n"
                }
            ], 
            "source": "for i, col in enumerate(list(np_data.dtype.names)):\n    print('{:2d}. {}: {}'.format(i, col, np_data.dtype.fields[col][0]))"
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 3. Create XGBoost model\n\nIn recent years, ensemble learning models took the lead and became popular among machine learning practitioners.\n\nXGBoost classifier makes its predictions based on the majority vote from collection of models which are a set of classification trees. It uses the combination of weak learners to create single strong learner. It\u2019s a sequential training process, new learners focus on the misclassified examples of previous learners.\n\nXGBoost is an open source library for ensemble based algorithms.\n\nIn this section you will learn how to train and test a XGBoost model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1: Prepare data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "After looking at description of features in our dataset, we can prepare our data for model building. We will use \"diagnosis\" columns as our target variable and remove id variable from set of predictors.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "y = 1.0*(np_data['diagnosis'] == 'M')\nX = np.array([list(r)[2:] for r in np_data])"
        }, 
        {
            "source": "We split the dataset into: train and test datasets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
                }
            ], 
            "source": "from sklearn.cross_validation import train_test_split \n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.35, random_state=143)"
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of training records: 369\nNumber of testing records : 200\n"
                }
            ], 
            "source": "print(\"Number of training records: \" + str(X_train.shape[0]))\nprint(\"Number of testing records : \" + str(X_test.shape[0]))"
        }, 
        {
            "source": "As you can see our data has been successfully split into two datasets:\n- The train data set, which is the largest group, is used for training.\n- The test data set will be used for model evaluation and is used to test the assumptions of the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2: Create XGBoost model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "We start with importing necessary libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n  DeprecationWarning)\n"
                }
            ], 
            "source": "from xgboost.sklearn import XGBClassifier\nfrom sklearn.grid_search import GridSearchCV\nfrom sklearn.metrics import accuracy_score"
        }, 
        {
            "source": "### 3.2.1: Model with default hyperparameters", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section we create XGBoost classifier with default values for hyperparameters. In next sections we will show how to improve this model base model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "xgb_model = XGBClassifier()"
        }, 
        {
            "source": "We can see the default parameters of the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 13, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=100, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "xgb_model"
        }, 
        {
            "source": "Once we have our XGBoost classifier is set up, we can train it by invoking fit method. We will also evaluate model during training process on train and test data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[0]\tvalidation_0-error:0.02168\tvalidation_1-error:0.05\n[1]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[2]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[3]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[4]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[5]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[6]\tvalidation_0-error:0.01897\tvalidation_1-error:0.05\n[7]\tvalidation_0-error:0.01626\tvalidation_1-error:0.05\n[8]\tvalidation_0-error:0.01355\tvalidation_1-error:0.05\n[9]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[10]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[11]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[12]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[13]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[14]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[15]\tvalidation_0-error:0.00813\tvalidation_1-error:0.05\n[16]\tvalidation_0-error:0.01084\tvalidation_1-error:0.05\n[17]\tvalidation_0-error:0.01084\tvalidation_1-error:0.04\n[18]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[19]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[20]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[21]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[22]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[23]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[24]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[25]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[26]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[27]\tvalidation_0-error:0.00813\tvalidation_1-error:0.04\n[28]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[29]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[30]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[31]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[32]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[33]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[34]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[35]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[36]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[37]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[38]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[39]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[40]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[41]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[42]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[43]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[44]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[45]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[46]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[47]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[48]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[49]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[50]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[51]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[52]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[53]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[54]\tvalidation_0-error:0.00542\tvalidation_1-error:0.04\n[55]\tvalidation_0-error:0.00542\tvalidation_1-error:0.045\n[56]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04\n[57]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04\n[58]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[59]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[60]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05\n[61]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[62]\tvalidation_0-error:0.00271\tvalidation_1-error:0.045\n[63]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05\n[64]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[65]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[66]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[67]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[68]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[69]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[70]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[71]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[72]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[73]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[74]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[75]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[76]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[77]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[78]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[79]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[80]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[81]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[82]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[83]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[84]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[85]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[86]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[87]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[88]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[89]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[90]\tvalidation_0-error:0\tvalidation_1-error:0.05\n[91]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[92]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[93]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[94]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[95]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[96]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[97]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[98]\tvalidation_0-error:0\tvalidation_1-error:0.055\n[99]\tvalidation_0-error:0\tvalidation_1-error:0.055\n"
                }, 
                {
                    "execution_count": 14, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=100, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "xgb_model.fit(X_train, y_train, eval_metric=['error'], eval_set=[((X_train, y_train)),(X_test, y_test)])"
        }, 
        {
            "source": "We plot the model performance evaluated during training process to assess model overfitting.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from matplotlib import pyplot\n\n%matplotlib inline"
        }, 
        {
            "execution_count": 19, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 19, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Text(0.5,0,'Number of iterations')"
                    }, 
                    "output_type": "execute_result"
                }, 
                {
                    "output_type": "display_data", 
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAGDCAYAAAAGSkjRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8XOV97/HPT5vlFdsjsXlfZMBsBoSJA8GEEAI0hQRIArdJWJK6uUlaunBb57bNQpqGNEsbAg2XBhNoUgiXtLnkFi4QgmUCBWz2xUiyDcbCBo/kFcu2PNLv/nHO2OPxSDOSZtGZ+b5fL70055xnzjwzDP7qOc9znsfcHRERERn5qkpdAREREcmNQltERCQiFNoiIiIRodAWERGJCIW2iIhIRCi0RUREIkKhLSIiEhEKbZGIMLPlZrbVzEaVui4iUhoKbZEIMLOZwAcABy4u4uvWFOu1RCQ7hbZINHwWeAr4KXBVcqeZjTaz75vZejPbbma/M7PR4bGzzOxJM9tmZhvM7Opw/3Iz+3zKOa42s9+lbLuZfcnM2oH2cN8Pw3PsMLNnzewDKeWrzex/mtlaM9sZHp9mZreY2fdT34SZ/drM/rQQH5BIJVBoi0TDZ4Gfhz8fMbMjwv3fA04D3g9MBv4S6DOz6cCDwI+ARmAB8MIgXu9jwBnA/HB7ZXiOycC/Af/bzOrDY38OXAlcBEwArgW6gTuBK82sCsDMGoAPAXcP5o2LyAEKbZERzszOAmYA97r7s8Ba4L+FYXgtcJ27v+3uve7+pLvvBf4A+I273+3u+9y9y90HE9rfdvct7r4bwN1/Fp4j4e7fB0YBx4RlPw/8jbu3euDFsOwzwHaCoAa4Alju7u8O8yMRqVgKbZGR7yrgYXfvDLf/LdzXANQThHi6af3sz9WG1A0z+wszWx1egt8GHBa+frbXuhP4dPj408C/DqNOIhVPg0xERrCwf/qTQLWZvRPuHgVMBI4C9gBzgBfTnroBWNjPaXcBY1K2j8xQZv/yf2H/9V8RtJhfdfc+M9sKWMprzQFeyXCenwGvmNnJwHHAr/qpk4jkQC1tkZHtY0AvQd/ygvDnOOBxgn7uZcAPzOzocEDYovCWsJ8D55nZJ82sxsxiZrYgPOcLwKVmNsbM5gKfy1KH8UACiAM1ZvZVgr7rpJ8A3zSzJgucZGYxAHfvIOgP/1fgl8nL7SIyNAptkZHtKuAOd3/L3d9J/gA3E/RbLwVeJgjGLcB3gCp3f4tgYNhfhPtfAE4Oz/mPQA/wLsHl659nqcNDBIPa2oD1BK371MvnPwDuBR4GdgC3A6NTjt8JnIgujYsMm7l79lIiIkNkZmcTXCaf6e59pa6PSJSppS0iBWNmtcB1wE8U2CLDp9AWkYIws+OAbQQD5v6pxNURKQu6PC4iIhIRammLiIhEhEJbREQkIkbc5CoNDQ0+c+bMUldDRESkaJ599tlOd2/MVm7EhfbMmTNZtWpVqashIiJSNGa2PpdyujwuIiISEQptERGRiFBoi4iIRMSI69POZN++fXR0dLBnz55SV6Vo6uvrmTp1KrW1taWuioiIjBCRCO2Ojg7Gjx/PzJkzMbPsT4g4d6erq4uOjg5mzZpV6uqIiMgIEYnL43v27CEWi1VEYAOYGbFYrKKuLIiISHaRCG2gYgI7qdLer4iIZBeZ0C6lrq4uFixYwIIFCzjyyCOZMmXK/u2enp6cznHNNdfQ2tpa4JqKiEg5i0SfdqnFYjFeeOEFAL7+9a8zbtw4rr/++oPKuDvuTlVV5r+D7rjjjoLXU0REylvWlraZLTOzzWb2Sj/HzcxuMrM1ZvaSmZ2acuwqM2sPf67KZ8VHgjVr1nDCCSfwhS98gVNPPZVNmzaxZMkSmpubOf7447nhhhv2lz3rrLN44YUXSCQSTJw4kaVLl3LyySezaNEiNm/eXMJ3ISIiUZFLS/unwM3AXf0cvxBoCn/OAH4MnGFmk4GvAc2AA8+a2f3uvnU4Ff7Gr1/ltY07hnOKQ8w/egJf+/3jh/Tc1157jTvuuINbb70VgBtvvJHJkyeTSCT44Ac/yOWXX878+fMPes727dtZvHgxN954I3/+53/OsmXLWLp06bDfh4iIlLesoe3uK8xs5gBFLgHu8mBh7qfMbKKZHQWcAzzi7lsAzOwR4ALg7uFWeiSZM2cOp59++v7tu+++m9tvv51EIsHGjRt57bXXDgnt0aNHc+GFFwJw2mmn8fjjjxe1ziKVbsuuHl7s2FbqakiEnXD0YTSOH1X0181Hn/YUYEPKdke4r7/9hzCzJcASgOnTpw/4YkNtERfK2LFj9z9ub2/nhz/8Ic888wwTJ07k05/+dMbbturq6vY/rq6uJpFIFKWuIhL4y/te4jer3y11NSTCbvvMaZx//JFFf918hHame5N8gP2H7nS/DbgNoLm5OWOZKNixYwfjx49nwoQJbNq0iYceeogLLrig1NUSkRR7E708saaTSxYczdXvn1nq6khEzW4YV5LXzUdodwDTUranAhvD/eek7V+eh9cbsU499VTmz5/PCSecwOzZsznzzDNLXSURSbPqza3s3tfLJQuO5pTpk0pdHZFBsaArOkuhoE/7/7r7CRmO/R7wZeAigoFoN7n7wnAg2rNAcjT5c8BpyT7u/jQ3N3v6etqrV6/muOOOy1rPclOp71ukkP7+gdX89Ik3eeFrH2ZMne56lZHBzJ519+Zs5bJ+Y83sboIWc4OZdRCMCK8FcPdbgQcIAnsN0A1cEx7bYmbfBFaGp7ohW2CLiBRaS2uc02dNUmBLJOUyevzKLMcd+FI/x5YBy4ZWNRGR/Nq0fTet7+7kstOOLXVVRIZE05iKSMVY0RYHYPG8w0tcE5GhUWiLSMVY0dbJkRPqmXdEaUb+igyXQltEKkKit4/H2+MsnteoVfQkshTaIlIRXuzYxo49Cc6e11jqqogMmUI7B/lYmhNg2bJlvPPOOwWsqYj0p6U1TpXBWXMbSl0VkSHTPQ85yGVpzlwsW7aMU089lSOPLP7UdyKVrqUtzinTJ3HYmNpSV0VkyNTSHqY777yThQsXsmDBAr74xS/S19dHIpHgM5/5DCeeeCInnHACN910E7/4xS944YUX+NSnPjXoFrqIDM+WXT289PZ2FuvSuERc9FraDy6Fd17O7zmPPBEuvHHQT3vllVf4j//4D5588klqampYsmQJ99xzD3PmzKGzs5OXXw7quW3bNiZOnMiPfvQjbr75ZhYsWJDf+ovIgB5vj+OOQlsiL3qhPYL85je/YeXKlTQ3BzPP7d69m2nTpvGRj3yE1tZWrrvuOi666CLOP//8EtdUZORYvWkHN/92Db19xVsbqPXdnUweW8eJUw4r2muKFEL0QnsILeJCcXeuvfZavvnNbx5y7KWXXuLBBx/kpptu4pe//CW33XZbCWooMvLc+eSbPPLau8xqGJu9cJ7UVVfx+Q/MoqpKt3pJtEUvtEeQ8847j8svv5zrrruOhoYGurq62LVrF6NHj6a+vp5PfOITzJo1iy984QsAjB8/np07d5a41iKl4+60tMX50HGH8+NPn1bq6ohEjkJ7GE488US+9rWvcd5559HX10dtbS233nor1dXVfO5zn8PdMTO+853vAHDNNdfw+c9/ntGjR/PMM89QV1dX4ncgUlztm99j0/Y9XPch9S2LDEVOS3MWk5bmPKBS37eUr588vo6/+8/VPLn0XI6eOLrU1REZMXJdmlO3fIlI0bS0xWk6fJwCW2SIFNoiUhTdPQmeXrdFt12JDINCW0SK4ul1W+jp7WPxMQptkaGKTGiPtL73Qqu09yvlr6UtTn1tFafPnFzqqohEViRCu76+nq6urooJMnenq6uL+vr6UldFJG9a2uIsmh2jvra61FURiaxI3PI1depUOjo6iMfjpa5K0dTX1zN16tRSV0MkL9Z37eKNzl1ctWhGqasiEmmRCO3a2lpmzZpV6mqIyBCtaAv+4NZa1iLDE4nL4yISbS1tcaZNHl3UqUtFypFCW0QKqifRx5Nru1g8rxEzzf0tMhyRuDwu0efu7NidGLDMqNqqAQcp7e7ppSfRl++qSYGtWr+F7p5eFs87vNRVEYk8hbYUxd/952pu/90bA5aZUF/D4391LoeNrj3k2Ovv7OCjN/2ORBGXc5T8qa02Fs2JlboaIpGn0JaCc3f+86VNLJg2kYtPPjpjmc0793Jry1qeWNPJRScedcjxh155l153/vqi46jW8oqRM+fwcYwbpX9uRIZL/xdJwbVvfo93duzhT89r4oqF0zOWSfT28fOn17OiLZ4xtFe0xzlp6kT+8OzZha6uiMiIpYFoUnAtrdlv96mpruKsuQ20tMUPmURne/c+nn9rq+asFpGKp9CWgmtpizPviOwrOy2e18im7Xto3/zeQft/t6aTPkehLSIVT6EtBdXdk+CZN3Jb2SnZEk+2zJNa2jYzob6Gk6ceVpA6iohEhUJbCuqpdV3Byk453O5z9MTRNB0+jhXtB0Lb3Wlpi/OBpkZqqvV1FZHKpn8FpaBaWuOMrq2meeaknMovntfI0+u20N0T3NPd+u5O3t2xV5fGRURQaEuBrWjv5H2zJ+e8stPiYxrp6e3j6XVbgNwGsYmIVAqFthRMcmWnwbSST585mfraKlrCBSZa2uIce+R4jjxMy5SKiCi0pWCSKzstPib36Svra6tZNDtGS1ucXXsTrHpTt3qJiCQptKVgWtriTJ88hpmxMYN63tnzGnmjcxf3rtpAT2+fLo2LiIQU2lIQexO9Q17ZKdmy/uGj7YMaxCYiUu4U2lIQz765NVzZafCt5FkNY5k2eTTbuvfx/jkxRtXkNohNRKTcae7xMrR5xx5ef2dnzpeVE7193L1yA917B146czCeXNs15JWdzIzF8xr52VNvsfgYXRoXEUlSaJehHzzSxr2rNvDs33yYSWPrspZ/9PXN/O2vXsl7PT5y/BGMHeLKTr9/0tH8+sVNfOi4I/JcKxGR6FJolxl3Z3lrnD6Hx9d09rsUZqqWtjjjRtXw5FfOpSaPy16OzvHe7EzOmB3jxa+dn7e6iIiUA4V2mUkugwnBLVfZQtvdWdEW5/1zYkyory1GFUVEZIg0EK3MJGcQO33mpIzLXKZb17mLjq271XcsIhIBCu0y09IW55gjxvPJ5mnEd+5l9aadA5dPThPapNAWERnpFNplZP8ymMc07r/VKjkdaH9a2uLMaRzLtMmDmwBFRESKT6FdRg4sg9nI4RPqOe6oCbS0be63/J59vTy1rkszjomIREROoW1mF5hZq5mtMbOlGY7PMLNHzewlM1tuZlNTjn3HzF4Jfz6Vz8rLwdKXwVw8r5FVb27lvX7uv376jS3sTfRpbm8RkYjIGtpmVg3cAlwIzAeuNLP5acW+B9zl7icBNwDfDp/7e8CpwALgDOB/mNmE/FVfUq1o72RRygxii+c1kuhz/mttV+bybXFG1VTxvtmDnwBFRESKL5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePpRyfD7S4e8LddwEvAhcMv9qSLtMymKfNmMTYuup+L5G3tMU5Y3Ys57WuRUSktHIJ7SnAhpTtjnBfqheBy8LHHwfGm1ks3H+hmY0xswbgg8C09BcwsyVmtsrMVsXjAw+cksz2L4OZEtp1NVUsmtPA8tZDb/3q2NrNms3v6dK4iEiE5BLamabISr/593pgsZk9DywG3gYS7v4w8ADwJHA38F/AIR2s7n6buze7e3Njo0JkKFra4syIjWFmw9iD9i8+ppGOrbt5o3PXQftXtHUGx+c1FK2OIiIyPLmEdgcHt46nAhtTC7j7Rne/1N1PAf463Lc9/P0td1/g7h8m+AOgPS81l/16En08ubYr473Wi5sy3/rV0raZKRNHM6dxXFHqKCIiw5dLaK8EmsxslpnVAVcA96cWMLMGM0ue6yvAsnB/dXiZHDM7CTgJeDhflZfAqvVb+l0Gc3psDLMbxu6/fA6wr7ePJ9cEt3oNdq1rEREpnaxzj7t7wsy+DDwEVAPL3P1VM7sBWOXu9wPnAN82MwdWAF8Kn14LPB4Gww7g0+6ev/UfI2jX3gRX3/EMXbt68nbOHbv3DbgM5tnzGvnZU+s59/vLAUj0Ojv3JtSfLSISMTktGOLuDxD0Tafu+2rK4/uA+zI8bw/BCHIJPd7eyco3t3LOMY2MG+KylZmcOn1Sv8tgfmbRDLZ195DoOzAU4aymBs7RfOMiIpGiVb6KLLkM5m2faaaupjgT0s1pHMc/XXFKUV5LREQKR9OYFlHqMpjFCmwRESkfSo4iWhvfxdvbtAymiIgMjUK7iJK3XWkZTBERGQqFdhG1tMWZrWUwRURkiBTaRbJnXy9Pr+vSbVYiIjJkCu0i0TKYIiIyXArtImlp1TKYIiIyPArtImlp26xlMEVEZFgU2kWwYUs3a+O7OLtJK2qJiMjQKbSLYEV7cKuXpg0VEZHhUGgXwYq2uJbBFBGRYVNoF9i+3j6e0DKYIiKSBwrtAntu/Vbe25tg8Tz1Z4uIyPAotAuspS1OTZXx/rkKbRERGR6FdoG1tMU5dfokJtTXlroqIiIScQrtAorv3MurG3doVS8REckLhXYBPR7e6qWpS0VEJB8U2gXU0hanYVwd84+aUOqqiIhIGVBoF0hvn7OiLc7ZTY1UVelWLxERGT6FdoG88vZ2tnbvU3+2iIjkjUK7QFa0xTGDs3Srl4iI5IlCu0Ba2uKcNOUwYuNGlboqIiJSJhTaBbC9ex/PvbWVszVqXERE8kihXQBPrO2kz3Wrl4iI5JdCuwBaWuOMr69hwbSJpa6KiIiUEYV2nrk7LW1xPtDUQE21Pl4REckfpUqetW9+j3d27NGlcRERyTuFdp61tAZTl2oQmoiI5JtCO89a2uIcc8R4jjpsdKmrIiIiZUahnUfdPQmeeWMLZ8/ThCoiIpJ/Cu08empdFz29fSyed3ipqyIiImVIoZ1HLa1xRtdW0zxzUqmrIiIiZUihnUctbXEWzYlRX1td6qqIiEgZUmjnyfquXbzZ1a1bvUREpGAU2nmyok23eomISGEptPOkpS3O9MljmBkbU+qqiIhImVJo58HeRC9Pru1i8bxGzKzU1RERkTKl0M6DZ9/cSndPr/qzRUSkoBTaedDSFqe22lg0J1bqqoiISBlTaOdBS1uc5hmTGTuqptRVERGRMlZxKePu9PZ53s737s69vP7OTpZeeGzezikiIpJJRYW2u3PxzU/w8tvb837us5vUny0iIoVVUaG9Nr6Ll9/ezu+ddBTHHjE+b+c9fMIojjsqf+cTERHJpKJCuyWcAGXpBccybbLupxYRkWjJaSCamV1gZq1mtsbMlmY4PsPMHjWzl8xsuZlNTTn2D2b2qpmtNrObrIQ3Mre0xZndOFaBLSIikZQ1tM2sGrgFuBCYD1xpZvPTin0PuMvdTwJuAL4dPvf9wJnAScAJwOnA4rzVfhD27Ovl6XVdupdaREQiK5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePpRx3oB6oA0YBtcC7w630UDz9xhb2JvoU2iIiElm5hPYUYEPKdke4L9WLwGXh448D480s5u7/RRDim8Kfh9x9dfoLmNkSM1tlZqvi8fhg30NOWlrjjKqp4n2zNQGKiIhEUy6hnakPOv1G5+uBxWb2PMHl77eBhJnNBY4DphIE/blmdvYhJ3O/zd2b3b25sbEwLeGWts2cMVtrXYuISHTlEtodwLSU7anAxtQC7r7R3S9191OAvw73bSdodT/l7u+5+3vAg8D78lLzQdiwpZu18V2c3dRQ7JcWERHJm1xCeyXQZGazzKwOuAK4P7WAmTWYWfJcXwGWhY/fImiB15hZLUEr/JDL44W2oj245H7OMerPFhGR6Moa2u6eAL4MPEQQuPe6+6tmdoOZXRwWOwdoNbM24AjgW+H++4C1wMsE/d4vuvuv8/sWslvRFmfKxNHMaRxX7JcWERHJm5wmV3H3B4AH0vZ9NeXxfQQBnf68XuCPhlnHYdnX28cTa7r4/ZOP1lrXIiISaWW/ytdz67fy3t6EbvUSEZHIK/vQbmmLU11lvH+ubvUSEZFoq4jQPm36JCbU15a6KiIiIsNS1qG9vXsfr7+zk8UaNS4iImWgrFf5OmxMLc/+zXl4+lQwIiIiEVTWoQ0wcUxdqasgIiKSF2V9eVxERKScKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGRU2ib2QVm1mpma8xsaYbjM8zsUTN7ycyWm9nUcP8HzeyFlJ89ZvaxfL8JERGRSpA1tM2sGrgFuBCYD1xpZvPTin0PuMvdTwJuAL4N4O6PufsCd18AnAt0Aw/nsf4iIiIVI5eW9kJgjbuvc/ce4B7gkrQy84FHw8ePZTgOcDnwoLt3D7WyIiIilSyX0J4CbEjZ7gj3pXoRuCx8/HFgvJnF0spcAdyd6QXMbImZrTKzVfF4PIcqiYiIVJ5cQtsy7PO07euBxWb2PLAYeBtI7D+B2VHAicBDmV7A3W9z92Z3b25sbMyp4iIiIpWmJocyHcC0lO2pwMbUAu6+EbgUwMzGAZe5+/aUIp8E/sPd9w2vuiIiIpUrl5b2SqDJzGaZWR3BZe77UwuYWYOZJc/1FWBZ2jmupJ9L4yIiIpKbrKHt7gngywSXtlcD97r7q2Z2g5ldHBY7B2g1szbgCOBbyeeb2UyClnpLXmsuIiJSYcw9vXu6tJqbm33VqlWlroaIiEjRmNmz7t6crZxmRBMREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiMgptM3sAjNrNbM1ZrY0w/EZZvaomb1kZsvNbGrKselm9rCZrTaz18xsZv6qLyIiUjmyhraZVQO3ABcC84ErzWx+WrHvAXe5+0nADcC3U47dBXzX3Y8DFgKb81FxERGRSlOTQ5mFwBp3XwdgZvcAlwCvpZSZD/xZ+Pgx4Fdh2flAjbs/AuDu7+Wp3jIc78Xh0W9AYm+pazLyzFgEzdcW5tzu8Lt/hPmXQGxO5jLP3glv/q4wr19MM8+C067KfCzeBr/7AfT1Htg3ZjKc/3dQXVuc+olEVC6hPQXYkLLdAZyRVuZF4DLgh8DHgfFmFgPmAdvM7N+BWcBvgKXu3pv6ZDNbAiwBmD59+hDehgzKusfg+X+Fw6ZBVS5fgQqxewu0PwSnXQNm+T//9o7gj6W9O+C8r2cu8+g3oDcRhFhUdW+Btb/tP7Rf/Dd48R6YNDPYTuyBnZvgxE/A1OaiVVMkinL5FzvTv16etn09cLOZXQ2sAN4GEuH5PwCcArwF/AK4Grj9oJO53wbcBtDc3Jx+bsm37q7g9x+tiHY45Nt//TM89JXg8xnbkP/zd60Jfne2Zz7evSV47Q9/E878k/y/frE8cRM88reweyuMnnTo8c52aGiCL68MtuNtcMvpweej0BYZUC4D0TqAaSnbU4GNqQXcfaO7X+rupwB/He7bHj73eXdf5+4Jgsvmp+al5jJ03VvAqqB+YqlrMrI0NAW/+wvV4UqGdvL3IcfXHlyPqNr/Ofb3PtdALOU9TpoJVl24z12kjOQS2iuBJjObZWZ1wBXA/akFzKzBzJLn+gqwLOW5k8ysMdw+l4P7wqUUuruCFlCV7vg7SGxu8LurQOGRDKUt6w7uz01Kvm4s4qGdrH+mP076eoP33zD3wL6auiC4C/W5i5SRrP9qhy3kLwMPAauBe939VTO7wcwuDoudA7SaWRtwBPCt8Lm9BJfOHzWzlwkutf9L3t+FDM7uLTBal8UPMXE6VI8qYEs7PG9vD2x769Djne3BGINJMwrz+sUyaUbwPjKF8La3gvef/odJQ1P/LXMR2S+nUUju/gDwQNq+r6Y8vg+4r5/nPgKcNIw6Sr51d8GYWKlrMfJUVcPk2f1fvh6uzjVw2HTY/lbwGpNnHXy8qx0mzYr+COrq2uB9ZPrjJ/nZpncBxObCuuXQ16crQCID0P8dlah7qwag9adhbmFa2vt2w/YNcMyFwXam1+hcE/3+7KSGpsx//HT20wXQ0BSMIt++4dDniMh+Cu1K1N2l0O5PbC5sfQN69+X3vF1rAYdpC4MBgOmXjpN9vbG5GZ8eObE5wXtO77vvag/ef/r3r9DjCUTKhEK70rirT3sgsSboS8DW9fk9bzKMGprC/tu0cNr2FvTuLZ+WdqwpeD/pLefk7V7p98HHsow4FxFAoV159nUHlyHV0s4sGZr5bvElwyg2Nwio9EvHye2ojxxP6u+2r/TbvZLGHQ6jJqilLZKFQrvSdG8JfmsgWmbJy7T57tfuaocJU6BubHDpeOcm2LvzwPHOlJZ4OYhl+ONn787gfTdk6AIwCz573astMiCFdqVJzoamy+OZjZkc/EGT95Z2+4E/CBoy3Mfc1Q71h5XPH1NjG4L3kxrC2a4m9Dd4TUT2U2hXmt1qaWcVazowO1k+uAfnS4Z1pv7b5GXjQswluPr2AAAT50lEQVR5Xgpm4eeYEtqd/dzulRRrgh1vQ8+uwtdPJKIU2pVm/+VxtbT7le/bvnbFYe/2A2E9eTZghwZauVwaT0qfMKWrPZg+d/LsfsonR5Dn8Q8mkTKj0K406tPOLtYEuzbDnu35Od/+/uowlGrrg9nXkvv3vgc7N5bP7V5JsbnB+9obrsjb2R6875pR/ZQv0CBAkTKi0K40yT5tLRbSv2wLXgxWpjnFG1IuHfc3S1jUpffdd7UPPDo+Ngcw3fYlMgCFdqXZvSUI7Gqto92vfLf4Otuhpj5Yvzz1NbrWBtN2ltvtXkmpC4f09R3cr59J7ejgM1JLW6RfCu1K071F/dnZ5HupyK41MHnOwXNqN8wN7pnfuTF8Heu/rzeqkn33ne3B+9zXnb0LoFDTyIqUCYV2pdFiIdnV1AUrVeWzpR2bc/C+/SPI24PXmTg96OsuJ7X1MDFsOe+fczxLaMfmBn/kuBe+fiIRpNCuNJrCNDexPC0VmeiBrW8eelk4tb83ObVnOYqFU7bm2m8fa4Ke92DnO4Wvm0gEKbQrTfcWtbRz0dAEW8I+5+HY+iZ476H91eOPgrpx0NkW9PWWW392UkPYd9/ZFrzf8UdlKa+FQ0QGotCuNOrTzk1sbn6WikxdKCSVWXDJ/I3HYd+uzFN7loPY3OD9vfF48H6zTR6T2m0gIodQaFeSfXuCf0AV2tnla+GQgfpyY00QX33gcTlKfo7x1bm9xwlToGa0pjMV6YdCu5IkpzBVn3Z2+VoqsqsdxjbC6Az3xTc0ZX5cTtLvTc+mqkoLh4gMQKFdSZITq6ilnd3+pSKHG9oD9FcnW9+59PVG1YSjoXZs8DjXGd8a5qqlLdIPhXYl0RSmuUsuFZmPy+P99VfvX0Akh77eqEr23UPuVxNiTbBtPST2Fq5eIhGlabEqiZblHJyGJnj9P+Hu/za053sfdHdmb2mXa392UkMTvPPSIFraTcFnd/cVQf82BPfOn/93cNjUzM/57bfg3VcP3nfa1TDv/CFXW2QkUmhXEi3LOTjHXwqbV8O2t4Z+jimnwdzzMh+rGwsLl/R/vFyc9KngD8W6sbmVn3kWTDsD3osH294Hm1+FGWfCwj88tHzPLljxD0EXw5iGYN+WddC7V6EtZUehXUmSl8dHTyptPaLimAuCn0K66LuFPf9IMO8jwU+uJhwNn3v4wLY7fHtq//3cyaU8L7gRjv9Y8Pi+a6Fj1dDqKzKCqU+7knRvCQZX1dSVuiYiuUuOL+hvRHmme+FjTcEVkn17Cl8/kSJSaFeS7i61siWaUpcyTde5hkMWXGloAjy4TC5SRhTalWS3pjCViIo1wbYNsG/3oce62oOFSWpHp5TXdKhSnhTalURTmEpUxebQb8u5s/3QkenJ28w0SYuUGYV2JdGynBJVDf3MSe4eDFBLv21u1PhgNLkmaZEyo9CuJLu36h5tiab+LnfvfCdYyjPTxC2aDlXKkEK7UiR6YO8OtbQlmurGBouJpM8F3zXAgizJwWvuha+fSJEotCvF7q3B7zEaPS4RlWla2c5+lj6F4JL5nu2wq7PwdRMpEoV2pdi/WIha2hJRDU1BSzu15dy1BmrHwPijM5dPlhEpEwrtSqFlOSXqYk2wdzvsih/Y19kejBSvyvBPmW77kjKk0K4UWpZToi65Wlrq4LKu9v4XXJk4HapHaTCalBWFdqXQspwSdclwTracE3uDqUr7W/KzqjqYJU2Xx6WMKLQrhZbllKg7bNrBLect64IVwAZa2rRBt31JeVFoV4rdW6F2LNTWl7omIkNTVRX0XydbzskwTs5+lklsLmx9A3r3Fb5+IkWg0K4U3V3qz5boS50wZaB7tPeXb4K+BGxdX/i6iRSBQrtSaN5xKQcNTbD1zWCyoM41MO5IqJ8wcHnQCHIpGwrtStHdpf5sib5YE3gvbFsfXCbvbxDa/vIZRpyLRJhCu1JoWU4pB6kLh3RlWN0r3ZjJwfdeLW0pEwrtSqHL41IOkiG94elgcGW2ljYErfOutYWtl0iRKLQrQW8C9mxTS1uib/REGNsIrQ8G2wPd7pWk276kjCi0K8GebcFv9WlLOYg1QWdr8Lghy+XxZPldm4PFQ0QiTqFdCTSFqZSTZFBX18HEGTmUT/aDa2Y0ib6cQtvMLjCzVjNbY2ZLMxyfYWaPmtlLZrbczKamHOs1sxfCn/vzWXnJ0f4pTBXaUgaSl8Qnzw6mKs21vAajSRmoyVbAzKqBW4APAx3ASjO7391fSyn2PeAud7/TzM4Fvg18Jjy2290X5LneMhhallPKSbLlnG3keNKkmWDV6teWspA1tIGFwBp3XwdgZvcAlwCpoT0f+LPw8WPAr/JZySHr64OffbzUtSi9ne8Gv9WnLeUgGda5hnZNHUyaAc//DN5eVbh6SW7GNsIl/xz8d8mmrxce/CtYuAQa5xW+bhGQS2hPATakbHcAZ6SVeRG4DPgh8HFgvJnF3L0LqDezVUACuNHdDwl0M1sCLAGYPn36oN/EgPbtzu/5oqh+Ahz/cZhwdKlrIjJ8k2fDqZ+FEy7N/Tmn/yG89iv9e1Bqe7bDuuWw6Etw9CnZy3e2wcp/gbENcM4hPbMVKZfQtgz7PG37euBmM7saWAG8TRDSANPdfaOZzQZ+a2Yvu/tBN026+23AbQDNzc3p5x66qir43MN5O52IjABV1XDxjwb3nEVfDH6ktN59DX68KBgUmFNotx/8W3IK7Q5gWsr2VGBjagF33whcCmBm44DL3H17yjHcfZ2ZLQdOATTTgYhIpZk8G7DcBwV2pS0OIzmNHl8JNJnZLDOrA64ADhoFbmYNZpY811eAZeH+SWY2KlkGOJOD+8JFRKRS1NbDxOm5t5yTt+l1rQXP30XYKMsa2u6eAL4MPASsBu5191fN7AYzuzgsdg7QamZtwBHAt8L9xwGrzOxFggFqN6aNOhcRkUrS0DT4lnbPe7BzU+HqFCG5XB7H3R8AHkjb99WUx/cB92V43pPAicOso4iIlItYE6x/Mri7p2qAdqN70CKfPAe2rA0eazCtZkQTEZEiapgL+7ph58aBy3V3BVMwH3NhsK1+bUChLSIixRRLWV51IMnjs86G2rGahjak0BYRkeJJzmjXlSWEky3rhiaIzVFLO6TQFhGR4hl/FNSNy62lnVwUpqFJ92qHFNoiIlI8Zrm1nLvWHFgUJtYE296CfXuKU8cRTKEtIiLFFWvK3kfd2X5gfvmGJsBhy7qCV22kU2iLiEhxNTTB9g39zwXfm4Ctbxy6olu2fvAKoNAWEZHiis1lwJbztvXQlzgw0jw2J/itwWgKbRERKbKGLLd9dbYfXG7U+GAAm277UmiLiEiR7b/c3U9oJ/enrpkem6uWNgptEREptrqxMGFK/y3nznYYE4Mxkw/sS972VeELhyi0RUSk+AZqOXetOdCfvb98UzCtaXdX4es2gim0RUSk+BrC274ytZw724M5ytPLJ49VMIW2iIgUX6wJ9m6HXfGD9+/ZDrs2Z2hpZ+kHrxAKbRERKb5kSzq95Zzs546ltbQnTofqUWppl7oCIiJSgfprOXel3e6VVFUdTGta4ROsKLRFRKT4DpuWueXc2Q5WDZNmHfqchrlqaZe6AiIiUoGqqsOFQ9Jazl3tMGkG1NQd+pxYUzC9ae++4tRxBFJoi4hIacQytJw7M9zulVq+LwFb1xe+biOUQltEREqjoQm2vgmJnmC7rw+2rD20Pzu1PFT0CPKaUldAREQqVKwJvBduPw+q64JWdGLPoSPH95cP9z/4V/D494tXz0zO+wbMPLPoL6vQFhGR0phzLhz7UdjXfWDfMRdB04czlx8zGc7479DZWpz6DaSqNPGp0BYRkdIYfwRc8fPBPefCGwtTl4hQn7aIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIRCW0REJCIU2iIiIhGh0BYREYkIhbaIiEhEKLRFREQiQqEtIiISEQptERGRiFBoi4iIRIS5e6nrcBAziwPr83zaBqAzz+esRPoc80OfY37oc8wPfY75MdzPcYa7N2YrNOJCuxDMbJW7N5e6HlGnzzE/9Dnmhz7H/NDnmB/F+hx1eVxERCQiFNoiIiIRUSmhfVupK1Am9Dnmhz7H/NDnmB/6HPOjKJ9jRfRpi4iIlINKaWmLiIhEXlmHtpldYGatZrbGzJaWuj5RYWbTzOwxM1ttZq+a2XXh/slm9oiZtYe/J5W6rlFgZtVm9ryZ/d9we5aZPR1+jr8ws7pS13GkM7OJZnafmb0efi8X6fs4eGb2Z+H/06+Y2d1mVq/vY3ZmtszMNpvZKyn7Mn7/LHBTmDsvmdmp+axL2Ya2mVUDtwAXAvOBK81sfmlrFRkJ4C/c/TjgfcCXws9uKfCouzcBj4bbkt11wOqU7e8A/xh+jluBz5WkVtHyQ+D/ufuxwMkEn6e+j4NgZlOAPwGa3f0EoBq4An0fc/FT4IK0ff19/y4EmsKfJcCP81mRsg1tYCGwxt3XuXsPcA9wSYnrFAnuvsndnwsf7yT4B3IKwed3Z1jsTuBjpalhdJjZVOD3gJ+E2wacC9wXFtHnmIWZTQDOBm4HcPced9+Gvo9DUQOMNrMaYAywCX0fs3L3FcCWtN39ff8uAe7ywFPARDM7Kl91KefQngJsSNnuCPfJIJjZTOAU4GngCHffBEGwA4eXrmaR8U/AXwJ94XYM2ObuiXBb38vsZgNx4I6wm+EnZjYWfR8Hxd3fBr4HvEUQ1tuBZ9H3caj6+/4VNHvKObQtwz4NlR8EMxsH/BL4U3ffUer6RI2ZfRTY7O7Ppu7OUFTfy4HVAKcCP3b3U4Bd6FL4oIV9rpcAs4CjgbEEl3LT6fs4PAX9f7ycQ7sDmJayPRXYWKK6RI6Z1RIE9s/d/d/D3e8mL/OEvzeXqn4RcSZwsZm9SdA9cy5By3tieHkS9L3MRQfQ4e5Ph9v3EYS4vo+Dcx7whrvH3X0f8O/A+9H3caj6+/4VNHvKObRXAk3hyMg6ggEX95e4TpEQ9rveDqx29x+kHLofuCp8fBXwf4pdtyhx96+4+1R3n0nw/futu/8B8BhweVhMn2MW7v4OsMHMjgl3fQh4DX0fB+st4H1mNib8fzz5Oer7ODT9ff/uBz4bjiJ/H7A9eRk9H8p6chUzu4igZVMNLHP3b5W4SpFgZmcBjwMvc6Av9n8S9GvfC0wn+AfgE+6ePjhDMjCzc4Dr3f2jZjaboOU9GXge+LS77y1l/UY6M1tAMJivDlgHXEPQ6ND3cRDM7BvApwjuEHke+DxBf6u+jwMws7uBcwhW8noX+BrwKzJ8/8I/iG4mGG3eDVzj7qvyVpdyDm0REZFyUs6Xx0VERMqKQltERCQiFNoiIiIRodAWERGJCIW2iIhIRCi0RYbJzNzMvp+yfb2ZfT1P5/6pmV2eveSwX+cT4epZj6XtP9rM7gsfLwhvo8zXa040sy9mei0RyUyhLTJ8e4FLzayh1BVJFa50l6vPAV909w+m7nT3je6e/KNhATCo0E6ZaSuTicD+0E57LRHJQKEtMnwJ4Dbgz9IPpLeUzey98Pc5ZtZiZveaWZuZ3Whmf2Bmz5jZy2Y2J+U055nZ42G5j4bPrzaz75rZynDN3j9KOe9jZvZvBJPjpNfnyvD8r5jZd8J9XwXOAm41s++mlZ8Zlq0DbgA+ZWYvmNmnzGxsuM7wynAhj0vC51xtZv/bzH4NPGxm48zsUTN7Lnzt5Gp7NwJzwvN9N/la4TnqzeyOsPzzZvbBlHP/u5n9PwvWMf6HlM/jp2FdXzazQ/5biJSDgf4KFpHc3QK8lAyRHJ0MHEew5N864CfuvtDMrgP+GPjTsNxMYDEwB3jMzOYCnyWYHvF0MxsFPGFmD4flFwInuPsbqS9mZkcTrJ18GsG6yQ+b2cfc/QYzO5dgxraMMze5e08Y7s3u/uXwfH9PMDXrtWY2EXjGzH4TPmURcFI4Q1QN8HF33xFejXjKzO4nWPTjBHdfEJ5vZspLfil83RPN7NiwrvPCYwsIVp7bC7Sa2Y8IVliaEq4TTVgfkbKjlrZIHoSroN0F/MkgnrYyXLt8L7AWSIbuywRBnXSvu/e5eztBuB8LnE8wv/ELBNPLxoCmsPwz6YEdOh1YHi4YkQB+TrBO9VCdDywN67AcqCeY0hHgkZQpRQ34ezN7CfgNwbSZR2Q591nAvwK4++vAeiAZ2o+6+3Z330Mwd/YMgs9ltpn9yMwuALQqnZQltbRF8uefgOeAO1L2JQj/OA7nJK5LOZY6v3NfynYfB/+/mT7XsBME4R+7+0OpB8I5znf1U79MSwYOhwGXuXtrWh3OSKvDHwCNwGnuvs+CVc/qczh3f1I/t16gxt23mtnJwEcIWumfBK7N6V2IRIha2iJ5ErYs7yUY1JX0JsHlaAjWMq4dwqk/YWZVYT/3bKAVeAj47xYsoYqZzTOzsVnO8zSw2MwawkFqVwItg6jHTmB8yvZDwB+Hf4xgZqf087zDCNYV3xf2Tc/o53ypVhCEPeFl8ekE7zuj8LJ7lbv/EvhbgqU7RcqOQlskv75PsBJQ0r8QBOUzQHoLNFetBOH6IPCF8LLwTwguDT8XDt76X2S5chYuD/gVgqUYXwSec/fBLMP4GDA/ORAN+CbBHyEvhXX4Zj/P+znQbGarCIL49bA+XQR98a+kD4AD/hmoNrOXgV8AV2dZeWoKsDy8VP/T8H2KlB2t8iUiIhIRammLiIhEhEJbREQkIhTaIiIiEaHQFhERiQiFtoiISEQotEVERCJCoS0iIhIRCm0REZGI+P/rnJtklTcHgAAAAABJRU5ErkJggg==\n", 
                        "text/plain": "<matplotlib.figure.Figure at 0x2b5d00486f98>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "xgb_eval = xgb_model.evals_result()\neval_steps = range(len(xgb_eval['validation_0']['error']))\n\nfig, ax = pyplot.subplots(1, 1, sharex=True, figsize=(8, 6))\n\nax.plot(eval_steps, [1-x for x in xgb_eval['validation_0']['error']], label='Train')\nax.plot(eval_steps, [1-x for x in xgb_eval['validation_1']['error']], label='Test')\nax.legend()\nax.set_title('Accuracy')\nax.set_xlabel('Number of iterations')"
        }, 
        {
            "source": "We can see that there is model overfitting. We see some model accuracy deacrease after about 60 iterations.\n\nWe select model trained after 30 iterations.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 20, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "n_trees = 30\ny_pred = xgb_model.predict(X_test, ntree_limit= n_trees)"
        }, 
        {
            "execution_count": 21, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 96.0%\n"
                }
            ], 
            "source": "accuracy = accuracy_score(y_test, y_pred)\n\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "**Note:** We see that model accuracy on test set. We will use accuracy on test data to compare with model selected using parameter tuning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2.1: Grid search for best model over hyperparameter space", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "We can tune our model now to achieve better accuracy by using grid search and cross validation.\n\nXGBoost has an extensive catalog of hyperparameters which provides great flexibility to shape algorithm\u2019s desired behavior. Here we will tune model optimizing which adds L1 penalty (`reg_alpha`).\nWe will use 10 fold crossvalidation because our training dataset is small.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In cell below, we create our XGBoost pipeline and setup parameter grid for search.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 22, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "xgb_model_gs = XGBClassifier()\nparameters = {'reg_alpha': [0.0, 1.0], 'reg_lambda': [0.0, 1.0], 'n_estimators': [n_trees], 'seed': [1337]}"
        }, 
        {
            "source": "We can search for the best parameters over specified parameters values above with GridSearchCV.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 23, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }, 
                {
                    "execution_count": 23, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "GridSearchCV(cv=10, error_score='raise',\n       estimator=XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=100, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1),\n       fit_params={}, iid=True, n_jobs=1,\n       param_grid={'seed': [1337], 'n_estimators': [30], 'reg_lambda': [0.0, 1.0], 'reg_alpha': [0.0, 1.0]},\n       pre_dispatch='2*n_jobs', refit=True, scoring='accuracy', verbose=0)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "clf = GridSearchCV(xgb_model_gs, parameters, scoring='accuracy', cv=10, verbose=0, refit=True)\nclf.fit(X_train, y_train)"
        }, 
        {
            "source": "We can see the performance result of all parameter combinations including the best parameter combination based on model performance from grid search scores", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 24, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 24, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "[mean: 0.94851, std: 0.03689, params: {'seed': 1337, 'n_estimators': 30, 'reg_lambda': 0.0, 'reg_alpha': 0.0},\n mean: 0.95393, std: 0.03813, params: {'seed': 1337, 'n_estimators': 30, 'reg_lambda': 1.0, 'reg_alpha': 0.0},\n mean: 0.95664, std: 0.03834, params: {'seed': 1337, 'n_estimators': 30, 'reg_lambda': 0.0, 'reg_alpha': 1.0},\n mean: 0.95393, std: 0.03616, params: {'seed': 1337, 'n_estimators': 30, 'reg_lambda': 1.0, 'reg_alpha': 1.0}]"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "clf.grid_scores_"
        }, 
        {
            "source": "Accuracy estimated using crossvalidation and hyperparameter values for best model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 25, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Best score: 95.7%\nBest parameter set: {'seed': 1337, 'n_estimators': 30, 'reg_lambda': 0.0, 'reg_alpha': 1.0}\n"
                }
            ], 
            "source": "print(\"Best score: %.1f%%\" % (clf.best_score_*100))\nprint(\"Best parameter set: %s\" % (clf.best_params_))"
        }, 
        {
            "source": "We can see the accuracy of best parameter combination on test set.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 96.0%\n"
                }, 
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "y_pred = clf.best_estimator_.predict(X_test, ntree_limit= n_trees)\n\naccuracy = accuracy_score(y_test, y_pred)\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "We see that accuracy on test set is about the same for tuned model as for trained with default values for hyperparameters. Altghough selected hyperparameters are different then default.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2.3: Model with pipeline data preprocessing", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Here you will see how to use XGBoost model within scikit-learn pipeline. \nLet's start with importing required objects.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn.pipeline import Pipeline\nfrom sklearn.decomposition import PCA"
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "pca = PCA(n_components=10)\nxgb_model_pca = XGBClassifier(n_estimators=n_trees)\npipeline = Pipeline(steps=[('pca', pca), ('xgb', xgb_model_pca)])"
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 29, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Pipeline(memory=None,\n     steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=10, random_state=None,\n  svd_solver='auto', tol=0.0, whiten=False)), ('xgb', XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,\n       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3,\n       min_child_weight=1, missing=None, n_estimators=30, nthread=-1,\n       objective='binary:logistic', reg_alpha=0, reg_lambda=1,\n       scale_pos_weight=1, seed=0, silent=True, subsample=1))])"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "pipeline.fit(X_train, y_train)"
        }, 
        {
            "source": "Now we are ready to evaluate accurracy of model trained on reduced set of features.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy: 95.5%\n"
                }, 
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "/opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n  if diff:\n"
                }
            ], 
            "source": "y_pred = pipeline.predict(X_test)\naccuracy = accuracy_score(y_test, y_pred)\nprint(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
        }, 
        {
            "source": "We can see that model has similar accuracy to trained using default hyperparameter values.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Let's see how we can save our XGBoost pipeline using Watson Machine Learning instance and deploy it for online scoring.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 4. Persist model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to store your XGBoost model in Watson Machine Learning repository by using python client libraries.\n\nFirst, you must import client libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting watson-machine-learning-client\n  Downloading watson_machine_learning_client-1.0.53-py3-none-any.whl (561kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 563kB 1.7MB/s eta 0:00:01\n\u001b[?25hRequirement not upgraded as not directly required: urllib3 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: pandas in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: certifi in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: lomond in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: tqdm in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: requests in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: tabulate in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement not upgraded as not directly required: python-dateutil>=2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement not upgraded as not directly required: pytz>=2011k in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement not upgraded as not directly required: numpy>=1.9.0 in /home/dsxuser/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement not upgraded as not directly required: six>=1.10.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement not upgraded as not directly required: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement not upgraded as not directly required: idna<2.7,>=2.5 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nInstalling collected packages: watson-machine-learning-client\nSuccessfully installed watson-machine-learning-client-1.0.53\n"
                }
            ], 
            "source": "!pip install watson-machine-learning-client --upgrade --user"
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "source": "Authenticate to Watson Machine Learning service on IBM Cloud.\n\n**Action**: Put authentication information from your instance of Watson Machine Learning service here.</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials = {\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"***\",\n  \"username\": \"***\",\n  \"password\": \"***\",\n  \"instance_id\": \"***\"\n}"
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "**Tip**: Service Credentials can be found on service dashboard on IBM Cloud.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "### 4.1: Save a XGBoost model in the ML Repository\n\nIn this subsection you will learn how to save model artifact to your Watson Machine Learning instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_props = {client.repository.ModelMetaNames.NAME: \"XGBoost model for breast cancer\"}\nmodel_details = client.repository.store_model(pipeline, model_props)"
        }, 
        {
            "source": "Get saved model metadata from Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "# 5. Deploy and score in a Cloud", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to create online scoring and to score a new data record by using the Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can list all stored models using method `list_models`.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  -------------------------------  ------------------------  -----------------\nGUID                                  NAME                             CREATED                   FRAMEWORK\n6b76f1d7-aff2-445d-b816-0c4c573fbf7d  Product Line Prediction          2018-02-18T19:51:57.776Z  mllib-2.0\n4b7c9d27-6ca3-4b89-b726-9dd32ccb3958  Tumor_Type_Detection_v1_pipe     2018-03-18T23:37:40.678Z  scikit-learn-0.19\n7c60098c-5c16-47f8-89e6-f72a629a9bff  My cool mnist model              2018-03-19T09:42:42.204Z  tensorflow-1.5\n5999a2d9-e22c-424e-839f-d1f9c37ba8c0  XGBoost model for breast cancer  2018-03-29T14:00:39.597Z  scikit-learn-0.19\nbc1104f5-9bba-4bf4-a8cc-34eff218ae4a  XGBoost model for breast cancer  2018-04-03T10:05:17.385Z  scikit-learn-0.19\n------------------------------------  -------------------------------  ------------------------  -----------------\n"
                }
            ], 
            "source": "client.repository.list_models()"
        }, 
        {
            "source": "We use model uid to create deployment. We can extract it from saved model details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "bc1104f5-9bba-4bf4-a8cc-34eff218ae4a\n"
                }
            ], 
            "source": "model_uid = client.repository.get_model_uid(model_details)\nprint(model_uid)"
        }, 
        {
            "source": "### 5.1: Create model deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you can create a deployment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: 'e63d45ac-c27d-4002-8795-81d16ea98f6d' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='e63d45ac-c27d-4002-8795-81d16ea98f6d'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "deployment_details = client.deployments.create(model_uid, 'Predict breast cancer')"
        }, 
        {
            "source": "Get the list of deployments.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ----------------------  ------  --------------  ------------------------  -----------------\nGUID                                  NAME                    TYPE    STATE           CREATED                   FRAMEWORK\n0e4ad207-9c30-4157-a55f-ac83808c9a2b  Predict breast cancer   online  DEPLOY_SUCCESS  2018-03-29T14:00:52.650Z  scikit-learn-0.19\n206ce253-eaaa-40f0-9aac-bc0d25e3fe88  online                  online  DEPLOY_SUCCESS  2018-02-18T19:52:22.583Z  mllib-2.0\n40f24853-f991-4aa1-9936-830f197aa758  Mnist model deployment  online  DEPLOY_SUCCESS  2018-03-19T09:42:58.735Z  tensorflow-1.5\n65e272bc-ff7e-4136-b210-c7a1a831a7f8  skl_xgb_cancer1_pipe    online  DEPLOY_SUCCESS  2018-03-18T23:38:07.407Z  scikit-learn-0.19\ne63d45ac-c27d-4002-8795-81d16ea98f6d  Predict breast cancer   online  DEPLOY_SUCCESS  2018-04-03T10:06:50.606Z  scikit-learn-0.19\n------------------------------------  ----------------------  ------  --------------  ------------------------  -----------------\n"
                }
            ], 
            "source": "client.deployments.list()"
        }, 
        {
            "source": "### 5. 3 Perform Prediction", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now, you have extract url endpoint used to send scoring requests.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/bc1104f5-9bba-4bf4-a8cc-34eff218ae4a/deployments/e63d45ac-c27d-4002-8795-81d16ea98f6d/online\n"
                }
            ], 
            "source": "scoring_url = client.deployments.get_scoring_url(deployment_details)\nprint(scoring_url)"
        }, 
        {
            "source": "Prepare scoring payload with values to score.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'values': [[12.23, 19.56, 78.54, 461.0, 0.09586, 0.08087, 0.04187, 0.04107, 0.1979, 0.06013, 0.3534, 1.326, 2.308, 27.24, 0.007514, 0.01779, 0.01401, 0.0114, 0.01503, 0.003338, 14.44, 28.36, 92.15, 638.4, 0.1429, 0.2042, 0.1377, 0.108, 0.2668, 0.08174]]}\n"
                }
            ], 
            "source": "payload_scoring = {\"values\": [X_test[0].tolist()]}\nprint(payload_scoring)"
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'fields': ['prediction', 'probability'], 'values': [[0.0, [0.9505169987678528, 0.04948299005627632]]]}\n"
                }
            ], 
            "source": "response_scoring = client.deployments.score(scoring_url, payload_scoring)\nprint(response_scoring)"
        }, 
        {
            "source": "As we can see, given patient record is classified as benign tumor.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 6. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You successfully completed this notebook! You learned how to use XGBoost machine learning as well as Watson Machine Learning for model creation and deployment. Check out our _[Online Documentation](https://console.bluemix.net/docs/services/PredictiveModeling/index.html#WMLgettingstarted)_ for more samples, tutorials, documentation, how-tos, and blog posts. \n\n### Author\n\nWojciech Sobala is Data Scientist at IBM.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}