From spark ml model to batch scoring.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Predicting customer churn with IBM Watson Machine Learning</b></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n  <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/blob/master/spark/customer-satisfaction-prediction/images/users_banner_2-03.png?raw=true\" width=\"600\" alt=\"Icon\"> </th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to develop a predictive model, and start scoring new data. This notebook introduces commands for getting data and for basic data cleaning and exploration, pipeline creation, model training, model persistance to Watson Machine Learning repository, model deployment, and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.5 and Apache\u00ae Spark 2.1.\n\nYou will use a data set, **Telco Customer Churn**, which details anonymous customers data of telecommunication company. Use the details of this data set to predict customer churn which is very critical to business as it's easier to retain existing customers rather than acquiring new ones.\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Load a CSV file into an Apache\u00ae Spark DataFrame.\n-  Explore data.\n-  Prepare data for training and evaluation.\n-  Create an Apache\u00ae Spark machine learning pipeline.\n-  Train and evaluate a model.\n-  Persist a pipeline and model in Watson Machine Learning repository.\n-  Explore and visualize prediction result using the plotly package.\n-  Deploy a model for batch scoring using Wastson Machine Learning API.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Setup](#setup)\n2.\t[Load and explore data](#load)\n3.\t[Create spark ml model](#model)\n4.\t[Persist model](#persistence)\n5.\t[Predict locally and visualize](#visualization)\n6.\t[Deploy and score in a Cloud](#scoring)\n7.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Setup\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance required for model storing and deployment (a free plan is offered).\n- Create a [Cloud Object Storage](https://console.bluemix.net/catalog/infrastructure/cloud-object-storage) instance required for storing scoring data (a free plan is offered).\n- Create a [Apache Spark Service](https://console.bluemix.net/catalog/services/apache-spark/) instance required for batch scoring (a free plan is offered).\n-  Make sure that you are using a Spark 2.0 kernel in notebook.\n\n### Create the Telco Customer Churn Data Asset  \n\nThe Telco Customer Churn data is available on project's Github page.\n\n1.  Run following cells to download dataset from **Telco Customer Churn project** page [data folder](https://github.com/pmservice/wml-sample-models/tree/master/spark/customer-satisfaction-prediction/data) to local machine filesystem.\n2.  After downloading the dataset to your local machine, read it into Spark DataFrame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "Install wget package if necessary by running following command <br>\n!pip install wget", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "WA_Fn%20UseC_%20Telco%20Customer%20Churn.csv\n"
                }
            ], 
            "source": "import wget\n\nlink_to_data = 'https://github.com/pmservice/wml-sample-models/raw/master/spark/customer-satisfaction-prediction/data/WA_Fn%20UseC_%20Telco%20Customer%20Churn.csv'\nfilename = wget.download(link_to_data)\n\nprint(filename)"
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 2. Load and explore data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will load the data as an Apache\u00ae Spark DataFrame and perform a basic exploration. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Once you see the file on **Files** section, you can insert it to your notebook by selecting **Insert SparkSession DataFrame**", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "spark = SparkSession.builder.getOrCreate()\n\ndf_data = spark.read\\\n  .format('org.apache.spark.sql.execution.datasources.csv.CSVFileFormat')\\\n  .option('header', 'true')\\\n  .option('inferSchema', 'true')\\\n  .option('nanValue', ' ')\\\n  .option('nullValue', ' ')\\\n  .load(filename)"
        }, 
        {
            "source": "Explore the loaded data by using the following Apache\u00ae Spark DataFrame methods:\n-  print schema\n-  count all records\n-  show distribution of label classes", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "root\n |-- customerID: string (nullable = true)\n |-- gender: string (nullable = true)\n |-- SeniorCitizen: integer (nullable = true)\n |-- Partner: string (nullable = true)\n |-- Dependents: string (nullable = true)\n |-- tenure: integer (nullable = true)\n |-- PhoneService: string (nullable = true)\n |-- MultipleLines: string (nullable = true)\n |-- InternetService: string (nullable = true)\n |-- OnlineSecurity: string (nullable = true)\n |-- OnlineBackup: string (nullable = true)\n |-- DeviceProtection: string (nullable = true)\n |-- TechSupport: string (nullable = true)\n |-- StreamingTV: string (nullable = true)\n |-- StreamingMovies: string (nullable = true)\n |-- Contract: string (nullable = true)\n |-- PaperlessBilling: string (nullable = true)\n |-- PaymentMethod: string (nullable = true)\n |-- MonthlyCharges: double (nullable = true)\n |-- TotalCharges: double (nullable = true)\n |-- Churn: string (nullable = true)\n\nNumber of fields:  21\n"
                }
            ], 
            "source": "df_data.printSchema()\n\nprint(\"Number of fields: %3g\" % len(df_data.schema))"
        }, 
        {
            "source": "As you can see, the data contains 21 fields. \"Churn\" field is the one we would like to predict (label).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Total number of records: 7043\n"
                }
            ], 
            "source": "print(\"Total number of records: \" + str(df_data.count()))"
        }, 
        {
            "source": "Data set contains 7043 records.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you will check if all records have complete data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of records with complete data: 7032\n"
                }
            ], 
            "source": "df_complete = df_data.dropna()\n\nprint(\"Number of records with complete data: %3g\" % df_complete.count())"
        }, 
        {
            "source": "You can see that there are some missing values you can investigate that all missing values are present in `TotalCharges` feature. We will use dataset with missing values removed for model training and evaluation.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you will inspect distribution of classes in label column.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+-----+-----+\n|Churn|count|\n+-----+-----+\n|   No| 5163|\n|  Yes| 1869|\n+-----+-----+\n\n"
                }
            ], 
            "source": "df_complete.groupBy('Churn').count().show()"
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 3. Create an Apache\u00ae Spark machine learning model\n\nIn this section you will learn how to prepare data, create an Apache\u00ae Spark machine learning pipeline, and train a model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1: Prepare data\n\nIn this subsection you will split your data into: train, test and predict datasets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of records for training: 5621\nNumber of records for evaluation: 1263\nNumber of records for prediction: 148\n"
                }
            ], 
            "source": "(train_data, test_data, predict_data) = df_complete.randomSplit([0.8, 0.18, 0.02], 24)\n\nprint(\"Number of records for training: \" + str(train_data.count()))\nprint(\"Number of records for evaluation: \" + str(test_data.count()))\nprint(\"Number of records for prediction: \" + str(predict_data.count()))"
        }, 
        {
            "source": "As you can see our data has been successfully split into three datasets: \n\n-  The train data set, which is the largest group, is used for training.\n-  The test data set will be used for model evaluation and is used to test the assumptions of the model.\n-  The predict data set will be used for prediction.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2: Create pipeline and train a model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will create an Apache\u00ae Spark machine learning pipeline and then train the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In the first step you need to import the Apache\u00ae Spark machine learning packages that will be needed in the subsequent steps.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from pyspark.ml.feature import StringIndexer, IndexToString, RFormula\nfrom pyspark.ml.classification import LogisticRegression\nfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator\nfrom pyspark.ml import Pipeline, Model"
        }, 
        {
            "source": "In the following step, convert all the predictors to features vector and label feature convert to numeric.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "lab = StringIndexer(inputCol = 'Churn', outputCol = 'label')\nfeatures = RFormula(formula = \"~ gender + SeniorCitizen +  Partner + Dependents + tenure + PhoneService + MultipleLines + InternetService + OnlineSecurity + OnlineBackup + DeviceProtection + TechSupport + StreamingTV + StreamingMovies + Contract + PaperlessBilling + PaymentMethod + MonthlyCharges + TotalCharges - 1\")"
        }, 
        {
            "source": "Next, define estimators you want to use for classification. Logistic Regression is used in the following example.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "lr = LogisticRegression(maxIter = 10)"
        }, 
        {
            "source": "Let's build the pipeline now. A pipeline consists of transformers and an estimator.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "pipeline_lr = Pipeline(stages = [features, lab , lr])"
        }, 
        {
            "source": "Now, you can train your Logistic Regression model by using the previously defined **pipeline** and **train data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_lr = pipeline_lr.fit(train_data)"
        }, 
        {
            "source": "You can check your **model accuracy** now. To evaluate the model, use **test data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Test dataset:\nAccuracy = 0.81\n"
                }
            ], 
            "source": "predictions = model_lr.transform(test_data)\nevaluator = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\", metricName=\"accuracy\")\naccuracy = evaluator.evaluate(predictions)\n\nprint(\"Test dataset:\")\nprint(\"Accuracy = %3.2f\" % accuracy)"
        }, 
        {
            "source": "You can tune your model now to achieve better accuracy. For simplicity of this example tuning section is omitted.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 4. Persist model", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "In this section you will learn how to store your pipeline and model in Watson Machine Learning repository by using python client libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "First, you must import Watson Machine Learning client libraries.\n\n**Note**: Apache\u00ae Spark 2.1 is required.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting watson-machine-learning-client\n  Downloading watson_machine_learning_client-1.0.53.tar.gz (215kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 215kB 1.7MB/s eta 0:00:01\n\u001b[?25hRequirement already up-to-date: requests in /usr/local/src/bluemix_jupyter_bundle.v86/notebook/lib/python2.7/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: urllib3 in /usr/local/src/bluemix_jupyter_bundle.v86/notebook/lib/python2.7/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: pandas in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python2.7/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: certifi in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python2.7/site-packages (from watson-machine-learning-client)\nCollecting tqdm (from watson-machine-learning-client)\n  Downloading tqdm-4.19.9-py2.py3-none-any.whl (52kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 61kB 1.0MB/s eta 0:00:01\n\u001b[?25hRequirement already up-to-date: lomond in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python2.7/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: tabulate in /usr/local/src/bluemix_jupyter_bundle.v86/notebook/lib/python2.7/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: idna<2.7,>=2.5 in /usr/local/src/bluemix_jupyter_bundle.v86/notebook/lib/python2.7/site-packages (from requests->watson-machine-learning-client)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python2.7/site-packages (from requests->watson-machine-learning-client)\nCollecting pytz>=2011k (from pandas->watson-machine-learning-client)\n  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 512kB 1.1MB/s eta 0:00:01\n\u001b[?25hCollecting numpy>=1.9.0 (from pandas->watson-machine-learning-client)\n  Downloading numpy-1.14.2-cp27-cp27mu-manylinux1_x86_64.whl (12.1MB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12.1MB 90kB/s  eta 0:00:01\n\u001b[?25hCollecting python-dateutil (from pandas->watson-machine-learning-client)\n  Downloading python_dateutil-2.7.2-py2.py3-none-any.whl (212kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 215kB 1.7MB/s eta 0:00:01\n\u001b[?25hRequirement already up-to-date: six>=1.10.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python2.7/site-packages (from lomond->watson-machine-learning-client)\nBuilding wheels for collected packages: watson-machine-learning-client\n  Running setup.py bdist_wheel for watson-machine-learning-client ... \u001b[?25ldone\n\u001b[?25h  Stored in directory: /gpfs/fs01/user/s86b-18e61b28c674e4-3fbaf243aed6/.cache/pip/wheels/e9/5a/a2/e5a4f33de365cc0cf8b4d938abd6294d7321dd816eefb4ff38\nSuccessfully built watson-machine-learning-client\nInstalling collected packages: tqdm, watson-machine-learning-client, pytz, numpy, python-dateutil\n  Found existing installation: tqdm 4.19.5\n    Uninstalling tqdm-4.19.5:\n      Successfully uninstalled tqdm-4.19.5\n  Found existing installation: watson-machine-learning-client 0.3.323\n    Uninstalling watson-machine-learning-client-0.3.323:\n      Successfully uninstalled watson-machine-learning-client-0.3.323\n  Found existing installation: pytz 2017.3\n    Uninstalling pytz-2017.3:\n      Successfully uninstalled pytz-2017.3\n  Found existing installation: numpy 1.14.0\n    Uninstalling numpy-1.14.0:\n      Successfully uninstalled numpy-1.14.0\n  Found existing installation: python-dateutil 2.6.1\n    Uninstalling python-dateutil-2.6.1:\n      Successfully uninstalled python-dateutil-2.6.1\nSuccessfully installed numpy-1.14.2 python-dateutil-2.7.2 pytz-2018.3 tqdm-4.19.9 watson-machine-learning-client-1.0.53\n"
                }
            ], 
            "source": "!pip install watson-machine-learning-client --upgrade"
        }, 
        {
            "execution_count": 17, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "source": "Authenticate to Watson Machine Learning service on Bluemix.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "**Action**: Put credentials from your instance of Watson Machine Learning service in Bluemix here.</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials = {\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"***\",\n  \"username\": \"***\",\n  \"password\": \"***\",\n  \"instance_id\": \"***\"\n}"
        }, 
        {
            "execution_count": 18, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "execution_count": 19, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "### 4.1: Save pipeline and model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will learn how to save pipeline and model artifacts to your Watson Machine Learning instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 22, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "saved_model = client.repository.store_model(model=model_lr, meta_props={'name':'my spark model'}, training_data=train_data, pipeline=pipeline_lr)"
        }, 
        {
            "source": "Get saved model metadata from Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 23, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Model Id: 0da45aa2-5378-4dc1-8617-3d209499f466\n"
                }
            ], 
            "source": "published_model_ID = client.repository.get_model_uid(saved_model)\n\nprint(\"Model Id: \" + str(published_model_ID))"
        }, 
        {
            "source": "**Model Id** can be used to retrive latest model version from Watson Machine Learning instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.2: Load model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will learn how to load back saved model from specified instance of Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 24, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "loaded_model = client.repository.load(published_model_ID)"
        }, 
        {
            "execution_count": 25, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<class 'pyspark.ml.pipeline.PipelineModel'>\n"
                }
            ], 
            "source": "print(type(loaded_model))"
        }, 
        {
            "source": "As you can see the name is correct. You have already learned how save and load the model from Watson Machine Learning repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"visualization\"></a>\n## 5. Predict locally and visualize", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to load data from batch scoring and visualize the prediction results with plotly package.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.1: Make local prediction using previously loaded model and test data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will score *predict_data* data set.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "predictions = loaded_model.transform(predict_data)"
        }, 
        {
            "source": "Preview the results by calling the *show()* method on the predictions DataFrame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+----------+------+-------------+-------+----------+------+------------+-------------+---------------+--------------+------------+----------------+-----------+-----------+---------------+--------------+----------------+--------------------+--------------+------------+-----+--------------------+-----+--------------------+--------------------+----------+\n|customerID|gender|SeniorCitizen|Partner|Dependents|tenure|PhoneService|MultipleLines|InternetService|OnlineSecurity|OnlineBackup|DeviceProtection|TechSupport|StreamingTV|StreamingMovies|      Contract|PaperlessBilling|       PaymentMethod|MonthlyCharges|TotalCharges|Churn|            features|label|       rawPrediction|         probability|prediction|\n+----------+------+-------------+-------+----------+------+------------+-------------+---------------+--------------+------------+----------------+-----------+-----------+---------------+--------------+----------------+--------------------+--------------+------------+-----+--------------------+-----+--------------------+--------------------+----------+\n|0378-XSZPU|  Male|            0|    Yes|        No|    58|         Yes|           No|            DSL|           Yes|         Yes|             Yes|         No|         No|             No|      One year|              No|Credit card (auto...|          60.3|      3563.8|  Yes|(30,[0,3,4,5,6,9,...|  1.0|[3.74236489664808...|[0.97685060080642...|       0.0|\n|0380-NEAVX|  Male|            1|     No|        No|    34|         Yes|           No|    Fiber optic|            No|         Yes|             Yes|         No|        Yes|            Yes|      One year|             Yes|    Electronic check|         99.85|     3343.15|   No|(30,[0,1,2,3,4,5,...|  0.0|[0.41319069172896...|[0.60185269953567...|       0.0|\n|0434-CSFON|Female|            0|    Yes|        No|    47|         Yes|          Yes|    Fiber optic|            No|          No|             Yes|         No|        Yes|            Yes|Month-to-month|             Yes|    Electronic check|         100.5|      4707.1|   No|(30,[3,4,5,7,8,10...|  0.0|[-0.0516148385850...|[0.48709915431274...|       1.0|\n|0461-CVKMU|Female|            0|    Yes|       Yes|    23|         Yes|           No|    Fiber optic|            No|         Yes|             Yes|        Yes|         No|             No|Month-to-month|              No|    Electronic check|          83.8|     1900.25|  Yes|(30,[4,5,6,8,10,1...|  1.0|[0.76974420811323...|[0.68346555821718...|       0.0|\n|0531-XBKMM|  Male|            0|     No|       Yes|    66|         Yes|          Yes|            DSL|           Yes|         Yes|              No|        Yes|         No|             No|      Two year|              No|Bank transfer (au...|          65.7|      4378.9|   No|(30,[0,2,4,5,7,9,...|  0.0|[5.07103459048971...|[0.99376321779986...|       0.0|\n+----------+------+-------------+-------+----------+------+------------+-------------+---------------+--------------+------------+----------------+-----------+-----------+---------------+--------------+----------------+--------------------+--------------+------------+-----+--------------------+-----+--------------------+--------------------+----------+\nonly showing top 5 rows\n\n"
                }
            ], 
            "source": "predictions.show(5)"
        }, 
        {
            "source": "By tabulating a count, you can see the split between labels.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+----------+-----+\n|prediction|count|\n+----------+-----+\n|0.0       |114  |\n|1.0       |34   |\n+----------+-----+\n\n"
                }
            ], 
            "source": "predictions.select(\"prediction\").groupBy(\"prediction\").count().show(truncate=False)"
        }, 
        {
            "source": "### 5.2: Sample visualization of data with Plotly package", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will explore prediction results with Plotly, which is an online analytics and data visualization tool.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**: First, you need to install required packages. You can do it by running the following code. Run it only one time.\n\n    !pip install plotly\n    !pip install cufflinks", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>", 
                        "text/vnd.plotly.v1+html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "import sys\nimport pandas\nimport plotly.plotly as py\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\nimport plotly.graph_objs as go\n\ninit_notebook_mode(connected=True)\nsys.path.append(\"\".join([os.environ[\"HOME\"]])) "
        }, 
        {
            "source": "Plot a pie chart that shows the predicted churn split. !!! By most important features: Contract, MonthlyCharges !!! predicted vs observed churns.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cumulative_stats = predictions.groupby(['label']).count()\nlabels_data_plot = ['No','Yes']\nvalues_data_plot = [cumulative_stats.select('count').collect()[x][0] for x in range(2)]"
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"14734a93-4085-477b-8999-4a4d77198a00\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"14734a93-4085-477b-8999-4a4d77198a00\", [{\"type\": \"pie\", \"labels\": [\"No\", \"Yes\"], \"values\": [111, 37]}], {\"title\": \"Churn\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"14734a93-4085-477b-8999-4a4d77198a00\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"14734a93-4085-477b-8999-4a4d77198a00\", [{\"type\": \"pie\", \"labels\": [\"No\", \"Yes\"], \"values\": [111, 37]}], {\"title\": \"Churn\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "type": "pie", 
                                    "labels": [
                                        "No", 
                                        "Yes"
                                    ], 
                                    "values": [
                                        111, 
                                        37
                                    ]
                                }
                            ], 
                            "layout": {
                                "title": "Churn"
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "product_data = [go.Pie(\n            labels=labels_data_plot,\n            values=values_data_plot,\n    )]\n\nproduct_layout = go.Layout(\n    title='Churn',\n)\n\nfig = go.Figure(data=product_data, layout=product_layout)\niplot(fig)"
        }, 
        {
            "source": "With this data set, you might want to do some analysis of Mean Monthly Charges per churn class", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "y_data_plot = [predictions.groupby(['label']).mean().collect()[x][3] for x in range(2)]"
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"853107c6-7a6a-4833-b727-afe7fb776369\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"853107c6-7a6a-4833-b727-afe7fb776369\", [{\"y\": [63.44144144144145, 81.10540540540539], \"x\": [\"No\", \"Yes\"], \"type\": \"bar\"}], {\"title\": \"Mean Monthly Charges per churn class\", \"xaxis\": {\"showline\": false, \"title\": \"Churn\"}, \"yaxis\": {\"title\": \"Mean Monthly Charges\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"853107c6-7a6a-4833-b727-afe7fb776369\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"853107c6-7a6a-4833-b727-afe7fb776369\", [{\"y\": [63.44144144144145, 81.10540540540539], \"x\": [\"No\", \"Yes\"], \"type\": \"bar\"}], {\"title\": \"Mean Monthly Charges per churn class\", \"xaxis\": {\"showline\": false, \"title\": \"Churn\"}, \"yaxis\": {\"title\": \"Mean Monthly Charges\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "y": [
                                        63.44144144144145, 
                                        81.10540540540539
                                    ], 
                                    "x": [
                                        "No", 
                                        "Yes"
                                    ], 
                                    "type": "bar"
                                }
                            ], 
                            "layout": {
                                "title": "Mean Monthly Charges per churn class", 
                                "xaxis": {
                                    "showline": false, 
                                    "title": "Churn"
                                }, 
                                "yaxis": {
                                    "title": "Mean Monthly Charges"
                                }
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "age_data = [go.Bar(\n            y= y_data_plot,\n            x= labels_data_plot\n            \n    )]\n\nage_layout = go.Layout(\n    title='Mean Monthly Charges per churn class',\n    xaxis=dict(\n        title = \"Churn\",\n        showline=False,),\n    yaxis=dict(\n        title = \"Mean Monthly Charges\",\n        ),\n)\n\nfig = go.Figure(data=age_data, layout=age_layout)\niplot(fig)"
        }, 
        {
            "source": "Based on bar plot you created, you might make the following conclusion: The mean monthly charges for churn customers is higher than non-churn customers as you may expect.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"scoring\"></a>\n## 6. Deploy model and score in a Cloud", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to create batch deployment and to score a new data record by using the Watson Machine Learning REST API. \nFor more information about REST APIs, see the [Swagger Documentation](http://watson-ml-api.mybluemix.net/).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### 6.1 Save data for prediction to Object Storage", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "ibm-cos-sdk library allows Python developers to manage Cloud Object Storage (COS).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import ibm_boto3\nfrom ibm_botocore.client import Config"
        }, 
        {
            "source": "<div>**Action**: Put credentials from Object Storage Service in Bluemix here.</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 101, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos_credentials = {\n  \"apikey\": \"***\",\n  \"cos_hmac_keys\": {\n    \"access_key_id\": \"***\",\n    \"secret_access_key\": \"***\"\n  },\n  \"endpoints\": \"***\",\n  \"iam_apikey_description\": \"***\",\n  \"iam_apikey_name\": \"***\",\n  \"iam_role_crn\": \"***\",\n  \"iam_serviceid_crn\": \"***\",\n  \"resource_instance_id\": \"***\"\n}"
        }, 
        {
            "execution_count": 64, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "connection_apikey = cos_credentials['apikey']\nconnection_resource_instance_id = cos_credentials[\"resource_instance_id\"]\nconnection_access_key_id = cos_credentials['cos_hmac_keys']['access_key_id']\nconnection_secret_access_key = cos_credentials['cos_hmac_keys']['secret_access_key']"
        }, 
        {
            "source": "**Action**: Define the service endpoint we will use. <br>\n**Tip**: You can find this information in Endpoints section of your Cloud Object Storage intance's dashbord.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "service_endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'"
        }, 
        {
            "source": "You also need IBM Cloud authorization endpoint to be able to create COS resource object.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "auth_endpoint = 'https://iam.bluemix.net/oidc/token'"
        }, 
        {
            "source": "We create COS resource to be able to write data to Cloud Object Storage.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos = ibm_boto3.resource('s3',\n                         ibm_api_key_id=cos_credentials['apikey'],\n                         ibm_service_instance_id=cos_credentials['resource_instance_id'],\n                         ibm_auth_endpoint=auth_endpoint,\n                         config=Config(signature_version='oauth'),\n                         endpoint_url=service_endpoint)"
        }, 
        {
            "source": "Now you will create bucket in COS and copy [dataset](https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/customer-satisfaction-prediction/data/scoreInput.csv) for batch scoring from **Telco Customer Churn project**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from uuid import uuid4\n\nbucket_uid = str(uuid4())\n\nscore_filename = \"scoreInput.csv\"\nbuckets = [\"batch-job-\" + bucket_uid]"
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Creating bucket \"batch-job-6595f0b0-182e-4bbd-b795-ea6909a183f5\"...\n"
                }
            ], 
            "source": "for bucket in buckets:\n    if not cos.Bucket(bucket) in cos.buckets.all():\n        print('Creating bucket \"{}\"...'.format(bucket))\n        try:\n            cos.create_bucket(Bucket=bucket)\n        except ibm_boto3.exceptions.ibm_botocore.client.ClientError as e:\n            print('Error: {}.'.format(e.response['Error']['Message']))"
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "data_links = ['https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/customer-satisfaction-prediction/data/scoreInput.csv']"
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Uploading data scoreInput.csv...\nscoreInput.csv is uploaded.\n"
                }
            ], 
            "source": "from urllib import urlopen\n\nbucket_obj = cos.Bucket(buckets[0])\n\nfor data_link in data_links:\n    filename=data_link.split('/')[-1]\n    print('Uploading data {}...'.format(filename))\n    scoring_data_connection = urlopen(data_link)\n    bucket_obj.upload_fileobj(scoring_data_connection, filename)\n    print('{} is uploaded.'.format(filename))"
        }, 
        {
            "source": "### 6.2: Create model deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you can create a batch scoring endpoint. Execute the following sample code that uses the **published Model Id** value to create the scoring endpoint to the Bluemix repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Create access token to Watson Machine Learning Service\n\nTo work with the Watson Machine Leraning REST API you must generate an access token. To do that you can use the following sample code:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "First, you must import standard python libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import urllib3, requests, json\n\nheaders = urllib3.util.make_headers(basic_auth='{username}:{password}'.format(username=wml_credentials['username'], password=wml_credentials['password']))\nurl = '{url}/v3/identity/token'.format(url=wml_credentials['url'])\nresponse = requests.get(url, headers=headers)\nmltoken = json.loads(response.text).get('token')"
        }, 
        {
            "source": "### 6.1: Create batch deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Get published_models url from instance details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<Response [200]>\n\"{\\\"metadata\\\":{\\\"guid\\\":\\\"37c88e6f-2a51-43f8-be5a-17a928580075\\\",\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075\\\",\\\"created_at\\\":\\\"2018-01-04T14:17:28.939Z\\\",\\\"modified_at\\\":\\\"2018-02-15T09:24:19.976Z\\\"},\\\"entity\\\":{\\\"source\\\":\\\"Bluemix\\\",\\\"published_models\\\":{\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models\\\"},\\\"usage\\\":{\\\"expiration_date\\\":\\\"2018-03-01T00:00:00.000Z\\\",\\\"computation_time\\\":{\\\"limit\\\":18000,\\\"current\\\":0},\\\"model_count\\\":{\\\"limit\\\":200,\\\"current\\\":125},\\\"prediction_count\\\":{\\\"limit\\\":5000,\\\"current\\\":0},\\\"deployment_count\\\":{\\\"limit\\\":5,\\\"current\\\":0}},\\\"plan_id\\\":\\\"3f6acf43-ede8-413a-ac69-f8af3bb0cbfe\\\",\\\"status\\\":\\\"Active\\\",\\\"organization_guid\\\":\\\"b0e61605-a82e-4f03-9e9f-2767973c084d\\\",\\\"region\\\":\\\"us-south\\\",\\\"account\\\":{\\\"id\\\":\\\"f52968f3dbbe7b0b53e15743d45e5e90\\\",\\\"name\\\":\\\"Umit Cakmak's Account\\\",\\\"type\\\":\\\"TRIAL\\\"},\\\"owner\\\":{\\\"ibm_id\\\":\\\"31000292EV\\\",\\\"email\\\":\\\"umit.cakmak@pl.ibm.com\\\",\\\"user_id\\\":\\\"43e0ee0e-6bfb-48fc-bcd8-d61e40d19253\\\",\\\"country_code\\\":\\\"POL\\\",\\\"beta_user\\\":true},\\\"deployments\\\":{\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/deployments\\\"},\\\"space_guid\\\":\\\"4c55eb1c-d6fe-4f0a-9390-35c9a7ecf27a\\\",\\\"plan\\\":\\\"lite\\\"}}\"\n"
                }
            ], 
            "source": "endpoint_instance = wml_credentials['url'] + \"/v3/wml_instances/\" + wml_credentials['instance_id']\nheader = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + mltoken}\n\nresponse_get_instance = requests.get(endpoint_instance, headers=header)\n\nprint(response_get_instance)\nprint(json.dumps(response_get_instance.text, indent=2))"
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models\n"
                }
            ], 
            "source": "endpoint_published_models = json.loads(response_get_instance.text).get('entity').get('published_models').get('url')\n\nprint(endpoint_published_models)"
        }, 
        {
            "source": "Execute the following sample code that uses the published_models endpoint to get deployments url.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Get the list of published models", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<Response [200]>\n\"{\\\"count\\\":1,\\\"resources\\\":[{\\\"metadata\\\":{\\\"guid\\\":\\\"b7655b89-657a-4f88-bf80-0478bcd2a678\\\",\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678\\\",\\\"created_at\\\":\\\"2018-02-15T09:20:42.995Z\\\",\\\"modified_at\\\":\\\"2018-02-15T09:20:43.111Z\\\"},\\\"entity\\\":{\\\"runtime_environment\\\":\\\"spark-2.0\\\",\\\"learning_configuration_url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/learning_configuration\\\",\\\"author\\\":{},\\\"name\\\":\\\"Customer Churn Model\\\",\\\"label_col\\\":\\\"Churn\\\",\\\"learning_iterations_url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/learning_iterations\\\",\\\"training_data_schema\\\":{\\\"fields\\\":[{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"customerID\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"gender\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"integer\\\",\\\"name\\\":\\\"SeniorCitizen\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Partner\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Dependents\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"integer\\\",\\\"name\\\":\\\"tenure\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PhoneService\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"MultipleLines\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"InternetService\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"OnlineSecurity\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"OnlineBackup\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"DeviceProtection\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"TechSupport\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"StreamingTV\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"StreamingMovies\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Contract\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PaperlessBilling\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PaymentMethod\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"double\\\",\\\"name\\\":\\\"MonthlyCharges\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"double\\\",\\\"name\\\":\\\"TotalCharges\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Churn\\\",\\\"nullable\\\":true}],\\\"type\\\":\\\"struct\\\"},\\\"feedback_url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/feedback\\\",\\\"latest_version\\\":{\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/ml_assets/models/b7655b89-657a-4f88-bf80-0478bcd2a678/versions/05b29be1-ffdb-4b2a-8832-a05520705f8a\\\",\\\"guid\\\":\\\"05b29be1-ffdb-4b2a-8832-a05520705f8a\\\",\\\"created_at\\\":\\\"2018-02-15T09:20:43.111Z\\\"},\\\"model_type\\\":\\\"mllib-2.0\\\",\\\"deployments\\\":{\\\"count\\\":0,\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/deployments\\\"},\\\"evaluation_metrics_url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/evaluation_metrics\\\",\\\"input_data_schema\\\":{\\\"fields\\\":[{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"customerID\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"gender\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"integer\\\",\\\"name\\\":\\\"SeniorCitizen\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Partner\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Dependents\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"integer\\\",\\\"name\\\":\\\"tenure\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PhoneService\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"MultipleLines\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"InternetService\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"OnlineSecurity\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"OnlineBackup\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"DeviceProtection\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"TechSupport\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"StreamingTV\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"StreamingMovies\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"Contract\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PaperlessBilling\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"string\\\",\\\"name\\\":\\\"PaymentMethod\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"double\\\",\\\"name\\\":\\\"MonthlyCharges\\\",\\\"nullable\\\":true},{\\\"metadata\\\":{},\\\"type\\\":\\\"double\\\",\\\"name\\\":\\\"TotalCharges\\\",\\\"nullable\\\":true}],\\\"type\\\":\\\"struct\\\"}}}]}\"\n"
                }
            ], 
            "source": "header = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + mltoken}\nresponse_get = requests.get(endpoint_published_models, headers=header)\n\nprint(response_get)\nprint(json.dumps(response_get.text, indent=2))"
        }, 
        {
            "source": "#### Get published model deployment url", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/deployments\n"
                }
            ], 
            "source": "[endpoint_deployments] = [x.get('entity').get('deployments').get('url') for x in json.loads(response_get.text).get('resources') if x.get('metadata').get('guid') == saved_model.uid]\n\nprint(endpoint_deployments)"
        }, 
        {
            "source": "#### Create batch deployment for published model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "First import standard python library.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 45, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import base64"
        }, 
        {
            "source": "<div>**Action**:  Put your credentials from Apache Spark Service in Bluemix here</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 102, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "spark_credentials = {\n  \"tenant_id\": \"***\",\n  \"tenant_id_full\": \"***\",\n  \"cluster_master_url\": \"***\",\n  \"tenant_secret\": \"***\",\n  \"instance_id\": \"***\",\n  \"plan\": \"***\"\n}"
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "encoded_spark_credentials_header = base64.b64encode(json.dumps({\"credentials\": spark_credentials, \"version\": \"2.0\"}))\nresult_filename = \"result.csv\""
        }, 
        {
            "execution_count": 103, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<Response [201]>\nINITIALIZING\n"
                }
            ], 
            "source": "header_batch = {'Content-Type': 'application/json', 'Authorization': \"Bearer \" + mltoken, 'X-Spark-Service-Instance': encoded_spark_credentials_header}\npayload_batch = {\"type\":\"batch\", \"name\": \"Customer Churn Prediction\", \"description\": \"Batch Deployment\", \"input\": {\"source\": { \"bucket\": buckets[0], \"filename\": score_filename, \"inferschema\":1,  \"fileformat\": \"csv\", \"type\": \"cloudobjectstorage\"}, \"connection\": {\"access_key\": connection_access_key_id, \"secret_key\": connection_secret_access_key, \"url\": service_endpoint}}, \"output\": { \"target\": { \"bucket\": buckets[0], \"filename\": result_filename, \"fileformat\": \"csv\", \"firstlineheader\":\"true\", \"type\": \"cloudobjectstorage\"}, \"connection\": {\"access_key\": connection_access_key_id, \"secret_key\": connection_secret_access_key, \"url\": service_endpoint}}}\n\nresponse_batch = requests.post(endpoint_deployments, json=payload_batch, headers=header_batch)\n\nprint(response_batch)\nprint(json.loads(response_batch.text).get(\"entity\").get(\"status\"))"
        }, 
        {
            "source": "You can see that your job has been submited to Spark.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Monitor job execution", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Here you can check status of your batch scoring. When status of Spark job is `FINNISHED` the results will be written to result.csv file in Object Storage.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 104, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<Response [200]>\n\"{\\\"count\\\":1,\\\"resources\\\":[{\\\"metadata\\\":{\\\"guid\\\":\\\"2a62e2b0-3504-4a08-ae35-9090eb9e6c3a\\\",\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678/deployments/2a62e2b0-3504-4a08-ae35-9090eb9e6c3a\\\",\\\"created_at\\\":\\\"2018-02-15T10:17:11.637Z\\\",\\\"modified_at\\\":\\\"2018-02-15T10:17:32.112Z\\\"},\\\"entity\\\":{\\\"runtime_environment\\\":\\\"spark-2.0\\\",\\\"name\\\":\\\"Customer Churn Prediction\\\",\\\"description\\\":\\\"Batch Deployment\\\",\\\"published_model\\\":{\\\"author\\\":{},\\\"name\\\":\\\"Customer Churn Model\\\",\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/37c88e6f-2a51-43f8-be5a-17a928580075/published_models/b7655b89-657a-4f88-bf80-0478bcd2a678\\\",\\\"guid\\\":\\\"b7655b89-657a-4f88-bf80-0478bcd2a678\\\",\\\"created_at\\\":\\\"2018-02-15T09:25:53.441Z\\\"},\\\"model_type\\\":\\\"mllib-2.0\\\",\\\"status\\\":\\\"COMPLETED\\\",\\\"status_message\\\":\\\"SUCCESS\\\",\\\"output\\\":{\\\"connection\\\":{\\\"access_key\\\":\\\"fe8d4a595ab8432bb8744707050be508\\\",\\\"url\\\":\\\"https://s3-api.us-geo.objectstorage.softlayer.net\\\",\\\"secret_key\\\":\\\"856554fa14c7526a38a226aa023ce871e176eb1d6c041f31\\\"},\\\"target\\\":{\\\"fileformat\\\":\\\"csv\\\",\\\"firstlineheader\\\":\\\"true\\\",\\\"filename\\\":\\\"result.csv\\\",\\\"bucket\\\":\\\"batch-job-6595f0b0-182e-4bbd-b795-ea6909a183f5\\\",\\\"type\\\":\\\"cloudobjectstorage\\\"}},\\\"type\\\":\\\"batch\\\",\\\"deployed_version\\\":{\\\"url\\\":\\\"https://ibm-watson-ml.mybluemix.net/v3/ml_assets/models/b7655b89-657a-4f88-bf80-0478bcd2a678/versions/05b29be1-ffdb-4b2a-8832-a05520705f8a\\\",\\\"guid\\\":\\\"05b29be1-ffdb-4b2a-8832-a05520705f8a\\\"},\\\"spark_service\\\":{\\\"credentials\\\":{\\\"tenant_id\\\":\\\"s4b8-61626aad40be90-35c9a7ecf27a\\\",\\\"cluster_master_url\\\":\\\"https://spark.bluemix.net\\\",\\\"tenant_id_full\\\":\\\"bc55d78e-aa17-41e6-b4b8-61626aad40be_4c55eb1c-d6fe-4f0a-9390-35c9a7ecf27a\\\",\\\"tenant_secret\\\":\\\"4ece7b65-8683-4327-986f-1be19c7c693d\\\",\\\"instance_id\\\":\\\"bc55d78e-aa17-41e6-b4b8-61626aad40be\\\",\\\"plan\\\":\\\"ibm.SparkService.PayGoPersonal\\\"},\\\"version\\\":\\\"2.0\\\"},\\\"input\\\":{\\\"source\\\":{\\\"fileformat\\\":\\\"csv\\\",\\\"inferschema\\\":1,\\\"filename\\\":\\\"scoreInput.csv\\\",\\\"bucket\\\":\\\"batch-job-6595f0b0-182e-4bbd-b795-ea6909a183f5\\\",\\\"type\\\":\\\"cloudobjectstorage\\\"},\\\"connection\\\":{\\\"access_key\\\":\\\"fe8d4a595ab8432bb8744707050be508\\\",\\\"url\\\":\\\"https://s3-api.us-geo.objectstorage.softlayer.net\\\",\\\"secret_key\\\":\\\"856554fa14c7526a38a226aa023ce871e176eb1d6c041f31\\\"}}}}]}\"\n"
                }
            ], 
            "source": "header_monitor_job = {'Content-Type': 'application/json', 'Authorization': \"Bearer \" + mltoken, 'X-Spark-Service-Instance': encoded_spark_credentials_header}\n\nresponse_monitor_job = requests.get(endpoint_deployments, headers=header_monitor_job)\n\nprint(response_monitor_job)\nprint(json.dumps(response_monitor_job.text, indent=2))"
        }, 
        {
            "source": "**Preview scored data**", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will load scored data.\n\nGet data with scores from Cloud Object Storage.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 105, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "bucket_obj.download_file(Key='result.csv',Filename='result.csv')"
        }, 
        {
            "source": "'result.csv' file is downloaded to your local file directory", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 106, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "result.csv\r\n"
                }
            ], 
            "source": "ls res*"
        }, 
        {
            "source": "Loading 'result.csv' file as pandas dataframe", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 109, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import pandas as pd\nresults_pdf = pd.read_csv('result.csv')"
        }, 
        {
            "source": "Preview the results by calling the *head()* method on the predictions in Pandas data frame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 110, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 110, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>customerID</th>\n      <th>gender</th>\n      <th>SeniorCitizen</th>\n      <th>Partner</th>\n      <th>Dependents</th>\n      <th>tenure</th>\n      <th>PhoneService</th>\n      <th>MultipleLines</th>\n      <th>InternetService</th>\n      <th>OnlineSecurity</th>\n      <th>...</th>\n      <th>PaymentMethod</th>\n      <th>MonthlyCharges</th>\n      <th>TotalCharges</th>\n      <th>Churn</th>\n      <th>SampleWeight</th>\n      <th>features</th>\n      <th>label</th>\n      <th>rawPrediction</th>\n      <th>probability</th>\n      <th>prediction</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>9237-HQITU</td>\n      <td>Female</td>\n      <td>0</td>\n      <td>No</td>\n      <td>No</td>\n      <td>2</td>\n      <td>Yes</td>\n      <td>No</td>\n      <td>Fiber optic</td>\n      <td>No</td>\n      <td>...</td>\n      <td>Electronic check</td>\n      <td>70.7</td>\n      <td>151.65</td>\n      <td>Yes</td>\n      <td>1.000</td>\n      <td>(30,[2,3,4,5,6,8,10,12,14,16,18,20,22,24,25,28...</td>\n      <td>1</td>\n      <td>[-0.6683985606091145,0.6683985606091145]</td>\n      <td>[0.3388555225599606,0.6611444774400395]</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3638-WEABW</td>\n      <td>Female</td>\n      <td>0</td>\n      <td>Yes</td>\n      <td>No</td>\n      <td>58</td>\n      <td>Yes</td>\n      <td>Yes</td>\n      <td>DSL</td>\n      <td>No</td>\n      <td>...</td>\n      <td>Credit card (automatic)</td>\n      <td>59.9</td>\n      <td>3505.10</td>\n      <td>No</td>\n      <td>2.768</td>\n      <td>(30,[3,4,5,7,9,10,13,14,17,18,20,23,24,28,29],...</td>\n      <td>0</td>\n      <td>[3.94407920167498,-3.94407920167498]</td>\n      <td>[0.9809989882724504,0.019001011727549564]</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>8665-UTDHZ</td>\n      <td>Male</td>\n      <td>0</td>\n      <td>Yes</td>\n      <td>Yes</td>\n      <td>1</td>\n      <td>No</td>\n      <td>No phone service</td>\n      <td>DSL</td>\n      <td>No</td>\n      <td>...</td>\n      <td>Electronic check</td>\n      <td>30.2</td>\n      <td>30.20</td>\n      <td>Yes</td>\n      <td>1.000</td>\n      <td>(30,[0,4,9,10,13,14,16,18,20,22,25,28,29],[1.0...</td>\n      <td>1</td>\n      <td>[-0.09214192316229042,0.09214192316229042]</td>\n      <td>[0.4769808032441,0.5230191967558999]</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8773-HHUOZ</td>\n      <td>Female</td>\n      <td>0</td>\n      <td>No</td>\n      <td>Yes</td>\n      <td>17</td>\n      <td>Yes</td>\n      <td>No</td>\n      <td>DSL</td>\n      <td>No</td>\n      <td>...</td>\n      <td>Mailed check</td>\n      <td>64.7</td>\n      <td>1093.10</td>\n      <td>Yes</td>\n      <td>1.000</td>\n      <td>(30,[2,4,5,6,9,10,12,14,16,19,21,22,24,26,28,2...</td>\n      <td>1</td>\n      <td>[0.747504546477989,-0.747504546477989]</td>\n      <td>[0.6786347093927774,0.32136529060722263]</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4080-IIARD</td>\n      <td>Female</td>\n      <td>0</td>\n      <td>Yes</td>\n      <td>No</td>\n      <td>13</td>\n      <td>Yes</td>\n      <td>Yes</td>\n      <td>DSL</td>\n      <td>Yes</td>\n      <td>...</td>\n      <td>Electronic check</td>\n      <td>76.2</td>\n      <td>981.45</td>\n      <td>No</td>\n      <td>2.768</td>\n      <td>(30,[3,4,5,7,9,11,13,14,17,19,20,22,24,25,28,2...</td>\n      <td>0</td>\n      <td>[0.5917942689227103,-0.5917942689227103]</td>\n      <td>[0.6437767283180145,0.3562232716819854]</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows \u00d7 27 columns</p>\n</div>", 
                        "text/plain": "   customerID  gender  SeniorCitizen Partner Dependents  tenure PhoneService  \\\n0  9237-HQITU  Female              0      No         No       2          Yes   \n1  3638-WEABW  Female              0     Yes         No      58          Yes   \n2  8665-UTDHZ    Male              0     Yes        Yes       1           No   \n3  8773-HHUOZ  Female              0      No        Yes      17          Yes   \n4  4080-IIARD  Female              0     Yes         No      13          Yes   \n\n      MultipleLines InternetService OnlineSecurity     ...      \\\n0                No     Fiber optic             No     ...       \n1               Yes             DSL             No     ...       \n2  No phone service             DSL             No     ...       \n3                No             DSL             No     ...       \n4               Yes             DSL            Yes     ...       \n\n             PaymentMethod MonthlyCharges TotalCharges Churn SampleWeight  \\\n0         Electronic check           70.7       151.65   Yes        1.000   \n1  Credit card (automatic)           59.9      3505.10    No        2.768   \n2         Electronic check           30.2        30.20   Yes        1.000   \n3             Mailed check           64.7      1093.10   Yes        1.000   \n4         Electronic check           76.2       981.45    No        2.768   \n\n                                            features label  \\\n0  (30,[2,3,4,5,6,8,10,12,14,16,18,20,22,24,25,28...     1   \n1  (30,[3,4,5,7,9,10,13,14,17,18,20,23,24,28,29],...     0   \n2  (30,[0,4,9,10,13,14,16,18,20,22,25,28,29],[1.0...     1   \n3  (30,[2,4,5,6,9,10,12,14,16,19,21,22,24,26,28,2...     1   \n4  (30,[3,4,5,7,9,11,13,14,17,19,20,22,24,25,28,2...     0   \n\n                                rawPrediction  \\\n0    [-0.6683985606091145,0.6683985606091145]   \n1        [3.94407920167498,-3.94407920167498]   \n2  [-0.09214192316229042,0.09214192316229042]   \n3      [0.747504546477989,-0.747504546477989]   \n4    [0.5917942689227103,-0.5917942689227103]   \n\n                                 probability  prediction  \n0    [0.3388555225599606,0.6611444774400395]           1  \n1  [0.9809989882724504,0.019001011727549564]           0  \n2       [0.4769808032441,0.5230191967558999]           1  \n3   [0.6786347093927774,0.32136529060722263]           0  \n4    [0.6437767283180145,0.3562232716819854]           0  \n\n[5 rows x 27 columns]"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "results_pdf.head()"
        }, 
        {
            "source": "You can see the predictions for all the records in pandas data frame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 7. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": " You successfully completed this notebook! You learned how to use Apache Spark machine learning as well as Watson Machine Learning for model creation and deployment. Check out our _[Online Documentation](https://console.ng.bluemix.net/docs/services/PredictiveModeling/index.html?pos=2)_ for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Authors\n\n**Umit Cakmak**, Data Scientist in IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5 with Spark 2.1", 
            "name": "python3-spark21", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "2.7.11", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython2", 
            "codemirror_mode": {
                "version": 2, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}