From spark ml model to online scoring.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Predict outdoor equipment purchase with `watson-machine-learning-client`</b></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr>\n   <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/blob/master/spark/product-line-prediction/images/products_graphics.png?raw=true\" alt=\"Icon\"> </th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to get data from the IBM Data Science Experience Community, create a predictive model, and start scoring new data. This notebook introduces commands for getting data and for basic data cleaning and exploration, pipeline creation, model training, model persistance to Watson Machine Learning repository, model deployment, and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.5 and Apache\u00ae Spark 2.1.\n\nYou will use a publicly available data set, **GoSales Transactions for Naive Bayes Model**, which details anonymous outdoor equipment purchases. Use the details of this data set to predict clients' interests in terms of product line, such as golf accessories, camping equipment, and others.\n\n**Note**: In this notebook we use GOSales data available on [Data Science Experience Community](https://apsportal.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600).\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Load a CSV file into an Apache\u00ae Spark DataFrame.\n-  Explore data.\n-  Prepare data for training and evaluation.\n-  Create an Apache\u00ae Spark machine learning pipeline.\n-  Train and evaluate a model.\n-  Persist a pipeline and model in Watson Machine Learning repository.\n-  Deploy a model for online scoring using Wastson Machine Learning API.\n-  Score sample scoring data using the Watson Machine Learning API.\n-  Explore and visualize prediction result using the plotly package.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Setup](#setup)\n2.\t[Load and explore data](#load)\n3.\t[Create spark ml model](#model)\n4.\t[Persist model](#persistence)\n5.\t[Predict locally and visualize](#visualization)\n6.\t[Deploy and score in a Cloud](#scoring)\n7.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Setup\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered). \n-  Upload **GoSales Transactions** data as a data asset in IBM Data Science Experience.\n-  Make sure that you are using a Spark 2.0 kernel.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 2. Load and explore data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will load the data as an Apache\u00ae Spark DataFrame and perform a basic exploration.\n\nLoad the data to the Spark DataFrame by using *wget* to upload the data to gpfs and then *read* method. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**: First, you need to install required packages. You can do it by running the following code. Run it only one time.<BR><BR>\n!pip install wget --user <BR>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "GoSales_Tx_NaiveBayes.csv\n"
                }
            ], 
            "source": "import wget\n\nlink_to_data = 'https://apsportal.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600'\nfilename = wget.download(link_to_data)\n\nprint(filename)"
        }, 
        {
            "source": "The csv file GoSales_Tx_NaiveBayes.csv is availble on gpfs now. Load the file to Apache\u00ae Spark DataFrame using code below.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "spark = SparkSession.builder.getOrCreate()\n\ndf_data = spark.read\\\n  .format('org.apache.spark.sql.execution.datasources.csv.CSVFileFormat')\\\n  .option('header', 'true')\\\n  .option('inferSchema', 'true')\\\n  .load(filename)"
        }, 
        {
            "source": "Explore the loaded data by using the following Apache\u00ae Spark DataFrame methods:\n-  print schema\n-  print top ten records\n-  count all records", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "root\n |-- PRODUCT_LINE: string (nullable = true)\n |-- GENDER: string (nullable = true)\n |-- AGE: integer (nullable = true)\n |-- MARITAL_STATUS: string (nullable = true)\n |-- PROFESSION: string (nullable = true)\n\n"
                }
            ], 
            "source": "df_data.printSchema()"
        }, 
        {
            "source": "As you can see, the data contains five fields. PRODUCT_LINE field is the one we would like to predict (label).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+--------------------+------+---+--------------+------------+\n|        PRODUCT_LINE|GENDER|AGE|MARITAL_STATUS|  PROFESSION|\n+--------------------+------+---+--------------+------------+\n|Personal Accessories|     M| 27|        Single|Professional|\n|Personal Accessories|     F| 39|       Married|       Other|\n|Mountaineering Eq...|     F| 39|       Married|       Other|\n|Personal Accessories|     F| 56|   Unspecified| Hospitality|\n|      Golf Equipment|     M| 45|       Married|     Retired|\n|      Golf Equipment|     M| 45|       Married|     Retired|\n|   Camping Equipment|     F| 39|       Married|       Other|\n|   Camping Equipment|     F| 49|       Married|       Other|\n|  Outdoor Protection|     F| 49|       Married|       Other|\n|      Golf Equipment|     M| 47|       Married|     Retired|\n|      Golf Equipment|     M| 47|       Married|     Retired|\n|Mountaineering Eq...|     M| 21|        Single|      Retail|\n|Personal Accessories|     F| 66|       Married|       Other|\n|   Camping Equipment|     F| 35|       Married|Professional|\n|Mountaineering Eq...|     M| 20|        Single|       Sales|\n|Mountaineering Eq...|     M| 20|        Single|       Sales|\n|Mountaineering Eq...|     M| 20|        Single|       Sales|\n|Personal Accessories|     F| 37|        Single|       Other|\n|   Camping Equipment|     M| 42|       Married|       Other|\n|   Camping Equipment|     F| 24|       Married|      Retail|\n+--------------------+------+---+--------------+------------+\nonly showing top 20 rows\n\n"
                }
            ], 
            "source": "df_data.show()"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "print(\"Number of records: \" + str(df_data.count()))"
        }, 
        {
            "source": "As you can see, the data set contains 60252 records.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 3. Create an Apache\u00ae Spark machine learning model\n\nIn this section you will learn how to prepare data, create an Apache\u00ae Spark machine learning pipeline, and train a model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1: Prepare data\n\nIn this subsection you will split your data into: train, test and predict datasets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 17, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of training records: 48176\nNumber of testing records : 10860\nNumber of prediction records : 1216\n"
                }
            ], 
            "source": "splitted_data = df_data.randomSplit([0.8, 0.18, 0.02], 24)\ntrain_data = splitted_data[0]\ntest_data = splitted_data[1]\npredict_data = splitted_data[2]\n\nprint(\"Number of training records: \" + str(train_data.count()))\nprint(\"Number of testing records : \" + str(test_data.count()))\nprint(\"Number of prediction records : \" + str(predict_data.count()))"
        }, 
        {
            "source": "As you can see our data has been successfully split into three datasets: \n\n-  The train data set, which is the largest group, is used for training.\n-  The test data set will be used for model evaluation and is used to test the assumptions of the model.\n-  The predict data set will be used for prediction.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2: Create pipeline and train a model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will create an Apache\u00ae Spark machine learning pipeline and then train the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In the first step you need to import the Apache\u00ae Spark machine learning packages that will be needed in the subsequent steps.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 18, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\nfrom pyspark.ml.classification import RandomForestClassifier\nfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator\nfrom pyspark.ml import Pipeline, Model"
        }, 
        {
            "source": "In the following step, convert all the string fields to numeric ones by using the StringIndexer transformer.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 19, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "stringIndexer_label = StringIndexer(inputCol=\"PRODUCT_LINE\", outputCol=\"label\").fit(df_data)\nstringIndexer_prof = StringIndexer(inputCol=\"PROFESSION\", outputCol=\"PROFESSION_IX\")\nstringIndexer_gend = StringIndexer(inputCol=\"GENDER\", outputCol=\"GENDER_IX\")\nstringIndexer_mar = StringIndexer(inputCol=\"MARITAL_STATUS\", outputCol=\"MARITAL_STATUS_IX\")"
        }, 
        {
            "source": "In the following step, create a feature vector by combining all features together.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 20, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "vectorAssembler_features = VectorAssembler(inputCols=[\"GENDER_IX\", \"AGE\", \"MARITAL_STATUS_IX\", \"PROFESSION_IX\"], outputCol=\"features\")"
        }, 
        {
            "source": "Next, define estimators you want to use for classification. Random Forest is used in the following example.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 21, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "rf = RandomForestClassifier(labelCol=\"label\", featuresCol=\"features\")"
        }, 
        {
            "source": "Finally, indexed labels back to original labels.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 22, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "labelConverter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=stringIndexer_label.labels)"
        }, 
        {
            "source": "Let's build the pipeline now. A pipeline consists of transformers and an estimator.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 23, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "pipeline_rf = Pipeline(stages=[stringIndexer_label, stringIndexer_prof, stringIndexer_gend, stringIndexer_mar, vectorAssembler_features, rf, labelConverter])"
        }, 
        {
            "source": "Now, you can train your Random Forest model by using the previously defined **pipeline** and **train data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 24, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "root\n |-- PRODUCT_LINE: string (nullable = true)\n |-- GENDER: string (nullable = true)\n |-- AGE: integer (nullable = true)\n |-- MARITAL_STATUS: string (nullable = true)\n |-- PROFESSION: string (nullable = true)\n\n"
                }
            ], 
            "source": "train_data.printSchema()"
        }, 
        {
            "execution_count": 25, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_rf = pipeline_rf.fit(train_data)"
        }, 
        {
            "source": "You can check your **model accuracy** now. To evaluate the model, use **test data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy = 0.585359\nTest Error = 0.414641\n"
                }
            ], 
            "source": "predictions = model_rf.transform(test_data)\nevaluatorRF = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\", metricName=\"accuracy\")\naccuracy = evaluatorRF.evaluate(predictions)\n\nprint(\"Accuracy = %g\" % accuracy)\nprint(\"Test Error = %g\" % (1.0 - accuracy))"
        }, 
        {
            "source": "You can tune your model now to achieve better accuracy. For simplicity of this example tuning section is omitted.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 4. Persist model", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "In this section you will learn how to store your pipeline and model in Watson Machine Learning repository by using python client libraries.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "First, you must install and import `watson-machine-learning-client` libraries.\n\n**Note**: Python 3.5 and Apache\u00ae Spark 2.1 is required.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement already up-to-date: watson-machine-learning-client in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages\nRequirement already up-to-date: tqdm in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: tabulate in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: urllib3 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: certifi in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: pandas in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: lomond in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: requests in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: python-dateutil>=2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: numpy>=1.9.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: pytz>=2011k in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: six>=1.10.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement already up-to-date: idna<2.7,>=2.5 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\n"
                }
            ], 
            "source": "!pip install watson-machine-learning-client --upgrade"
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "source": "Authenticate to Watson Machine Learning service on Cloud.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "**Action**: Put authentication information from your instance of Watson Machine Learning service here.</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials={\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"***\",\n  \"username\": \"***\",\n  \"password\": \"***\",\n  \"instance_id\": \"***\"\n}"
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "**Tip**: credentials can be found on **Service Credentials** tab of service instance created in Bluemix. If you cannot see **instance_id** field in **Serice Credentials** generate new credentials by pressing **New credential (+)** button. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "### 4.1: Save pipeline and model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will learn how to save pipeline and model artifacts to your Watson Machine Learning instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "published_model_details = client.repository.store_model(model=model_rf, meta_props={'name':'Product line model'}, training_data=train_data, pipeline=pipeline_rf)"
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "70a8b8ad-ef04-4d9f-a1e7-679719577b62\n"
                }
            ], 
            "source": "model_uid = client.repository.get_model_uid(published_model_details)\nprint(model_uid)"
        }, 
        {
            "source": "Get saved model metadata from Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip**: Use `client.repository.ModelMetaNames.show()` to get the list of available props.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "-----------------------  ----  --------\nMETA_PROP NAME           TYPE  REQUIRED\nNAME                     str   Y\nDESCRIPTION              str   N\nAUTHOR_NAME              str   N\nAUTHOR_EMAIL             str   N\nFRAMEWORK_NAME           str   N\nFRAMEWORK_VERSION        str   N\nFRAMEWORK_LIBRARIES      list  N\nRUNTIME_NAME             str   N\nRUNTIME_VERSION          str   N\nTRAINING_DATA_REFERENCE  dict  N\nEVALUATION_METHOD        str   N\nEVALUATION_METRICS       list  N\n-----------------------  ----  --------\n"
                }
            ], 
            "source": "client.repository.ModelMetaNames.show()"
        }, 
        {
            "source": "### 4.2: Load model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will learn how to load back saved model from specified instance of Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "loaded_model = client.repository.load(model_uid)"
        }, 
        {
            "source": "You can print for example model name to make sure that model has been loaded correctly.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "<class 'pyspark.ml.pipeline.PipelineModel'>\n"
                }
            ], 
            "source": "print(type(loaded_model))"
        }, 
        {
            "source": "<a id=\"visualization\"></a>\n## 5. Predict locally and visualize", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to score test data using loaded model and visualize the prediction results with plotly package.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.1: Make local prediction using previously loaded model and test data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will score *predict_data* data set.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "predictions = loaded_model.transform(predict_data)"
        }, 
        {
            "source": "Preview the results by calling the *show()* method on the predictions DataFrame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+-----------------+------+---+--------------+----------+-----+-------------+---------+-----------------+------------------+--------------------+--------------------+----------+--------------------+\n|     PRODUCT_LINE|GENDER|AGE|MARITAL_STATUS|PROFESSION|label|PROFESSION_IX|GENDER_IX|MARITAL_STATUS_IX|          features|       rawPrediction|         probability|prediction|      predictedLabel|\n+-----------------+------+---+--------------+----------+-----+-------------+---------+-----------------+------------------+--------------------+--------------------+----------+--------------------+\n|Camping Equipment|     F| 20|        Single|     Other|  0.0|          0.0|      1.0|              1.0|[1.0,20.0,1.0,0.0]|[5.40697857263576...|[0.27034892863178...|       1.0|Personal Accessories|\n|Camping Equipment|     F| 20|        Single|     Other|  0.0|          0.0|      1.0|              1.0|[1.0,20.0,1.0,0.0]|[5.40697857263576...|[0.27034892863178...|       1.0|Personal Accessories|\n|Camping Equipment|     F| 20|        Single|     Other|  0.0|          0.0|      1.0|              1.0|[1.0,20.0,1.0,0.0]|[5.40697857263576...|[0.27034892863178...|       1.0|Personal Accessories|\n|Camping Equipment|     F| 20|        Single|     Other|  0.0|          0.0|      1.0|              1.0|[1.0,20.0,1.0,0.0]|[5.40697857263576...|[0.27034892863178...|       1.0|Personal Accessories|\n|Camping Equipment|     F| 21|       Married|     Other|  0.0|          0.0|      1.0|              0.0|[1.0,21.0,0.0,0.0]|[7.86046445392534...|[0.39302322269626...|       0.0|   Camping Equipment|\n+-----------------+------+---+--------------+----------+-----+-------------+---------+-----------------+------------------+--------------------+--------------------+----------+--------------------+\nonly showing top 5 rows\n\n"
                }
            ], 
            "source": "predictions.show(5)"
        }, 
        {
            "source": "By tabulating a count, you can see which product line is the most popular.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+------------------------+-----+\n|predictedLabel          |count|\n+------------------------+-----+\n|Camping Equipment       |752  |\n|Golf Equipment          |67   |\n|Mountaineering Equipment|42   |\n|Personal Accessories    |355  |\n+------------------------+-----+\n\n"
                }
            ], 
            "source": "predictions.select(\"predictedLabel\").groupBy(\"predictedLabel\").count().show(truncate=False)"
        }, 
        {
            "source": "### 5.2: Sample visualization of data with Plotly package", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will explore prediction results with Plotly, which is an online analytics and data visualization tool.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**: First, you need to install required packages. You can do it by running the following code. Run it only one time.<BR><BR>\n!pip install plotly --user <BR>\n!pip install cufflinks==0.8.2 --user", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Import Plotly and other required packages.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>", 
                        "text/vnd.plotly.v1+html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "import sys\nimport pandas\nimport plotly.plotly as py\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\nimport plotly.graph_objs as go\n\ninit_notebook_mode(connected=True)\nsys.path.append(\"\".join([os.environ[\"HOME\"]])) "
        }, 
        {
            "source": "Convert the Apache Spark DataFrame to a Pandas DataFrame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "predictions_pdf = predictions.select(\"prediction\", \"predictedLabel\", \"GENDER\", \"AGE\", \"PROFESSION\", \"MARITAL_STATUS\").toPandas()"
        }, 
        {
            "source": "Plot a pie chart that shows the predicted product-line interest.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 45, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"615bee7e-8e44-4bfe-bbe9-d6efe0923ea0\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"615bee7e-8e44-4bfe-bbe9-d6efe0923ea0\", [{\"type\": \"pie\", \"values\": [752, 67, 42, 355], \"labels\": [\"Camping Equipment\", \"Golf Equipment\", \"Mountaineering Equipment\", \"Personal Accessories\"]}], {\"title\": \"Predicted product line client interest distribution\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"615bee7e-8e44-4bfe-bbe9-d6efe0923ea0\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"615bee7e-8e44-4bfe-bbe9-d6efe0923ea0\", [{\"type\": \"pie\", \"values\": [752, 67, 42, 355], \"labels\": [\"Camping Equipment\", \"Golf Equipment\", \"Mountaineering Equipment\", \"Personal Accessories\"]}], {\"title\": \"Predicted product line client interest distribution\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "type": "pie", 
                                    "labels": [
                                        "Camping Equipment", 
                                        "Golf Equipment", 
                                        "Mountaineering Equipment", 
                                        "Personal Accessories"
                                    ], 
                                    "values": [
                                        752, 
                                        67, 
                                        42, 
                                        355
                                    ]
                                }
                            ], 
                            "layout": {
                                "title": "Predicted product line client interest distribution"
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "cumulative_stats = predictions_pdf.groupby(['predictedLabel']).count()\nproduct_data = [go.Pie(labels=cumulative_stats.index, values=cumulative_stats['GENDER'])]\nproduct_layout = go.Layout(title='Predicted product line client interest distribution')\n\nfig = go.Figure(data=product_data, layout=product_layout)\niplot(fig)"
        }, 
        {
            "source": "With this data set, you might want to do some analysis of the mean AGE per product line by using a bar chart.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"19a76b92-68a4-44a8-8932-76fd4416a585\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"19a76b92-68a4-44a8-8932-76fd4416a585\", [{\"y\": [32.986702127659576, 50.985074626865675, 39.214285714285715, 32.12112676056338], \"type\": \"bar\", \"x\": [\"Camping Equipment\", \"Golf Equipment\", \"Mountaineering Equipment\", \"Personal Accessories\"]}], {\"title\": \"Mean AGE per predicted product line\", \"xaxis\": {\"title\": \"Product Line\", \"showline\": false}, \"yaxis\": {\"title\": \"Mean AGE\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"19a76b92-68a4-44a8-8932-76fd4416a585\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"19a76b92-68a4-44a8-8932-76fd4416a585\", [{\"y\": [32.986702127659576, 50.985074626865675, 39.214285714285715, 32.12112676056338], \"type\": \"bar\", \"x\": [\"Camping Equipment\", \"Golf Equipment\", \"Mountaineering Equipment\", \"Personal Accessories\"]}], {\"title\": \"Mean AGE per predicted product line\", \"xaxis\": {\"title\": \"Product Line\", \"showline\": false}, \"yaxis\": {\"title\": \"Mean AGE\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "y": [
                                        32.986702127659576, 
                                        50.985074626865675, 
                                        39.214285714285715, 
                                        32.12112676056338
                                    ], 
                                    "x": [
                                        "Camping Equipment", 
                                        "Golf Equipment", 
                                        "Mountaineering Equipment", 
                                        "Personal Accessories"
                                    ], 
                                    "type": "bar"
                                }
                            ], 
                            "layout": {
                                "title": "Mean AGE per predicted product line", 
                                "xaxis": {
                                    "showline": false, 
                                    "title": "Product Line"
                                }, 
                                "yaxis": {
                                    "title": "Mean AGE"
                                }
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "age_data = [go.Bar(y=predictions_pdf.groupby(['predictedLabel']).mean()[\"AGE\"], x=cumulative_stats.index)]\n\nage_layout = go.Layout(\n    title='Mean AGE per predicted product line',\n    xaxis=dict(title = \"Product Line\", showline=False),\n    yaxis=dict(title = \"Mean AGE\"))\n\nfig = go.Figure(data=age_data, layout=age_layout)\niplot(fig)"
        }, 
        {
            "source": "Based on bar plot you created, you might make the following conclusion: The mean age for clients that are interested in golf equipment is predicted to be over 50 years old.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"scoring\"></a>\n## 6. Deploy and score in a Cloud", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to create online scoring and to score a new data record using `watson-machine-learning-client`.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Note:** You can also use REST API to deploy and score.\nFor more information about REST APIs, see the [Swagger Documentation](http://watson-ml-api.mybluemix.net/).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 6.1: Create online scoring endpoint", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now you can create an online scoring endpoint. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Create online deployment for published model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: '3112f5b0-4c72-4e78-ac33-325b2dc38bbe' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='3112f5b0-4c72-4e78-ac33-325b2dc38bbe'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "deployment_details = client.deployments.create(model_uid=model_uid, name='Product line model deployment')"
        }, 
        {
            "execution_count": 49, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/70a8b8ad-ef04-4d9f-a1e7-679719577b62/deployments/3112f5b0-4c72-4e78-ac33-325b2dc38bbe/online\n"
                }
            ], 
            "source": "scoring_url = client.deployments.get_scoring_url(deployment_details)\nprint(scoring_url)"
        }, 
        {
            "source": "Now, you can send new scoring records (new data) for which you would like to get predictions. To do that, execute the following sample code: ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 50, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 50, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'fields': ['GENDER',\n  'AGE',\n  'MARITAL_STATUS',\n  'PROFESSION',\n  'PRODUCT_LINE',\n  'label',\n  'PROFESSION_IX',\n  'GENDER_IX',\n  'MARITAL_STATUS_IX',\n  'features',\n  'rawPrediction',\n  'probability',\n  'prediction',\n  'predictedLabel'],\n 'values': [['M',\n   23,\n   'Single',\n   'Student',\n   'Camping Equipment',\n   0.0,\n   6.0,\n   0.0,\n   1.0,\n   [0.0, 23.0, 1.0, 6.0],\n   [5.258712981131878,\n    6.723421733525897,\n    6.0132018750333724,\n    0.218060296213069,\n    1.7866031140957863],\n   [0.26293564905659383,\n    0.3361710866762948,\n    0.3006600937516686,\n    0.010903014810653448,\n    0.0893301557047893],\n   1.0,\n   'Personal Accessories'],\n  ['M',\n   55,\n   'Single',\n   'Executive',\n   'Camping Equipment',\n   0.0,\n   3.0,\n   0.0,\n   1.0,\n   [0.0, 55.0, 1.0, 3.0],\n   [3.6000096318286547,\n    4.060237825988732,\n    3.8628119611335734,\n    8.339956386992652,\n    0.13698419405638748],\n   [0.18000048159143273,\n    0.20301189129943661,\n    0.19314059805667866,\n    0.4169978193496326,\n    0.006849209702819374],\n   3.0,\n   'Golf Equipment']]}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "payload_scoring = {\"fields\": [\"GENDER\",\"AGE\",\"MARITAL_STATUS\",\"PROFESSION\"],\"values\": [[\"M\",23,\"Single\",\"Student\"],[\"M\",55,\"Single\",\"Executive\"]]}\n\nclient.deployments.score(scoring_url, payload_scoring)"
        }, 
        {
            "source": "As we can see we predict that a 23-year-old male student is interested in personal accessories (predictedLabel: Personal Accessories, prediction: 1.0). We can also see that single, 55 years old man is interested in golf equipment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 7. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": " You successfully completed this notebook! You learned how to use Apache Spark machine learning as well as Watson Machine Learning for model creation and deployment. Check out our _[Online Documentation](https://console.ng.bluemix.net/docs/services/PredictiveModeling/pm_service_api_spark.html)_ for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Authors\n\n**Lukasz Cmielowski**, PhD, is a Automation Architect and Data Scientist in IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5 with Spark 2.1", 
            "name": "python3-spark21", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}