Use learning system to predict best heart drug.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<div style=\"background:#F5F7FA; height:100px; padding: 2em; font-size:14px;\">\n<span style=\"font-size:18px;color:#152935;\">Want to do more?</span><span style=\"border: 1px solid #3d70b2;padding: 15px;float:right;margin-right:40px; color:#3d70b2; \"><a href=\"https://ibm.co/wsnotebooks\" target=\"_blank\" style=\"color: #3d70b2;text-decoration: none;\">Sign Up</a></span><br>\n<span style=\"color:#5A6872;\"> Try out this notebook with your free trial of IBM Watson Studio.</span>\n</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Use learning system to predict best heart drug</b></font></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr> \n   <tr style=\"border: none\">\n       <td style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/raw/master/spark/drug-selection/images/learning_banner-05.png\" width=\"600\" alt=\"Icon\"></td>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to configure **continuous learning system** using wml client, and start scoring new data. This notebook introduces commands for getting data, model persistance to Watson Machine Learning repository, model deployment, continuous learning system configuration and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3 and Apache\u00ae Spark 2.1.\n\nYou will use published on git data set, **drug_feedback_data.csv**, which details anonymous patients records. Use the details of this data set to predict best drug for heart disease treatment.\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Prepare feedback data set in Db2 Warehouse on Cloud on Bluemix.\n-  Publish a sample model in Watson Machine Learning repository.\n-  Configure continuous learning system for published model using Watson Machine Learning API.\n-  Deploy a model for online scoring using Watson Machine Learning API.\n-  Track model performance changes after learning system iteration using Watson Machine Learning API.\n-  Explore and visualize model performance using the plotly package and Watson Machine Learning API.\n\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Setup](#setup)\n2.\t[Create spark ml model](#model)\n3.\t[Persist model](#load)\n4.\t[Configure continuous learning system](#configuration)\n5.\t[Track model performance](#performance)\n6.\t[Visualization of model performance](#visualization)\n7.\t[Feedback records](#visualization)\n8.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Setup\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning (WML) Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered and information about how to create the instance is [here](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html))\n- Create a [Spark Service](https://console.ng.bluemix.net/catalog/services/spark/) instance (an entry plan is offered).\n- Create a [Db2 Warehouse on Cloud Service](https://console.bluemix.net/catalog/services/db2-warehouse-on-cloud/) instance (an entry plan is offered).\n- Create the **DRUG_TRAIN_DATA_UPDATED** and **DRUG_FEEDBACK_DATA** tables in **Db2 Warehouse on Cloud**. \n  + Download [drug_train_data_updated.csv](https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/drug-selection/data/drug_train_data_updated.csv) and [drug_feedback_data.csv](https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/drug-selection/data/drug_feedback_data.csv) files from git repository.\n  + Click the **Open the console** to get started with **Db2 Warehouse on Cloud** icon.\n  + Select the **Load Data** and **Desktop** load type.\n  + **Drag and drop** previously downloaded file and press **Next**.\n  + Select **Schema** to import data and click **New Table**. \n  + Write name for **new table** than click **Next** to finish data import.\n  + Use `;` as **field separator**.\n  + Click **Next** to create table with uploaded data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 2. Create spark ml model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to prepare data, create an Apache\u00ae Spark machine learning pipeline, and train a model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 2.1 Load training data from Db2 Warehouse on Cloud", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Using below code you will load DRUG_TRAIN_DATA_UPDATED table content into Spark Data Frame.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "db2_service_credentials = {\n  \"port\": 50000,\n  \"db\": \"BLUDB\",\n  \"username\": \"***\",\n  \"ssljdbcurl\": \"jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;\",\n  \"host\": \"dashdb-entry-yp-dal10-01.services.dal.bluemix.net\",\n  \"https_url\": \"https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443\",\n  \"dsn\": \"***\",\n  \"hostname\": \"dashdb-entry-yp-dal10-01.services.dal.bluemix.net\",\n  \"jdbcurl\": \"***\",\n  \"ssldsn\": \"***\",\n  \"uri\": \"***\",\n  \"password\": \"***\"\n}"
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "db2_credentials = {\n    'jdbcurl': db2_service_credentials['jdbcurl'],\n    'user': db2_service_credentials['username'],\n    'password': db2_service_credentials['password']\n}"
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "tablename = \"{schema}.{table}\".format(schema=db2_credentials['user'], table='DRUG_TRAIN_DATA_UPDATED')"
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "DRUG_TRAIN_DATA_UPDATED_data = spark.read.jdbc(db2_credentials['jdbcurl'], table=tablename, properties=db2_credentials)"
        }, 
        {
            "source": "**Tip:** All required fields can be found on Service Credentials tab of Db2 Warehouse on Cloud service instance created in IBM Cloud. In case you do not have any credential, you can create one by clicking \"New credential\"", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+---+---+----+-----------+--------+--------+-----+\n|AGE|SEX|  BP|CHOLESTEROL|      NA|       K| DRUG|\n+---+---+----+-----------+--------+--------+-----+\n| 43|  M|HIGH|       HIGH|0.656371|0.046979|drugA|\n| 32|  M|HIGH|     NORMAL|0.529750|0.056087|drugA|\n| 37|  F|HIGH|       HIGH|0.559171|0.042713|drugA|\n| 24|  M|HIGH|     NORMAL|0.613261|0.064726|drugA|\n| 29|  M|HIGH|       HIGH|0.625272|0.048637|drugA|\n+---+---+----+-----------+--------+--------+-----+\nonly showing top 5 rows\n\n"
                }
            ], 
            "source": "DRUG_TRAIN_DATA_UPDATED_data.show(5)"
        }, 
        {
            "source": "DRUG column is the target/label column.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 2.2: Prepare data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will split your data into: train and test datasets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of records for training: 150\nNumber of records for evaluation: 31\n"
                }
            ], 
            "source": "(train_data, test_data) = DRUG_TRAIN_DATA_UPDATED_data.randomSplit([0.8, 0.2], 24)\n\nprint(\"Number of records for training: \" + str(train_data.count()))\nprint(\"Number of records for evaluation: \" + str(test_data.count()))"
        }, 
        {
            "source": "As you can see our data has been successfully split into two datasets:\n - The train data set, which is the largest group, is used for training.\n - The test data set will be used for model evaluation.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 2.3: Create pipeline and train a model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will create an Apache\u00ae Spark machine learning pipeline and then train the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In the first step you need to import the Apache\u00ae Spark machine learning packages that will be needed in the subsequent steps.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\nfrom pyspark.ml.classification import DecisionTreeClassifier\nfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator\nfrom pyspark.ml import Pipeline, Model"
        }, 
        {
            "source": "In the following step, convert all the string fields to numeric ones by using the StringIndexer transformer.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "stringIndexer_sex = StringIndexer(inputCol = 'SEX', outputCol = 'SEX_IX')\nstringIndexer_bp = StringIndexer(inputCol = 'BP', outputCol = 'BP_IX')\nstringIndexer_chol = StringIndexer(inputCol = 'CHOLESTEROL', outputCol = 'CHOL_IX')\nstringIndexer_label = StringIndexer(inputCol=\"DRUG\", outputCol=\"label\").fit(DRUG_TRAIN_DATA_UPDATED_data)"
        }, 
        {
            "source": "In the following step, create a feature vector by combining all features together.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "vectorAssembler_features = VectorAssembler(inputCols=[\"AGE\", \"SEX_IX\", \"BP_IX\", \"CHOL_IX\", \"NA\", \"K\"], outputCol=\"features\")"
        }, 
        {
            "source": "Next, define estimators you want to use for classification. Decision Tree is used in the following example.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 17, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "dt = DecisionTreeClassifier(labelCol=\"label\", featuresCol=\"features\")"
        }, 
        {
            "source": "Finally, indexed labels back to original labels.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 18, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "labelConverter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=stringIndexer_label.labels)"
        }, 
        {
            "source": "Let's build the pipeline now. A pipeline consists of transformers and an estimator.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 19, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "pipeline_dt = Pipeline(stages=[stringIndexer_label, stringIndexer_sex, stringIndexer_bp, stringIndexer_chol, vectorAssembler_features, dt, labelConverter])"
        }, 
        {
            "source": "Now, you can train your Decision Tree model by using the previously defined pipeline and train data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 20, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model = pipeline_dt.fit(train_data)"
        }, 
        {
            "source": "You can check your model accuracy now. To evaluate the model, use test data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 21, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy = 0.870968\n"
                }
            ], 
            "source": "predictions = model.transform(test_data)\nevaluatorDT = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\", metricName=\"accuracy\")\naccuracy = evaluatorDT.evaluate(predictions)\n\nprint(\"Accuracy = %g\" % accuracy)"
        }, 
        {
            "source": "You can tune your model now to achieve better accuracy. For simplicity of this example tuning section is omitted.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 3. Persist model", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "In this section you will learn how to store sample model in Watson Machine Learning repository by using repository client.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "First, install and import client library.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 22, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 23, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Collecting watson-machine-learning-client\n  Downloading watson_machine_learning_client-1.0.53-py3-none-any.whl (561kB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 563kB 1.1MB/s eta 0:00:01\n\u001b[?25hCollecting tqdm (from watson-machine-learning-client)\n  Using cached tqdm-4.19.9-py2.py3-none-any.whl\nRequirement already up-to-date: tabulate in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: urllib3 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: certifi in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: pandas in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: lomond in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: requests in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nCollecting python-dateutil>=2 (from pandas->watson-machine-learning-client)\n  Using cached python_dateutil-2.7.2-py2.py3-none-any.whl\nCollecting numpy>=1.9.0 (from pandas->watson-machine-learning-client)\n  Downloading numpy-1.14.2-cp35-cp35m-manylinux1_x86_64.whl (12.1MB)\n\u001b[K    100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12.1MB 98kB/s  eta 0:00:01   24% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                        | 2.9MB 34.9MB/s eta 0:00:01\n\u001b[?25hCollecting pytz>=2011k (from pandas->watson-machine-learning-client)\n  Using cached pytz-2018.3-py2.py3-none-any.whl\nRequirement already up-to-date: six>=1.10.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement already up-to-date: idna<2.7,>=2.5 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/s86b-18e61b28c674e4-3fbaf243aed6/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nInstalling collected packages: tqdm, watson-machine-learning-client, python-dateutil, numpy, pytz\n  Found existing installation: tqdm 4.19.5\n    Uninstalling tqdm-4.19.5:\n      Successfully uninstalled tqdm-4.19.5\n  Found existing installation: watson-machine-learning-client 0.3.900\n    Uninstalling watson-machine-learning-client-0.3.900:\n      Successfully uninstalled watson-machine-learning-client-0.3.900\n  Found existing installation: python-dateutil 2.6.1\n    Uninstalling python-dateutil-2.6.1:\n      Successfully uninstalled python-dateutil-2.6.1\n  Found existing installation: numpy 1.14.0\n    Uninstalling numpy-1.14.0:\n      Successfully uninstalled numpy-1.14.0\n  Found existing installation: pytz 2017.3\n    Uninstalling pytz-2017.3:\n      Successfully uninstalled pytz-2017.3\nSuccessfully installed numpy-1.14.2 python-dateutil-2.7.2 pytz-2018.3 tqdm-4.19.9 watson-machine-learning-client-1.0.53\n"
                }
            ], 
            "source": "!pip install watson-machine-learning-client --upgrade"
        }, 
        {
            "source": "**Note**: Apache\u00ae Spark 2.1 is required.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 24, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient\nimport json"
        }, 
        {
            "source": "Authenticate to the Watson Machine Learning service on IBM Cloud.\n\n**Tip**: Authentication information (your credentials) can be found in the [Service Credentials](https://console.bluemix.net/docs/services/service_credentials.html#service_credentials) tab of the service instance that you created on IBM Cloud. \n\nIf you cannot see the **instance_id** field in **Service Credentials**, click **New credential (+)** to generate new authentication information. \n\n**Action**: Enter your Watson Machine Learning service instance credentials here.\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 25, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials={\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"***\",\n  \"username\": \"***\",\n  \"password\": \"***\",\n  \"instance_id\": \"***\"\n}"
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "Create WatsonMachineLearningAPIClient.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "#### Metadata preparation", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Prepare additional information to be saved as model's metadata:\n* TRAINING_DATA_REF\n* EVALUATION_METHOD: **multiclass**\n* EVALUATION_METRICS name: **accuracy** (metric name used to evaluate the model)\n* EVALUATION_METRICS value: **0.87** (accuracy value calculated few steps above)\n* EVALUATION_METRICS threshold: **0.8** (if the accuracy after evaluation using feedback data is below this threshold auto-retraining is triggered)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip**: If the accuracy value goes below the threshold retraining action is required.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Prepare training data reference that will be required by continuous learning system to trigger retraning action.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "training_data_reference = {\n \"name\": \"DRUG feedback\",\n \"connection\": db2_service_credentials,\n \"source\": {\n  \"tablename\": \"DRUG_TRAIN_DATA_UPDATED\",\n  \"type\": \"dashdb\"\n }\n}"
        }, 
        {
            "source": "**Tip**: All required fields can be found on Service Credentials tab of Db2 Warehouse on Cloud service instance created in IBM Cloud.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Add all information to model meta props.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_props = {\n    client.repository.ModelMetaNames.NAME: \"Best Heart Drug Selection\",\n    client.repository.ModelMetaNames.TRAINING_DATA_REFERENCE: training_data_reference,\n    client.repository.ModelMetaNames.EVALUATION_METHOD: \"multiclass\",\n    client.repository.ModelMetaNames.EVALUATION_METRICS: [\n        {\n           \"name\": \"accuracy\",\n           \"value\": accuracy,\n           \"threshold\": 0.8\n        }\n    ]\n}"
        }, 
        {
            "source": "Store model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "published_model_details = client.repository.store_model(model=model, meta_props=model_props, training_data=train_data, pipeline=pipeline_dt)\nmodel_uid = client.repository.get_model_uid(published_model_details)"
        }, 
        {
            "source": "**Tip**: Use `client.repository.ModelMetaNames.show()` to get the list of available props.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This way you can check your models details:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'metadata': {'created_at': '2018-03-29T08:35:31.303Z', 'url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264', 'modified_at': '2018-03-29T08:35:31.484Z', 'guid': '298832f7-d21d-4a07-a028-b5716dba1264'}, 'entity': {'latest_version': {'created_at': '2018-03-29T08:35:31.484Z', 'url': 'https://ibm-watson-ml.mybluemix.net/v3/ml_assets/models/298832f7-d21d-4a07-a028-b5716dba1264/versions/f97de42d-c0ea-42f2-a216-8c01744e85bd', 'guid': 'f97de42d-c0ea-42f2-a216-8c01744e85bd'}, 'runtime_environment': 'spark-2.1', 'name': 'Best Heart Drug Selection', 'learning_configuration_url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264/learning_configuration', 'model_type': 'mllib-2.1', 'input_data_schema': {'type': 'struct', 'fields': [{'metadata': {'name': 'AGE', 'scale': 0}, 'nullable': True, 'type': 'integer', 'name': 'AGE'}, {'metadata': {'name': 'SEX', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'SEX'}, {'metadata': {'name': 'BP', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'BP'}, {'metadata': {'name': 'CHOLESTEROL', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'CHOLESTEROL'}, {'metadata': {'name': 'NA', 'scale': 6}, 'nullable': True, 'type': 'decimal(12,6)', 'name': 'NA'}, {'metadata': {'name': 'K', 'scale': 6}, 'nullable': True, 'type': 'decimal(13,6)', 'name': 'K'}]}, 'feedback_url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264/feedback', 'label_col': 'DRUG', 'deployments': {'url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264/deployments', 'count': 0}, 'training_data_schema': {'type': 'struct', 'fields': [{'metadata': {'name': 'AGE', 'scale': 0}, 'nullable': True, 'type': 'integer', 'name': 'AGE'}, {'metadata': {'name': 'SEX', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'SEX'}, {'metadata': {'name': 'BP', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'BP'}, {'metadata': {'name': 'CHOLESTEROL', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'CHOLESTEROL'}, {'metadata': {'name': 'NA', 'scale': 6}, 'nullable': True, 'type': 'decimal(12,6)', 'name': 'NA'}, {'metadata': {'name': 'K', 'scale': 6}, 'nullable': True, 'type': 'decimal(13,6)', 'name': 'K'}, {'metadata': {'name': 'DRUG', 'scale': 0}, 'nullable': True, 'type': 'string', 'name': 'DRUG'}]}, 'learning_iterations_url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264/learning_iterations', 'evaluation_metrics_url': 'https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/298832f7-d21d-4a07-a028-b5716dba1264/evaluation_metrics', 'author': {}}}\n"
                }
            ], 
            "source": "print(published_model_details)"
        }, 
        {
            "source": "<a id=\"configuration\"></a>\n## 4. Configure continuous learning system", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will learn how to configure continuous learning system with Watson Machine Learning REST API Client.\nTo check underlying REST APIs, see the [Swagger Documentation](http://watson-ml-api.mybluemix.net/).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Continuous learning system provides you:\n- monitoring of model quality\n- model retraining if quality is below specified threshold\n- model redeployment if retrained model performs better", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Prepare Authorization header that combines Watson Machine Learning token and Spark instance credentials.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "spark_credentials = {\n  \"tenant_id\": \"***\",\n  \"tenant_id_full\": \"***\",\n  \"cluster_master_url\": \"https://spark.bluemix.net\",\n  \"tenant_secret\": \"***\",\n  \"instance_id\": \"***\",\n  \"plan\": \"ibm.SparkService.PayGoPersonal\"\n}"
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "### Set configuration for published model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip**: Note that only ```tablename``` is different comparing to ```training_data_reference```.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "feedback_data_reference = {\n \"name\": \"DRUG feedback\",\n \"connection\": db2_service_credentials,\n \"source\": {\n  \"tablename\": \"DRUG_FEEDBACK_DATA\",\n  \"type\": \"dashdb\"\n }\n}"
        }, 
        {
            "source": "Define values of the following fields to finalize payload:\n- ```min_feedback_data_size``` - this is minimal number of records in feedback dataset to start continuous learning system iteration\n- ```auto_retrain``` [never, always, conditionally] - this parameter specifies if retraining process should be triggered (conditionally will trigger retraining process when evaluation result is below specified threshold)\n- ```auto_redeploy``` [never, always, conditionally] - this paramete specifies if retrained model should be deployed (conditionally will trigger redeployment when newly trained model quality is better)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 37, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'auto_redeploy': 'always',\n 'auto_retrain': 'conditionally',\n 'feedback_data_reference': {'connection': {'db': 'BLUDB',\n   'dsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50000;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;',\n   'host': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'hostname': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'https_url': 'https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443',\n   'jdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'password': '5338f7276f54',\n   'port': 50000,\n   'ssldsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50001;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;Security=SSL;',\n   'ssljdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;',\n   'uri': 'db2://dash6973:5338f7276f54@dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'username': 'dash6973'},\n  'source': {'tablename': 'DRUG_FEEDBACK_DATA', 'type': 'dashdb'}},\n 'min_feedback_data_size': 10,\n 'spark_service': {'credentials': {'cluster_master_url': 'https://spark.bluemix.net',\n   'instance_id': '9c87ff2a-99c9-40a4-a86b-18e61b28c674',\n   'plan': 'ibm.SparkService.PayGoPersonal',\n   'tenant_id': 's86b-18e61b28c674e4-3fbaf243aed6',\n   'tenant_id_full': '9c87ff2a-99c9-40a4-a86b-18e61b28c674_e6d9f948-d4d4-411b-9ce4-3fbaf243aed6',\n   'tenant_secret': '09350d00-a602-49b9-8160-910fdb339747'},\n  'version': '2.0'},\n 'training_data_reference': {'connection': {'db': 'BLUDB',\n   'dsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50000;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;',\n   'host': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'hostname': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'https_url': 'https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443',\n   'jdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'password': '5338f7276f54',\n   'port': 50000,\n   'ssldsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50001;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;Security=SSL;',\n   'ssljdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;',\n   'uri': 'db2://dash6973:5338f7276f54@dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'username': 'dash6973'},\n  'source': {'tablename': 'DRUG_TRAIN_DATA_UPDATED', 'type': 'dashdb'}}}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "system_config = {\n    client.learning_system.ConfigurationMetaNames.FEEDBACK_DATA_REFERENCE: feedback_data_reference,\n    client.learning_system.ConfigurationMetaNames.MIN_FEEDBACK_DATA_SIZE: 10,\n    client.learning_system.ConfigurationMetaNames.SPARK_REFERENCE: spark_credentials,\n    client.learning_system.ConfigurationMetaNames.AUTO_RETRAIN: \"conditionally\",\n    client.learning_system.ConfigurationMetaNames.AUTO_REDEPLOY: \"always\"\n}\n\nclient.learning_system.setup(model_uid=model_uid, meta_props=system_config)"
        }, 
        {
            "source": "Learning configuration has been specified successfuly. You can check the details with below GET call.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "learning_details = client.learning_system.get_details(model_uid)"
        }, 
        {
            "source": "### Patch configuration for published model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "To update learning configuration you can use PATCH request as shown below.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "feedback_data_reference_updated = {\n    \"connection\": db2_service_credentials,\n    \"source\": {\n        \"type\": \"dashdb\",\n         \"tablename\": \"DRUG_FEEDBACK_DATA\"\n    }\n}"
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 44, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'auto_redeploy': 'always',\n 'auto_retrain': 'conditionally',\n 'evaluation_definition': {'method': 'multiclass',\n  'metrics': [{'name': 'accuracy', 'threshold': 0.8}]},\n 'feedback_data_reference': {'connection': {'db': 'BLUDB',\n   'dsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50000;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;',\n   'host': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'hostname': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'https_url': 'https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443',\n   'jdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'password': '5338f7276f54',\n   'port': 50000,\n   'ssldsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50001;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;Security=SSL;',\n   'ssljdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;',\n   'uri': 'db2://dash6973:5338f7276f54@dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'username': 'dash6973'},\n  'source': {'tablename': 'DRUG_FEEDBACK_DATA', 'type': 'dashdb'}},\n 'min_feedback_data_size': 10,\n 'spark_service': {'credentials': {'cluster_master_url': 'https://spark.bluemix.net',\n   'instance_id': '9c87ff2a-99c9-40a4-a86b-18e61b28c674',\n   'plan': 'ibm.SparkService.PayGoPersonal',\n   'tenant_id': 's86b-18e61b28c674e4-3fbaf243aed6',\n   'tenant_id_full': '9c87ff2a-99c9-40a4-a86b-18e61b28c674_e6d9f948-d4d4-411b-9ce4-3fbaf243aed6',\n   'tenant_secret': '09350d00-a602-49b9-8160-910fdb339747'},\n  'version': '2.0'},\n 'training_data_reference': {'connection': {'db': 'BLUDB',\n   'dsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50000;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;',\n   'host': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'hostname': 'dashdb-entry-yp-dal10-01.services.dal.bluemix.net',\n   'https_url': 'https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443',\n   'jdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'password': '5338f7276f54',\n   'port': 50000,\n   'ssldsn': 'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal10-01.services.dal.bluemix.net;PORT=50001;PROTOCOL=TCPIP;UID=dash6973;PWD=5338f7276f54;Security=SSL;',\n   'ssljdbcurl': 'jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;',\n   'uri': 'db2://dash6973:5338f7276f54@dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50000/BLUDB',\n   'username': 'dash6973'},\n  'source': {'tablename': 'DRUG_TRAIN_DATA_UPDATED', 'type': 'dashdb'}}}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "updated_config = {\n    client.learning_system.ConfigurationMetaNames.FEEDBACK_DATA_REFERENCE: feedback_data_reference_updated\n}\n\nclient.learning_system.update(model_uid, updated_config)"
        }, 
        {
            "source": "<a id=\"performance\"></a>\n## 5. Track model performance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "To start iteration of learning system use run(...) method of client. Within iteration published model will be evaluated. If the evaluated accuracy is below specified threshold model retraining will be triggered. Both data sets: training and feedback are used for retraining and evaluation.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "`aynchronous` parameter set to False provides iteration state monitoring.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################\n\nSynchronous run for uid: '604d5ef2-879d-49f0-a7a8-33be2eaaeab4' started\n\n#######################################################################\n\n\nINITIALIZED\nRUNNING................\nCOMPLETED\n\n\n--------------------------------------------------------------------------------------------\nSuccessfully finished learning iteration run, run_uid='604d5ef2-879d-49f0-a7a8-33be2eaaeab4'\n--------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "run_details = client.learning_system.run(model_uid, asynchronous=False)"
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "run_uid = client.learning_system.get_run_uid(run_details)"
        }, 
        {
            "source": "#### Get run details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 53, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "learning_run_details = client.learning_system.get_run_details(run_uid)"
        }, 
        {
            "source": "#### Get evaluation values", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 51, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "metrics = client.learning_system.get_metrics(model_uid)\n# print(json.dumps(metrics, indent=2))"
        }, 
        {
            "source": "**Tip**: To see evaluation result you need to wait for iteration completion.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Action**: to display evaluation details in form of table you need to install ```tabulate``` package.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 52, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "----------  ------------------------  -----------  ------------------  --------------  -----------------------------------\nPHASE       TIMESTAMP                 METRIC NAME  METRIC VALUE        METRIC THRESH.  VERSION\nsetup       2018-03-29T08:35:31.382Z  accuracy     0.8709677419354839  0.8             f97de42d-c0ea-42f2-a216-8c01744e85b\nmonitoring  2018-03-29T08:46:59.079Z  accuracy     0.75                0.8             f97de42d-c0ea-42f2-a216-8c01744e85b\ntraining    2018-03-29T08:47:33.793Z  accuracy     0.8398066941113299  0.8             acffe913-247e-4085-a335-540e53aaced\n----------  ------------------------  -----------  ------------------  --------------  -----------------------------------\n"
                }
            ], 
            "source": "client.learning_system.list_metrics(model_uid)"
        }, 
        {
            "source": "You can see that this iteration of continuous learning loop consists of the following phases:\n- monitoring - using feedback data published model quality was checked (evaluation). \n- training - since evaluation result (0.75) is below specified threshold (0.8) model retraining was triggered. Evaluation of retrained model shows accuracy at 0.92.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip**: If `auto_redeploy` option is set to conditionally the newly trained model will be redeployed since it shows better accuracy than original one.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"visualization\"></a>\n## 6. Visualization of model performance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this subsection you will visualize iteration results with Plotly, which is an online analytics and data visualization tool.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Example**:  First, you need to install required packages. You can do it by running the following code. Run it only one time.\n\n!pip install plotly --user\n\n!pip install cufflinks --user", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "!pip install plotly --user \n!pip install cufflinks --user", 
            "cell_type": "raw", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "Import Plotly and other required packages.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!pip install cufflinks"
        }, 
        {
            "execution_count": 56, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>", 
                        "text/vnd.plotly.v1+html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "import sys\nimport pandas\nimport plotly.plotly as py\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\nimport plotly.graph_objs as go\n\ninit_notebook_mode(connected=True)\nsys.path.append(\"\".join([os.environ[\"HOME\"]])) "
        }, 
        {
            "source": "#### Prepare data for plotly", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 57, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "phases = []\nevaluation_values = []\nthreshold_values = []\n\nfor i,x in enumerate(metrics['resources']):\n    phases.append(x['phase'] + '_' + str(i))\n    evaluation_values.append(x['values'][0]['value'])\n    threshold_values.append(x['values'][0]['threshold'])"
        }, 
        {
            "source": "#### Plot linear chart", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 58, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"e1cd8cc6-145d-47c2-bd17-e406fb2b704a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"e1cd8cc6-145d-47c2-bd17-e406fb2b704a\", [{\"y\": [0.8709677419354839, 0.75, 0.8398066941113299], \"type\": \"scatter\", \"name\": \"accuracy\", \"x\": [\"setup_0\", \"monitoring_1\", \"training_2\"], \"mode\": \"lines+markers\"}, {\"y\": [0.8, 0.8, 0.8], \"type\": \"scatter\", \"name\": \"threshold\", \"x\": [\"setup_0\", \"monitoring_1\", \"training_2\"], \"mode\": \"lines\"}], {\"title\": \"Model performance\", \"xaxis\": {\"title\": \"Phase\"}, \"yaxis\": {\"title\": \"Evaluation result\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"e1cd8cc6-145d-47c2-bd17-e406fb2b704a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"e1cd8cc6-145d-47c2-bd17-e406fb2b704a\", [{\"y\": [0.8709677419354839, 0.75, 0.8398066941113299], \"type\": \"scatter\", \"name\": \"accuracy\", \"x\": [\"setup_0\", \"monitoring_1\", \"training_2\"], \"mode\": \"lines+markers\"}, {\"y\": [0.8, 0.8, 0.8], \"type\": \"scatter\", \"name\": \"threshold\", \"x\": [\"setup_0\", \"monitoring_1\", \"training_2\"], \"mode\": \"lines\"}], {\"title\": \"Model performance\", \"xaxis\": {\"title\": \"Phase\"}, \"yaxis\": {\"title\": \"Evaluation result\"}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "y": [
                                        0.8709677419354839, 
                                        0.75, 
                                        0.8398066941113299
                                    ], 
                                    "x": [
                                        "setup_0", 
                                        "monitoring_1", 
                                        "training_2"
                                    ], 
                                    "type": "scatter", 
                                    "mode": "lines+markers", 
                                    "name": "accuracy"
                                }, 
                                {
                                    "y": [
                                        0.8, 
                                        0.8, 
                                        0.8
                                    ], 
                                    "x": [
                                        "setup_0", 
                                        "monitoring_1", 
                                        "training_2"
                                    ], 
                                    "type": "scatter", 
                                    "mode": "lines", 
                                    "name": "threshold"
                                }
                            ], 
                            "layout": {
                                "title": "Model performance", 
                                "xaxis": {
                                    "title": "Phase"
                                }, 
                                "yaxis": {
                                    "title": "Evaluation result"
                                }
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "trace1 = go.Scatter(\n    x = phases,\n    y = evaluation_values,\n    mode = 'lines+markers',\n    name = 'accuracy'\n)\n\ntrace2 = go.Scatter(\n    x = phases,\n    y = threshold_values,\n    mode = 'lines',\n    name = 'threshold'\n)\n\nlayout = dict(title = 'Model performance',\n              xaxis = dict(title = 'Phase'),\n              yaxis = dict(title = 'Evaluation result'),\n              )\n\nfig = dict(data=[trace1, trace2], layout=layout)\niplot(fig)"
        }, 
        {
            "source": "Within single Continuous Learning System iteration we can notice two phases:\n* monitoring - in that phase initial model is evaluated using feedback data\n* training - in that phase model is retrained using combination of training and feedback data. Next, model is evaluated.\n<BR><BR>\nAfter retraining model accuracy increased to desired level (above specified threshold).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"feedback\"></a>\n## 7. Feedback records   ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can use feedback endpoint to send new records to feedback data store.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Let's generate some records based on training data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 59, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from pyspark.sql.functions import UserDefinedFunction, col, column\nfrom pyspark.sql.types import IntegerType\n\ncol_name = 'AGE'\nudf_add = UserDefinedFunction(lambda x: x + 1, IntegerType())\nnew_records_df = train_data.select(*[udf_add(column).alias(col_name) if column == col_name else column for column in train_data.columns])\nnew_records_df = new_records_df.withColumn(\"K\", col(\"K\").cast(\"double\")).withColumn(\"NA\", col(\"NA\").cast(\"double\"))\nnew_records_pdf = new_records_df.toPandas()"
        }, 
        {
            "execution_count": 60, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "records=[]\n\nimport numpy as np\n\nfor i in range(new_records_pdf.shape[0]):\n    records.append([x.tolist() if type(x).__module__ == np.__name__ else x for x in new_records_pdf.loc[i].values.tolist()])"
        }, 
        {
            "source": "In next step feedback payload is sent.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 61, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 61, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'rows_inserted': 150}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.learning_system.send_feedback(model_uid, records, fields=train_data.columns)"
        }, 
        {
            "source": "**Tip:** Now, you can run another iteration of learning system using new feedback data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 8. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": " You successfully completed this notebook! You learned how to use Continuous Learning System of Watson Machine Learning. Check out our [Online Documentation](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html)\n for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Authors\n\n**Lukasz Cmielowski**, PhD, is a Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n\n**Maria Oleszkiewicz**, MSc, is developer who took part in building the wml api client used in this notebook.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2018 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5 with Spark 2.1", 
            "name": "python3-spark21", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}