From Keras experiment with HPO to scoring.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"4\" color=\"black\"><b>From keras experiment to scoring with</b></font><font face=\"monospace\" size=\"4\" color=\"darkblue\"><b> watson-machine-learning-client</b></font></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr> \n   <tr style=\"border: none\">\n       <td style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/raw/master/tensorflow/hand-written-digit-recognition/images/experiment_banner.png\" width=\"600\" alt=\"Icon\"></td>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to demonstrate support of deep learning experiments in Watson Machine Learning Service. This notebook introduces commands for getting data, training_definition persistance, experiment training, model persistance, model deployment and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.\n\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Working with Watson Machine Learning experiments to train Deep Learning models (Keras).\n-  Saving trained models in Watson Machine Learning repository.\n-  Online deployment and scoring of trained model.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Setup](#setup)\n2.\t[Training definition](#definition)\n3.\t[Experiment definition](#experiment)\n4.\t[Experiment Run](#run)\n5.\t[Deploy and Score](#scoring)\n6.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Setup\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a lite plan is offered). \n-  Create [Cloud Object Storage (COS)](https://console.bluemix.net/catalog/infrastructure/cloud-object-storage) instance (a lite plan is offered).\n    - After you create COS instance, go to your COS dashboard.\n    - In \"Service credentials\" tab, click on \"New Credential\",\n    - Add inline configuration parameter: {\"HMAC\":true}, click \"Add\".\n\n    This configuration parameter will add section below to instance credentials which will be used later on,\n    ```\n      \"cos_hmac_keys\": {\n            \"access_key_id\": \"722432c254bc4eaa96e05897bf2779e2\",\n            \"secret_access_key\": \"286965ac10ecd4de8b44306288c7f5a3e3cf81976a03075c\"\n       }\n    ```", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 1.1 Working with Cloud Object Storage", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Boto library allows Python developers to manage Cloud Object Storage.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** If ibm_boto3 is not preinstalled in you environment please install it by running the following command: `!pip install ibm-cos-sdk`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import ibm_boto3\nfrom ibm_botocore.client import Config\nimport os\nimport json\nimport warnings\nimport time"
        }, 
        {
            "source": "We define the endpoint we will use. You can find this information in \"Endpoint\" section of your Cloud Object Storage intance's dashbord.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos_credentials = {\n  \"apikey\": \"----\",\n  \"cos_hmac_keys\": {\n    \"access_key_id\": \"---\",\n    \"secret_access_key\": \"---\"\n  },\n  \"endpoints\": \"https://cos-service.bluemix.net/endpoints\",\n  \"iam_apikey_description\": \"Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/bc6f89c9dbf0a21338f4ee6fa02cf6cf:d2901ba2-3196-4663-aee9-7ddfbb49eb7d::\",\n  \"iam_apikey_name\": \"auto-generated-apikey-cd81fb86-8f99-4bbf-944d-6cc4bc54f2be\",\n  \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Writer\",\n  \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::a/bc6f89c9dbf0a21338f4ee6fa02cf6cf::serviceid:ServiceId-b6138191-c9a3-4349-8b20-ddc89f13c78a\",\n  \"resource_instance_id\": \"crn:v1:bluemix:public:cloud-object-storage:global:a/bc6f89c9dbf0a21338f4ee6fa02cf6cf:d2901ba2-3196-4663-aee9-7ddfbb49eb7d::\"\n}\n\napi_key = cos_credentials['apikey']\nservice_instance_id = cos_credentials['resource_instance_id']\nauth_endpoint = 'https://iam.bluemix.net/oidc/token'\nservice_endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'"
        }, 
        {
            "source": "We create Boto resource by providing type, endpoint_url and credentials.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos = ibm_boto3.resource('s3',\n                         ibm_api_key_id=api_key,\n                         ibm_service_instance_id=service_instance_id,\n                         ibm_auth_endpoint=auth_endpoint,\n                         config=Config(signature_version='oauth'),\n                         endpoint_url=service_endpoint)"
        }, 
        {
            "source": "Let's create the buckets we will use to store training data and training results.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Note:**: Bucket name has to be unique - please update below ones to any unique name.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Creating bucket \"mnist-keras-data-example\"...\nCreating bucket \"mnist-keras-results-example\"...\n"
                }
            ], 
            "source": "buckets = ['mnist-keras-data-example', 'mnist-keras-results-example']\nfor bucket in buckets:\n    if not cos.Bucket(bucket) in cos.buckets.all():\n        print('Creating bucket \"{}\"...'.format(bucket))\n        try:\n            cos.create_bucket(Bucket=bucket)\n        except ibm_boto3.exceptions.ibm_botocore.client.ClientError as e:\n            print('Error: {}.'.format(e.response['Error']['Message']))"
        }, 
        {
            "source": "Now we should have our buckets created.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[s3.Bucket(name='mnist-keras-data-example'), s3.Bucket(name='mnist-keras-results-example'), s3.Bucket(name='style-data-example'), s3.Bucket(name='style-results-example')]\n"
                }
            ], 
            "source": "print(list(cos.buckets.all()))"
        }, 
        {
            "source": "### 1.2 Downloading MNIST data and upload it to COS buckets", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "We will work with Keras **MNIST** sample dataset. Let's download our training data and upload them to 'mnist-keras-data' bucket.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Below cell will create 'MNIST_KERAS_DATA' folder and download the file from link.\n\n**Tip:** First, please install wget library by the command below\n`!pip install wget`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "link = 'https://s3.amazonaws.com/img-datasets/mnist.npz'"
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "mnist.npz\r\n"
                }
            ], 
            "source": "import wget, os\n\ndata_dir = 'MNIST_KERAS_DATA'\nif not os.path.isdir(data_dir):\n    os.mkdir(data_dir)\n\nif not os.path.isfile(os.path.join(data_dir, os.path.join(link.split('/')[-1]))):\n    wget.download(link, out=data_dir)  \n        \n!ls MNIST_KERAS_DATA"
        }, 
        {
            "source": "Upload the data files to created buckets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "mnist.npz is uploaded.\n"
                }
            ], 
            "source": "bucket_name = buckets[0]\nbucket_obj = cos.Bucket(bucket_name)\n\nfor filename in os.listdir(data_dir):\n    with open(os.path.join(data_dir, filename), 'rb') as data: \n        bucket_obj.upload_file(os.path.join(data_dir, filename), filename)\n        print('{} is uploaded.'.format(filename))"
        }, 
        {
            "source": "Let's see the list of all buckets and their contents.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Object key: mnist.npz\nObject size (kb): 11221.126953125\n"
                }
            ], 
            "source": "for obj in bucket_obj.objects.all():\n    print('Object key: {}'.format(obj.key))\n    print('Object size (kb): {}'.format(obj.size/1024))"
        }, 
        {
            "source": "We are done with Cloud Object Storage, we are ready to train our model!", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 1.3 Work with the Watson Machine Learning instance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Authenticate to Watson Machine Learning service on Bluemix.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import urllib3, requests, json, base64, time, os\nwarnings.filterwarnings('ignore')"
        }, 
        {
            "source": "**Action**: Put authentication information from your instance of Watson Machine Learning service here.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials = {\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"YrY6e1xi6rJemA1yEB2iaTojb0OKZTpi79o+4MSFrSG3BM2IFZaDZwvBcoQXGXStpxSFKe9cZoFYLlzgPf++qpWZYcc+6fawL9S0V+2V79Adc+zik+ZHJYrsBRl9GAcs\",\n  \"username\": \"---\",\n  \"password\": \"---\",\n  \"instance_id\": \"---\"\n}"
        }, 
        {
            "source": "#### Install `watson-machine-learning-client` from pypi", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "!pip install --upgrade watson-machine-learning-client", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement already up-to-date: watson-machine-learning-client in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages\nRequirement already up-to-date: tqdm in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: tabulate in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: urllib3 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: certifi in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: pandas in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: lomond in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: requests in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: python-dateutil>=2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: numpy>=1.9.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: pytz>=2011k in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: six>=1.10.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement already up-to-date: idna<2.7,>=2.5 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\n"
                }
            ], 
            "source": "!pip install --upgrade watson-machine-learning-client"
        }, 
        {
            "source": "#### Import `watson-machine-learning-client` and authenticate to service instance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "1.0.12\n"
                }
            ], 
            "source": "print(client.version)"
        }, 
        {
            "source": "<a id=\"definition\"></a>\n## 2. Training definitions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "For purpose of this example two Keras model definitions have been prepared:\n* Multilayer Perceptron (MLP)\n* Convolution Neural Network (CNN)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 2.2 Save training definition ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Prepare training definition metadata", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_definition_1_metadata = {\n            client.repository.DefinitionMetaNames.NAME: \"MNIST-MLP\",\n            client.repository.DefinitionMetaNames.AUTHOR_EMAIL: \"JohnSmith@ibm.com\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_NAME: \"tensorflow\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_VERSION: \"1.5\",\n            client.repository.DefinitionMetaNames.RUNTIME_NAME: \"python\",\n            client.repository.DefinitionMetaNames.RUNTIME_VERSION: \"3.5\",\n            client.repository.DefinitionMetaNames.EXECUTION_COMMAND: \"python3 mnist_mlp.py\"\n            }"
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_definition_2_metadata = {\n            client.repository.DefinitionMetaNames.NAME: \"MNIST-CNN\",\n            client.repository.DefinitionMetaNames.AUTHOR_EMAIL: \"JohnSmith@ibm.com\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_NAME: \"tensorflow\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_VERSION: \"1.5\",\n            client.repository.DefinitionMetaNames.RUNTIME_NAME: \"python\",\n            client.repository.DefinitionMetaNames.RUNTIME_VERSION: \"3.5\",\n            client.repository.DefinitionMetaNames.EXECUTION_COMMAND: \"python3 mnist_cnn.py\"\n            }"
        }, 
        {
            "source": "#### Get sample model definition content files from git (python scripts with CNN and MLP)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf MNIST.zip"
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--2018-03-16 07:35:25--  https://github.com/pmservice/wml-sample-models/raw/master/keras/mnist/MNIST.zip\nResolving github.com (github.com)... 192.30.253.112, 192.30.253.113\nConnecting to github.com (github.com)|192.30.253.112|:443... connected.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://raw.githubusercontent.com/pmservice/wml-sample-models/master/keras/mnist/MNIST.zip [following]\n--2018-03-16 07:35:26--  https://raw.githubusercontent.com/pmservice/wml-sample-models/master/keras/mnist/MNIST.zip\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 3836 (3.7K) [application/zip]\nSaving to: \u2018MNIST.zip\u2019\n\n100%[======================================>] 3,836       --.-K/s   in 0s      \n\n2018-03-16 07:35:26 (28.6 MB/s) - \u2018MNIST.zip\u2019 saved [3836/3836]\n\nMNIST.zip\n"
                }
            ], 
            "source": "filename_mnist = 'MNIST.zip'\n\nif os.path.isfile(filename_mnist):\n    !ls 'MNIST.zip'\nelse:\n    !wget https://github.com/pmservice/wml-sample-models/raw/master/keras/mnist/MNIST.zip\n    !ls 'MNIST.zip'"
        }, 
        {
            "source": "#### Publish training definitions in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "##### Store definition 1", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/ml_assets/training_definitions/d56f3240-b49e-4b30-98e8-fb1ca8ab6dd6\n"
                }
            ], 
            "source": "definition_1_details = client.repository.store_definition(filename_mnist, model_definition_1_metadata)\n\ndefinition_1_url = client.repository.get_definition_url(definition_1_details)\ndefinition_1_uid = client.repository.get_definition_uid(definition_1_details)\nprint(definition_1_url)"
        }, 
        {
            "source": "##### Store definition 2", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/ml_assets/training_definitions/970df64d-d363-4524-881d-75320928c695\n"
                }
            ], 
            "source": "definition_2_details = client.repository.store_definition(filename_mnist, model_definition_2_metadata)\n\ndefinition_2_url = client.repository.get_definition_url(definition_2_details)\ndefinition_2_uid = client.repository.get_definition_uid(definition_2_details)\nprint(definition_2_url)"
        }, 
        {
            "source": "#### LIST stored definitions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ------------------------  ------------------------  ----------\nGUID                                  NAME                      CREATED                   FRAMEWORK\nea8c0dd5-8a6b-4e5e-a9cd-74a0c573c1de  MNIST-MLP                 2018-03-15T10:20:58.646Z  tensorflow\n8095a87c-5c94-4f43-b5fd-468b2d72a3c8  MNIST-CNN                 2018-03-15T10:21:00.033Z  tensorflow\nbebc485e-bcc9-45b7-9bce-1ba20acc3b53  MNIST-MLP                 2018-03-15T10:35:04.997Z  tensorflow\nf26539ff-43b5-4100-8915-7b0be22b2211  MNIST-CNN                 2018-03-15T10:35:05.815Z  tensorflow\nbed45da1-6372-4cb1-b889-e0d402cfe62a  MNIST-MLP                 2018-03-15T15:41:40.335Z  tensorflow\nc80aaeef-1c03-4474-8891-535aa6dae81e  MNIST-CNN                 2018-03-15T15:41:41.174Z  tensorflow\nf819f724-8e1c-4903-a04c-eee5dfc048a4  style transfer van gogh   2018-03-15T20:33:03.354Z  tensorflow\nf5c4e713-27d9-4529-a473-10dc88383a52  style transfer kandinsky  2018-03-15T20:33:05.032Z  tensorflow\naf72a944-cc83-4aaf-9152-9f25c1912bf4  style transfer van gogh   2018-03-15T20:57:13.232Z  tensorflow\n4fb3987a-f979-4e1e-a26f-1acbe11a08e7  style transfer kandinsky  2018-03-15T20:57:13.892Z  tensorflow\n1421337a-c2a8-4153-8554-accdc55bbd27  style transfer van gogh   2018-03-15T21:45:46.889Z  tensorflow\n0f8fa74a-f821-475c-85c9-a2b507ed6c63  style transfer kandinsky  2018-03-15T21:45:48.344Z  tensorflow\ncd31e290-6636-4586-8097-e3c3ac7fef8c  style transfer van gogh   2018-03-16T08:37:18.880Z  tensorflow\n76b144fc-100c-45ee-80ae-ef0d44a87cd6  style transfer kandinsky  2018-03-16T08:37:19.383Z  tensorflow\n7dc7e7cc-ae5b-4db2-8705-32174324010a  MNIST-MLP                 2018-03-16T09:59:57.138Z  tensorflow\n6a024dde-d6bc-45d3-8463-895452134c29  MNIST-CNN                 2018-03-16T09:59:57.888Z  tensorflow\n32c64603-13d2-4ec6-b3ca-4ee4650fbd8e  style transfer van gogh   2018-03-16T10:04:24.779Z  tensorflow\n18498298-3e01-43a0-a1f7-955ecc118a4c  style transfer kandinsky  2018-03-16T10:04:25.661Z  tensorflow\n9e446ccc-8b1d-4e46-a549-655f9b16a577  style transfer van gogh   2018-03-16T10:33:54.588Z  tensorflow\n1f54975c-0104-4fcb-910c-ae57e3272d7c  style transfer kandinsky  2018-03-16T10:33:55.170Z  tensorflow\n3501d90f-8599-4154-a8a5-d6ddafac15a8  MNIST-MLP                 2018-03-16T11:53:19.116Z  tensorflow\n7c1b279f-7556-4beb-bee3-5516a0daa433  MNIST-CNN                 2018-03-16T11:53:19.907Z  tensorflow\nd56f3240-b49e-4b30-98e8-fb1ca8ab6dd6  MNIST-MLP                 2018-03-16T12:35:29.322Z  tensorflow\n970df64d-d363-4524-881d-75320928c695  MNIST-CNN                 2018-03-16T12:35:30.591Z  tensorflow\n------------------------------------  ------------------------  ------------------------  ----------\n"
                }
            ], 
            "source": "client.repository.list_definitions()"
        }, 
        {
            "source": "<a id=\"experiment\"></a>\n## 3. Experiment definition", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1 Save experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Get the list of supported configuration params", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--------------------------  ----  --------\nMETA_PROP NAME              TYPE  REQUIRED\nNAME                        str   Y\nTAGS                        list  N\nDESCRIPTION                 str   N\nAUTHOR_NAME                 str   N\nAUTHOR_EMAIL                str   Y\nEVALUATION_METHOD           str   N\nEVALUATION_METRICS          list  N\nTRAINING_REFERENCES         list  Y\nTRAINING_DATA_REFERENCE     dict  Y\nTRAINING_RESULTS_REFERENCE  dict  Y\n--------------------------  ----  --------\n"
                }
            ], 
            "source": "client.repository.ExperimentMetaNames.show()"
        }, 
        {
            "source": "#### Experiment configuration dictionary\nCreate experiment that will train models based on previously stored definitions.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "`TRAINING_DATA_REFERENCE` - location of traininng data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "TRAINING_DATA_REFERENCE = {\n                            \"connection\": {\n                                \"endpoint_url\": service_endpoint,\n                                \"aws_access_key_id\": cos_credentials['cos_hmac_keys']['access_key_id'],\n                                \"aws_secret_access_key\": cos_credentials['cos_hmac_keys']['secret_access_key']\n                            },\n                            \"source\": {\n                                \"bucket\": buckets[0],\n                            },\n                            \"type\": \"s3\"\n}"
        }, 
        {
            "source": "`TRAINING_RESULTS_REFERENCE` - location of training results", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "TRAINING_RESULTS_REFERENCE = {\n                                \"connection\": {\n                                    \"endpoint_url\": service_endpoint,\n                                    \"aws_access_key_id\": cos_credentials['cos_hmac_keys']['access_key_id'],\n                                    \"aws_secret_access_key\": cos_credentials['cos_hmac_keys']['secret_access_key']\n                                },\n                                \"target\": {\n                                    \"bucket\": buckets[1],\n                                },\n                                \"type\": \"s3\"\n}"
        }, 
        {
            "source": "Configure hyper parameters optimizer for you experiment. We are interested in `val_acc` metric so we need to put it as optimizer `objective`. `num_optimizer_steps` tells the optimizer how many models we want to train based on hyper parameters values combinations.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "HPO = {\n        \"method\": {\n            \"name\": \"rbfopt\",\n            \"parameters\": [\n                client.experiments.HPOMethodParam(\"objective\", \"val_acc\"),\n                client.experiments.HPOMethodParam(\"maximize_or_minimize\", \"maximize\"),\n                client.experiments.HPOMethodParam(\"num_optimizer_steps\", 3)\n            ]\n        },\n        \"hyper_parameters\": [\n            client.experiments.HPOParameter('learning_rate', min=0.0001, max=0.001, step=0.0003)\n        ]\n     }           "
        }, 
        {
            "source": "Configure your experiment. `TRAINING_REFERENCES` links previously stored training definitions and provides information about `compute_configuration` that will be used to run the training.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 45, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_metadata = {\n            client.repository.ExperimentMetaNames.NAME: \"MNIST experiment\",\n            client.repository.ExperimentMetaNames.DESCRIPTION: \"Best model for MNIST.\",\n            client.repository.ExperimentMetaNames.AUTHOR_EMAIL: \"JohnSmith@ibm.com\",\n            client.repository.ExperimentMetaNames.EVALUATION_METHOD: \"multiclass\",\n            client.repository.ExperimentMetaNames.EVALUATION_METRICS: [\"val_acc\"],\n            #client.repository.ExperimentMetaNames.EVALUATION_METRICS: [{\"name\": \"val_acc\", \"maximize\": True}],\n            client.repository.ExperimentMetaNames.TRAINING_DATA_REFERENCE: TRAINING_DATA_REFERENCE,\n            client.repository.ExperimentMetaNames.TRAINING_RESULTS_REFERENCE: TRAINING_RESULTS_REFERENCE,\n            client.repository.ExperimentMetaNames.TRAINING_REFERENCES: [\n                        {\n                            \"name\": \"MNIST_MLP\",\n                            \"training_definition_url\": definition_1_url,\n                            \"compute_configuration\": {\"name\": \"k80x2\"},\n                            \"hyper_parameters_optimization\": HPO\n                            \n                        },\n                        {\n                            \"name\": \"MNIST_CNN\",\n                            \"training_definition_url\": definition_2_url,\n                            \"compute_configuration\": {\"name\": \"k80x2\"},\n                        }],\n            }"
        }, 
        {
            "source": "#### Store experiment in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "dc222702-508e-4c27-aae5-a0af07c3a810\n"
                }
            ], 
            "source": "experiment_details = client.repository.store_experiment(meta_props=experiment_metadata)\n\nexperiment_uid = client.repository.get_experiment_uid(experiment_details)\nprint(experiment_uid)"
        }, 
        {
            "source": "#### LIST stored experiments", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ----------------  ------------------------\nGUID                                  NAME              CREATED\n0c8803bf-122f-4066-be77-39d6aef4de3d  MNIST experiment  2018-03-15T10:35:54.116Z\n207928e4-2634-48b2-80d1-acbc3eed3db2  MNIST experiment  2018-03-15T10:21:23.857Z\n298cc42b-3138-43a2-a690-bf18292e7f65  STYLE experiment  2018-03-16T10:34:03.736Z\n3b274502-a203-430f-b2b2-903f1e713d40  MNIST experiment  2018-03-16T10:00:46.078Z\n496a28ea-7887-4b3e-b4c0-28d21b831333  MNIST experiment  2018-03-15T15:41:47.219Z\n4bae6fcb-f490-45b7-bdaf-559030b6a0f9  STYLE experiment  2018-03-15T21:46:05.795Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  STYLE experiment  2018-03-16T08:37:30.628Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  STYLE experiment  2018-03-15T20:57:23.876Z\ndc222702-508e-4c27-aae5-a0af07c3a810  MNIST experiment  2018-03-16T12:35:40.804Z\nf35cea6e-4461-45d2-b1b7-d2250fc1a59e  STYLE experiment  2018-03-15T20:35:12.928Z\nff7bada4-7530-4c1d-8de1-59684727e021  STYLE experiment  2018-03-16T10:04:35.729Z\n------------------------------------  ----------------  ------------------------\n"
                }
            ], 
            "source": "client.repository.list_experiments()"
        }, 
        {
            "source": "**Hint:** You can update experiment definition by calling below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "updated_experiment_details = client.repository.update_experiment(experiment_uid, experiment_metadata)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "#### GET experiment definition details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "details = client.repository.get_experiment_details(experiment_uid)"
        }, 
        {
            "source": "#### Delete experiment definition from repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "client.repository.delete(experiment_uid)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"run\"></a>\n## 4. Run experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.1 Running experiments", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 49, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "experiment_run_details = client.experiments.run(experiment_uid, asynchronous=True)"
        }, 
        {
            "source": "As we can see experiment run has been triggered.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### LIST experiment runs\nYou can list experiment run using `client.experiments.list_runs()` method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 50, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ------------------------------------  -----------------  ---------  --------------------\nGUID (experiment)                     GUID (run)                            NAME (experiment)  STATE      CREATED\n207928e4-2634-48b2-80d1-acbc3eed3db2  1b1b2396-2f7a-46d1-801b-4e36dc1c5029  MNIST experiment   completed  2018-03-15T10:21:40Z\n298cc42b-3138-43a2-a690-bf18292e7f65  bc8655b4-5723-47d2-ab98-778f0f433b9c  STYLE experiment   completed  2018-03-16T10:34:08Z\n298cc42b-3138-43a2-a690-bf18292e7f65  dab3c0e5-caaf-4eef-9898-1547919df058  STYLE experiment   completed  2018-03-16T10:34:40Z\n3b274502-a203-430f-b2b2-903f1e713d40  d8b27753-4eee-4ad0-a356-ba92818867f3  MNIST experiment   completed  2018-03-16T10:00:54Z\n496a28ea-7887-4b3e-b4c0-28d21b831333  bfb9774b-b730-4a1d-adc0-4e6ba84185be  MNIST experiment   completed  2018-03-15T15:42:27Z\n4bae6fcb-f490-45b7-bdaf-559030b6a0f9  3dd16a25-eb54-445d-85f4-eb7291d0ba60  STYLE experiment   completed  2018-03-15T21:46:13Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  0e8690af-565f-44ac-95b7-9f1e0db4755b  STYLE experiment   completed  2018-03-16T09:32:59Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  c6b6af18-409e-46ed-9dc4-5c4159332536  STYLE experiment   completed  2018-03-16T08:37:37Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  e328d165-acc6-4d4f-abd3-edb05838cfcc  STYLE experiment   completed  2018-03-16T09:14:40Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  eb89050e-ee91-404d-8b66-add9e68f6f66  STYLE experiment   completed  2018-03-16T08:39:53Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  feae525e-e654-46d9-aad9-b01e43f78147  STYLE experiment   completed  2018-03-16T09:01:35Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  feb1c972-c010-42fa-92d8-3fcd6598fd31  STYLE experiment   completed  2018-03-16T08:48:27Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  92a128f3-ba26-4114-a4a4-8a0f7c4c45dd  STYLE experiment   completed  2018-03-15T21:36:18Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  ccc0e8b0-614c-42d8-a740-7c3de5c9b41d  STYLE experiment   completed  2018-03-15T20:57:25Z\ndc222702-508e-4c27-aae5-a0af07c3a810  a8fc6bf2-dc47-425f-97d0-0939b69afd12  MNIST experiment   pending    2018-03-16T12:39:34Z\nf35cea6e-4461-45d2-b1b7-d2250fc1a59e  b59c47b4-154f-4f43-a668-da6b5d5bbb7c  STYLE experiment   completed  2018-03-15T20:36:01Z\nff7bada4-7530-4c1d-8de1-59684727e021  2d5d811f-a4fb-4a48-8ee4-2e440e2a4146  STYLE experiment   completed  2018-03-16T10:04:41Z\n------------------------------------  ------------------------------------  -----------------  ---------  --------------------\n"
                }
            ], 
            "source": "client.experiments.list_runs()"
        }, 
        {
            "source": "#### GET experiment run UID", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 51, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "a8fc6bf2-dc47-425f-97d0-0939b69afd12\n"
                }
            ], 
            "source": "experiment_run_uid = client.experiments.get_run_uid(experiment_run_details)\nprint(experiment_run_uid)"
        }, 
        {
            "source": "#### LIST training runs triggered by experiment run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Hint:** Please run the cell below several times durign the run to see updates.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 71, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--------------------  ---------  ---------  --------------------  --------------------  --------------------\nGUID (training)       NAME       STATE      SUBMITTED             FINISHED              PERFORMANCE\ntraining-3rLt7_giR    MNIST_CNN  completed  2018-03-16T12:39:44Z  2018-03-16T12:42:44Z  test:acc=0.9864\n                                                                                        test:loss=0.0475\n                                                                                        test:val_acc=0.9891\n                                                                                        test:val_loss=0.0313\ntraining-S3ttnlgig    MNIST_MLP  completed  2018-03-16T12:39:36Z  2018-03-16T12:43:11Z\ntraining-S3ttnlgig_0  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9616\n                                                                                        test:val_loss=0.1249\ntraining-S3ttnlgig_1  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9753\n                                                                                        test:val_loss=0.0874\ntraining-S3ttnlgig_2  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9784\n                                                                                        test:val_loss=0.0710\n--------------------  ---------  ---------  --------------------  --------------------  --------------------\n"
                }
            ], 
            "source": "client.experiments.list_training_runs(experiment_run_uid)"
        }, 
        {
            "source": "#### GET run details\nUse below method to get particular experiment run details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 53, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_run_details = client.experiments.get_run_details(experiment_run_uid)"
        }, 
        {
            "source": "#### GET experiment run status\nYou can check experiment run status by calling `client.experiments.get_status(run_uid)` method. This is quite useful when running experiment in background.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 54, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 54, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'current_at': '2018-03-16T12:39:34Z',\n 'current_iteration': 1,\n 'state': 'pending',\n 'submitted_at': '2018-03-16T12:39:34Z'}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.experiments.get_status(experiment_run_uid)"
        }, 
        {
            "source": "#### GET experiment details\nUse below method to get all information about particular experiment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 55, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_details = client.experiments.get_details(experiment_uid)"
        }, 
        {
            "source": "You can use below print method to display experiment details (change cell type to code).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "print(json.dumps(experiment_details, indent=2))", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "#### GET training runs uids", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 79, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "training-3rLt7_giR\ntraining-S3ttnlgig\ntraining-S3ttnlgig_0\ntraining-S3ttnlgig_1\ntraining-S3ttnlgig_2\n"
                }
            ], 
            "source": "experiment_run_details = client.experiments.get_run_details(experiment_run_uid)\ntraining_run_uids = client.experiments.get_training_uids(experiment_run_details)\n\nfor i in training_run_uids:\n    print(i)"
        }, 
        {
            "source": "#### Delete experiment-run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can delete experiment runs and results by running below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** the `delete` method can be also used to terminate experiment run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "client.experiments.delete(experiment_run_id)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "### 4.2 Monitoring experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### MONITOR experiment run\nYou can monitor experiment run by calling `client.experiments.monitor_logs(run_uid)`. This method will stream training logs content to console.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** You can also monitor particular training run by calling `client.training.monitor_logs(training_run_uid)`.\nTo get training_run_uid you can call method `client.experiments.list_training_runs(experiment_run_uid)`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client.experiments.monitor_logs(experiment_run_uid)"
        }, 
        {
            "source": "**Note:** Before going to next section make sure that your experiment run has completed.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 58, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 58, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "'completed'"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.experiments.get_status(experiment_run_uid)['state']"
        }, 
        {
            "source": "### 4.3 Quality metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Accessing metrics data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can get final evaluation metrics by running below cell.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 59, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "metrics = client.experiments.get_latest_metrics(experiment_run_uid)"
        }, 
        {
            "source": "You can get all evaluation metrics by running below code.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 60, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "all_metrics = client.experiments.get_metrics(experiment_run_uid)"
        }, 
        {
            "source": "#### Visualize training results using plotly.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "!pip install cufflinks", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "execution_count": 61, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>", 
                        "text/vnd.plotly.v1+html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "import sys\nimport pandas\nimport plotly.plotly as py\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\nimport plotly.graph_objs as go\n\ninit_notebook_mode(connected=True)\nsys.path.append(\"\".join([os.environ[\"HOME\"]])) "
        }, 
        {
            "source": "Let's put our final metrics data into pandas dataframe.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 62, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 62, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>GUID</th>\n      <th>NAME</th>\n      <th>METRIC NAME</th>\n      <th>METRIC VALUE</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>acc</td>\n      <td>0.986400</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>loss</td>\n      <td>0.047485</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>val_acc</td>\n      <td>0.989100</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>val_loss</td>\n      <td>0.031273</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>training-S3ttnlgig_0</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.961600</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>training-S3ttnlgig_0</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.124866</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>training-S3ttnlgig_1</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.975300</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>training-S3ttnlgig_1</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.087398</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>training-S3ttnlgig_2</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.978400</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>training-S3ttnlgig_2</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.071028</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "                   GUID       NAME METRIC NAME  METRIC VALUE\n0    training-3rLt7_giR  MNIST_CNN         acc      0.986400\n1    training-3rLt7_giR  MNIST_CNN        loss      0.047485\n2    training-3rLt7_giR  MNIST_CNN     val_acc      0.989100\n3    training-3rLt7_giR  MNIST_CNN    val_loss      0.031273\n4  training-S3ttnlgig_0  MNIST_MLP     val_acc      0.961600\n5  training-S3ttnlgig_0  MNIST_MLP    val_loss      0.124866\n6  training-S3ttnlgig_1  MNIST_MLP     val_acc      0.975300\n7  training-S3ttnlgig_1  MNIST_MLP    val_loss      0.087398\n8  training-S3ttnlgig_2  MNIST_MLP     val_acc      0.978400\n9  training-S3ttnlgig_2  MNIST_MLP    val_loss      0.071028"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "import pandas as pd\n\nmetrics_df = pd.DataFrame(columns=['GUID', 'NAME', 'METRIC NAME', 'METRIC VALUE'])\n\nfor m in metrics:\n    for v in m['metrics']['values']:\n            metrics_df = metrics_df.append({'GUID': m['training_guid'], 'NAME': m['training_reference_name'], 'METRIC NAME': v['name'], 'METRIC VALUE': v['value']}, ignore_index=True)\n    \nmetrics_df"
        }, 
        {
            "source": "Visualize metrics data using grouped bar chart.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 63, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\", [{\"y\": [0.9864000082015991], \"type\": \"bar\", \"name\": \"acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.047485120594501495], \"type\": \"bar\", \"name\": \"loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.9890999794006348, 0.9616, 0.9753, 0.9784], \"type\": \"bar\", \"name\": \"val_acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}, {\"y\": [0.03127295523881912, 0.1248657464325428, 0.08739820090634748, 0.07102841361872853], \"type\": \"bar\", \"name\": \"val_loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}], {\"barmode\": \"group\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\", [{\"y\": [0.9864000082015991], \"type\": \"bar\", \"name\": \"acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.047485120594501495], \"type\": \"bar\", \"name\": \"loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.9890999794006348, 0.9616, 0.9753, 0.9784], \"type\": \"bar\", \"name\": \"val_acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}, {\"y\": [0.03127295523881912, 0.1248657464325428, 0.08739820090634748, 0.07102841361872853], \"type\": \"bar\", \"name\": \"val_loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}], {\"barmode\": \"group\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "y": [
                                        0.9864000082015991
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)"
                                    ], 
                                    "type": "bar", 
                                    "name": "acc"
                                }, 
                                {
                                    "y": [
                                        0.047485120594501495
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)"
                                    ], 
                                    "type": "bar", 
                                    "name": "loss"
                                }, 
                                {
                                    "y": [
                                        0.9890999794006348, 
                                        0.9616, 
                                        0.9753, 
                                        0.9784
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)", 
                                        "training-S3ttnlgig_0 (MNIST_MLP)", 
                                        "training-S3ttnlgig_1 (MNIST_MLP)", 
                                        "training-S3ttnlgig_2 (MNIST_MLP)"
                                    ], 
                                    "type": "bar", 
                                    "name": "val_acc"
                                }, 
                                {
                                    "y": [
                                        0.03127295523881912, 
                                        0.1248657464325428, 
                                        0.08739820090634748, 
                                        0.07102841361872853
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)", 
                                        "training-S3ttnlgig_0 (MNIST_MLP)", 
                                        "training-S3ttnlgig_1 (MNIST_MLP)", 
                                        "training-S3ttnlgig_2 (MNIST_MLP)"
                                    ], 
                                    "type": "bar", 
                                    "name": "val_loss"
                                }
                            ], 
                            "layout": {
                                "barmode": "group"
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "data = []\n\nfor i in list(pd.unique(metrics_df['METRIC NAME'])):\n    data.append(go.Bar(x=metrics_df[metrics_df['METRIC NAME'].isin([i])]['GUID'] + ' (' + metrics_df[metrics_df['METRIC NAME'].isin([i])]['NAME'] + ')', y=metrics_df[metrics_df['METRIC NAME'].isin([i])]['METRIC VALUE'], name=i))\n\n\nlayout = go.Layout(\n    barmode='group'\n)\n\nfig = go.Figure(data=data, layout=layout)\n\niplot(fig)"
        }, 
        {
            "source": "## 5. Create online deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can deployed stored model as webservice (online) by running below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.1 Store trained model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Call this method to save model in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 64, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "training-3rLt7_giR (MNIST_CNN)\n"
                }
            ], 
            "source": "status = client.experiments.get_status(experiment_run_uid)\nbest_model_uid = status['best_results']['experiment_best_model']['training_guid']\nbest_model_name = status['best_results']['experiment_best_model']['training_reference_name']\n\nprint(best_model_uid + ' (' +  best_model_name  + ')')"
        }, 
        {
            "execution_count": 81, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "saved_model_details = client.repository.store_model(best_model_uid, {'name': 'MNIST best model'})"
        }, 
        {
            "source": "**Tip:** Run below code to extract stored model guid from dictionary.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 82, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Saved model guid: f7219c8f-fccb-4d26-a1bd-f51544090eae\n"
                }
            ], 
            "source": "model_guid = client.repository.get_model_uid(saved_model_details)\nprint(\"Saved model guid: \" + model_guid)"
        }, 
        {
            "source": "#### Call this method to list stored models", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 83, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  -------------------------------  ------------------------  --------------\nGUID                                  NAME                             CREATED                   FRAMEWORK\n43986a55-0f0e-493b-9b98-0712f918b0ba  MNIST CNN model                  2018-03-15T16:12:06.503Z  tensorflow-1.5\na5a952e1-fa6f-4c64-88f7-293b367b7e1f  MNIST best model                 2018-03-16T10:12:16.773Z  tensorflow-1.5\nc755c740-dab6-4d5d-b873-1060885d55d5  MNIST best model                 2018-03-16T10:12:31.990Z  tensorflow-1.5\n861b6b73-6861-49c0-867c-d171d6d9d545  MNIST best model                 2018-03-16T13:00:40.461Z  tensorflow-1.5\nf7219c8f-fccb-4d26-a1bd-f51544090eae  training-S3ttnlgig_2 best model  2018-03-16T13:32:33.747Z  tensorflow-1.5\n------------------------------------  -------------------------------  ------------------------  --------------\n"
                }
            ], 
            "source": "client.repository.list_models()"
        }, 
        {
            "source": "### 5.2 Create online deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 68, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: '1bf606e8-e8c4-4143-8727-5b2d1ac97b7e' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_IN_PROGRESS..\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='1bf606e8-e8c4-4143-8727-5b2d1ac97b7e'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "deployment_details = client.deployments.create(name=\"MNIST keras deployment\", model_uid=model_guid)"
        }, 
        {
            "source": "Extract `scoring endpoint` from deployment details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 72, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/861b6b73-6861-49c0-867c-d171d6d9d545/deployments/1bf606e8-e8c4-4143-8727-5b2d1ac97b7e/online\n"
                }
            ], 
            "source": "scoring_url = client.deployments.get_scoring_url(deployment_details)\nprint(scoring_url)"
        }, 
        {
            "source": "## 6. Scoring", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "Prepare sample scoring data to score deployed model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 73, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "Using TensorFlow backend.\n"
                }
            ], 
            "source": "from keras.datasets import mnist\n\n(x_train, y_train), (x_test, y_test) = mnist.load_data()"
        }, 
        {
            "source": "Let's plot two digits.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 74, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np"
        }, 
        {
            "execution_count": 75, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT4AAACFCAYAAADCQpQyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAB3NJREFUeJzt3ctLFn8Ux/GxfpGU3aQwg5JIsKLM\nhRBGYLeFRUUlXUCiRStdtGjRQtp0oZtGi2pVf4BQUYl0gYQgicggKjBwURlEFJlEC8mKfotfHL5n\nfjMP4+PMOI/n/Vqdw3eYmcX4YZ6v35kp+vPnjwcAlkwa7xMAgLQRfADMIfgAmEPwATCH4ANgDsEH\nwByCD4A5BB8Acwg+AOb8k/LxeEwkO4rG+wQmEK7r7Ih0XXPHB8Acgg+AOQQfAHMIPgDmEHwAzCH4\nAJhD8AEwh+ADYA7BB8Acgg+AOWk/sgYgQe3t7aofHh6W+uXLl2rs+vXroftpbm5WfV1dndT79+8f\nyylmAnd8AMwh+ACYU5Tyd3V5i0V28HaW+Izrdb13716pr127lsgxKisrpX7w4IEaW7RoUSLHzBNv\nZwGAIAQfAHMIPgDmMMdnF3N88Un1unbn9Dwv+rze0qVLVd/Q0CD1mzdv1FhnZ2fofk6ePKn61tbW\nSMdPCXN8ABCE4ANgDk9uAAXg2bNnUt+8eTN0uxUrVqje/ck6d+5cNVZSUiL1yMiIGlu9erXqX7x4\nIfXg4GCEM8427vgAmEPwATCH4ANgTsHP8fnfMHHlyhWpFyxYoMaKi4ulbmpqUmPz58+X2n08B8iC\njx8/Su1fgubO692/f1+NlZeXR9q//60ur1+/Dt1269atkfaZZdzxATCH4ANgTsE/ubF48WLVv3v3\nLq/9zJw5U+rly5eP5ZTysnDhQqmPHDmixmpra5M4JE9uxCfVP6KBgQHVz5gxQ+rS0tK89rlq1SrV\nv3r1KnTb7u5u1a9fvz6vYyaEJzcAIAjBB8Acgg+AOQW/nOXq1auqdx+t8c/V9fX1Sf38+XM19vDh\nQ6mfPHmixtw3zL5//z7yuU2ZMkX17iND7vIE/zHd+T7PS2yODwWqoqIilv20tbVJ3d/fn3Nb9xE2\n/+NshYg7PgDmEHwAzCn45SxxGRoaktr/M9j9qdnb2xt5n1OnTlV9VVWV1P6XQn79+lXqy5cvq7GW\nlpbIxxwFlrPEJ7PXtaurq0v1u3fvlvrHjx9qrKysTPUdHR1S19fXJ3B2sWE5CwAEIfgAmEPwATCn\n4JezxGXOnDlSb9iwIXS7jRs35n2MGzduSO3OKXqe51VXV0u9b9++vI8BhHHf4ux5/5/Xc/k/aJTx\neb1R444PgDkEHwBzWM6SoM+fP6t+5cqVoWPuC1UbGxuTPbH/sJwlPpm9rnfs2CG1/yWl7k/dAwcO\nqLGLFy+q3v0wUcaxnAUAghB8AMwh+ACYw3KWBPkfPXPn9WbPnq3G3MfZgHz53/rz+PFjqf3LV+bN\nmyf10aNH1VgBzenlhTs+AOYQfADM4aduzHp6eqQ+c+ZM6Ha3b99WvfttVCBfu3btUv2XL19Ct3W/\nLb1kyZLEzimLuOMDYA7BB8Acgg+AOczxxezOnTtSj4yMqLFNmzZJXVdXl9o5YWLr7OyU2v/2cNe6\ndetUf/z48aROKfO44wNgDsEHwByCD4A5zPGN0fDwsOrv3bsntf8ra8eOHZPa/7FxIKrBwUHVnzp1\nSmr/vLKrpqZG9RP9sbRcuOMDYA7BB8AcfuqOUVtbm+rd5QSbN29WY2vWrEnlnDCxnT9/XvVPnz4N\n3dZ9A7Pl5St+3PEBMIfgA2AOwQfAHL6yNkpdXV2q37lzp+qnT58u9d27d9VYxh5T4ytr8Un1ui4u\nLlZ9riUsHz58kLq8vDyxc8oQvrIGAEEIPgDmsJwlAnel/KFDh9TYr1+/VL9lyxapM/bTFga51+5Y\nnhaaNWtW6H5+/vwp9bdv30L3MTQ0pPoLFy5EOvbkyZNVf/bsWamnTZsWaR9+3PEBMIfgA2AOwQfA\nHOb4Avz+/Vv1DQ0NUr99+1aNVVZWqv7EiRPJnRgwStXV1bHsZ8+ePVL7l8V8+vRJ6o6OjliOl0tZ\nWZnU/g+hR8UdHwBzCD4A5vDkRoD+/n7VV1VVhW7rfujF8zxv27ZtiZxTAnhyIz6pXtf+j4bfunUr\nzcOPirv0ZdKk8Pus7du3q762tjZ027Vr10odsGSMJzcAIAjBB8Acgg+AOczx/TUwMCB1fX196Fh7\ne7saO3z4sOqLigpm6qxgTrQAjOt1fe7cOalzvanFr6+vT+rRLEM5ePCg6isqKkK3bWxslHrZsmWR\njzEGzPEBQBCCD4A5/NT9q7W1VerTp0+Hbtfb26v6XP92zzh+6sYns9e1QfzUBYAgBB8Acwg+AOaY\nfTvLo0ePVH/p0qVxOhMAaeOOD4A5BB8Ac8z+1O3p6VH99+/fQ7d1XzZaUlKS2DkBSAd3fADMIfgA\nmEPwATDH7BxfLjU1Narv7u6WurS0NO3TARAz7vgAmEPwATCHt7PYxdtZ4sN1nR28nQUAghB8AMwh\n+ACYk/ZyFuaVMBFxXRcY7vgAmEPwATCH4ANgDsEHwByCD4A5BB8Acwg+AOYQfADMIfgAmEPwATCH\n4ANgDsEHwByCD4A5BB8Acwg+AOYQfADMIfgAmEPwATCH4ANgDsEHwByCD4A5BB8Acwg+AOb8C3bx\nik91/Qo6AAAAAElFTkSuQmCC\n", 
                        "text/plain": "<matplotlib.figure.Figure at 0x7fe6c7066d30>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "for i, image in enumerate([x_test[0], x_test[1]]):\n    plt.subplot(2, 2, i + 1)\n    plt.axis('off')\n    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')"
        }, 
        {
            "source": "Our input node expects to get data with shape (28,28,1) so we need to reshape our two digits.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 76, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "image_1 = np.expand_dims(x_test[0], axis=2)\nimage_2 = np.expand_dims(x_test[1], axis=2)"
        }, 
        {
            "source": "Prepare scoring payload and score.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 77, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "scoring_data = {'values': [image_1.tolist(), image_2.tolist()]}"
        }, 
        {
            "execution_count": 78, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Scoring result: {'fields': ['prediction', 'prediction_classes', 'probability'], 'values': [[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0], 7, [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]], [[0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 2, [0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]]}\n"
                }
            ], 
            "source": "predictions = client.deployments.score(scoring_url, scoring_data)\nprint(\"Scoring result: \" + str(predictions))"
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 7. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": " You successfully completed this notebook! You learned how to use `watson-machine-learning-client` to run experiments. Check out our _[Online Documentation](https://console.ng.bluemix.net/docs/services/PredictiveModeling/index.html)_ for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Authors\n\n**Lukasz Cmielowski**, PhD, is a Automation Architect and Data Scientist in IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}