Use Keras experiment with HPO to predict digits.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<div style=\"background:#F5F7FA; height:100px; padding: 2em; font-size:14px;\">\n<span style=\"font-size:18px;color:#152935;\">Want to do more?</span><span style=\"border: 1px solid #3d70b2;padding: 15px;float:right;margin-right:40px; color:#3d70b2; \"><a href=\"https://ibm.co/wsnotebooks\" target=\"_blank\" style=\"color: #3d70b2;text-decoration: none;\">Sign Up</a></span><br>\n<span style=\"color:#5A6872;\"> Try out this notebook with your free trial of IBM Watson Studio.</span>\n</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"4\" color=\"black\"><b>From keras experiment to scoring with watson-machine-learning-client</b></font></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr> \n   <tr style=\"border: none\">\n       <td style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/raw/master/tensorflow/hand-written-digit-recognition/images/experiment_banner.png\" width=\"600\" alt=\"Icon\"></td>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook contains steps and code to demonstrate support of deep learning experiments in Watson Machine Learning Service. This notebook introduces commands for getting data, training_definition persistance, experiment training, model persistance, model deployment and scoring.\n\nSome familiarity with Python is helpful. This notebook uses Python 3.\n\n\n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Working with Watson Machine Learning experiments to train Deep Learning models (Keras).\n-  Saving trained models in Watson Machine Learning repository.\n-  Online deployment and scoring of trained model.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Set up the environment](#setup)\n2.\t[Training definition](#definition)\n3.\t[Experiment definition](#experiment)\n4.\t[Experiment Run](#run)\n5.\t[Deploy and Score](#scoring)\n6.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Set up the environment\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning (WML) Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered and information about how to create the instance is [here](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html))\n-  Create a [Cloud Object Storage (COS)](https://console.bluemix.net/catalog/infrastructure/cloud-object-storage) instance (a lite plan is offered and information about how to order storage is [here](https://console.bluemix.net/docs/services/cloud-object-storage/basics/order-storage.html#order-storage)).\n    - After you create COS instance, go to your COS dashboard.\n    - In **Service credentials** tab, click **New Credential**.\n    - Add the inline configuration parameter: {[\"HMAC\"](https://console.bluemix.net/docs/services/cloud-object-storage/hmac/credentials.html#using-hmac-credentials):true}, click **Add**.\n\n    This configuration parameter adds the following section to the instance credentials, (for use later in this notebook):\n    ```\n      \"cos_hmac_keys\": {\n            \"access_key_id\": \"722432c254bc4eaa96e05897bf2779e2\",\n            \"secret_access_key\": \"286965ac10ecd4de8b44306288c7f5a3e3cf81976a03075c\"\n       }", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 1.1 Work with Cloud Object Storage (COS)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Import the Boto library, which allows Python developers to manage COS.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** If `ibm_boto3` is not preinstalled in you environment, run the following command to install it: ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Run command if ibm_boto3 is not installed.\n#!pip install ibm-cos-sdk"
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import ibm_boto3\nfrom ibm_botocore.client import Config\nimport os\nimport json\nimport warnings\nimport time"
        }, 
        {
            "source": "Define the endpoint you will use. \n\n1. To do this, first enter your COS credentials in the following cell. \nYou can find these credentials in your COS instance dashboard under the **Service credentials** tab.\n**Note** the HMAC key, described in [set up the environment](#setup) is included in these credentials.\n\n2. Go to the **Endpoint** tab in the COS instance's dashboard to get the endpoint information.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos_credentials = {\n  \"apikey\": \"----\",\n  \"cos_hmac_keys\": {\n    \"access_key_id\": \"---\",\n    \"secret_access_key\": \"---\"\n  },\n  \"endpoints\": \"https://cos-service.bluemix.net/endpoints\",\n  \"iam_apikey_description\": \"Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/bc6f89c9dbf0a21338f4ee6fa02cf6cf:d2901ba2-3196-4663-aee9-7ddfbb49eb7d::\",\n  \"iam_apikey_name\": \"auto-generated-apikey-cd81fb86-8f99-4bbf-944d-6cc4bc54f2be\",\n  \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Writer\",\n  \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::a/bc6f89c9dbf0a21338f4ee6fa02cf6cf::serviceid:ServiceId-b6138191-c9a3-4349-8b20-ddc89f13c78a\",\n  \"resource_instance_id\": \"crn:v1:bluemix:public:cloud-object-storage:global:a/bc6f89c9dbf0a21338f4ee6fa02cf6cf:d2901ba2-3196-4663-aee9-7ddfbb49eb7d::\"\n}\n\napi_key = cos_credentials['apikey']\nservice_instance_id = cos_credentials['resource_instance_id']\nauth_endpoint = 'https://iam.bluemix.net/oidc/token'\nservice_endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'"
        }, 
        {
            "source": "We create Boto resource by providing type, endpoint_url and credentials.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "cos = ibm_boto3.resource('s3',\n                         ibm_api_key_id=api_key,\n                         ibm_service_instance_id=service_instance_id,\n                         ibm_auth_endpoint=auth_endpoint,\n                         config=Config(signature_version='oauth'),\n                         endpoint_url=service_endpoint)"
        }, 
        {
            "source": "Let's create the buckets we will use to store training data and training results.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Note:**: Bucket name has to be unique - please update rename the buckets you create and give them any unique name.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Creating bucket \"mnist-keras-data-example\"...\nCreating bucket \"mnist-keras-results-example\"...\n"
                }
            ], 
            "source": "buckets = ['mnist-keras-data-example', 'mnist-keras-results-example']\nfor bucket in buckets:\n    if not cos.Bucket(bucket) in cos.buckets.all():\n        print('Creating bucket \"{}\"...'.format(bucket))\n        try:\n            cos.create_bucket(Bucket=bucket)\n        except ibm_boto3.exceptions.ibm_botocore.client.ClientError as e:\n            print('Error: {}.'.format(e.response['Error']['Message']))"
        }, 
        {
            "source": "Now we should have our buckets created.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[s3.Bucket(name='mnist-keras-data-example'), s3.Bucket(name='mnist-keras-results-example'), s3.Bucket(name='style-data-example'), s3.Bucket(name='style-results-example')]\n"
                }
            ], 
            "source": "print(list(cos.buckets.all()))"
        }, 
        {
            "source": "### 1.2 Downloading MNIST data and upload it to COS buckets", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "We will work with Keras **MNIST** sample dataset. Let's download our training data and upload them to 'mnist-keras-data' bucket.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Run the code in the cell below to create 'MNIST_KERAS_DATA' folder and download the file from the link.\n\n**Tip:** First, install wget library with this command:\n`!pip install wget`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "link = 'https://s3.amazonaws.com/img-datasets/mnist.npz'"
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "mnist.npz\r\n"
                }
            ], 
            "source": "import wget, os\n\ndata_dir = 'MNIST_KERAS_DATA'\nif not os.path.isdir(data_dir):\n    os.mkdir(data_dir)\n\nif not os.path.isfile(os.path.join(data_dir, os.path.join(link.split('/')[-1]))):\n    wget.download(link, out=data_dir)  \n        \n!ls MNIST_KERAS_DATA"
        }, 
        {
            "source": "Upload the data files to created buckets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "mnist.npz is uploaded.\n"
                }
            ], 
            "source": "bucket_name = buckets[0]\nbucket_obj = cos.Bucket(bucket_name)\n\nfor filename in os.listdir(data_dir):\n    with open(os.path.join(data_dir, filename), 'rb') as data: \n        bucket_obj.upload_file(os.path.join(data_dir, filename), filename)\n        print('{} is uploaded.'.format(filename))"
        }, 
        {
            "source": "Let's see the list of all buckets and their contents.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Object key: mnist.npz\nObject size (kb): 11221.126953125\n"
                }
            ], 
            "source": "for obj in bucket_obj.objects.all():\n    print('Object key: {}'.format(obj.key))\n    print('Object size (kb): {}'.format(obj.size/1024))"
        }, 
        {
            "source": "We are done with Cloud Object Storage, we are ready to train our model!", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 1.3 Work with the Watson Machine Learning instance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Authenticate to Watson Machine Learning service on IBM Cloud.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import urllib3, requests, json, base64, time, os\nwarnings.filterwarnings('ignore')"
        }, 
        {
            "source": "Authenticate to the Watson Machine Learning service on IBM Cloud.\n\n**Tip**: Authentication information (your credentials) can be found in the [Service Credentials](https://console.bluemix.net/docs/services/service_credentials.html#service_credentials) tab of the service instance that you created on IBM Cloud. <BR>If you cannot see the **instance_id** field in **Service Credentials**, click **New credential (+)** to generate new authentication information. \n\n**Action**: Enter your Watson Machine Learning service instance credentials here.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "wml_credentials = {\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"YrY6e1xi6rJemA1yEB2iaTojb0OKZTpi79o+4MSFrSG3BM2IFZaDZwvBcoQXGXStpxSFKe9cZoFYLlzgPf++qpWZYcc+6fawL9S0V+2V79Adc+zik+ZHJYrsBRl9GAcs\",\n  \"username\": \"---\",\n  \"password\": \"---\",\n  \"instance_id\": \"---\"\n}"
        }, 
        {
            "source": "#### Install `watson-machine-learning-client` from pypi", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "!pip install --upgrade watson-machine-learning-client", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client"
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement already up-to-date: watson-machine-learning-client in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages\nRequirement already up-to-date: tqdm in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: tabulate in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: urllib3 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: certifi in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: pandas in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: lomond in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: requests in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from watson-machine-learning-client)\nRequirement already up-to-date: python-dateutil>=2 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: numpy>=1.9.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: pytz>=2011k in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from pandas->watson-machine-learning-client)\nRequirement already up-to-date: six>=1.10.0 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from lomond->watson-machine-learning-client)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/local/src/conda3_runtime.v28/home/envs/DSX-Python35-Spark/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\nRequirement already up-to-date: idna<2.7,>=2.5 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/sec7-2ac43fd194bc9a-8ef090487f81/.local/lib/python3.5/site-packages (from requests->watson-machine-learning-client)\n"
                }
            ], 
            "source": "!pip install --upgrade watson-machine-learning-client"
        }, 
        {
            "source": "#### Import `watson-machine-learning-client` and authenticate to service instance", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "1.0.12\n"
                }
            ], 
            "source": "print(client.version)"
        }, 
        {
            "source": "<a id=\"definition\"></a>\n## 2. Training definitions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "For purpose of this example two Keras model definitions have been prepared:\n* Multilayer Perceptron (MLP)\n* Convolution Neural Network (CNN)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 2.2 Save training definition ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Prepare training definition metadata", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_definition_1_metadata = {\n            client.repository.DefinitionMetaNames.NAME: \"MNIST-MLP\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_NAME: \"tensorflow\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_VERSION: \"1.5\",\n            client.repository.DefinitionMetaNames.RUNTIME_NAME: \"python\",\n            client.repository.DefinitionMetaNames.RUNTIME_VERSION: \"3.5\",\n            client.repository.DefinitionMetaNames.EXECUTION_COMMAND: \"python3 mnist_mlp.py\"\n            }"
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "model_definition_2_metadata = {\n            client.repository.DefinitionMetaNames.NAME: \"MNIST-CNN\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_NAME: \"tensorflow\",\n            client.repository.DefinitionMetaNames.FRAMEWORK_VERSION: \"1.5\",\n            client.repository.DefinitionMetaNames.RUNTIME_NAME: \"python\",\n            client.repository.DefinitionMetaNames.RUNTIME_VERSION: \"3.5\",\n            client.repository.DefinitionMetaNames.EXECUTION_COMMAND: \"python3 mnist_cnn.py\"\n            }"
        }, 
        {
            "source": "#### Get sample model definition content files from git (python scripts with CNN and MLP)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!rm -rf MNIST.zip"
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--2018-03-16 07:35:25--  https://github.com/pmservice/wml-sample-models/raw/master/keras/mnist/MNIST.zip\nResolving github.com (github.com)... 192.30.253.112, 192.30.253.113\nConnecting to github.com (github.com)|192.30.253.112|:443... connected.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://raw.githubusercontent.com/pmservice/wml-sample-models/master/keras/mnist/MNIST.zip [following]\n--2018-03-16 07:35:26--  https://raw.githubusercontent.com/pmservice/wml-sample-models/master/keras/mnist/MNIST.zip\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 3836 (3.7K) [application/zip]\nSaving to: \u2018MNIST.zip\u2019\n\n100%[======================================>] 3,836       --.-K/s   in 0s      \n\n2018-03-16 07:35:26 (28.6 MB/s) - \u2018MNIST.zip\u2019 saved [3836/3836]\n\nMNIST.zip\n"
                }
            ], 
            "source": "filename_mnist = 'MNIST.zip'\n\nif os.path.isfile(filename_mnist):\n    !ls 'MNIST.zip'\nelse:\n    !wget https://github.com/pmservice/wml-sample-models/raw/master/keras/mnist/MNIST.zip\n    !ls 'MNIST.zip'"
        }, 
        {
            "source": "#### Publish training definitions in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "##### Store definition 1", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/ml_assets/training_definitions/d56f3240-b49e-4b30-98e8-fb1ca8ab6dd6\n"
                }
            ], 
            "source": "definition_1_details = client.repository.store_definition(filename_mnist, model_definition_1_metadata)\n\ndefinition_1_url = client.repository.get_definition_url(definition_1_details)\ndefinition_1_uid = client.repository.get_definition_uid(definition_1_details)\nprint(definition_1_url)"
        }, 
        {
            "source": "##### Store definition 2", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/ml_assets/training_definitions/970df64d-d363-4524-881d-75320928c695\n"
                }
            ], 
            "source": "definition_2_details = client.repository.store_definition(filename_mnist, model_definition_2_metadata)\n\ndefinition_2_url = client.repository.get_definition_url(definition_2_details)\ndefinition_2_uid = client.repository.get_definition_uid(definition_2_details)\nprint(definition_2_url)"
        }, 
        {
            "source": "#### LIST stored definitions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ------------------------  ------------------------  ----------\nGUID                                  NAME                      CREATED                   FRAMEWORK\nea8c0dd5-8a6b-4e5e-a9cd-74a0c573c1de  MNIST-MLP                 2018-03-15T10:20:58.646Z  tensorflow\n8095a87c-5c94-4f43-b5fd-468b2d72a3c8  MNIST-CNN                 2018-03-15T10:21:00.033Z  tensorflow\nbebc485e-bcc9-45b7-9bce-1ba20acc3b53  MNIST-MLP                 2018-03-15T10:35:04.997Z  tensorflow\nf26539ff-43b5-4100-8915-7b0be22b2211  MNIST-CNN                 2018-03-15T10:35:05.815Z  tensorflow\nbed45da1-6372-4cb1-b889-e0d402cfe62a  MNIST-MLP                 2018-03-15T15:41:40.335Z  tensorflow\nc80aaeef-1c03-4474-8891-535aa6dae81e  MNIST-CNN                 2018-03-15T15:41:41.174Z  tensorflow\nf819f724-8e1c-4903-a04c-eee5dfc048a4  style transfer van gogh   2018-03-15T20:33:03.354Z  tensorflow\nf5c4e713-27d9-4529-a473-10dc88383a52  style transfer kandinsky  2018-03-15T20:33:05.032Z  tensorflow\naf72a944-cc83-4aaf-9152-9f25c1912bf4  style transfer van gogh   2018-03-15T20:57:13.232Z  tensorflow\n4fb3987a-f979-4e1e-a26f-1acbe11a08e7  style transfer kandinsky  2018-03-15T20:57:13.892Z  tensorflow\n1421337a-c2a8-4153-8554-accdc55bbd27  style transfer van gogh   2018-03-15T21:45:46.889Z  tensorflow\n0f8fa74a-f821-475c-85c9-a2b507ed6c63  style transfer kandinsky  2018-03-15T21:45:48.344Z  tensorflow\ncd31e290-6636-4586-8097-e3c3ac7fef8c  style transfer van gogh   2018-03-16T08:37:18.880Z  tensorflow\n76b144fc-100c-45ee-80ae-ef0d44a87cd6  style transfer kandinsky  2018-03-16T08:37:19.383Z  tensorflow\n7dc7e7cc-ae5b-4db2-8705-32174324010a  MNIST-MLP                 2018-03-16T09:59:57.138Z  tensorflow\n6a024dde-d6bc-45d3-8463-895452134c29  MNIST-CNN                 2018-03-16T09:59:57.888Z  tensorflow\n32c64603-13d2-4ec6-b3ca-4ee4650fbd8e  style transfer van gogh   2018-03-16T10:04:24.779Z  tensorflow\n18498298-3e01-43a0-a1f7-955ecc118a4c  style transfer kandinsky  2018-03-16T10:04:25.661Z  tensorflow\n9e446ccc-8b1d-4e46-a549-655f9b16a577  style transfer van gogh   2018-03-16T10:33:54.588Z  tensorflow\n1f54975c-0104-4fcb-910c-ae57e3272d7c  style transfer kandinsky  2018-03-16T10:33:55.170Z  tensorflow\n3501d90f-8599-4154-a8a5-d6ddafac15a8  MNIST-MLP                 2018-03-16T11:53:19.116Z  tensorflow\n7c1b279f-7556-4beb-bee3-5516a0daa433  MNIST-CNN                 2018-03-16T11:53:19.907Z  tensorflow\nd56f3240-b49e-4b30-98e8-fb1ca8ab6dd6  MNIST-MLP                 2018-03-16T12:35:29.322Z  tensorflow\n970df64d-d363-4524-881d-75320928c695  MNIST-CNN                 2018-03-16T12:35:30.591Z  tensorflow\n------------------------------------  ------------------------  ------------------------  ----------\n"
                }
            ], 
            "source": "client.repository.list_definitions()"
        }, 
        {
            "source": "<a id=\"experiment\"></a>\n## 3. Experiment definition", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1 Save experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Get the list of supported configuration params", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--------------------------  ----  --------\nMETA_PROP NAME              TYPE  REQUIRED\nNAME                        str   Y\nTAGS                        list  N\nDESCRIPTION                 str   N\nAUTHOR_NAME                 str   N\nAUTHOR_EMAIL                str   Y\nEVALUATION_METHOD           str   N\nEVALUATION_METRICS          list  N\nTRAINING_REFERENCES         list  Y\nTRAINING_DATA_REFERENCE     dict  Y\nTRAINING_RESULTS_REFERENCE  dict  Y\n--------------------------  ----  --------\n"
                }
            ], 
            "source": "client.repository.ExperimentMetaNames.show()"
        }, 
        {
            "source": "#### Experiment configuration dictionary\nCreate experiment that will train models based on previously stored definitions.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "`TRAINING_DATA_REFERENCE` - location of traininng data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "TRAINING_DATA_REFERENCE = {\n                            \"connection\": {\n                                \"endpoint_url\": service_endpoint,\n                                \"aws_access_key_id\": cos_credentials['cos_hmac_keys']['access_key_id'],\n                                \"aws_secret_access_key\": cos_credentials['cos_hmac_keys']['secret_access_key']\n                            },\n                            \"source\": {\n                                \"bucket\": buckets[0],\n                            },\n                            \"type\": \"s3\"\n}"
        }, 
        {
            "source": "`TRAINING_RESULTS_REFERENCE` - location of training results", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "TRAINING_RESULTS_REFERENCE = {\n                                \"connection\": {\n                                    \"endpoint_url\": service_endpoint,\n                                    \"aws_access_key_id\": cos_credentials['cos_hmac_keys']['access_key_id'],\n                                    \"aws_secret_access_key\": cos_credentials['cos_hmac_keys']['secret_access_key']\n                                },\n                                \"target\": {\n                                    \"bucket\": buckets[1],\n                                },\n                                \"type\": \"s3\"\n}"
        }, 
        {
            "source": "Configure hyper parameters optimizer for you experiment. We are interested in `val_acc` metric so we need to put it as optimizer `objective`. `num_optimizer_steps` tells the optimizer how many models we want to train based on hyper parameters values combinations.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "HPO = {\n        \"method\": {\n            \"name\": \"rbfopt\",\n            \"parameters\": [\n                client.experiments.HPOMethodParam(\"objective\", \"val_acc\"),\n                client.experiments.HPOMethodParam(\"maximize_or_minimize\", \"maximize\"),\n                client.experiments.HPOMethodParam(\"num_optimizer_steps\", 3)\n            ]\n        },\n        \"hyper_parameters\": [\n            client.experiments.HPOParameter('learning_rate', min=0.0001, max=0.001, step=0.0003)\n        ]\n     }           "
        }, 
        {
            "source": "Configure your experiment. `TRAINING_REFERENCES` links previously stored training definitions and provides information about `compute_configuration` that will be used to run the training.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 45, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_metadata = {\n            client.repository.ExperimentMetaNames.NAME: \"MNIST experiment\",\n            client.repository.ExperimentMetaNames.DESCRIPTION: \"Best model for MNIST.\",\n            client.repository.ExperimentMetaNames.EVALUATION_METHOD: \"multiclass\",\n            client.repository.ExperimentMetaNames.EVALUATION_METRICS: [\"val_acc\"],\n            #client.repository.ExperimentMetaNames.EVALUATION_METRICS: [{\"name\": \"val_acc\", \"maximize\": True}],\n            client.repository.ExperimentMetaNames.TRAINING_DATA_REFERENCE: TRAINING_DATA_REFERENCE,\n            client.repository.ExperimentMetaNames.TRAINING_RESULTS_REFERENCE: TRAINING_RESULTS_REFERENCE,\n            client.repository.ExperimentMetaNames.TRAINING_REFERENCES: [\n                        {\n                            \"name\": \"MNIST_MLP\",\n                            \"training_definition_url\": definition_1_url,\n                            \"compute_configuration\": {\"name\": \"k80x2\"},\n                            \"hyper_parameters_optimization\": HPO\n                            \n                        },\n                        {\n                            \"name\": \"MNIST_CNN\",\n                            \"training_definition_url\": definition_2_url,\n                            \"compute_configuration\": {\"name\": \"k80x2\"},\n                        }],\n            }"
        }, 
        {
            "source": "#### Store experiment in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "dc222702-508e-4c27-aae5-a0af07c3a810\n"
                }
            ], 
            "source": "experiment_details = client.repository.store_experiment(meta_props=experiment_metadata)\n\nexperiment_uid = client.repository.get_experiment_uid(experiment_details)\nprint(experiment_uid)"
        }, 
        {
            "source": "#### LIST stored experiments", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ----------------  ------------------------\nGUID                                  NAME              CREATED\n0c8803bf-122f-4066-be77-39d6aef4de3d  MNIST experiment  2018-03-15T10:35:54.116Z\n207928e4-2634-48b2-80d1-acbc3eed3db2  MNIST experiment  2018-03-15T10:21:23.857Z\n298cc42b-3138-43a2-a690-bf18292e7f65  STYLE experiment  2018-03-16T10:34:03.736Z\n3b274502-a203-430f-b2b2-903f1e713d40  MNIST experiment  2018-03-16T10:00:46.078Z\n496a28ea-7887-4b3e-b4c0-28d21b831333  MNIST experiment  2018-03-15T15:41:47.219Z\n4bae6fcb-f490-45b7-bdaf-559030b6a0f9  STYLE experiment  2018-03-15T21:46:05.795Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  STYLE experiment  2018-03-16T08:37:30.628Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  STYLE experiment  2018-03-15T20:57:23.876Z\ndc222702-508e-4c27-aae5-a0af07c3a810  MNIST experiment  2018-03-16T12:35:40.804Z\nf35cea6e-4461-45d2-b1b7-d2250fc1a59e  STYLE experiment  2018-03-15T20:35:12.928Z\nff7bada4-7530-4c1d-8de1-59684727e021  STYLE experiment  2018-03-16T10:04:35.729Z\n------------------------------------  ----------------  ------------------------\n"
                }
            ], 
            "source": "client.repository.list_experiments()"
        }, 
        {
            "source": "**Hint:** You can update experiment definition by calling below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "updated_experiment_details = client.repository.update_experiment(experiment_uid, experiment_metadata)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "#### GET experiment definition details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "details = client.repository.get_experiment_details(experiment_uid)"
        }, 
        {
            "source": "#### Delete experiment definition from repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "client.repository.delete(experiment_uid)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"run\"></a>\n## 4. Run experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.1 Running experiments", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 49, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "experiment_run_details = client.experiments.run(experiment_uid, asynchronous=True)"
        }, 
        {
            "source": "As we can see experiment run has been triggered.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### LIST experiment runs\nYou can list experiment run using `client.experiments.list_runs()` method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 50, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ------------------------------------  -----------------  ---------  --------------------\nGUID (experiment)                     GUID (run)                            NAME (experiment)  STATE      CREATED\n207928e4-2634-48b2-80d1-acbc3eed3db2  1b1b2396-2f7a-46d1-801b-4e36dc1c5029  MNIST experiment   completed  2018-03-15T10:21:40Z\n298cc42b-3138-43a2-a690-bf18292e7f65  bc8655b4-5723-47d2-ab98-778f0f433b9c  STYLE experiment   completed  2018-03-16T10:34:08Z\n298cc42b-3138-43a2-a690-bf18292e7f65  dab3c0e5-caaf-4eef-9898-1547919df058  STYLE experiment   completed  2018-03-16T10:34:40Z\n3b274502-a203-430f-b2b2-903f1e713d40  d8b27753-4eee-4ad0-a356-ba92818867f3  MNIST experiment   completed  2018-03-16T10:00:54Z\n496a28ea-7887-4b3e-b4c0-28d21b831333  bfb9774b-b730-4a1d-adc0-4e6ba84185be  MNIST experiment   completed  2018-03-15T15:42:27Z\n4bae6fcb-f490-45b7-bdaf-559030b6a0f9  3dd16a25-eb54-445d-85f4-eb7291d0ba60  STYLE experiment   completed  2018-03-15T21:46:13Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  0e8690af-565f-44ac-95b7-9f1e0db4755b  STYLE experiment   completed  2018-03-16T09:32:59Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  c6b6af18-409e-46ed-9dc4-5c4159332536  STYLE experiment   completed  2018-03-16T08:37:37Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  e328d165-acc6-4d4f-abd3-edb05838cfcc  STYLE experiment   completed  2018-03-16T09:14:40Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  eb89050e-ee91-404d-8b66-add9e68f6f66  STYLE experiment   completed  2018-03-16T08:39:53Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  feae525e-e654-46d9-aad9-b01e43f78147  STYLE experiment   completed  2018-03-16T09:01:35Z\nceb102b9-a05a-4e42-8212-4ac1e0fe115e  feb1c972-c010-42fa-92d8-3fcd6598fd31  STYLE experiment   completed  2018-03-16T08:48:27Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  92a128f3-ba26-4114-a4a4-8a0f7c4c45dd  STYLE experiment   completed  2018-03-15T21:36:18Z\nd8cf0def-3a2c-4cee-83b9-79b5eedff2eb  ccc0e8b0-614c-42d8-a740-7c3de5c9b41d  STYLE experiment   completed  2018-03-15T20:57:25Z\ndc222702-508e-4c27-aae5-a0af07c3a810  a8fc6bf2-dc47-425f-97d0-0939b69afd12  MNIST experiment   pending    2018-03-16T12:39:34Z\nf35cea6e-4461-45d2-b1b7-d2250fc1a59e  b59c47b4-154f-4f43-a668-da6b5d5bbb7c  STYLE experiment   completed  2018-03-15T20:36:01Z\nff7bada4-7530-4c1d-8de1-59684727e021  2d5d811f-a4fb-4a48-8ee4-2e440e2a4146  STYLE experiment   completed  2018-03-16T10:04:41Z\n------------------------------------  ------------------------------------  -----------------  ---------  --------------------\n"
                }
            ], 
            "source": "client.experiments.list_runs()"
        }, 
        {
            "source": "#### GET experiment run UID", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 51, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "a8fc6bf2-dc47-425f-97d0-0939b69afd12\n"
                }
            ], 
            "source": "experiment_run_uid = client.experiments.get_run_uid(experiment_run_details)\nprint(experiment_run_uid)"
        }, 
        {
            "source": "#### LIST training runs triggered by experiment run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Hint:** Please run the cell below several times durign the run to see updates.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 71, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--------------------  ---------  ---------  --------------------  --------------------  --------------------\nGUID (training)       NAME       STATE      SUBMITTED             FINISHED              PERFORMANCE\ntraining-3rLt7_giR    MNIST_CNN  completed  2018-03-16T12:39:44Z  2018-03-16T12:42:44Z  test:acc=0.9864\n                                                                                        test:loss=0.0475\n                                                                                        test:val_acc=0.9891\n                                                                                        test:val_loss=0.0313\ntraining-S3ttnlgig    MNIST_MLP  completed  2018-03-16T12:39:36Z  2018-03-16T12:43:11Z\ntraining-S3ttnlgig_0  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9616\n                                                                                        test:val_loss=0.1249\ntraining-S3ttnlgig_1  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9753\n                                                                                        test:val_loss=0.0874\ntraining-S3ttnlgig_2  MNIST_MLP  completed  2018-03-16T12:39:36Z  -                     test:val_acc=0.9784\n                                                                                        test:val_loss=0.0710\n--------------------  ---------  ---------  --------------------  --------------------  --------------------\n"
                }
            ], 
            "source": "client.experiments.list_training_runs(experiment_run_uid)"
        }, 
        {
            "source": "#### GET run details\nUse below method to get particular experiment run details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 53, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_run_details = client.experiments.get_run_details(experiment_run_uid)"
        }, 
        {
            "source": "#### GET experiment run status\nYou can check experiment run status by calling `client.experiments.get_status(run_uid)` method. This is quite useful when running experiment in background.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 54, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 54, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{'current_at': '2018-03-16T12:39:34Z',\n 'current_iteration': 1,\n 'state': 'pending',\n 'submitted_at': '2018-03-16T12:39:34Z'}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.experiments.get_status(experiment_run_uid)"
        }, 
        {
            "source": "#### GET experiment details\nUse below method to get all information about particular experiment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 55, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "experiment_details = client.experiments.get_details(experiment_uid)"
        }, 
        {
            "source": "You can use below print method to display experiment details (change cell type to code).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "print(json.dumps(experiment_details, indent=2))", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "#### GET training runs uids", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 79, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "training-3rLt7_giR\ntraining-S3ttnlgig\ntraining-S3ttnlgig_0\ntraining-S3ttnlgig_1\ntraining-S3ttnlgig_2\n"
                }
            ], 
            "source": "experiment_run_details = client.experiments.get_run_details(experiment_run_uid)\ntraining_run_uids = client.experiments.get_training_uids(experiment_run_details)\n\nfor i in training_run_uids:\n    print(i)"
        }, 
        {
            "source": "#### Delete experiment-run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can delete experiment runs and results by running below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** the `delete` method can be also used to terminate experiment run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "client.experiments.delete(experiment_run_id)", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "source": "### 4.2 Monitoring experiment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### MONITOR experiment run\nYou can monitor experiment run by calling `client.experiments.monitor_logs(run_uid)`. This method will stream training logs content to console.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Tip:** You can also monitor particular training run by calling `client.training.monitor_logs(training_run_uid)`.\nTo get training_run_uid you can call method `client.experiments.list_training_runs(experiment_run_uid)`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client.experiments.monitor_logs(experiment_run_uid)"
        }, 
        {
            "source": "**Note:** Before going to next section make sure that your experiment run has completed.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 58, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 58, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "'completed'"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.experiments.get_status(experiment_run_uid)['state']"
        }, 
        {
            "source": "### 4.3 Quality metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Accessing metrics data", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can get final evaluation metrics by running below cell.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 59, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "metrics = client.experiments.get_latest_metrics(experiment_run_uid)"
        }, 
        {
            "source": "You can get all evaluation metrics by running below code.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 60, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "all_metrics = client.experiments.get_metrics(experiment_run_uid)"
        }, 
        {
            "source": "#### Visualize training results using plotly.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "!pip install cufflinks", 
            "cell_type": "raw", 
            "metadata": {}
        }, 
        {
            "execution_count": 61, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>", 
                        "text/vnd.plotly.v1+html": "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "import sys\nimport pandas\nimport plotly.plotly as py\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\nimport plotly.graph_objs as go\n\ninit_notebook_mode(connected=True)\nsys.path.append(\"\".join([os.environ[\"HOME\"]])) "
        }, 
        {
            "source": "Let's put our final metrics data into pandas dataframe.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 62, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 62, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>GUID</th>\n      <th>NAME</th>\n      <th>METRIC NAME</th>\n      <th>METRIC VALUE</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>acc</td>\n      <td>0.986400</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>loss</td>\n      <td>0.047485</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>val_acc</td>\n      <td>0.989100</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>training-3rLt7_giR</td>\n      <td>MNIST_CNN</td>\n      <td>val_loss</td>\n      <td>0.031273</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>training-S3ttnlgig_0</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.961600</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>training-S3ttnlgig_0</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.124866</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>training-S3ttnlgig_1</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.975300</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>training-S3ttnlgig_1</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.087398</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>training-S3ttnlgig_2</td>\n      <td>MNIST_MLP</td>\n      <td>val_acc</td>\n      <td>0.978400</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>training-S3ttnlgig_2</td>\n      <td>MNIST_MLP</td>\n      <td>val_loss</td>\n      <td>0.071028</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "                   GUID       NAME METRIC NAME  METRIC VALUE\n0    training-3rLt7_giR  MNIST_CNN         acc      0.986400\n1    training-3rLt7_giR  MNIST_CNN        loss      0.047485\n2    training-3rLt7_giR  MNIST_CNN     val_acc      0.989100\n3    training-3rLt7_giR  MNIST_CNN    val_loss      0.031273\n4  training-S3ttnlgig_0  MNIST_MLP     val_acc      0.961600\n5  training-S3ttnlgig_0  MNIST_MLP    val_loss      0.124866\n6  training-S3ttnlgig_1  MNIST_MLP     val_acc      0.975300\n7  training-S3ttnlgig_1  MNIST_MLP    val_loss      0.087398\n8  training-S3ttnlgig_2  MNIST_MLP     val_acc      0.978400\n9  training-S3ttnlgig_2  MNIST_MLP    val_loss      0.071028"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "import pandas as pd\n\nmetrics_df = pd.DataFrame(columns=['GUID', 'NAME', 'METRIC NAME', 'METRIC VALUE'])\n\nfor m in metrics:\n    for v in m['metrics']['values']:\n            metrics_df = metrics_df.append({'GUID': m['training_guid'], 'NAME': m['training_reference_name'], 'METRIC NAME': v['name'], 'METRIC VALUE': v['value']}, ignore_index=True)\n    \nmetrics_df"
        }, 
        {
            "source": "Visualize metrics data using grouped bar chart.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 63, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<div id=\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\", [{\"y\": [0.9864000082015991], \"type\": \"bar\", \"name\": \"acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.047485120594501495], \"type\": \"bar\", \"name\": \"loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.9890999794006348, 0.9616, 0.9753, 0.9784], \"type\": \"bar\", \"name\": \"val_acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}, {\"y\": [0.03127295523881912, 0.1248657464325428, 0.08739820090634748, 0.07102841361872853], \"type\": \"bar\", \"name\": \"val_loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}], {\"barmode\": \"group\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "text/vnd.plotly.v1+html": "<div id=\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"1dccb433-7e6a-43f8-a8e6-2b8dc0884078\", [{\"y\": [0.9864000082015991], \"type\": \"bar\", \"name\": \"acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.047485120594501495], \"type\": \"bar\", \"name\": \"loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\"]}, {\"y\": [0.9890999794006348, 0.9616, 0.9753, 0.9784], \"type\": \"bar\", \"name\": \"val_acc\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}, {\"y\": [0.03127295523881912, 0.1248657464325428, 0.08739820090634748, 0.07102841361872853], \"type\": \"bar\", \"name\": \"val_loss\", \"x\": [\"training-3rLt7_giR (MNIST_CNN)\", \"training-S3ttnlgig_0 (MNIST_MLP)\", \"training-S3ttnlgig_1 (MNIST_MLP)\", \"training-S3ttnlgig_2 (MNIST_MLP)\"]}], {\"barmode\": \"group\"}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>", 
                        "application/vnd.plotly.v1+json": {
                            "data": [
                                {
                                    "y": [
                                        0.9864000082015991
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)"
                                    ], 
                                    "type": "bar", 
                                    "name": "acc"
                                }, 
                                {
                                    "y": [
                                        0.047485120594501495
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)"
                                    ], 
                                    "type": "bar", 
                                    "name": "loss"
                                }, 
                                {
                                    "y": [
                                        0.9890999794006348, 
                                        0.9616, 
                                        0.9753, 
                                        0.9784
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)", 
                                        "training-S3ttnlgig_0 (MNIST_MLP)", 
                                        "training-S3ttnlgig_1 (MNIST_MLP)", 
                                        "training-S3ttnlgig_2 (MNIST_MLP)"
                                    ], 
                                    "type": "bar", 
                                    "name": "val_acc"
                                }, 
                                {
                                    "y": [
                                        0.03127295523881912, 
                                        0.1248657464325428, 
                                        0.08739820090634748, 
                                        0.07102841361872853
                                    ], 
                                    "x": [
                                        "training-3rLt7_giR (MNIST_CNN)", 
                                        "training-S3ttnlgig_0 (MNIST_MLP)", 
                                        "training-S3ttnlgig_1 (MNIST_MLP)", 
                                        "training-S3ttnlgig_2 (MNIST_MLP)"
                                    ], 
                                    "type": "bar", 
                                    "name": "val_loss"
                                }
                            ], 
                            "layout": {
                                "barmode": "group"
                            }
                        }
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "data = []\n\nfor i in list(pd.unique(metrics_df['METRIC NAME'])):\n    data.append(go.Bar(x=metrics_df[metrics_df['METRIC NAME'].isin([i])]['GUID'] + ' (' + metrics_df[metrics_df['METRIC NAME'].isin([i])]['NAME'] + ')', y=metrics_df[metrics_df['METRIC NAME'].isin([i])]['METRIC VALUE'], name=i))\n\n\nlayout = go.Layout(\n    barmode='group'\n)\n\nfig = go.Figure(data=data, layout=layout)\n\niplot(fig)"
        }, 
        {
            "source": "## 5. Create online deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can deployed stored model as webservice (online) by running below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 5.1 Store trained model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Call this method to save model in Watson Machine Learning repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 64, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "training-3rLt7_giR (MNIST_CNN)\n"
                }
            ], 
            "source": "status = client.experiments.get_status(experiment_run_uid)\nbest_model_uid = status['best_results']['experiment_best_model']['training_guid']\nbest_model_name = status['best_results']['experiment_best_model']['training_reference_name']\n\nprint(best_model_uid + ' (' +  best_model_name  + ')')"
        }, 
        {
            "execution_count": 81, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "saved_model_details = client.repository.store_model(best_model_uid, {'name': 'MNIST best model'})"
        }, 
        {
            "source": "**Tip:** Run below code to extract stored model guid from dictionary.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 82, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Saved model guid: f7219c8f-fccb-4d26-a1bd-f51544090eae\n"
                }
            ], 
            "source": "model_guid = client.repository.get_model_uid(saved_model_details)\nprint(\"Saved model guid: \" + model_guid)"
        }, 
        {
            "source": "#### Call this method to list stored models", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 83, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  -------------------------------  ------------------------  --------------\nGUID                                  NAME                             CREATED                   FRAMEWORK\n43986a55-0f0e-493b-9b98-0712f918b0ba  MNIST CNN model                  2018-03-15T16:12:06.503Z  tensorflow-1.5\na5a952e1-fa6f-4c64-88f7-293b367b7e1f  MNIST best model                 2018-03-16T10:12:16.773Z  tensorflow-1.5\nc755c740-dab6-4d5d-b873-1060885d55d5  MNIST best model                 2018-03-16T10:12:31.990Z  tensorflow-1.5\n861b6b73-6861-49c0-867c-d171d6d9d545  MNIST best model                 2018-03-16T13:00:40.461Z  tensorflow-1.5\nf7219c8f-fccb-4d26-a1bd-f51544090eae  training-S3ttnlgig_2 best model  2018-03-16T13:32:33.747Z  tensorflow-1.5\n------------------------------------  -------------------------------  ------------------------  --------------\n"
                }
            ], 
            "source": "client.repository.list_models()"
        }, 
        {
            "source": "### 5.2 Create online deployment", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 68, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: '1bf606e8-e8c4-4143-8727-5b2d1ac97b7e' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_IN_PROGRESS..\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='1bf606e8-e8c4-4143-8727-5b2d1ac97b7e'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "deployment_details = client.deployments.create(name=\"MNIST keras deployment\", model_uid=model_guid)"
        }, 
        {
            "source": "Extract `scoring endpoint` from deployment details.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 72, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/3f6e5c2b-4880-46aa-9d79-62e90ccc9d56/published_models/861b6b73-6861-49c0-867c-d171d6d9d545/deployments/1bf606e8-e8c4-4143-8727-5b2d1ac97b7e/online\n"
                }
            ], 
            "source": "scoring_url = client.deployments.get_scoring_url(deployment_details)\nprint(scoring_url)"
        }, 
        {
            "source": "## 6. Scoring", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "Prepare sample scoring data to score deployed model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 73, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stderr", 
                    "text": "Using TensorFlow backend.\n"
                }
            ], 
            "source": "from keras.datasets import mnist\n\n(x_train, y_train), (x_test, y_test) = mnist.load_data()"
        }, 
        {
            "source": "Let's plot two digits.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 74, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np"
        }, 
        {
            "execution_count": 75, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT4AAACFCAYAAADCQpQyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAB3NJREFUeJzt3ctLFn8Ux/GxfpGU3aQwg5JIsKLM\nhRBGYLeFRUUlXUCiRStdtGjRQtp0oZtGi2pVf4BQUYl0gYQgicggKjBwURlEFJlEC8mKfotfHL5n\nfjMP4+PMOI/n/Vqdw3eYmcX4YZ6v35kp+vPnjwcAlkwa7xMAgLQRfADMIfgAmEPwATCH4ANgDsEH\nwByCD4A5BB8Acwg+AOb8k/LxeEwkO4rG+wQmEK7r7Ih0XXPHB8Acgg+AOQQfAHMIPgDmEHwAzCH4\nAJhD8AEwh+ADYA7BB8Acgg+AOWk/sgYgQe3t7aofHh6W+uXLl2rs+vXroftpbm5WfV1dndT79+8f\nyylmAnd8AMwh+ACYU5Tyd3V5i0V28HaW+Izrdb13716pr127lsgxKisrpX7w4IEaW7RoUSLHzBNv\nZwGAIAQfAHMIPgDmMMdnF3N88Un1unbn9Dwv+rze0qVLVd/Q0CD1mzdv1FhnZ2fofk6ePKn61tbW\nSMdPCXN8ABCE4ANgDk9uAAXg2bNnUt+8eTN0uxUrVqje/ck6d+5cNVZSUiL1yMiIGlu9erXqX7x4\nIfXg4GCEM8427vgAmEPwATCH4ANgTsHP8fnfMHHlyhWpFyxYoMaKi4ulbmpqUmPz58+X2n08B8iC\njx8/Su1fgubO692/f1+NlZeXR9q//60ur1+/Dt1269atkfaZZdzxATCH4ANgTsE/ubF48WLVv3v3\nLq/9zJw5U+rly5eP5ZTysnDhQqmPHDmixmpra5M4JE9uxCfVP6KBgQHVz5gxQ+rS0tK89rlq1SrV\nv3r1KnTb7u5u1a9fvz6vYyaEJzcAIAjBB8Acgg+AOQW/nOXq1auqdx+t8c/V9fX1Sf38+XM19vDh\nQ6mfPHmixtw3zL5//z7yuU2ZMkX17iND7vIE/zHd+T7PS2yODwWqoqIilv20tbVJ3d/fn3Nb9xE2\n/+NshYg7PgDmEHwAzCn45SxxGRoaktr/M9j9qdnb2xt5n1OnTlV9VVWV1P6XQn79+lXqy5cvq7GW\nlpbIxxwFlrPEJ7PXtaurq0v1u3fvlvrHjx9qrKysTPUdHR1S19fXJ3B2sWE5CwAEIfgAmEPwATCn\n4JezxGXOnDlSb9iwIXS7jRs35n2MGzduSO3OKXqe51VXV0u9b9++vI8BhHHf4ux5/5/Xc/k/aJTx\neb1R444PgDkEHwBzWM6SoM+fP6t+5cqVoWPuC1UbGxuTPbH/sJwlPpm9rnfs2CG1/yWl7k/dAwcO\nqLGLFy+q3v0wUcaxnAUAghB8AMwh+ACYw3KWBPkfPXPn9WbPnq3G3MfZgHz53/rz+PFjqf3LV+bN\nmyf10aNH1VgBzenlhTs+AOYQfADM4aduzHp6eqQ+c+ZM6Ha3b99WvfttVCBfu3btUv2XL19Ct3W/\nLb1kyZLEzimLuOMDYA7BB8Acgg+AOczxxezOnTtSj4yMqLFNmzZJXVdXl9o5YWLr7OyU2v/2cNe6\ndetUf/z48aROKfO44wNgDsEHwByCD4A5zPGN0fDwsOrv3bsntf8ra8eOHZPa/7FxIKrBwUHVnzp1\nSmr/vLKrpqZG9RP9sbRcuOMDYA7BB8AcfuqOUVtbm+rd5QSbN29WY2vWrEnlnDCxnT9/XvVPnz4N\n3dZ9A7Pl5St+3PEBMIfgA2AOwQfAHL6yNkpdXV2q37lzp+qnT58u9d27d9VYxh5T4ytr8Un1ui4u\nLlZ9riUsHz58kLq8vDyxc8oQvrIGAEEIPgDmsJwlAnel/KFDh9TYr1+/VL9lyxapM/bTFga51+5Y\nnhaaNWtW6H5+/vwp9bdv30L3MTQ0pPoLFy5EOvbkyZNVf/bsWamnTZsWaR9+3PEBMIfgA2AOwQfA\nHOb4Avz+/Vv1DQ0NUr99+1aNVVZWqv7EiRPJnRgwStXV1bHsZ8+ePVL7l8V8+vRJ6o6OjliOl0tZ\nWZnU/g+hR8UdHwBzCD4A5vDkRoD+/n7VV1VVhW7rfujF8zxv27ZtiZxTAnhyIz6pXtf+j4bfunUr\nzcOPirv0ZdKk8Pus7du3q762tjZ027Vr10odsGSMJzcAIAjBB8Acgg+AOczx/TUwMCB1fX196Fh7\ne7saO3z4sOqLigpm6qxgTrQAjOt1fe7cOalzvanFr6+vT+rRLEM5ePCg6isqKkK3bWxslHrZsmWR\njzEGzPEBQBCCD4A5/NT9q7W1VerTp0+Hbtfb26v6XP92zzh+6sYns9e1QfzUBYAgBB8Acwg+AOaY\nfTvLo0ePVH/p0qVxOhMAaeOOD4A5BB8Ac8z+1O3p6VH99+/fQ7d1XzZaUlKS2DkBSAd3fADMIfgA\nmEPwATDH7BxfLjU1Narv7u6WurS0NO3TARAz7vgAmEPwATCHt7PYxdtZ4sN1nR28nQUAghB8AMwh\n+ACYk/ZyFuaVMBFxXRcY7vgAmEPwATCH4ANgDsEHwByCD4A5BB8Acwg+AOYQfADMIfgAmEPwATCH\n4ANgDsEHwByCD4A5BB8Acwg+AOYQfADMIfgAmEPwATCH4ANgDsEHwByCD4A5BB8Acwg+AOb8C3bx\nik91/Qo6AAAAAElFTkSuQmCC\n", 
                        "text/plain": "<matplotlib.figure.Figure at 0x7fe6c7066d30>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "for i, image in enumerate([x_test[0], x_test[1]]):\n    plt.subplot(2, 2, i + 1)\n    plt.axis('off')\n    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')"
        }, 
        {
            "source": "Our input node expects to get data with shape (28,28,1) so we need to reshape our two digits.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 76, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "image_1 = np.expand_dims(x_test[0], axis=2)\nimage_2 = np.expand_dims(x_test[1], axis=2)"
        }, 
        {
            "source": "Prepare scoring payload and score.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 77, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "scoring_data = {'values': [image_1.tolist(), image_2.tolist()]}"
        }, 
        {
            "execution_count": 78, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Scoring result: {'fields': ['prediction', 'prediction_classes', 'probability'], 'values': [[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0], 7, [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]], [[0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 2, [0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]]}\n"
                }
            ], 
            "source": "predictions = client.deployments.score(scoring_url, scoring_data)\nprint(\"Scoring result: \" + str(predictions))"
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 7. Summary and next steps     ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": " You successfully completed this notebook! You learned how to use `watson-machine-learning-client` to run experiments. Check out our [Online Documentation](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html) for more samples, tutorials, documentation, how-tos, and blog posts. ", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Authors\n\n**Lukasz Cmielowski**, PhD, is a Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 2017, 2018 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}