notebooks/AI OpenScale and Azure ML Studio Engine.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Azure Machine Learning Studio engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to monitor the model deployed on Azure Machine Learning Studio using Watson OpenScale python sdk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contents\n",
    "\n",
    "1. Setup\n",
    "2. Binding machine learning engine\n",
    "3. Subscriptions\n",
    "4. Performance monitor, scoring and payload logging\n",
    "5. Quality monitor and feedback logging\n",
    "6. Fairness, Drift monitor and explainability\n",
    "7. Payload analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample model creation using [Azure Machine Learning Studio](https://studio.azureml.net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Download training data set from [here](https://github.com/pmservice/wml-sample-models/raw/master/spark/product-line-prediction/data/GoSales_Tx.csv)\n",
    "- Copy [credit risk experiment](https://gallery.cortanaintelligence.com/Experiment/German-credit-risk-created-on-1-9-2019) from Azure ML Studio Gallery \n",
    "- Run experiment to train a model\n",
    "- Create (deploy) web service (new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Classic web services are not supported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation and authentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed ibm-ai-openscale-2.2.1\n",
      "Successfully installed py4j-0.10.7 pyspark-2.4.5\n",
      "Successfully installed lime-0.1.1.32\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade ibm-ai-openscale --no-cache | tail -n 1\n",
    "!pip install --upgrade pyspark --no-cache | tail -n 1\n",
    "!pip install --upgrade lime==0.1.1.32 --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Action:** Restart the kernel (Kernel->Restart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import and initiate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale import APIClient\n",
    "from ibm_ai_openscale.supporting_classes import PayloadRecord\n",
    "from ibm_ai_openscale.supporting_classes.enums import InputDataType, ProblemType\n",
    "from ibm_ai_openscale.engines import *\n",
    "from ibm_ai_openscale.utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ACTION: Get Watson OpenScale `apikey`\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "How to get api key using bluemix console:\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "API_KEY = 'USE IBM CLOUD API KEY'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's define some constants required to set up data mart:\n",
    "\n",
    "- AIOS_CREDENTIALS\n",
    "- POSTGRES_CREDENTIALS\n",
    "- SCHEMA_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.utils import get_instance_guid\n",
    "\n",
    "AIOS_CREDENTIALS = {\n",
    "  \"url\": \"https://api.aiopenscale.cloud.ibm.com\",\n",
    "  \"instance_guid\": get_instance_guid(api_key=API_KEY),\n",
    "  \"apikey\": API_KEY\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "POSTGRES_CREDENTIALS = {\n",
    "    \"db_type\": \"postgresql\",\n",
    "    \"uri_cli_1\": \"xxx\",\n",
    "    \"maps\": [],\n",
    "    \"instance_administration_api\": {\n",
    "        \"instance_id\": \"xxx\",\n",
    "        \"root\": \"xxx\",\n",
    "        \"deployment_id\": \"xxx\"\n",
    "    },\n",
    "    \"name\": \"xxx\",\n",
    "    \"uri_cli\": \"xxx\",\n",
    "    \"uri_direct_1\": \"xxx\",\n",
    "    \"ca_certificate_base64\": \"xxx\",\n",
    "    \"deployment_id\": \"xxx\",\n",
    "    \"uri\": \"xxx\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SCHEMA_NAME = 'azure_dm'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create schema for data mart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "create_postgres_schema(postgres_credentials=POSTGRES_CREDENTIALS, schema_name=SCHEMA_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "client = APIClient(AIOS_CREDENTIALS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.2.1'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download and preview training data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-01 18:43:42--  https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/credit-risk/data/credit_risk_training.csv\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 694222 (678K) [text/plain]\n",
      "Saving to: ‘credit_risk_training.csv’\n",
      "\n",
      "100%[======================================>] 694,222     --.-K/s   in 0.04s   \n",
      "\n",
      "2020-04-01 18:43:42 (16.3 MB/s) - ‘credit_risk_training.csv’ saved [694222/694222]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -rf credit_risk_training.csv\n",
    "!wget \"https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/credit-risk/data/credit_risk_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "training_data_pd = pd.read_csv(\n",
    "        \"credit_risk_training.csv\",\n",
    "        dtype={'LoanDuration': int, 'LoanAmount': int, 'InstallmentPercent': int, 'CurrentResidenceDuration': int, 'Age': int, 'ExistingCreditsCount': int, 'Dependents': int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CheckingStatus</th>\n",
       "      <th>LoanDuration</th>\n",
       "      <th>CreditHistory</th>\n",
       "      <th>LoanPurpose</th>\n",
       "      <th>LoanAmount</th>\n",
       "      <th>ExistingSavings</th>\n",
       "      <th>EmploymentDuration</th>\n",
       "      <th>InstallmentPercent</th>\n",
       "      <th>Sex</th>\n",
       "      <th>OthersOnLoan</th>\n",
       "      <th>...</th>\n",
       "      <th>OwnsProperty</th>\n",
       "      <th>Age</th>\n",
       "      <th>InstallmentPlans</th>\n",
       "      <th>Housing</th>\n",
       "      <th>ExistingCreditsCount</th>\n",
       "      <th>Job</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>Telephone</th>\n",
       "      <th>ForeignWorker</th>\n",
       "      <th>Risk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>31</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>other</td>\n",
       "      <td>1889</td>\n",
       "      <td>100_to_500</td>\n",
       "      <td>less_1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>less_0</td>\n",
       "      <td>18</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>car_new</td>\n",
       "      <td>462</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>37</td>\n",
       "      <td>stores</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>less_0</td>\n",
       "      <td>15</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>furniture</td>\n",
       "      <td>250</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>real_estate</td>\n",
       "      <td>28</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>28</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>retraining</td>\n",
       "      <td>3693</td>\n",
       "      <td>less_100</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>no_checking</td>\n",
       "      <td>28</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>education</td>\n",
       "      <td>6235</td>\n",
       "      <td>500_to_1000</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>unknown</td>\n",
       "      <td>57</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>Risk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  CheckingStatus  LoanDuration           CreditHistory LoanPurpose  \\\n",
       "0       0_to_200            31    credits_paid_to_date       other   \n",
       "1         less_0            18    credits_paid_to_date     car_new   \n",
       "2         less_0            15  prior_payments_delayed   furniture   \n",
       "3       0_to_200            28    credits_paid_to_date  retraining   \n",
       "4    no_checking            28  prior_payments_delayed   education   \n",
       "\n",
       "   LoanAmount ExistingSavings EmploymentDuration  InstallmentPercent     Sex  \\\n",
       "0        1889      100_to_500             less_1                   3  female   \n",
       "1         462        less_100             1_to_4                   2  female   \n",
       "2         250        less_100             1_to_4                   2    male   \n",
       "3        3693        less_100          greater_7                   3    male   \n",
       "4        6235     500_to_1000          greater_7                   3    male   \n",
       "\n",
       "  OthersOnLoan  ...       OwnsProperty Age  InstallmentPlans Housing  \\\n",
       "0         none  ...  savings_insurance  32              none     own   \n",
       "1         none  ...  savings_insurance  37            stores     own   \n",
       "2         none  ...        real_estate  28              none     own   \n",
       "3         none  ...  savings_insurance  32              none     own   \n",
       "4         none  ...            unknown  57              none     own   \n",
       "\n",
       "  ExistingCreditsCount      Job Dependents  Telephone ForeignWorker     Risk  \n",
       "0                    1  skilled          1       none           yes  No Risk  \n",
       "1                    2  skilled          1       none           yes  No Risk  \n",
       "2                    2  skilled          1        yes            no  No Risk  \n",
       "3                    1  skilled          1       none           yes  No Risk  \n",
       "4                    2  skilled          1       none           yes     Risk  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data_pd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataMart setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "client.data_mart.setup(db_credentials=POSTGRES_CREDENTIALS, schema=SCHEMA_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_mart_details = client.data_mart.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"binding\"></a>\n",
    "## 2. Bind machine learning engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bind  `Azure` machine learning studio\n",
    "\n",
    "Provide credentials using following fields:\n",
    "- `client_id`\n",
    "- `client_secret`\n",
    "- `subscription_id`\n",
    "- `tenant`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "AZURE_ENGINE_CREDENTIALS = {\n",
    "    \"client_id\": \"***\",\n",
    "    \"client_secret\": \"***\",\n",
    "    \"subscription_id\": \"***\",\n",
    "    \"tenant\": \"***\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "binding_uid = client.data_mart.bindings.add('My Azure ML Studio engine', AzureMachineLearningStudioInstance(AZURE_ENGINE_CREDENTIALS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bindings_details = client.data_mart.bindings.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Service bindings</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>service_type</th><th style='border: 1px solid #dddddd'>created</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>My Azure ML Studio engine</td><td style='border: 1px solid #dddddd'>azure_machine_learning</td><td style='border: 1px solid #dddddd'>2020-04-01T18:44:08.993Z</td></tr><tr><td style='border: 1px solid #dddddd'>87d766e0-8b4d-42fd-85c1-c313f189d411</td><td style='border: 1px solid #dddddd'>WML Instance</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2020-03-06T21:02:52.670Z</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "client.data_mart.bindings.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"subsciption\"></a>\n",
    "## 3. Subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List available deployments.\n",
    "\n",
    "**Note:** Depending on number of assets it may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Available assets (binding_uid='6ac4d3c4-862f-4702-8053-9f531466686a')</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>source_uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>created</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>frameworks</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>is_subscribed</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2e0bf2a9b0567d419d37b3f992d40749</td><td style='border: 1px solid #dddddd'>balancescale.multiclass</td><td style='border: 1px solid #dddddd'>2019-06-06T02:17:19.3204159Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>5acb22f1b2d22b6ecb44d5bac5d6f44f</td><td style='border: 1px solid #dddddd'>bankmarket.binary</td><td style='border: 1px solid #dddddd'>2019-05-28T14:50:01.6576057Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>6a3f4095de199d0e35a8ede7d72a491f</td><td style='border: 1px solid #dddddd'>loanlogisticregression</td><td style='border: 1px solid #dddddd'>2019-05-24T12:12:46.4120458Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>3f3bb16a7bc9971fecbe9045bea764fd</td><td style='border: 1px solid #dddddd'>lendingclub.loan.multiclass</td><td style='border: 1px solid #dddddd'>2019-05-15T14:16:40.4861116Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>8b3fa6de5b35bb77fdc640877f5ee16b</td><td style='border: 1px solid #dddddd'>forestcover.multiclass</td><td style='border: 1px solid #dddddd'>2019-05-15T09:52:01.6785081Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>417532dc6dae852809393ae097703de5</td><td style='border: 1px solid #dddddd'>adultcensusincome.logistic.regression</td><td style='border: 1px solid #dddddd'>2019-05-08T08:09:09.3483819Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>690e5e9d2c03859de56a0c7570097ec7</td><td style='border: 1px solid #dddddd'>incomeprediction.census</td><td style='border: 1px solid #dddddd'>2019-05-07T17:12:53.5401795Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>31816b6dfa672e0d18dd0485cb82c4be</td><td style='border: 1px solid #dddddd'>bikerental.linear.regression</td><td style='border: 1px solid #dddddd'>2019-05-07T16:25:47.8119975Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>20a23d9de1ab5762b880d7bc4ce09ff2</td><td style='border: 1px solid #dddddd'>videogameverdict.binary</td><td style='border: 1px solid #dddddd'>2019-05-06T17:05:16.3846249Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>78108c0d5e657065f65c7217b90351bd</td><td style='border: 1px solid #dddddd'>letterrecognition.multiclass</td><td style='border: 1px solid #dddddd'>2019-05-06T16:58:04.7056361Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>5202b312258ed793823d67c6ece36987</td><td style='border: 1px solid #dddddd'>bikerental.regression</td><td style='border: 1px solid #dddddd'>2019-05-06T16:51:07.9481437Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>76611220d2a21271860e391d01269235</td><td style='border: 1px solid #dddddd'>walkingactivity.multiclass</td><td style='border: 1px solid #dddddd'>2019-05-06T15:40:44.3115174Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>c08abd85de8de5a1cf691a3535d28a2e</td><td style='border: 1px solid #dddddd'>adult.census.binary</td><td style='border: 1px solid #dddddd'>2019-05-06T11:33:11.2195519Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>c60ed0933ee963672f74778a87289b14</td><td style='border: 1px solid #dddddd'>IncomePrediction.2019.3.15.10.54.53.233</td><td style='border: 1px solid #dddddd'>2019-03-15T10:56:01.3464451Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>517d90acd374f5a5778fd5e83bdf8a40</td><td style='border: 1px solid #dddddd'>Sklearn-linear-regression-1</td><td style='border: 1px solid #dddddd'>2019-02-22T16:41:12.3994798Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>84a9175980d4a0b306c9c79df5086db4</td><td style='border: 1px solid #dddddd'>Sklearn-Iris-1</td><td style='border: 1px solid #dddddd'>2019-02-21T23:23:44.8789721Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>20166ed5d8979ec15a6e03aa2db78e48</td><td style='border: 1px solid #dddddd'>ProductlinetestFeb15</td><td style='border: 1px solid #dddddd'>2019-02-15T07:21:16.5368409Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>7d56bf771bfcd6d9b7d9a58bf3fdf24d</td><td style='border: 1px solid #dddddd'>ExecutePythonScr.2019.2.13.8.6.20.493</td><td style='border: 1px solid #dddddd'>2019-02-13T08:06:27.6046787Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>3d9d326a906661b64ab9e3f6b232189b</td><td style='border: 1px solid #dddddd'>hung-TensorflowMNISTE.2019.2.13.7.18.28.543</td><td style='border: 1px solid #dddddd'>2019-02-13T07:18:46.3150458Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>6de5e798b8b64cc5d09e91734a149197</td><td style='border: 1px solid #dddddd'>DaveGermanCredit.2019.2.12.17.16.52.617</td><td style='border: 1px solid #dddddd'>2019-02-12T17:18:04.5957903Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>fe4eb4722074a5922ef474d62e3899e8</td><td style='border: 1px solid #dddddd'>Hung-IncomeExperiment</td><td style='border: 1px solid #dddddd'>2019-02-08T17:24:47.2672878Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>eb4c24fd0e14d53007a50aef50dbb501</td><td style='border: 1px solid #dddddd'>IncomeExperiment.2019.2.6.23.10.17.731</td><td style='border: 1px solid #dddddd'>2019-02-06T23:11:50.528911Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>4fff12e2b55fe1038b2e04d98be759c5</td><td style='border: 1px solid #dddddd'>FastpathGermanCreditRisk.2019.2.1.17.17.21.412</td><td style='border: 1px solid #dddddd'>2019-02-01T17:17:55.575148Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>986fd3e779b52d0e23a2bde5b6da996c</td><td style='border: 1px solid #dddddd'>ScottdaAzureML12.2019.1.29.21.29.50.528</td><td style='border: 1px solid #dddddd'>2019-01-29T21:30:38.2982264Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>9b6abb9a934cecf33a465f07d73aabe6</td><td style='border: 1px solid #dddddd'>MultiClass-ProductLineTestAutomationNC</td><td style='border: 1px solid #dddddd'>2019-01-21T06:50:06.8313304Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>33ee79fb8e64881e9182dd4fd3475586</td><td style='border: 1px solid #dddddd'>ClaimInsuranceRegressionTestAutomationCat</td><td style='border: 1px solid #dddddd'>2019-01-21T06:46:49.6811946Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>97521fdfaf0830239fadfbac0d5d0091</td><td style='border: 1px solid #dddddd'>ClaimInsuranceRegressionTestAutomationNC</td><td style='border: 1px solid #dddddd'>2019-01-21T06:41:04.8380274Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>a7474537b6166d3cc01481b647423769</td><td style='border: 1px solid #dddddd'>Germancreditrisk.2019.1.17.14.9.9.994</td><td style='border: 1px solid #dddddd'>2019-01-17T14:09:46.5811815Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>085460ef94636166aea5800e9ea26168</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>2019-01-09T10:42:59.7933412Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>77919091f862ab6a128edc60ccafc064</td><td style='border: 1px solid #dddddd'>ASHDrugModelPred.2019.1.8.9.48.15.590</td><td style='border: 1px solid #dddddd'>2019-01-08T09:48:33.9184573Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>920beeba5513b8059799900ecfcbdeca</td><td style='border: 1px solid #dddddd'>MultiClass-ProductLineTestAutomation</td><td style='border: 1px solid #dddddd'>2019-01-05T05:55:41.6061263Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>56f60f0e2f997ed58d851e5a063eb554</td><td style='border: 1px solid #dddddd'>ClaimInsuranceRegressionTestAutomation</td><td style='border: 1px solid #dddddd'>2019-01-04T10:43:02.5032489Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>e5a8d57a6002c42fac5524948683295d</td><td style='border: 1px solid #dddddd'>InsuranceClaimBinaryTestAutomation</td><td style='border: 1px solid #dddddd'>2019-01-04T05:52:25.0479481Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>dc0a9a92cfb377234b6266dd223ae4d0</td><td style='border: 1px solid #dddddd'>frauddetectionpr.2019.1.2.7.59.28.838</td><td style='border: 1px solid #dddddd'>2019-01-02T08:00:03.8203084Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>bc108cd66382ae734b60ae853c5242a1</td><td style='border: 1px solid #dddddd'>DPJC-Titanic-Retraining.2018.12.21.9.21.43.439</td><td style='border: 1px solid #dddddd'>2018-12-21T09:22:14.1837647Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>12df12e7f50e1114978bcd54471381d0</td><td style='border: 1px solid #dddddd'>DPJC-Titanic.2018.12.19.10.23.28.594</td><td style='border: 1px solid #dddddd'>2018-12-19T10:24:03.9383824Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>08ba1404374d7635f8eb8638c105f7eb</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-ProductLine-LinearRegression(Age)-20181213</td><td style='border: 1px solid #dddddd'>2018-12-13T09:23:36.1493809Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>efa7c594c5a0deac76bc6866ffc18dc7</td><td style='border: 1px solid #dddddd'>ClaimInsuranceRegression.2018.12.12.5.12.15.584</td><td style='border: 1px solid #dddddd'>2018-12-12T05:13:51.021202Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>2f1c3b2552bb9d96a134ef671c716059</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-ProductLine-LinearRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T03:01:58.8467204Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>48c6ebc2fa6cfd359923e3fe59bdefcc</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-ProductLine-TwoClassLogisticRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T02:59:33.7864639Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>f49e4115dbc90378d3c5cb4bbc27d2d8</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-ProductLine-MulticlassLogisticRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T02:58:25.3151546Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>68efeda5fa73738bd028e1a290807367</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-DrugSelection-TwoClassLogisticRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T02:49:34.4638846Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>5d14ed9ef7fa12fc86a96254e4d5061f</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-DrugSelection-LinearRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T02:49:04.2670654Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>920b3ea9eb29a41e4fa93bfa5c96ac63</td><td style='border: 1px solid #dddddd'>AzureNew-Yuki-DrugSelection-MulticlassLogisticRegression-20181212</td><td style='border: 1px solid #dddddd'>2018-12-12T02:47:43.887378Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>3d07cae29805365453230e8beb638088</td><td style='border: 1px solid #dddddd'>MultiClassProductline</td><td style='border: 1px solid #dddddd'>2018-12-11T19:21:01.9441141Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>f39da17aaccec60ee9bd67207ae272f6</td><td style='border: 1px solid #dddddd'>neelimaregressio.2018.12.11.18.31.12.560</td><td style='border: 1px solid #dddddd'>2018-12-11T18:31:22.7972314Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>f7ba1231cc1a7cd200b996d8e514f758</td><td style='border: 1px solid #dddddd'>InsuranceClaimBinaryClass2</td><td style='border: 1px solid #dddddd'>2018-12-11T18:16:33.0425079Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>fbda2dcb71e4f339696c778e81d6a922</td><td style='border: 1px solid #dddddd'>neelimaincomepre.2018.12.11.17.52.18.494</td><td style='border: 1px solid #dddddd'>2018-12-11T17:52:31.6527702Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>a5951aaca215e4b2208a87495594ab2d</td><td style='border: 1px solid #dddddd'>CI-Regression-Dec11</td><td style='border: 1px solid #dddddd'>2018-12-11T16:39:39.0675549Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>90b4be34a6c987d29066076233d46b1a</td><td style='border: 1px solid #dddddd'>neelimaproductli.2018.12.11.16.20.34.185</td><td style='border: 1px solid #dddddd'>2018-12-11T16:20:51.0606809Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>d8dd110f1b75f27b7854928b1d705932</td><td style='border: 1px solid #dddddd'>JimExperimentDoc01</td><td style='border: 1px solid #dddddd'>2018-12-11T14:44:22.4454173Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>4fe40f7168e087c27998ffd5fec2ed8c</td><td style='border: 1px solid #dddddd'>productlineregre.2018.12.10.6.56.13.808</td><td style='border: 1px solid #dddddd'>2018-12-10T06:56:24.9336379Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>bd7f7335d9283525ce4d5d4d10d4f481</td><td style='border: 1px solid #dddddd'>mnistimage.2018.12.7.5.46.5.751</td><td style='border: 1px solid #dddddd'>2018-12-07T05:46:25.8962975Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>454d8ded446f62842197e5b0ec1cfcaf</td><td style='border: 1px solid #dddddd'>productlineregre.2018.12.5.6.21.44.165</td><td style='border: 1px solid #dddddd'>2018-12-05T06:22:10.0824452Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>7b6f7b05a7aa19c7669546864be4636c</td><td style='border: 1px solid #dddddd'>Irisclassificati.me2018.12.4.10.40.56.847</td><td style='border: 1px solid #dddddd'>2018-12-04T10:41:13.1176858Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>2cfbf15dca01a453faa4833009dabcf6</td><td style='border: 1px solid #dddddd'>ProductLine-Yuki-20181204-1537</td><td style='border: 1px solid #dddddd'>2018-12-04T06:37:14.1095386Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>0a9d0563eeb1ab13abd01eae14aacfcd</td><td style='border: 1px solid #dddddd'>DrugSelection-Yuki-20181204-1520</td><td style='border: 1px solid #dddddd'>2018-12-04T06:20:52.6833542Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>e88428224af3cb9f4711a39aba102a1d</td><td style='border: 1px solid #dddddd'>InsuranceClaimBinaryClass1</td><td style='border: 1px solid #dddddd'>2018-11-29T13:44:14.5020986Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>f8a3f2462f0f23f967bf48c2f4d51471</td><td style='border: 1px solid #dddddd'>ProductLineMulticlassification</td><td style='border: 1px solid #dddddd'>2018-11-27T12:31:40.6453108Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>91f11e0d31fa00be92c1967bbf0bbf45</td><td style='border: 1px solid #dddddd'>InsuranceClaimBClassification</td><td style='border: 1px solid #dddddd'>2018-11-26T17:00:45.734513Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>a996478db6f24b67d722ecbe637d2ca7</td><td style='border: 1px solid #dddddd'>InsuranceClaimRegressionModel</td><td style='border: 1px solid #dddddd'>2018-11-21T08:35:02.627712Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>72519e05bd423560d4a919153e6dc029</td><td style='border: 1px solid #dddddd'>ProductLineClass.2018.11.2.11.40.22.845</td><td style='border: 1px solid #dddddd'>2018-11-02T11:40:59.563349Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>0b64ea667b5e40a339f948c10d0adeff</td><td style='border: 1px solid #dddddd'>Irisclassificati.2018.10.22.7.39.3.217</td><td style='border: 1px solid #dddddd'>2018-10-22T07:39:14.9178443Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>f0cad1153dac876126ad53158b0d3cd3</td><td style='border: 1px solid #dddddd'>ProductLineAgeRe.2018.10.22.7.33.27.127</td><td style='border: 1px solid #dddddd'>2018-10-22T07:33:43.7417975Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>d18d5fb45bfcd9253f52fb76657c27ea</td><td style='border: 1px solid #dddddd'>Irisclassificati.2018.10.19.9.49.7.418</td><td style='border: 1px solid #dddddd'>2018-10-19T09:49:27.4251325Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>False</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "client.data_mart.bindings.list_assets(binding_uid=binding_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Action:** Assign your credit risk source_uid to `source_uid` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "source_uid = '085460ef94636166aea5800e9ea26168'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "feature_columns = ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker']\n",
    "categorical_columns = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription = client.data_mart.subscriptions.add(\n",
    "    AzureMachineLearningStudioAsset(source_uid=source_uid,\n",
    "                                      binding_uid=binding_uid,\n",
    "                                      input_data_type=InputDataType.STRUCTURED,\n",
    "                                      problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "                                      label_column='Risk',\n",
    "                                      prediction_column='Scored Labels',\n",
    "                                      probability_column='Scored Probabilities',\n",
    "                                      feature_columns=feature_columns,\n",
    "                                      categorical_columns=categorical_columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get subscriptions list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscriptions = client.data_mart.subscriptions.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f']\n"
     ]
    }
   ],
   "source": [
    "subscriptions_uids = client.data_mart.subscriptions.get_uids()\n",
    "print(subscriptions_uids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>created</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>2020-04-01T18:45:23.693Z</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "client.data_mart.subscriptions.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"scoring\"></a>\n",
    "## 4. Scoring and payload logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the credit risk model and measure response time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Results\": {\n",
      "    \"output1\": [\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"31\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"other\",\n",
      "        \"LoanAmount\": \"1889\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"32\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0140035906806588\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"13\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"1389\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"25\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.439973294734955\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import json\n",
    "\n",
    "subscription_details = subscription.get_details()\n",
    "scoring_url = subscription_details['entity']['deployments'][0]['scoring_endpoint']['url']\n",
    "\n",
    "data = {\n",
    "            \"Inputs\": {\n",
    "                \"input1\":\n",
    "                    [\n",
    "                        {\n",
    "                            'CheckingStatus': \"0_to_200\", 'LoanDuration': 31, 'CreditHistory': \"credits_paid_to_date\", 'LoanPurpose': \"other\",\n",
    "                            'LoanAmount': 1889, 'ExistingSavings': \"100_to_500\",'EmploymentDuration': \"less_1\",'InstallmentPercent': 3,'Sex': \"female\",\n",
    "                            'OthersOnLoan': \"none\",'CurrentResidenceDuration': 3, 'OwnsProperty': \"savings_insurance\", 'Age': 32,'InstallmentPlans': \"none\",\n",
    "                            'Housing': \"own\",'ExistingCreditsCount': 1,'Job': \"skilled\",'Dependents': 1,'Telephone': \"none\",'ForeignWorker': \"yes\",\n",
    "                        },\n",
    "                        {\n",
    "                            'CheckingStatus': \"no_checking\", 'LoanDuration': 13, 'CreditHistory': \"credits_paid_to_date\", 'LoanPurpose': \"car_new\",\n",
    "                            'LoanAmount': 1389, 'ExistingSavings': \"100_to_500\",'EmploymentDuration': \"1_to_4\",'InstallmentPercent': 2,'Sex': \"male\",\n",
    "                            'OthersOnLoan': \"none\",'CurrentResidenceDuration': 3, 'OwnsProperty': \"savings_insurance\", 'Age': 25,'InstallmentPlans': \"none\",\n",
    "                            'Housing': \"own\",'ExistingCreditsCount': 2,'Job': \"skilled\",'Dependents': 1,'Telephone': \"none\",'ForeignWorker': \"yes\",\n",
    "                        }\n",
    "                    ],\n",
    "            },\n",
    "            \"GlobalParameters\": {\n",
    "            }\n",
    "        }\n",
    "\n",
    "body = str.encode(json.dumps(data))\n",
    "\n",
    "token = subscription_details['entity']['deployments'][0]['scoring_endpoint']['credentials']['token']\n",
    "headers = subscription_details['entity']['deployments'][0]['scoring_endpoint']['request_headers']\n",
    "headers['Authorization'] = ('Bearer ' + token)\n",
    "\n",
    "start_time = time.time()\n",
    "response = requests.post(url=scoring_url, data=body, headers=headers)\n",
    "response_time = int(time.time() - start_time)*1000\n",
    "result = response.json()\n",
    "\n",
    "print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the request and response in payload logging table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "records_list = []\n",
    "\n",
    "for i in range(1, 10):\n",
    "    records_list.append(PayloadRecord(request=data, response=result, response_time=response_time))\n",
    "\n",
    "subscription.payload_logging.store(records=records_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       LoanDuration   LoanAmount  InstallmentPercent  \\\n",
      "count     18.000000    18.000000           18.000000   \n",
      "mean      22.000000  1639.000000            2.500000   \n",
      "std        9.260924   257.247878            0.514496   \n",
      "min       13.000000  1389.000000            2.000000   \n",
      "25%       13.000000  1389.000000            2.000000   \n",
      "50%       22.000000  1639.000000            2.500000   \n",
      "75%       31.000000  1889.000000            3.000000   \n",
      "max       31.000000  1889.000000            3.000000   \n",
      "\n",
      "       CurrentResidenceDuration       Age  ExistingCreditsCount  Dependents  \\\n",
      "count                      18.0  18.00000             18.000000        18.0   \n",
      "mean                        3.0  28.50000              1.500000         1.0   \n",
      "std                         0.0   3.60147              0.514496         0.0   \n",
      "min                         3.0  25.00000              1.000000         1.0   \n",
      "25%                         3.0  25.00000              1.000000         1.0   \n",
      "50%                         3.0  28.50000              1.500000         1.0   \n",
      "75%                         3.0  32.00000              2.000000         1.0   \n",
      "max                         3.0  32.00000              2.000000         1.0   \n",
      "\n",
      "       Scored Probabilities  prediction_probability  \n",
      "count             18.000000               18.000000  \n",
      "mean               0.226988                0.773012  \n",
      "std                0.219160                0.219160  \n",
      "min                0.014004                0.560027  \n",
      "25%                0.014004                0.560027  \n",
      "50%                0.226988                0.773012  \n",
      "75%                0.439973                0.985996  \n",
      "max                0.439973                0.985996  \n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "subscription.payload_logging.describe_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check transactions performance data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>PerformanceMetrics (binding_id=6ac4d3c4-862f-4702-8053-9f531466686a, subscription_id=7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>scoring_time</th><th style='border: 1px solid #dddddd'>scoring_records</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>process</th><th style='border: 1px solid #dddddd'>asset_revision</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:45:38.121000+00:00</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.performance_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** It may take few seconds before performance data is populated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"feedback\"></a>\n",
    "## 5. Feedback logging & quality (accuracy) monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enable quality monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to provide the monitoring `threshold` and `min_records` (minimal number of feedback records)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.quality_monitoring.enable(threshold=0.7, min_records=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedback records logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feedback records are used to evaluate your model. The predicted values are compared to real values (feedback records)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the schema of feedback table using below method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Schema of Feedback_7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>nullable</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>CheckingStatus</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanDuration</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>CreditHistory</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanPurpose</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanAmount</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ExistingSavings</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>EmploymentDuration</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>InstallmentPercent</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Sex</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>OthersOnLoan</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>CurrentResidenceDuration</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>OwnsProperty</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Age</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>InstallmentPlans</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Housing</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ExistingCreditsCount</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Job</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Dependents</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Telephone</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ForeignWorker</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>record_timestamp</td><td style='border: 1px solid #dddddd'>timestamp</td><td style='border: 1px solid #dddddd'>False</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.feedback_logging.print_table_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feedback records can be send to feedback table using below code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "records = [\n",
    "    [\"no_checking\",\"28\",\"outstanding_credit\",\"appliances\",\"5990\",\"500_to_1000\",\"greater_7\",\"5\",\"male\",\"co-applicant\",\"3\",\"car_other\",\"55\",\"none\",\"free\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "    [\"greater_200\",\"22\",\"all_credits_paid_back\",\"car_used\",\"3376\",\"less_100\",\"less_1\",\"3\",\"female\",\"none\",\"2\",\"car_other\",\"32\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "    [\"no_checking\",\"39\",\"credits_paid_to_date\",\"vacation\",\"6434\",\"unknown\",\"greater_7\",\"5\",\"male\",\"none\",\"4\",\"car_other\",\"39\",\"none\",\"own\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "    [\"0_to_200\",\"20\",\"credits_paid_to_date\",\"furniture\",\"2442\",\"less_100\",\"unemployed\",\"3\",\"female\",\"none\",\"1\",\"real_estate\",\"42\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "    [\"greater_200\",\"4\",\"all_credits_paid_back\",\"education\",\"4206\",\"less_100\",\"unemployed\",\"1\",\"female\",\"none\",\"3\",\"savings_insurance\",\"27\",\"none\",\"own\",\"1\",\"management_self-employed\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "    [\"greater_200\",\"23\",\"credits_paid_to_date\",\"car_used\",\"2963\",\"greater_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"46\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"none\",\"yes\",\"Risk\"],\n",
    "    [\"no_checking\",\"31\",\"prior_payments_delayed\",\"vacation\",\"2673\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"2\",\"real_estate\",\"35\",\"stores\",\"rent\",\"1\",\"skilled\",\"2\",\"none\",\"yes\",\"Risk\"],\n",
    "    [\"no_checking\",\"37\",\"prior_payments_delayed\",\"other\",\"6971\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"3\",\"savings_insurance\",\"54\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "    [\"0_to_200\",\"39\",\"prior_payments_delayed\",\"appliances\",\"5685\",\"100_to_500\",\"1_to_4\",\"4\",\"female\",\"none\",\"2\",\"unknown\",\"37\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "    [\"no_checking\",\"38\",\"prior_payments_delayed\",\"appliances\",\"4990\",\"500_to_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"50\",\"bank\",\"own\",\"2\",\"unemployed\",\"2\",\"yes\",\"yes\",\"Risk\"]]\n",
    "\n",
    "fields = feature_columns.copy()\n",
    "fields.append('Risk')\n",
    "\n",
    "subscription.feedback_logging.store(feedback_data=records, fields=fields)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run quality monitoring on demand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, quality monitoring is run on hourly schedule. You can also trigger it on demand using below code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "================================================================================\n",
      "\n",
      " Waiting for end of quality monitoring run 61a40239-55eb-4ffe-86fe-3ca542be14cd \n",
      "\n",
      "================================================================================\n",
      "\n",
      "\n",
      "\n",
      "completed\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "run_details = subscription.quality_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show the quality metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>QualityMetrics (binding_id=6ac4d3c4-862f-4702-8053-9f531466686a, subscription_id=7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower limit</th><th style='border: 1px solid #dddddd'>upper limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.8571428571428571</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.9285714285714286</td><td style='border: 1px solid #dddddd'>0.7</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.923076923076923</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.9</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.10192821288228596</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>area_under_pr</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.9785714285714285</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:46:29.070000+00:00</td><td style='border: 1px solid #dddddd'>recall</td><td style='border: 1px solid #dddddd'>999201ac-c64b-4316-bdfa-d1d203fdad48</td><td style='border: 1px solid #dddddd'>0.8571428571428571</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.quality_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all calculated metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deployment_uids = subscription.get_deployment_uids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  {\n",
      "    \"asset_id\": \"085460ef94636166aea5800e9ea26168\",\n",
      "    \"binding_id\": \"6ac4d3c4-862f-4702-8053-9f531466686a\",\n",
      "    \"tags\": [\n",
      "      {\n",
      "        \"id\": \"model_type\",\n",
      "        \"value\": \"original\"\n",
      "      }\n",
      "    ],\n",
      "    \"ts\": \"2020-04-01T18:46:29.070Z\",\n",
      "    \"measurement_id\": \"999201ac-c64b-4316-bdfa-d1d203fdad48\",\n",
      "    \"monitor_definition_id\": \"quality\",\n",
      "    \"subscription_id\": \"7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f\",\n",
      "    \"metrics\": [\n",
      "      {\n",
      "        \"id\": \"true_positive_rate\",\n",
      "        \"value\": 0.8571428571428571\n",
      "      },\n",
      "      {\n",
      "        \"lower_limit\": 0.7,\n",
      "        \"id\": \"area_under_roc\",\n",
      "        \"value\": 0.9285714285714286\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"precision\",\n",
      "        \"value\": 1.0\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"f1_measure\",\n",
      "        \"value\": 0.923076923076923\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"accuracy\",\n",
      "        \"value\": 0.9\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"log_loss\",\n",
      "        \"value\": 0.10192821288228596\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"false_positive_rate\",\n",
      "        \"value\": 0.0\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"area_under_pr\",\n",
      "        \"value\": 0.9785714285714285\n",
      "      },\n",
      "      {\n",
      "        \"id\": \"recall\",\n",
      "        \"value\": 0.8571428571428571\n",
      "      }\n",
      "    ],\n",
      "    \"deployment_id\": \"563f01d37f720857b95c557dc76176ad\"\n",
      "  }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "metrics = subscription.quality_monitoring.get_metrics(deployment_uid=deployment_uids[0])\n",
    "print(json.dumps(metrics, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get metrics as pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAD8CAYAAAAsc076AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X2c1lWd//HXm3GEEIVU3CZRRw3vEAUZb/AW8ya3zJvCxDRB3fhlKavlbvRz62dZu1q/X7lZrqEpVlYEarrQhq6CqKkwI/coqz/BFTTFm1BUCPCzf3zP6OUwzFwz37mua27ez8djHvP9nu/5nvM5F6OfOed857oUEZiZmVn79ap0AGZmZl2dk6mZmVlOTqZmZmY5OZmamZnl5GRqZmaWk5OpmZlZTk6mZmZmOTmZmpmZ5eRkamZmltM2lQ7AymPnnXeO2traSodhZtZlNDQ0vBIRA4up62TaQ9TW1lJfX1/pMMzMugxJzxVb18u8ZmZmOTmZmpmZ5eRkamZmlpP3TM3MDICNGzeyatUq1q9fX+lQyqpPnz4MGjSI6urqdrfhZNpDLF69ltqJMyodhpl1YlceuxOH7P1Rtun3ESR94NpBgwZUKKrSigheffVVVq1axZ577tnudrzMa2ZmAOwxoJpt+m6/RSLtziSx00475Z6NO5mamRkAQj0qkTbqiDE7mXZBkmolLUnHoyRNr3RMZmY9mfdMy0jZrz+KiHcrHYuZWWtO+8kjHdreyms+1aHt9evXj3Xr1nVom+3lmWmJpVnkk5JuAJ4AviDpUUlPSJoqqV+qd6ikP0laKGmupO3TvQ+luk9IOrKyozEzs+Y4mZbHvsAvgJOAi4ATI+IQoB74qqRtgSnA30fEwcCJwDvAy8BJqe7ZwI/b0qmk8ZLqJdVvfnttx43GzKwEvv71r3PDDTe8d37VVVfx7W9/mxNOOIFDDjmEoUOHcvfdd29x3+zZszn11FPfO7/kkkuYPHkyAA0NDRx33HGMGDGCT3ziE7z44oslid3JtDyei4jHgCOAA4BHJC0AxgJ7kCXbFyNiHkBEvBERm4Bq4CZJi4Gp6d6iRcSkiKiLiLqqvv07cDhmZh1vzJgxTJky5b3z3/3ud1xwwQXcddddPPHEE8yaNYuvfe1rRERR7W3cuJFLL72UadOm0dDQwIUXXsiVV15Zkti9Z1oeb6XvAu6LiHMKL0o6CGjup+Ny4CXgYLJffHrWX1KbWY8yfPhwXn75ZV544QXWrFnDhz/8YWpqarj88suZM2cOvXr1YvXq1bz00kt85CMfabW95cuXs2TJEk466SQANm/eTE1NTUlidzItr8eAn0r6WEQ8I6kvMAh4CviopEMjYp6k7cmWefsDqyLiXUljgarKhW5mVnqjR49m2rRp/PnPf2bMmDHcfvvtrFmzhoaGBqqrq6mtrd3ib0K32WYb3n33/ec6G69HBEOGDOHRRx8tedxe5i2jiFgDjAN+I2kRWXLdLyL+SrYner2khcB9QB/gBmCspMeAfXh/hmtm1i2NGTOG3/72t0ybNo3Ro0ezdu1adtllF6qrq5k1axbPPbflp6LtscceLFu2jA0bNrB27Vruv/9+APbdd1/WrFnzXjLduHEjS5cuLUncnpmWWESsBA4sOH8AOLSZevPI9lQLPQ0cVHD+jaZtRsRsYHbHRWxmlrnnkqPeOy7X2wkOGTKEN998k1133ZWamhrOPfdcPv3pT1NXV8ewYcPYb7/9trhnt91243Of+xwHHXQQgwcPZvjw4QBsu+22TJs2jQkTJrB27Vo2bdrEZZddxpAhQzo8bhW7kWtdW11dXfjDwc2sJU8++ST7779/pcOoiObGLqkhIuqKud/LvGZmZjk5mZqZmeXkZGpmZu/piVt/HTFmJ1MzMwOyD8l+9dVXe1RCbfw80z59+uRqx0/zmpkZAIMGDWLVqlWsWbOm0qGUVZ8+fRg0aFCuNpxMzcwMgOrqavbcc89Kh9EleZnXzMwsJydTMzOznJxMzczMcnIyNTMzy8nJ1MzMLCcnUzMzs5z8pzE9xOLVa6mdOKPSYZiZdbiV13yq0iF4ZmpmZpaXk6mZmVlOTqYlImmUpOmVjsPMzEqv2yVTSVWVjqE9JOXav+6q4zYz6w66XDKV9HtJDZKWShqfytZJ+o6kx4GRkkZIejDVmympJtX7oqR5khZKukNS3xb6mSxpdMH5uvR9lKTZkqZJekrS7ZKUrp2Syh4GPlNw73aSbkl9z5d0eiofJ2mqpH8H7t1KHKMkzZF0l6Rlkm6U1Ku5ced7Zc3MrL26XDIFLoyIEUAdMEHSTsB2wJKIOBx4HLgeGJ3q3QJ8L917Z0QcGhEHA08CF7UzhuHAZcABwF7AUZL6ADcBnwaOAT5SUP9K4IGIOBQ4HviBpO3StZHA2Ij4eAv9HQZ8DRgK7M37ifq9cUfEw01vkjReUr2k+s1vr23nUM3MrDVd8U9jJkg6Mx3vBgwGNgN3pLJ9gQOB+9KEsQp4MV07UNJ3gQFAP2BmO2OYGxGrACQtAGqBdcCKiHg6lf8KGJ/qnwycJumKdN4H2D0d3xcRrxXR37Op3d8ARwPT+OC4txARk4BJAL1rBvecDyg0MyuzLpVMJY0CTgRGRsTbkmaTJab1EbG5sRqwNCKaW/acDJwREQsljQNGtdDdJtLMPS3jbltwbUPB8Wbefx23lrAEfDYiljcZz+HAWy3E0Khpu43nheM2M7MK6WrLvP2B11Mi3Q84opk6y4GBkkYCSKqWNCRd2x54UVI1cG4rfa0ERqTj04HqVuo/Bewpae90fk7BtZnApQV7q8NbaaupwyTtmfZKzwa2WNI1M7PK6WrJ9I/ANpIWAVcDjzWtEBF/BUYD10paCCwAjkyXv0m2p3ofWfJryU3AcZLmAq3OICNiPdmy7oz0ANJzBZevJkvGiyQtSedt8ShwDbAEWAHc1cb7zcyshBThrbTOLC1tXxERp+Zpp3fN4KgZe13HBGVm1omU6u0EJTVERF0xdbvUnqm139Bd+1PfCd6/0sysO+rxyVTSlcBZTYqnRsT3mqtfwjiGAr9sUrwh/bnP7HLGYmZmbdPjk2lKmmVNnFuJYzEwrNJxmJlZ23W1B5DMzMw6HSdTMzOznJxMzczMcnIyNTMzy8nJ1MzMLCcnUzMzs5ycTM3MzHJyMjUzM8vJydTMzCwnJ1MzM7OcnEzNzMxycjI1MzPLqWTJVNIESU9Kun0r10dJml6q/ttD0pcknZ+Ox0n6aMG1myUdUIGYPhCHmZl1PqX81JgvA38bEStK2EeHiogbC07HAUuAF9K1vytVv5KqImLzVi5/IA4zM+t8SjIzlXQjsBdwj6SvS/qTpPnp+77N1D9O0oL0NV/S9qn8HyTNk7RI0rdb6K9W0lOSbkt1p0nqm66dkNpcLOkWSb1T+TWSlqX6/zeVXSXpCkmjgTrg9hTThyTNllQn6WJJ3y/oe5yk69PxeZLmpnt+JqmqhZjXSfqOpMeBkZK+lca6RNIkZZqLY4SkByU1SJopqabN/0BmZtahSpJMI+JLZDOp44F/A46NiOHAt4B/buaWK4CvRMQw4BjgHUknA4OBw8g+53OEpGNb6HZfYFJEHAS8AXxZUh9gMnB2RAwlm4lfLGlH4ExgSKr/3SbxTwPqgXMjYlhEvFNweRrwmYLzs4EpkvZPx0elcWwGzm0h3u2AJRFxeEQ8DPwkIg6NiAOBDwGnNo0D2ARcD4yOiBHALbTwWaySxkuql1S/Zs2aFkIxM7M8yvEAUn9gqqQlwI+AIc3UeQT4oaQJwICI2AScnL7mA08A+5El1615PiIeSce/Ao4mS7ArIuK/UvltwLFkyXY9cLOkzwBvFzuYiFgDPCvpCEk7pT4eAU4ARgDzJC1I53u10NRm4I6C8+MlPS5pMfBxmn+d9gUOBO5LffwTMKiFWCdFRF1E1A0cOLDYIZqZWRuVcs+00dXArIg4U1ItMLtphYi4RtIM4JPAY5JOBAT8S0T8rMh+oplzNVsxYpOkw8gS3hjgErIEVqwpwOeAp4C7IiIkCbgtIr5RZBvrG/dJ0wz6BqAuIp6XdBXQp5l7BCyNiJFtiNXMzEqsXDPT1el4XHMVJO0dEYsj4lqyZc39gJnAhZL6pTq7StqlhX52l9SYZM4BHiZLdrWSPpbKvwA8mNrsHxF/AC4jW0Zu6k1g+630dSdwRupnSiq7HxjdGKOkHSXt0UK8hRoT5yspttFbiWM5MLBxnJKqJTU3gzUzszIqx8z0+8Btkr4KPLCVOpdJOp5s6XMZ8B8RsSHtQz6aTfpYB5wHvLyVNp4Exkr6GfA08G8RsV7SBWTLzNsA84AbgR2Bu9OMUMDlzbQ3GbhR0jvAB2aCEfG6pGXAARExN5Utk/RPwL2SegEbga8Az7Xy+hARf5F0E7AYWJni3Foco4EfS+pP9u93HbC0tT7MzKx0FNF0dbTrScvH09PDO9aMurq6qK+vr3QYZmZdhqSGiKgrpq7fAcnMzCyncizzdpj09Oz9zVw6obPOStPfkfZuUvyFiFhciXjMzKzjdalkGhGv0vzDQp1WRBxe6RjMzKy0vMxrZmaWk5OpmZlZTk6mZmZmOTmZmpmZ5eRkamZmlpOTqZmZWU5OpmZmZjk5mZqZmeXkZGpmZpZTl3oHJGu/xavXUjtxRsX6X3nNpyrWt5lZqXlmamZmlpOTqZmZWU5Opm0kaV0HtzdbUlGfl2dmZp2Tk6mZmVlOTqbtpMwPJC2RtFjS2am8l6QbJC2VNF3SHySNLrLNc1JbSyRdm8qqJE0u6OfyVD5B0jJJiyT9tnQjNTOz1vhp3vb7DNlnqx4M7AzMkzQHOAqoBYYCuwBPAre01pikjwLXAiOA14F7JZ0BPA/s2vjh55IGpFsmAntGxIaCsqZtjgfGA1TtMLB9ozQzs1Z5Ztp+RwO/iYjNEfES8CBwaCqfGhHvRsSfgVlFtncoMDsi1kTEJuB24FjgWWAvSddLOgV4I9VfBNwu6TxgU3MNRsSkiKiLiLqqvv3bO04zM2uFk2n7qY3l7WovIl4nm/3OBr4C3JwufQr4KdlMtkGSVxnMzCrEybT95gBnpz3NgWSzyLnAw8Bn097p3wCjimzvceA4STtLqgLOAR6UtDPQKyLuAL4JHCKpF7BbRMwC/hEYAPTryMGZmVnxPJtpv7uAkcBCIIB/jIg/S7oDOAFYAvwXWZJc21pjEfGipG+QLQsL+ENE3C3pYODWlEABvgFUAb+S1D/V/VFE/KVjh2dmZsVSRFQ6hm5HUr+IWCdpJ7LZ6lFp/7RietcMjpqx11Wsf7+doJl1NZIaIqKo9wHwzLQ0pqcnbLcFrq50IgUYumt/6p3QzMxKwsm0BCJiVNMySXcBezYp/npEzCxLUGZmVjJOpmUSEWdWOgYzMysNP81rZmaWk5OpmZlZTk6mZmZmOTmZmpmZ5eRkamZmlpOTqZmZWU5OpmZmZjk5mZqZmeXkZGpmZpaT3wGph1i8ei21E2dUOgwz62F6yodceGZqZmaWk5OpmZlZTk6mFSbJS+1mZl1ci/8jl/SZlq5HxJ0dG07nIun3wG5AH+BfI2KSpFOAfwaqgFci4gRJ/YDrgToggG9HxB2S1kVEv9TWaODUiBgnaTLwGjAceELSFOA64EPAO8AFEbFcUhVwLfCJ1O5NwDLgksZPoZF0EnBxRLT4b2VmZqXT2qzo0+n7LsCRwAPp/HhgNtCtkylwYUS8JulDwDxJd5MltGMjYoWkHVO9bwJrI2IogKQPF9H2PsCJEbFZ0g6pzU2STiRL1p8FxpN9BurwdG1H4HXgp5IGRsQa4ALg1g4cs5mZtVGLyTQiLgCQNB04ICJeTOc1wE9LH17FTZDU+Dmku5EltzkRsQIgIl5L104ExjTeFBGvF9H21IjYnI77A7dJGkw2A60uaPfGiNhU2J+kXwLnSboVGAmc31wHksanmKnaYWARIZmZWXsUu2da25hIk5fIZlbdlqRRZMlsZEQcDMwHFpIluy2qb6W8sKxPk2tvFRxfDcyKiAPJVgMa626t3VuB84BzyJLypubGEBGTIqIuIuqq+vZvroqZmXWAYpPpbEkzJY2TNBaYAcwqYVydQX/g9Yh4W9J+wBFAb+A4SXsCFCzz3gtc0nhjwTLvS5L2l9QLOJOt6w+sTsfjCsrvBb7U+JBSY38R8QLwAvBPwOT2DtDMzDpGUck0Ii4BfgYcDAwDJkXEpaUMrBP4I7CNpEVkM8fHgDVky6Z3SloITEl1vwt8WNKSVH58Kp8ITCfbay6c2Tf1feBfJD1C9mBTo5uB/wYWpXY/X3DtduD5iFiWY4xmZtYBFNHcKqJ1dpJ+AsyPiJ8XU793zeCoGXtdiaMyM/ugrvwOSJIaIqKumLqt/WnMwxFxtKQ3+eDenYCIiB1yxGntJKmBbM/1a5WOxczMPDPtMerq6qK+vr7SYZiZdRltmZn6HZDMzMxycjI1MzPLycnUzMwsJydTMzOznJxMzczMcnIyNTMzy8nJ1MzMLCcnUzMzs5ycTM3MzHJyMjUzM8vJydTMzCynFt/o3rqPxavXUjtxRqXDMDN7T1f+RJmmPDM1MzPLycnUzMwspx6RTCVNkPSkpDskPSppg6QrKh2XmZl1Dz1lz/TLwN+SfaD2HsAZlQ2nfSRtExGbKh2HmZl9ULefmUq6EdgLuAc4NyLmARuLuK9W0lOSbpa0RNLtkk6U9IikpyUdluptJ+kWSfMkzZd0esH9D0l6In0dmcprJM2RtCC1e0wqX1fQ92hJk9PxZEk/lDQLuHZr/ZmZWeV0+5lpRHxJ0inA8RHxShtv/xhwFjAemAd8HjgaOA3432Qz3CuBByLiQkkDgLmS/hN4GTgpItZLGgz8BqhLbcyMiO9JqgL6FhHHPsCJEbFZ0j83119EvNX0JknjU+xU7TCwjUM3M7NidftkmtOKiFgMIGkpcH9EhKTFQG2qczJwWsEebB9gd+AF4CeShgGbyRIiZEn5FknVwO8jYkERcUyNiM2t9Pdk05siYhIwCaB3zeAocsxmZtZGTqYt21Bw/G7B+bu8/9oJ+GxELC+8UdJVwEvAwWTL6esBImKOpGOBTwG/lPSDiPgFUJjs+jSJo3DW2Wx/ZmZWOd1+z7QMZgKXShKApOGpvD/wYkS8C3wBqErX9wBejoibgJ8Dh6T6L0naX1Iv4Mx29GdmZhXSo2amkj4C1AM7AO9Kugw4ICLeyNHs1cB1wKKU4FYCpwI3AHdIOguYxfuzy1HAP0jaCKwDzk/lE4HpwPPAEqBfG/szM7MKUYS30nqC3jWDo2bsdZUOw8zsPZ397QQlNUREXTF1e9TMtCcbumt/6jv5D66ZWVfV45OppJ2A+5u5dEJEvFrueMzMrOvp8ck0JcxhlY7DzMy6Lj/Na2ZmlpOTqZmZWU5OpmZmZjk5mZqZmeXkZGpmZpaTk6mZmVlOTqZmZmY5OZmamZnl5GRqZmaWU49/B6SeYvHqtdROnFHpMMzMyqacb6TvmamZmVlOTqZmZmY5OZmWmaQ/tXL9D5IGlCseMzPLz3umOUiqiojNbbknIo5s5fon80VlZmbl5pnpVkiqlfSUpNskLZI0TVJfSSslfUvSw8BZkvaW9EdJDZIekrRfuv9vJN0laWH6OjKVr0vfayTNkbRA0hJJx6TylZJ2TsdfTdeWSLqsIK4nJd0kaamkeyV9qCIvkpmZAU6mrdkXmBQRBwFvAF9O5esj4uiI+C0wCbg0IkYAVwA3pDo/Bh6MiIOBQ4ClTdr+PDAzIoYBBwMLCi9KGgFcABwOHAF8UdLwdHkw8NOIGAL8Bfhsc8FLGi+pXlL95rfXtu8VMDOzVnmZt2XPR8Qj6fhXwIR0PAVAUj/gSGCqpMZ7eqfvHwfOB0hLwU2z2TzgFknVwO8jYkGT60cDd0XEW6mvO4FjgHuAFQX1G4Da5oKPiElkyZ7eNYOjuCGbmVlbeWbasqYJqPH8rfS9F/CXiBhW8LV/UQ1HzAGOBVYDv5R0fpMq2vKu92woON6MfykyM6soJ9OW7S5pZDo+B3i48GJEvAGskHQWgDIHp8v3Axen8ipJOxTeK2kP4OWIuAn4OdlScKE5wBlpn3Y74EzgoY4bmpmZdRQn05Y9CYyVtAjYEfi3ZuqcC1wkaSHZvujpqfzvgeMlLSZbih3S5L5RwAJJ88n2PP+18GJEPAFMBuYCjwM3R8T8DhiTmZl1MEV4K605kmqB6RFxYIVD6RC9awZHzdjrKh2GmVnZ5H07QUkNEVFXTF3vtfUQQ3ftT30Z36fSzKwncTLdiohYCXSLWamZmZWW90zNzMxycjI1MzPLycnUzMwsJydTMzOznJxMzczMcnIyNTMzy8nJ1MzMLCcnUzMzs5ycTM3MzHJyMjUzM8vJydTMzCwnvzdvD7F49VpqJ86odBhmZlvI++kunYFnpmZmZjk5mZqZmeXkZNoBJI2SNL3ScZiZWWV06WQqqarSMbSHpFb3qrvq2MzMeqJOnUwl/V5Sg6SlksansnWSviPpcWCkpBGSHkz1ZkqqSfW+KGmepIWS7pDUt4V+JksaXXC+Ln0fJWm2pGmSnpJ0uySla6eksoeBzxTcu52kW1Lf8yWdnsrHSZoq6d+Be7cSxyhJsyT9Glicyr4qaUn6uqyg7vmSFqXx/bK9r7GZmeXX2Z/mvTAiXpP0IWCepDuA7YAlEfEtSdXAg8DpEbFG0tnA94ALgTsj4iYASd8FLgKub0cMw4EhwAvAI8BRkuqBm4CPA88AUwrqXwk8EBEXShoAzJX0n+naSOCgiHithf4OAw6MiBWSRgAXAIcDAh6X9CDw19TPURHxiqQdm2so/QIyHqBqh4HtGLqZmRWjsyfTCZLOTMe7AYOBzcAdqWxf4EDgvjRhrAJeTNcOTEl0ANAPmNnOGOZGxCoASQuAWmAdsCIink7lvyIlLeBk4DRJV6TzPsDu6fi+VhJpY38r0vHRwF0R8Vbq507gGCCAaRHxCsDW2oyIScAkgN41g6PoEZuZWZt02mQqaRRwIjAyIt6WNJssMa2PiM2N1YClETGymSYmA2dExEJJ44BRLXS3ibTknZZxty24tqHgeDPvv2ZbS04CPhsRy5uM53DgrRZiaFRYRy304eRoZtZJdOY90/7A6ymR7gcc0Uyd5cBASSMBJFVLGpKubQ+8mJaCz22lr5XAiHR8OlDdSv2ngD0l7Z3Ozym4NhO4tGBvdXgrbbVkDnCGpL6StgPOBB4C7gc+J2mn1Eezy7xmZlYenTmZ/hHYRtIi4GrgsaYVIuKvwGjgWkkLgQXAkenyN4HHgfvIkl9LbgKOkzSXbH+yxRlkRKwnW9adkR5Aeq7g8tVkyXiRpCXpvF0i4gmyGfZcsrHcHBHzI2Ip2d7wg2ncP2xvH2Zmlp8ivFrYE/SuGRw1Y6+rdBhmZlvorG8nKKkhIuqKqdtp90ytYw3dtT/1nfQH1sysq+tRyVTSlcBZTYqnRsT3yhzHUKDp34ZuiIjDyxmHmZl1jB6VTFPSLGvi3Eoci4FhlY7DzMw6Rmd+AMnMzKxLcDI1MzPLycnUzMwsJydTMzOznJxMzczMcnIyNTMzy8nJ1MzMLCcnUzMzs5ycTM3MzHLqUe+A1JMtXr2W2okzKh2GmXVRnfXN6DsLz0zNzMxycjI1MzPLqSzJVNIASV8uR195SfpT+l4r6fMF5XWSflyBeD4Qh5mZdT7lmpkOALZIppKqytR/0SLiyHRYC3y+oLw+IiaUok9JLe1dfyAOMzPrfMqVTK8B9pa0QNI8SbMk/RpYnGZeSxorSrpC0lXpeG9Jf5TUIOkhSfttrQNJkyXdmOr9l6RTU3kfSbdKWixpvqTjU/kQSXNTTIskDU7l6wpiPiZdv1zSKEnTJfWStFLSgIK+n5H0N5IGSrojjXGepKNaiPcqSZMk3Qv8Ir0OD0l6In01JvWmcVRJ+kFqf5Gk/9X2fw4zM+tI5XqadyJwYEQMkzQKmJHOV0iqbeG+ScCXIuJpSYcDNwAfb6F+LXAcsDcwS9LHgK8ARMTQlIzvlbQP8CXgXyPidknbAk1nyROBKyKiMSmPSu28K+lu4Ezg1hTXyoh4Kf2C8KOIeFjS7sBMYP8W4h0BHB0R70jqC5wUEetTYv8NUNdMHOOBtRFxqKTewCOS7o2IFU0bT3XHA1TtMLCFMMzMLI9K/WnM3Ob+519IUj/gSGCqpMbi3q20+7uIeBd4WtKzwH7A0cD1ABHxlKTngH2AR4ErJQ0C7oyIp9sQ/xTgW8CtwJh0DnAicEBBvDtI2j4i3txKO/dExDvpuBr4iaRhwOYUY3NOBg6SNDqd9wcGA1u8nhExiewXEnrXDI4ix2ZmZm1UqWT6VsHxJj643Nwnfe8F/CUihrWh3aYJIwA1WzHi15IeBz4FzJT0dxHxQJH9PAp8TNJA4AzguwUxjyxIkK0pfB0uB14CDk7trN/KPQIujYiZRfZhZmYlVq490zeB7bdy7SVgF0k7pWXLUwEi4g1ghaSzAJQ5uJV+zkp7mnsDewHLgTnAuamNfYDdgeWS9gKejYgfA/cABxUbc0QEcBfwQ+DJiHg1XboXuKSxXpplFqs/8GKaWX+B95edm8YxE7hYUnXjmCRt14Z+zMysg5UlmaZk80h60OgHTa5tBL4DPA5MB54quHwucJGkhcBS4PRWuloOPAj8B9le63qyfdYqSYvJlmPHRcQG4GxgiaQFZMvBv2jS1iJgk6SFki5vpq8pwHm8v8QLMAGoSw8GLSPbly3WDcBYSY+RLfE2zlqbxnEzsAx4Ir2eP8PvZGVmVlHKJlldn6TJwPSImFbpWDqj3jWDo2bsdZUOw8y6qJ74doKSGiKirpi6ntH0EEN37U99D/yPwcysHLpcMpV0JXBWk+KpETGuAuG0StIFwN83KX4kIr5SiXjMzKzjdZtlXmtZXV1d1NfXVzoMM7Muoy3LvH6jezMzs5ycTM3MzHJyMjUzM8vJe6Y9hKQ3yf4OtyfaGXil0kFUkMfv8ffU8ecd+x4RUdQbm3e5p3mt3ZYXu5He3Uiq76ljB4/f4++54y/n2L3Ma2ZmlpOTqZmZWU5Opj3HpEoHUEE9eezwmJGYAAADrUlEQVTg8Xv8PVfZxu4HkMzMzHLyzNTMzCwnJ9NuRNIpkpZLekbSxGau95Y0JV1/XFJt+aMsnSLG/1VJy9JH5N0vaY9KxFkqrY2/oN5oSSGpWz3hWcz4JX0u/QwslfTrcsdYKkX87O8uaZak+enn/5OViLNUJN0i6eX0sZTNXZekH6fXZ5GkQzo8iIjwVzf4Ivsw8f9P9qHo2wILgQOa1PkycGM6HgNMqXTcZR7/8UDfdHxxTxt/qrc9MAd4DKirdNxl/vcfDMwHPpzOd6l03GUc+yTg4nR8ALCy0nF38GtwLHAIsGQr1z9J9jnXAo4AHu/oGDwz7T4OA56JiGcj4q/Ab9nyw9RPB25Lx9OAEySpjDGWUqvjj4hZEfF2On0MGFTmGEupmH9/gKuB7wPryxlcGRQz/i8CP42I1wEi4uUyx1gqxYw9gB3ScX/ghTLGV3IRMQd4rYUqpwO/iMxjwABJNR0Zg5Np97Er8HzB+apU1mydiNgErAV2Kkt0pVfM+AtdRPabanfR6vglDQd2i4jp5QysTIr5998H2EfSI5Iek3RK2aIrrWLGfhVwnqRVwB+AS8sTWqfR1v8/tJnfAan7aG6G2fRR7WLqdFVFj03SeUAdcFxJIyqvFscvqRfwI2BcuQIqs2L+/bchW+odRbYq8ZCkAyPiLyWOrdSKGfs5wOSI+H+SRgK/TGN/t/ThdQol/3+fZ6bdxypgt4LzQWy5lPNeHUnbkC33tLQ00pUUM34knQhcCZwWERvKFFs5tDb+7YEDgdmSVpLtG93TjR5CKvbn/+6I2BgRK8jeq3pwmeIrpWLGfhHwO4CIeBToQ/a+tT1FUf9/yMPJtPuYBwyWtKekbckeMLqnSZ17gLHpeDTwQKTd+W6g1fGnZc6fkSXS7rJf1qjF8UfE2ojYOSJqI6KWbM/4tIjoLp8YX8zP/+/JHkJD0s5ky77PljXK0ihm7P8NnAAgaX+yZLqmrFFW1j3A+emp3iOAtRHxYkd24GXebiIiNkm6BJhJ9nTfLRGxVNJ3gPqIuAf4OdnyzjNkM9IxlYu4YxU5/h8A/YCp6bmr/46I0yoWdAcqcvzdVpHjnwmcLGkZsBn4h4h4tXJRd4wix/414CZJl5Mtb47rRr9II+k3ZMv3O6d94f8DVANExI1k+8SfBJ4B3gYu6PAYutHraWZmVhFe5jUzM8vJydTMzCwnJ1MzM7OcnEzNzMxycjI1MzPLycnUzMwsJydTMzOznJxMzczMcvofqvzfHeQejREAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "quality_metrics_pandas = subscription.quality_monitoring.get_table_content()\n",
    "\n",
    "%matplotlib inline\n",
    "quality_metrics_pandas.plot.barh(x='id', y='value');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Fairness, Drift monitoring and explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairness monitor configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes.feature import Feature\n",
    "\n",
    "subscription.fairness_monitoring.enable(\n",
    "            training_data=training_data_pd,\n",
    "            features=[\n",
    "                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n",
    "                Feature(\"Age\", majority=[[26, 75]], minority=[[18, 25]], threshold=0.95)\n",
    "            ],\n",
    "            favourable_classes=['No Risk'],\n",
    "            unfavourable_classes=['Risk'],\n",
    "            min_records=5\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run fairness monitor on demand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================\n",
      "\n",
      " Counting bias for deployment_uid=563f01d37f720857b95c557dc76176ad \n",
      "\n",
      "===================================================================\n",
      "\n",
      "\n",
      "\n",
      "RUNNING\n",
      "FINISHED\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "fairness_run = subscription.fairness_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show run results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>FairnessMetrics (binding_id=6ac4d3c4-862f-4702-8053-9f531466686a, subscription_id=7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>feature</th><th style='border: 1px solid #dddddd'>feature_value</th><th style='border: 1px solid #dddddd'>fairness_biased</th><th style='border: 1px solid #dddddd'>fairness_value</th><th style='border: 1px solid #dddddd'>fairness_fav_class</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>process</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-04-01 18:47:05.781993+00:00</td><td style='border: 1px solid #dddddd'>Sex</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>100.0</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-04-01 18:47:05.781993+00:00</td><td style='border: 1px solid #dddddd'>Age</td><td style='border: 1px solid #dddddd'>[18, 25]</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>1.667</td><td style='border: 1px solid #dddddd'>100.0</td><td style='border: 1px solid #dddddd'>6ac4d3c4-862f-4702-8053-9f531466686a</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'></td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.fairness_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explainability configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.explainability.enable(training_data=training_data_pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get sample transaction/scoring id. You can also use `subscription.payload_logging.show_table()` to see list of your transactions (`scoring_id` column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample transaction/scoring id:  219dbfc5-18e7-48cf-bcab-94de377ec475-1\n"
     ]
    }
   ],
   "source": [
    "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'][0]\n",
    "print('Sample transaction/scoring id: ', transaction_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "====================================================================\n",
      "\n",
      " Looking for explanation for 219dbfc5-18e7-48cf-bcab-94de377ec475-1 \n",
      "\n",
      "====================================================================\n",
      "\n",
      "\n",
      "\n",
      "in_progress..\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "explain_run = subscription.explainability.run(transaction_id=transaction_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAD8CAYAAABw+Q3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcVNWd///XW0RBQYmKxiWk0bihYqsFCirimomjCUaMGszIxCUmY9y+OkO2CSaaaHTiwyXG4IYmakzcwk+NoEYEV2ywWV0xOCEyiiYqKKDg5/fHPa2Xspfq7iq6u3g/H4969K17zzn3U7cKPnXOPXWvIgIzMzOrXut0dABmZmZWWU72ZmZmVc7J3szMrMo52ZuZmVU5J3szM7Mq52RvZmZW5ZzszczMqpyTvZmZWZVzsjczM6ty63Z0AGYAm222WdTU1HR0GGZmXcr06dPfjIi+LZVzsrdOoaamhrq6uo4Ow8ysS5H0ainlPIxvZmZW5ZzszczMqpyTvZmZWZXzOXszM+tUPvzwQxYuXMjy5cs7OpROo0ePHmyzzTZ07969TfWd7M2s1QrjCh0dQqPqTvUkz2qwcOFCevfuTU1NDZI6OpwOFxG89dZbLFy4kP79+7epDQ/jm5lZp7J8+XI23XRTJ/pEEptuumm7Rjqc7K3VJC1tZttwSfeuyXjMrPo40a+uvcfDyd7MzKzK+Zy9tYmyr5m/AL4EBHBBRNyeNm8k6W5gR2AK8J2I+KhjIjWzrq7cc0QqObfj5JNP5pxzzmHAgAFNlhk9ejRHHHEEI0eOXG39ggULeOKJJ/j6179e9rjcs7e2+ipQC+wOHAJcImnLtG0w8P+A3YDtUtlPkXSqpDpJdYsXL14DIZuZVdZ1113XbKJvzoIFC7j11lvLHFHGyd7aaj/gtohYFRGvA48Cg9K2aRHxSkSsAm5LZT8lIsZFRCEiCn37tnhpZzOzNeYXv/gFV1xxBQBnn302Bx10EAAPP/wwJ5xwApMmTWLIkCHsueeeHHPMMSxdmk1lGj58+MeX/r7++uvZYYcdGD58OKeccgqnn376x+1PmTKFoUOHsu2223LHHXcAMGbMGKZOnUptbS2XXXZZWV+Pk721VXOzRaKF52ZmndqwYcOYOnUqAHV1dSxdupQPP/yQxx57jN12240LLriAhx56iBkzZlAoFPjlL3+5Wv3XXnuNn/70pzz11FM8+OCDPP/886ttX7RoEY899hj33nsvY8aMAeCiiy5i//33p76+nrPPPrusr8fJ3tpqCnCspG6S+gLDgGlp22BJ/SWtAxwLPNZRQZqZtcVee+3F9OnTWbJkCeuvvz5Dhgyhrq6OqVOn0rNnT+bNm8e+++5LbW0tN910E6++uvr9aKZNm8YBBxzAJptsQvfu3TnmmGNW2z5ixAjWWWcdBgwYwOuvv17x1+MJetYqktYFVgB3A0OAmWQ99/+MiP+TtBPwJHAR2Tn7KamsmVmX0b17d2pqarjxxhsZOnQoAwcO5JFHHmH+/Pn079+fQw89lNtuu63J+hHND2iuv/76JZctB/fsrbV2AeZH5ryI2DUidmuYiR8RkyPioIg4NiIGRMRpnolvZl3RsGHDuPTSSxk2bBj7778/11xzDbW1teyzzz48/vjjvPzyywC8//77vPjii6vVHTx4MI8++ij//Oc/WblyJXfeeWeL++vduzdLliypyGtxz95KJuk04AzgrI6OxTqWL0tra1JHfd72339/LrzwQoYMGcKGG25Ijx492H///enbty/jx4/n+OOPZ8WKFQBccMEF7LDDDh/X3Xrrrfn+97/P3nvvzVZbbcWAAQPYeOONm93fwIEDWXfdddl9990ZPXp0Wc/ba00MH5i1pFAoRMMMVjNbuz333HPsvPPOHR1Guy1dupRevXqxcuVKjjrqKL75zW9y1FFHtbm9xo6LpOkR0eKFCDyMb2ZmVgFjx46ltraWXXfdlf79+zNixIgOi8XD+GZmZhVw6aWXdnQIH3PP3szMOh2fYl5de4+Hk72ZmXUqPXr04K233nLCTxruZ9+jR482t+FhfDMz61S22WYbFi5ciO+Z8YkePXqwzTbbtLm+k72ZmXUq3bt3p3///h0dRlXxML6ZmVmVc7I3MzOrck72ZmZmVc7n7M2sUYVxLV6Uq9PxZXzNGrfW9uwlLW1jvRGSBpRQbqykc9PyeEkj27K/VsQ1WtJWueeTJb0gaaakxyXtWMn9NxNXScfLzMwqZ61N9u0wAuiMyWs0sFXRulERsTtwE3BJqQ2l29iWS2c9XmZma421PtlLGp56wXdIel7SLZKUtl0kaZ6kWZIulTQU+DJwiaR6SdtJOkXSM6kHfaekDVrY3wJJP5P0pKQ6SXtKmihpfrqrXEO581K7sySdn9bVSHpO0rWS5kqaJKlnGjUoALekuHoW7XYK8IXUxl6SHpU0Pe13y7R+corrUeBMSVtIuju9rpnptSPpBEnT0n5+I6lbWr9U0oWp7FOp/qeOV7vfMDMza7W1Ptkne5DdtnUAsC2wr6RNgKOAXSJiIHBBRDwBTADOi4jaiJgP3BURg1IP+jngpBL297eIGAJMBcYDI4F9gJ8ASDoM2B4YDNQCe0kalupuD/wqInYB3gaOjog7gDqynnxtRCwr2t+RwGxJ3YErgZERsRdwA3BhrlyfiDggIv4HuAJ4NL2uPYG5knYGjgX2jYhaYBUwKtXdEHgqlZ8CnNLE8TIzszXME/Qy0yJiIYCkeqAGeApYDlwn6T7g3ibq7irpAqAP0AuYWML+JqS/s4FeEbEEWCJpuaQ+wGHp8Wwq14ssyf8v8NeIqE/rp6dYm3KLpGXAAuC7wI7ArsCDafCiG7AoV/723PJBwL8BRMQq4B1J3wD2Ap5J9XsCb6TyH/DJMZoOHNrSQZB0KnAqQL9+/VoqbmZmbeRkn1mRW14FrBsRKyUNBg4GjgNOJ0uAxcYDIyJipqTRwPBW7O+jon1/RPaeCPh5RPwmX0lSTSOxFg/Z542KiI+nJ6cvEnPTqEJj3mshbgE3RcT3Gtn2YXxyIetVlPDZiohxwDjI7mffUnkzM2sbD+M3QVIvYOOIuJ9siL82bVoC9M4V7Q0sSkPkoyiPicA3UwxI2lrS5i3UKY6rMS8AfSUNSe12l7RLE2UfBr6dynWTtFFaN7IhFkmbSPp8GeIyM7MKcrJvWm/gXkmzgEeBs9P63wPnSXo2TTj7EfA08CDwfDl2HBGTgFuBJyXNBu6g5YQ5HrimiQl6De1+QDY/4GJJM4F6YGgT7Z0JHJj2P51s7sI84IfApHRcHgS2bCGu4uNlZmZrmHwLQesMCoVC1NX5giidiS+qY9b5SZoeES3+Y/U5ezNrlBOnWfXwML6ZmVmVc7I3MzOrck72ZmZmVc7J3szMrMo52ZuZmVU5J3szM7Mq52RvZmZW5ZzszczMqpyTvZmZWZVzsjczM6tyvlyumXXJ6+A3xpf4NWuce/ZmZmZVruLJXtJnJf1e0nxJ8yTdL2mHSu83t/8+kr6Te14jaVm6Few8STene9G3pe37JfVpZP1YSee2J+5WxDBW0t/T63lJ0l2SBpSx/RpJX889L0i6olztm5lZ5VU02UsScDcwOSK2i4gBwPeBLUqs363oeVtOO/QBvlO0bn5E1AK7AdsAX2tDu0TE4RHxdlvqltllEVEbEdsDtwN/kdS31MotHNca4ONkHxF1EXFGmyM1M7M1rtI9+wOBDyPimoYVEVEPdJN0b8M6SVdJGp2WF0j6b0mPAcdImizpZ5IeBc6U1FfSnZKeSY99U72xkm5I5V+R1JCQLgK2Sz3fS/LBRcQqYBqwdWqjm6RLUruzJH0rrd9S0pTUxhxJ++di3Swt/0DSC5IeAnbMvbbtJD0gabqkqZJ2SuvHS7pC0hMp3pG5Ov8pabakmZIuaq6dYhFxOzCJlKCLYixImpw7XuMkTQJuTj34qZJmpMfQ3PHbP732syUNb3jvJG0i6Z50rJ6SNLCF98LMzDpApSfo7QpMb0O95RGxH4Ck04A+EXFAen4rWU/2MUn9gInAzqneTmRfMHoDL0j6NTAG2DX15JFU07ATST2AvYEz06qTgHciYpCk9YHHUzL8KjAxIi5Mow0b5IOVtBdwHLAH2TGdkXvd44DTIuIlSXsDVwMHpW1bAvuluCcAd0j6EjAC2Dsi3pe0SQntFJuR2mzJXsB+EbFM0gbAoRGxXNL2wG1AIR2/cyPiiPRah+fqnw88GxEjJB0E3AzUpm2fei8i4sMSYjIzszLrrLPxb2/m+SHAgOwMAQAbSeqdlu+LiBXACklv0PTpgu0k1QPbA3dExKy0/jBgYK6XvXEq8wxwQzq3f08ancjbH7g7It4HkDQh/e0FDAX+mIt3/Vy9eyLiI2CepIZYDwFubGgrIv5RQjvF1My2vAkRsSwtdweuklQLrAJKmVexH3B0ivMvkjaVtHHa1th7sXC1IKVTgVMB+vXrV2LIZmbWWpVO9nOBkY2sX8nqpxB6FG1/r5nn6wBDckkKgJQEV+RWraLp1zc/ImolbQlMlvTliJhAliS/GxETiytIGgb8K/BbSZdExM1FRaKR/awDvN0wqtCIfLzK/S1uq6V2iu0BNPwGKX+smzvOZwOvA7un8stL2E9jXyoaYm/xvYiIcWQjFhQKhcaOn5mZlUGlz9n/BVhf0ikNKyQNArqR9c7XTz3Bg1vR5iTg9Fx7LSXAJWRDyZ8SEYvIhqm/l1ZNBL6devBI2kHShpI+D7wREdcC1wN7FjU1BThKUs80ynBkav9d4K+SjkntSdLuJby+b6ZhdSRt0pp2JB1NNkJxW1q1gGy4HlIvvAkbA4vSSMM3yN4jaOb4kb3uUWm/w4E3U6xmZtaJVDTZR0QARwGHKvvp3VxgLPAa8AdgFnAL8Gwrmj0DKKRJYfOA01qI4S2yc+9zVDRBL7kH2CBNursOmAfMkDQH+A1Zj3Q4UC/pWbKEeXnRPmaQnWqoB+4EpuY2jwJOkjSTbKTjKy3E+wDZ+fu6dKqh4Sd8zbVzdppA9xJwAnBQRCxO284HLpc0layH3ZSrgRMlPUU2hN/Q658FrEyTBc8uqjOW9F6QTeQ7sbnXZmZmHUNZPjbrWIVCIerqfPWzjuIr6Jl1TZKmR0SL/4A76wQ9M1uDnCTNqpsvl2tmZlblnOzNzMyqnJO9mZlZlXOyNzMzq3JO9mZmZlXOyd7MzKzKOdmbmZlVOSd7MzOzKudkb2ZmVuWc7M3MzKqcL5drZlWjWq7x3xJf3thayz17MzOzKudk3w6SPivp9+n2vfMk3S/pVEn3lqHt4Y21I+nLksa0o90jJD2bblk7T9K30voRkgaUUL+kcmZm1nk42beRJAF3A5MjYruIGAB8H9iikvuNiAkRcVFb6krqDowDjoyI3YE9gMlp8wiglCReajkzM+sknOzb7kDgw4i4pmFFRNQDU4Feku6Q9LykW9IXAyTtJelRSdMlTZS0ZVr/BUkPpd72DEnb5XckaVDqjW8rabSkq9L68ZKukPSEpFckjUzr15F0taS5ku5NIw4jgd5k8zTeSvGuiIgXJA0FvgxcIqle0naSTpH0TIrpTkkbNFFusqRC2u9mkhak5V0kTUvlZknavmLvhJmZNatVyV7ShpUKpAvaFZjexLY9gLPIesDbAvumXvWVwMiI2Au4Abgwlb8F+FXqbQ8FFjU0lBLsNcBXIuKVRva1JbAfcATQ0OP/KlAD7AacDAwBiIh/ABOAVyXdJmmUpHUi4om0/ryIqI2I+cBdETEoxfQccFIT5ZpyGnB5RNQCBWBhcYF0yqNOUt3ixYubacrMzNqjpGQvaaikeWT/6SNpd0lXVzSyrm1aRCyMiI+AerLEuyPZF4QHJdUDPwS2kdQb2Doi7gaIiOUR8X5qZ2c+GXb/3yb2dU9EfBQR8/jkFMJ+wB/T+v8DHmkoHBEnAwcD04Bzyb50NGZXSVMlzQZGAbu08hg8CXxf0n8Bn4+IZcUFImJcRBQiotC3b99WNm9mZqUqtWd/GfBFPhn+nQkMq1RQXcRcYK8mtq3ILa8iGzoXMDf1iGsjYreIOCytb8oiYDnZSEFT8vtS0d9GRcTsiLgMOBQ4uoli44HTI2I34HygRxPlVvLJ5+jjMhFxK9mQ/zJgoqSDmovJzMwqp+Rh/Ij4W9GqVWWOpav5C7C+pFMaVkgaBBzQRPkXgL6ShqSy3SXtEhHvAgsljUjr15e0QarzNvCvwM8kDW9FbI8BR6dz91sAw1PbvYraqQVeTctLyM7pN+gNLEqnH0bl1heXW8AnX3pGNqyUtC3wSkRcQTb0P7AV8ZuZWRmVmuz/ls4dh6T1JJ1LGtJfW0VEAEcBh6af3s0FxgKvNVH+A7JkeLGkmWTD+0PT5m8AZ0iaBTwBfDZX73XgSOBXkvYuMbw7yc6RzwF+AzwNvEPW4/9PSS+kUwnnA6NTnd8D56WJgNsBP0r1HgSez7VdXO5S4NuSngA2y5U7FpiT9rMTcHOJsZuZWZkpy1ktFJI2Ay4HDiFLGJOAMyPircqGZ20lqVdELJW0Kdn5+X3T+ftOqVAoRF2drwpm7eMr6NnaRtL0iGjxg1/S5XIj4k1WH8q1zu9eSX2A9YCfduZEb1YuToJmjSsp2UvqD3yXbFb5x3Ui4suVCcvaKyKGd3QMZmbWOZR6I5x7gOuB/w/4qHLhmJmZWbmVmuyXp1nVZmZm1sWUmuwvl/Rjsol5H/+uOyJmVCQqMzMzK5tSk/1uZD8PO4hPhvEjPTczM7NOrNRkfxSwbfqtuJmZmXUhpV5UZybQp5KBmJmZWWWU2rPfAnhe0jOsfs7eP70zMzPr5EpN9j+uaBRmZmZWMaVeQe/RSgdiZmZmlVHqFfT2Aa4ku7/6ekA34L2I2KiCsZlZI9aW67+3hS+Xa9a4UifoXQUcD7wE9AROTuvMzMysk2vN/exfBrpFxKqIuJF0j/S1gaSlbaw3QtKAEsqNTbcNRtJ4SSNbqtMekkZL2ir3fHK67e1MSY9L2jG33t1IM7MurtRk/76k9YB6Sb+QdDawYQXjqhYjgBaTfQcYDWxVtG5UROwO3ARcssYjMjOziik12X8jlT0deA/4HHB0pYLqrCQNT73dOyQ9L+kWSUrbLpI0T9IsSZdKGgp8GbhEUr2k7SSdIumZ1IO+U9IGLexvgaSfSXpSUp2kPSVNlDRf0mm5cueldmdJOj+tq5H0nKRrJc2VNElSzzRqUABuSXH1LNrtFOALjcTy6xTD3IZ95GI8X9IMSbMl7ZTWH5Dar5f0rKTebTvqZmbWXqXOxn81LS4Hzm+u7FpgD2AX4DXgcWBfSfPIrjK4U0SEpD4R8bakCcC9EXEHgKS3I+LatHwBcBLZxMfm/C0ihki6DBgP7Av0AOYC10g6DNgeGAwImCBpGPC/af3xEXGKpD8AR0fE7ySdDpwbEXUplvz+jgRmNxLHDyLiH5K6AQ9LGhgRs9K2NyNiT0nfAc4lm9NxLvAfEfG4pF5kn53VSDoVOBWgX79+LRwGMzNrq5J69pL2lfSgpBclvdLwqHRwndS0iFgYER8B9UAN8C5ZMrtO0leB95uou6ukqZJmA6PIvjS0ZEL6Oxt4OiKWRMRiYLmkPsBh6fEsMAPYiSzJA/w1IurT8vQUa1NukVRP9mXi3Ea2f03SjLSfXVj99MRdjezjceCXks4A+kTEyuIGI2JcRBQiotC3b99mQjMzs/Yo9aI61wNnk/1nvqpy4XQJK3LLq4B1I2KlpMHAwcBxZKc7GrtJ0HhgRETMlDSa0iY5Nuzvo6J9f0T2/gn4eUT8Jl9JUk0jsRYP2eeNaujpF5PUn+wLwKCI+Kek8WSjC8UxrkoxEREXSboPOBx4StIhEfF8M/s3M7MKKfWc/TsR8eeIeCMi3mp4VDSyLiQNU28cEfcDZwG1adMSIH+uujewSFJ3sp59OUwEvpliQNLWkjZvoU5xXC3ZiGyuxjuStgC+1FIFSdtFxOyIuBioIxtxMDOzDlBqz/4RSZeQDdf6fvaf1hv4k6QeZD3ts9P63wPXpqHskcCPgKeBV8mG5ds9aS0iJknaGXgynXtfCpxA8yMw48nO9y8DhpSwj5mSniWbJ/AK2RB9S86SdGCKYx7w5xLqmJlZBSgiWi4kPdLI6ogI38/eyqJQKERdna9+VgpfQa9pvoKerW0kTY+IFv9TKHU2/oEt7OzEiLip1ODMrO2c0MystUq+gl4LzixTO2ZmZlZm5Ur2armImZmZdYRyJfuWT/ybmZlZh3DP3szMrMqVK9mX8lMsMzMz6wClXi53C0nXS/pzej5A0kkN2yPi9EoFaGZmZu1Tas9+PNmV2hpui/oi2ZXizMzMrJMrNdlvFhF/ILseO+mmJmv7NfLNzMy6hFKT/XuSNiXNupe0D/BOxaIyMzOzsin12vjnkN1qdTtJjwN9ya71bmZmZp1ci8le0jpktzM9ANiR7Gd2L0TEhxWOzcysVXzfgOb5UstrrxaH8SPiI+B/ImJlRMyNiDmdNdFLWiWpPvcY00L5+yX1aWb7WZI2KLV8CfENljRF0guSnpd0Xb79NrQ3XtLItHydpAFp+fsl1O0u6SJJL0maI2mapBZvXdvK+Gokfb2cbZqZWeuVes5+kqSjle6h2okti4ja3OOi5gpHxOER8XYzRc4CNmhF+Sal+8D/EfiviNgR2Bl4gKLb3Eoq9dTKaiLi5IiYl562mOyBnwJbArtGxK7AkcWxlEEN4GRvZtbBSk3255AlqhWS3pW0RNK7FYyrbCRtnHrSO6bnt0k6JS0vkLSZpA0l3SdpZurlHpvuQb8V8EjDLX5z5WskPSfpWklzJU2S1DOVGSRplqQnJV0iaU4K5T+AmyLiScjuDxwRd0TE65LGShonaRJws6Ruqe4zqa1vpbYl6SpJ8yTdB2yee52TJRUkXQT0TCMbtzTx2jYATgG+GxErUjyvp19cIOl4SbNT+Ytz+1iaWx4paXxaHi/pCklPSHqlYbQBuAjYP8VydvneVTMza42Skn1E9I6IdSJivYjYKD3fqNLBtUFDkmt4HBsR7wCnA+MlHQd8JiKuLar3L8BrEbF76uU+EBFXAK8BBzZxi9/tgV9FxC7A28DRaf2NwGkRMYTVf564KzC9mdj3Ar4SEV8HTgLeiYhBwCDgFEn9gaPI5k3sRpashxY3EhFj+GSEY1Rjrw34AvC/EfGpL2yStgIuBg4CaoFBkkY0E3eDLYH9gCPIkjzAGGBqiuWyEtowM7MKKGnIWNKwxtZHxJTyhtNuyyKitnhlRDwo6RjgV8DujdSbDVyaerH3RsTUEvb114ioT8vTgZp0Pr93RDyR1t9KlvxKMSEilqXlw4CBuR7yxmRfLoYBt0XEKuA1SX8pod1PvbYWzsYMAiZHxGIASbek/d7Twn7uSfM75qVTFi2SdCpwKkC/fv1KqWJmZm1Q6vnh83LLPYDBZAnuoLJHVAHpFwU7A8uATYCF+e0R8aKkvYDDgZ9LmhQRP2mh2RW55VVAT5q/IdBcst77n5rY/l4+ZLIh9olFr+NwWnmHwcZeG3Ap0E9S74hYUlSludeQ33ePom3541HS3I6IGAeMAygUCr5zoplZhZQ6jH9k7nEo2ZD065UNrazOBp4DjgdukNQ9vzENXb8fEb8jS4R7pk1LaMWktYj4J7BE2UWHAI7Lbb4KOFHS3rn9niDps400NRH4dkOcknaQtCEwBTgundPfEmjs9ALAh7m6n3ptEfE+cD1whaT1UrktJZ0APA0ckOYmdCM7Zo+mdl+XtHP68nRUCYekVcfPzMwqo00zv8l6xruWM5Ay6SmpPvf8AeAG4GRgcEQskTQF+CHw41y53YBLJH0EfAh8O60fB/xZ0qImzts35iTgWknvAZNJVxpME/GOIxtS35zs0sNTgLsaaeM6spnsM9IvIBYDI4C7yUZTZpPdn+DRRuo2xD1L0gzg5iZe2w+BC8iG3ZeTjSz8d0QskvQ94BGyHvr9EdEwGjEGuBf4GzAH6NXCsZgFrJQ0Exjv8/ZmZh1DES2Pnkq6kk+GcNchm7i1ICJOqGBsXZKkXhGxNC2PAbaMiDM7OKxOr1AoRF2dL/hh7eOL6jTPF9WpPpKmR0SLH/xSe/b5T8hKsklivod94/419YzXBV4FRndsOGZrDyczs8aVmuz7RMTl+RWSzixeZxARtwO3d3QcZmZmDUq9qM6JjawbXcY4zMzMrEKa7dlLOp7scqf9JU3IbeoNvFXJwMzMzKw8WhrGfwJYBGwG/E9u/RKymdZmZmbWyTWb7CPiVbJJZkPWTDhmZmZWbiWds5e0T7opy1JJHyi7lWyXuBGOmZnZ2q7UCXpXkV1J7SWyy8KeDFxZqaDMzMysfEq+gl5EvCypW7oJy42SnmixkpmZmXW4UpP9++ka6vWSfkE2aW/DyoVlZmZm5VLqMP43UtnTya6h/jk+uX+7mZmZdWIl9ewj4lVJPcmu835+hWMyM2sTXxu/NL6s8Nqn1Nn4RwL1ZHeRQ1Jt0UV2zMzMrJMqdRh/LDAYeBsgIurJbsFqFSRpU0n16fF/kv6ee75eE3UWSurTxv09Jqm2fVGbmVlnU+oEvZUR8U52a3VbUyLiLbLbCSNpLLA0Ii7t0KDMzKzLKbVnP0fS14FukrZP97f3T+86kKQTJU1LvfyrJX3qvWysjKR1Jb0t6TJJMyQ9KGnTXLXjUp0XJA1N7fSUdJOk2anOsLT+ZEl3SJoo6SVJP8/t+0uSnkzlb5fkX2+YmXWQZpO9pN+mxfnALsAK4DbgXeCsyoZmTZG0K3AUMDQiaslGaI5rRZmNgaciYk/gSeBH+aoRMRg4D/jvtO4M4IOI2I3slxm/zZ1G2B0YCQwETpC0laTNgTHAwWkfs4Azy3YAzMysVVoaxt9L0ueBY4EDWf1mOBsAyysVmDXrEGAQUJdOrfQE/taKMiuBP6bl3wG35urdlf5O55N5GfsBlwBExFxJrwFfSNseioglAJKeB/oBnwUGAE+kfa8HPFb8IiSdCpwK0K9fvxJfupmZtVZLyf4ashn42wL532oIiLTe1jwBN0TEj1pbRtK6ZO9dXv75ivR3FZ98PpqbrLEqyxvtAAARPklEQVQit9xQR8ADEfGNZuoREeOAcQCFQqE4JjMzK5Nmh/Ej4oqI2JksaWybe/SPCCf6jvMQ8DVJm8HHs/aLu8bNlekOfDUtf51Get1FpgCjUjs7A1sCLzdT/gngAEnbpjobStq+pFdmZmZlV9IEvYj4dqUDsdJFxGzgfOAhSbOAScAWrSjzDrCnpBlkQ/QXtLDLK4GekmYDtwD/FhEfNBPf68BJwO2SZpIl/x1a9yrNzKxcFOHR07VJGsZ/MyLa9Fv8SikUClFX56t6Wfv4Cnql8RX0qoek6RHR4ge/5LvemZl1dk5iZo1zsl/LRMRKoFP16s3MrLJKvaiOmZmZdVFO9mZmZlXOyd7MzKzKOdmbmZlVOSd7MzOzKudkb2ZmVuWc7M3MzKqck72ZmVmVc7I3MzOrcr6CnnV5vh66NfDlcs0a5569mZlZlXOyXwMkLa1w+6sk1UuaK2mmpHMkle29lTRa0la559dJGlCu9s3MrLI8jF8dlkVELYCkzYFbgY2BH5fagKRuEbGqic2jgTnAawARcXK7ojUzszXKPfsOIunzkh6WNCv97ZfWHynpaUnPSnpI0hZp/VhJN0iaLOkVSWc01m5EvAGcCpyuzGhJV+X2e6+k4Wl5qaSfSHoaGCLpvyU9I2mOpHGp/kigANySRg96phgKqY3jJc1OdS7O7WeppAvTSMNTDa/DzMzWPCf7jnMVcHNEDARuAa5I6x8D9omIPYDfA/+Zq7MT8EVgMPBjSd0bazgiXiF7bzdvIYYNgTkRsXdEPAZcFRGDImJXoCdwRETcAdQBoyKiNiKWNVROQ/sXAwcBtcAgSSNybT8VEbsDU4BTincu6VRJdZLqFi9e3EKoZmbWVk72HWcI2XA7wG+B/dLyNsBESbOB84BdcnXui4gVEfEm8AbQXG9ZJcSwCrgz9/zANKowmyyB79J4tY8NAiZHxOKIWEn2pWVY2vYBcG9ang7UFFeOiHERUYiIQt++fUsI18zM2sLJvvOI9PdKsh72bsC3gB65Mityy6toYs6FpG3T9jeAlaz+PufbW95wnl5SD+BqYGTa97VFZRvdVTPbPoyIhtfUZKxmZlZ5TvYd5wnguLQ8imz4HrKJdX9Pyye2tlFJfYFryL4wBLAAqJW0jqTPkZ0CaExDYn9TUi9gZG7bEqB3I3WeBg6QtJmkbsDxwKOtjdnMzCrLva01YwNJC3PPfwmcAdwg6TxgMfDvadtY4I+S/g48BfQvof2ekuqB7mQ9+d+mfQA8DvwVmE02o35GYw1ExNuSrk3lFgDP5DaPB66RtIzs9ENDnUWSvgc8QtbLvz8i/lRCvGZmtgbpk5FWs45TKBSirq5tVz/zFfSsga+gZ2sbSdMjosX/BN2zty7P/8GbmTXP5+zNzMyqnJO9mZlZlXOyNzMzq3JO9mZmZlXOyd7MzKzKOdmbmZlVOSd7MzOzKudkb2ZmVuWc7M3MzKqck72ZmVmV8+Vyzaxq+D4JtqZ0tct0u2dvZmZW5aom2UtaJak+9xhTpnYXSNqsHG2Vg6Q+kr6Te14jaZmkZyU9J2mapBPLvM+zJG2Qe36/pD7l3IeZmVVONQ3jL4uI2o4OYg3oA3wHuDq3bn5E7AEgaVvgLknrRMSNpTQoSWS3O/6oiSJnAb8D3geIiMPbGryZma15VdOzb0rqmf9M0pOS6iTtKWmipPmSTktlhkuaIuluSfMkXSPpU8dG0jmS5qTHWWndTyWdmStzoaQzUpuPSvqDpBclXSRpVOp5z5a0XSrfV9Kdkp5Jj33T+rGSbpA0WdIrks5Iu7gI2C6NXlxSHGNEvAKcA5yRa+fcXHxz0mhATRoJuBqYAXxO0q/TMZor6fxU/gxgK+ARSY/kjulmzRyThravTW1NktSzPe+jmZm1XTUl+55Fw/jH5rb9LSKGAFOB8cBIYB/gJ7kyg4H/B+wGbAd8Nd+4pL2Afwf2TnVPkbQHcD1wYiqzDnAccEuqtjtwZmrzG8AOETEYuA74bipzOXBZRAwCjk7bGuwEfDHF9mNJ3YExZD352og4r4ljMSPVbcmOwM0RsUdEvAr8ICIKwEDgAEkDI+IK4DXgwIg4sMRjArA98KuI2AV4O702iuqfmr5c1C1evLiEcM3MrC3WlmH8CenvbKBXRCwBlkhanjv3PC31ipF0G7AfcEeujf2AuyPivVTmLmD/iLhC0lspyW0BPBsRb2Uj4zwTEYtS+fnApFwcDYnzEGBAKg+wkaTeafm+iFgBrJD0Rmq/FGq5CACvRsRTuedfk3Qq2ediS2AAMKuZ+o0eE7Lj/deIqE/lpgM1xZUjYhwwDqBQKESJMZuZWStVU7Jvzor096PccsPzhmNQnGyKnzeXQK8DRgOfBW5oZL/F+87vdx1gSEQsW21nWfLP119F6e/XHsBzaXklq4/g9Mgtv5fbX3/gXGBQRPxT0viiso1p7pgUx+5hfDOzDlJNw/jtNVhS/zQUfyzwWNH2KcAISRtI2hA4iuy0AMDdwL8Ag4CJrdzvJOD0hieSWppkuATo3dRGSTXApcCVadUCYM+0bU+gfxNVNyJL/u9I2gL4Ugn7bO6YmJlZJ1FNPfuekupzzx+IiNb8/O5Jsslvu5ElsbvzGyNiRurtTkurrouIZ9O2D9LktbcjYlUr4z4D+JWkWWTvxxTgtKYKp1MEj0uaA/wZ+BXZhL1nyXriS4ArczPx7wT+LR2bZ4AXm2h3ZmpjLvAK8Hhu8zjgz5IW5c/bN3VM0hcOMzPrJBThU6WShgPnRsQRbay/DtmkuGMi4qVyxra2KBQKUVfXta5IZZ2Pr6Bna0pnuYKepOlpYnWzqqln3yEkDQDuJZuo5kRv1oE6y3/AZp2Nkz0QEZOByW2sOw/YtpzxmJmZlZMn6JmZmVU5J3szM7Mq52RvZmZW5ZzszczMqpyTvZmZWZVzsjczM6tyTvZmZmZVzsnezMysyvmiOmZWNXy5XOuK1sSVH92zNzMzq3JO9hUgaRtJf5L0kqT5ki6XtJ6kWkmH58qNlXRuhWI4VdLz6TFN0n4l1Bkt6apKxGNmZh3Hyb7MJAm4C7gnIrYHdgB6ARcCtcDhzVRv7b66NbH+COBbwH4RsRPZLXNvlfTZcu3bzMy6Dif78jsIWN5wP/l0f/uzgZOBXwDHSqqXdGwqP0DSZEmvSDqjoRFJJ6Qeeb2k3zQkdklLJf1E0tPAEEkXSZonaZakS1P1/wLOi4g3UwwzgJuA/0htLJB0vqQZkmZL2qm5FyTp+FRujqSLc+t/LalO0lxJ5+fWt6p9MzOrLCf78tsFmJ5fERHvAguAC4DbI6I2Im5Pm3cCvggMBn4sqbuknYFjgX0johZYBYxK5TcE5kTE3sA84Chgl4gYmNpvNAagLq1v8GZE7An8GmjyVIKkrYCLyb7E1AKDJI1Im3+Q7qM8EDhA0sDWtm9mZpXnZF9+AqIV6++LiBWpF/4GsAVwMLAX8Iyk+vS84Ta6q4A70/K7wHLgOklfBd5vRVx3pb/TgZpm6g0CJkfE4ohYCdwCDEvbviZpBvAs2ReJAa1pP80rqJNUt3jx4mZCMDOz9nCyL7+5wGq//5G0EfA5skRdbEVueRXZzyEF3JRGAGojYseIGJvKLE+nBkjJdzBZ8h8BPJDKzCP7spC3Z1pfvN+GfTZFja6U+pP12A9Oowr3AT1a035EjIuIQkQU+vbt20wIZmbWHk725fcwsIGkf4OPJ9H9DzAeeB3oXWIbIyVtntrYRNLniwtJ6gVsHBH3A2eRDbNDNjfgYkmbpnK1wGjg6ja8nqfJhug3S6/leOBRYCPgPeAdSVsAX2pD22Zmtgb4ojplFhEh6Sjgakk/IvtCdT/wfbLz7WPS0PzPm2ljnqQfApMkrQN8SDa57tWior2BP0nqQdYDPzvVnyBpa+AJSQEsAU6IiEUlvITRuXPyAPsA3wMeSfu4PyL+BCDpWbKRjFeAx0to28zMOoAiGjuNbLZmFQqFqKur/FWkrLr5CnrWFbXnCnqSpqeJ0s1yz97MqsaauOyoWVfkc/ZmZmZVzsnezMysyjnZm5mZVTknezMzsyrnZG9mZlbl/NM76xQkLebT1xFojc2AN8sUzprQleJ1rJXTleJ1rJXTnng/HxEtXoLUyd6qgqS6Un5r2ll0pXgda+V0pXgda+WsiXg9jG9mZlblnOzNzMyqnJO9VYtxHR1AK3WleB1r5XSleB1r5VQ8Xp+zNzMzq3Lu2ZuZmVU5J3vrMiRtIulBSS+lv59potwDkt6WdG/R+vGS/iqpPj1qO3Gs/SU9nerfLmm9SsXaynhPTGVeknRibv1kSS/kju3mFYjxX9I+XpY0ppHt66dj9XI6djW5bd9L61+Q9MVyx1auWCXVSFqWO47XVDrWEuMdJmmGpJWSRhZta/Qz0UljXZU7thM6QaznSJonaZakhyV9PretvMc1Ivzwo0s8gF8AY9LyGODiJsodDBwJ3Fu0fjwwsovE+gfguLR8DfDtjo4X2AR4Jf39TFr+TNo2GShUML5uwHxgW2A9YCYwoKjMd4Br0vJxwO1peUAqvz7QP7XTrZPGWgPMWROf0VbGWwMMBG7O/xtq7jPR2WJN25Z2suN6ILBBWv527nNQ9uPqnr11JV8BbkrLNwEjGisUEQ8DS9ZUUE1oc6ySBBwE3NFS/TIqJd4vAg9GxD8i4p/Ag8C/VDiuBoOBlyPilYj4APg9Wcx5+ddwB3BwOpZfAX4fESsi4q/Ay6m9zhhrR2gx3ohYEBGzgI+K6q7pz0R7Yl3TSon1kYh4Pz19CtgmLZf9uDrZW1eyRUQsAkh/2zJUfGEaMrtM0vrlDW817Yl1U+DtiFiZni8Eti5zfMVKiXdr4G+558Vx3ZiGR39UgcTV0r5XK5OO3Ttkx7KUuuXUnlgB+kt6VtKjkvavYJyfiiVpzfHpjMe2OT0k1Ul6SlKlv0C3NtaTgD+3sW6L1m1PZbNyk/QQ8NlGNv2gDM1/D/g/siG1ccB/AT9pa2MVjLWxRNnun82UId7m4hoVEX+X1Bu4E/gG2TBquZRyTJoqU5Hj2Yz2xLoI6BcRb0naC7hH0i4R8W65gywhlkrXbYv27q9fRLwmaVvgL5JmR8T8MsVWrORYJZ0AFIADWlu3VE721qlExCFNbZP0uqQtI2KRpC2BN1rZ9qK0uELSjcC57Qi1krG+CfSRtG7q9W0DvNaeWKEs8S4Ehueeb0N2rp6I+Hv6u0TSrWRDmOVM9guBzxXtu/iYNJRZKGldYGPgHyXWLac2xxrZCdsVABExXdJ8YAegroPjba7u8KK6k8sSVdP7a/N7GRGvpb+vSJoM7EF2Xr0SSopV0iFkX7gPiIgVubrDi+pObk8wHsa3rmQC0DAr9UTgT62pnJJYwznxEcCcska3ujbHmv7DfwRomEnc6tfaBqXEOxE4TNJn0mz9w4CJktaVtBmApO7AEZT/2D4DbK/sVwrrkU1qK55NnX8NI4G/pGM5ATguzYDvD2wPTCtzfGWJVVJfSd0AUu9ze7LJWZVUSrxNafQzUaE4oR2xphjXT8ubAfsC8yoWaQmxStoD+A3w5YjIf8Eu/3FdUzMT/fCjvQ+yc5oPAy+lv5uk9QXguly5qcBiYBnZN+QvpvV/AWaTJaLfAb06cazbkiWkl4E/Aut3kmP7zRTTy8C/p3UbAtOBWcBc4HIqMNsdOBx4kawn9oO07idk/1EC9EjH6uV07LbN1f1BqvcC8KU18FltU6zA0ekYzgRmAEdWOtYS4x2UPp/vAW8Bc5v7THTGWIGh6d//zPT3pE4Q60PA60B9ekyo1HH1FfTMzMyqnIfxzczMqpyTvZmZWZVzsjczM6tyTvZmZmZVzsnezMysyjnZm5mZVTknezMzsyrnZG9mZlbl/n9ee/grYSrDwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f636a022f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "explanation_result = pd.DataFrame.from_dict(explain_run['entity']['predictions'][0]['explanation_features'])\n",
    "explanation_result.plot.barh(x='feature_name', y='weight', color='g', alpha=0.8);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift monitor configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drift requires a trained model to be uploaded manually for Azure. You can train, create and download a drift detection model using template given ( check for Drift detection model generation) [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/training_statistics_notebook.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-01 18:47:33--  https://github.com/pmservice/ai-openscale-tutorials/blob/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz?raw=true\n",
      "Resolving github.com (github.com)... 140.82.112.3\n",
      "Connecting to github.com (github.com)|140.82.112.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github.com/pmservice/ai-openscale-tutorials/raw/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz [following]\n",
      "--2020-04-01 18:47:33--  https://github.com/pmservice/ai-openscale-tutorials/raw/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz [following]\n",
      "--2020-04-01 18:47:33--  https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 229416 (224K) [application/octet-stream]\n",
      "Saving to: ‘creditrisk_azure_drift_detection_model.tar.gz’\n",
      "\n",
      "100%[======================================>] 229,416     --.-K/s   in 0.02s   \n",
      "\n",
      "2020-04-01 18:47:34 (13.4 MB/s) - ‘creditrisk_azure_drift_detection_model.tar.gz’ saved [229416/229416]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -rf creditrisk_azure_drift_detection_model.tar.gz\n",
    "!wget -O creditrisk_azure_drift_detection_model.tar.gz https://github.com/pmservice/ai-openscale-tutorials/blob/master/assets/models/creditrisk_azure_drift_detection_model.tar.gz?raw=true "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'config_status': {'state': 'finished'},\n",
       " 'data_drift_enabled': True,\n",
       " 'drift_threshold': 0.1,\n",
       " 'is_schedule_enabled': True,\n",
       " 'min_samples': 5,\n",
       " 'model_drift_enabled': True,\n",
       " 'next_scheduled_run_timestamp': '2020-04-01T18:49:38.709431Z',\n",
       " 'schedule_repeat_interval': 3,\n",
       " 'schedule_repeat_type': 'hour'}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.drift_monitoring.enable(threshold=0.1, min_records=5,model_path=\"creditrisk_azure_drift_detection_model.tar.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Drift monitor on demand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==========================================\n",
      "\n",
      " Waiting for end of drift monitoring run  \n",
      "\n",
      "==========================================\n",
      "\n",
      "\n",
      "\n",
      "RUNNING\n",
      "COMPLETED\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drift_run_details = subscription.drift_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ts</th>\n",
       "      <th>id</th>\n",
       "      <th>measurement_id</th>\n",
       "      <th>value</th>\n",
       "      <th>lower limit</th>\n",
       "      <th>upper limit</th>\n",
       "      <th>tags</th>\n",
       "      <th>binding_id</th>\n",
       "      <th>subscription_id</th>\n",
       "      <th>deployment_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-04-01 18:47:51.233402+00:00</td>\n",
       "      <td>data_drift_magnitude</td>\n",
       "      <td>f181bb7a-359c-4efe-88b1-23e660cdd0eb</td>\n",
       "      <td>0.000</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>6ac4d3c4-862f-4702-8053-9f531466686a</td>\n",
       "      <td>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td>\n",
       "      <td>563f01d37f720857b95c557dc76176ad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-04-01 18:47:51.233402+00:00</td>\n",
       "      <td>drift_magnitude</td>\n",
       "      <td>f181bb7a-359c-4efe-88b1-23e660cdd0eb</td>\n",
       "      <td>0.413</td>\n",
       "      <td></td>\n",
       "      <td>0.1</td>\n",
       "      <td></td>\n",
       "      <td>6ac4d3c4-862f-4702-8053-9f531466686a</td>\n",
       "      <td>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td>\n",
       "      <td>563f01d37f720857b95c557dc76176ad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-04-01 18:47:51.233402+00:00</td>\n",
       "      <td>predicted_accuracy</td>\n",
       "      <td>f181bb7a-359c-4efe-88b1-23e660cdd0eb</td>\n",
       "      <td>0.500</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>6ac4d3c4-862f-4702-8053-9f531466686a</td>\n",
       "      <td>7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f</td>\n",
       "      <td>563f01d37f720857b95c557dc76176ad</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                ts                    id  \\\n",
       "0 2020-04-01 18:47:51.233402+00:00  data_drift_magnitude   \n",
       "1 2020-04-01 18:47:51.233402+00:00       drift_magnitude   \n",
       "2 2020-04-01 18:47:51.233402+00:00    predicted_accuracy   \n",
       "\n",
       "                         measurement_id  value lower limit upper limit tags  \\\n",
       "0  f181bb7a-359c-4efe-88b1-23e660cdd0eb  0.000                                \n",
       "1  f181bb7a-359c-4efe-88b1-23e660cdd0eb  0.413                     0.1        \n",
       "2  f181bb7a-359c-4efe-88b1-23e660cdd0eb  0.500                                \n",
       "\n",
       "                             binding_id                       subscription_id  \\\n",
       "0  6ac4d3c4-862f-4702-8053-9f531466686a  7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f   \n",
       "1  6ac4d3c4-862f-4702-8053-9f531466686a  7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f   \n",
       "2  6ac4d3c4-862f-4702-8053-9f531466686a  7c3a3f6c-b11c-4d38-a7ca-a179227f5f9f   \n",
       "\n",
       "                      deployment_id  \n",
       "0  563f01d37f720857b95c557dc76176ad  \n",
       "1  563f01d37f720857b95c557dc76176ad  \n",
       "2  563f01d37f720857b95c557dc76176ad  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.drift_monitoring.get_table_content()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Payload analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run data distributions calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===============================================================================\n",
      "\n",
      " Waiting for end of data distribution run 04e01728-bf9d-4e69-aa45-8f50d27af8aa \n",
      "\n",
      "===============================================================================\n",
      "\n",
      "\n",
      "\n",
      "completed\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "start_date = \"2018-01-01T00:00:00.00Z\"\n",
    "end_date = datetime.utcnow().isoformat() + \"Z\"\n",
    "\n",
    "sex_distribution = subscription.payload_logging.data_distribution.run(\n",
    "            start_date=start_date,\n",
    "            end_date=end_date,\n",
    "            group=['Scored Labels', 'Sex'],\n",
    "            agg=['count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get data distributions as pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sex_distribution_run_uid = sex_distribution['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Scored Labels</th>\n",
       "      <th>Sex</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>No Risk</td>\n",
       "      <td>female</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No Risk</td>\n",
       "      <td>male</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Scored Labels     Sex  count\n",
       "0       No Risk  female      9\n",
       "1       No Risk    male      9"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributions_pd = subscription.payload_logging.data_distribution.get_run_result(run_id=sex_distribution_run_uid, format='pandas')\n",
    "distributions_pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAElCAYAAADjk4nIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFzhJREFUeJzt3X+QXWWd5/H3BwhEIIBAcIAASSkLiJEftjDAjMUPBWQU1NI1LM6GRSuiWEIN6jIqyurozNZErRJGmDAwUQRkWSRmFBR0YZFCzQ/ygygggY3axpIQNJAokMh3/+gbq2lu/0jfm3TCeb+qbvU5z3nuc76dgs8997nPPZ2qQpLUHNuNdQGSpC3L4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGmaHsS6gnb333rsmT5481mVI0jZj4cKFT1TVxJH03SqDf/LkySxYsGCsy5CkbUaSX4y0r1M9ktQwBr8kNYzBL0kNs1XO8UsSwPr16+nt7eWZZ54Z61K2GuPHj2fSpEmMGzdu1GMY/JK2Wr29vUyYMIHJkyeTZKzLGXNVxerVq+nt7WXKlCmjHsepHklbrWeeeYa99trL0G9Jwl577dXxOyCDX9JWzdB/oW78exj8ktQwzvG/1F22+1hXIA3usjWb1H3yJd/p6ulX/NPfDNvny1/+MldeeSVHH300119/fVfPD3DZZZex66678pGPfKTrYw/G4JekIXzlK1/h9ttv7+jD1K2NwS9Jgzj//PN57LHHOPPMM5k2bRqPPvooDzzwABs2bOCyyy7jrLPOYvbs2cyZM4c//elPLFu2jIsvvpjnnnuO6667jp122onbbruNPffck6uvvppZs2bx3HPP8apXvYrrrruOnXfe+QXne/TRR7ngggtYtWoVO++8M1dffTWHHnpo13+vYef4k1yb5PEky/q13ZRkceuxIsniQZ67IskDrX7efEfSNuWqq65iv/3246677mLdunWcfPLJzJ8/n7vuuouPfvSjrFu3DoBly5Zxww03MG/ePD7xiU+w8847s2jRIo477ji+9rWvAfCOd7yD+fPns2TJEg477DCuueaaF51vxowZXH755SxcuJCZM2fywQ9+cLP8XiO54p8NXAF8bWNDVb1743aSLwBDTdSdVFVPjLZASdoa3HHHHcydO5eZM2cCfUtNf/nLXwJw0kknMWHCBCZMmMDuu+/OW9/6VgCmTp3K0qVLgb4Xh09+8pP8/ve/Z+3atZx22mkvGH/t2rXcd999vOtd7/pz27PPPrtZfpdhg7+q7kkyud2x9K0r+s/Ayd0tS5K2LlXFLbfcwiGHHPKC9p/85CfstNNOf97fbrvt/ry/3XbbsWHDBgDOPfdc5syZwxFHHMHs2bO5++67XzDO888/zx577MHixW0nULqq0+Wcfw38tqoeGeR4AXckWZhkxlADJZmRZEGSBatWreqwLEnqrtNOO43LL7+cqgJg0aJFm/T8p59+mn333Zf169e3XR202267MWXKFG6++Wag74VmyZIlnRfeRqcf7p4N3DjE8ROqamWSfYA7kzxUVfe061hVs4BZAD09PdVhXZJegkay/HJzufTSS7nooot47WtfS1UxefJkvv3tb4/4+Z/97Gc59thjOeigg5g6dSpPP/30i/pcf/31fOADH+Af/uEfWL9+PdOmTeOII47o5q8BQDa+eg3ZqW+q59tV9Zp+bTsAvwZeV1W9IxjjMmBtVc0crm9PT0/5h1i6xHX82poNs47/wQcf5LDDDttCxWw72v27JFlYVT0jeX4nUz1vBB4aLPST7JJkwsZt4FRgWbu+kqQtZyTLOW8EfgQckqQ3yXtbh6YxYJonyX5JbmvtvgK4N8kSYB7wnar6bvdKlySNxkhW9Zw9SPu5bdpWAme0th8Duj85JUnqiDdpk6SGMfglqWEMfklqGG/SJmnb0e3lyZt4W+hNdffddzNz5sxNWu+/JXjFL0kNY/BL0hBWrFjBoYceyvve9z5e85rXcM455/D973+fE044gYMPPph58+Yxb948jj/+eI466iiOP/54Hn744ReNs27dOs477zxe//rXc9RRR/Gtb31rDH6bPga/JA1j+fLlXHjhhSxdupSHHnqIG264gXvvvZeZM2fy+c9/nkMPPZR77rmHRYsW8ZnPfIaPf/zjLxrjc5/73KC3dd7SnOOXpGFMmTKFqVOnAnD44YdzyimnkISpU6eyYsUK1qxZw/Tp03nkkUdIwvr16180xmC3dR6LW1IY/JI0jOFuu3zppZdy0kknceutt7JixQpOPPHEF40x2G2dx4JTPZLUoTVr1rD//vsDMHv27LZ9Or2tczd5xS9p27GZl1+O1sc+9jGmT5/OF7/4RU4+uf3fper0ts7dNKLbMm9p3pa5i7wts7Zm3pZ5VMbytsySpG2QwS9JDWPwS9qqbY3T0WOpG/8eBr+krdb48eNZvXq14d9SVaxevZrx48d3NI6reiRttSZNmkRvby+rVq0a61K2GuPHj2fSpEkdjWHwS9pqjRs3jilTpox1GS85TvVIUsMY/JLUMMMGf5JrkzyeZFm/tsuS/DrJ4tbjjEGee3qSh5MsT3JJNwuXJI3OSK74ZwOnt2n/UlUd2XrcNvBgku2BfwHeDLwaODvJqzspVpLUuWGDv6ruAZ4cxdjHAMur6rGqeg74BnDWKMaRJHVRJ3P8H0qytDUV9PI2x/cHftVvv7fV1laSGUkWJFng0i1J2nxGG/xXAq8EjgR+A3yhTZ+0aRv0WxhVNauqeqqqZ+LEiaMsS5I0nFEFf1X9tqr+VFXPA1fTN60zUC9wQL/9ScDK0ZxPktQ9owr+JPv22307sKxNt/nAwUmmJNkRmAbMHc35JEndM+w3d5PcCJwI7J2kF/g0cGKSI+mbulkBvL/Vdz/g36rqjKrakORDwPeA7YFrq+qnm+W3kCSN2LDBX1Vnt2m+ZpC+K4Ez+u3fBrxoqackaez4zV1JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGGTb4k1yb5PEky/q1/XOSh5IsTXJrkj0Gee6KJA8kWZxkQTcLlySNzkiu+GcDpw9ouxN4TVW9Fvg58PdDPP+kqjqyqnpGV6IkqZuGDf6qugd4ckDbHVW1obX7Y2DSZqhNkrQZdGOO/zzg9kGOFXBHkoVJZnThXJKkDu3QyZOTfALYAFw/SJcTqmplkn2AO5M81HoH0W6sGcAMgAMPPLCTsiRJQxj1FX+S6cBbgHOqqtr1qaqVrZ+PA7cCxww2XlXNqqqequqZOHHiaMuSJA1jVMGf5HTgvwNnVtUfBumzS5IJG7eBU4Fl7fpKkrackSznvBH4EXBIkt4k7wWuACbQN32zOMlVrb77Jbmt9dRXAPcmWQLMA75TVd/dLL+FJGnEhp3jr6qz2zRfM0jflcAZre3HgCM6qk6S1HV+c1eSGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JapgRBX+Sa5M8nmRZv7Y9k9yZ5JHWz5cP8tzprT6PJJnercIlSaMz0iv+2cDpA9ouAX5QVQcDP2jtv0CSPYFPA8cCxwCfHuwFQpK0ZYwo+KvqHuDJAc1nAV9tbX8VeFubp54G3FlVT1bV74A7efELiCRpC+pkjv8VVfUbgNbPfdr02R/4Vb/93lbbiySZkWRBkgWrVq3qoCxJ0lA294e7adNW7TpW1ayq6qmqnokTJ27msiSpuToJ/t8m2Reg9fPxNn16gQP67U8CVnZwTklShzoJ/rnAxlU604FvtenzPeDUJC9vfah7aqtNkjRGRrqc80bgR8AhSXqTvBf4J+BNSR4B3tTaJ0lPkn8DqKongc8C81uPz7TaJEljZIeRdKqqswc5dEqbvguA9/Xbvxa4dlTVSZK6zm/uSlLDGPyS1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwI1rOqW3X5GduGOsSpEGtGOsCGsorfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqmFEHf5JDkizu93gqyUUD+pyYZE2/Pp/qvGRJUidGfVvmqnoYOBIgyfbAr4Fb23T9YVW9ZbTnkSR1V7emek4BHq2qX3RpPEnSZtKt4J8G3DjIseOSLElye5LDBxsgyYwkC5IsWLVqVZfKkiQN1HHwJ9kROBO4uc3h+4GDquoI4HJgzmDjVNWsquqpqp6JEyd2WpYkaRDduOJ/M3B/Vf124IGqeqqq1ra2bwPGJdm7C+eUJI1SN4L/bAaZ5knyF0nS2j6mdb7VXTinJGmUOvpj60l2Bt4EvL9f2/kAVXUV8E7gA0k2AH8EplVVdXJOSVJnOgr+qvoDsNeAtqv6bV8BXNHJOSRJ3eU3dyWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqGINfkhrG4Jekhuk4+JOsSPJAksVJFrQ5niRfTrI8ydIkR3d6TknS6O3QpXFOqqonBjn2ZuDg1uNY4MrWT0nSGNgSUz1nAV+rPj8G9kiy7xY4rySpjW4EfwF3JFmYZEab4/sDv+q339tqe4EkM5IsSLJg1apVXShLktRON4L/hKo6mr4pnQuSvGHA8bR5Tr2ooWpWVfVUVc/EiRO7UJYkqZ2Og7+qVrZ+Pg7cChwzoEsvcEC//UnAyk7PK0kanY6CP8kuSSZs3AZOBZYN6DYX+K+t1T1/Caypqt90cl5J0uh1uqrnFcCtSTaOdUNVfTfJ+QBVdRVwG3AGsBz4A/DfOjynJKkDHQV/VT0GHNGm/ap+2wVc0Ml5JEnd4zd3JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGmbUwZ/kgCR3JXkwyU+TXNimz4lJ1iRZ3Hp8qrNyJUmd6uSPrW8ALq6q+5NMABYmubOqfjag3w+r6i0dnEeS1EWjvuKvqt9U1f2t7aeBB4H9u1WYJGnz6Mocf5LJwFHAT9ocPi7JkiS3Jzm8G+eTJI1eJ1M9ACTZFbgFuKiqnhpw+H7goKpam+QMYA5w8CDjzABmABx44IGdliVJGkRHV/xJxtEX+tdX1TcHHq+qp6pqbWv7NmBckr3bjVVVs6qqp6p6Jk6c2ElZkqQhdLKqJ8A1wINV9cVB+vxFqx9Jjmmdb/VozylJ6lwnUz0nAH8LPJBkcavt48CBAFV1FfBO4ANJNgB/BKZVVXVwTklSh0Yd/FV1L5Bh+lwBXDHac0iSus9v7kpSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMAa/JDWMwS9JDWPwS1LDGPyS1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMAa/JDWMwS9JDdNR8Cc5PcnDSZYnuaTN8Z2S3NQ6/pMkkzs5nySpc6MO/iTbA/8CvBl4NXB2klcP6PZe4HdV9SrgS8D/HO35JEnd0ckV/zHA8qp6rKqeA74BnDWgz1nAV1vb/xs4JUk6OKckqUM7dPDc/YFf9dvvBY4drE9VbUiyBtgLeGLgYElmADNau2uTPNxBbdLmsjdt/vvV6MQ5gG46aKQdOwn+dlfuNYo+fY1Vs4BZHdQjbXZJFlRVz1jXIXWik6meXuCAfvuTgJWD9UmyA7A78GQH55QkdaiT4J8PHJxkSpIdgWnA3AF95gLTW9vvBP5PVbW94pckbRmjnuppzdl/CPgesD1wbVX9NMlngAVVNRe4BrguyXL6rvSndaNoaQw5HaltXrwAl6Rm8Zu7ktQwBr8kNYzBL0kNY/BLQ0iyU5u2PceiFqlbDH5paN9MMm7jTpJ9gTvHsB6pYwa/NLQ5wM1Jtm/dXfZ7wN+PaUVSh1zOKQ0jyQXA6cBk4P1Vdd/YViR1xuCX2kjyd/13gb8FHgAWAVTVF8eiLqkbOrlJm/RSNmHA/q2DtEvbHK/4pRFKsh2wa1U9Nda1SJ3ww11pCEluSLJbkl2AnwEPJ/noWNcldcLgl4b26tYV/tuA24AD6Zvvl7ZZBr80tHGtdfxvA75VVesZ5I8JSdsKg18a2r8CK4BdgHuSHAQ4x69tmh/uSpsoyQ5VtWGs65BGy+WcUhtJ3lNVXx+wnr8/1/Frm2XwS+3t0vrpun295DjVI22iJLtU1bqxrkMaLT/clQaRZP8kPUl2bO3vk+TzwCNjXJrUEYNfaiPJRcBi4HLgx0mmAw8CLwNeN5a1SZ1yqkdqI8nPgL+qqieTHAgsB95QVT8e49KkjnnFL7X3TFU9CVBVvwR+bujrpcJVPVJ7k5J8ud/+Pv33q+rDY1CT1BUGv9TewBuxLRyTKqTNwDl+SWoY5/glqWEMfklqGINfkhrG4JeGkGRSkluTrEry2yS3JJk01nVJnTD4paH9OzAX2BfYH/iPVpu0zXJVjzSEJIur6sjh2qRtiVf80tCeSPKeJNu3Hu8BVo91UVInvOKXhtC6T88VwHH0/a3d+4ALq+oXY1qY1AGDX5Iaxls2SG0k+dQQh6uqPrvFipG6zCt+qY0kF7dp3gV4L7BXVe26hUuSusbgl4aRZAJwIX2h/7+AL1TV42NblTR6TvVIg0iyJ/B3wDnAV4Gjq+p3Y1uV1DmDX2ojyT8D7wBmAVOrau0YlyR1jVM9UhtJngeeBTbQt4zzz4fo+3B3tzEpTOoCg1+SGsZv7kpSwxj8ktQwBr8kNYzBr21Ckk8k+WmSpUkWJzl2C577siQfGWn7EONs0sqgTR1fGimXc2qrl+Q44C30raN/NsnewI4djrlDVW3oSoHSNsYrfm0L9gWeqKpnAarqiapaCZDk9UnuS7IkybwkE5KMT/LvSR5IsijJSa2+5ya5Ocl/AHe02j6aZH7rncT/2HjC1juMh5N8HzhkU4pNMifJwtY7lBkDjn0hyf1JfpBkYqvtlUm+23rOD5Mc2mbMDyf5WavOb2zSv540gFf82hbcAXwqyc+B7wM3VdX/TbIjcBPw7qqan2Q34I/03V6BqpraCtE7kvyn1ljHAa+tqieTnAocDBxD3/r8uUneAKwDpgFH0ff/yP3Awk2o97zW+C8D5ie5papW03evn/ur6uLWTeA+DXyIvi+JnV9Vj7SmsL4CnDxgzEuAKa13PHtsQi3Sixj82upV1dokrwP+GjgJuCnJJfSF8W+qan6r31MASf4KuLzV9lCSXwAbg//OqnqytX1q67Gotb8rfS8EE4Bbq+oPrfHmbmLJH07y9tb2Aa0xVwPP0/dCBfB14JtJdgWOB25OsvH5O7UZcylwfZI5wJxNrEd6AYNf24Sq+hNwN3B3kgeA6fRdibf7BmLatG20bkC/f6yqf33Bk5OLBhl3WElOBN4IHFdVf0hyNzB+kO5F33Tr70fwpxz/BngDcCZwaZLD/YxCo+Ucv7Z6SQ5JcnC/piOBXwAPAfsleX2r34QkOwD30HdjNVpTPAcCD7cZ+nvAea2rbpLsn2Sf1vPfnuRlrTtzvnUTyt0d+F0r9A8F/rLfse2Ad7a2/wtwb+tdyv9L8q5WDUlyxIDffzvggKq6C/gYsAd9706kUfGKX9uCXYHLW3PbG4DlwIyqei7Ju1vHXkbf/P4b6Zsjv6r1zmADcG5rbvwFg1bVHUkOA37UOrYWeE9V3Z/kJmAxfS8wPxyitk+23iFs9Erg/CRL6Xux+XG/Y+uAw5MsBNYA7261nwNcmeSTwDjgG8CSfs/bHvh6kt3pe5fypar6/TD/ZtKgvFePJDWMUz2S1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0kN8/8BXtqVqD+KGjYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.payload_logging.data_distribution.show_chart(sex_distribution_run_uid);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have finished the tutorial for IBM Watson OpenScale and Azure Machine Learning Studio. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authors\n",
    "Lukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n",
    "\n",
    "Rounok Salehin,Data Scientist and Cognitive Developer at IBM with experties in enterprise-level machine learning/AI applications development emphasize on turning data into descriptive/actionable outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}