notebooks/Watson OpenScale and Watson ML Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Watson Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook should be run in a Watson Studio project, using **Default Spark Python** runtime environment. **If you are viewing this in Watson Studio and do not see Python 3.6 with Spark in the upper right corner of your screen, please update the runtime now.** It requires service credentials for the following Cloud services:\n",
    "  * Watson OpenScale\n",
    "  * Watson Machine Learning\n",
    "  \n",
    "If you have a paid Cloud account, you may also provision a **Databases for PostgreSQL** or **Db2 Warehouse** service to take full advantage of integration with Watson Studio and continuous learning services. If you choose not to provision this paid service, you can use the free internal PostgreSQL storage with OpenScale, but will not be able to configure continuous learning for your model.\n",
    "\n",
    "The notebook will train, create and deploy a German Credit Risk model, configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contents\n",
    "\n",
    "- [Setup](#setup)\n",
    "- [Model building and deployment](#model)\n",
    "- [OpenScale configuration](#openscale)\n",
    "- [Quality monitor and feedback logging](#quality)\n",
    "- [Fairness, drift monitoring and explanations](#fairness)\n",
    "- [Custom monitors and metrics](#custom)\n",
    "- [Payload analytics](#analytics)\n",
    "- [Business application](#application)\n",
    "- [Historical data](#historical)\n",
    "- [Run Business Application](#run_application)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup <a name=\"setup\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20200110100850-0000\n",
      "KERNEL_ID = b9f4a2ed-5612-493d-94e9-23c0f3f61726\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from pyspark.sql import SparkSession\n",
    "except:\n",
    "    print('Error: Spark runtime is missing. If you are using Watson Studio change the notebook runtime to Spark.')\n",
    "    raise "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Package installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[31mibm-cos-sdk-core 2.4.3 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.7 which is incompatible.\u001b[0m\n",
      "\u001b[31mbotocore 1.12.82 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.7 which is incompatible.\u001b[0m\n",
      "Successfully installed certifi-2019.11.28 chardet-3.0.4 h5py-2.10.0 ibm-ai-openscale-2.1.20 idna-2.8 numpy-1.18.1 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3 requests-2.22.0 six-1.13.0 tabulate-0.8.6 urllib3-1.25.7\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[31mbotocore 1.12.82 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.7 which is incompatible.\u001b[0m\n",
      "Successfully installed certifi-2019.11.28 chardet-3.0.4 docutils-0.15.2 ibm-cos-sdk-2.6.0 ibm-cos-sdk-core-2.6.0 ibm-cos-sdk-s3transfer-2.6.0 idna-2.8 jmespath-0.9.4 lomond-0.3.3 numpy-1.18.1 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3 requests-2.22.0 six-1.13.0 tabulate-0.8.6 tqdm-4.41.1 urllib3-1.25.7 watson-machine-learning-client-1.0.378\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy-1.18.1.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/bin already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "Successfully installed numpy-1.18.1\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy-1.18.1.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/bin already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "Successfully installed SciPy-1.4.1 numpy-1.18.1\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[31mbotocore 1.12.82 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.7 which is incompatible.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/__pycache__ already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/six.py already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/scipy already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/six-1.13.0.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/numpy-1.18.1.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/dateutil already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/scipy-1.4.1.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/python_dateutil-2.8.1.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3.6/bin already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "Successfully installed PyWavelets-1.1.1 cycler-0.10.0 decorator-4.4.1 imageio-2.6.1 joblib-0.14.1 kiwisolver-1.1.0 lime-0.1.1.37 matplotlib-3.1.2 networkx-2.4 numpy-1.18.1 pillow-7.0.0 progressbar-2.5 pyparsing-2.4.6 python-dateutil-2.8.1 scikit-image-0.16.2 scikit-learn-0.22.1 scipy-1.4.1 setuptools-44.0.0 six-1.13.0\n"
     ]
    }
   ],
   "source": [
    "!rm -rf /home/spark/shared/user-libs/python3.6*\n",
    "\n",
    "!pip install --upgrade ibm-ai-openscale --no-cache | tail -n 1\n",
    "!pip install --upgrade watson-machine-learning-client | tail -n 1\n",
    "!pip install numpy --no-cache | tail -n 1\n",
    "!pip install SciPy --no-cache | tail -n 1\n",
    "!pip install lime --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: ibm-ai-openscale\r\n",
      "Version: 2.1.20\r\n",
      "Summary: Python SDK for IBM Watson OpenScale\r\n",
      "Home-page: http://ai-openscale-python-client.mybluemix.net\r\n",
      "Author: IBM\r\n",
      "Author-email: lukasz.cmielowski@pl.ibm.com, maria.oleszkiewicz@pl.ibm.com, wojciech.sobala@pl.ibm.com\r\n",
      "License: BSD\r\n",
      "Location: /home/spark/shared/user-libs/python3.6\r\n",
      "Requires: pandas, requests, tabulate, h5py\r\n",
      "Required-by: \r\n"
     ]
    }
   ],
   "source": [
    "!pip show ibm-ai-openscale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provision services and configure credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** You can also get OpenScale `API_KEY` using IBM CLOUD CLI.\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "How to get api key using console:\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = \"PASTE HERE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next you will need credentials for Watson Machine Learning. If you already have a WML instance, you may use credentials for it. To provision a new Lite instance of WML, use the [Cloud catalog](https://cloud.ibm.com/catalog/services/machine-learning), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your WML credentials into the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "WML_CREDENTIALS = {\n",
    "    \"apikey\": \"key\",\n",
    "    \"iam_apikey_description\": \"description\",\n",
    "    \"iam_apikey_name\": \"auto-generated-apikey\",\n",
    "    \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Writer\",\n",
    "    \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::\",\n",
    "    \"instance_id\": \"instance_id\",\n",
    "    \"url\": \"https://us-south.ml.cloud.ibm.com\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial can use Databases for PostgreSQL, Db2 Warehouse, or a free internal verison of PostgreSQL to create a datamart for OpenScale.\n",
    "\n",
    "If you have previously configured OpenScale, it will use your existing datamart, and not interfere with any models you are currently monitoring. Do not update the cell below.\n",
    "\n",
    "If you do not have a paid Cloud account or would prefer not to provision this paid service, you may use the free internal PostgreSQL service with OpenScale. Do not update the cell below.\n",
    "\n",
    "To provision a new instance of Db2 Warehouse, locate [Db2 Warehouse in the Cloud catalog](https://cloud.ibm.com/catalog/services/db2-warehouse), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Db2 Warehouse credentials into the cell below.\n",
    "\n",
    "To provision a new instance of Databases for PostgreSQL, locate [Databases for PostgreSQL in the Cloud catalog](https://cloud.ibm.com/catalog/services/databases-for-postgresql), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Databases for PostgreSQL credentials into the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DB_CREDENTIALS = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__If you previously configured OpenScale to use the free internal version of PostgreSQL, you can switch to a new datamart using a paid database service.__ If you would like to delete the internal PostgreSQL configuration and create a new one using service credentials supplied in the cell above, set the __KEEP_MY_INTERNAL_POSTGRES__ variable below to __False__ below. In this case, the notebook will remove your existing internal PostgreSQL datamart and create a new one with the supplied credentials. __*NO DATA MIGRATION WILL OCCUR.*__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEEP_MY_INTERNAL_POSTGRES = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the notebook\n",
    "\n",
    "At this point, the notebook is ready to run. You can either run the cells one at a time, or click the **Kernel** option above and select **Restart and Run All** to run all the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model building and deployment <a name=\"model\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you will learn how to train Spark MLLib model and next deploy it as web-service using Watson Machine Learning service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the training data from github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 674K Jan 10 10:28 german_credit_data_biased_training.csv\r\n"
     ]
    }
   ],
   "source": [
    "from IPython.utils import io\n",
    "\n",
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv -O german_credit_data_biased_training.csv\n",
    "!ls -lh german_credit_data_biased_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(CheckingStatus='0_to_200', LoanDuration=31, CreditHistory='credits_paid_to_date', LoanPurpose='other', LoanAmount=1889, ExistingSavings='100_to_500', EmploymentDuration='less_1', InstallmentPercent=3, Sex='female', OthersOnLoan='none', CurrentResidenceDuration=3, OwnsProperty='savings_insurance', Age=32, InstallmentPlans='none', Housing='own', ExistingCreditsCount=1, Job='skilled', Dependents=1, Telephone='none', ForeignWorker='yes', Risk='No Risk')"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "import pandas as pd\n",
    "import json\n",
    "import datetime\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n",
    "df_data = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- CheckingStatus: string (nullable = true)\n",
      " |-- LoanDuration: integer (nullable = true)\n",
      " |-- CreditHistory: string (nullable = true)\n",
      " |-- LoanPurpose: string (nullable = true)\n",
      " |-- LoanAmount: integer (nullable = true)\n",
      " |-- ExistingSavings: string (nullable = true)\n",
      " |-- EmploymentDuration: string (nullable = true)\n",
      " |-- InstallmentPercent: integer (nullable = true)\n",
      " |-- Sex: string (nullable = true)\n",
      " |-- OthersOnLoan: string (nullable = true)\n",
      " |-- CurrentResidenceDuration: integer (nullable = true)\n",
      " |-- OwnsProperty: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- InstallmentPlans: string (nullable = true)\n",
      " |-- Housing: string (nullable = true)\n",
      " |-- ExistingCreditsCount: integer (nullable = true)\n",
      " |-- Job: string (nullable = true)\n",
      " |-- Dependents: integer (nullable = true)\n",
      " |-- Telephone: string (nullable = true)\n",
      " |-- ForeignWorker: string (nullable = true)\n",
      " |-- Risk: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records: 5000\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of records: \" + str(df_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records for training: 4016\n",
      "Number of records for evaluation: 984\n"
     ]
    }
   ],
   "source": [
    "spark_df = df_data\n",
    "(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n",
    "\n",
    "MODEL_NAME = \"Spark German Risk Model - Final\"\n",
    "DEPLOYMENT_NAME = \"Spark German Risk Deployment - Final\"\n",
    "\n",
    "print(\"Number of records for training: \" + str(train_data.count()))\n",
    "print(\"Number of records for evaluation: \" + str(test_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below creates a Random Forest Classifier with Spark, setting up string indexers for the categorical features and the label column. Finally, this notebook creates a pipeline including the indexers and the model, and does an initial Area Under ROC evaluation of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml import Pipeline, Model\n",
    "from pyspark.ml.feature import SQLTransformer\n",
    "\n",
    "features = [x for x in spark_df.columns if x != 'Risk']\n",
    "categorical_features = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']\n",
    "categorical_num_features = [x + '_IX' for x in categorical_features]\n",
    "si_list = [StringIndexer(inputCol=x, outputCol=y) for x, y in zip(categorical_features, categorical_num_features)]\n",
    "va_features = VectorAssembler(inputCols=categorical_num_features + [x for x in features if x not in categorical_features], outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "si_label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_label.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "\n",
    "classifier = RandomForestClassifier(featuresCol=\"features\")\n",
    "feature_filter = SQLTransformer(statement=\"SELECT * FROM __THIS__\")\n",
    "pipeline = Pipeline(stages= si_list + [si_label, va_features, classifier, label_converter, feature_filter])\n",
    "model = pipeline.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If you want filter features from model output please replace **`*`** with feature names to be retained in **`SQLTransformer`** statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "areaUnderROC = 0.697506\n"
     ]
    }
   ],
   "source": [
    "predictions = model.transform(test_data)\n",
    "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "area_under_curve = evaluatorDT.evaluate(predictions)\n",
    "\n",
    "print(\"areaUnderROC = %g\" % area_under_curve)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publish the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, the notebook uses the supplied Watson Machine Learning credentials to save the model (including the pipeline) to the WML instance. Previous versions of the model are removed so that the notebook can be run again, resetting all data for another demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from watson_machine_learning_client import WatsonMachineLearningAPIClient\n",
    "import json\n",
    "\n",
    "wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing model and deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----  ----  -------  ---------\n",
      "GUID  NAME  CREATED  FRAMEWORK\n",
      "----  ----  -------  ---------\n"
     ]
    }
   ],
   "source": [
    "model_deployment_ids = wml_client.deployments.get_uids()\n",
    "for deployment_id in model_deployment_ids:\n",
    "    deployment = wml_client.deployments.get_details(deployment_id)\n",
    "    model_id = deployment['entity']['deployable_asset']['guid']\n",
    "    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n",
    "        print('Deleting deployment id', deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print('Deleting model id', model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 14K Jan 10 10:28 credit-risk-meta.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/credit-risk/meta/credit-risk-meta.json -O credit-risk-meta.json\n",
    "!ls -lh credit-risk-meta.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('credit-risk-meta.json') as f:\n",
    "    [training_data_reference, *_] = json.load(f)['model_meta']['training_data_reference']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'connection': {'db': 'BLUDB', 'host': 'dashdb-txn-sbox-yp-dal09-03.services.dal.bluemix.net', 'password': 'khhz72v+6mcwwkfv', 'username': 'cmb91569'}, 'name': 'German credit risk training data', 'source': {'tablename': 'CREDIT_RISK_TRAIN_DATA', 'type': 'db2'}}\n"
     ]
    }
   ],
   "source": [
    "print(training_data_reference)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# COS training data reference example format\n",
    "# Get connection info from COS credentials (url: endpoint from COS credentials)\n",
    "# Only if using training data reference from COS, uncomment this section\n",
    "\n",
    "\n",
    "# training_data_reference = {\n",
    "#         \"type\": \"cos\",\n",
    "#         \"location\": {\n",
    "#             \"bucket\": \"***\",\n",
    "#             \"file_name\": \"***\",\n",
    "#             \"firstlineheader\": True,\n",
    "#             \"file_format\": 'csv'\n",
    "#         },\n",
    "#         'connection': {\n",
    "#             'resource_instance_id': \"***\",\n",
    "#             'url': \"***\", \n",
    "#             'api_key': \"***\"\n",
    "#         },\n",
    "#         'name': 'COS training data reference.'\n",
    "#     }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_props = {\n",
    "    wml_client.repository.ModelMetaNames.NAME: \"{}\".format(MODEL_NAME),\n",
    "    wml_client.repository.ModelMetaNames.EVALUATION_METHOD: \"binary\",\n",
    "    wml_client.repository.ModelMetaNames.TRAINING_DATA_REFERENCE: training_data_reference,\n",
    "    wml_client.repository.ModelMetaNames.EVALUATION_METRICS: [\n",
    "        {\n",
    "           \"name\": \"areaUnderROC\",\n",
    "           \"value\": area_under_curve,\n",
    "           \"threshold\": 0.7\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storing model ...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "wml_models = wml_client.repository.get_details()\n",
    "model_uid = None\n",
    "for model_in in wml_models['models']['resources']:\n",
    "    if MODEL_NAME == model_in['entity']['name']:\n",
    "        model_uid = model_in['metadata']['guid']\n",
    "        break\n",
    "\n",
    "if model_uid is None:\n",
    "    print(\"Storing model ...\")\n",
    "\n",
    "    published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props, training_data=train_data, pipeline=pipeline)\n",
    "    model_uid = wml_client.repository.get_model_uid(published_model_details)\n",
    "    print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ac325cbf-f50b-40e2-bb08-f1c8f085c1b2'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_uid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next section of the notebook deploys the model as a RESTful web service in Watson Machine Learning. The deployed model will have a scoring URL you can use to send data to the model for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deploying model...\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: 'ac325cbf-f50b-40e2-bb08-f1c8f085c1b2' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "INITIALIZING\n",
      "DEPLOY_SUCCESS\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='a58d369a-f7c8-4cb4-a9f1-68a6183114c8'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Model id: ac325cbf-f50b-40e2-bb08-f1c8f085c1b2\n",
      "Deployment id: a58d369a-f7c8-4cb4-a9f1-68a6183114c8\n"
     ]
    }
   ],
   "source": [
    "wml_deployments = wml_client.deployments.get_details()\n",
    "deployment_uid = None\n",
    "for deployment in wml_deployments['resources']:\n",
    "    if DEPLOYMENT_NAME == deployment['entity']['name']:\n",
    "        deployment_uid = deployment['metadata']['guid']\n",
    "        break\n",
    "\n",
    "if deployment_uid is None:\n",
    "    print(\"Deploying model...\")\n",
    "\n",
    "    deployment = wml_client.deployments.create(artifact_uid=model_uid, name=DEPLOYMENT_NAME, asynchronous=False)\n",
    "    deployment_uid = wml_client.deployments.get_uid(deployment)\n",
    "    \n",
    "print(\"Model id: {}\".format(model_uid))\n",
    "print(\"Deployment id: {}\".format(deployment_uid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure OpenScale <a name=\"openscale\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook will now import the necessary libraries and set up a Python OpenScale client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale import APIClient\n",
    "from ibm_ai_openscale.engines import *\n",
    "from ibm_ai_openscale.utils import *\n",
    "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n",
    "from ibm_ai_openscale.supporting_classes.enums import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Watson OpenScale GUID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each instance of OpenScale has a unique ID. We can get this value using the Cloud API key specified at the beginning of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cacaf1bc-1285-46da-ba9a-49464bc683a3\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from ibm_ai_openscale.utils import get_instance_guid\n",
    "\n",
    "\n",
    "WOS_GUID = get_instance_guid(api_key=CLOUD_API_KEY)\n",
    "WOS_CREDENTIALS = {\n",
    "    \"instance_guid\": WOS_GUID,\n",
    "    \"apikey\": CLOUD_API_KEY,\n",
    "    \"url\": \"https://api.aiopenscale.cloud.ibm.com\"\n",
    "}\n",
    "\n",
    "if WOS_GUID is None:\n",
    "    print('Watson OpenScale GUID NOT FOUND')\n",
    "else:\n",
    "    print(WOS_GUID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Watson OpenScale GUID has not been found you need provide the GUID to variabe `WOS_GUID`.\n",
    "\n",
    "#### Get OpenScale `instance_guid` \n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "\n",
    "\n",
    "How to get your OpenScale instance GUID\n",
    "- log in to IBM Cloud:\n",
    "```\n",
    "bx login --sso\n",
    "```\n",
    "- if your resource group is different than `default` switch to resource group containing OpenScale instance\n",
    "```\n",
    "bx target -g <myResourceGroup>\n",
    "```\n",
    "- get details of the instance\n",
    "```\n",
    "bx resource service-instance 'AI-OpenScale-instance_name'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.20'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ai_client = APIClient(aios_credentials=WOS_CREDENTIALS)\n",
    "ai_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create schema and datamart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up datamart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were **not** supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there **unless** there is an existing datamart **and** the **KEEP_MY_INTERNAL_POSTGRES** variable is set to **True**. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.\n",
    "\n",
    "Prior instances of the German Credit model will be removed from OpenScale monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using existing internal datamart.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    data_mart_details = ai_client.data_mart.get_details()\n",
    "    if 'internal_database' in data_mart_details and data_mart_details['internal_database']:\n",
    "        if KEEP_MY_INTERNAL_POSTGRES:\n",
    "            print('Using existing internal datamart.')\n",
    "        else:\n",
    "            if DB_CREDENTIALS is None:\n",
    "                print('No postgres credentials supplied. Using existing internal datamart')\n",
    "            else:\n",
    "                print('Switching to external datamart')\n",
    "                ai_client.data_mart.delete(force=True)\n",
    "                ai_client.data_mart.setup(db_credentials=DB_CREDENTIALS)\n",
    "    else:\n",
    "        print('Using existing external datamart')\n",
    "except:\n",
    "    if DB_CREDENTIALS is None:\n",
    "        print('Setting up internal datamart')\n",
    "        ai_client.data_mart.setup(internal_db=True)\n",
    "    else:\n",
    "        print('Setting up external datamart')\n",
    "        try:\n",
    "            ai_client.data_mart.setup(db_credentials=DB_CREDENTIALS)\n",
    "        except:\n",
    "            print('Setup failed, trying Db2 setup')\n",
    "            ai_client.data_mart.setup(db_credentials=DB_CREDENTIALS, schema=DB_CREDENTIALS['username'])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_mart_details = ai_client.data_mart.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bind machine learning engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If this binding already exists, this code will output a warning message and use the existing binding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can bind more than one engine instance if needed by calling `ai_client.data_mart.bindings.add` method. Next, you can refer to particular binding using `binding_uid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Service bindings</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>service_type</th><th style='border: 1px solid #dddddd'>created</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>WML Instance 4 Tutorial</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2020-01-10T10:29:06.989Z</td></tr><tr><td style='border: 1px solid #dddddd'>becebffa-9309-4349-8d0e-297099ebc91e</td><td style='border: 1px solid #dddddd'>WML Cloud Instance</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2020-01-10T10:20:41.019Z</td></tr><tr><td style='border: 1px solid #dddddd'>2fa0701a-b6af-49b1-a46b-e5dea58897a9</td><td style='border: 1px solid #dddddd'>My custom engine</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2019-12-06T11:22:59.439Z</td></tr><tr><td style='border: 1px solid #dddddd'>909f6985-4c29-4aa8-a889-d9f8017e1f58</td><td style='border: 1px solid #dddddd'>My Custom deployment</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2019-10-21T09:29:13.972Z</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "binding_uid = ai_client.data_mart.bindings.add('WML Instance 4 Tutorial', WatsonMachineLearningInstance(WML_CREDENTIALS))\n",
    "bindings_details = ai_client.data_mart.bindings.get_details()\n",
    "\n",
    "if binding_uid is None:\n",
    "    binding_uid = [binding['metadata']['guid'] for binding in bindings_details['service_bindings'] if binding['entity']['name']=='WML Cloud Instance'][0]\n",
    "\n",
    "ai_client.data_mart.bindings.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "409c1c90-3666-4ef3-8186-1b376c0f2c22\n"
     ]
    }
   ],
   "source": [
    "print(binding_uid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Available assets (binding_uid='409c1c90-3666-4ef3-8186-1b376c0f2c22')</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>source_uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>created</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>frameworks</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>is_subscribed</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>ac325cbf-f50b-40e2-bb08-f1c8f085c1b2</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>2020-01-10T10:28:57.481Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>mllib-2.3</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>False</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ai_client.data_mart.bindings.list_assets(binding_uid=binding_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing credit risk subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code removes previous subscriptions to the German Credit model to refresh the monitors with the new model and new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing subscription for Spark German Risk Model - Final\n"
     ]
    }
   ],
   "source": [
    "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "for subscription in subscriptions_uids:\n",
    "    sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n",
    "    if sub_name == MODEL_NAME:\n",
    "        ai_client.data_mart.subscriptions.delete(subscription)\n",
    "        print('Deleted existing subscription for', MODEL_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset(\n",
    "    model_uid,\n",
    "    problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "    input_data_type=InputDataType.STRUCTURED,\n",
    "    label_column='Risk',\n",
    "    prediction_column='predictedLabel',\n",
    "    probability_column='probability',\n",
    "    transaction_id_column='transaction_id',\n",
    "    feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "    categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n",
    "))\n",
    "\n",
    "if subscription is None:\n",
    "    print('Subscription already exists; get the existing one')\n",
    "    subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "    for sub in subscriptions_uids:\n",
    "        if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:\n",
    "            subscription = ai_client.data_mart.subscriptions.get(sub)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get subscription list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>created</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>2020-01-10T10:29:18.367Z</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "ai_client.data_mart.subscriptions.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription_details = subscription.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'8a8842d9-5b8a-42ec-baca-5cd54dde47df'"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.uid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the model so we can configure monitors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a58d369a-f7c8-4cb4-a9f1-68a6183114c8\n",
      "https://us-south.ml.cloud.ibm.com/v3/wml_instances/409c1c90-3666-4ef3-8186-1b376c0f2c22/deployments/a58d369a-f7c8-4cb4-a9f1-68a6183114c8/online\n"
     ]
    }
   ],
   "source": [
    "credit_risk_scoring_endpoint = None\n",
    "print(deployment_uid)\n",
    "\n",
    "for deployment in wml_client.deployments.get_details()['resources']:\n",
    "    if deployment_uid in deployment['metadata']['guid']:\n",
    "        credit_risk_scoring_endpoint = deployment['entity']['scoring_url']\n",
    "        \n",
    "print(credit_risk_scoring_endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Single record scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'LoanPurpose_IX', 'ExistingSavings_IX', 'EmploymentDuration_IX', 'Sex_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'InstallmentPlans_IX', 'Housing_IX', 'Job_IX', 'Telephone_IX', 'ForeignWorker_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, [20, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 46.0, 2.0, 1.0]], [13.441885143125948, 6.558114856874052], [0.6720942571562973, 0.3279057428437026], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}\n",
    "scoring_response = wml_client.deployments.score(credit_risk_scoring_endpoint, payload_scoring)\n",
    "\n",
    "print('Single record scoring result:', '\\n fields:', scoring_response['fields'], '\\n values: ', scoring_response['values'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "subscription.payload_logging.get_records_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quality monitoring and feedback logging <a name=\"quality\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable quality monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n",
    "\n",
    "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "time.sleep(10)\n",
    "subscription.quality_monitoring.enable(threshold=0.7, min_records=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedback logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 17K Jan 10 10:29 additional_feedback_data.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data.json -O additional_feedback_data.json\n",
    "!ls -lh additional_feedback_data.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('additional_feedback_data.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)\n",
    "    \n",
    "subscription.feedback_logging.store(additional_feedback_data['data'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Feedback_8a8842d9-5b8a-42ec-baca-5cd54dde47df (binding_id=409c1c90-3666-4ef3-8186-1b376c0f2c22, subscription_id=8a8842d9-5b8a-42ec-baca-5cd54dde47df)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>CheckingStatus</th><th style='border: 1px solid #dddddd'>LoanDuration</th><th style='border: 1px solid #dddddd'>CreditHistory</th><th style='border: 1px solid #dddddd'>LoanPurpose</th><th style='border: 1px solid #dddddd'>LoanAmount</th><th style='border: 1px solid #dddddd'>ExistingSavings</th><th style='border: 1px solid #dddddd'>EmploymentDuration</th><th style='border: 1px solid #dddddd'>InstallmentPercent</th><th style='border: 1px solid #dddddd'>Sex</th><th style='border: 1px solid #dddddd'>OthersOnLoan</th><th style='border: 1px solid #dddddd'>CurrentResidenceDuration</th><th style='border: 1px solid #dddddd'>OwnsProperty</th><th style='border: 1px solid #dddddd'>Age</th><th style='border: 1px solid #dddddd'>InstallmentPlans</th><th style='border: 1px solid #dddddd'>Housing</th><th style='border: 1px solid #dddddd'>ExistingCreditsCount</th><th style='border: 1px solid #dddddd'>Job</th><th style='border: 1px solid #dddddd'>Dependents</th><th style='border: 1px solid #dddddd'>Telephone</th><th style='border: 1px solid #dddddd'>ForeignWorker</th><th style='border: 1px solid #dddddd'>Risk</th><th style='border: 1px solid #dddddd'>record_timestamp</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>10</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>250</td><td style='border: 1px solid #dddddd'>500_to_1000</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>real_estate</td><td style='border: 1px solid #dddddd'>23</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>23</td><td style='border: 1px solid #dddddd'>prior_payments_delayed</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>6964</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>39</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>30</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>3464</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>greater_7</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>guarantor</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>51</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>23</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>car_used</td><td style='border: 1px solid #dddddd'>2681</td><td style='border: 1px solid #dddddd'>500_to_1000</td><td style='border: 1px solid #dddddd'>greater_7</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>unskilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>18</td><td style='border: 1px solid #dddddd'>prior_payments_delayed</td><td style='border: 1px solid #dddddd'>furniture</td><td style='border: 1px solid #dddddd'>1673</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>1_to_4</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>30</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>44</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>3476</td><td style='border: 1px solid #dddddd'>unknown</td><td style='border: 1px solid #dddddd'>greater_7</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>unknown</td><td style='border: 1px solid #dddddd'>60</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>8</td><td style='border: 1px solid #dddddd'>no_credits</td><td style='border: 1px solid #dddddd'>education</td><td style='border: 1px solid #dddddd'>803</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>unemployed</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>19</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>7</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>250</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>unemployed</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>real_estate</td><td style='border: 1px solid #dddddd'>19</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>3548</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>1_to_4</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>28</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr><tr><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>24</td><td style='border: 1px solid #dddddd'>prior_payments_delayed</td><td style='border: 1px solid #dddddd'>retraining</td><td style='border: 1px solid #dddddd'>4158</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>greater_7</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>35</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>unskilled</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>2020-01-10 10:29:46.715645+00:00</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.feedback_logging.show_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "================================================================================\n",
      "\n",
      " Waiting for end of quality monitoring run 8e1e0633-1829-490c-9bff-19f9f5514cae \n",
      "\n",
      "================================================================================\n",
      "\n",
      "\n",
      "\n",
      "initializing\n",
      "completed\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "run_details = subscription.quality_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>QualityMetrics (binding_id=409c1c90-3666-4ef3-8186-1b376c0f2c22, subscription_id=8a8842d9-5b8a-42ec-baca-5cd54dde47df)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower limit</th><th style='border: 1px solid #dddddd'>upper limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.3939393939393939</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.6585081585081586</td><td style='border: 1px solid #dddddd'>0.7</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.7222222222222222</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.5098039215686274</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.7448979591836735</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.4318227213152725</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.07692307692307693</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>area_under_pr</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.605407819693534</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:29:51.146000+00:00</td><td style='border: 1px solid #dddddd'>recall</td><td style='border: 1px solid #dddddd'>fe61429f-e807-47e7-ae06-5e4df6c5b896</td><td style='border: 1px solid #dddddd'>0.3939393939393939</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "subscription.quality_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAD4CAYAAACKcG2KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de3hV1Z3/8fcHjFAF0aKOGVGDlooCIhAvUC9YtfUZrZcKpWpb0E4Zq5Vq6+9X5mc7jzPaVtvp1ctYdCydllYFtTrSFqmCF6pCwv0itSN0ClJEq1G0UAjf3x9nRY/hJDk7JOck5PN6nvNk77XWXuu7dwLfrL12zlFEYGZmZsXrVu4AzMzMOhsnTzMzs4ycPM3MzDJy8jQzM8vIydPMzCyjPcodgJXG/vvvH1VVVeUOw8ys06itrX0lIg4oVOfk2UVUVVVRU1NT7jDMzDoNSX9sqs63bc3MzDJy8jQzM8vIydPMzCwjr3mamRkA27ZtY926dWzZsqXcoZRUz5496devHxUVFUUf4+TZRSxbX0fV5JnlDsOsoLU3nV3uEAxYt24dvXv3pqqqCknlDqckIoJXX32VdevW0b9//6KP821bMzMDYMuWLfTt27fLJE4ASfTt2zfzbNvJ08zM3tGVEmeD1pyzk2cnJKlK0vK0PVrSI+WOycysK/GaZwkp9+uNImJHuWMxM2tJWz8n0dZr27169WLz5s1t2mexPPNsZ2mWuFrSfwHLgU9LekbSQknTJfVK7Y6T9DtJSyTNl9Q7HftUartQ0qjyno2ZmYGTZ6kMAG4HTgU+C5wREcOBGuBLkvYE7gW+GBFDgTOAvwIvA2emtuOAH2YZVNJESTWSaurfrmu7szEzaweTJ0/mtttue2f/+uuv58Ybb+T0009n+PDhDBkyhIceemin4+bOncs555zzzv4XvvAFpk6dCkBtbS2nnnoqI0aM4KMf/SgbNmxok1idPEvjjxHxLHAicDQwT9JiYDxwGHAksCEiFgBExBsRsR2oAO6UtAyYno4tWkRMiYjqiKjuvlefNjwdM7O2N27cOO6777539u+77z7Gjx/Pgw8+yMKFC5kzZw5f/vKXiYii+tu2bRtXXXUVM2bMoLa2lssuu4zrrruuTWL1mmdpvJW+CpgdERflV0oa0sRx1wAbgaHkftHpWn+5bGZdyrBhw3j55Zd56aWX2LRpE/vttx8HHXQQ11xzDU8++STdunVj/fr1bNy4kYMOOqjF/lavXs3y5cs588wzAaivr6eysrJNYnXyLK1ngdskfSAi/iBpb+BgYDVQKem4iFggqTe527Z9gHURsUPSeKB7+UI3M2t/Y8eOZcaMGfz5z39m3LhxTJs2jU2bNlFbW0tFRQVVVVU7/U3mHnvswY4d7z6H2VAfEQwaNIhnnnmmzeP0bdsSiohNwATgF5KWAs8AAyPib+TWNG+RtASYDfQkt046PpUN5N0ZrJnZbmncuHHcc889zJgxg7Fjx1JXV8eBBx5IRUUFc+bM4Y9/3PlTwg477DBWrlzJ1q1bef3113nssccAOPLII9m0adM7yXPbtm2sWLGiTeL0zLOdRcRaYHDe/uPAcQXaLSC3JprvBeCYvP2vNO4zIuYCc9suYjOznHK8beKgQYN48803Ofjgg6msrOSSSy7hYx/7GEOGDKG6upqBAwfudMwhhxzCJz7xCQYPHkz//v0ZNmwYAHvuuSczZsxg0qRJ1NXVsX37dq6++moGDRq0y3Gq2IVX69yqq6vDH4ZtZs1ZtWoVRx11VLnDKItC5y6pNiKqC7X3bVszM7OMnDzNzMwycvI0M7N3dMWlvNacs5OnmZkBuQ+FfvXVV7tUAm34PM+ePXtmOs5P25qZGQD9+vVj3bp1bNq0qdyhlFTPnj3p169fpmOcPM3MDICKigr69+9f7jA6Bd+2NTMzy8jJ08zMLCMnTzMzs4ycPM3MzDJy8jQzM8vIydPMzCwj/6lKF7FsfR1Vk2eWOwwzK7NyfFLK7sgzTzMzs4ycPM3MzDJy8mwnkkZLeqTccZiZWdvb7ZKnpO7ljqE1JO3S+nNnPW8zs86o0yVPSb+UVCtphaSJqWyzpO9IWgKMlDRC0hOp3SxJland5yQtkLRE0v2S9mpmnKmSxuTtb05fR0uaK2mGpOclTZOkVHdWKlsIfDzv2L0l3S1pvqRFks5L5RMkPSzpceCxJuIYLelJSTMlrZZ0h6Ruhc57166smZkVq9MlT+CyiBgBVAOTJPUF9gaei4ihwHPALcCY1O5u4Ovp2Aci4rjUbhXw2VbGMAy4GjgaOBz4kKSewJ3Ax4ARwEF57a8DHo+I44HTgG9L2jvVDU+xntrMeMcDV6XxjuDdxPzOeUfE040PkjRRUo2kmvq361p5qmZm1lhn/FOVSZIuSNuHAAOAeuD+VHYkMBiYnSaE3YENqW6wpBuBfYFewKxWxjA/ItYBSFoMVAGbgTUR8UIq/xkwMbX/CHCupGvTfk/g0LQ9OyL+UsR4L6Z+fwGcBMzgvee9k4iYAkwB6FE5oOt8QJ+ZWTvrVMlT0mjgDGBkRLwtaS65RLQlIuobmgErIqLQbcypwPkRsUTSBGB0M8NtJ83M023SPfPqtuZt19PydRRwYUSsbnQ+JwBvtXAsQOPE17Cff95mZlYine22bR/gtZQ4BwInFmizGjhA0kgASRWSBqW63sAGSRXAJS2MtZbc7VeAc4GKFto/D1RJOiLtX5RXNwu4Km9tdFgLfTV2vKT+KYmPA3a6RWtmZqXT2ZLnb4A9JK0CbgKebdwgIv4GjAFuTg/SLAZGpeqvkVsTnUcu2TXnTuDUvIdxmp0hRsQWcrdpZ6YHhl7Oq76BXPJdKmlF2s9iAXAruXXaNcCDGY83M7M2pAgvhXVk6Vb1tRFxzq7006NyQFSO/37bBGVmnZbfnq94kmojorpQXada87TWG3JwH2r8j8bMrE10+eQp6TpgbKPi6RHx9ULt2zGOIcBPGxVvjYgTgLmljMXMzJrX5ZNnSpIlTZRNxLEMOLbccZiZWcs62wNDZmZmZefkaWZmlpGTp5mZWUZOnmZmZhk5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll1OXfYairWLa+jqrJM8sag9+Q2sx2F555mpmZZeTkaWZmlpGTp5mZWUbtljwlTZK0StK0JupHS3qkvcZvDUmXS/pM2p4g6e/z6u6SdHQZYnpPHGZmVn7t+cDQFcAZEbGuHcdoUxFxR97uBGA58FKq+8f2GldS94iob6L6PXGYmVn5tcvMU9IdwOHAryV9RdIzkhZJ+p2kIwu0P1XS4vRaJKl3Kv8/khZIWirpX5sZr0rS85KmpdnuDEl7pbrTU5/LJN0tqUcqv0nSytT3v6ey6yVdK2kMUA1MSzG9T9JcSdVpdvrtvLEnSLo1bX9K0vx0zI8kdW8m5s2SviNpCTBS0r+kc10uaYpyCsUxQtITkmolzZJU2cwYEyXVSKqpf7uume+YmZll0S7JMyIuJzdTOg34D+DkiBgG/AvwjQKHXAtcGRHHAicDf5X0EWAAcDy5D4keIemUZoY9Erg9Io4C3gCukNQTmAqMi4gh5Gban5fUF7gAGBQRxwA3Nop/BlADXBIRx0bEX/Oq70/HNhgH3CPpqLT9oXQe9cAlzcS7N/BcRAyNiKeBWyPiuIgYDLwPOKdxHMB24BZgTESMAO6mmQ/yjogpEVEdEdXd9+rTTChmZpZFKR4Y6gNMl7Qc+B4wqECbecB3JU0C9o2I7cBH0msRsBAYSC6ZNuVPETEvbf8MOIlcQl0TEb9P5T8BTgHqgC3Af0r6OPB2sScTEZuAFyWdmJLwwBT/6cAIYIGkxWn/8Ga6qieXiBucJuk5ScuAD1P4Oh0JDAZmpzG+CvQrNnYzM2sbpXiThBuAORFxgaQqYG7jBhFxk6SZwD8A8yR9FBDwzYj4UZHjRAv7+eNtl3Q8uQQ3BvgCuYRVrHuATwDPAw9GREgS8JOI+Oci+9jSsM6ZZsi3A9UR8SdJ1wM9CxwjYEVEjMwQq5mZtbFSzTzXp+0JhRpIOiIilkXEzcACcrO5WcBlknqlNgdLOrCZcQ6V1JBULgaeBlYDVZI+kMo/DTyR+uwTEb8CrgGGFujvTaB3E2M9CJwHXEQukQI8BoxpiFHS+yUd1ky8+RoS5SsptjFNxLEaOKDhPCVVSCo0QzUzs3ZUiuT5LeCbkhbR9Ez36vSgzFJgG/DriHgU+DnwTLqVOYOmkxnkEsuVklYB+wH/ERFbgEvJ3TZeBuwA7kj9PJLGexr4UoH+pgJ3NDyok18REa8Bq4DDImJ+KltJ7jbqo6nf2UCTD/M06u914E5yT9XOIvcLxE5xAN3JJdab04NGi4FRxYxhZmZtRxFN3t3sNNLt4EfSwzZWQHV1ddTU1JQ7DDOzTkNSbURUF6rzOwyZmZll1Kk+VSU93fpYgarTO+qsU9JzQI9GxZ+OiGXliMfMzHZdp0qeEfEqub/57DQi4oRyx2BmZm3Lt23NzMwycvI0MzPLyMnTzMwsIydPMzOzjJw8zczMMnLyNDMzy8jJ08zMLCMnTzMzs4w61ZskWOstW19H1eSZ5Q7DrF2svenscodgXYxnnmZmZhk5eZqZmWXk5GlmZpaRk2dGkja3cX9zJRX8vDgzM+uYnDzNzMwycvJsJeV8W9JyScskjUvl3STdLul5SbMl/UrSmCL7vCj1tVzSzamsu6SpeeNck8onSVopaamke5rob6KkGkk19W/XtdWpm5l1ef5Tldb7OLnPFh0K7A8skPQk8CGgCjgaOBBYBdzdUmeS/h64GRgBvAY8Kul84E/AwQ0f9i1p33TIZKB/RGzNK3uPiJgCTAHoUTkgWneaZmbWmGeerXcS8IuIqI+IjcATwHGpfHpE7IiIPwNziuzvOGBuRGyKiO3ANOAU4EXgcEm3SDoLeCO1XwpMk/QpYHvbnZaZmbXEybODi4jXyM1u5wKXA3elqrOB24Dh5Ga9votgZlYiTp6t9xQwLq1JHkBuljgfmAdcmNY+/w4YXWR/84FTJe0vqTtwEfCEpP2BbhFxP/BVYLikbsAhETEH+ArQB+jVlidnZmZN82yl9R4ERgJLgAD+b0T8WdL9wOnASnLrlQuBFp/WiYgNkiaTu80rYGZEPCRpKPDjlDAB/hnoDvxMUp/U9ocR8Xrbnp6ZmTVFEX6OpK1J6hURmyX1JTej/FBa/yyb6urqqKmpKWcIZmadiqTaiCj4d/ieebaPR9ITsHsCN5Q7cZqZWdty8mwHETG6cZmkB4H+jYq/EhGzShKUmZm1GSfPEomIC8odg5mZtQ0/bWtmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaWkZOnmZlZRk6eZmZmGTl5mpmZZeQ3Segilq2vo2ryzHKHYWZWMmtvOrvd+vbM08zMLCMnTzMzs4ycPM3MzDJy8iwzSV53NjPrZJr9j1vSx5urj4gH2jacjkXSL4FDgJ7ADyJiiqSzgG8A3YFXIuJ0Sb2AW4BqIIB/jYj7JW2OiF6przHAORExQdJUYAswDJgn6R7gB2mcvwKXRsRqSd2Bm4GzgB3AncAKYFJEnJ/6PRO4wp/aYmZWOi3Nej6Wvh4IjAIeT/unAb8DduvkCVwWEX+R9D5ggaSHyCWwUyJijaT3p3ZfA+oiYgiApP2K6LsfMCoi6iXtA5wcEdslnUEuOV8ITASqgGNT3fuB14DbJR0QEZuAS4G7Cw0gaWLqg+77HNCqC2BmZjtrNnlGxKUAkh4Fjo6IDWm/Epja7tGV3yRJDTO6Q8gloicjYg1ARPwl1Z0BfLLhoIh4rYi+p0dEfdruA/xE0gByM9eKvH7viIjt+eNJ+inwKUk/BkYCnyk0QERMAaYA9KgcEEXEZGZmRSh2ve2QhsSZbAQObYd4OgxJo8klr5ER8bakucBiYGCGbvITVs9GdW/lbd8AzImICyRVAXNb6PfHwH+Tu/U7vSG5mplZaRT7wNBjkmZJmiBpAjAT+G37hdUh9AFeS4lzIHAiuQR4iqT+AHm3bWcDVzYcmHfbdqOkoyR1A5pbk+wDrE/bE/LKZwP/1PBQUcN4EfES8BLwVXKJ1MzMSqio5BkRXwB+BAxNrykRcVV7BtYB/AbYQ9Iq4CbgWWATuVu3D0haAtyb2t4I7CdpeSo/LZVPBh4htz6cP3Nv7FvANyUt4r13A+4C/hdYmvq9OK9uGvCniFi1C+doZmatoAgvhXVGkm4FFkXEfxbTvkflgKgc//12jsrMrOPY1bfnk1QbEdWF6lr6U5WnI+IkSW/y3vU7ARER++xSZNYqkmrJrZl+udhjhhzch5p2fJ9HM7OupKWnbU9KX3uXJhwrRkSMKHcMZmZdmd9hyMzMLCMnTzMzs4ycPM3MzDJy8jQzM8vIydPMzCwjJ08zM7OMnDzNzMwycvI0MzPLyMnTzMwsIydPMzOzjIr9PE/r5Jatr6Nq8sxyh2G2W9vVNyK3zsMzTzMzs4ycPM3MzDJy8jQzM8uoSyRPSZMkrZJ0v6RnJG2VdG254zIzs86pqzwwdAVwBvA34DDg/PKG0zqS9oiI7eWOw8ysq9vtZ56S7gAOB34NXBIRC4BtRRxXJel5SVMl/V7SNElnSJon6QVJx6d2e0u6W9J8SYsknZd3/FOSFqbXqFReKelJSYslLZd0cirfnDf2GElT0/ZUSXdIeg74VlPjNXEOEyXVSKqpf7uutZfQzMwa2e1nnhFxuaSzgNMi4pWMh38AGAtcBiwALgZOAs4F/h+5Gex1wOMRcZmkfYH5kn4LvAycGRFbJA0AfgFUpz5mRcTXJXUH9ioijn7AqIiol/SNQuNFxFsFzn0KMAWgR+WAyHjuZmbWhN0+ee6iNRGxDEDSCuCxiAhJy4Cq1OYjwLl5a6g9gUOBl4BbJR0L1AMfTPULgLslVQC/jIjFRcQxPSLqWxhvVWtP0szMsnHybN7WvO0defs7ePfaCbgwIlbnHyjpemAjMJTc7fEtABHxpKRTgLOBqZK+GxH/BeTPDHs2iiN/VllwPDMzK53dfs2zBGYBV0kSgKRhqbwPsCEidgCfBrqn+sOAjRFxJ3AXMDy13yjpKEndgAtaMZ6ZmZVIl0qekg6StA74EvBVSesk7bOL3d4AVABL063dG1L57cB4SUuAgbw7exwNLJG0CBgH/CCVTwYeAX4HbGjFeGZmViKK8HMkXUF1dXXU1NSUOwwzs05DUm1EVBeq61IzTzMzs7bQ5R8YktQXeKxA1ekR8Wqp4zEzs46vyyfPlCCPLXccZmbWefi2rZmZWUZOnmZmZhk5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaWUZd/k4SuYtn6Oqomzyx3GGZmbWLtTWeXdXzPPM3MzDJy8jQzM8vIybPEJP2uhfpfSdq3VPGYmVl2XvPcBZK6R0R9lmMiYlQL9f+wa1GZmVl788yzCZKqJD0vaZqkVZJmSNpL0lpJN0taCIyVdISk30iqlfSUpIHp+L+T9KCkJek1KpVvTl8rJT0pabGk5ZJOTuVrJe2ftr+U6pZLujovrlWS7pS0QtKjkt5XlotkZtZFOXk270jg9og4CngDuCKVvxoRwyPiHmAKcFVEjACuBW5PbX4IPBERQ4HhwIpGfV8MzIqIY4GhwOL8SkkjgEuBE4ATgc9JGpaqBwC3RcQg4HXgwkLBS5ooqUZSTf3bda27AmZmthPftm3enyJiXtr+GTApbd8LIKkXMAqYLqnhmB7p64eBzwCkW7uNs9cC4G5JFcAvI2Jxo/qTgAcj4q001gPAycDDwJq89rVAVaHgI2IKueROj8oBUdwpm5lZSzzzbF7jhNOw/1b62g14PSKOzXsdVVTHEU8CpwDrgamSPpMhrq152/X4lyAzs5Jy8mzeoZJGpu2LgafzKyPiDWCNpLEAyhmaqh8DPp/Ku0vqk3+spMOAjRFxJ3AXuVu7+Z4Czk/rrHsDF6QyMzMrMyfP5q0GrpS0CtgP+I8CbS4BPitpCbl1zfNS+ReB0yQtI3dr9ehGx40GlkhaBIwDfpBfGRELganAfOA54K6IWNQG52RmZrtIEV4KK0RSFfBIRAwucyhtokflgKgc//1yh2Fm1iZK8fZ8kmojorpQndfKuoghB/ehpszvBWlmtrtw8mxCRKwFdotZp5mZtS2veZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaWkZOnmZlZRk6eZmZmGTl5mpmZZeTkaWZmlpGTp5mZWUZ+b9suYtn6Oqomzyx3GGa2myrFp5x0JJ55mpmZZeTkaWZmlpGTZxuQNFrSI+WOw8zMSqNTJ09J3csdQ2tIanGtuZg2ZmZWHh06eUr6paRaSSskTUxlmyV9R9ISYKSkEZKeSO1mSapM7T4naYGkJZLul7RXM+NMlTQmb39z+jpa0lxJMyQ9L2maJKW6s1LZQuDjecfuLeluSfMlLZJ0XiqfIOlhSY8DjzURx2hJT0l6GFiZyr4kaXl6XZ3X9jOSlqbz+2lrr7GZmWXX0Wc3l0XEXyS9D1gg6X5gb+C5iPiypArgCeC8iNgkaRzwdeAy4IGIuBNA0o3AZ4FbWhHDMGAQ8BIwD/iQpBrgTuDDwB+Ae/PaXwc8HhGXSdoXmC/pt6luOHBMRPylmfGGA4MjYo2kEcClwAmAgOckPQH8DfgqMCoiXpH0/kIdpV84JgJ03+eAVpy6mZkV0tGT5yRJF6TtQ4ABQD1wfyo7EhgMzE4Twu7AhlQ3OCXNfYFewKxWxjA/ItYBSFoMVAGbgTUR8UIq/xkpSQEfAc6VdG3a7wkcmrZnt5A4G8Zbk7ZPAh6MiLfSOA8AJwMBTI+IVwCa6jMipgBTAHpUDoiiz9jMzJrVYZOnpNHAGcDIiHhb0lxyiWhLRNQ3NANWRMTIAl1MBc6PiCWSJgCjmxluO+kWtqRuwJ55dVvztutp+ZoJuDAiVjc6nxOAt1o4liLbmJlZGXXkNc8+wGspcQ4ETizQZjVwgKSRAJIqJA1Kdb2BDenW7iUtjLUWGJG2zwUqWmj/PFAl6Yi0f1Fe3Szgqry10WEt9NWcp4DzJe0laW/gglT2ODBWUt80RsHbtmZm1j46cvL8DbCHpFXATcCzjRtExN+AMcDN6QGixcCoVP014Dly65TPtzDWncCpDQ8h0cLsLyK2kLtNOzM9MPRyXvUN5JLvUkkr0n6rRMRCcjPo+eTO5a6IWBQRK8it7T6RYv5ua8cwM7PsFOGlsK6gR+WAqBz//XKHYWa7qd3x7fkk1UZEdaG6DrvmaW1ryMF9qNkNf7jNzMqhSyVPSdcBYxsVT4+Ir5c4jiFA47/N3BoRJ5QyDjMza50ulTxTkixpomwijmXAseWOw8zMWqcjPzBkZmbWITl5mpmZZeTkaWZmlpGTp5mZWUZOnmZmZhk5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZdal3GOrKlq2vo2ryzHKHYVbQ7vim4rZ788zTzMwsIydPMzOzjEqSPCXtK+mKUoy1qyT9Ln2tknRxXnm1pB+WIZ73xGFmZuVXqpnnvsBOyVNSh1tzjYhRabMKuDivvCYiJrXHmC1ch/fEYWZm5Veq5HkTcISkxZIWSHpK0sPAyjSzWt7QUNK1kq5P20dI+o2k2nTMwKYGkDRV0h2SaiT9XtI5qbynpB9LWiZpkaTTUvkgSfNTTEslDUjlm/NiPjnVXyNptKRHJHWTtFbSvnljvyDp7yQdIOn+dI4LJH2omXivl/RTSfOAn6br8JSkhenVkMQbx9Fd0rdT/0sl/VP2b4eZme2KUs38JgODI+JYSaOBmWl/jaSqZo6bAlweES9IOgG4HfhwM+2rgOOBI4A5kj4AXAlERAxJyfdRSR8ELgd+EBHTJO0JdC8Q87UR0ZCER5PraIekh4ALgB+nuP4YERsl/Rz4XkQ8LelQYBZwVDPxHg2cFBF/lbQXcGZEbEmJ/BdAdYE4JgJ1EXGcpB7APEmPRsSaxp2nthMBuu9zQDNhmJlZFuW6bTq/0H/2+ST1AkYB0yU1FPdood/7ImIH8IKkF4GBwEnALQAR8bykPwIfBJ4BrpPUD3ggIl7IEP+9wL8APwY+mfYBzgCOzot3H0m9ImLzzl0A8HBE/DVtVwC3SjoWqE8xFvIR4BhJY9J+H2AAsNP1jIgp5H4BoUflgCjy3MzMrAXlSp5v5W1v5723j3umr92A1yPi2Az9Nk4QTSaMiPi5pOeAs4FfSfqniHi8yHGeAT4g6QDgfODGvJhPjIgtRfaTfx2uATYCQ1M/TfUh4KqImFXkGGZm1sZKteb5JtC7ibqNwIGS+qbbkOcARMQbwBpJYwGUM7SFccamNckjgMOB1cBTwCWpjw8ChwKrJR0OvBgRPwQeAo4pNuaICOBB4LvAqoh4NVU9ClzV0C7NIovVB9iQZs6f5t3byI3jmAV8XlJFwzlJ2jvDOGZmtotKkjxTcpmXHgz6dqO6bcC/AfOB2cDzedWXAJ+VtARYAZzXwlD/m/r5Nbm10i3k1km7SVpG7vbqhIjYCnwCWC5pMTAY+K9GfS0F6iUtkXRNgbHuBT7Fu7dsASYB1elBnpXk1lWLdTswPp3rQN6dlTaO4y5gJbAwXc8f4XeKMjMrKeUmUZ2fpKnAIxExo9yxdEQ9KgdE5fjvlzsMs4L89nzWEUmqjYjqQnWesXQRQw7uQ43/gzIzaxOdLnlKug4Y26h4ekRMKEM4LZJ0KfDFRsXzIuLKcsRjZma7bre5bWvNq66ujpqamnKHYWbWaTR329ZvDG9mZpaRk6eZmVlGTp5mZmYZec2zi5D0Jrk3jejo9gdeKXcQRXKs7cOxtr3OEid0rFgPi4iCbwze6Z62tVZb3dTCd0ciqaYzxAmOtb041rbXWeKEzhOrb9uamZll5ORpZmaWkZNn1zGl3AEUqbPECY61vTjWttdZ4oROEqsfGDIzM8vIM08zM7OMnDzNzMwycvLcjUg6S9JqSX+QNLlAfQ9J96b65yRVlT7Kd2JpKdZTJC2UtF3SmHLEmBdLS7F+SdLK9Dmuj0k6rBxxplhaivVyScskLVw7NVcAAANkSURBVJb0tKSjyxFniqXZWPPaXSgpJJXlzxeKuKYTJG1K13SxpH8sR5wplhavqaRPpJ/XFZJ+XuoY8+Jo6bp+L++a/l7S6+WIs0kR4ddu8AK6A/8DHA7sCSwBjm7U5grgjrT9SeDeDhxrFXAMuQ8pH9PBr+tpwF5p+/Md/Lruk7d9LvCbjhpratcbeBJ4FqjuiHECE4Bby3EdWxHrAGARsF/aP7Cjxtqo/VXA3eW+xvkvzzx3H8cDf4iIFyPib8A9wHmN2pwH/CRtzwBOl6QSxtigxVgjYm1ELAV2lCG+fMXEOici3k67zwL9Shxjg2JifSNvd2+gXE8MFvPzCnADcDOwpZTB5Sk2zo6gmFg/B9wWEa8BRMTLJY6xQdbrehHwi5JEViQnz93HwcCf8vbXpbKCbSJiO1AH9C1JdE3EkRSKtaPIGutngV+3a0RNKypWSVdK+h/gW8CkEsXWWIuxShoOHBIRM0sZWCPFfv8vTLftZ0g6pDSh7aSYWD8IfFDSPEnPSjqrZNG9V9H/rtIySH/g8RLEVTQnT7M2IulTQDXw7XLH0pyIuC0ijgC+Any13PEUIqkb8F3gy+WOpQj/DVRFxDHAbN69u9MR7UHu1u1ocrO5OyXtW9aIWvZJYEZE1Jc7kHxOnruP9UD+b7z9UlnBNpL2APoAr5YkuibiSArF2lEUFaukM4DrgHMjYmuJYmss63W9Bzi/XSNqWkux9gYGA3MlrQVOBB4uw0NDLV7TiHg173t+FzCiRLE1Vsz3fx3wcERsi4g1wO/JJdNSy/Kz+kk62C1bwA8M7S4vcr9Rvkju9kbDAvygRm2u5L0PDN3XUWPNazuV8j4wVMx1HUbu4YcBneBnYEDe9seAmo4aa6P2cynPA0PFXNPKvO0LgGc76jUFzgJ+krb3J3frtG9HjDW1GwisJb2hT0d6lT0Av9rwmwn/QO43yf8Brktl/0ZuNgTQE5gO/AGYDxzegWM9jtxvyW+Rmx2v6MCx/hbYCCxOr4c7cKw/AFakOOc0l7DKHWujtmVJnkVe02+ma7okXdOBHfWaAiJ3O3wlsAz4ZEeNNe1fD9xUrhibe/nt+czMzDLymqeZmVlGTp5mZmYZOXmamZll5ORpZmaWkZOnmZlZRk6eZmZmGTl5mpmZZfT/AbqxlldWeFsjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "quality_pd = subscription.quality_monitoring.get_table_content(format='pandas')\n",
    "quality_pd.plot.barh(x='id', y='value');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness, drift monitoring and explanations <a name=\"fairness\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairness configuration\n",
    "\n",
    "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n",
    "\n",
    "  * Which model feature to monitor\n",
    "  * One or more **majority** groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n",
    "  * One or more **minority** groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n",
    "  * The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 95%)\n",
    "\n",
    "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.fairness_monitoring.enable(\n",
    "            features=[\n",
    "                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n",
    "                Feature(\"Age\", majority=[[26,75]], minority=[[18,25]], threshold=0.95)\n",
    "            ],\n",
    "            favourable_classes=['No Risk'],\n",
    "            unfavourable_classes=['Risk'],\n",
    "            min_records=200,\n",
    "            training_data=pd_data\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'config_status': {'model_name': None, 'state': 'new'},\n",
       " 'data_drift_enabled': True,\n",
       " 'drift_threshold': 0.05,\n",
       " 'is_schedule_enabled': True,\n",
       " 'min_samples': 200,\n",
       " 'model_drift_enabled': True,\n",
       " 'schedule_repeat_interval': 3,\n",
       " 'schedule_repeat_type': 'hour'}"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.drift_monitoring.enable(threshold=0.05, min_records=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10:31:28 new\n",
      "10:31:58 in_progress\n",
      "10:32:28 in_progress\n",
      "10:32:59 in_progress\n",
      "10:33:29 in_progress\n",
      "finished\n"
     ]
    }
   ],
   "source": [
    "drift_status = None\n",
    "while drift_status != 'finished':\n",
    "    drift_details = subscription.drift_monitoring.get_details()\n",
    "    drift_status = drift_details['parameters']['config_status']['state']\n",
    "    if drift_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), drift_status)\n",
    "        time.sleep(30)\n",
    "print(drift_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score the model again now that monitoring is configured"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 3.0M Jan 10 10:34 german_credit_feed.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_feed.json -O german_credit_feed.json\n",
    "!ls -lh german_credit_feed.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Score 200 randomly chosen records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "with open('german_credit_feed.json', 'r') as scoring_file:\n",
    "    scoring_data = json.load(scoring_file)\n",
    "\n",
    "fields = scoring_data['fields']\n",
    "values = []\n",
    "for _ in range(200):\n",
    "    values.append(random.choice(scoring_data['values']))\n",
    "payload_scoring = {\"fields\": fields, \"values\": values}\n",
    "\n",
    "\n",
    "scoring_response = wml_client.deployments.score(credit_risk_scoring_endpoint, payload_scoring)\n",
    "time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Now in payload table should be total 208 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records in payload table:  198\n"
     ]
    }
   ],
   "source": [
    "print('Number of records in payload table: ', subscription.payload_logging.get_records_count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run fairness monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=======================================================================\n",
      "\n",
      " Counting bias for deployment_uid=a58d369a-f7c8-4cb4-a9f1-68a6183114c8 \n",
      "\n",
      "=======================================================================\n",
      "\n",
      "\n",
      "\n",
      "RUNNING......\n",
      "FINISHED\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "fairness_run_details = subscription.fairness_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>FairnessMetrics (binding_id=409c1c90-3666-4ef3-8186-1b376c0f2c22, subscription_id=8a8842d9-5b8a-42ec-baca-5cd54dde47df)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>feature</th><th style='border: 1px solid #dddddd'>feature_value</th><th style='border: 1px solid #dddddd'>fairness_biased</th><th style='border: 1px solid #dddddd'>fairness_value</th><th style='border: 1px solid #dddddd'>fairness_fav_class</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>process</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-01-10 10:34:10.194728+00:00</td><td style='border: 1px solid #dddddd'>Sex</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>True</td><td style='border: 1px solid #dddddd'>0.92</td><td style='border: 1px solid #dddddd'>75.0</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:34:10.194728+00:00</td><td style='border: 1px solid #dddddd'>Age</td><td style='border: 1px solid #dddddd'>[18, 25]</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>1.068</td><td style='border: 1px solid #dddddd'>86.5</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td><td style='border: 1px solid #dddddd'></td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "\n",
    "subscription.fairness_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run drift monitor\n",
    "\n",
    "\n",
    "Kick off a drift monitor run on current data. The monitor runs every hour, but can be manually initiated using the Python client, the REST API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==========================================\n",
      "\n",
      " Waiting for end of drift monitoring run  \n",
      "\n",
      "==========================================\n",
      "\n",
      "\n",
      "\n",
      "RUNNING\n",
      "COMPLETED\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drift_run_details = subscription.drift_monitoring.run(background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ts</th>\n",
       "      <th>id</th>\n",
       "      <th>measurement_id</th>\n",
       "      <th>value</th>\n",
       "      <th>lower limit</th>\n",
       "      <th>upper limit</th>\n",
       "      <th>tags</th>\n",
       "      <th>binding_id</th>\n",
       "      <th>subscription_id</th>\n",
       "      <th>deployment_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-01-10 10:35:14.498424+00:00</td>\n",
       "      <td>data_drift_magnitude</td>\n",
       "      <td>92ca70ca-50a8-40d5-984d-8d233f6608a3</td>\n",
       "      <td>0.096154</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>409c1c90-3666-4ef3-8186-1b376c0f2c22</td>\n",
       "      <td>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td>\n",
       "      <td>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-01-10 10:35:14.498424+00:00</td>\n",
       "      <td>drift_magnitude</td>\n",
       "      <td>92ca70ca-50a8-40d5-984d-8d233f6608a3</td>\n",
       "      <td>0.059692</td>\n",
       "      <td></td>\n",
       "      <td>0.05</td>\n",
       "      <td></td>\n",
       "      <td>409c1c90-3666-4ef3-8186-1b376c0f2c22</td>\n",
       "      <td>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td>\n",
       "      <td>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-01-10 10:35:14.498424+00:00</td>\n",
       "      <td>predicted_accuracy</td>\n",
       "      <td>92ca70ca-50a8-40d5-984d-8d233f6608a3</td>\n",
       "      <td>0.721308</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>409c1c90-3666-4ef3-8186-1b376c0f2c22</td>\n",
       "      <td>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td>\n",
       "      <td>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                ts                    id  \\\n",
       "0 2020-01-10 10:35:14.498424+00:00  data_drift_magnitude   \n",
       "1 2020-01-10 10:35:14.498424+00:00       drift_magnitude   \n",
       "2 2020-01-10 10:35:14.498424+00:00    predicted_accuracy   \n",
       "\n",
       "                         measurement_id     value lower limit upper limit  \\\n",
       "0  92ca70ca-50a8-40d5-984d-8d233f6608a3  0.096154                           \n",
       "1  92ca70ca-50a8-40d5-984d-8d233f6608a3  0.059692                    0.05   \n",
       "2  92ca70ca-50a8-40d5-984d-8d233f6608a3  0.721308                           \n",
       "\n",
       "  tags                            binding_id  \\\n",
       "0       409c1c90-3666-4ef3-8186-1b376c0f2c22   \n",
       "1       409c1c90-3666-4ef3-8186-1b376c0f2c22   \n",
       "2       409c1c90-3666-4ef3-8186-1b376c0f2c22   \n",
       "\n",
       "                        subscription_id                         deployment_id  \n",
       "0  8a8842d9-5b8a-42ec-baca-5cd54dde47df  a58d369a-f7c8-4cb4-a9f1-68a6183114c8  \n",
       "1  8a8842d9-5b8a-42ec-baca-5cd54dde47df  a58d369a-f7c8-4cb4-a9f1-68a6183114c8  \n",
       "2  8a8842d9-5b8a-42ec-baca-5cd54dde47df  a58d369a-f7c8-4cb4-a9f1-68a6183114c8  "
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.drift_monitoring.get_table_content()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can disable drift monitor running code in cell below."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "subscription.drift_monitoring.disable()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we provide OpenScale with the training data to enable and configure the explainability features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import *\n",
    "\n",
    "subscription.explainability.enable(training_data=pd_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "explainability_details = subscription.explainability.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run explanation for sample record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80fd89244879c09b0b461354674e50b3-200\n"
     ]
    }
   ],
   "source": [
    "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'].values[0]\n",
    "\n",
    "print(transaction_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==================================================================\n",
      "\n",
      " Looking for explanation for 80fd89244879c09b0b461354674e50b3-200 \n",
      "\n",
      "==================================================================\n",
      "\n",
      "\n",
      "\n",
      "in_progress\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "explain_run = subscription.explainability.run(transaction_id=transaction_id, background_mode=False, cem=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAD4CAYAAAAAX/TLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de5xd49338c9XRIIEjaSo0EkcShBDdkIQQrXa3g5BFE1V2mqqrTo93E3rcYuWF0qpQ9WdElF1up09qLMQlJhEDhJUELcQROoUlZD4PX+sa2JlzGHPzN4zkz3f9+s1r1l7reu61u9ae5Lfvq619lqKCMzMzKyyrNbeAZiZmVnpOcGbmZlVICd4MzOzCuQEb2ZmVoGc4M3MzCrQ6u0dgBlA7969o6qqqr3DMDNbpUydOvWdiOhT3zYneOsQqqqqqKmpae8wzMxWKZJebWibp+jNzMwqkBO8mZlZBXKCNzMzq0A+B29mZh3Kp59+yvz581myZEl7h9JhdO/enb59+9K1a9ei6zjBm5m1scL4QnuHUHI1Y0p3kez8+fPp2bMnVVVVSCpZu6uqiGDRokXMnz+ffv36FV3PU/RmZtahLFmyhPXXX9/JPZHE+uuv3+wZDSf4MpLUV9Ltkl6U9JKkCyWtUYb9jJP0uqTpkp6VtH+p91FEDL9p632aWeVycl9ZS46HE3yZKHs3bgFui4gtgC2BHsCZZdrlBRFRDRwCTJC00nsrqSynY5RZDXCCNzPrQHwOvnz2ApZExJUAEbFc0gnAK5IKwHERMVPSM8CtEfFbSb8FXgNeBMYB7wDbAlOB70dESDob2B9YBtwXESfldxoRz0laBvSW9HtgCbAD8LikM4AJQH/g38CYFMM4YDNgc6A38PuI+AuApJOB7wLdUpynSaoC7gWeAgYBU4A1JU0HZgMvAf+KiD+mNs4E3o6IC0t1cM2s8yj1NQulvF6grqOOOooTTzyRAQMGNFhm9OjR7LvvvowcOXKl9fPmzeOJJ57ge9/7XklicYIvn23IEvMKEfGBpP8FHgaGpTsQLQN2TUWGAUcDG5El5W2AN4DHgV0lPQccCGyVkv16dXcqaSfgM2BhWtUX2CV9wLgYeCYiRkjaC/grUJ3KDQR2BtYGnpF0F9mHiy2AIYCAOyTtDvxvWn9kRDyZ9ntImkEgfQC4BfhjGt0fltqoG+sYYAzApptu2uQBNTPr6C6//PIW1503bx7XXnttyRK8p+jbxyPA7mSJ/S6gh6S1gH4R8UIqMyUi5kfEZ8B0oAp4n2xEfoWkg8hG4bVOSCPo84BDIyLS+hsjYnla3g24GiAiHgLWl7RO2nZ7RHwcEe+QfQAZAnwz/TwDTAO2IkvsAK/WJve6ImIesEjSDrX1I2JRPeXGR0QhIgp9+tR7K2Uzs3Zx7rnnctFFFwFwwgknsNdeewHw0EMPMWrUKO677z6GDh3KjjvuyCGHHMLixYsBGD58+Irbbl9xxRVsueWWDBkyhJ/85Cccc8wxK9p/9NFH2WWXXejfvz833XQTAGPHjmXy5MlUV1dzwQUXtLoPTvDlM4ds+nqFlEw3JUuYBbIR+6Pp9U9YecS/NLe8HFg9IpaRJd6bgH2Be3JlLoiI6ogYFhGTc+s/KjLeqOe1gLNSu9URsXlEXFFku5cDo4Efkp0WMDNbZQwbNozJk7P/Smtqali8eDGffvopkydPZuDAgZxxxhk88MADTJs2jUKhwPnnn79S/TfeeIPf/e53PPnkkzz++OM8//zzK21fsGABjz32GHfeeSdjx44F4Oyzz2bYsGFMnz6dE044odV9cIIvnweBtST9AEBSF+APwMSI+IDsXPshwD+AycBJZMm+QZJ6AOtGxN3ACcD2zYxpMjAqtTUceCfFAnCApO6S1geGA0+TnWf/UdovkjaW9OUG2v5UUv4ODLcC3wIGp3bMzFYZgwYNYurUqXzwwQd069aNoUOHUlNTw+TJk1lzzTWZM2cOu+66K9XV1Vx11VW8+urKz3yZMmUKe+yxB7169aJr164ccsghK20fMWIEq622GgMGDOCtt94qSx98Dr5M0jnyA4FLJZ1K9mHqbj6/2nwy8PWI+FjSZLJz5ZPrb22FnsDtkrqTja5PbGZY48iusJ9JNr1/ZG7bTLKp+d7A7yLiDeANSVsD/0hf0VgMfJ9sRqGu8cBMSdMiYlREfCLpYeC93CkCM7NVQteuXenXrx8TJ05kl112YeDAgTz88MPMnTuXfv368Y1vfIPrrruuxe1369ZtxfLnZ1RLywm+jCLiNWC/BradCpyalt8gS9i12yYBk3Kvj8lV/cLFahExroF9jK7z+l/AiAbCnRkRP6injQuB+q5+37ZOuV8Bv6p9nS6u25lslsLMbJUzbNgwzjvvPCZMmMB2223HiSeeyKBBg9h55535xS9+wdy5c9l888356KOPeP3119lyyy1X1B08eDDHH3887777Lj179uTmm29mu+22a3R/PXv25MMPPyxZ/E7wVnKSBgB3kn2t7sX2jsesoynn17QqUXsdr2HDhnHmmWcydOhQ1l57bbp3786wYcPo06cPEydO5PDDD2fp0uxyqTPOOGOlBL/xxhvzm9/8hiFDhtCrVy+22mor1l133Ub3N3DgQLp06cL222/P6NGjW30eXuWaGjBrjkKhELVXnppZ5/bcc8+x9dZbt3cYrbZ48WJ69OjBsmXLOPDAA/nRj37EgQce2OL26jsukqZGRL03CvBFdmZmZmUwbtw4qqur2XbbbenXrx8jRjR0hrQ8PEVvZmZWBuedd1677t8jeDMz63B8+nhlLTkeTvBmZtahdO/enUWLFjnJJ7XPg+/evXuz6nmK3szMOpS+ffsyf/58Fi5c2HThTqJ79+707du3WXWc4M3MrEOpvcmMtY6n6M3MzCqQE7yZmVkFcoI3MzOrQD4Hb2bWxgrj673xWKfg2/S2HY/gzczMKpATvBVF0imSZkuaKWm6pJ3aOyYzM2uYp+itSZKGAvsCO0bEUkm9gTXaOSwzM2uER/BWjI2AdyJiKUBEvBMRb0gaJOkRSVMl3StpI0mrS3pa0nAASWdJOrM9gzcz64yc4K0Y9wGbSPqnpEsl7SGpK3AxMDIiBgETgDMjYhkwGvizpL2BbwGn19eopDGSaiTV+I5VZmal5Sl6a1JELJY0CBgG7AncAJwBbAvcLwmgC7AglZ8t6WrgTmBoRHzSQLvjgfGQPQ++3P0wM+tMnOCtKBGxHJgETJI0C/gFMDsihjZQZTvgPeDLbROhmZnleYremiTpa5K2yK2qBp4D+qQL8JDUVdI2afkgoBewO3CxpPXaOmYzs87OI3grRg8+T9TLgLnAGLLp9YskrUv2t/RHSW8BZwNfj4jXJF0CXAgc2T6hm5l1TvLzdq0jKBQKUVPjO1yZmTWHpKkRUe+tET1Fb2ZmVoGc4M3MzCqQE7yZmVkFcoI3MzOrQE7wZmZmFcgJ3szMrAI5wZuZmVUgJ3gzM7MK5ARvZmZWgZzgzczMKpDvRW9m1sYK4+u9s2inUzPGt6cuJ4/gzczMKpATvJmZWQVygm8FSRtKul7SS5KmSrpb0hhJd5ag7eH1tSNpf0ljW9HuvpKekTRD0hxJP03rR0gaUET9osqZmVn7coJvIUkCbgUmRcRmETEI+DWwQTn3GxF3RMTZLakrqSvZM9z3i4jtgR2ASWnzCKCYxF1sOTMza0dO8C23J/BpRFxWuyIiZgCTgR6SbpL0vKRr0ocBJA2S9Ega7d8raaO0fnNJD6RR9TRJm+V3JGlwGnVvJmm0pEvS+omSLpL0hKSXJY1M61eTdGna//1pZmEk0JPswspFKd6lEfGCpF2A/YFzJU1P+/mJpKdTTDdLWquBcpMkFdJ+e0ual5a3kTQllZspaYuyvRNmZvYFTvAtty0wtYFtOwDHk410+wO7ptHzxcDINNqfAJyZyl8D/CmNqncBFtQ2lJLqZcABEfFSPfvaCNgN2BeoHdkfBFSl/R8BDAWIiH8BdwCvSrpO0ihJq0XEE2n9yRFRnfZzS0QMTjE9B/y4gXINORq4MCKqgQIwv26BdDqjRlLNwoULG2nKzMyay1+TK48pETEfQNJ0smT7HtmHgvvTgL4LsEBST2DjiLgVICKWpHoAW5NNqX8zIt5oYF+3RcRnwBxJtacHdgNuTOvflPRwbeGIOErSdsDewEnAN4DR9bS7raQzgPWAHsC9zTwG/wBOkdSX7MPCi3ULRMT41D8KhUI0s30zM2uER/AtNxsY1MC2pbnl5WQfpATMTiPf6ojYLiK+2cQ+FgBLyGYEGpLfl5poD4CImBURF5Al94MbKDYROCYitgNOB7o3UG4Zn/8drSgTEdeSTed/DNwtaa9iYjMzs9Jwgm+5h4BuksbUrpA0EBjWQPkXgD6ShqayXSVtExEfAvMljUjru0laK9V5D/gP4CxJw5sR2+PAwelc/AbA8NR2jzrtVAOvpuUPyc7R1+pJNsPQFRiVW1+33Dw+/6AzsnalpP7AyxFxEXA7MLAZ8ZuZWSs5wbdQRARwILB3+prcbOAs4M0Gyn9ClgDPkTQDmE52vh2y8+THSpoJPAFsmKv3Ftn59T9J2qnI8G4mO+c9B/gbMA14n2yE/5+SXkinDk7n8+n564GTay/mA04FniL7sPB8ru265c4DfibpGaB3rtx3gWfTfrYF/lpk7GZmVgLK8pRVGkk9ImKxpPWBKcCuEVHvh4+OoFAoRE2Nb1tpZtYckqZGRL33PvZFdpXrTknrAWsAv+vIyd3MzErPCb5CRcTw9o7BzMzaj8/Bm5mZVSAneDMzswrkBG9mZlaBnODNzMwqkBO8mZlZBXKCNzMzq0BO8GZmZhXICd7MzKwC+UY3ZmZtrDC+3juLdmo1Y3yr6lLzCN7MzKwCOcG3A0mL22AfIySFpK3Kva8m4jg+9/hbMzNrI07wletw4LH0uz0dDzjBm5m1MSf4DkJSlaSHJM2U9KCkTdP6/SQ9lZ6//oCkDdL6cZImSJok6WVJx+ba6gHsBvwYOCy3frikRyTdnuqcLWmUpCmSZqXnuzcWy0RJI3PtLc61O0nSTZKel3SNMscCXwEelvRw2Q+imZmt4ATfcVwMXBURA4FrgIvS+seAnSNiB+B64D9zdbYC9gGGAKdJ6prWHwDcExH/BBZJGpSrsz1wNLA1cASwZUQMAS4HftlELI3ZgWy0PgDoT/b8+YuAN4A9I2LPuhUkjZFUI6lm4cKFRezCzMyK1awE73OpZTUUuDYtX002AgfoC9wraRZwMrBNrs5dEbE0It4B3gY2SOsPJ/swQPqdn6Z/OiIWRMRS4CXgvrR+FlDVRCyNmRIR8yPiM2B6rq0GRcT4iChERKFPnz5F7MLMzIpVVIKXtIukOcDz6fX2ki4ta2RW62LgkojYDvgp0D23bWlueTmwuqRewF7A5ZLmkX0o+K4k1VPns9zrz2j6a5PLSH8zklYD1mgslibaMjOzMip2BH8B2VTwIoCImAHsXq6gOqkn+Px8+ShgclpeF3g9LR9ZRDsjgasj4qsRURURmwCvAMNKEMs8oHa6f3+gK037EOjZjH2bmVkJFD1FHxGv1Vm1vMSxdCZrSZqf+zmR7Pz3DyXNJDs3flwqOw64UdJU4J0i2j4cuLXOuptp3tX0DcXyF2APSTPIpvE/KqKt8cA9vsjOzKxtKSKaLiTdBJwPXALsRPYffiEiDmu0olmRCoVC1NT4TlbWOfhOdl/kO9m1jKSpEVHvH1Sx50mPBi4ENiabLr4P+EVpwjMz61yczKwtFJXg01Xao8oci5mZmZVIUQleUj+y87JV+ToRsX95wjIzM7PWKHaK/jbgCuD/kX2dyszMzDqwYhP8knRXMjMzM1sFFJvgL5R0GtnFdStuaBIR08oSlZmZmbVKsQl+O7LvQ+/F51P0kV6bmZlZB1Nsgj8E6B8Rn5QzGDMzMyuNYu9k9yywXjkDMTMzs9IpdgS/HvC8pKdZ+Ry8vyZnZmbWARWb4E8raxRmZmZWUsXeye6RcgdiZq3j+5uvOnyrWmsLxT4PfmdJT0taLOkTScslfVDu4MzMzKxlir3I7hKyx42+CKwJHAX8qVxBtUT60DE99zO2RO3Ok9S7FG2VgqT1JP0897pK0seSnpH0nKQpkkaXeJ/HS1or9/puSb7o0sysAyv2HDwRMVdSl4hYDlwp6Rng1+ULrdk+jojq9g6iDawH/By4NLfupYjYAUBSf+AWSYqIK4tpUJLIHh3c0G2Ijwf+BvwbICK+09LgzcysbRQ7gv+3pDWA6ZJ+L+mEZtRtV2kEflYa1ddI2lHSvZJeknR0KjNc0qOS7pL0gqTLJH2hf5JOlPRs+jk+rftt7XJ6faak41Kbj0i6XdLLks6WNCqNsGdJ2iyV7yPp5nQK5GlJu6b14yRNkDQp1T827eJsYLPUn3PrxhgRLwMnAsfm2jkpF9+zadRflfr6V7KvQW4i6c/pGM2WdHoqfyzwFeBhSQ/njmnvRo5JVZpN+Etq6z5Ja7bmfTQzs+YpNkkfkcoeA3wEbAIcXK6gWmjNOlP0h+a2/W8a3U8GJgIjgZ2B03NlhpA9MW8AsBlwUL5xSYOAHwI7pbo/kbQDMAH4QSqzGnAY2WgXYHvgaGBrsmO4ZUQMAS5P+wK4ELggIgaTHdPLc7vdCtgnxXaapK7AWLIRe3VEnNzAsZiW6jZlC+DSiNgmIl4FTomIAjAQ2EPSwPQMgjeAPSNizyKPSW3bf4qIbYD3qOfvRdKY9IGiZuHChUWEa2ZmxSr2KvpX0+ISVk6KHUljU/R3pN+zgB4R8SHwoaSluXPJU9LoF0nXAbsBN+Xa2A24NSI+SmVuAYZFxEWSFqXEtgHwTEQsyma9eToiFqTyL5Hdy782jtpkuTcwIJUHWEdSj7R8V0QsBZZKeju1Xww1XQSAVyPiydzr70oaQ/Z3sRHZh52ZjdSv95iQHe9XImJ6KjeV7FHDK4mI8cB4gEKhEEXGbGZmRSj2efC7AuOAr7Ly8+D7lyeskqu9Oc9nueXa17X9qZtgmpNwLgdGAxuSjejr7rfuvvP7XQ3YOSKW5BtMCT9ffznFXzOxA/BcWl7GyjM13XPLH+X21w84CRgcEe9KmlinbHPVjd1T9GZmbajYKforgPPJRmyDcz+VZIikfmma/VDgsTrbJwMjJK0laW3gwLQO4FbgW2TH5N5m7vc+Pp+uR1JTFwp+CPRsaKOkKuA84OK0ah6wY9q2I9CvgarrkCX89yVtAHy7iH02dkzMzKwdFTsifD8i/l7WSFpvTUnTc6/viYjmfFXuabKvA24OPEyWtFeIiGlpVDslrbo8Ip5J2z5JF6C9l75l0BzHAn+SNJPs/XiU7Lx9vdL0/+OSngX+TvZ1xc3Stxq6kyXjiyJiYqpyM/ADSbOBp4B/NtDujNTG88BrwOO5zeOBeyS9kT8P39AxSR8yzMysHSmi6ZloSWcDXYBbqMDnwUsaDpwUEfu2sP5qZBe2HRIRL5Yyts6iUChETY3v7tUavpPdqsN3srNSkTQ1XRz9BcWO4HdKv/ON+HnwgKQBwJ1kF5s5uVu7cdIws7xir6Lfs7Htko6MiKtKE1Lbi4hJwKQW1p0DrCoXG5qZWSdRqpvVHFeidszMzKwESpXgi/3etZmZmbWBUiV436TEzMysA/EI3szMrAKVKsE/3nQRMzMzaytFJXhJG0i6QtLf0+sBkn5cuz0ijilXgGZmZtZ8xY7gJ5LdgvUr6fU/yZ4RbmZmZh1QsQm+d0T8D9lDUoiIZWQPEDEzM7MOqNgE/5Gk9UlXy0vaGXi/bFGZmZlZqxR7q9oTyZ7xvZmkx4E+wMiyRWVmVsH83IBV26pyW+gmE3x6kEp3YA/ga2RfiXshIj4tc2xmZmbWQk1O0UfEZ8CfImJZRMyOiGed3Bsnqa+k2yW9KOklSRdKWkNStaTv5MqNk3RSmWIYI+n59DNF0m5F1Bkt6ZJyxGNmZm2r2HPwD0o6WJJvaNOEdIxuAW6LiC2ALYEewJlANfCdRqo3d19dGli/L/BTYLeI2Irs+fLXStqwVPs2M7OOrdgE/1PgRmCppA8kfSjpgzLGtSrbC1gSEVcCRMRy4ATgKOD3wKGSpks6NJUfIGmSpJclHVvbiKTvp5H3dEn/XZvMJS2W9AdJM4Chks6WNEfSTEnnpeq/Ak6OiHdSDNOAq4BfpDbmSTpd0jRJsyRt1ViHJB2eyj0r6Zzc+j9LqpE0W9LpufXNat/MzEqvqAQfET0jYrWIWCMi1kmv1yl3cKuobYCp+RUR8QEwDzgDuCEiqiPihrR5K2AfYAhwmqSukrYGDgV2jYhqsq8kjkrl1waeiojtgeeAA4FtImJgar/eGICatL7WOxGxI/BnoMHTBJK+ApxD9sGlGhgsaUTafEpEFICBwB6SBjan/XQaoUZSzcKFCxsKwczMWqCoq+gl7V7f+oh4tLThdEp3RcRSstmRt4ENgK8Dg4Cn01mRNYG3U/nlwM1p+X1gCXCFpDuBO5ux31vS76nAQY2UGwxMioiFAJKuAXYHbgO+K2kM2d/RRsAAYGax7UfEeGA8QKFQ8AOLzMxKqNivyZ2cW+5ONtqcSjaqs5XNoc5XCCWtA2wKLKun/NLc8nKy90TAVRHx63rKL0nT/kTEMklDyD4QjASOIXtP5pB9QHgoV28QMLue/dbus1kk9SMbmQ+OiHclTST72yhJ+2Zm1jrFTtHvl/v5BrAt8G55Q1tlPQisJekHsOJCuD+Q3e73LaBnkW2MlPTl1EYvSV+tW0hSD2DdiLib7Dz/9mnT74Fz0s2JkFQNjAYubUF/ppBNv/dOfTkceARYB/gIeF/SBsC3W9C2mZmVSUtHVvOBrUsZSKWIiJB0IHCppFPJPkTdDfyG7Pz5WEnTgbMaaWOOpP8L3JfuQ/Ap2QVyr9Yp2hO4XVJ3slH/ian+HZI2Bp6QFMCHwPcjYkERXRidO8cOsDMwFng47eOuiLgdQNIzwPPAa/iJgmZmHYoimj71Keli0m1qyRJWNTAvIr5fxtisEykUClFTs2rcHcrMrKOQNDVd7PwFxY7g8//zLgOuiwiP2MzMzDqoYhP8ehFxYX6FpOPqrjMzM7OOodgb3RxZz7rRJYzDzMzMSqjREbykw4HvAf0k3ZHb1BP4VzkDMzMzs5Zraor+CWAB0Jvsq161PuTzG5qYmZlZB9Nogo+IV8m+mjW0bcIxMzOzUijqHLyknSU9nR508omk5X7YjJmZWcdV7EV2l5DdwexFsvuiHwX8qVxBmZmZWesUm+CJiLlAl4hYnh6F+q3yhWVmZmatUez34P8taQ1guqTfk114V/SHAzMzM2tbxSb4I8gS+jFkDzXZBDi4XEGZWf0K4+u9I6WtYmrG+LbMVn5FJfiIeFXSmsBGEXF6mWMyMzOzVir2Kvr9gOnAPel1dZ0b35iZmVkHUux59HHAEOA9gIiYDvQrU0wVR9LiMre/XNJ0SbMlzZD0f9JjZkvV/mhJX8m9vlzSgFK1b2ZmpVfsOfhPI+J9Sfl1TT9n1trKxxFRDSDpy8C1wDrAacU2IKlLRCxvYPNo4FngDYCIOKpV0ZqZWdkVO8qbLel7QBdJW6Tnwz9RxrgqnqQqSQ9JminpQUmbpvX7SXpK0jOSHpC0QVo/TtIESZMkvSzp2PrajYi3gTHAMcqMlnRJbr93ShqelhdL+oOkGcBQSf+Vbmj0rKTxqf5IoABck2YJ1kwxFFIbh0ualeqck9vPYklnphmFJ2v7YWZmbaPRBC/p6rT4ErANsBS4DvgAOL68oVW8i4GrImIgcA1wUVr/GLBzROwAXA/8Z67OVsA+ZKdLTpPUtb6GI+JloAvw5SZiWBt4KiK2j4jHgEsiYnBEbEt2Q6N9I+ImoAYYFRHVEfFxbeU0bX8OsBdQDQyWNCLX9pMRsT3wKPCTujuXNEZSjaSahQsXNhGqmZk1R1Mj+EHpP/FDyR42sw/wzbS8Vpljq3RDyabSAa4GdkvLfYF7Jc0CTib7YFXrrohYGhHvAG8DrR0VLwduzr3eM80ezCJL2tvUX22FwcCkiFgYEcvIPqjsnrZ9AtyZlqcCVXUrR8T4iChERKFPnz6t6IaZmdXV1Dn4y4AHgf5ko7haIjsH379McXVmFwPnR8QdaSp9XG7b0tzychp4/yT1T9vfBpax8ge57rnlJbXn3SV1By4FChHxmqRxdco216cRUXudRoOxmplZeTQ6go+IiyJia2BCRPTP/fSLCCf31nkCOCwtjwImp+V1gdfT8pHNbVRSH7IPZpekBDsPqJa0mqRNyKb361ObzN+R1AMYmdv2IdCznjpTgD0k9ZbUhex5BY80N2YzMyu9Ym9087NyB1Lh1pI0P/f6fOCXwJWSTgYWAj9M28YBN0p6F3iI4r6OuKak6UBXshH71WkfAI8DrwBzgOeAafU1EBHvSfoL2dXybwJP5zZPBC6T9DG5RwdHxAJJY4GHyWZ17oqI24uI18zMykyfz6KatZ9CoRA1Nb59p5lZc0iaGhH13sPaD4wxMzOrQE7wZmZmFcgJ3szMrAI5wZuZmVUgJ3gzM7MK5ARvZmZWgZzgzczMKpATvJmZWQVygjczM6tATvBmZmYVyE/4MjNrY4Xx9d5Z1DqpmjHluU23R/BmZmYVyAneiiJphKSQtFV7x2JmZk1zgrdiHQ48ln6bmVkH5wRvTZLUA9gN+DFwWFq3mqRLJT0v6X5Jd0sambYNkvSIpKmS7pW0UTuGb2bWKTnBWzEOAO6JiH8CiyQNAg4CqoABwBHAUABJXYGLgZERMQiYAJxZX6OSxkiqkVSzcOHC8vfCzKwT8VX0VozDgQvT8vXp9erAjRHxGfCmpIfT9q8B2wL3SwLoAiyor9GIGA+MBygUClG26M3MOiEneGuUpF7AXsB2koIsYQdwa0NVgNkRMbSNQjQzs3p4it6aMhK4OiK+GhFVEbEJ8ArwL+DgdC5+A2B4Kv8C0EfSiil7Sdu0R+BmZp2ZE7w15XC+OFq/GdgQmA/MAf4GTAPej4hPyD4UnCNpBjAd2KXtwjUzM/AUvTUhIvasZ91FkF1dHxGLJa0PTAFmpe3Tgd3bNCtbwhAAAAx5SURBVFAzM1uJE7y1xp2S1gPWAH4XEW+2d0Bmq4Jy3ZrULM8J3losIoa3dwxmZlY/n4M3MzOrQE7wZmZmFcgJ3szMrAI5wZuZmVUgJ3gzM7MK5ARvZmZWgZzgzczMKpATvJmZWQVygjczM6tAvpOddWqF8YX2DsE6Id+q1tqCR/BmZmYVyAm+A5O0vqTp6edNSa/nXq/RQJ356QEwLdnfY5KqWxe1mZl1BJ6i78AiYhFQDSBpHLA4Is5r16DMzGyV4BH8KkrSkZKmpNH8pZK+8F7WV0bS6pLek3SRpNmS7k/Pc691WKrzgqRdUjtrSrpK0ixJ0yTtntYfJekmSfdKelHSWbl9f1vSP1L5GyStXfaDYmZmKzjBr4IkbQscCOwSEdVkMzGHNaPMusDjEbEN8A/g1HzViBgCnAz8V1p3LLA0IrYDjgCuzp0i2B44BBgIfF/SVyR9GRgLfD0idgRmAsfV048xkmok1SxcuLAVR8TMzOryFP2qaW9gMFAjCWBN4LVmlFkG3JiW/wZcm6t3S/o9FahKy7sB5wJExGxJbwCbp20PRMQHAJKeBzYFNgQGAE+kfa8BPFa3ExExHhgPUCgUosi+m5lZEZzgV00CJkTEqc0tI6m+9zyfXJem38sp7u9jaW65to6AeyLiiCLqm5lZGXiKftX0APBdSb1hxdX2mzajzOrAQWn5e9Qzuq5jMjAqtbM1sBEwt5HyTwB7SOqf6qwtaYuiemZmZiXhBL8KiohZwOnAA5JmAvcBGzSjzPvAMEmzyabfz2hilxcDa0qaBVwD/CAiPmkkvreAHwM3SJpBlvC3bF4vzcysNRThU5+dSZqifyciWvRd+XIpFApRU9P2d/fyneysPfhOdlYqkqZGRL3/kfkcvHVq/o/WzCqVE3wnExHLgA41ejczs9LzOXgzM7MK5ARvZmZWgZzgzczMKpATvJmZWQVygjczM6tATvBmZmYVyAnezMysAjnBm5mZVSDf6MZWeb7drK1qfAdFawsewZuZmVUgJ3gzM7MKVPYEL2lDSddLeknSVEl3S2qzR4dKWk/Sz3OvqyR9LGm6pDmS/iqpawvbvlvSF+7rLmmcpJNaE3czYhgn6fXUnxcl3SJpQAnbr5L0vdzrgqSLStW+mZmVR1kTvCQBtwKTImKziBgE/Jo6zy5vpH6XOq9bcs3AesDP66x7KSKqge2AvsB3W9AuEfGdiHivJXVL7IKIqI6ILYAbgIck9Sm2chPHtQpYkeAjoiYijm1xpGZm1ibKPYLfE/g0Ii6rXRERM4Auku6sXSfpEkmj0/I8SedImgYcImmSpD9KqgGOk9RH0s2Snk4/u6Z64yRNSOVfllSbhM4GNksj3HPzwUXEcmAKsHFqo4ukc1O7MyX9NK3fSNKjqY1nJQ3Lxdo7LZ8i6Z+SHgO+luvbZpLuSbMXkyVtldZPlHSRpCdSvCNzdX4laZakGZLObqyduiLiBuA+UlKuE2NB0qTc8bpa0uPA1WmkPlnStPSzS+74DUt9P0HS8Nr3TlIvSbelY/WkpIFNvBdmZtZGyn0V/bbA1BbUWxQROwJIOhpYo/aB9pKuJRuxPiZpU+BeYOtUbyuyDxU9gRck/RkYC2ybRuxIqqrdiaTuwE7AcWnVj4H3I2KwpG7A45LuAw4C7o2IM9Oswlr5YCUNAg4DqsmO6bRcv8cDR0fEi5J2Ai4F9krbNgJ2S3HfAdwk6dvAAcBOEfFvSb2KaKeuaanNpgwAdouIjyWtBXwjIpZI2gK4Diik43dSROyb+jo8V/904JmIGCFpL+Cv6RhAPe9FRHxa57iNAcYAbLrppkWEa2ZmxeqoX5O7oZHXewMDstl/ANaR1CMt3xURS4Glkt6m4VMBm0maDvRLdWam9d8EBuZG0+sCWwBPAxPSufrbImJ6nfaGAbdGxL8BJN2RfvcAdgFuzMXbLVfvtoj4DJgjqTbWvYEra9uKiH8V0U5damRb3h0R8XFa7gpcIqkaWA4Uc53EbsDBKc6HJK0vaZ20rb73Yn6+ckSMJ/vgQqFQiCJjNjOzIpQ7wc8GRtazfhkrnx7oXmf7R428Xg3YOSKW5AukxLc0t2o5DffvpYioTlPXj0vaPyLuIEuMv4yIe+tWkLQ78B/AREnnR8RfG2g7bzXgvdrZg3rk420sKTfVTl07ALVftM0f68aO8wnAW8D2qfwSWqfY98LMzMqg3OfgHwK6palYANJ5WpGNwrspuwr9681o8z7gl7n2mkp6H5JNE39BRLxDNgX967TqXuBnaaSOpC0lrS3pq8BbEfEX4HJgxzpNPQqMkLSmpJ7Afqn9D4BXJB2S2pOk7ZuI937gh2nKHEm9mtOOpIPJZiKuS6vmAYPS8sGN7HddYEGaUTgCqL3AscHjB0wGRqX9DgfeSbGamVk7K2uCj4gADgT2VvY1udnAWcCbwP8Az6bfzzSj2WOBQrqwaw5wdBMxLCIbpT+rOhfZJbcBa6UL5y4H5gDTJD0L/DfZyHM4MEPSM8ChwIV19jGN7DTCDODvZFP6tUYBP5Y0g2xG44Am4r2H7Hx8TTqNUPt1u8baOSFdBPci8H1gr4hYmLadDlyo7CLF5Y3s+lLgyNT+Vnw+up8JLE8X/J1Qp844YJCkmWQX4x3ZWN/MzKztKMvBZu2rUChETY1v32lm1hySptZehF6X72RnZmZWgZzgzczMKpATvJmZWQVygjczM6tATvBmZmYVyFfRW4cgaSHwajvtvjfwTjvtuxzcn46v0vrk/rSfr0ZEvQ8Xc4K3Tk9STUNfM1kVuT8dX6X1yf3pmDxFb2ZmVoGc4M3MzCqQE7xZeqJdBXF/Or5K65P70wH5HLyZmVkF8gjezMysAjnBm5mZVSAneKt4knpJul/Si+n3lxood4+k9yTdWWf9REmvpEfyTpdU3TaRN6wEfeon6SlJcyXdIGmNtom8fs3oz5GpzIuSjsytnyTphdx79OW2i36l+L6V4pgraWw927ul4z03Hf+q3LZfp/UvSNqnLeNuTEv7JKlK0se59+Syto69PkX0Z3dJ0yQtkzSyzrZ6//46rIjwj38q+gf4PTA2LY8Fzmmg3NeB/YA766yfCIxs736UuE//AxyWli8DftbR+wP0Al5Ov7+Ulr+Utk0CCu3chy7AS0B/YA1gBjCgTpmfA5el5cOAG9LygFS+G9AvtdOlA/ydtaZPVcCz7d2HFvSnChgI/DX/776xv7+O+uMRvHUGBwBXpeWrgBH1FYqIB4EP2yqoVmpxnyQJ2Au4qan6baiY/uwD3B8R/4qId4H7gW+1UXzFGALMjYiXI+IT4HqyfuXl+3kT8PX0fhwAXB8RSyPiFWBuaq+9taZPHVGT/YmIeRExE/isTt2O/vf3BU7w1hlsEBEL0vKbwAYtaONMSTMlXSCpWwlja6nW9Gl94L2IWJZezwc2LmVwLVBMfzYGXsu9rhv3lWkq+NR2SjBNxbdSmXT83yd7P4qp2x5a0yeAfpKekfSIpGHlDrYIrTnOHfU9atDq7R2AWSlIegDYsJ5Np+RfRERIau53Q39NlnTWIPt+7K+A37YkzuYoc5/aXJn7MyoiXpfUE7gZOIJsitXazwJg04hYJGkQcJukbSLig/YOrLNwgreKEBF7N7RN0luSNoqIBZI2At5uZtu1I8ulkq4ETmpFqM3Zb7n6tAhYT9LqacTVF3i9leE2qQT9eR0Ynnvdl+zcOxHxevr9oaRryaZi2zrBvw5sUie+use1tsx8SasD65K9H8XUbQ8t7lNkJ66XAkTEVEkvAVsCNWWPumGtOc4N/v11VJ6it87gDqD2itcjgdubUzklnNpz1yOAZ0saXcu0uE/pP96HgdorhJt9TMqgmP7cC3xT0pfSVfbfBO6VtLqk3gCSugL70j7v0dPAFukbCmuQXXB2R50y+X6OBB5K78cdwGHpivR+wBbAlDaKuzEt7pOkPpK6AEjqT9anl9so7oYU05+G1Pv3V6Y4S6O9r/Lzj3/K/UN2PvBB4EXgAaBXWl8ALs+VmwwsBD4mO7+2T1r/EDCLLGn8DehRAX3qT5ZA5gI3At1Wkf78KMU8F/hhWrc2MBWYCcwGLqSdrkAHvgP8k+xK7VPSut8C+6fl7ul4z03Hv3+u7imp3gvAt9v7b6y1fQIOTu/HdGAasF9796XI/gxO/1Y+Iptdmd3Y319H/vGtas3MzCqQp+jNzMwqkBO8mZlZBXKCNzMzq0BO8GZmZhXICd7MzKwCOcGbmZlVICd4MzOzCvT/AR69Miak4JZfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "explain_result = pd.DataFrame.from_dict(explain_run['entity']['predictions'][0]['explanation_features'])\n",
    "explain_result.plot.barh(x='feature_name', y='weight', color='g', alpha=0.8);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom monitors and metrics <a name=\"custom\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register custom monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_definition(monitor_name):\n",
    "    monitors_definitions = ai_client.data_mart.monitors.get_details()['monitor_definitions']\n",
    "    \n",
    "    for definition in monitors_definitions:\n",
    "        if monitor_name == definition['entity']['name']:\n",
    "            return definition\n",
    "    \n",
    "    return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import Metric, Tag\n",
    "\n",
    "monitor_name = 'my model performance'\n",
    "metrics = [Metric(name='sensitivity', lower_limit_default=0.8), Metric(name='specificity', lower_limit_default=0.75)]\n",
    "tags = [Tag(name='region', description='customer geographical region')]\n",
    "\n",
    "existing_definition = get_definition(monitor_name)\n",
    "\n",
    "if existing_definition is None:\n",
    "    my_monitor = ai_client.data_mart.monitors.add(name=monitor_name, metrics=metrics, tags=tags)\n",
    "else:\n",
    "    my_monitor = existing_definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List monitors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get monitors uids and details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_model_performance\n"
     ]
    }
   ],
   "source": [
    "monitor_uid = my_monitor['metadata']['guid']\n",
    "\n",
    "print(monitor_uid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "monitor definition details {'entity': {'metrics': [{'expected_direction': 'increasing', 'id': 'sensitivity', 'name': 'sensitivity', 'required': True, 'thresholds': [{'default': 0.8, 'type': 'lower_limit'}]}, {'expected_direction': 'increasing', 'id': 'specificity', 'name': 'specificity', 'required': True, 'thresholds': [{'default': 0.75, 'type': 'lower_limit'}]}], 'name': 'my model performance', 'tags': [{'description': 'customer geographical region', 'id': 'region', 'name': 'region', 'required': True}]}, 'metadata': {'guid': 'my_model_performance', 'url': '/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/monitor_definitions/my_model_performance', 'created_at': '2019-11-14T12:51:16.792Z'}}\n"
     ]
    }
   ],
   "source": [
    "my_monitor = ai_client.data_mart.monitors.get_details(monitor_uid=monitor_uid)\n",
    "print('monitor definition details', my_monitor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable custom monitor for subscription"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import Threshold\n",
    "\n",
    "thresholds = [Threshold(metric_uid='sensitivity', lower_limit=0.9)]\n",
    "subscription.monitoring.enable(monitor_uid=monitor_uid, thresholds=thresholds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get monitor configuration details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'enabled': True,\n",
       " 'monitor_definition': {'entity': {'metrics': [{'expected_direction': 'increasing',\n",
       "     'id': 'sensitivity',\n",
       "     'name': 'sensitivity',\n",
       "     'required': True,\n",
       "     'thresholds': [{'default': 0.9, 'type': 'lower_limit'}]},\n",
       "    {'expected_direction': 'increasing',\n",
       "     'id': 'specificity',\n",
       "     'name': 'specificity',\n",
       "     'required': True}],\n",
       "   'name': 'my model performance',\n",
       "   'tags': [{'description': 'customer geographical region',\n",
       "     'id': 'region',\n",
       "     'name': 'region',\n",
       "     'required': True}]},\n",
       "  'metadata': {'guid': 'my_model_performance',\n",
       "   'url': '/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/monitor_definitions/my_model_performance',\n",
       "   'created_at': '2019-11-14T12:51:16.792Z'}},\n",
       " 'thresholds': [{'metric_id': 'sensitivity',\n",
       "   'type': 'lower_limit',\n",
       "   'value': 0.9}]}"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription.monitoring.get_details(monitor_uid=monitor_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'asset_id': 'ac325cbf-f50b-40e2-bb08-f1c8f085c1b2',\n",
       "  'binding_id': '409c1c90-3666-4ef3-8186-1b376c0f2c22',\n",
       "  'measurement_id': '13ba26a5-e0d3-4e9c-a820-5638dc6f4b2f',\n",
       "  'metrics': [{'region': 'us-south',\n",
       "    'sensitivity': 0.67,\n",
       "    'specificity': 0.78}],\n",
       "  'monitor_definition_id': 'my_model_performance',\n",
       "  'subscription_id': '8a8842d9-5b8a-42ec-baca-5cd54dde47df',\n",
       "  'timestamp': '2020-01-10T10:43:53.637038Z'}]"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = {\"specificity\": 0.78, \"sensitivity\": 0.67, \"region\": \"us-south\"}\n",
    "\n",
    "subscription.monitoring.store_metrics(monitor_uid=monitor_uid, metrics=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List and get custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>CustomMetrics (binding_id=409c1c90-3666-4ef3-8186-1b376c0f2c22, subscription_id=8a8842d9-5b8a-42ec-baca-5cd54dde47df)</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower limit</th><th style='border: 1px solid #dddddd'>upper limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2020-01-10 10:43:53.637038+00:00</td><td style='border: 1px solid #dddddd'>sensitivity</td><td style='border: 1px solid #dddddd'>13ba26a5-e0d3-4e9c-a820-5638dc6f4b2f</td><td style='border: 1px solid #dddddd'>0.67</td><td style='border: 1px solid #dddddd'>0.9</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>region: us-south</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr><tr><td style='border: 1px solid #dddddd'>2020-01-10 10:43:53.637038+00:00</td><td style='border: 1px solid #dddddd'>specificity</td><td style='border: 1px solid #dddddd'>13ba26a5-e0d3-4e9c-a820-5638dc6f4b2f</td><td style='border: 1px solid #dddddd'>0.78</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>region: us-south</td><td style='border: 1px solid #dddddd'>409c1c90-3666-4ef3-8186-1b376c0f2c22</td><td style='border: 1px solid #dddddd'>8a8842d9-5b8a-42ec-baca-5cd54dde47df</td><td style='border: 1px solid #dddddd'>a58d369a-f7c8-4cb4-a9f1-68a6183114c8</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "\n",
    "subscription.monitoring.show_table(monitor_uid=monitor_uid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'asset_id': 'ac325cbf-f50b-40e2-bb08-f1c8f085c1b2',\n",
       "  'binding_id': '409c1c90-3666-4ef3-8186-1b376c0f2c22',\n",
       "  'tags': [{'id': 'region', 'value': 'us-south'}],\n",
       "  'ts': '2020-01-10T10:43:53.637038Z',\n",
       "  'measurement_id': '13ba26a5-e0d3-4e9c-a820-5638dc6f4b2f',\n",
       "  'monitor_definition_id': 'my_model_performance',\n",
       "  'subscription_id': '8a8842d9-5b8a-42ec-baca-5cd54dde47df',\n",
       "  'metrics': [{'lower_limit': 0.9, 'id': 'sensitivity', 'value': 0.67},\n",
       "   {'id': 'specificity', 'value': 0.78}]}]"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "custom_metrics = subscription.monitoring.get_metrics(monitor_uid=monitor_uid, deployment_uid='credit')\n",
    "custom_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAD4CAYAAABMtfkzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAARpklEQVR4nO3de5CddX3H8fcXNphRwmUCSCTIhooGQoCE9QKVKkUqQ8BLuQTBCmJrQQFJ6YyZYkemTdsoM4IUpgz1Ei8gxK0pWEBFCOIliFnIFQaqNkyXcIkZiUEMDeHbP86zeNxscs6SPef8Nvt+zeyc5/J7zvPZZzf55LlkNzITSZJKsEunA0iSNMBSkiQVw1KSJBXDUpIkFcNSkiQVo6vTAUazffbZJ7u7uzsdQ5JGlb6+vl9l5r5DrbOUdkB3dzdLly7tdAxJGlUi4vFtrfPynSSpGJaSJKkYlpIkqRjeU5KkFtu8eTP9/f1s2rSp01Haavz48UyePJlx48Y1vY2lJEkt1t/fz4QJE+ju7iYiOh2nLTKT9evX09/fz5QpU5rezst3ktRimzZtYuLEiWOmkAAigokTJw777NBSkqQ2GEuFNOCVfM6WkiSpGN5TkqQ26557+4i+35r5s0b0/XbffXeee+65EX3PZnmmJEkqhqUkSTu5uXPnct111708f8UVVzBv3jxOOOEEZs6cyfTp07n11lu32u7ee+/llFNOeXn+oosuYsGCBQD09fXxjne8g6OPPpp3v/vdPPnkkyOS1VKSpJ3c7NmzWbhw4cvzCxcu5Nxzz2XRokU8+OCDLF68mMsuu4zMbOr9Nm/ezMUXX0xvby99fX2cf/75XH755SOS1XtKkrSTmzFjBs888wxr165l3bp17L333uy///7MmTOH++67j1122YUnnniCp59+mv3337/h+z366KOsWrWKE088EYAtW7YwadKkEclqKUnSGHDGGWfQ29vLU089xezZs7nxxhtZt24dfX19jBs3ju7u7q3+T1FXVxcvvfTSy/MD6zOTadOmsWTJkhHP6eU7SRoDZs+ezc0330xvby9nnHEGGzZsYL/99mPcuHEsXryYxx/f+rdJHHTQQTz88MO88MILPPvss9x9990AvOlNb2LdunUvl9LmzZtZvXr1iOT0TEmS2mykH+FuxrRp09i4cSMHHHAAkyZN4pxzzuHUU09l+vTp9PT0MHXq1K22OfDAAznzzDM5/PDDmTJlCjNmzABgt912o7e3l0suuYQNGzbw4osvcumllzJt2rQdzhnN3tjS1np6etJf8iepkUceeYRDDz200zE6YqjPPSL6MrNnqPFevpMkFcNSkiQVw1KSpDYYi7dKXsnnbClJUouNHz+e9evXj6liGvh9SuPHjx/Wdj59J0ktNnnyZPr7+1m3bl2no7TVwG+eHQ5LSZJabNy4ccP67atjmZfvJEnFsJQkScWwlCRJxfCe0g5Y+cSGEf8NkpJUulb+mCTPlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFGJWlFBF3RMRe1fQlEfFIRNwYEe+JiLkNtv1J9dodEWe3I68kqTldnQ7wSmTmyXWzHwPelZn91fxtDbY9tprsBs4GbhrxgJKkV6RlZ0oR8ZqIuD0ilkfEqoiYHRFrIuKzEbEyIh6IiDdUY/eNiP+IiJ9VH39cLd89Ir5cjV8REadVy9dExD4RcT1wMHBnRMyJiPMi4tpqzGsjYlG1/+URcWy1/Lkq4nzguIhYVm17X0QcVZf/RxFxZKuOjyRpa628fHcSsDYzj8zMw4HvVMs3ZOZ04Frg6mrZ54GrMvPNwGnAF6rlfz8wPjOPAO6p30FmXgCsBY7PzKsG7f8a4AeZeSQwE1g9aP1c4IeZeVS17ReB8wAi4o3A+MxcPviTioiPRsTSiFi65fkNwzkekqQGWllKK4ETI+IzEXFcZg78Df6Nutdjqul3AddGxDJql9/2iIjdq+XXDbxhZv56GPv/U+Dfqu221O1/W74JnBIR44DzgQVDDcrMGzKzJzN7dn31nsOII0lqpGX3lDLzsYiYCZwMzIuIuwdW1Q+rXncB3paZm+rfIyJaFW8rmfl8RNwFvBc4Ezi6bTuXJAGtvaf0OuD5zPw6cCW1S2gAs+tel1TT3wMurtt24N7OXcDH65bvPYwIdwMXVtvtGhGDT2s2AhMGLfsCtct+PxvmWZkkaQS08vLddOCB6pLcp4F51fK9I2IF8AlgTrXsEqCnepjhYeCCavm8avyqiFgOHD+M/X8COD4iVgJ9wGGD1q8AtlQPQcwByMw+4DfAl4fziUqSRkZkZuNRI7WziDVAT2b+qm07HYbq7O5eYGpmvtRo/KsmHZKTzr260TBJ2qmsmT9rh7aPiL7M7Blq3aj8z7OtEBEfAn4KXN5MIUmSRl5b//NsZna3c3/DkZlfBb7a6RySNJZ5piRJKoalJEkqhqUkSSqGpSRJKoalJEkqhqUkSSqGpSRJKoalJEkqhqUkSSqGpSRJKoalJEkqhqUkSSqGpSRJKoalJEkqhqUkSSqGpSRJKoalJEkqhqUkSSqGpSRJKkZXpwOMZtMP2JOl82d1OoYk7TQ8U5IkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFaNreysj4s+3tz4zvzWycSRJY9l2Swk4tXrdDzgWuKeaPx74CWApSZJGzHZLKTM/DBAR3wMOy8wnq/lJwIKWp5MkjSnN3lM6cKCQKk8Dr29BHknSGNbo8t2AuyPiu8A3qvnZwPdbE0mSNFY1VUqZeVH10MNx1aIbMnNR62JJksaiZs+UBp6088EGSVLLNHok/EeZ+faI2Ahk/SogM3OPlqaTJI0pjZ6+e3v1OqE9cSRJY5k/0UGSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVIyuTgcYzVY+sYHuubd3OoakUWzN/FmdjlAUz5QkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxRiVpRQRP6leuyPi7LrlPRFxTYNtL4iID1XT50XE61qbVpLUrK5OB3glMvPYarIbOBu4qVq+FFjaYNvr62bPA1YBa0c8pCRp2Fp2phQRr4mI2yNieUSsiojZEXF0RPwgIvoi4rsRMakae29EfCYiHoiIxyLiuGr5tGrZsohYERGHVMufq3YzHziuWj8nIt4ZEf8VEbtExJqI2Ksuz39HxGsj4oqI+NuIOB3oAW6stp8VEf9ZN/7EiFjUquMjSdpaKy/fnQSszcwjM/Nw4DvAvwKnZ+bRwJeAf6ob35WZbwEuBT5dLbsA+HxmHkWtQPoH7WMu8MPMPCozrxpYmJkvAbcC7weIiLcCj2fm03VjeqmdVZ1Tvf8dwNSI2Lca8uEq4x+IiI9GxNKIWLrl+Q3DPyqSpG1qZSmtBE6szoCOAw4EDgfuiohlwKeAyXXjv1W99lG7LAewBPi7iPgkcFBm/m4Y+78FmF1Nn1XNb1NmJvA14IPVGdYxwJ1DjLshM3sys2fXV+85jDiSpEZadk8pMx+LiJnAycA84B5gdWYes41NXqhetwzkysybIuKnwCzgjoj468y8p8kIS4A3VGc+76syNPJl4NvAJuCbmflik/uSJI2AVt5Teh3wfGZ+HbgSeCuwb0QcU60fFxHTGrzHwcAvM/Maapfjjhg0ZCMwYahtqzOfRcDngEcyc/0Qw/5g+8xcS+2hh09RKyhJUhu18um76cCVEfESsBm4EHgRuCYi9qz2fTWwejvvcSbwFxGxGXgK+OdB61cAWyJiObAAeGjQ+luAn1F7ym4oC4DrI+J3wDHV5cEbgX0z85EmPkdJ0giK2gmFBkTEtcBDmfnFRmNfNemQnHTu1W1IJWlntWb+rE5HaLuI6MvMnqHWjcr/p9QqEdEH/Ba4rNNZJGksspTqVI+qS5I6ZFT+mCFJ0s7JUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFaOr0wFGs+kH7MnS+bM6HUOSdhqeKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKYSlJkophKUmSimEpSZKKEZnZ6QyjVkRsBB7tdI4G9gF+1ekQDZhxZJhxZJSesfR80DjjQZm571ArulqTZ8x4NDN7Oh1ieyJiqRl3nBlHhhl3XOn5YMcyevlOklQMS0mSVAxLacfc0OkATTDjyDDjyDDjjis9H+xARh90kCQVwzMlSVIxLCVJUjEspSZExEkR8WhE/Dwi5g6x/lURcUu1/qcR0V1gxj+JiAcj4sWIOL3d+ZrM+DcR8XBErIiIuyPioAIzXhARKyNiWUT8KCIOKy1j3bjTIiIjoq2PDzdxDM+LiHXVMVwWEX/ZznzNZKzGnFl9P66OiJtKyxgRV9Udw8ci4tkCM74+IhZHxEPVn+uTG75pZvqxnQ9gV+AXwMHAbsBy4LBBYz4GXF9NnwXcUmDGbuAI4KvA6YUex+OBV1fTFxZ6HPeom34P8J3SMlbjJgD3AfcDPSXlA84Drm339+AwMx4CPATsXc3vV1rGQeMvBr5UWkZqDzxcWE0fBqxp9L6eKTX2FuDnmfnLzPw/4GbgvYPGvBf4SjXdC5wQEVFSxsxck5krgJfamKteMxkXZ+bz1ez9wOQCM/6mbvY1QLufFGrm+xHgH4HPAJvaGY7m83VSMxn/CrguM38NkJnPFJix3geAb7Ql2e81kzGBParpPYG1jd7UUmrsAOB/6+b7q2VDjsnMF4ENwMS2pBu0/8pQGTttuBk/AtzZ0kRbaypjRHw8In4BfBa4pE3ZBjTMGBEzgQMz8/Z2Bqs0+3U+rbqc0xsRB7Yn2suayfhG4I0R8eOIuD8iTmpbupqm/7xUl7mnAPe0IVe9ZjJeAXwwIvqBO6id0W2XpaTiRMQHgR7gyk5nGUpmXpeZfwR8EvhUp/PUi4hdgM8Bl3U6y3Z8G+jOzCOAu/j9VYaSdFG7hPdOamch/x4Re3U00badBfRm5pZOBxnCB4AFmTkZOBn4WvU9uk2WUmNPAPX/kptcLRtyTER0UTtNXd+WdIP2XxkqY6c1lTEi3gVcDrwnM19oU7YBwz2ONwPva2mirTXKOAE4HLg3ItYAbwNua+PDDg2PYWaur/vafgE4uk3ZBjTzde4HbsvMzZn5P8Bj1EqqXYbzvXgW7b90B81l/AiwECAzlwDjqf2w1m1r542x0fhB7V9Mv6R2ejxwM2/aoDEf5w8fdFhYWsa6sQvozIMOzRzHGdRunB5S8Nf6kLrpU4GlpWUcNP5e2vugQzPHcFLd9PuB+0s7hsBJwFeq6X2oXaaaWFLGatxUYA3VD0Io8DjeCZxXTR9K7Z7SdrO29ZMYrR/UTjsfq/7CvLxa9g/U/jUPtfb/JvBz4AHg4AIzvpnav/5+S+0sbnWBGb8PPA0sqz5uKzDj54HVVb7F2yuETmUcNLatpdTkMfyX6hgur47h1NKOIRDULoM+DKwEziotYzV/BTC/3dmGcRwPA35cfa2XAX/W6D39MUOSpGJ4T0mSVAxLSZJUDEtJklQMS0mSVAxLSZJUDEtJklQMS0mSVIz/B76dmY+MtrhbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "custom_metrics_pandas = subscription.monitoring.get_table_content(monitor_uid=monitor_uid)\n",
    "\n",
    "%matplotlib inline\n",
    "custom_metrics_pandas.plot.barh(x='id', y='value');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Payload analytics <a name=\"analytics\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run data distributions calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===============================================================================\n",
      "\n",
      " Waiting for end of data distribution run fbe700c2-bf8a-46c9-9c94-576416182a53 \n",
      "\n",
      "===============================================================================\n",
      "\n",
      "\n",
      "\n",
      "completed\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "start_date = \"2018-01-01T00:00:00.00Z\"\n",
    "end_date = datetime.utcnow().isoformat() + \"Z\"\n",
    "\n",
    "sex_distribution = subscription.payload_logging.data_distribution.run(\n",
    "            start_date=start_date,\n",
    "            end_date=end_date,\n",
    "            group=['predictedLabel', 'Sex'],\n",
    "            agg=['count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get data distributions as pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predictedLabel</th>\n",
       "      <th>Sex</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>No Risk</td>\n",
       "      <td>female</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No Risk</td>\n",
       "      <td>male</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Risk</td>\n",
       "      <td>female</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Risk</td>\n",
       "      <td>male</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  predictedLabel     Sex  count\n",
       "0        No Risk  female     68\n",
       "1        No Risk    male    101\n",
       "2           Risk  female      4\n",
       "3           Risk    male     35"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sex_distribution_run_uid = sex_distribution['id']\n",
    "distributions_pd = subscription.payload_logging.data_distribution.get_run_result(run_id=sex_distribution_run_uid, format='pandas')\n",
    "distributions_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEiCAYAAAACg5K6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAYrklEQVR4nO3dfZhedX3n8fcHgqQgiMBIQwZ3ssqzkAZHBFN3QVSgxcL6wEKxhspeqSvrQ3WlorJSXbx0y2oFVt0oNLgCiojC2ocVLKxalRAIz4SSYoChWKYgUXCDiXz3jzmhw2RCZuaeyT1z8n5d11z3Ob/zO+f+Tq7kc//yu89DqgpJUrts0+0CJEmTz3CXpBYy3CWphQx3SWohw12SWshwl6QWmtXtAgB233336uvr63YZkjSj3HTTTf9cVT2jbZsW4d7X18fy5cu7XYYkzShJ7t/UNqdlJKmFDHdJaiHDXZJaaFrMuUvSBuvWrWNgYIC1a9d2u5RpY/bs2fT29rLddtuNeR/DXdK0MjAwwE477URfXx9Jul1O11UVjz76KAMDA8ybN2/M+zktI2laWbt2LbvttpvB3kjCbrvtNu7/yRjukqYdg/3ZJvLnYbhL0gjnnXce+++/P6eccsqUHP/ss8/m3HPPnZJjb+Cc+3ic/YJuV9AuZ6/pdgWaAfo++JeTerzVn/zdzfb53Oc+x7XXXktvb++kvveW5MhdkoZ5xzvewX333cexxx7LOeecw9vf/nYOPfRQFixYwFVXXQXA0qVLOeGEE3jd615HX18fF1xwAZ/+9KdZsGABhx12GI899hgAX/ziF3nFK17B/PnzedOb3sQvf/nLjd7vH/7hHzjmmGN4+ctfzqtf/WpWrlw5Kb+H4S5Jw3zhC19gzz335LrrruPJJ5/kNa95DcuWLeO6667jAx/4AE8++SQAd9xxB1deeSU33ngjH/7wh9lhhx1YsWIFhx9+OF/+8pcBeOMb38iNN97Irbfeyv7778+FF1640fstXryY888/n5tuuolzzz2Xd77znZPyezgtI0mb8J3vfIerr776mfnxtWvX8sADDwBw5JFHstNOO7HTTjvxghe8gDe84Q0AHHTQQdx2223A0AfARz7yER5//HGeeOIJjj766Gcd/4knnuCHP/whb3nLW55pe+qppyaldsNdkjahqvjGN77Bvvvu+6z2G264ge233/6Z9W222eaZ9W222Yb169cDcOqpp/Ktb32L+fPns3TpUq6//vpnHefpp59ml1124ZZbbpn02p2WkaRNOProozn//POpKgBWrFgxrv1/8YtfMGfOHNatW8cll1yy0fadd96ZefPm8fWvfx0Y+jC59dZbOy8cw12SNumss85i3bp1HHzwwRx44IGcddZZ49r/4x//OK985StZuHAh++2336h9LrnkEi688ELmz5/PgQce+MyXtp3Khk+kburv768ZcT93T4WcXJ4KqVHcfffd7L///t0uY9oZ7c8lyU1V1T9a/82O3JNclOSRJHeMaH9XkpVJ7kzy34a1n5lkVZJ7khy98RElSVNtLF+oLgUuAL68oSHJkcDxwPyqeirJi5r2A4CTgAOBPYFrk+xTVb+e7MIlSZu22ZF7VX0PeGxE838EPllVTzV9Hmnajwe+WlVPVdVPgFXAoZNYryRpDCb6heo+wKuT3JDk/yZ5RdM+F3hwWL+Bpm0jSRYnWZ5k+eDg4ATLkCSNZqLhPgvYFTgM+ABwecZ527KqWlJV/VXV39Mz6sO7JUkTNNFwHwCurCHLgKeB3YGHgL2G9ett2iRJW9BEw/1bwJEASfYBngf8M3A1cFKS7ZPMA/YGlk1GoZI03V1//fUcd9xx3S4DGMPZMkkuA44Adk8yAHwUuAi4qDk98lfAoho6Yf7OJJcDdwHrgdM9U0ZSRyb7+pKt5PqKsZwtc3JVzamq7aqqt6ourKpfVdVbq+plVXVIVf3tsP7nVNVLqmrfqvrrqS1fkibX6tWr2W+//Tj11FPZZ599OOWUU7j22mtZuHAhe++9N8uWLWPZsmUcfvjhLFiwgFe96lXcc889Gx3nySefHPV2wVuKtx+QpBFWrVrF+9//flauXMnKlSu59NJL+cEPfsC5557LJz7xCfbbbz++//3vs2LFCj72sY/xoQ99aKNjnHPOOZu8XfCW4F0hJWmEefPmcdBBBwFw4IEHctRRR5GEgw46iNWrV7NmzRoWLVrEvffeSxLWrVu30TE2dbvgLXVrBcNdkkbY3O18zzrrLI488ki++c1vsnr1ao444oiNjrGp2wVvKU7LSNI4rVmzhrlzh67PXLp06ah9Or1dcKcMd0kapzPOOIMzzzyTBQsWPPNgjpE6vV1wp7zl73h4y9/JtZWckqbx8Za/o5v0W/5KkmYew12SWshwl6QWMtwlTTvT4bvA6WQifx6Gu6RpZfbs2Tz66KMGfKOqePTRR5k9e/a49vMiJknTSm9vLwMDA/gQn38xe/Zsent7x7WP4S5pWtluu+2YN29et8uY8ZyWkaQWMtwlqYU2G+5JLkrySPNgjpHb3p+kkuzerCfJeUlWJbktySFTUbQk6bmNZeS+FDhmZGOSvYDXAw8Maz6WoUfr7Q0sBj7feYmSpPEay5OYvgc8NsqmzwBnAMPPVzoe+HLz4OwfA7skmTMplUqSxmxCc+5JjgceqqpbR2yaCzw4bH2gaZMkbUHjPhUyyQ7AhxiakpmwJIsZmrrhxS9+cSeHkiSNMJGR+0uAecCtSVYDvcDNSX4TeAjYa1jf3qZtI1W1pKr6q6q/p6dnAmVIkjZl3OFeVbdX1Yuqqq+q+hiaejmkqn4KXA28rTlr5jBgTVU9PLklS5I2ZyynQl4G/AjYN8lAktOeo/tfAfcBq4AvAu+clColSeOy2Tn3qjp5M9v7hi0XcHrnZUmSOuEVqpLUQoa7JLWQ4S5JLWS4S1ILGe6S1EKGuyS1kOEuSS1kuEtSCxnuktRChrsktZDhLkktZLhLUgsZ7pLUQoa7JLWQ4S5JLWS4S1ILjeVJTBcleSTJHcPa/izJyiS3Jflmkl2GbTszyaok9yQ5eqoKlyRt2lhG7kuBY0a0XQO8rKoOBv4eOBMgyQHAScCBzT6fS7LtpFUrSRqTzYZ7VX0PeGxE23eqan2z+mOgt1k+HvhqVT1VVT9h6Fmqh05ivZKkMZiMOfe3A3/dLM8FHhy2baBp20iSxUmWJ1k+ODg4CWVIkjboKNyTfBhYD1wy3n2raklV9VdVf09PTydlSJJGmDXRHZOcChwHHFVV1TQ/BOw1rFtv0yZJ2oImNHJPcgxwBvB7VfXLYZuuBk5Ksn2SecDewLLOy5QkjcdmR+5JLgOOAHZPMgB8lKGzY7YHrkkC8OOqekdV3ZnkcuAuhqZrTq+qX09V8ZKk0W023Kvq5FGaL3yO/ucA53RSlCSpM16hKkktZLhLUgsZ7pLUQoa7JLWQ4S5JLWS4S1ILGe6S1EKGuyS1kOEuSS1kuEtSCxnuktRChrsktZDhLkktZLhLUgsZ7pLUQpsN9yQXJXkkyR3D2nZNck2Se5vXFzbtSXJeklVJbktyyFQWL0ka3VhG7kuBY0a0fRD4blXtDXy3WQc4lqFH6+0NLAY+PzllSpLGY7PhXlXfAx4b0Xw8cHGzfDFwwrD2L9eQHwO7JJkzWcVKksZmonPue1TVw83yT4E9muW5wIPD+g00bZKkLajjL1SrqoAa735JFidZnmT54OBgp2VIkoaZaLj/04bplub1kab9IWCvYf16m7aNVNWSquqvqv6enp4JliFJGs1Ew/1qYFGzvAi4alj725qzZg4D1gybvpEkbSGzNtchyWXAEcDuSQaAjwKfBC5PchpwP3Bi0/2vgN8BVgG/BP5wCmqWJG3GZsO9qk7exKajRulbwOmdFiVJ6oxXqEpSCxnuktRChrsktZDhLkktZLhLUgsZ7pLUQoa7JLWQ4S5JLWS4S1ILGe6S1EKGuyS1kOEuSS1kuEtSCxnuktRChrsktZDhLkkt1FG4J/njJHcmuSPJZUlmJ5mX5IYkq5J8LcnzJqtYSdLYTDjck8wF3g30V9XLgG2Bk4BPAZ+pqpcCPwNOm4xCJUlj1+m0zCzgN5LMAnYAHgZeA1zRbL8YOKHD95AkjdOEw72qHgLOBR5gKNTXADcBj1fV+qbbADB3tP2TLE6yPMnywcHBiZYhSRpFJ9MyLwSOB+YBewI7AseMdf+qWlJV/VXV39PTM9EyJEmj6GRa5rXAT6pqsKrWAVcCC4FdmmkagF7goQ5rlCSNUyfh/gBwWJIdkgQ4CrgLuA54c9NnEXBVZyVKksarkzn3Gxj64vRm4PbmWEuAPwHel2QVsBtw4STUKUkah1mb77JpVfVR4KMjmu8DDu3kuJKkzniFqiS1kOEuSS1kuEtSCxnuktRChrsktZDhLkktZLhLUgsZ7pLUQoa7JLVQR1eobm361l7a7RJaZXW3C5BazJG7JLWQ4S5JLWS4S1ILGe6S1EKGuyS1UEfhnmSXJFckWZnk7iSHJ9k1yTVJ7m1eXzhZxUqSxqbTkftngb+pqv2A+cDdwAeB71bV3sB3m3VJ0hY04XBP8gLg39A8Rq+qflVVjwPHAxc33S4GTui0SEnS+HQycp8HDAJ/kWRFki8l2RHYo6oebvr8FNij0yIlSePTSbjPAg4BPl9VC4AnGTEFU1UF1Gg7J1mcZHmS5YODgx2UIUkaqZNwHwAGquqGZv0KhsL+n5LMAWheHxlt56paUlX9VdXf09PTQRmSpJEmHO5V9VPgwST7Nk1HAXcBVwOLmrZFwFUdVShJGrdObxz2LuCSJM8D7gP+kKEPjMuTnAbcD5zY4XtIksapo3CvqluA/lE2HdXJcSVJnfEKVUlqIcNdklrIcJekFjLcJamFDHdJaiHDXZJayHCXpBYy3CWphQx3SWohw12SWshwl6QWMtwlqYUMd0lqIcNdklrIcJekFjLcJamFOg73JNsmWZHk2836vCQ3JFmV5GvNU5okSVvQZIzc3wPcPWz9U8BnquqlwM+A0ybhPSRJ49BRuCfpBX4X+FKzHuA1wBVNl4uBEzp5D0nS+HU6cv9z4Azg6WZ9N+DxqlrfrA8Ac0fbMcniJMuTLB8cHOywDEnScBMO9yTHAY9U1U0T2b+qllRVf1X19/T0TLQMSdIoZnWw70Lg95L8DjAb2Bn4LLBLklnN6L0XeKjzMiU9p7Nf0O0K2uXsNd2uoGMTHrlX1ZlV1VtVfcBJwN9W1SnAdcCbm26LgKs6rlKSNC5TcZ77nwDvS7KKoTn4C6fgPSRJz6GTaZlnVNX1wPXN8n3AoZNxXEnSxHiFqiS1kOEuSS1kuEtSCxnuktRChrsktZDhLkktZLhLUgsZ7pLUQoa7JLWQ4S5JLWS4S1ILGe6S1EKGuyS1kOEuSS1kuEtSC3XyDNW9klyX5K4kdyZ5T9O+a5JrktzbvL5w8sqVJI1FJyP39cD7q+oA4DDg9CQHAB8EvltVewPfbdYlSVtQJ89Qfbiqbm6WfwHcDcwFjgcubrpdDJzQaZGSpPGZlDn3JH3AAuAGYI+qerjZ9FNgj8l4D0nS2HUc7kmeD3wDeG9V/Xz4tqoqoDax3+Iky5MsHxwc7LQMSdIwHYV7ku0YCvZLqurKpvmfksxpts8BHhlt36paUlX9VdXf09PTSRmSpBE6OVsmwIXA3VX16WGbrgYWNcuLgKsmXp4kaSJmdbDvQuAPgNuT3NK0fQj4JHB5ktOA+4ETOytRkjReEw73qvoBkE1sPmqix5Ukdc4rVCWphQx3SWohw12SWshwl6QWMtwlqYUMd0lqIcNdklrIcJekFjLcJamFDHdJaiHDXZJayHCXpBYy3CWphQx3SWohw12SWqiTh3VImib61l7a7RJaZXW3C5gEUzZyT3JMknuSrErywal6H0nSxqYk3JNsC/wP4FjgAODkJAdMxXtJkjY2VSP3Q4FVVXVfVf0K+Cpw/BS9lyRphKmac58LPDhsfQB45fAOSRYDi5vVJ5LcM0W1bI12B/6520VsTj7V7QrUBf7dnFz/alMbuvaFalUtAZZ06/3bLMnyqurvdh3SSP7d3HKmalrmIWCvYeu9TZskaQuYqnC/Edg7ybwkzwNOAq6eoveSJI0wJdMyVbU+yX8C/g+wLXBRVd05Fe+lUTndpenKv5tbSKqq2zVIkiaZtx+QpBYy3CWphQz3Fkiy/Shtu3ajFknTg+HeDlcm2W7DSpI5wDVdrEd6RpKXj9J2XDdq2ZoY7u3wLeDyJNsm6WPoLKUzu1qR9C++mORlG1aSnAyc1cV6tgqeLdMSSU4HjgH6gD+qqh92tyJpSJJ/DVwB/D7wauBtwHFVtaarhbWc4T6DJXnf8FWG/tHcBqwAqKpPd6MuaaQk+zD0P8wHgH9XVf+vyyW1ng/rmNl2GrF+5SbapS0uye3A8NHjrgxd1HhDEqrq4O5UtnVw5N4ySbYBnl9VP+92Ldq6JdnkHQsBqur+LVXL1sgvVFsgyaVJdk6yI3AHcFeSD3S7Lm3dqur+JsBnAT9tlucx9GwH59unmOHeDgc0I/UTgL9m6B/QH3S3JOkZ3wB+neSlDN1bZi/Ah75OMcO9HbZrznM/Abi6qtbx7LlOqZuerqr1wBuB86vqA8CcLtfUeoZ7O/xPhh7YviPwvWau0zl3TRfrmnPb3wZ8u2nb7jn6axL4hWpLJZnVjJakrkpyAPAO4EdVdVmSecCJVTVzHmY3AxnuM1iSt1bVV0ac7/4Mz3OXtl6e5z6z7di8el67pp0kl1fViaOc7w7gee5TzJF7SyXZsaqe7HYd2nolmVNVD2/qfHfPc59afqE6wyWZm6S/eVYtSV6U5BPAvV0uTVu5qnq4eb1/+A/wIPDb3a2u/Qz3GSzJe4FbgPOBHyf5D8DdwG8AG91mVdqSmgvrzkxyQZLXZ8i7gPuAE7tdX9s5LTODJbkL+O2qeizJi4G/BxZW1U1dLk0iyVXAz4AfAUcBL2LoBnfvqapbulnb1sBwn8GS3FxVhwxbv7Wq5nezJmmDJLdX1UHN8rbAw8CLq2ptdyvbOni2zMzWm+S8Yetzhq9X1bu7UJO0wboNC1X16yQDBvuW48h9Bkuy6Lm2V9XFW6oWaaQkvwY2nLEVhr4L+mWzXFW1c7dq2xoY7pLUQp4tI0ktZLhLUgsZ7pLUQoZ7CyTpTfLNJINJHknyjSS93a5LUvcY7u3wF8DVDD0AYU/gfzdtkrZSni3TAkluqarf2lybpK2HI/d2eDTJW5Ns2/y8FXi020VJ6h5H7i3Q3FL1fOBwhu6b/UPg3VX1QFcLk9Q1hrsktZD3lpnBkvyX59hcVfXxLVaMpGnFkfsMluT9ozTvCJwG7FZVz9/CJUmaJgz3lkiyE/AehoL9cuC/V9Uj3a1KUrc4LTPDJdkVeB9wCnAxcEhV/ay7VUnqNsN9BkvyZ8AbgSXAQVX1RJdLkjRNOC0zgyV5GngKWM/QKZDPbML7ZUtbNcNdklrIK1QlqYUMd0lqIcNdklrIcFerJXmied0zyRWb6fveJDuM8/hHJPl2s3xqkgvGse/qJLuPo/+4jq+tm+GuGSfJtuPdp6r+sarevJlu7wXGFe7SdGW4a1pJ0pdkZZJLktyd5IokOzSj3E8luRl4S5KXJPmbJDcl+X6S/Zr95yX5UZLbk/zXEce9o1neNsm5Se5IcluSdyV5N0MPOrkuyXVNv9c3x7o5ydeTPL9pP6ap8WaGrjPY3O/0+STLk9yZ5E9HbD6jqXVZkpc2/Xuap2nd2PwsnIQ/Wm1lDHdNR/sCn6uq/YGfA+9s2h+tqkOq6qsMXbj1rqp6OfCfgc81fT4LfL6qDgIe3sTxFwN9wG9V1cHAJVV1HvCPwJFVdWQzXfIR4LVVdQiwHHhfktnAF4E3AC8HfnMMv8+Hq6ofOBj4t0kOHrZtTVPrBcCfD/sdPlNVrwDeBHxpDO8hPYtXqGo6erCq/q5Z/grw7mb5awDNCPpVwNeTbNhn++Z1IUOBCPC/gE+NcvzXAl+oqvUAVfXYKH0OAw4A/q55j+cBPwL2A35SVfc2tXyFoQ+L53JiksUM/Xub0xz3tmbbZcNePzOsvgOG/W47b/hfgzRWhrumo5FX1m1Yf7J53QZ4/DkeIzgZV+YFuKaqTn5WYzKuRxcmmcfQ/yxeUVU/S7IUmD2sS42yvA1wWFWtHXGs8by1tnJOy2g6enGSw5vl3wd+MHxjVf0c+EmStwBkyPxm898BJzXLp2zi+NcAf5RkVrP/rk37L4CdmuUfAwuHzYPvmGQfYCXQl+QlTb9nhf8odmboQ2lNkj2AY0ds//fDXn/ULH8HeNeGDuP9QJHAcNf0dA9wepK7gRcCnx+lzynAaUluBe4Ejm/a39PsezswdxPH/xLwAHBbs//vN+1LgL9Jcl1VDQKnApcluY1mSqYZTS8G/rL5QnXkbZVPTTKw4YehZ9muYOhD4VKGPnyGe2Fz/PcAf9y0vRvob77svQt4xyZ+D2mTvLeMppUkfcC3q+plXS5FmtEcuUtSCzlyl6QWcuQuSS1kuEtSCxnuktRChrsktZDhLkktZLhLUgv9fyKOCYFaUtPYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "subscription.payload_logging.data_distribution.show_chart(sex_distribution_run_uid);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===============================================================================\n",
      "\n",
      " Waiting for end of data distribution run ebc82ff0-71b1-4e57-b2a1-73e7cacf8296 \n",
      "\n",
      "===============================================================================\n",
      "\n",
      "\n",
      "\n",
      "completed\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "credit_history_distribution = subscription.payload_logging.data_distribution.run(\n",
    "            start_date=start_date,\n",
    "            end_date=end_date,\n",
    "            group=['predictedLabel', 'CreditHistory'],\n",
    "            agg=['count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEiCAYAAAACg5K6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXgUVdr///cNBkNEAQEVhZmgoyxZgQQiS2RRQEDZBxWUgMioKKDjghsgLg8+8nMBUR8cCeqAoijgD2UcFRhABEw0bCIqMxFRRjFMkH2R8/2jOz0he9IhTSqf13VxpevUqaq7Y7xTOX3qPuacQ0REvKVaqAMQEZHyp+QuIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQaeFOgCA+vXru8jIyFCHISJSqaSnp//inGtQ0L5TIrlHRkaSlpYW6jBERCoVM/uusH0alhER8SAldxERD1JyFxHxoFNizF2ksjh69Cg7duzg0KFDoQ5FqpDw8HAaNWpEWFhYiY9RchcphR07dnDmmWcSGRmJmYU6HKkCnHNkZWWxY8cOmjRpUuLjNCwjUgqHDh2iXr16SuxSYcyMevXqlfqvRSV3kVJSYpeKVpafOSV3EREP0ph7Kcy4eWmoQ/CU0S92CXUIQYsc/165ni9zSq8yH5vzMGD9+vWpVasW+/btK8fISnbddu3asXr1ajIzM1m9ejXXXXfdSb/+hAkTSE5O5vLLLz+hffny5UydOpXFixcXeNykSZOoVasWd911V1DXT0lJoXfv3gwcODCo85Q33bmLSD7Hjh0r03GrV68GIDMzk7lz55ZnSIWaPHlyvsQuSu4ilVLfvn1p3bo1UVFRzJw5s9THP/HEE8TExBAXF8f48eMB6NSpE+PGjSMhIYFnn32WXbt2MWDAABITE0lMTOSTTz4BICsri27duhEVFcXIkSPJvVRnrVq1ABg/fjwrV64kPj6ep59+ms2bN9OmTRvi4+OJjY3lm2++KTCuzMxMmjVrxpAhQ2jevDkDBw7kwIEDgC+JJyYmEh0dzahRowLXTUlJYf78+QD87W9/o1mzZrRq1Yp33nmn2O/D+vXrufTSS7n44ot56aWXANi3bx9du3alVatWxMTEsGjRokD/V199ldjYWOLi4rj++uvzne+hhx4iJSWF3377rdhrn2walhGphGbNmsXZZ5/NwYMHSUxMZMCAASU+dsmSJSxatIi1a9cSERHB7t27A/uOHDkSqPN03XXXcccdd9ChQwe2b99O9+7d2bJlCw8//DAdOnRgwoQJvPfee7z88sv5rjFlypQThkRuv/12xo4dy5AhQzhy5EiRyW/r1q28/PLLtG/fnhEjRvD8889z1113cdtttzFhwgQArr/+ehYvXsxVV10VOO7QoUPcdNNNLF26lD/84Q8MHjy42O/Fhg0bWLNmDfv376dly5b06tWLc845hwULFnDWWWfxyy+/kJSUxNVXX82XX37Jo48+yurVq6lfv/4J3zeAu+++m71795KamnpKfOiuO3eRSmjatGnExcWRlJTE999/X+idcEE++ugjhg8fTkREBABnn312YF/uhPjRRx9x2223ER8fz9VXX82vv/7Kvn37WLFiBUOHDgWgV69e1K1bt9hrXnrppTz++OM88cQTfPfdd9SsWbPQvo0bN6Z9+/YADB06lFWrVgGwbNky2rZtS0xMDEuXLmXz5s0nHPfVV1/RpEkTLr74YswsEGNR+vTpQ82aNalfvz6dO3dm3bp1OOe4//77iY2N5fLLL+eHH37gp59+YunSpQwaNIj69evn+7498sgj7NmzhxdffPGUSOygO3eRSmf58uV89NFHfPrpp0RERNCpU6dye2L2jDPOCLw+fvw4a9asITw8POjzXnfddbRt25b33nuPnj178n//93906VLwB+p5k6OZcejQIW699VbS0tJo3LgxkyZNKpf3XNC15syZw65du0hPTycsLIzIyMhir5WYmEh6ejq7d+8+IemHku7cRSqZPXv2ULduXSIiIvjqq69Ys2ZNqY6/4oorSE1NDYxl5x1eyNGtWzemT58e2M7IyAAgOTk58GHpkiVL+M9//pPv2DPPPJO9e/cGtv/5z39y4YUXMmbMGPr06cOGDRsKjW/79u18+umnAMydO5cOHToEkmv9+vXZt29fYIw9t2bNmpGZmcm2bdsAeP311wv/JvgtWrSIQ4cOkZWVxfLly0lMTGTPnj2cc845hIWFsWzZMr77zldVt0uXLrz11ltkZWUBJ37fevTowfjx4+nVq9cJ7zuUdOcuEoRgpi6WVY8ePXjxxRdp3rw5TZs2JSkpqdTHZ2RkkJCQQI0aNejZsyePP/54vn7Tpk1j9OjRxMbGcuzYMZKTk3nxxReZOHEi1157LVFRUbRr147f/e53+Y6NjY2levXqxMXFkZKSwuHDh3nttdcICwvjvPPO4/777y80vqZNmzJjxgxGjBhBixYtuOWWW4iIiOCmm24iOjqa8847j8TExHzHhYeHM3PmTHr16kVERAQdO3YsNtHGxsbSuXNnfvnlFx566CHOP/98hgwZwlVXXUVMTAwJCQk0a9YMgKioKB544AEuu+wyqlevTsuWLZk9e3bgXIMGDWLv3r1cffXVvP/++0UOPVUEy/1Jd4EdzGYBvYGfnXPRudpvB0YDvwHvOefu8bffB9zobx/jnPuguCASEhJcZVisQ/Pcy1dlnOe+ZcsWmjdvHuowPCszM5PevXuzadOmUIdyyinoZ8/M0p1zCQX1L8md+2zgOeDVXCfsDPQB4pxzh83sHH97C+AaIAo4H/jIzC5xzoV+XpCISBVSbHJ3zq0ws8g8zbcAU5xzh/19fva39wHe8Lf/y8y+BdoAn5ZbxCJSIhs3bsw3F/v0009n7dq1IYrov7KysujatWu+9o8//rjc79pTU1N59tlnT2hr3749M2bMKNfrnGrKOuZ+CdDRzB4DDgF3Oec+Ay4Acn+6s8Pflo+ZjQJGAQWO2YlIcGJiYgIfgp5q6tWrV2GxDR8+nOHDh1fItU4lZZ0tcxpwNpAE3A28aaWc3Omcm+mcS3DOJTRoUODi3SIiUkZlTe47gHeczzrgOFAf+AFonKtfI3+biIhUoLIm94VAZwAzuwSoAfwCvAtcY2anm1kT4GJgXXkEKiIiJVfsmLuZvQ50Auqb2Q5gIjALmGVmm4AjwDDnm1O52czeBL4EjgGjNVNGRKTilWS2zLWF7CqwcINz7jHgsWCCEqk0JtUu5/PtKd/zFaNTp05MnTqVhIQEevbsGXjydO7cudx6660n/fovvvgiERER3HDDDSe0FzffPSMjgx9//JGePXsGHUPuevSFefzxx4t88OpUpPIDIh5T1lrs77//PnXq1CE7O5vnn3++nKMq2M0335wvsZdERkYG77///kmIqGAFPcF7qlNyF6mE8tYVT0lJ4eabb6Zt27bcc8897N+/nxEjRtCmTRtatmwZqEl+8OBBrrnmGpo3b06/fv04ePBg4JyRkZH88ssvjB8/nm3bthEfH8/dd9/Nzp07SU5OJj4+nujoaFauXFloXLVq1eKOO+4gKiqKrl27smvXLgBeeuklEhMTiYuLY8CAAYG6NpMmTWLq1KkApKenExcXR1xcXJFz0I8cOcKECROYN28e8fHxzJs3j927d9O3b19iY2NJSkoqsnZNUfXoC6qTP378eA4ePEh8fDxDhgwB4K9//WugPv2f/vSnU6J+e15K7iKVzObNm3n00UdZunQp69evDzygs2PHDlavXs1TTz3FY489RpcuXVi3bh3Lli3j7rvvZv/+/bzwwgtEREQE6rKnp6fnO/+UKVO46KKLyMjI4Mknn2Tu3Ll0796djIwM1q9fT3x8fKGx7d+/n4SEBDZv3sxll13Gww8/DED//v357LPPWL9+Pc2bNy+wBvzw4cOZPn0669evL/L916hRg8mTJzN48GAyMjIYPHgwEydOpGXLlmzYsIHHH3+8yL8GcurRb968mX79+rF9+/bAvlmzZpGenk5aWhrTpk0jKyuLKVOmULNmTTIyMpgzZw5btmxh3rx5fPLJJ2RkZFC9enXmzJlTZMyhoMJhIpVMYXXFBw0aRPXq1QH4+9//zrvvvhu4Kz506BDbt29nxYoVjBkzBvAVzYqNjS32eomJiYwYMYKjR4/St2/fIpN7tWrVAjXhhw4dSv/+/QHYtGkTDz74INnZ2ezbt4/u3bufcFx2djbZ2dkkJycDvsU4lixZUuLvyapVq3j77bcBX/XGrKwsfv31V84666x8fVesWBFYpSlvPfpp06axYMECgECd/Hr16p1w/Mcff0x6enqgeNnBgwc555xzShxrRVFyF/GI3LXYnXO8/fbbNG3aNOjzJicns2LFCt577z1SUlK48847SzxOnvNsY0pKCgsXLiQuLo7Zs2ezfPnyoOMqbyWtk++cY9iwYfzP//xPCKIsOQ3LiFQyRdUVz9G9e3emT58eGE/+4osvgBNrsW/atKnAsem8tdi/++47zj33XG666SZGjhzJ559/Xmhsx48fD9Raz6nFDrB3714aNmzI0aNHCxzCqFOnDnXq1AmsulTcMEfeGDt27Bg4Zvny5dSvX7/Au/a834Pc9eiLqpMfFhbG0aNHAejatSvz58/n5599JbV2794dqPl+KtGdu0gwKnjqIhRcVzyvhx56iHHjxhEbG8vx48dp0qQJixcv5pZbbmH48OE0b96c5s2b07p163zH1qtXj/bt2xMdHc2VV15JdHQ0Tz75JGFhYdSqVYtXX3013zE5zjjjDNatW8ejjz7KOeecw7x58wDfMnRt27alQYMGtG3btsA666mpqYwYMQIzo1u3bkV+Dzp37syUKVOIj4/nvvvuY9KkSYwYMYLY2FgiIiJ45ZVXCj22sHr0RdXJHzVqFLGxsbRq1Yo5c+bw6KOP0q1bN44fP05YWBgzZszg97//fZExV7Ri67lXBNVzr5pUz917atWqxb59+0IdhieVtp67hmVERDxIwzIiUmpt27bl8OHDJ7S99tpr5X7X/sEHH3Dvvfee0NakSZPAjJaiVNU67jmU3EWk1CpqwY/u3bvnmzZZUlW1jnsODcuIiHiQkruIiAcpuYuIeJCSu4iIB5VksY5ZQG/gZ+dcdJ59fwamAg2cc7/411F9FugJHABSnHOFP84mUsnFvBJTrufbOGxjuZ6vIsyePZu0tDSee+65E+qzz549m27dunH++eeHOsQqqSR37rOBHnkbzawx0A3Ynqv5SnxL610MjAJeCD5EEaloZa0Jn7s+++zZs/nxxx/LMywphWKTu3NuBZC/eAU8DdwD5H7EtQ/wqn/h7DVAHTNrWC6RigjgW6WoefPm3HTTTURFRdGtWzcOHjxIRkYGSUlJxMbG0q9fv0DNlIJ8++23XH755cTFxdGqVSu2bdvG8uXL6dixI1dffTUtWrQACq9bnpqayiWXXEKbNm345JNPAufNqc8+f/580tLSGDJkCPHx8Rw8eJDx48fTokULYmNjueuuu07uN0nKNuZuZn2AH5xzeQsvXwB8n2t7h79NRMrRN998w+jRo9m8eTN16tTh7bff5oYbbuCJJ55gw4YNxMTEBGqpF2TIkCGMHj2a9evXs3r1aho29N2Dff755zz77LN8/fXXhdYt37lzJxMnTuSTTz5h1apVfPnll/nOP3DgQBISEpgzZw4ZGRkcOHCABQsWsHnzZjZs2MCDDz540r434lPqh5jMLAK4H9+QTJmZ2Sh8QzeBwj0iUjJNmjQJ1FVv3bo127ZtIzs7m8suuwyAYcOGMWjQoAKP3bt3Lz/88AP9+vUDIDw8PLCvTZs2NGnSBCi8bvnatWvp1KkTDRo0AGDw4MF8/fXXRcZbu3ZtwsPDufHGG+nduze9e/cO4t1LSZTlzv0ioAmw3swygUbA52Z2HvAD0DhX30b+tnycczOdcwnOuYScHxIRKZnTTz898Lp69epkZ2eXy3nz1oQfNmwYGRkZZGRksHXrViZNmlSm85522mmsW7eOgQMHsnjxYnr0yPcxnpSzUid359xG59w5zrlI51wkvqGXVs65fwPvAjeYTxKwxzm3s3xDFpG8ateuTd26dQPrm7722muBu/i8zjzzTBo1asTChQsBOHz4cGBN09wKq1vetm1b/vGPf5CVlcXRo0d56623Cr1OTmnfffv2sWfPHnr27MnTTz9d7FJ6ErySTIV8HegE1DezHcBE51z+BRB93sc3DfJbfFMhq25hB6kSTqWpi6+88go333wzBw4c4MILLyQ1NbXQvq+99hp/+tOfmDBhAmFhYQUm6BYtWhRYtzwpKYlJkyZx6aWXUqdOnUKX3ctZtLtmzZosWbKEPn36cOjQIZxzPPXUU+X2vqVgqudeCqrnXr5Uz12k5FTPXUREVPJXxMtGjx59wjx0gLFjx1bpUrhVhZK7iIdVlYUpJD8Ny4iIeJCSu4iIBym5i4h4kJK7iIgH6QNVkSBsaVa+c96bf7WlXM8HlKiu+sKFC7nkkksC1SCDlZKSQu/evRk4cCAjR47kzjvvLLdzl7fIyEjS0tKoX78+7dq1Y/Xq1WRmZrJ69Wquu+66UIdXZrpzF/G4ktRVX7hwYYHVHcvDX/7ylwpP7GWtR7969WrAV1Z57ty55RlShVNyF6mEnnrqKaKjo4mOjuaZZ54hMzOT6Oj/LpQ2depUJk2aVKK66qtXr+bdd9/l7rvvJj4+nm3btvHSSy+RmJhIXFwcAwYMCNSeSUlJYcyYMbRr144LL7yQ+fPnA74iY7fddhtNmzbl8ssvD9SjAejUqRM5T6DXqlWLBx54gLi4OJKSkvjpp58A2LZtG0lJScTExPDggw9Sq1atIt//E088QUxMDHFxcYwfPz5wnXHjxpGQkMCzzz7Lrl27GDBgAImJiSQmJgbm+2dlZdGtWzeioqIYOXIkuZ/Sz7nu+PHjWblyJfHx8Tz99NNB/bcKFSV3kUomPT2d1NRU1q5dy5o1a3jppZcKXZijJHXV27Vrx9VXX82TTz5JRkYGF110Ef379+ezzz5j/fr1NG/enJdf/m85qZ07d7Jq1SoWL14cSKwLFixg69atfPnll7z66quBO+C89u/fT1JSEuvXryc5OZmXXnoJ8D1YNXbsWDZu3EijRo2KfP9Llixh0aJFrF27lvXr13PPPfcE9h05coS0tDT+/Oc/M3bsWO644w4+++wz3n77bUaOHAnAww8/TIcOHdi8eTP9+vVj+/bt+a4xZcoUOnbsSEZGBnfccUeR8ZyqNOYuUsmsWrWKfv36Bcrz9u/fP1ANsjglrau+adMmHnzwQbKzs9m3bx/du3cP7Ovbty/VqlWjRYsWgTvvFStWcO2111K9enXOP/98unQpuG5QjRo1Atds3bo1H374IQCffvppoErlddddV+RKTR999BHDhw8nIiICgLPPPjuwb/DgwSf0yz3U9Ouvv7Jv3z5WrFjBO++8A0CvXr2oW7duodeqzJTcRTwgOzub48ePB7YPHTpUYL+cuuoff/wx8+fP57nnnmPp0vwF8VJSUli4cCFxcXHMnj2b5cuXB/blriVf2sKDYWFhmBngq0Nf1rHxwuSuR3/8+HHWrFlzwmIkVYmGZUQqmY4dO7Jw4UIOHDjA/v37WbBgAVdeeSU///wzWVlZHD58mMWLFwf6l6Sueu4+4FutqWHDhhw9epQ5c+YUG1NycjLz5s3jt99+Y+fOnSxbtqxU7ykpKYm3334bgDfeeKPIvldccQWpqamBzwF27y5oiWfo1q0b06dPD2xnZGQEYs35sHTJkiUFDmnl/X5URrpzFwnCyZi6WJxWrVqRkpJCmzZtABg5ciSJiYlMmDCBNm3acMEFF9CsWbNA/5LUVb/mmmu46aabmDZtGvPnz+eRRx6hbdu2NGjQgLZt2xab6Pr168fSpUtp0aIFv/vd77j00ktL9Z6eeeYZhg4dymOPPUaPHj2oXbt2oX179OhBRkYGCQkJ1KhRg549e/L444/n6zdt2jRGjx5NbGwsx44dIzk5mRdffJGJEydy7bXXEhUVRbt27Qpc5jM2Npbq1asTFxdHSkpKpRx3Vz33UlA99/Kleu6S48CBA9SsWRMz44033uD1119n0aJFoQ7rlFLaeu4lWYlpFtAb+Nk5F+1vexK4CjgCbAOGO+ey/fvuA24EfgPGOOc+KPvbEZGqID09ndtuuw3nHHXq1GHWrFmhDqnSK8mwzGzgOeDVXG0fAvc5546Z2RPAfcC9ZtYCuAaIAs4HPjKzS5xzv5Vv2CLiJR07dsy3rurGjRu5/vrrT2g7/fTTWbt2bUWGVmkVm9ydcyvMLDJP299zba4BBvpf9wHecM4dBv5lZt8CbYBPyyVaEakyYmJiAh+CSumVx2yZEcAS/+sLgO9z7dvhb8vHzEaZWZqZpe3atascwhARkRxBJXczewA4BhQ/VyoP59xM51yCcy6hQYMGwYQhIiJ5lHkqpJml4Pugtav775SbH4DGubo18rd5Qpflo0MdgsdU/DRCkaqiTHfuZtYDuAe42jl3INeud4FrzOx0M2sCXAysCz5MEREpjZJMhXwd6ATUN7MdwER8s2NOBz70P0q8xjl3s3Nus5m9CXyJb7hmtGbKiJeV97MP5Tn3f8KECSQnJ3P55ZeX2zlPRRkZGfz444/07NkzqPPkrkEfTJ9gZGZm0rt3bzZt2hT0uUoyW+baAppfLqAtp/9jwGPBBCUiwfntt9+YPHlyqY+pXr36SYro5MnIyCAtLS3o5O41qi0jUslkZmbSrFkzhgwZQvPmzRk4cCAHDhwgMjKSe++9l1atWvHWW2+RkpISqLf+8ccf07JlS2JiYhgxYgSHDx8GyHdMQTp16sTYsWOJj48nOjqadet8I63r1q3j0ksvpWXLlrRr146tW7cCvtotuacwdujQgfXr1zNp0iSGDRtGx44d+f3vf88777zDPffcQ0xMDD169ODo0aOA74Gmyy67jNatW9O9e3d27twZiOPee++lTZs2XHLJJaxcuZIjR44wYcIE5s2bR3x8PPPmzeMf//gH8fHxxMfH07Jly0JLJxRVg76wGHKbPHkyiYmJREdHM2rUKJxzbNu2jVatWgX6fPPNN4Htws6Znp5OXFwccXFxzJgxo7j//CWm5C5SCW3dupVbb72VLVu2cNZZZ/H8888DUK9ePT7//HOuueaaQN9Dhw6RkpLCvHnz2LhxI8eOHeOFF14I7C/omLwOHDhARkYGzz//PCNGjACgWbNmrFy5ki+++ILJkydz//33A3DjjTcye/ZsAL7++msOHTpEXFwc4FuUY+nSpbz77rsMHTqUzp07s3HjRmrWrMl7773H0aNHuf3225k/fz7p6emMGDGCBx54IBDHsWPHWLduHc888wwPP/wwNWrUYPLkyQwePJiMjAwGDx7M1KlTmTFjBhkZGaxcuZKaNWsW+J4Kq0FfXAw5brvtNj777DM2bdrEwYMHWbx4MRdddBG1a9cO/HJLTU1l+PDhRZ5z+PDhTJ8+Pd9DXMFS4TCRSqhx48a0b98egKFDhzJt2jTgxHrmObZu3UqTJk245JJLABg2bBgzZsxg3LhxhR6T17XX+kZnk5OT+fXXX8nOzmbv3r0MGzaMb775BjML3HkPGjSIRx55hCeffJJZs2aRkpISOM+VV15JWFgYMTEx/Pbbb/To0QPwPbCUmZnJ1q1b2bRpE1dccQXgGypq2LBh4Pj+/fsDvlrwmZmZBcbavn177rzzToYMGUL//v0LXfyjsBr0xcWQY9myZfzv//4vBw4cYPfu3URFRXHVVVcxcuRIUlNTeeqpp5g3bx7r1q0r9JzZ2dlkZ2eTnJwMwPXXX8+SJUvyXasslNxFKqGcmuh5t3PXMy+pkhxT0PUeeughOnfuzIIFC8jMzKRTp04AREREcMUVV7Bo0SLefPNN0tPTA8fl1IKvVq3aCbXdq1WrxrFjx3DOERUVxaefFvxQe87xRdWCHz9+PL169eL999+nffv2fPDBBydUySxOcTGA76+hW2+9lbS0NBo3bsykSZMCNfQHDBjAww8/TJcuXWjdujX16tXjxx9/LPCc2dnZJY6rtDQsI1IJbd++PZAo5s6dS4cOHQrt27RpUzIzM/n2228BeO2117jssstKdb158+YBvlWgateuTe3atdmzZw8XXOB7AD1nGCbHyJEjGTNmDImJiaVa6ahp06bs2rUr8N6OHj3K5s2bizwmb+31bdu2ERMTw7333ktiYiJfffVVgccVVoO+JDHkJPL69euzb9++wGcbAOHh4XTv3p1bbrmF4cOHF3nOOnXqUKdOHVatWgVQotr5JaU7d5EghKpscdOmTZkxYwYjRoygRYsW3HLLLScsTJFbeHg4qampDBo0iGPHjpGYmMjNN99cquuFh4fTsmVLjh49GqjYeM899zBs2DAeffRRevXqdUL/1q1bc9ZZZwWSW0nVqFGD+fPnM2bMGPbs2cOxY8cYN24cUVFRhR7TuXNnpkyZQnx8PPfddx+rVq1i2bJlVKtWjaioKK688soCjyusBn1JYqhTpw433XQT0dHRnHfeeSQmJp5w7iFDhrBgwQK6detW7DlTU1MZMWIEZhboXx5Uz70UtjRTHe/yFIqFLoJ1KtRzL8+50CXRqVMnpk6dSkJCgWXDC/Tjjz/SqVMnvvrqK6pVq3oDBFOnTmXPnj088sgj5XbOcq/nLiJSGq+++ioPPPAATz31VJVM7P369QvMCgolJXeRSiYyMvKk3LWPHj2aTz755IS2sWPHnrA4dknccMMN3HDDDeUYWXAqui78ggULTsp5S0vJXUQAyvUBmlNJVa0LX/X+ZhIJ0qnwOZVULWX5mVNyFymF8PBwsrKylOClwjjnyMrKIjw8vFTHaVhGpBQaNWrEjh070OphUpHCw8MLfdK2MEruIqUQFhZGkyZNQh2GSLGU3Evhj/fp21WeNoY6ABEPK3bM3cxmmdnPZrYpV9vZZvahmX3j/1rX325mNs3MvjWzDWbWqvAzi4jIyVKSD1RnAz3ytI0HPnbOXQx87N8GuBLf0noXA6OAFxARkQpXbHJ3zq0Adudp7iCG/yEAAA84SURBVAO84n/9CtA3V/urzmcNUMfM8tfKFBGRk6qsUyHPdc7lLE3yb+Bc/+sLgO9z9dvhbxMRkQoU9Dx355vwW+pJv2Y2yszSzCxN08pERMpXWad//GRmDZ1zO/3DLjmLD/4ANM7Vr5G/LR/n3ExgJviqQpYxjgq18V/bQx2CiEiJlPXO/V1gmP/1MGBRrvYb/LNmkoA9uYZvRESkghR7525mrwOdgPpmtgOYCEwB3jSzG4HvgD/6u78P9AS+BQ4ApavULyIi5aLY5O6cu7aQXV0L6OuA0cEGJSIiwVHhMBERD1JyFxHxICV3EREPUnIXEfEgJXcREQ9SchcR8SAldxERD1JyFxHxICV3EREPUnIXEfEgJXcREQ9SchcR8SAldxERD1JyFxHxICV3EREPUnIXEfGgoJK7md1hZpvNbJOZvW5m4WbWxMzWmtm3ZjbPzGqUV7AiIlIyZU7uZnYBMAZIcM5FA9WBa4AngKedc38A/gPcWB6BiohIyQU7LHMaUNPMTgMigJ1AF2C+f/8rQN8gryEiIqVU5uTunPsBmApsx5fU9wDpQLZz7pi/2w7ggoKON7NRZpZmZmm7du0qaxgiIlKAYIZl6gJ9gCbA+cAZQI+SHu+cm+mcS3DOJTRo0KCsYYiISAGCGZa5HPiXc26Xc+4o8A7QHqjjH6YBaAT8EGSMIiJSSsEk9+1AkplFmJkBXYEvgWXAQH+fYcCi4EIUEZHSCmbMfS2+D04/Bzb6zzUTuBe408y+BeoBL5dDnCIiUgqnFd+lcM65icDEPM3/BNoEc14REQmOnlAVEfEgJXcREQ9SchcR8SAldxERD1JyFxHxICV3EREPUnIXEfEgJXcREQ9SchcR8SAldxERD1JyFxHxICV3EREPUnIXEfEgJXcREQ9SchcR8aCgkruZ1TGz+Wb2lZltMbNLzexsM/vQzL7xf61bXsGKiEjJBHvn/izwN+dcMyAO2AKMBz52zl0MfOzfFhGRClTm5G5mtYFk/MvoOeeOOOeygT7AK/5urwB9gw1SRERKJ5g79ybALiDVzL4ws7+Y2RnAuc65nf4+/wbODTZIEREpnWCS+2lAK+AF51xLYD95hmCccw5wBR1sZqPMLM3M0nbt2hVEGCIiklcwyX0HsMM5t9a/PR9fsv/JzBoC+L/+XNDBzrmZzrkE51xCgwYNgghDRETyKnNyd879G/jezJr6m7oCXwLvAsP8bcOARUFFKCIipXZakMffDswxsxrAP4Hh+H5hvGlmNwLfAX8M8hoiIlJKQSV351wGkFDArq7BnFdERIKjJ1RFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg4JO7mZW3cy+MLPF/u0mZrbWzL41s3n+VZpERKQClced+1hgS67tJ4CnnXN/AP4D3FgO1xARkVIIKrmbWSOgF/AX/7YBXYD5/i6vAH2DuYaIiJResHfuzwD3AMf92/WAbOfcMf/2DuCCgg40s1FmlmZmabt27QoyDBERya3Myd3MegM/O+fSy3K8c26mcy7BOZfQoEGDsoYhIiIFOC2IY9sDV5tZTyAcOAt4FqhjZqf5794bAT8EH6aIFGXGzUtDHYKnjH6xS6hDCFqZ79ydc/c55xo55yKBa4ClzrkhwDJgoL/bMGBR0FGKiEipnIx57vcCd5rZt/jG4F8+CdcQEZEiBDMsE+CcWw4s97/+J9CmPM4rIiJloydURUQ8SMldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSDlNxFRDxIyV1ExIPKpfxAVRF5aG6oQ/CUzFAHIOJhunMXEfEgJXcREQ9SchcR8SAldxERDwpmDdXGZrbMzL40s81mNtbffraZfWhm3/i/1i2/cEVEpCSCuXM/BvzZOdcCSAJGm1kLYDzwsXPuYuBj/7aIiFSgYNZQ3emc+9z/ei+wBbgA6AO84u/2CtA32CBFRKR0ymXM3cwigZbAWuBc59xO/65/A+eWxzVERKTkgk7uZlYLeBsY55z7Nfc+55wDXCHHjTKzNDNL27VrV7BhiIhILkE9oWpmYfgS+xzn3Dv+5p/MrKFzbqeZNQR+LuhY59xMYCZAQkJCgb8ARKRkuiwfHeoQPGZLqAMIWjCzZQx4GdjinHsq1653gWH+18OARWUPT0REyiKYO/f2wPXARjPL8LfdD0wB3jSzG4HvgD8GF6KIiJRWmZO7c24VYIXs7lrW84qISPD0hKqIiAcpuYuIeJCSu4iIBym5i4h4kJK7iIgHKbmLiHiQkruIiAcpuYuIeJCSu4iIBym5i4h4kJK7iIgHKbmLiHiQkruIiAcpuYuIeJCSu4iIBym5i4h40ElL7mbWw8y2mtm3Zjb+ZF1HRETyOynJ3cyqAzOAK4EWwLVm1uJkXEtERPI7WXfubYBvnXP/dM4dAd4A+pyka4mISB7BLJBdlAuA73Nt7wDa5u5gZqOAUf7NfWa29STFUhXVB34JdRDFsSdCHYGEQKX42cQKWx76lPP7wnacrOReLOfcTGBmqK7vZWaW5pxLCHUcInnpZ7PinKxhmR+Axrm2G/nbRESkApys5P4ZcLGZNTGzGsA1wLsn6VoiIpLHSRmWcc4dM7PbgA+A6sAs59zmk3EtKZCGu+RUpZ/NCmLOuVDHICIi5UxPqIqIeJCSu4iIBym5e4CZnV5A29mhiEVETg1K7t7wjpmF5WyYWUPgwxDGIxJgZq0LaOsdiliqEiV3b1gIvGlm1c0sEt8spftCGpHIf71kZtE5G2Z2LfBQCOOpEjRbxiPMbDTQA4gE/uScWx3aiER8zOxCYD5wHdARuAHo7ZzbE9LAPE7JvRIzsztzb+L7n2YD8AWAc+6pUMQlkpeZXYLvL8ztQD/n3MEQh+R5IastI+XizDzb7xTSLlLhzGwjkPvu8Wx8DzWuNTOcc7Ghiaxq0J27x5hZNaCWc+7XUMciVZuZFVqxEMA5911FxVIV6QNVDzCzuWZ2lpmdAWwCvjSzu0Mdl1Rtzrnv/An8NODf/tdN8K3toPH2k0zJ3Rta+O/U+wJL8P0PdH1oQxIJeBv4zcz+gK+2TGNgbmhD8j4ld28I889z7wu865w7yoljnSKhdNw5dwzoD0x3zt0NNAxxTJ6n5O4N/wdkAmcAK/xjnRpzl1PFUf/c9huAxf62sCL6SznQB6oeZWan+e+WRELKzFoANwOfOudeN7MmwB+dc1po8SRScq/EzGyoc+6veea7B2ieu0jVpXnuldsZ/q+a1y6nHDN70zn3xwLmuwNonvtJpjt3jzKzM5xz+0Mdh1RdZtbQObezsPnumud+cukD1UrOzC4wswT/WrWY2Tlm9jjwTYhDkyrOObfT//W73P+A74EOoY3O+5TcKzEzGwdkANOBNWY2EtgC1ATylVkVqUj+B+vuM7PnzKyb+dwO/BP4Y6jj8zoNy1RiZvYl0ME5t9vMfgd8DbR3zqWHODQRzGwR8B/gU6ArcA6+AndjnXMZoYytKlByr8TM7HPnXKtc2+udc3GhjEkkh5ltdM7F+F9XB3YCv3POHQptZFWDZstUbo3MbFqu7Ya5t51zY0IQk0iOozkvnHO/mdkOJfaKozv3SszMhhW13zn3SkXFIpKXmf0G5MzYMnyfBR3wv3bOubNCFVtVoOQuIuJBmi0jIuJBSu4iIh6k5C4i4kFK7h5gZo3MbIGZ7TKzn83sbTNrFOq4RCR0lNy9IRV4F98CCOcD/7+/TUSqKM2W8QAzy3DOxRfXJiJVh+7cvSHLzIaaWXX/v6FAVqiDEpHQ0Z27B/hLqk4HLsVXN3s1MMY5tz2kgYlIyCi5i4h4kGrLVGJmNqGI3c4590iFBSMipxTduVdiZvbnAprPAG4E6jnnalVwSCJyilBy9wgzOxMYiy+xvwn8f865n0MblYiEioZlKjkzOxu4ExgCvAK0cs79J7RRiUioKblXYmb2JNAfmAnEOOf2hTgkETlFaFimEjOz48Bh4Bi+KZCBXahetkiVpuQuIuJBekJVRMSDlNxFRDxIyV1ExIOU3MXTzGyf/+v5Zja/mL7jzCyilOfvZGaL/a9TzOy5UhybaWb1S9G/VOeXqk3JXSodM6te2mOccz865wYW020cUKrkLnKqUnKXU4qZRZrZV2Y2x8y2mNl8M4vw3+U+YWafA4PM7CIz+5uZpZvZSjNr5j++iZl9amYbzezRPOfd5H9d3cymmtkmM9tgZreb2Rh8C50sM7Nl/n7d/Of63MzeMrNa/vYe/hg/x/ecQXHv6QUzSzOzzWb2cJ7d9/hjXWdmf/D3b+BfTesz/7/25fCtlSpGyV1ORU2B551zzYFfgVv97VnOuVbOuTfwPbh1u3OuNXAX8Ly/z7PAC865GGBnIecfBUQC8c65WGCOc24a8CPQ2TnX2T9c8iBwuXOuFZAG3Glm4cBLwFVAa+C8EryfB5xzCUAscJmZxebat8cf63PAM7new9POuURgAPCXElxD5AR6QlVORd875z7xv/4rMMb/eh6A/w66HfCWmeUcc7r/a3t8CRHgNeCJAs5/OfCic+4YgHNudwF9koAWwCf+a9QAPgWaAf9yzn3jj+Wv+H5ZFOWPZjYK3/9vDf3n3eDf93qur0/niq9Frvd2Vs5fDSIlpeQup6K8T9blbO/3f60GZBexjGB5PJlnwIfOuWtPaDQr1dKFZtYE318Wic65/5jZbCA8VxdXwOtqQJJz7lCec5Xm0lLFaVhGTkW/M7NL/a+vA1bl3umc+xX4l5kNAjCfOP/uT4Br/K+HFHL+D4E/mdlp/uPP9rfvBc70v14DtM81Dn6GmV0CfAVEmtlF/n4nJP8CnIXvl9IeMzsXuDLP/sG5vn7qf/134PacDqX9hSICSu5yatoKjDazLUBd4IUC+gwBbjSz9cBmoI+/faz/2I3ABYWc/y/AdmCD//jr/O0zgb+Z2TLn3C4gBXjdzDbgH5Lx302PAt7zf6Cat6xyipntyPmHby3bL/D9UpiL75dPbnX95x8L3OFvGwMk+D/s/RK4uZD3IVIo1ZaRU4qZRQKLnXPRIQ5FpFLTnbuIiAfpzl1ExIN05y4i4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh70/wB6oLB4fNmUkAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "credit_history_distribution_run_uid = credit_history_distribution['id']\n",
    "\n",
    "subscription.payload_logging.data_distribution.show_chart(credit_history_distribution_run_uid);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Business Application <a name=\"application\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import BusinessMetric, AggregationMethods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In next cells `business payload fields` and `business metrics` are desribed for aplication. `business payload fields` is a list of columns of business payload loaded at the end of section \"Historical data\". `business metrics` defines metrics (BKPIs) which will be calculated in created application. The application is run in [section](#run_application) after loading historical data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "business_payload_fields = [\n",
    "                {\n",
    "                    \"name\": \"LoanDuration\",\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"Duration of the loan\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"LoanPurpose\",\n",
    "                    \"type\": \"string\",\n",
    "                    \"description\": \"Purpose of the loan\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"LoanAmount\",\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"Amount of the loan\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"InstallmentPercent\",\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"Installment percents\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"AcceptedPercent\",\n",
    "                    \"type\": \"number\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"AmountGranted\",\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"Risk percent\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"Accepted\",\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"Number of loans accepted\"\n",
    "                }\n",
    "            ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "business_metrics = [BusinessMetric(metric_name='Accepted Credits', description=\"Accepted Credits Daily\",\n",
    "                                           field_name='Accepted', aggregation=AggregationMethods.SUM,\n",
    "                                   time_unit='day', time_count=1, lower_limit=2150),\n",
    "                    BusinessMetric(metric_name='Credit Amount Granted', description=\"Credit Amount Granted Daily\",\n",
    "                                   field_name='AmountGranted', aggregation=AggregationMethods.SUM,\n",
    "                                   time_unit='day', time_count=1, lower_limit=5500000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add business application and get details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "application_details = ai_client.data_mart.applications.add(name=\"Tutorial Credit Risk Application\",\n",
    "                                                           description=\"Tutorial Business Application\",\n",
    "                                                           subscription_ids=[subscription.uid],\n",
    "                                                           business_metrics=business_metrics,\n",
    "                                                           payload_schema=business_payload_fields)\n",
    "application_id = application_details['metadata']['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10:44:57 preparing\n",
      "Final application status: active\n"
     ]
    }
   ],
   "source": [
    "while application_details['entity']['status']['state'] == 'preparing':\n",
    "    print(datetime.utcnow().strftime('%H:%M:%S'), application_details['entity']['status']['state'] )\n",
    "    time.sleep(10)\n",
    "    application_details = ai_client.data_mart.applications.get_details(application_id)\n",
    "print(\"Final application status: {}\".format(application_details['entity']['status']['state']) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomennt cell to show details:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "application_details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Historical data <a name=\"historical\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "historyDays = 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Insert historical payloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next section of the notebook downloads and writes historical data to the payload and measurement tables to simulate a production model that has been monitored and receiving regular traffic for the last seven days. This historical data can be viewed in the Watson OpenScale user interface. The code uses the Python and REST APIs to write this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_0.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_1.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_2.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_3.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_4.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_5.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 5.0M Jan 10 10:45 history_payloads_with_transaction_id_6.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_0.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_1.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_2.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_3.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_4.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_5.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_6.json\n",
    "!ls -lh history_payloads_with_transaction_id_*.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading historical scoring payload...\n",
      "Day 0 injection.\n",
      "Daily loading finished.\n",
      "Day 1 injection.\n",
      "Daily loading finished.\n",
      "Day 2 injection.\n",
      "Daily loading finished.\n",
      "Day 3 injection.\n",
      "Daily loading finished.\n",
      "Day 4 injection.\n",
      "Daily loading finished.\n",
      "Day 5 injection.\n",
      "Daily loading finished.\n",
      "Day 6 injection.\n",
      "Daily loading finished.\n"
     ]
    }
   ],
   "source": [
    "from ibm_ai_openscale.utils.inject_demo_data import DemoData\n",
    "import os\n",
    "\n",
    "historicalData = DemoData(aios_credentials=WOS_CREDENTIALS)\n",
    "historical_data_path=os.getcwd()\n",
    "\n",
    "historicalData.load_historical_scoring_payload(subscription, deployment_uid,file_path=historical_data_path, day_template=\"history_payloads_with_transaction_id_{}.json\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.aiopenscale.cloud.ibm.com/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/metrics\n"
     ]
    }
   ],
   "source": [
    "performance_metrics_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/metrics'\n",
    "print(performance_metrics_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical fairness metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 1.4M Jan 10 10:46 history_fairness.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_fairness.json -O history_fairness.json\n",
    "!ls -lh history_fairness.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "with open('history_fairness.json', 'r') as history_file:\n",
    "    payloads = json.load(history_file)\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Loading day', day + 1)\n",
    "    metrics = []\n",
    "    \n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n",
    "        \n",
    "        metric = {\n",
    "            'metric_type': 'fairness',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': model_uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': payloads[index]\n",
    "        }\n",
    "        metrics.append(metric)\n",
    "    response = requests.post(performance_metrics_url, json=metrics, headers=ai_client._get_headers())\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical debias metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 553K Jan 10 10:46 history_debias.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_debias.json -O history_debias.json\n",
    "!ls -lh history_debias.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "with open('history_debias.json', 'r') as history_file:\n",
    "    payloads = json.load(history_file)\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Loading day', day + 1)\n",
    "    debias_metrics = []\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n",
    "\n",
    "        debiasMetric = {\n",
    "            'metric_type': 'debiased_fairness',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': subscription.uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': payloads[index]\n",
    "        }\n",
    "\n",
    "        debias_metrics.append(debiasMetric)\n",
    "    response = requests.post(performance_metrics_url, json=debias_metrics, headers=ai_client._get_headers())\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical quality metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Day 1\n",
      "Day 2\n",
      "Day 3\n",
      "Day 4\n",
      "Day 5\n",
      "Day 6\n",
      "Day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]\n",
    "for day in range(historyDays):\n",
    "    quality_metrics = []\n",
    "    print('Day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        qualityMetric = {\n",
    "            'metric_type': 'quality',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': subscription.uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': {\n",
    "                'quality': measurements[day],\n",
    "                'threshold': 0.7,\n",
    "                'metrics': [\n",
    "                    {\n",
    "                        'name': 'auroc',\n",
    "                        'value': measurements[day],\n",
    "                        'threshold': 0.7\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        }\n",
    "        \n",
    "        quality_metrics.append(qualityMetric)\n",
    "    \n",
    "    response = requests.post(performance_metrics_url, json=quality_metrics, headers=ai_client._get_headers())\n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical confusion matrixes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 79K Jan 10 10:46 history_quality_metrics.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_quality_metrics.json -O history_quality_metrics.json\n",
    "!ls -lh history_quality_metrics.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.aiopenscale.cloud.ibm.com/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/measurements\n"
     ]
    }
   ],
   "source": [
    "measurements_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/measurements'\n",
    "print(measurements_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Day 1\n",
      "Day 2\n",
      "Day 3\n",
      "Day 4\n",
      "Day 5\n",
      "Day 6\n",
      "Day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "with open('history_quality_metrics.json') as json_file:\n",
    "    records = json.load(json_file)\n",
    "    \n",
    "for day in range(historyDays):\n",
    "    index = 0\n",
    "    measurments = []\n",
    "    print('Day', day + 1)\n",
    "    \n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "\n",
    "        measurement = {\n",
    "            \"monitor_definition_id\": 'quality',\n",
    "            \"binding_id\": subscription.binding_uid,\n",
    "            \"subscription_id\": subscription.uid,\n",
    "            \"asset_id\": subscription.source_uid,\n",
    "            'metrics': [records[index]['metrics']],\n",
    "            'sources': [records[index]['sources']],\n",
    "            'timestamp': score_time\n",
    "        }\n",
    "\n",
    "        measurments.append(measurement)\n",
    "        index+=1\n",
    "\n",
    "    response = requests.post(measurements_url, json=measurments, headers=ai_client._get_headers())\n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical performance metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Day 1\n",
      "Day 2\n",
      "Day 3\n",
      "Day 4\n",
      "Day 5\n",
      "Day 6\n",
      "Day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "for day in range(historyDays):\n",
    "    performance_metrics = []\n",
    "    print('Day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        score_count = random.randint(60, 600)\n",
    "        score_resp = random.uniform(60, 300)\n",
    "\n",
    "        performanceMetric = {\n",
    "            'metric_type': 'performance',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': subscription.uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': {\n",
    "                'response_time': score_resp,\n",
    "                'records': score_count\n",
    "            }\n",
    "        }\n",
    "        performance_metrics.append(performanceMetric)\n",
    "\n",
    "    response = requests.post(performance_metrics_url, json=performance_metrics, headers=ai_client._get_headers())\n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical manual labeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.aiopenscale.cloud.ibm.com/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/manual_labelings\n"
     ]
    }
   ],
   "source": [
    "manual_labeling_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/manual_labelings'\n",
    "print(manual_labeling_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 926K Jan 10 10:46 history_manual_labeling.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_manual_labeling.json -O history_manual_labeling.json\n",
    "!ls -lh history_manual_labeling.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "with open('history_manual_labeling.json', 'r') as history_file:\n",
    "    records = json.load(history_file)\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Loading day', day + 1)\n",
    "    record_json = []\n",
    "    for hour in range(24):\n",
    "        for record in records:\n",
    "            if record['fastpath_history_day'] == day and record['fastpath_history_hour'] == hour:\n",
    "                record['binding_id'] = binding_uid\n",
    "                record['subscription_id'] = subscription.uid\n",
    "                record['asset_revision'] = model_uid\n",
    "                record['deployment_id'] = deployment_uid\n",
    "                record['scoring_timestamp'] = (datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "                record_json.append(record)\n",
    "    response = requests.post(manual_labeling_url, json=record_json, headers=ai_client._get_headers())\n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical drift measurements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 832K Jan 10 10:46 history_drift_measurement_0.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 868K Jan 10 10:46 history_drift_measurement_1.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 870K Jan 10 10:46 history_drift_measurement_2.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 910K Jan 10 10:46 history_drift_measurement_3.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 841K Jan 10 10:46 history_drift_measurement_4.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 836K Jan 10 10:46 history_drift_measurement_5.json\r\n",
      "-rw-r--r-- 1 spark 4294967294 840K Jan 10 10:46 history_drift_measurement_6.json\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_0.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_1.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_2.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_3.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_4.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_5.json\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_drift_measurement_6.json\n",
    "!ls -lh history_drift_measurement_*.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.aiopenscale.cloud.ibm.com/v1/data_marts/cacaf1bc-1285-46da-ba9a-49464bc683a3/measurements\n"
     ]
    }
   ],
   "source": [
    "print(measurements_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Daily loading finished.\n",
      "Loading day 2\n",
      "Daily loading finished.\n",
      "Loading day 3\n",
      "Daily loading finished.\n",
      "Loading day 4\n",
      "Daily loading finished.\n",
      "Loading day 5\n",
      "Daily loading finished.\n",
      "Loading day 6\n",
      "Daily loading finished.\n",
      "Loading day 7\n",
      "Daily loading finished.\n"
     ]
    }
   ],
   "source": [
    "for day in range(historyDays):\n",
    "    measurements = []\n",
    "\n",
    "    with open(\"history_drift_measurement_{}.json\".format(day), 'r') as history_file:\n",
    "        drift_daily_measurements = json.load(history_file)\n",
    "    print('Loading day', day + 1)\n",
    "\n",
    "    #Historical data contains 8 records per day - each represents 3 hour drift window.\n",
    "    \n",
    "    for nb_window, records in enumerate(drift_daily_measurements):\n",
    "        for record in records:\n",
    "            window_start =  (datetime.utcnow() + timedelta(hours=(-(24 * day + (nb_window+1)*3 + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ') # first_payload_record_timestamp_in_window (oldest)\n",
    "            window_end = (datetime.utcnow() + timedelta(hours=(-(24 * day + nb_window*3 + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')# last_payload_record_timestamp_in_window (most recent)\n",
    "            #modify start and end time for each record\n",
    "            record['sources'][0]['data']['start'] = window_start\n",
    "            record['sources'][0]['data']['end'] = window_end\n",
    "            \n",
    "            measurement = {\n",
    "                \"monitor_definition_id\": \"drift\",\n",
    "                \"binding_id\": subscription.binding_uid,\n",
    "                \"subscription_id\": subscription.uid,\n",
    "                \"asset_id\": subscription.source_uid,\n",
    "                'metrics': record['metrics'],\n",
    "                'sources': record['sources'],\n",
    "                'timestamp': window_start,\n",
    "                'process': 'Drift run for subscription_{}'.format(subscription.uid)\n",
    "            }\n",
    "\n",
    "            measurements.append(measurement)\n",
    "    response = requests.post(measurements_url, json=measurements, headers=ai_client._get_headers())\n",
    "    print(\"Daily loading finished.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert Business payload "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following section triggers two minute waiting for all historical records to be loaded succesfully. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 spark 4294967294 1.6M Jan 10 10:46 history_business_payloads_week.csv\r\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_business_payloads_week.csv -O history_business_payloads_week.csv\n",
    "!ls -lh history_business_payloads_week.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "business_payload_data_set_id = application_details['entity']['business_payload_data_set_id']\n",
    "\n",
    "historicalData.load_historical_business_payload(business_payload_data_set_id, file_path=historical_data_path, file_name=\"history_business_payloads_week.csv\")\n",
    "time.sleep(120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Business Application  <a id=\"run_application\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_details = ai_client.data_mart.applications.run(application_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional data to help debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: ac325cbf-f50b-40e2-bb08-f1c8f085c1b2\n",
      "Deployment: a58d369a-f7c8-4cb4-a9f1-68a6183114c8\n",
      "Binding: 409c1c90-3666-4ef3-8186-1b376c0f2c22\n",
      "Scoring URL: https://us-south.ml.cloud.ibm.com/v3/wml_instances/409c1c90-3666-4ef3-8186-1b376c0f2c22/deployments/a58d369a-f7c8-4cb4-a9f1-68a6183114c8/online\n"
     ]
    }
   ],
   "source": [
    "print('Model:', model_uid)\n",
    "print('Deployment:', deployment_uid)\n",
    "print('Binding:', binding_uid)\n",
    "print('Scoring URL:', credit_risk_scoring_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify transactions for Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scoring_id</th>\n",
       "      <th>predictedLabel</th>\n",
       "      <th>probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-200</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.6010296029951443, 0.39897039700485576]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-199</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.5252259404132618, 0.47477405958673813]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-198</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.6058498809703368, 0.39415011902966324]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-197</td>\n",
       "      <td>Risk</td>\n",
       "      <td>[0.48709690351550117, 0.5129030964844988]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-196</td>\n",
       "      <td>Risk</td>\n",
       "      <td>[0.17206640991365046, 0.8279335900863496]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-195</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.8587600876373713, 0.14123991236262867]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-194</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.6736928636502977, 0.3263071363497023]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-193</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.592557171897104, 0.4074428281028961]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-192</td>\n",
       "      <td>Risk</td>\n",
       "      <td>[0.36256174271349495, 0.6374382572865052]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-191</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.941578376354431, 0.05842162364556904]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-190</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.9272040466599101, 0.07279595334008997]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-189</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.5030006786565658, 0.49699932134343416]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-188</td>\n",
       "      <td>Risk</td>\n",
       "      <td>[0.34934564744084157, 0.6506543525591584]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-187</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.8982745883190519, 0.10172541168094798]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-186</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.5124893436756256, 0.4875106563243744]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-185</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.6299218144216273, 0.3700781855783727]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-184</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.6506091805406601, 0.34939081945934]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-183</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.9313739658632384, 0.06862603413676159]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-182</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.9493095125702398, 0.05069048742976015]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>80fd89244879c09b0b461354674e50b3-181</td>\n",
       "      <td>No Risk</td>\n",
       "      <td>[0.845783450389388, 0.15421654961061212]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              scoring_id predictedLabel  \\\n",
       "0   80fd89244879c09b0b461354674e50b3-200        No Risk   \n",
       "1   80fd89244879c09b0b461354674e50b3-199        No Risk   \n",
       "2   80fd89244879c09b0b461354674e50b3-198        No Risk   \n",
       "3   80fd89244879c09b0b461354674e50b3-197           Risk   \n",
       "4   80fd89244879c09b0b461354674e50b3-196           Risk   \n",
       "5   80fd89244879c09b0b461354674e50b3-195        No Risk   \n",
       "6   80fd89244879c09b0b461354674e50b3-194        No Risk   \n",
       "7   80fd89244879c09b0b461354674e50b3-193        No Risk   \n",
       "8   80fd89244879c09b0b461354674e50b3-192           Risk   \n",
       "9   80fd89244879c09b0b461354674e50b3-191        No Risk   \n",
       "10  80fd89244879c09b0b461354674e50b3-190        No Risk   \n",
       "11  80fd89244879c09b0b461354674e50b3-189        No Risk   \n",
       "12  80fd89244879c09b0b461354674e50b3-188           Risk   \n",
       "13  80fd89244879c09b0b461354674e50b3-187        No Risk   \n",
       "14  80fd89244879c09b0b461354674e50b3-186        No Risk   \n",
       "15  80fd89244879c09b0b461354674e50b3-185        No Risk   \n",
       "16  80fd89244879c09b0b461354674e50b3-184        No Risk   \n",
       "17  80fd89244879c09b0b461354674e50b3-183        No Risk   \n",
       "18  80fd89244879c09b0b461354674e50b3-182        No Risk   \n",
       "19  80fd89244879c09b0b461354674e50b3-181        No Risk   \n",
       "\n",
       "                                  probability  \n",
       "0   [0.6010296029951443, 0.39897039700485576]  \n",
       "1   [0.5252259404132618, 0.47477405958673813]  \n",
       "2   [0.6058498809703368, 0.39415011902966324]  \n",
       "3   [0.48709690351550117, 0.5129030964844988]  \n",
       "4   [0.17206640991365046, 0.8279335900863496]  \n",
       "5   [0.8587600876373713, 0.14123991236262867]  \n",
       "6    [0.6736928636502977, 0.3263071363497023]  \n",
       "7     [0.592557171897104, 0.4074428281028961]  \n",
       "8   [0.36256174271349495, 0.6374382572865052]  \n",
       "9    [0.941578376354431, 0.05842162364556904]  \n",
       "10  [0.9272040466599101, 0.07279595334008997]  \n",
       "11  [0.5030006786565658, 0.49699932134343416]  \n",
       "12  [0.34934564744084157, 0.6506543525591584]  \n",
       "13  [0.8982745883190519, 0.10172541168094798]  \n",
       "14   [0.5124893436756256, 0.4875106563243744]  \n",
       "15   [0.6299218144216273, 0.3700781855783727]  \n",
       "16     [0.6506091805406601, 0.34939081945934]  \n",
       "17  [0.9313739658632384, 0.06862603413676159]  \n",
       "18  [0.9493095125702398, 0.05069048742976015]  \n",
       "19   [0.845783450389388, 0.15421654961061212]  "
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "payload_data = subscription.payload_logging.get_table_content(limit=20)\n",
    "payload_data.filter(items=['scoring_id', 'predictedLabel', 'probability'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating new item: 2020-01-10 10:48:48.243765 PASS\n",
      "Item: 2020-01-10 10:48:48.243765 PASS created!\n"
     ]
    }
   ],
   "source": [
    "# The code was removed by Watson Studio for sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "\n",
    "You have finished the hands-on lab for IBM Watson OpenScale. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n",
    "\n",
    "## Next steps\n",
    "\n",
    "OpenScale shows model performance over time. You have two options to keep data flowing to your OpenScale graphs:\n",
    "  * Download, configure and schedule the [model feed notebook](https://raw.githubusercontent.com/emartensibm/german-credit/master/german_credit_scoring_feed.ipynb). This notebook can be set up with your WML credentials, and scheduled to provide a consistent flow of scoring requests to your model, which will appear in your OpenScale monitors.\n",
    "  * Re-run this notebook. Running this notebook from the beginning will delete and re-create the model and deployment, and re-create the historical data. Please note that the payload and measurement logs for the previous deployment will continue to be stored in your datamart, and can be deleted if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n",
    "\n",
    "Eric Martens, is a technical specialist having expertise in analysis and description of business processes, and their translation into functional and non-functional IT requirements. He acts as the interpreter between the worlds of IT and business.\n",
    "\n",
    "Lukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 with Spark",
   "language": "python3",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}