notebooks/OpenScale and OpenPages model risk management on Cloud Pak for Data.ipynb

{"cells": [{"metadata": {}, "cell_type": "markdown", "source": "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"}, {"metadata": {}, "cell_type": "markdown", "source": "This notebook should be run using with Python 3.x with Spark runtime environment. If you are viewing this in Watson Studio and do not see Python 3.x with Spark in the upper right corner of your screen, please update the runtime now. It requires service credentials for the following services:\n\n<li>Watson OpenScale\n<li>Watson Machine Learning\n<li>Db2"}, {"metadata": {}, "cell_type": "markdown", "source": "## Credentials for IBM Cloud Pak for Data\nIn the following code boxes, you need to replace the sample data with your own credentials. You can acquire the information from your system administrator or through the Cloud Pak for Data dashboard."}, {"metadata": {}, "cell_type": "markdown", "source": "### Obtaining your Watson OpenScale credentials\nYou can retrieve the URL by running the following command: `oc get route -n namespace1 --no-headers | awk '{print $2}'`\nReplace the `namespace1` variable with your namespace.\n\nYou should have been assigned a username and password when you were added to the Cloud Pak for Data system. You might need to ask either your database administrator or your system administrator for some of the information."}, {"metadata": {}, "cell_type": "code", "source": "############################################################################################\n# Paste your Watson OpenScale credentials into the following section and then run this cell.\n############################################################################################\nWOS_CREDENTIALS = {\n      \"url\": \"https://CLUSTER_NAME\",\n      \"username\": \"******\",\n      \"password\": \"******\"\n  }", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "### Your Watson OpenScale GUID\nFor most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default."}, {"metadata": {}, "cell_type": "code", "source": "WOS_GUID=\"00000000-0000-0000-0000-000000000000\"", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "### Your Watson OpenScale GUID\nFor most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default."}, {"metadata": {}, "cell_type": "code", "source": "WML_CREDENTIALS = WOS_CREDENTIALS.copy()\nWML_CREDENTIALS['instance_id']='openshift'\nWML_CREDENTIALS['version']='2.5.0'", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "### Your database credentials and schema\nNormally, you must obtain the database credentials and schema name from your database administrator, however, if you deployed one of the database options from the Cloud Pak for Data UI, you should be able to retrieve credentials yourself by going to My data and right-clicking on the database tile."}, {"metadata": {}, "cell_type": "code", "source": "####################################################################################\n# Paste your database credentials into the following section and then run this cell.\n####################################################################################\nDATABASE_CREDENTIALS = {\n      \"db\": \"SAMPLE\",\n      \"db_type\": \"db2\",\n      \"hostname\": \"******\",\n      \"password\": \"******\",\n      \"port\": \"******\",\n      \"username\": \"******\"\n  }", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "SCHEMA_NAME = \"YOUR_SCHEMA_NAME_HERE\"", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Integration system credentials (IBM OpenPages MRG) \nAs part of the closed beta, you were sent an email message from IBM that provides the URL, username, and password of an IBM OpenPages system that you can use. Replace the following credentials with the ones that you received in that email message."}, {"metadata": {}, "cell_type": "code", "source": "#########################################################################################\n# Paste your IBM OpenPages credentials into the following section and then run this cell.\n#########################################################################################\nOPENPAGES_CREDENTIALS = {\n     \"url\": \"https://softlayer-dev1.op-ibm.com\",\n     \"username\": \"******\",\n     \"password\": \"******\"\n  }", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Model ID\nFor the model that you create, copy the `model_id` from the OpenPages Model URL from the browser address bar. For example, if you create a model and view it in the OpenPages UI, you might see a URL such as \"https://mrgbeta.op-ibm.com/app/jspview/react/grc/task-view/7916?\". In this example, the `model_id` is **7916**. Paste that value into the following code cell."}, {"metadata": {}, "cell_type": "code", "source": "####################################################################\n# Paste your model ID in the following field and then run this cell.\n####################################################################\nopenpages_model_id=\"paste_model_id_here\"", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Package installation\nThe following opensource packages must be installed into this notebook instance so that they are available to use during processing."}, {"metadata": {}, "cell_type": "code", "source": "!rm -rf /home/spark/shared/user-libs/python3.6*\n\n!pip install pyspark==2.3 --no-cache | tail -n 1\n!pip install numpy==1.16.4 --no-cache | tail -n 1\n!pip install scikit-learn==0.20.2 --no-cache | tail -n 1\n!pip install --upgrade SciPy --no-cache | tail -n 1\n!pip install --upgrade ibm-ai-openscale --no-cache | tail -n 1", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Load the training data from Github\nSo you don't have to manually generate training data, we've provided a sample and placed it in a publicly available Github repo."}, {"metadata": {}, "cell_type": "code", "source": "!rm german_credit_data_biased_training.csv\n!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "import json\nimport requests\nimport base64\nfrom requests.auth import HTTPBasicAuth\nimport time", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "# Deploy the models\n\nThe following cells deploy both the pre-prod and challenger models into the Watson Machine Learning instance."}, {"metadata": {}, "cell_type": "code", "source": "PRE_PROD_MODEL_NAME=\"German Credit Risk Model - PreProd\"\nPRE_PROD_DEPLOYMENT_NAME=\"German Credit Risk Model - PreProd\"\n\nPRE_PROD_CHALLENGER_MODEL_NAME=\"German Credit Risk Model - Challenger\"\nPRE_PROD_CHALLENGER_DEPLOYMENT_NAME=\"German Credit Risk Model - Challenger\"\n\nPRE_PROD_SPACE_NAME=\"pre-prod\"", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Initialize WML client and set the space_id parameter\nAnalytic spaces are used to categorize assets and can be associated with a project inside of Watson Studio."}, {"metadata": {}, "cell_type": "code", "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient\nwml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)\nprint(wml_client.service_instance.get_url())\n\nwml_client.spaces.list()\n\n# Find and set the default space\nspace_name=PRE_PROD_SPACE_NAME\nspaces = wml_client.spaces.get_details()['resources']\nspace_id = None\nfor space in spaces:\n    if space['entity']['name'] == space_name:\n        space_id = space[\"metadata\"][\"guid\"]\nif space_id is None:\n    space_id = wml_client.spaces.store(\n        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"guid\"]\nwml_client.set.default_space(space_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Deploy the Spark Credit Risk Model to WML\n\nThe following cell deploys the Spark version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. You'll notice that this version of the German Credit Risk model has an auc-roc score around 71%."}, {"metadata": {}, "cell_type": "code", "source": "import numpy \nnumpy.version.version\n\nimport pandas as pd\nimport json\n\nfrom pyspark import SparkContext, SQLContext\nfrom pyspark.ml import Pipeline\nfrom pyspark.ml.classification import RandomForestClassifier,GBTClassifier\nfrom pyspark.ml.evaluation import BinaryClassificationEvaluator\nfrom pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString\nfrom pyspark.sql.types import StructType, DoubleType, StringType, ArrayType\n\nfrom pyspark.sql import SparkSession\nfrom pyspark import SparkFiles\n\nspark = SparkSession.builder.getOrCreate()\npd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\nspark_df = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\nspark_df.head()\n\n(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)\nprint(\"Number of records for training: \" + str(train_data.count()))\nprint(\"Number of records for evaluation: \" + str(test_data.count()))\n\nsi_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')\nsi_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')\nsi_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')\nsi_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')\nsi_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')\nsi_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')\nsi_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')\nsi_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')\nsi_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')\nsi_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')\nsi_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')\nsi_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')\nsi_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')\nsi_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\nlabel_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)\n\nva_features = VectorAssembler(\ninputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\",\n           \"EmploymentDuration_IX\", \"Sex_IX\", \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\",\n           \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \"LoanDuration\", \"LoanAmount\",\n           \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\",\n           \"Dependents\"], outputCol=\"features\")\n\nclassifier=GBTClassifier(featuresCol=\"features\")\n\npipeline = Pipeline(\nstages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,\n        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\n        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n\nmodel = pipeline.fit(train_data)\npredictions = model.transform(test_data)\nevaluator = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\nauc = evaluator.evaluate(predictions)\n\nprint(\"Accuracy = %g\" % auc)\n\n# Remove existing model and deployment\nMODEL_NAME=PRE_PROD_MODEL_NAME\nDEPLOYMENT_NAME=PRE_PROD_DEPLOYMENT_NAME\n\ndeployment_details = wml_client.deployments.get_details()\nfor deployment in deployment_details['resources']:\n    deployment_id = deployment['metadata']['guid']\n    model_id = deployment['entity']['asset']['href'].split('/')[3].split('?')[0]\n    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n        print('Deleting deployment id', deployment_id)\n        wml_client.deployments.delete(deployment_id)\n        print('Deleting model id', model_id)\n        wml_client.repository.delete(model_id)\nwml_client.repository.list_models()\n\n# Save Model\nmodel_props_rf = {\n    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.RUNTIME_UID: \"spark-mllib_2.3\",\n    wml_client.repository.ModelMetaNames.TYPE: \"mllib_2.3\"\n}\n\npublished_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)\nprint(published_model_details)\n\n# List models in the repository\nwml_client.repository.list_models()\n\n# Get the model UID\npre_prod_model_uid = wml_client.repository.get_model_uid(published_model_details)\npre_prod_model_uid\n\n\n# Deploy model\nwml_deployments = wml_client.deployments.get_details()\npre_prod_deployment_uid = None\nfor deployment in wml_deployments['resources']:\n    if DEPLOYMENT_NAME == deployment['entity']['name']:\n        pre_prod_deployment_uid = deployment['metadata']['guid']\n        break\n\nif pre_prod_deployment_uid is None:\n    print(\"Deploying model...\")\n    meta_props = {\n        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n    }\n    deployment = wml_client.deployments.create(artifact_uid=pre_prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)\n    pre_prod_deployment_uid = wml_client.deployments.get_uid(deployment)\n\nprint(\"Model id: {}\".format(pre_prod_model_uid))\nprint(\"Deployment id: {}\".format(pre_prod_deployment_uid))\n\npre_prod_deployment_uid=wml_client.deployments.get_uid(deployment)\npre_prod_deployment_uid\n\nfields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\nvalues = [\n  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n]\n\npayload_scoring = {\"fields\": fields,\"values\": values}\npayload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)\n\nscoring_response", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Deploy the Scikit-Learn Credit Risk Model to Watson Machine Learning\n\nThe following cell deploys the Scikit-learn version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. This version of the German Credit Risk model has an auc-roc score around 85% and will be called the \"Challenger.\""}, {"metadata": {}, "cell_type": "code", "source": "import pandas as pd\nimport json\nimport sys\nimport numpy\nimport sklearn\nimport sklearn.ensemble\nnumpy.set_printoptions(threshold=sys.maxsize)\nfrom sklearn.utils.multiclass import type_of_target\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import StandardScaler, OrdinalEncoder\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.metrics import get_scorer\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.metrics import classification_report\n\ndata_df=pd.read_csv (\"german_credit_data_biased_training.csv\")\n\ndata_df.head()\n\ntarget_label_name = \"Risk\"\nfeature_cols= data_df.drop(columns=[target_label_name])\nlabel= data_df[target_label_name]\n\n# Set model evaluation properties\noptimization_metric = 'roc_auc'\nrandom_state = 33\ncv_num_folds = 3\nholdout_fraction = 0.1\n\nif type_of_target(label.values) in ['multiclass', 'binary']:\n    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state, stratify=label.values)\nelse:\n    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state)\n\n# Data preprocessing transformer generation\n\nnumeric_transformer = Pipeline(steps=[\n    ('imputer', SimpleImputer(strategy='median')),\n    ('scaler', StandardScaler())])\ncategorical_transformer = Pipeline(steps=[\n    ('imputer', SimpleImputer(strategy='most_frequent')),\n    ('OrdinalEncoder', OrdinalEncoder(categories='auto',dtype=numpy.float64 ))])\n\nnumeric_features = feature_cols.select_dtypes(include=['int64', 'float64']).columns\ncategorical_features = feature_cols.select_dtypes(include=['object']).columns\n\npreprocessor = ColumnTransformer(\n    transformers=[\n        ('num', numeric_transformer, numeric_features),\n        ('cat', categorical_transformer, categorical_features)])\n\n# Initiate model and create pipeline\nmodel=sklearn.ensemble.gradient_boosting.GradientBoostingClassifier()\ngbt_pipeline = Pipeline(steps=[('preprocessor', preprocessor), ('classifier', model)])\nmodel_gbt=gbt_pipeline.fit(X_train, y_train)\n\ny_pred = model_gbt.predict(X_holdout)\n\n\n# Evaluate model performance on test data and Cross validation\nscorer = get_scorer(optimization_metric)\nscorer(model_gbt,X_holdout, y_holdout)\n\n# Cross validation -3 folds\ncv_results = cross_validate(model_gbt,X_train,y_train, scoring={optimization_metric:scorer})\nnumpy.mean(cv_results['test_' + optimization_metric])\n\nprint(classification_report(y_pred, y_holdout))\n\n\n# Remove existing model and deployment\nMODEL_NAME=PRE_PROD_CHALLENGER_MODEL_NAME\nDEPLOYMENT_NAME=PRE_PROD_CHALLENGER_DEPLOYMENT_NAME\n\ndeployment_details = wml_client.deployments.get_details()\nfor deployment in deployment_details['resources']:\n    deployment_id = deployment['metadata']['guid']\n    model_id = deployment['entity']['asset']['href'].split('/')[3].split('?')[0]\n    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n        print('Deleting deployment id', deployment_id)\n        wml_client.deployments.delete(deployment_id)\n        print('Deleting model id', model_id)\n        wml_client.repository.delete(model_id)\nwml_client.repository.list_models()\n\n# Store Model\nmodel_props_gbt = {\n    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.RUNTIME_UID: \"scikit-learn_0.20-py3.6\",\n    wml_client.repository.ModelMetaNames.TYPE: \"scikit-learn_0.20\"\n}\n\npublished_model_details = wml_client.repository.store_model(model=model_gbt, meta_props=model_props_gbt, training_data=feature_cols,training_target=label)\nprint(published_model_details)\n\n# List models in the repository\nwml_client.repository.list_models()\n\n# Get the model UID\nchallenger_model_uid = wml_client.repository.get_model_uid(published_model_details)\nchallenger_model_uid\n\n\n# Deploy model\nwml_deployments = wml_client.deployments.get_details()\nchallenger_deployment_uid = None\nfor deployment in wml_deployments['resources']:\n    if DEPLOYMENT_NAME == deployment['entity']['name']:\n        challenger_deployment_uid = deployment['metadata']['guid']\n        break\n\nif challenger_deployment_uid is None:\n    print(\"Deploying model...\")\n    meta_props = {\n        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n    }\n    deployment = wml_client.deployments.create(artifact_uid=challenger_model_uid, meta_props=meta_props)\n    challenger_deployment_uid = wml_client.deployments.get_uid(deployment)\n\nprint(\"Model id: {}\".format(challenger_model_uid))\nprint(\"Deployment id: {}\".format(challenger_deployment_uid))\n\nchallenger_deployment_uid=wml_client.deployments.get_uid(deployment)\nchallenger_deployment_uid\n\n\n# Sample scoring\nfields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\nvalues = [\n  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n]\n\npayload_scoring = {\"fields\": fields,\"values\": values}\npayload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)\n\nscoring_response\n", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "# Configure OpenScale \nThe notebook will now import the necessary libraries and set up a Python OpenScale client."}, {"metadata": {}, "cell_type": "code", "source": "from ibm_ai_openscale import APIClient4ICP\nfrom ibm_ai_openscale.engines import *\nfrom ibm_ai_openscale.utils import *\nfrom ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\nfrom ibm_ai_openscale.supporting_classes.enums import *", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "ai_client = APIClient4ICP(aios_credentials=WOS_CREDENTIALS)\nai_client.version", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Create schema and datamart\n\n### Set up datamart\nWatson OpenScale uses a database to store payload logs and calculated metrics. If an OpenScale datamart exists in Db2, the existing datamart will be used and no data will be overwritten."}, {"metadata": {}, "cell_type": "code", "source": "try:\n    data_mart_details = ai_client.data_mart.get_details()\n    print('Using existing external datamart')\nexcept:\n    print('Setting up external datamart')\n    ai_client.data_mart.setup(db_credentials=DATABASE_CREDENTIALS, schema=SCHEMA_NAME)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "data_mart_details = ai_client.data_mart.get_details()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "data_mart_details", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Generate an ICP token\n\nThe following is a function that will generate an ICP access token used to interact with the Watson OpenScale APIs"}, {"metadata": {}, "cell_type": "code", "source": "def generate_access_token():\n    headers={}\n    headers[\"Accept\"] = \"application/json\"\n    auth = HTTPBasicAuth(WOS_CREDENTIALS[\"username\"], WOS_CREDENTIALS[\"password\"])\n    \n    ICP_TOKEN_URL= WOS_CREDENTIALS[\"url\"] + \"/v1/preauth/validateAuth\"\n    \n    response = requests.get(ICP_TOKEN_URL, headers=headers, auth=auth, verify=False)\n    json_data = response.json()\n    icp_access_token = json_data['accessToken']\n    return icp_access_token", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Bind WML machine learning instance as Pre-Prod\n\nWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Pre-Prod\" already exists, this code will delete that binding a create a new one.\n\n**Note**: Binding with name `WML Pre-Prod` is assumed to be only created by this notebook."}, {"metadata": {}, "cell_type": "code", "source": "all_bindings = ai_client.data_mart.bindings.get_details()['service_bindings']\nfor binding in all_bindings:\n    if binding['entity']['name'] == \"WML Pre-Prod\":\n        binding_uid = binding['metadata']['guid']\n        ai_client.data_mart.bindings.delete(binding_uid)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "import uuid\n\nheaders = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\npayload = {\n    \"name\": \"WML Pre-Prod\",\n    \"description\": \"WML Instance designated as Pre-Production\",\n    \"service_type\": \"watson_machine_learning\",\n    \"instance_id\": str(uuid.uuid4()),\n    \"credentials\": {\n    },\n    \"operational_space_id\": \"pre_production\",\n    \"deployment_space_id\": space_id\n}\n\nbinding_uid = str(uuid.uuid4())\n\nSERVICE_BINDINGS_URL = WOS_CREDENTIALS[\"url\"] + \"/v1/data_marts/{0}/service_bindings/{1}\".format(WOS_GUID, binding_uid)\n\nresponse = requests.put(SERVICE_BINDINGS_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "print(binding_uid)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "ai_client.data_mart.bindings.get_details(binding_uid)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Create an integration to IBM OpenPages\n\nTo push metrics from Watson OpenScale to IBM OpenPages, a connection must be established between the two services."}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nINTEGRATED_SYSTEMS_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/integrated_systems\".format(WOS_GUID)\n\npayload = {\n    \"name\": \"OpenPages Connection\",\n    \"type\": \"open_pages\",\n    \"description\": \"Integration with OpenPages\",\n    \"credentials\": OPENPAGES_CREDENTIALS\n}\n\nresponse = requests.post(INTEGRATED_SYSTEMS_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint(json_data)\n\nif \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n    integrated_system_id = json_data[\"metadata\"][\"id\"]\n    print(integrated_system_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Subscriptions\n### Remove existing PreProd and Challenger credit risk subscriptions\nThis code removes previous subscriptions with name `German Credit Risk Model - PreProd` and `German Credit Risk Model - Challenger` to refresh the monitors with the new model and new data."}, {"metadata": {}, "cell_type": "code", "source": "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\nfor subscription in subscriptions_uids:\n    sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n    if sub_name == PRE_PROD_MODEL_NAME or sub_name == PRE_PROD_CHALLENGER_MODEL_NAME:\n        ai_client.data_mart.subscriptions.delete(subscription)\n        print('Deleted existing subscription for', sub_name)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "pre_prod_subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset(\n    pre_prod_model_uid,\n    binding_uid=binding_uid,\n    problem_type=ProblemType.BINARY_CLASSIFICATION,\n    input_data_type=InputDataType.STRUCTURED,\n    label_column='Risk',\n    prediction_column='predictedLabel',\n    probability_column='probability',\n    feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n    categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n))\n\nif pre_prod_subscription is None:\n    print('Subscription already exists; get the existing one')\n    subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n    for sub in subscriptions_uids:\n        if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == PRE_PROD_MODEL_NAME:\n            pre_prod_subscription = ai_client.data_mart.subscriptions.get(sub)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "challenger_subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset(\n    challenger_model_uid,\n    binding_uid=binding_uid,\n    problem_type=ProblemType.BINARY_CLASSIFICATION,\n    input_data_type=InputDataType.STRUCTURED,\n    label_column='Risk',\n    prediction_column='prediction',\n    probability_column='probability',\n    feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n    categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n))\n\nif challenger_subscription is None:\n    print('Subscription already exists; get the existing one')\n    subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n    for sub in subscriptions_uids:\n        if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == PRE_PROD_CHALLENGER_MODEL_NAME:\n            challenger_subscription = ai_client.data_mart.subscriptions.get(sub)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "ai_client.data_mart.subscriptions.list()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "pre_prod_subscription.uid", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "challenger_subscription.uid", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Patch the training data reference in both the preprod & challenger subscription"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\ntraining_data_reference = {\n  \"connection\": {\n    \"connection_string\": \"jdbc:db2://dashdb-txn-sbox-yp-dal09-03.services.dal.bluemix.net:50000/BLUDB:retrieveMessagesFromServerOnGetMessage=true;\",\n    \"database_name\": \"BLUDB\",\n    \"hostname\": \"dashdb-txn-sbox-yp-dal09-03.services.dal.bluemix.net\",\n    \"password\": \"khhz72v+6mcwwkfv\",\n    \"username\": \"cmb91569\"\n  },\n  \"location\": {\n    \"schema_name\": \"CMB91569\",\n    \"table_name\": \"CREDIT_RISK_TRAIN_DATA\"\n  },\n  \"name\": \"German credit risk training data\",\n  \"type\": \"db2\"\n}\n\npayload = [\n {\n   \"op\": \"replace\",\n   \"path\": \"/asset_properties/training_data_reference\",\n   \"value\": training_data_reference\n }\n]", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "SUBSCRIPTION_URL = WOS_CREDENTIALS[\"url\"] + \"/v1/data_marts/{0}/service_bindings/{1}/subscriptions/{2}\".format(WOS_GUID, binding_uid, pre_prod_subscription.uid)\n\nresponse = requests.patch(SUBSCRIPTION_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint(json_data)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "SUBSCRIPTION_URL = WOS_CREDENTIALS[\"url\"] + \"/v1/data_marts/{0}/service_bindings/{1}/subscriptions/{2}\".format(WOS_GUID, binding_uid, challenger_subscription.uid)\n\nresponse = requests.patch(SUBSCRIPTION_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint(json_data)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "### Score the model so we can configure monitors\nNow that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions."}, {"metadata": {}, "cell_type": "code", "source": "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\nvalues = [\n  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n]\n\npayload_scoring = {\"fields\": fields,\"values\": values}", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "payload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)\n\nprint('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "time.sleep(10)\npre_prod_subscription.payload_logging.get_records_count()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "payload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)\n\nprint('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "time.sleep(10)\nchallenger_subscription.payload_logging.get_records_count()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "# Quality monitoring\n\n## Enable quality monitoring\nThe code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 80%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n\nThe second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint."}, {"metadata": {}, "cell_type": "code", "source": "time.sleep(10)\npre_prod_subscription.quality_monitoring.enable(threshold=0.8, min_records=100)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "time.sleep(10)\nchallenger_subscription.quality_monitoring.enable(threshold=0.8, min_records=100)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "# Fairness, drift monitoring and explanations \n\n## Fairness configuration\nThe code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n\nWhich model feature to monitor\nOne or more majority groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\nOne or more minority groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\nThe threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 80%)\nAdditionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 100 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data."}, {"metadata": {}, "cell_type": "code", "source": "import pandas as pd\npd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n\npre_prod_subscription.fairness_monitoring.enable(\n            features=[\n                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.80),\n                Feature(\"Age\", majority=[[26,74]], minority=[[19,25]], threshold=0.80)\n            ],\n            favourable_classes=['No Risk'],\n            unfavourable_classes=['Risk'],\n            min_records=100,\n            training_data=pd_data\n        )", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "challenger_subscription.fairness_monitoring.enable(\n            features=[\n                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.80),\n                Feature(\"Age\", majority=[[26,74]], minority=[[19,25]], threshold=0.80)\n            ],\n            favourable_classes=['No Risk'],\n            unfavourable_classes=['Risk'],\n            min_records=100,\n            training_data=pd_data\n        )", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Drift configuration\n\nEnable the drift configuration for both the subscription created with a threshold of 10% and minimal sample as 100 records."}, {"metadata": {}, "cell_type": "code", "source": "pre_prod_subscription.drift_monitoring.enable(threshold=0.10, min_records=100)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "drift_status = None\nwhile drift_status != 'finished':\n    drift_details = pre_prod_subscription.drift_monitoring.get_details()\n    drift_status = drift_details['parameters']['config_status']['state']\n    if drift_status != 'finished':\n        print(datetime.utcnow().strftime('%H:%M:%S'), drift_status)\n        time.sleep(30)\nprint(drift_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "challenger_subscription.drift_monitoring.enable(threshold=0.10, min_records=100)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "drift_status = None\nwhile drift_status != 'finished':\n    drift_details = challenger_subscription.drift_monitoring.get_details()\n    drift_status = drift_details['parameters']['config_status']['state']\n    if drift_status != 'finished':\n        print(datetime.utcnow().strftime('%H:%M:%S'), drift_status)\n        time.sleep(30)\nprint(drift_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Configure Explainability\nFinally, we provide OpenScale with the training data to enable and configure the explainability features."}, {"metadata": {}, "cell_type": "code", "source": "from ibm_ai_openscale.supporting_classes import *\n\npre_prod_subscription.explainability.enable(training_data=pd_data)\nchallenger_subscription.explainability.enable(training_data=pd_data)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Enable model risk management (MRM) \n\nWe enable the MRM configuration for both the subscriptions"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\npayload = {\n  \"data_mart_id\": WOS_GUID,\n  \"monitor_definition_id\": \"mrm\",\n  \"target\": {\n    \"target_id\": pre_prod_subscription.uid,\n    \"target_type\": \"subscription\"\n  },\n  \"parameters\": {\n  },\n  \"managed_by\": \"user\"\n}\n\nMONITOR_INSTANCES_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances\".format(WOS_GUID)\n\nresponse = requests.post(MONITOR_INSTANCES_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint(json_data)\nif \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n    pre_prod_mrm_instance_id = json_data[\"metadata\"][\"id\"]", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\npayload = {\n  \"data_mart_id\": WOS_GUID,\n  \"monitor_definition_id\": \"mrm\",\n  \"target\": {\n    \"target_id\": challenger_subscription.uid,\n    \"target_type\": \"subscription\"\n  },\n  \"parameters\": {\n  },\n  \"managed_by\": \"user\"\n}\n\nMONITOR_INSTANCES_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances\".format(WOS_GUID)\n\nresponse = requests.post(MONITOR_INSTANCES_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint(json_data)\nif \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n    challenger_mrm_instance_id = json_data[\"metadata\"][\"id\"]", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Patch the integration reference in the pre-prod subscription"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\npayload = [\n  {\n    \"op\": \"add\",\n    \"path\": \"/integration_reference\",\n    \"value\": {\n        \"integrated_system_id\": integrated_system_id,\n        \"external_id\": openpages_model_id\n    }\n  }\n]\n\nSUBSCRIPTIONS_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/subscriptions/{1}\".format(WOS_GUID, pre_prod_subscription.uid)\n\nresponse = requests.patch(SUBSCRIPTIONS_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Create test data sets from the training data "}, {"metadata": {}, "cell_type": "code", "source": "test_data_1 = pd_data[1:201]\ntest_data_1.to_csv(\"german_credit_risk_test_data_1.csv\", encoding=\"utf-8\", index=False)\ntest_data_2 = pd_data[201:401]\ntest_data_2.to_csv(\"german_credit_risk_test_data_2.csv\", encoding=\"utf-8\", index=False)\ntest_data_3 = pd_data[401:601]\ntest_data_3.to_csv(\"german_credit_risk_test_data_3.csv\", encoding=\"utf-8\", index=False)\ntest_data_4 = pd_data[601:801]\ntest_data_4.to_csv(\"german_credit_risk_test_data_4.csv\", encoding=\"utf-8\", index=False)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Function to upload, evaluate and check the status of the evaluation\n\nThis function will upload the test data CSV and trigger the risk evaluation. It will iterate and check the status of the evaluation until its finished with a finite wait duration"}, {"metadata": {}, "cell_type": "code", "source": "def upload_and_evaluate(file_name, mrm_instance_id):\n    \n    print(\"Running upload and evaluate for {}\".format(file_name))\n    import json\n    import time\n    from datetime import datetime\n\n    status = None\n    monitoring_run_id = None\n    GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES = 32\n    GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL = 10\n    \n    if file_name is not None:\n        \n        headers = {}\n        headers[\"Content-Type\"] = \"text/csv\"\n        headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n        \n        POST_EVALUATIONS_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitoring_services/mrm/monitor_instances/{1}/risk_evaluations?test_data_set_name={2}\".format(WOS_GUID, mrm_instance_id, file_name)\n\n        with open(file_name) as file:\n            f = file.read()\n            b = bytearray(f, 'utf-8')\n\n        response = requests.post(POST_EVALUATIONS_URL, data=bytes(b), headers=headers, verify=False)\n        if response.ok is False:\n            print(\"Upload and evalaute for {0} failed with {1}: {2}\".format(file_name, response.status_code, response.reason))\n            return\n        \n        headers = {}\n        headers[\"Content-Type\"] = \"application/json\"\n        headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\n        GET_EVALUATIONS_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitoring_services/mrm/monitor_instances/{1}/risk_evaluations\".format(WOS_GUID, mrm_instance_id)\n        \n        for i in range(GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES):\n        \n            response = requests.get(GET_EVALUATIONS_URL, headers=headers, verify=False)\n            if response.ok is False:\n                print(\"Getting status of upload and evalaute for {0} failed with {1}: {2}\".format(file_name, response.status_code, response.reason))\n                return\n\n            response = json.loads(response.text)\n            if \"metadata\" in response and \"id\" in response[\"metadata\"]:\n                monitoring_run_id = response[\"metadata\"][\"id\"]\n            if \"entity\" in response and \"status\" in response[\"entity\"]:\n                status = response[\"entity\"][\"status\"][\"state\"]\n            \n            if status is not None:\n                print(datetime.utcnow().strftime('%H:%M:%S'), status.lower())\n                if status.lower() in [\"finished\", \"completed\"]:\n                    break\n                elif \"error\" in status.lower():\n                    print(response)\n                    break\n\n            time.sleep(GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL)\n\n    return status, monitoring_run_id", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Perform Risk Evaluations\n\nWe now start performing evaluations of smaller data sets against both the PreProd and Challenger subscriptions"}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", pre_prod_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", pre_prod_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", pre_prod_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", pre_prod_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", challenger_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", challenger_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", challenger_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", challenger_mrm_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Explore the Model Risk Management UI\n\nHere is a quick recap of what we have done so far.\n\n1. We've deployed two Credit Risk Model to a WML instance that is designated as Pre-Production\n2. We've created subscriptions of these two model deployments in OpenScale\n3. Configured all monitors supported by OpenScale for these subscriptions\n4. We've performed a few risk evaluations against both these susbscription with the same set of test data\n\nNow, please explore the Model Risk Management UI to visualize the results, compare the performance of models, download the evaluation report as PDF. For more information, refer to the Beta Guide section \"Work in Watson OpenScale.\"\n\nLink to OpenScale : https://CLUSTER_URL/aiopenscale/insights"}, {"metadata": {}, "cell_type": "markdown", "source": "# Promote pre-production model to production \n\nAfter you have reviewed the evaluation results of the PreProd Vs Challenger and if you make the decision to promote the Challenger model to Production, the first thing you need to do is to deploy the model into a WML instance that is designated as Production instance"}, {"metadata": {}, "cell_type": "markdown", "source": "## Deploy model to production WML instance "}, {"metadata": {}, "cell_type": "code", "source": "PROD_MODEL_NAME=\"German Credit Risk Model - Prod\"\nPROD_DEPLOYMENT_NAME=\"German Credit Risk Model - Prod\"\n\nPROD_SPACE_NAME=\"prod\"", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "wml_client.spaces.list()\n\n# Find and set the default space\nspace_name=PROD_SPACE_NAME\nspaces = wml_client.spaces.get_details()['resources']\nspace_id = None\nfor space in spaces:\n    if space['entity']['name'] == space_name:\n        space_id = space[\"metadata\"][\"guid\"]\nif space_id is None:\n    space_id = wml_client.spaces.store(\n        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"guid\"]\nwml_client.set.default_space(space_id)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "import numpy \nnumpy.version.version\n\nimport pandas as pd\nimport json\n\nfrom pyspark import SparkContext, SQLContext\nfrom pyspark.ml import Pipeline\nfrom pyspark.ml.classification import RandomForestClassifier,GBTClassifier\nfrom pyspark.ml.evaluation import BinaryClassificationEvaluator\nfrom pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString\nfrom pyspark.sql.types import StructType, DoubleType, StringType, ArrayType\n\nfrom pyspark.sql import SparkSession\nfrom pyspark import SparkFiles\n\nspark = SparkSession.builder.getOrCreate()\npd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\nspark_df = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\nspark_df.head()\n\n(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)\nprint(\"Number of records for training: \" + str(train_data.count()))\nprint(\"Number of records for evaluation: \" + str(test_data.count()))\n\nsi_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')\nsi_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')\nsi_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')\nsi_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')\nsi_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')\nsi_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')\nsi_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')\nsi_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')\nsi_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')\nsi_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')\nsi_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')\nsi_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')\nsi_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')\nsi_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\nlabel_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)\n\nva_features = VectorAssembler(\ninputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\",\n           \"EmploymentDuration_IX\", \"Sex_IX\", \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\",\n           \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \"LoanDuration\", \"LoanAmount\",\n           \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\",\n           \"Dependents\"], outputCol=\"features\")\n\nclassifier=GBTClassifier(featuresCol=\"features\")\n\npipeline = Pipeline(\nstages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,\n        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\n        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n\nmodel = pipeline.fit(train_data)\npredictions = model.transform(test_data)\nevaluator = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\nauc = evaluator.evaluate(predictions)\n\nprint(\"Accuracy = %g\" % auc)\n\n# Remove existing model and deployment\nMODEL_NAME=PROD_MODEL_NAME\nDEPLOYMENT_NAME=PROD_DEPLOYMENT_NAME\n\ndeployment_details = wml_client.deployments.get_details()\nfor deployment in deployment_details['resources']:\n    deployment_id = deployment['metadata']['guid']\n    model_id = deployment['entity']['asset']['href'].split('/')[3].split('?')[0]\n    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n        print('Deleting deployment id', deployment_id)\n        wml_client.deployments.delete(deployment_id)\n        print('Deleting model id', model_id)\n        wml_client.repository.delete(model_id)\nwml_client.repository.list_models()\n\n# Save Model\nmodel_props_rf = {\n    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n    wml_client.repository.ModelMetaNames.RUNTIME_UID: \"spark-mllib_2.3\",\n    wml_client.repository.ModelMetaNames.TYPE: \"mllib_2.3\"\n}\n\npublished_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)\nprint(published_model_details)\n\n# List models in the repository\nwml_client.repository.list_models()\n\n# Get the model UID\nprod_model_uid = wml_client.repository.get_model_uid(published_model_details)\nprod_model_uid\n\n\n# Deploy model\nwml_deployments = wml_client.deployments.get_details()\nprod_deployment_uid = None\nfor deployment in wml_deployments['resources']:\n    if DEPLOYMENT_NAME == deployment['entity']['name']:\n        prod_deployment_uid = deployment['metadata']['guid']\n        break\n\nif prod_deployment_uid is None:\n    print(\"Deploying model...\")\n    meta_props = {\n        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n    }\n    deployment = wml_client.deployments.create(artifact_uid=prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)\n    prod_deployment_uid = wml_client.deployments.get_uid(deployment)\n\nprint(\"Model id: {}\".format(prod_model_uid))\nprint(\"Deployment id: {}\".format(prod_deployment_uid))\n\nprod_deployment_uid=wml_client.deployments.get_uid(deployment)\nprod_deployment_uid\n\nfields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\nvalues = [\n  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n]\n\npayload_scoring = {\"fields\": fields,\"values\": values}\npayload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(prod_deployment_uid, payload)\n\nprint('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Bind WML machine learning instance as Prod\n\nWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Prod\" already exists, this code will delete that binding a create a new one.\n\n**Note**: Binding with name `WML Prod` is assumed to be only created by this notebook."}, {"metadata": {}, "cell_type": "code", "source": "all_bindings = ai_client.data_mart.bindings.get_details()['service_bindings']\nfor binding in all_bindings:\n    if binding['entity']['name'] == \"WML Prod\":\n        binding_uid = binding['metadata']['guid']\n        ai_client.data_mart.bindings.delete(binding_uid)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "import uuid\n\nheaders = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\npayload = {\n    \"name\": \"WML Prod\",\n    \"description\": \"WML Instance designated as Production\",\n    \"service_type\": \"watson_machine_learning\",\n    \"instance_id\": str(uuid.uuid4()),\n    \"credentials\": {\n    },\n    \"operational_space_id\": \"production\",\n    \"deployment_space_id\": space_id\n}\n\nbinding_uid = str(uuid.uuid4())\n\nSERVICE_BINDINGS_URL = WOS_CREDENTIALS[\"url\"] + \"/v1/data_marts/{0}/service_bindings/{1}\".format(WOS_GUID, binding_uid)\n\nresponse = requests.put(SERVICE_BINDINGS_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "print(binding_uid)\nai_client.data_mart.bindings.get_details(binding_uid)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Remove existing prod subscription\n\nThis code removes previous subscription that matches the name `German Credit Risk Model - Prod` as it is expected this subscription is created only via this notebook."}, {"metadata": {}, "cell_type": "code", "source": "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\nfor subscription in subscriptions_uids:\n    sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n    if sub_name == PROD_MODEL_NAME:\n        ai_client.data_mart.subscriptions.delete(subscription)\n        print('Deleted existing subscription for', sub_name)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "# Import configuration settings from pre-prod model\n\nWith MRM we provide a important feature that lets you copy the configuration settings of your pre-production subscription to the production subscription. To try this out\n\n1. Navigate to Model Monitors view in Insights dashboard of OpenScale\n2. Click on the Add to dashboard\n3. Select the production model deployment from WML production machine learning provider and click on Configure\n4. In Selections saved dialog, click on Configure monitors\n4. In Model Governance, click on Begin and choose the same OpenPages model that was linked to the pre-production model and click on Save\n5. You will be presented with an dialog about the OpenPages model being associated with another subscription and if you'd like to import settings.\n6. Click on Import and confirm\n\nAll the configuration settings are now copied into the production subscription\n\n\n<b>Note: The next set of cells should be executed only after finishing the import settings from the OpenScale dashboard</b>"}, {"metadata": {}, "cell_type": "markdown", "source": "## Score the production model so that we can trigger monitors\n\nNow that the production subscription is configured by copying the configuration, there would be schedules created for each of the monitors to run on a scheduled basis. \nQuality, Fairness and Mrm will run hourly. Drift will run once in three hours.\n\nFor this demo purpose, we will trigger the monitors on-demand so that we can see the model summary dashboard without having to wait the entire hour. \nTo do that lets first push some records in the Payload Logging table."}, {"metadata": {}, "cell_type": "code", "source": "df = pd_data.sample(n=400)\ndf = df.drop(['Risk'], axis=1)\nfields = df.columns.tolist()\nvalues = df.values.tolist()\n\npayload_scoring = {\"fields\": fields,\"values\": values}", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "payload = {\n    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n}\nscoring_response = wml_client.deployments.score(prod_deployment_uid, payload)\n\nprint('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "prod_subscription = ai_client.data_mart.subscriptions.get(name=PROD_DEPLOYMENT_NAME)\nprod_subscription.uid", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "time.sleep(10)\nprod_subscription.payload_logging.get_records_count()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Fetch all monitor instances"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nMONITOR_INSTANCES_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances?target.target_id={1}&target.target_type=subscription\".format(WOS_GUID, prod_subscription.uid)\nprint(MONITOR_INSTANCES_URL)\n\nresponse = requests.get(MONITOR_INSTANCES_URL, headers=headers, verify=False)\nmonitor_instances = response.json()[\"monitor_instances\"]\n\ndrift_monitor_instance_id = None\nquality_monitor_instance_id = None\nmrm_monitor_instance_id = None\n\nif monitor_instances is not None:\n    for monitor_instance in monitor_instances:\n        if \"entity\" in monitor_instance and \"monitor_definition_id\" in monitor_instance[\"entity\"]:\n            monitor_name = monitor_instance[\"entity\"][\"monitor_definition_id\"]\n            if \"metadata\" in monitor_instance and \"id\" in monitor_instance[\"metadata\"]:\n                id = monitor_instance[\"metadata\"][\"id\"]\n                if monitor_name == \"drift\":\n                    drift_monitor_instance_id = id\n                elif monitor_name == \"quality\":\n                    quality_monitor_instance_id = id\n                elif monitor_name == \"mrm\":\n                    mrm_monitor_instance_id = id\n                    \nprint(\"Quality monitor instance id - {0}\".format(quality_monitor_instance_id))\nprint(\"Drift monitor instance id - {0}\".format(drift_monitor_instance_id))\nprint(\"MRM monitor instance id - {0}\".format(mrm_monitor_instance_id))", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Function to get the monitoring run details"}, {"metadata": {}, "cell_type": "code", "source": "def get_monitoring_run_details(monitor_instance_id, monitoring_run_id):\n    \n    headers = {}\n    headers[\"Content-Type\"] = \"application/json\"\n    headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n    \n    MONITORING_RUNS_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances/{1}/runs/{2}\".format(WOS_GUID, monitor_instance_id, monitoring_run_id)\n    response = requests.get(MONITORING_RUNS_URL, headers=headers, verify=False)\n    return response.json()", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Run on-demand Quality"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nif quality_monitor_instance_id is not None:\n    MONITOR_RUN_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, quality_monitor_instance_id)\n    payload = {\n        \"triggered_by\": \"user\"\n    }\n    print(\"Triggering Quality computation with {}\".format(MONITOR_RUN_URL))\n    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n    json_data = response.json()\n    print()\n    print(json_data)\n    print()\n    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n        quality_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n    print(\"Done triggering Quality computation\")", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "from datetime import datetime\n\nquality_run_status = None\nwhile quality_run_status != 'finished':\n    monitoring_run_details = get_monitoring_run_details(quality_monitor_instance_id, quality_monitoring_run_id)\n    quality_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n    if quality_run_status == \"error\":\n        print(monitoring_run_details)\n        break\n    if quality_run_status != 'finished':\n        print(datetime.utcnow().strftime('%H:%M:%S'), quality_run_status)\n        time.sleep(10)\nprint(quality_run_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Run on-demand Drift"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nif drift_monitor_instance_id is not None:\n    MONITOR_RUN_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, drift_monitor_instance_id)\n    payload = {\n        \"triggered_by\": \"user\"\n    }\n    print(\"Triggering Drift computation with {}\".format(MONITOR_RUN_URL))\n    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n    json_data = response.json()\n    print()\n    print(json_data)\n    print()\n    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n        drift_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n    print(\"Done triggering Drift computation\")", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "from datetime import datetime\n\ndrift_run_status = None\nwhile drift_run_status != 'finished':\n    monitoring_run_details = get_monitoring_run_details(drift_monitor_instance_id, drift_monitoring_run_id)\n    drift_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n    if drift_run_status == \"error\":\n        print(monitoring_run_details)\n        break\n    if drift_run_status != 'finished':\n        print(datetime.utcnow().strftime('%H:%M:%S'), drift_run_status)\n        time.sleep(10)\nprint(drift_run_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Run on-demand Fairness"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nFAIRNESS_RUNS_URL = WOS_CREDENTIALS[\"url\"] + \"/v1/fairness_monitoring/{0}/runs\".format(prod_subscription.uid)\npayload = {\n  \"binding_id\": binding_uid,\n  \"subscription_id\": prod_subscription.uid,\n  \"deployment_id\": prod_deployment_uid,\n  \"data_mart_id\": WOS_GUID\n}\n\nprint(\"Triggering Fairness computation with {}\".format(FAIRNESS_RUNS_URL))\nresponse = requests.post(FAIRNESS_RUNS_URL, json=payload, headers=headers, verify=False)\njson_data = response.json()\nprint()\nprint(json_data)\nprint()\nprint(\"Done triggering Fairness computation\")", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "from datetime import datetime\n\nfairness_run_status = None\ntime.sleep(5)\nwhile fairness_run_status != 'FINISHED':\n    fairness_monitoring_details = prod_subscription.fairness_monitoring.get_details()\n    fairness_run_status = fairness_monitoring_details[\"parameters\"][\"run_status\"][prod_deployment_uid][\"run_status\"]\n    if fairness_run_status == \"FINISHED WITH ERRORS\":\n        print(fairness_monitoring_details)\n        break\n    if fairness_run_status != 'FINISHED':\n        print(datetime.utcnow().strftime('%H:%M:%S'), fairness_run_status)\n        time.sleep(10)\nprint(fairness_run_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Run on-demand MRM"}, {"metadata": {}, "cell_type": "code", "source": "headers = {}\nheaders[\"Content-Type\"] = \"application/json\"\nheaders[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n\nif mrm_monitor_instance_id is not None:\n    MONITOR_RUN_URL = WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, mrm_monitor_instance_id)\n    payload = {\n        \"triggered_by\": \"user\"\n    }\n    print(\"Triggering MRM computation with {}\".format(MONITOR_RUN_URL))\n    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n    json_data = response.json()\n    print()\n    print(json_data)\n    print()\n    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n        mrm_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n    print(\"Done triggering MRM computation\")", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "code", "source": "from datetime import datetime\n\nmrm_run_status = None\nwhile mrm_run_status != 'finished':\n    monitoring_run_details = get_monitoring_run_details(mrm_monitor_instance_id, mrm_monitoring_run_id)\n    mrm_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n    if mrm_run_status == \"error\":\n        print(monitoring_run_details)\n        break\n    if mrm_run_status != 'finished':\n        print(datetime.utcnow().strftime('%H:%M:%S'), mrm_run_status)\n        time.sleep(10)\nprint(mrm_run_status)", "execution_count": null, "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": "## Refresh the model summary of the production subscription in the OpenScale dashboard\n\nThis brings us to the end of this demo exercise. Thank you for trying it out."}, {"metadata": {}, "cell_type": "code", "source": "", "execution_count": null, "outputs": []}], "metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3.6", "language": "python"}, "language_info": {"name": "python", "version": "3.6.10", "mimetype": "text/x-python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py"}}, "nbformat": 4, "nbformat_minor": 1}