CARS4U-Business Area and Action-AI function.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<table style=\"border: none\" align=\"left\">\n    <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://raw.githubusercontent.com/pmservice/cars-4-you/master/static/images/logo.png\" width=\"200\" alt=\"Icon\"></th>\n       <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Area and Action - AI function</b></th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "\n<img align=left src=\"https://github.com/pmservice/cars-4-you/raw/master/static/images/ai_function.png\" alt=\"Icon\" width=\"618\"> \n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "AI function with logic to sequentially call two models: BUSINESS_AREA and ACTION.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Contents\n\n- [0. Setup](#setup)\n- [1. Itroduction](#introduction)\n- [2. Define AI function](#ai_function)\n- [3. Store the AI function in the repository](#persistence)\n- [4. Deploy AI function](#deploy)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 0. Setup\n\nIn this section please use below cell to upgrade the `watson-machine-learning-client`.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement already up-to-date: watson-machine-learning-client==1.0.260 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages\nRequirement not upgraded as not directly required: lomond in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: certifi in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: pandas in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: requests in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: tqdm in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: urllib3 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: ibm-cos-sdk in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: tabulate in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: six>=1.10.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from lomond->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: python-dateutil>=2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: pytz>=2011k in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: numpy>=1.9.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: idna<2.7,>=2.5 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: ibm-cos-sdk-core==2.*,>=2.0.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from ibm-cos-sdk->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: ibm-cos-sdk-s3transfer==2.*,>=2.0.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from ibm-cos-sdk->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: docutils>=0.10 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from ibm-cos-sdk-core==2.*,>=2.0.0->ibm-cos-sdk->watson-machine-learning-client==1.0.260)\nRequirement not upgraded as not directly required: jmespath<1.0.0,>=0.7.1 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from ibm-cos-sdk-core==2.*,>=2.0.0->ibm-cos-sdk->watson-machine-learning-client==1.0.260)\n"
                }
            ], 
            "source": "!rm -rf $PIP_BUILD/watson-machine-learning-client\n!pip install --upgrade watson-machine-learning-client==1.0.260"
        }, 
        {
            "source": "<a id=\"introduction\"></a>\n## 1. Introduction\n\nModels to predict Business Area and Action based on client feedback will be embeded into AI function. An example of payload logging using AI function will be introduced.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"ai_function\"></a>\n## 2. Define AI function", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section we define AI function and test it locally.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Template\n\nWe start with template version of AI function to present its structure in simplest case.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "    # Template version\n\n    def score_template(payload):\n        \"\"\"AI function example.\n\n        # Here we present data schema to be used by function:\n        Example:\n          {\n            \"fields\": [\"Gender\", \"Status\", \"Children\", \"Age\", \"Customer_Status\"],\n            \"values\": [\n              [\"Male\", \"M\", 2, 48, \"Inactive\"],\n              [\"Female\", \"S\", 0, 23, \"Inactive\"]\n            ]\n          }\n        \"\"\"\n        fields = payload['fields'] + ['Prediction', 'Probability']\n        values = [record + [int(record[0] == 'Male'), 0.9] for record in payload['values']]\n        return { 'fields': fields, 'values': values }\n\n    template_payload = {\"fields\": [],\n                        \"values\": [[\"Male\", \"M\", 2, 48, \"Inactive\"], [\"Female\", \"S\", 0, 23, \"Inactive\"]]}\n\n    print(score_template(template_payload))", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Deployed models metadata extraction\n\nWe have to identify model deployments to be used before we define AI function. We extract scoring url and information about model version.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from watson_machine_learning_client import WatsonMachineLearningAPIClient"
        }, 
        {
            "source": "**Tip:** Put your Watson Machine Learning service credentials here.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by Watson Studio for sharing."
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = WatsonMachineLearningAPIClient(wml_credentials)"
        }, 
        {
            "source": "We list all deployed models predicting `Action` and `Business Area` to select one to be used in the AI function.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "------------------------------------  ---------------------------------------------------------  ------  --------------  ------------------------  --------------  ----------\nGUID                                  NAME                                                       TYPE    STATE           CREATED                   FRAMEWORK       ASSET TYPE\nf221801b-362e-48c1-8a5e-9850b4c57109  CARS4U - Satisfaction Prediction - AI Function Deployment  online  DEPLOY_SUCCESS  2018-07-31T13:48:12.424Z  n/a             function\n95702a10-e783-410a-9c84-2f0499554296  CARS4U - Satisfaction Prediction Model Deployment          online  DEPLOY_SUCCESS  2018-07-31T13:47:49.984Z  tensorflow-1.5  model\n7712d6b7-aca4-4b67-8d77-fbbce1680fc9  CARS4U - Action Model Deployment                           online  DEPLOY_SUCCESS  2018-07-31T13:45:11.869Z  mllib-2.1       model\n4bb27d68-f2ba-4825-b451-c745450d83b4  CARS4U - Business Area Prediction Model Deployment         online  DEPLOY_SUCCESS  2018-07-31T13:43:31.252Z  mllib-2.1       model\n------------------------------------  ---------------------------------------------------------  ------  --------------  ------------------------  --------------  ----------\n"
                }
            ], 
            "source": "client.deployments.list()"
        }, 
        {
            "source": "We select deployed model to predict `Business Area` and model to recommend `Action`. We have to extract scoring endpoints for two models.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Action url:  https://us-south.ml.cloud.ibm.com/v3/wml_instances/aaed6937-c0e7-4307-8a17-361aca257c7e/deployments/7712d6b7-aca4-4b67-8d77-fbbce1680fc9/online\nBusiness area url:  https://us-south.ml.cloud.ibm.com/v3/wml_instances/aaed6937-c0e7-4307-8a17-361aca257c7e/deployments/4bb27d68-f2ba-4825-b451-c745450d83b4/online\n"
                }
            ], 
            "source": "deployments = client.deployments.get_details()['resources']\n\nfor x in deployments:\n    if (x['entity']['name'] == 'CARS4U - Action Model Deployment'):\n        action_scoring_url = client.deployments.get_scoring_url(x)\n    if (x['entity']['name'] == 'CARS4U - Business Area Prediction Model Deployment'):\n        business_area_scoring_url = client.deployments.get_scoring_url(x)\n        \nprint(\"Action url: \", action_scoring_url)\nprint(\"Business area url: \", business_area_scoring_url)"
        }, 
        {
            "source": "### AI function definition\n\nWe define AI function to be deployed in the IBM Cloud.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Put all parameters required to define AI function in dictionary.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "ai_params = {\"wml_credentials\": wml_credentials, \n             \"business_area_scoring_url\": business_area_scoring_url, \n             \"action_scoring_url\": action_scoring_url}"
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "def score_generator(params=ai_params):\n    \n    from watson_machine_learning_client import WatsonMachineLearningAPIClient\n\n    wml_credentials = params[\"wml_credentials\"]\n    business_area_scoring_url = params[\"business_area_scoring_url\"]\n    action_scoring_url = params[\"action_scoring_url\"]\n\n    client = WatsonMachineLearningAPIClient(wml_credentials)\n\n    def score(payload):\n        \"\"\"AI function with model version.\n\n        Example:\n          {\"fields\": [\"ID\", \"Gender\", \"Status\", \"Children\", \"Age\", \"Customer_Status\", \"Car_Owner\", \"Customer_Service\", \"Satisfaction\"],\n           \"values\": [[2624, 'Male', 'S', 0, 49.27, 'Active', 'No', \"Good experience with all the rental co.'s I contacted. I Just called with rental dates and received pricing and selected rental co.\", 1]]}\n        \"\"\"\n        \n        scores_area = client.deployments.score(business_area_scoring_url, payload)\n        idx_area = [i for i, x in enumerate(scores_area[\"fields\"]) if (x == 'probability') or (x == 'predictedLabel')]\n\n        fields_from_action = payload[\"fields\"] + [\"predictedLabel\"]\n        idx_from_action = set([i for i, x in enumerate(scores_area[\"fields\"]) if x in fields_from_action])\n        values_action = [[v for i, v in enumerate(rec) if i in idx_from_action] for rec in scores_area[\"values\"]]\n        \n        payload_action = {\"fields\": payload[\"fields\"] + [\"Business_Area\"], \n                          \"values\": values_action}\n        \n        scores_action = client.deployments.score(action_scoring_url, payload_action)\n        idx_action = [i for i, x in enumerate(scores_action[\"fields\"]) if (x == 'probability') or (x == 'predictedLabel')]\n\n        values = [rec + [scores_area['values'][i][idx_area[0]], scores_area['values'][i][idx_area[1]], scores_action['values'][i][idx_action[0]], scores_action['values'][i][idx_action[1]]] for i, rec in enumerate(payload['values'])]\n\n        fields = payload['fields'] + ['Probability_Area', 'Prediction_Area'] + ['Probability_Action', 'Prediction_Action']\n\n        return {'fields': fields, 'values': values}\n\n    return score"
        }, 
        {
            "source": "### AI function local test\n\nLet's test defined AI function in notebook before we deploy it in the IBM Cloud. We start with preparing sample payload for `score` function.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "sample_payload = {\"fields\": [\"ID\", \"Gender\", \"Status\", \"Children\", \"Age\", \"Customer_Status\", \"Car_Owner\", \"Customer_Service\", \"Satisfaction\"],\n                  \"values\": [[2624, 'Male', 'S', 0, 49.27, 'Active', 'No', \"Good experience with all the rental co.'s I contacted. I Just called with rental dates and received pricing and selected rental co.\", 1]]}       "
        }, 
        {
            "source": "Let's test AI function locally before we deploy it to the IBM Cloud.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'values': [[2624, 'Male', 'S', 0, 49.27, 'Active', 'No', \"Good experience with all the rental co.'s I contacted. I Just called with rental dates and received pricing and selected rental co.\", 1, [0.3574468085106383, 0.3191489361702128, 0.0425531914893617, 0.06808510638297872, 0.07234042553191489, 0.059574468085106386, 0.05106382978723404, 0.029787234042553193], 'Service: Knowledge', [1.0, 0.0, 0.0, 0.0, 0.0], 'NA']], 'fields': ['ID', 'Gender', 'Status', 'Children', 'Age', 'Customer_Status', 'Car_Owner', 'Customer_Service', 'Satisfaction', 'Probability_Area', 'Prediction_Area', 'Probability_Action', 'Prediction_Action']}\n"
                }
            ], 
            "source": "score = score_generator()\nscores_ai = score(sample_payload)\nprint(scores_ai)"
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 3. Store the AI function in the repository", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section we store AI function to Watson machine Learning repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Required prerequisite to store AI function is runtime specification.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "runtime_meta = {\n            client.runtime_specs.ConfigurationMetaNames.NAME: \"Basic runtime specification\",\n            client.runtime_specs.ConfigurationMetaNames.DESCRIPTION: \"Runtime for AI function\",\n            client.runtime_specs.ConfigurationMetaNames.PLATFORM: {\n               \"name\": \"python\",\n               \"version\": \"3.5\"\n             }}"
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://us-south.ml.cloud.ibm.com/v4/runtimes/ae473ac0-c10e-413b-b349-1adbc5b1ab61\n"
                }
            ], 
            "source": "runtime_details = client.runtime_specs.create(runtime_meta)\nruntime_url = client.runtime_specs.get_url(runtime_details)\nprint(runtime_url)"
        }, 
        {
            "source": "Now we are ready to save function to Watson Machine Learning repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Recognized generator function.\n"
                }
            ], 
            "source": "meta_data = {\n    client.repository.FunctionMetaNames.NAME: 'CARS4U - Business area and Action Prediction - AI Function',\n    client.repository.FunctionMetaNames.RUNTIME_URL: runtime_url\n}\n\nfunction_details = client.repository.store_function(meta_props=meta_data, function=score_generator)"
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "4dbe7c73-9a1d-4162-96e9-6518b0781e19\n"
                }
            ], 
            "source": "ai_function_uid = client.repository.get_function_uid(function_details)\nprint(ai_function_uid)"
        }, 
        {
            "source": "<a id=\"deploy\"></a>\n## 4. Deploy AI function", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section we deploy AI function in the IBM Cloud and test created deployment using sample payload.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n#######################################################################################\n\nSynchronous deployment creation for uid: '4dbe7c73-9a1d-4162-96e9-6518b0781e19' started\n\n#######################################################################################\n\n\nINITIALIZING\nDEPLOY_IN_PROGRESS....\nDEPLOY_SUCCESS\n\n\n------------------------------------------------------------------------------------------------\nSuccessfully finished deployment creation, deployment_uid='f466aea9-62f3-444a-a79a-5eff98053029'\n------------------------------------------------------------------------------------------------\n\n\n"
                }
            ], 
            "source": "function_deployment_details = client.deployments.create(ai_function_uid, \"CARS4U - Business area and Action Prediction - AI Function Deployment\")"
        }, 
        {
            "source": "### AI function online deployment test\n\nNow we are ready to test created deployment.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "scoring_url = client.deployments.get_scoring_url(function_deployment_details)"
        }, 
        {
            "execution_count": 17, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'values': [[2624, 'Male', 'S', 0, 49.27, 'Active', 'No', \"Good experience with all the rental co.'s I contacted. I Just called with rental dates and received pricing and selected rental co.\", 1, [0.3574468085106383, 0.3191489361702128, 0.0425531914893617, 0.06808510638297872, 0.07234042553191489, 0.059574468085106386, 0.05106382978723404, 0.029787234042553193], 'Service: Knowledge', [1.0, 0.0, 0.0, 0.0, 0.0], 'NA']], 'fields': ['ID', 'Gender', 'Status', 'Children', 'Age', 'Customer_Status', 'Car_Owner', 'Customer_Service', 'Satisfaction', 'Probability_Area', 'Prediction_Area', 'Probability_Action', 'Prediction_Action']}\n"
                }
            ], 
            "source": "scoring_results = client.deployments.score(scoring_url, sample_payload)\nprint(scoring_results)"
        }, 
        {
            "source": "---", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.5", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}