cpd4.5/notebooks/python_sdk/deployments/custom_library/Use scikit-learn and custom library to predict temperature.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Use scikit-learn and custom library to predict temperature with `ibm-watson-machine-learning`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook contains steps and code to train a Scikit-Learn model that uses a custom defined transformer and use it with Watson Machine Learning service. Once the model is trained, this notebook contains steps to persist the model and custom defined transformer to Watson Machine Learning Repository, deploy and score it using Watson Machine Learning python client.\n",
        "\n",
        "In this notebook, we use GNFUV dataset that contains mobile sensor readings data about humidity and temperature from Unmanned Surface Vehicles in a test-bed in Athens, to train a Scikit-Learn model for predicting the temperature. \n",
        "\n",
        "Some familiarity with Python is helpful. This notebook uses Python-3.8 and scikit-learn."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Learning goals\n",
        "\n",
        "The learning goals of this notebook are:\n",
        "\n",
        "- Train a model with custom defined transformer\n",
        "- Persist the custom defined transformer and the model in Watson Machine Learning repository.\n",
        "- Deploy the model using Watson Machine Learning Service\n",
        "- Perform predictions using the deployed model\n",
        "\n",
        "## Contents\n",
        "1.\t[Set up the environment](#setup)\n",
        "2.\t[Install python library containing custom transformer implementation](#install_lib)\n",
        "3.  [Prepare training data](#load)\n",
        "4.\t[Train the scikit-learn model](#train)\n",
        "5.\t[Save the model and library to WML Repository](#upload)\n",
        "6.\t[Deploy and score data](#deploy)\n",
        "7.  [Clean up](#cleanup)\n",
        "8.\t[Summary and next steps](#summary)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"setup\"></a>\n",
        "## 1. Set up the environment\n",
        "\n",
        "Before you use the sample code in this notebook, you must perform the following setup tasks:\n",
        "\n",
        "-  Contact with your Cloud Pack for Data administrator and ask him for your account credentials"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Connection to WML\n",
        "\n",
        "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "username = 'PASTE YOUR USERNAME HERE'\n",
        "api_key = 'PASTE YOUR API_KEY HERE'\n",
        "url = 'PASTE THE PLATFORM URL HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "wml_credentials = {\n",
        "    \"username\": username,\n",
        "    \"apikey\": api_key,\n",
        "    \"url\": url,\n",
        "    \"instance_id\": 'openshift',\n",
        "    \"version\": '4.5'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Alternatively you can use `username` and `password` to authenticate WML services.\n",
        "\n",
        "```\n",
        "wml_credentials = {\n",
        "    \"username\": ***,\n",
        "    \"password\": ***,\n",
        "    \"url\": ***,\n",
        "    \"instance_id\": 'openshift',\n",
        "    \"version\": '4.5'\n",
        "}\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install and import the `ibm-watson-machine-learning` package\n",
        "**Note:** `ibm-watson-machine-learning` documentation can be found <a href=\"http://ibm-wml-api-pyclient.mybluemix.net/\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install -U ibm-watson-machine-learning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "from ibm_watson_machine_learning import APIClient\n",
        "\n",
        "client = APIClient(wml_credentials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Working with spaces\n",
        "\n",
        "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
        "\n",
        "- Click New Deployment Space\n",
        "- Create an empty space\n",
        "- Go to space `Settings` tab\n",
        "- Copy `space_id` and paste it below\n",
        "\n",
        "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
        "\n",
        "**Action**: Assign space ID below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "space_id = 'PASTE YOUR SPACE ID HERE'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can use `list` method to print all existing spaces."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "client.spaces.list(limit=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To be able to interact with all resources available in Watson Machine Learning, you need to set **space** which you will be using."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.set.default_space(space_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"install_lib\"></a>\n",
        "\n",
        "## 2. Install the library containing custom transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Library - `linalgnorm-0.1` is a python distributable package that contains the implementation of a user defined Scikit-Learn transformer - `LNormalizer` . <br>\n",
        "Any 3rd party libraries that are required for the custom transformer must be defined as the dependency for the corresponding library that contains implementation of the transformer. \n",
        "\n",
        "\n",
        "In this section, we will create the library and install it in the current notebook environment. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "!mkdir -p linalgnorm-0.1/linalg_norm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define a custom scikit transformer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Writing linalgnorm-0.1/linalg_norm/sklearn_transformers.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile linalgnorm-0.1/linalg_norm/sklearn_transformers.py\n",
        "\n",
        "from sklearn.base import BaseEstimator, TransformerMixin\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "class LNormalizer(BaseEstimator, TransformerMixin):\n",
        "    def __init__(self, norm_ord=2):\n",
        "        self.norm_ord = norm_ord\n",
        "        self.row_norm_vals = None\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        self.row_norm_vals = np.linalg.norm(X, ord=self.norm_ord, axis=0)\n",
        "\n",
        "    def transform(self, X, y=None):\n",
        "        return X / self.row_norm_vals\n",
        "\n",
        "    def fit_transform(self, X, y=None):\n",
        "        self.fit(X, y)\n",
        "        return self.transform(X, y)\n",
        "\n",
        "    def get_norm_vals(self):\n",
        "        return self.row_norm_vals"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Wrap created code into Python source distribution package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Writing linalgnorm-0.1/linalg_norm/__init__.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile linalgnorm-0.1/linalg_norm/__init__.py\n",
        "\n",
        "__version__ = \"0.1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Writing linalgnorm-0.1/README.md\n"
          ]
        }
      ],
      "source": [
        "%%writefile linalgnorm-0.1/README.md\n",
        "\n",
        "A simple library containing a simple custom scikit estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Writing linalgnorm-0.1/setup.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile linalgnorm-0.1/setup.py\n",
        "\n",
        "from setuptools import setup\n",
        "\n",
        "VERSION='0.1'\n",
        "setup(name='linalgnorm',\n",
        "      version=VERSION,\n",
        "      url='https://github.ibm.com/NGP-TWC/repository/',\n",
        "      author='IBM',\n",
        "      author_email='ibm@ibm.com',\n",
        "      license='IBM',\n",
        "      packages=[\n",
        "            'linalg_norm'\n",
        "      ],\n",
        "      zip_safe=False\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "running sdist\n",
            "running egg_info\n",
            "creating linalgnorm.egg-info\n",
            "writing linalgnorm.egg-info/PKG-INFO\n",
            "writing dependency_links to linalgnorm.egg-info/dependency_links.txt\n",
            "writing top-level names to linalgnorm.egg-info/top_level.txt\n",
            "writing manifest file 'linalgnorm.egg-info/SOURCES.txt'\n",
            "reading manifest file 'linalgnorm.egg-info/SOURCES.txt'\n",
            "writing manifest file 'linalgnorm.egg-info/SOURCES.txt'\n",
            "running check\n",
            "creating linalgnorm-0.1\n",
            "creating linalgnorm-0.1/linalg_norm\n",
            "creating linalgnorm-0.1/linalgnorm.egg-info\n",
            "copying files to linalgnorm-0.1...\n",
            "copying README.md -> linalgnorm-0.1\n",
            "copying setup.py -> linalgnorm-0.1\n",
            "copying linalg_norm/__init__.py -> linalgnorm-0.1/linalg_norm\n",
            "copying linalg_norm/sklearn_transformers.py -> linalgnorm-0.1/linalg_norm\n",
            "copying linalgnorm.egg-info/PKG-INFO -> linalgnorm-0.1/linalgnorm.egg-info\n",
            "copying linalgnorm.egg-info/SOURCES.txt -> linalgnorm-0.1/linalgnorm.egg-info\n",
            "copying linalgnorm.egg-info/dependency_links.txt -> linalgnorm-0.1/linalgnorm.egg-info\n",
            "copying linalgnorm.egg-info/not-zip-safe -> linalgnorm-0.1/linalgnorm.egg-info\n",
            "copying linalgnorm.egg-info/top_level.txt -> linalgnorm-0.1/linalgnorm.egg-info\n",
            "Writing linalgnorm-0.1/setup.cfg\n",
            "creating dist\n",
            "creating 'dist/linalgnorm-0.1.zip' and adding 'linalgnorm-0.1' to it\n",
            "adding 'linalgnorm-0.1'\n",
            "adding 'linalgnorm-0.1/linalg_norm'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info'\n",
            "adding 'linalgnorm-0.1/PKG-INFO'\n",
            "adding 'linalgnorm-0.1/README.md'\n",
            "adding 'linalgnorm-0.1/setup.py'\n",
            "adding 'linalgnorm-0.1/setup.cfg'\n",
            "adding 'linalgnorm-0.1/linalg_norm/sklearn_transformers.py'\n",
            "adding 'linalgnorm-0.1/linalg_norm/__init__.py'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info/PKG-INFO'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info/not-zip-safe'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info/SOURCES.txt'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info/top_level.txt'\n",
            "adding 'linalgnorm-0.1/linalgnorm.egg-info/dependency_links.txt'\n",
            "removing 'linalgnorm-0.1' (and everything under it)\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "\n",
        "cd linalgnorm-0.1\n",
        "python setup.py sdist --formats=zip\n",
        "cd ..\n",
        "mv linalgnorm-0.1/dist/linalgnorm-0.1.zip .\n",
        "rm -rf linalgnorm-0.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Install the downloaded library using `pip` command"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Processing ./linalgnorm-0.1.zip\n",
            "Building wheels for collected packages: linalgnorm\n",
            "  Building wheel for linalgnorm (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Created wheel for linalgnorm: filename=linalgnorm-0.1-py3-none-any.whl size=1670 sha256=5416b34c623f8502515a75d8f9de1f6fce41fe55cd31ab9ab87863e6f7f9df23\n",
            "  Stored in directory: /Users/jansoltysik/Library/Caches/pip/wheels/78/00/7b/c263b6176f7c38c807f442edaa5f11a3e7a2cbcc5fa07b2673\n",
            "Successfully built linalgnorm\n",
            "Installing collected packages: linalgnorm\n",
            "  Attempting uninstall: linalgnorm\n",
            "    Found existing installation: linalgnorm 0.1\n",
            "    Uninstalling linalgnorm-0.1:\n",
            "      Successfully uninstalled linalgnorm-0.1\n",
            "Successfully installed linalgnorm-0.1\n"
          ]
        }
      ],
      "source": [
        "!pip install linalgnorm-0.1.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"load\"></a>\n",
        "\n",
        "## 3. Download training dataset and prepare training data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Download the data from UCI repository - https://archive.ics.uci.edu/ml/machine-learning-databases/00452/GNFUV%20USV%20Dataset.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "!rm -rf dataset\n",
        "!mkdir dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2020-12-08 12:45:12--  https://archive.ics.uci.edu/ml/machine-learning-databases/00452/GNFUV%20USV%20Dataset.zip\n",
            "Resolving archive.ics.uci.edu... 128.195.10.252\n",
            "Connecting to archive.ics.uci.edu|128.195.10.252|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 501978 (490K) [application/x-httpd-php]\n",
            "Saving to: 'dataset/gnfuv_dataset.zip'\n",
            "\n",
            "dataset/gnfuv_datas 100%[===================>] 490.21K   119KB/s    in 4.1s    \n",
            "\n",
            "2020-12-08 12:45:17 (119 KB/s) - 'dataset/gnfuv_dataset.zip' saved [501978/501978]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://archive.ics.uci.edu/ml/machine-learning-databases/00452/GNFUV%20USV%20Dataset.zip --output-document=dataset/gnfuv_dataset.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Archive:  dataset/gnfuv_dataset.zip\n",
            "  inflating: dataset/pi2/gnfuv-temp-exp1-55d487b85b-5g2xh_1.0.csv  \n",
            "  inflating: dataset/pi3/gnfuv-temp-exp1-55d487b85b-2bl8b_1.0.csv  \n",
            "  inflating: dataset/pi4/gnfuv-temp-exp1-55d487b85b-xcl97_1.0.csv  \n",
            "  inflating: dataset/pi5/gnfuv-temp-exp1-55d487b85b-5ztk8_1.0.csv  \n",
            "  inflating: dataset/README.pdf      \n"
          ]
        }
      ],
      "source": [
        "!unzip dataset/gnfuv_dataset.zip -d dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create pandas datafame based on the downloaded dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import os\n",
        "from datetime import datetime\n",
        "from json import JSONDecodeError"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [],
      "source": [
        "home_dir = './dataset'\n",
        "pi_dirs = os.listdir(home_dir)\n",
        "\n",
        "data_list = []\n",
        "base_time = None\n",
        "columns = None\n",
        "\n",
        "for pi_dir in pi_dirs:\n",
        "    if 'pi' not in pi_dir:\n",
        "        continue\n",
        "    curr_dir = os.path.join(home_dir, pi_dir)\n",
        "    data_file = os.path.join(curr_dir, os.listdir(curr_dir)[0])\n",
        "    with open(data_file, 'r') as f:\n",
        "        line = f.readline().strip().replace(\"'\", '\"')\n",
        "        while line != '':\n",
        "            try:\n",
        "                input_json = json.loads(line)\n",
        "                sensor_datetime = datetime.fromtimestamp(input_json['time'])\n",
        "                if base_time is None:\n",
        "                    base_time = datetime(sensor_datetime.year, sensor_datetime.month, sensor_datetime.day, 0, 0, 0, 0)\n",
        "                input_json['time'] = (sensor_datetime - base_time).seconds\n",
        "                data_list.append(list(input_json.values()))\n",
        "                if columns is None:\n",
        "                    columns = list(input_json.keys())\n",
        "            except JSONDecodeError as je:\n",
        "                pass\n",
        "            line = f.readline().strip().replace(\"'\", '\"')\n",
        "\n",
        "data_df = pd.DataFrame(data_list, columns=columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>device</th>\n",
              "      <th>humidity</th>\n",
              "      <th>temperature</th>\n",
              "      <th>experiment</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>gnfuv-temp-exp1-55d487b85b-5g2xh</td>\n",
              "      <td>21.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>69557</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>gnfuv-temp-exp1-55d487b85b-5g2xh</td>\n",
              "      <td>21.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>69571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>gnfuv-temp-exp1-55d487b85b-5g2xh</td>\n",
              "      <td>21.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>69577</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>gnfuv-temp-exp1-55d487b85b-5g2xh</td>\n",
              "      <td>21.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>69583</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>gnfuv-temp-exp1-55d487b85b-5g2xh</td>\n",
              "      <td>22.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>69589</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                             device  humidity  temperature  experiment   time\n",
              "0  gnfuv-temp-exp1-55d487b85b-5g2xh      21.0         40.0         1.0  69557\n",
              "1  gnfuv-temp-exp1-55d487b85b-5g2xh      21.0         40.0         1.0  69571\n",
              "2  gnfuv-temp-exp1-55d487b85b-5g2xh      21.0         40.0         1.0  69577\n",
              "3  gnfuv-temp-exp1-55d487b85b-5g2xh      21.0         40.0         1.0  69583\n",
              "4  gnfuv-temp-exp1-55d487b85b-5g2xh      22.0         40.0         1.0  69589"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create training and test datasets from the downloaded GNFUV-USV dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "Y = data_df['temperature']\n",
        "X = data_df.drop('temperature', axis=1)\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=143)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"train\"></a>\n",
        "\n",
        "## 4. Train a model\n",
        "\n",
        "In this section, you will use the custom transformer as a stage in the Scikit-Learn `Pipeline` and train a model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Import the custom transformer \n",
        "Here, import the custom transformer that has been defined in `linalgnorm-0.1.zip` and create an instance of it that will inturn be used as stage in `sklearn.Pipeline`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [],
      "source": [
        "from linalg_norm.sklearn_transformers import LNormalizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [],
      "source": [
        "lnorm_transf = LNormalizer()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Import other objects required to train a model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.linear_model import LinearRegression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now, you can create a `Pipeline` with user defined transformer as one of the stages and train the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Pipeline(steps=[('normalizer', LNormalizer()),\n",
              "                ('regression_estimator', LinearRegression())])"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "skl_pipeline = Pipeline(steps=[('normalizer', lnorm_transf), ('regression_estimator', LinearRegression())])\n",
        "skl_pipeline.fit(X_train.loc[:, ['time', 'humidity']].values, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "RMSE: 2.213758431322581\n"
          ]
        }
      ],
      "source": [
        "y_pred = skl_pipeline.predict(X_test.loc[:, ['time', 'humidity']].values)\n",
        "rmse = np.mean((np.round(y_pred) - y_test.values)**2)**0.5\n",
        "print('RMSE: {}'.format(rmse))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"upload\"></a>\n",
        "\n",
        "## 5. Persist the model and custom library\n",
        "\n",
        "In this section, using `ibm-watson_machine_learning` SDK, you will ...\n",
        "- save the library `linalgnorm-0.1.zip` in WML Repository by creating a package extension resource\n",
        "- create a Software Specification resource and bind the package resource to it. This Software Specification resource will be used to configure the online deployment runtime environment for a model \n",
        "- bind Software Specification resource to the model and save the model to WML Repository"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create package extension"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the meta data required to create package extension resource. <br>\n",
        "\n",
        "The value for `file_path` in `client.package_extensions.LibraryMetaNames.store()` contains the library file name that must be uploaded to the WML.\n",
        "\n",
        "**Note:** You can also use conda environment configuration file `yaml` as package extension input. In such case set the `TYPE` to `conda_yml` and `file_path` to yaml file.\n",
        "```\n",
        "client.package_extensions.ConfigurationMetaNames.TYPE = \"conda_yml\"\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating package extensions\n",
            "SUCCESS\n"
          ]
        }
      ],
      "source": [
        "meta_prop_pkg_extn = {\n",
        "    client.package_extensions.ConfigurationMetaNames.NAME: \"K_Linag_norm_skl\",\n",
        "    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: \"Pkg extension for custom lib\",\n",
        "    client.package_extensions.ConfigurationMetaNames.TYPE: \"pip_zip\"\n",
        "}\n",
        "\n",
        "pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path=\"linalgnorm-0.1.zip\")\n",
        "pkg_extn_uid = client.package_extensions.get_uid(pkg_extn_details)\n",
        "pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Display the details of the package extension resource that was created in the above cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [],
      "source": [
        "details = client.package_extensions.get_details(pkg_extn_uid)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create software specification and add custom library"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the meta data required to create software spec resource and bind the package. This software spec resource will be used to configure the online deployment runtime environment for a model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "---------------------------  ----  --------  --------------------------------\n",
            "META_PROP NAME               TYPE  REQUIRED  SCHEMA\n",
            "NAME                         str   Y\n",
            "DESCRIPTION                  str   N\n",
            "PACKAGE_EXTENSIONS           list  N\n",
            "SOFTWARE_CONFIGURATION       dict  N         {'platform(required)': 'string'}\n",
            "BASE_SOFTWARE_SPECIFICATION  dict  Y\n",
            "---------------------------  ----  --------  --------------------------------\n"
          ]
        }
      ],
      "source": [
        "client.software_specifications.ConfigurationMetaNames.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### List base software specifications"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "client.software_specifications.list()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Select base software specification to extend"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [],
      "source": [
        "base_sw_spec_uid = client.software_specifications.get_uid_by_name(\"runtime-22.1-py3.9\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define new software specification based on base one and custom library"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "SUCCESS\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "meta_prop_sw_spec = {\n",
        "    client.software_specifications.ConfigurationMetaNames.NAME: \"linalgnorm-0.1\",\n",
        "    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: \"Software specification for linalgnorm-0.1\",\n",
        "    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {\"guid\": base_sw_spec_uid}\n",
        "}\n",
        "\n",
        "sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)\n",
        "sw_spec_uid = client.software_specifications.get_uid(sw_spec_details)\n",
        "\n",
        "\n",
        "client.software_specifications.add_package_extension(sw_spec_uid, pkg_extn_uid)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Save the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the metadata to save the trained model to WML Repository along with the information about the software spec resource required for the model. \n",
        "\n",
        "The `client.repository.ModelMetaNames.SOFTWARE_SPEC_UID` metadata property is used to specify the GUID of the software spec resource that needs to be associated with the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_props = {\n",
        "    client.repository.ModelMetaNames.NAME: \"Temp prediction model with custom lib\",\n",
        "    client.repository.ModelMetaNames.TYPE: 'scikit-learn_1.0',\n",
        "    client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: sw_spec_uid\n",
        "    \n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Save the model to the WML Repository and display its saved metadata. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [],
      "source": [
        "published_model = client.repository.store_model(model=skl_pipeline, meta_props=model_props)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"entity\": {\n",
            "    \"software_spec\": {\n",
            "      \"id\": \"dae27ebc-97b7-450e-bd27-60bd1e5ca198\",\n",
            "      \"name\": \"linalgnorm-0.1\"\n",
            "    },\n",
            "    \"type\": \"scikit-learn_0.23\"\n",
            "  },\n",
            "  \"metadata\": {\n",
            "    \"created_at\": \"2020-12-08T11:45:55.651Z\",\n",
            "    \"id\": \"a6a27638-71ee-493c-b6b6-d8488958b974\",\n",
            "    \"modified_at\": \"2020-12-08T11:45:57.475Z\",\n",
            "    \"name\": \"Temp prediction model with custom lib\",\n",
            "    \"owner\": \"1000330999\",\n",
            "    \"space_id\": \"83b00166-9047-4159-b777-83dcb498e7ab\"\n",
            "  },\n",
            "  \"system\": {\n",
            "    \"warnings\": []\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "published_model_uid = client.repository.get_model_id(published_model)\n",
        "model_details = client.repository.get_details(published_model_uid)\n",
        "print(json.dumps(model_details, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"deploy\"></a>\n",
        "\n",
        "## 6 Deploy and Score\n",
        "\n",
        "In this section, you will deploy the saved model that uses the custom transformer and perform predictions. You will use WML client to perform these tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Deploy the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "Synchronous deployment creation for uid: 'a6a27638-71ee-493c-b6b6-d8488958b974' started\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "\n",
            "initializing.\n",
            "ready\n",
            "\n",
            "\n",
            "------------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_uid='0d94523d-a2e7-4d75-b753-648fae333747'\n",
            "------------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "metadata = {\n",
        "    client.deployments.ConfigurationMetaNames.NAME: \"Deployment of custom lib model\",\n",
        "    client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
        "}\n",
        "\n",
        "created_deployment = client.deployments.create(published_model_uid, meta_props=metadata)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"score\"></a>\n",
        "### Predict using the deployed model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Note**: Here we use deployment `uid` saved in published_model object. In next section, we show how to retrive deployment url from Watson Machine Learning instance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [],
      "source": [
        "deployment_uid = client.deployments.get_id(created_deployment)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now you can print an online scoring endpoint. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "https://wmlgmc-cpd-wmlgmc.apps.wmlautoai.cp.fyre.ibm.com/ml/v4/deployments/0d94523d-a2e7-4d75-b753-648fae333747/predictions\n"
          ]
        }
      ],
      "source": [
        "scoring_endpoint = client.deployments.get_scoring_href(created_deployment)\n",
        "print(scoring_endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Prepare the payload for prediction. The payload contains the input records for which predictions has to be performed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [],
      "source": [
        "scoring_payload = {\n",
        "    \"input_data\": [{\n",
        "        'fields': [\"time\", \"humidity\"],\n",
        "        'values': [[79863, 47]]}]\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Execute the method to perform online predictions and display the prediction results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [],
      "source": [
        "predictions = client.deployments.score(deployment_uid, scoring_payload)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"predictions\": [\n",
            "    {\n",
            "      \"fields\": [\n",
            "        \"prediction\"\n",
            "      ],\n",
            "      \"values\": [\n",
            "        [\n",
            "          14.629242312262988\n",
            "        ]\n",
            "      ]\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(json.dumps(predictions, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"cleanup\"></a>\n",
        "## 7. Clean up "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you want to clean up all created assets:\n",
        "- experiments\n",
        "- trainings\n",
        "- pipelines\n",
        "- model definitions\n",
        "- models\n",
        "- functions\n",
        "- deployments\n",
        "\n",
        "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"summary\"></a>\n",
        "\n",
        "## 8. Summary\n",
        "\n",
        "You successfully completed this notebook! \n",
        " \n",
        "You learned how to use a scikit-learn model with custom transformer in Watson Machine Learning service to deploy and score.\n",
        "\n",
        "Check out our [Online Documentation](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=deploying-managing-models-functions) for more samples, tutorials, documentation, how-tos, and blog posts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Author\n",
        "\n",
        "**Krishnamurthy Arthanarisamy**, is a senior technical lead in IBM Watson Machine Learning team. Krishna works on developing cloud services that caters to different stages of machine learning and deep learning modeling life cycle.\n",
        "\n",
        "**Lukasz Cmielowski**, PhD, is a Software Architect and Data Scientist at IBM."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}