cloud/notebooks/python_sdk/deployments/foundation_models/Use watsonx, and LangChain to make a series of calls to a language model.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "![image](https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cloud/notebooks/headers/watsonx-Prompt_Lab-Notebook.png)\n",
        "# Use watsonx, and LangChain to make a series of calls to a language model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Disclaimers\n",
        "\n",
        "- Use only Projects and Spaces that are available in watsonx context.\n",
        "\n",
        "\n",
        "## Notebook content\n",
        "\n",
        "This notebook contains the steps and code to demonstrate Sequential Chain using langchain integration with watsonx models.\n",
        "\n",
        "Some familiarity with Python is helpful. This notebook uses Python 3.11.\n",
        "\n",
        "\n",
        "## Learning goal\n",
        "\n",
        "The goal of this notebook is to demonstrate how to chain `google/flan-ul2` and `google/flan-t5-xxl` models to generate a sequence of creating a random question on a given topic and an answer to that question and also to make the user friends with LangChain framework, using simple chain (LLMChain) and the extended chain (SequentialChain) with the WatsonxLLM.\n",
        "\n",
        "\n",
        "## Contents\n",
        "\n",
        "This notebook contains the following parts:\n",
        "\n",
        "- [Setup](#setup)\n",
        "- [Foundation Models on watsonx](#models)\n",
        "- [LangChain integration](#watsonxllm)\n",
        "- [Sequential Chain experiment](#experiment)\n",
        "- [Python function](#function)\n",
        "- [Custom inference endpoint](#deployment)\n",
        "- [Scoring](#scoring)\n",
        "- [Summary](#summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"setup\"></a>\n",
        "## Set up the environment\n",
        "\n",
        "Before you use the sample code in this notebook, you must perform the following setup tasks:\n",
        "\n",
        "-  Create a <a href=\"https://cloud.ibm.com/catalog/services/watsonxai-runtime\" target=\"_blank\" rel=\"noopener no referrer\">watsonx.ai Runtime Service</a> instance (a free plan is offered and information about how to create the instance can be found <a href=\"https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html?context=wx&audience=wdp\" target=\"_blank\" rel=\"noopener no referrer\">here</a>)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Install and import the `datasets` and dependecies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "!pip install -U \"langchain>=0.3,<0.4\" | tail -n 1\n",
        "!pip install -U \"langchain_ibm>=0.3,<0.4\" | tail -n 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Defining the watsonx.ai credentials\n",
        "This cell defines the watsonx.ai credentials required to work with watsonx Foundation Model inferencing.\n",
        "\n",
        "**Action:** Provide the IBM Cloud user API key. For details, see <a href=\"https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui\" target=\"_blank\" rel=\"noopener no referrer\">documentation</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "import getpass\n",
        "from ibm_watsonx_ai import Credentials\n",
        "\n",
        "credentials = Credentials(\n",
        "    url=\"https://us-south.ml.cloud.ibm.com\",\n",
        "    api_key=getpass.getpass(\"Please enter your watsonx.ai api key (hit enter): \"),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Defining the project id\n",
        "The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "try:\n",
        "    project_id = os.environ[\"PROJECT_ID\"]\n",
        "except KeyError:\n",
        "    project_id = input(\"Please enter your project_id (hit enter): \")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create an instance of APIClient with authentication details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai import APIClient\n",
        "\n",
        "api_client = APIClient(credentials=credentials, project_id=project_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"models\"></a>\n",
        "## Foundation Models on `watsonx.ai`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### List available models\n",
        "\n",
        "All avaliable models are presented under `TextModels` class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'MT0_XXL': 'bigscience/mt0-xxl', 'CODELLAMA_34B_INSTRUCT_HF': 'codellama/codellama-34b-instruct-hf', 'JAIS_13B_CHAT': 'core42/jais-13b-chat', 'ELYZA_JAPANESE_LLAMA_2_7B_INSTRUCT': 'elyza/elyza-japanese-llama-2-7b-instruct', 'FLAN_T5_XL': 'google/flan-t5-xl', 'FLAN_T5_XXL': 'google/flan-t5-xxl', 'FLAN_UL2': 'google/flan-ul2', 'GRANITE_13B_CHAT_V2': 'ibm/granite-13b-chat-v2', 'GRANITE_13B_INSTRUCT_V2': 'ibm/granite-13b-instruct-v2', 'GRANITE_20B_CODE_INSTRUCT': 'ibm/granite-20b-code-instruct', 'GRANITE_20B_MULTILINGUAL': 'ibm/granite-20b-multilingual', 'GRANITE_3_2B_INSTRUCT': 'ibm/granite-3-2b-instruct', 'GRANITE_3_8B_INSTRUCT': 'ibm/granite-3-8b-instruct', 'GRANITE_34B_CODE_INSTRUCT': 'ibm/granite-34b-code-instruct', 'GRANITE_3B_CODE_INSTRUCT': 'ibm/granite-3b-code-instruct', 'GRANITE_7B_LAB': 'ibm/granite-7b-lab', 'GRANITE_8B_CODE_INSTRUCT': 'ibm/granite-8b-code-instruct', 'GRANITE_8B_JAPANESE': 'ibm/granite-8b-japanese', 'GRANITE_GUARDIAN_3_2B': 'ibm/granite-guardian-3-2b', 'GRANITE_GUARDIAN_3_8B': 'ibm/granite-guardian-3-8b', 'LLAMA_2_13B_CHAT': 'meta-llama/llama-2-13b-chat', 'LLAMA_3_1_70B_INSTRUCT': 'meta-llama/llama-3-1-70b-instruct', 'LLAMA_3_1_8B_INSTRUCT': 'meta-llama/llama-3-1-8b-instruct', 'LLAMA_3_2_11B_VISION_INSTRUCT': 'meta-llama/llama-3-2-11b-vision-instruct', 'LLAMA_3_2_1B_INSTRUCT': 'meta-llama/llama-3-2-1b-instruct', 'LLAMA_3_2_3B_INSTRUCT': 'meta-llama/llama-3-2-3b-instruct', 'LLAMA_3_2_90B_VISION_INSTRUCT': 'meta-llama/llama-3-2-90b-vision-instruct', 'LLAMA_3_405B_INSTRUCT': 'meta-llama/llama-3-405b-instruct', 'LLAMA_3_70B_INSTRUCT': 'meta-llama/llama-3-70b-instruct', 'LLAMA_3_8B_INSTRUCT': 'meta-llama/llama-3-8b-instruct', 'LLAMA_GUARD_3_11B_VISION': 'meta-llama/llama-guard-3-11b-vision', 'MISTRAL_LARGE': 'mistralai/mistral-large', 'MIXTRAL_8X7B_INSTRUCT_V01': 'mistralai/mixtral-8x7b-instruct-v01', 'PIXTRAL_12B': 'mistralai/pixtral-12b', 'LLAMA2_13B_DPO_V7': 'mncai/llama2-13b-dpo-v7', 'ALLAM_1_13B_INSTRUCT': 'sdaia/allam-1-13b-instruct'}\n"
          ]
        }
      ],
      "source": [
        "api_client.foundation_models.TextModels.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You need to specify `model_id`'s that will be used for inferencing:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "model_id_1 = \"google/flan-ul2\"\n",
        "model_id_2 = \"google/flan-t5-xxl\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Defining the model parameters\n",
        "\n",
        "You might need to adjust model `parameters` for different models or tasks, to do so please refer to documentation under `GenTextParamsMetaNames` class.\n",
        "\n",
        "**Action:** If any complications please refer to the <a href=\"https://ibm.github.io/watsonx-ai-python-sdk/index.html\" target=\"_blank\" rel=\"noopener no referrer\">documentation</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams\n",
        "from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods\n",
        "\n",
        "parameters = {\n",
        "    GenParams.DECODING_METHOD: DecodingMethods.SAMPLE.value,\n",
        "    GenParams.MAX_NEW_TOKENS: 100,\n",
        "    GenParams.MIN_NEW_TOKENS: 1,\n",
        "    GenParams.TEMPERATURE: 0.5,\n",
        "    GenParams.TOP_K: 50,\n",
        "    GenParams.TOP_P: 1\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"watsonxllm\"></a>\n",
        "## LangChain integration\n",
        "\n",
        "`WatsonxLLM` is a wrapper around watsonx.ai models that provide chain integration around the models.\n",
        "\n",
        "**Action:** For more details about `CustomLLM` check the <a href=\"https://python.langchain.com/docs/modules/model_io/models/llms/custom_llm\" target=\"_blank\" rel=\"noopener no referrer\">LangChain documentation</a>\n",
        "\n",
        "\n",
        "### Initialize the `WatsonxLLM` class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from langchain_ibm import WatsonxLLM\n",
        "\n",
        "flan_ul2_llm = WatsonxLLM(\n",
        "    model_id=model_id_1,\n",
        "    url=credentials[\"url\"],\n",
        "    apikey=credentials[\"apikey\"],\n",
        "    project_id=project_id,\n",
        "    params=parameters\n",
        "    )\n",
        "flan_t5_llm = WatsonxLLM(\n",
        "    model_id=model_id_2,\n",
        "    url=credentials[\"url\"],\n",
        "    apikey=credentials[\"apikey\"],\n",
        "    project_id=project_id\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can print all set data about the WatsonxLLM object using the `dict()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'model_id': 'google/flan-ul2',\n",
              " 'deployment_id': None,\n",
              " 'params': {'decoding_method': 'sample',\n",
              "  'max_new_tokens': 100,\n",
              "  'min_new_tokens': 1,\n",
              "  'temperature': 0.5,\n",
              "  'top_k': 50,\n",
              "  'top_p': 1},\n",
              " 'project_id': 'd2436d2e-5814-4370-bd06-1754670e7d46',\n",
              " 'space_id': None,\n",
              " '_type': 'IBM watsonx.ai'}"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "flan_ul2_llm.dict()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"experiment\"></a>\n",
        "## Sequential Chain experiment\n",
        "The simplest type of sequential chain is called a `SequentialChain`, in which each step has a single input and output and the output of one step serves as the input for the following step.\n",
        "\n",
        "The experiment will consist in generating a random question about any topic and answer the following question."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "An object called `PromptTemplate` assists in generating prompts using a combination of user input, additional non-static data, and a fixed template string.\n",
        "\n",
        "In our case we would like to create two `PromptTemplate` objects which will be responsible for creating a random question and answering it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from langchain_core.prompts import PromptTemplate\n",
        "\n",
        "prompt_1 = PromptTemplate(\n",
        "    input_variables=[\"topic\"], \n",
        "    template=\"Generate a random question about {topic}: Question: \"\n",
        ")\n",
        "prompt_2 = PromptTemplate(\n",
        "    input_variables=[\"question\"],\n",
        "    template=\"Answer the following question: {question}\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "We would like to add functionality around language models using `LLMChain` chain.\n",
        "\n",
        "`prompt_to_flan_ul2` chain formats the prompt template whose task is to generate random question, passes the formatted string to LLM and returns the LLM output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from langchain.chains import LLMChain\n",
        "\n",
        "prompt_to_flan_ul2 = LLMChain(llm=flan_ul2_llm, prompt=prompt_1, output_key='question')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "`flan_to_t5` chain formats the prompt template whose task is to answer the question we got from `prompt_to_flan_ul2` chain, passes the formatted string to LLM and returns the LLM output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "flan_to_t5 = LLMChain(llm=flan_t5_llm, prompt=prompt_2, output_key='answer')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "This is the overall chain where we run `prompt_to_flan_ul2` and `flan_to_t5` chains in sequence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from langchain.chains import SequentialChain\n",
        "\n",
        "qa = SequentialChain(chains=[prompt_to_flan_ul2, flan_to_t5], input_variables=[\"topic\"], output_variables=['question', 'answer'], verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Generate random question and answer to topic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'topic': 'life',\n",
              " 'question': 'What is the name of the largest zoo in the world?',\n",
              " 'answer': 'zoo sao paulo brazil'}"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "qa.invoke({\"topic\": \"life\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"function\"></a>\n",
        "## Python function\n",
        "Let's wrap the chain code within python function that can be used as a content of inference endpoint."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Function implementation.\n",
        "Let's wrap the above chain code into function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "ai_params = {\n",
        "    \"url\": credentials[\"url\"],\n",
        "    \"apikey\": credentials[\"apikey\"],\n",
        "    \"project_id\": project_id,\n",
        "    \"generation_parameters\": parameters\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "def chain_text_generator(params=ai_params):\n",
        "    \n",
        "    from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes\n",
        "    from langchain import PromptTemplate\n",
        "    from langchain.chains import LLMChain, SequentialChain\n",
        "    from langchain_ibm import WatsonxLLM\n",
        "\n",
        "    url = params[\"url\"]\n",
        "    apikey = params[\"apikey\"]\n",
        "    project_id = params['project_id']\n",
        "    parameters = params['generation_parameters']\n",
        "    flan_ul2_llm = WatsonxLLM(model_id=ModelTypes.FLAN_UL2.value, url=url, apikey=apikey, project_id=project_id, params=parameters)\n",
        "    flan_t5_llm = WatsonxLLM(model_id=ModelTypes.FLAN_T5_XXL.value, url=url, apikey=apikey, project_id=project_id)\n",
        "    prompt_1 = PromptTemplate(input_variables=[\"topic\"], template=\"Generate a random question about {topic}: Question: \")\n",
        "    prompt_2 = PromptTemplate(input_variables=[\"question\"], template=\"Answer the following question: {question}\")\n",
        "    prompt_to_flan_ul2 = LLMChain(llm=flan_ul2_llm, prompt=prompt_1, output_key='question')\n",
        "    flan_to_t5 = LLMChain(llm=flan_t5_llm, prompt=prompt_2, output_key='answer')\n",
        "    chain = SequentialChain(chains=[prompt_to_flan_ul2, flan_to_t5], input_variables=[\"topic\"], output_variables=['question', 'answer'])\n",
        "\n",
        "    def score(payload):\n",
        "        \"\"\"Generates question based on provided topic and returns the answer.\"\"\"\n",
        "\n",
        "        answer = chain.invoke({\"topic\": payload[\"input_data\"][0]['values'][0][0]})\n",
        "        return {'predictions': [{'fields': ['topic', 'question', 'answer'], 'values': [answer['topic'], answer['question'], answer['answer']]}]}\n",
        "\n",
        "    return score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Test the function\n",
        "It is good practice to validate the code locally first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'predictions': [{'fields': ['topic', 'question', 'answer'],\n",
              "   'values': ['life',\n",
              "    'What is the term for the process of a living thing gaining nutrients from dead organisms?',\n",
              "    'decomposition']}]}"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sample_payload = {\n",
        "    \"input_data\": [\n",
        "        {\n",
        "            \"fields\": [\"topic\"],\n",
        "            \"values\": [[\"life\"]]\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "\n",
        "inference = chain_text_generator()\n",
        "inference(sample_payload)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"deployment\"></a>\n",
        "## Custom inference endpoint\n",
        "Create the online deployment of python function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Custom software specification\n",
        "Create new software specification based on default Python 3.11 environment extended by langchain package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "config_yml =\\\n",
        "\"\"\"\n",
        "name: python311\n",
        "channels:\n",
        "  - empty\n",
        "dependencies:\n",
        "  - pip:\n",
        "    - langchain_ibm==0.3.3\n",
        "    - langchain==0.3.7\n",
        "prefix: /opt/anaconda3/envs/python311\n",
        "\"\"\"\n",
        "\n",
        "with open(\"config.yaml\", \"w\", encoding=\"utf-8\") as f:\n",
        "    f.write(config_yml)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "space_id = 'PASTE YOUR SPACE ID HERE'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Now you need to store new package extension with APIClient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating package extensions\n",
            "SUCCESS\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai import APIClient\n",
        "\n",
        "client = APIClient(credentials)\n",
        "client.set.default_space(space_id)\n",
        "base_sw_spec_id = client.software_specifications.get_id_by_name(\"runtime-24.1-py3.11\")\n",
        "meta_prop_pkg_extn = {\n",
        "    client.package_extensions.ConfigurationMetaNames.NAME: \"langchain watsonx.ai env\",\n",
        "    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: \"Environment with langchain\",\n",
        "    client.package_extensions.ConfigurationMetaNames.TYPE: \"conda_yml\"\n",
        "}\n",
        "\n",
        "pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path=\"config.yaml\")\n",
        "pkg_extn_id = client.package_extensions.get_id(pkg_extn_details)\n",
        "pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Create new software specification and add created package extension to it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "SUCCESS\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "meta_prop_sw_spec = {\n",
        "    client.software_specifications.ConfigurationMetaNames.NAME: \"langchain watsonx.ai custom software specification\",\n",
        "    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: \"Software specification for statsmodels\",\n",
        "    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {\"guid\": base_sw_spec_id}\n",
        "}\n",
        "\n",
        "sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)\n",
        "sw_spec_id = client.software_specifications.get_id(sw_spec_details)\n",
        "client.software_specifications.add_package_extension(sw_spec_id, pkg_extn_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Store the function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "meta_props = {\n",
        "    client.repository.FunctionMetaNames.NAME: \"SequenceChain LLM function\",\n",
        "    client.repository.FunctionMetaNames.SOFTWARE_SPEC_ID: sw_spec_id\n",
        "}\n",
        "\n",
        "function_details = client.repository.store_function(meta_props=meta_props, function=chain_text_generator)\n",
        "function_id = client.repository.get_function_id(function_details)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Create online deployment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "######################################################################################\n",
            "\n",
            "Synchronous deployment creation for id: 'd71113c7-bcc7-4adf-ba23-fe8ea3c120c6' started\n",
            "\n",
            "######################################################################################\n",
            "\n",
            "\n",
            "initializing\n",
            "Note: online_url and serving_urls are deprecated and will be removed in a future release. Use inference instead.\n",
            "...........\n",
            "ready\n",
            "\n",
            "\n",
            "-----------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_id='8d7ec29e-51d5-40ae-813b-3d37102ded54'\n",
            "-----------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "metadata = {\n",
        "    client.deployments.ConfigurationMetaNames.NAME: \"Deployment of LLMs chain function\",\n",
        "    client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
        "}\n",
        "\n",
        "function_deployment = client.deployments.create(function_id, meta_props=metadata)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"scoring\"></a>\n",
        "## Scoring\n",
        "Generate text using custom inference endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'predictions': [{'fields': ['topic', 'question', 'answer'],\n",
              "   'values': ['life',\n",
              "    'What is the name of the smallest unit of life?',\n",
              "    'cell']}]}"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "deployment_id = client.deployments.get_id(function_deployment)\n",
        "client.deployments.score(deployment_id, sample_payload)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"summary\"></a>\n",
        "## Summary and next steps\n",
        "\n",
        " You successfully completed this notebook!.\n",
        " \n",
        " You learned how to use Sequential Chain using custom llm WastonxLLM.\n",
        " \n",
        "Check out our _<a href=\"https://ibm.github.io/watson-machine-learning-sdk/samples.html\" target=\"_blank\" rel=\"noopener no referrer\">Online Documentation</a>_ for more samples, tutorials, documentation, how-tos, and blog posts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Authors:\n",
        " **Lukasz Cmielowski, PhD**, Senior Technical Staff Member at watsonx.ai.\n",
        " \n",
        " **Mateusz Szewczyk**, Software Engineer at watsonx.ai."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Copyright © 2024-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "rt241_v2",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 2
}