Gemma/Translator_of_Old_Korean_Literature.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h7W3Nhq8Cpnv"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "TinKT_mJCsxf"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o3z5WeA7CvId"
      },
      "source": [
        "# Gemma - Translator of Old Korean Literature\n",
        "\n",
        "The Korean alphabet, or Hangul, has undergone changes over time, resulting in several letters no longer used in modern Korean. These obsolete letters include:\n",
        "\n",
        "1. ㆍ (Arae-a): This dot vowel represents a short 'a' sound.\n",
        "2. ㆆ (Yeorin-hieut): Pronounced as a 'light h,' akin to a softer version of the English 'h.'\n",
        "3. ㅿ (Bansiot): Represents the 'z' sound.\n",
        "4. ㆁ (Yet-ieung): A velar nasal sound comparable to 'ng' in the word 'sing.'\n",
        "\n",
        "For Korean speakers, reading older literature presents a challenge due to the utilization of now-obsolete letters. Early Hangul lacked spaces between words, further complicating readability. In contrast, modern Hangul employs spaces, consistent with most alphabetic systems.\n",
        "\n",
        "However, with the capabilities provided by Gemma, it becomes possible to create a translator that can aid in understanding and bridging the gap between contemporary and archaic Korean.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/Translator_of_Old_Korean_Literature.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNW5xe6H-pjx"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:\n",
        "\n",
        "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n",
        "2. Select **Change runtime type**.\n",
        "3. Under **Hardware accelerator**, select **T4 GPU**.\n",
        "\n",
        "\n",
        "### Gemma setup on Kaggle\n",
        "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n",
        "\n",
        "* Get access to Gemma on kaggle.com.\n",
        "* Select a Colab runtime with sufficient resources to run the Gemma 2B model.\n",
        "* Generate and configure a Kaggle username and API key.\n",
        "\n",
        "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_-wBeyQSDkCG"
      },
      "source": [
        "### Set environment variables\n",
        "\n",
        "Set environment variables for `KAGGLE_USERNAME` and `KAGGLE_KEY`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KM03mq1eDlPq"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata, drive\n",
        "\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate for your system.\n",
        "os.environ[\"KAGGLE_USERNAME\"] = userdata.get(\"KAGGLE_USERNAME\")\n",
        "os.environ[\"KAGGLE_KEY\"] = userdata.get(\"KAGGLE_KEY\")\n",
        "\n",
        "# Mounting gDrive for to store artifacts\n",
        "drive.mount(\"/content/drive\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D9iAtD0MD2Nw"
      },
      "source": [
        "### Install dependencies\n",
        "\n",
        "Install Keras and KerasNLP."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ttUGCQLS-SbC"
      },
      "outputs": [],
      "source": [
        "# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n",
        "!pip install -q -U keras-nlp datasets\n",
        "!pip install -q -U keras\n",
        "\n",
        "# Set the backbend before importing Keras\n",
        "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
        "# Avoid memory fragmentation on JAX backend.\n",
        "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n",
        "\n",
        "import keras_nlp\n",
        "import keras\n",
        "\n",
        "# Run at half precision.\n",
        "#keras.config.set_floatx(\"bfloat16\")\n",
        "\n",
        "# Training Configurations\n",
        "token_limit = 256\n",
        "lora_name = \"translator\"\n",
        "lora_rank = 4\n",
        "lr_value = 1e-4\n",
        "train_epoch = 5\n",
        "model_id = \"gemma2_instruct_2b_en\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJuLO9SF-5sY"
      },
      "source": [
        "## Load Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQZ3FuAFESRd"
      },
      "source": [
        "Here's [the dataset](https://huggingface.co/datasets/bebechien/HongGildongJeon) from Hong Gildong jeon (Korean: 홍길동전), which is a Korean novel written during the Joseon Dynasty. The [original source](https://ko.wikisource.org/wiki/%ED%99%8D%EA%B8%B8%EB%8F%99%EC%A0%84_36%EC%9E%A5_%EC%99%84%ED%8C%90%EB%B3%B8) is in public domain. You will use a [modern translation](https://ko.wikisource.org/wiki/%ED%99%8D%EA%B8%B8%EB%8F%99%EC%A0%84_36%EC%9E%A5_%EC%99%84%ED%8C%90%EB%B3%B8/%ED%98%84%EB%8C%80%EC%96%B4_%ED%95%B4%EC%84%9D) in a [creative commons license](https://creativecommons.org/licenses/by-sa/4.0/), translated by `직지프로`.\n",
        "\n",
        "To simplify the task, you will adopt the following structure for fine-tuning the model. The model will generate contemporary Korean text based on the user's input in Early Hangul.\n",
        "\n",
        "```\n",
        "<start_of_turn>user\\n\n",
        "됴션국셰둉ᄃᆡ왕즉위십오연의홍희문밧긔ᄒᆞᆫᄌᆡ상이잇스되\n",
        "<end_of_turn>\\n\n",
        "<start_of_turn>model\\n\n",
        "조선국 세종대왕 즉위 십오년에 홍회문 밖에 한 재상이 있으되,\n",
        "```\n",
        "\n",
        "> NOTE: korean text means, In the fifteenth year of the reign of King Sejong of Joseon, there was a prime minister outside Honghoemun Gate.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ATGYi1g6-7cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset({\n",
            "    features: ['original', 'modern translation'],\n",
            "    num_rows: 447\n",
            "})\n",
            "302\n",
            "<start_of_turn>user\n",
            "됴션국셰둉ᄃᆡ왕즉위십오연의홍희문밧긔ᄒᆞᆫᄌᆡ상이잇스되<end_of_turn>\n",
            "<start_of_turn>model\n",
            "조선국 세종대왕 즉위 십오년에 홍회문 밖에 한 재상이 있으되,<end_of_turn>\n",
            "<start_of_turn>user\n",
            "셩은홍이요명은문이니위인이쳥염강직ᄒᆞ여덩망이거록ᄒᆞ니당셰의영웅이라<end_of_turn>\n",
            "<start_of_turn>model\n",
            "성은 홍이요, 명은 문이니, 위인이 청렴강직하여 덕망이 거룩하니 당세의 영웅이라.<end_of_turn>\n",
            "<start_of_turn>user\n",
            "일직용문의올나벼살이할림의쳐ᄒᆞ엿더니명망이됴졍의읏듬되ᄆᆡ젼하그덕망을승이녀긔ᄉᆞ벼살을도도와이조판셔로좌으졍을ᄒᆞ이시니승상이국은을감동ᄒᆞ야갈츙보국ᄒᆞ니ᄉᆞ방의일이업고도젹이업스ᄆᆡ시화연풍ᄒᆞ여나라이ᄐᆡ평ᄒᆞ더라<end_of_turn>\n",
            "<start_of_turn>model\n",
            "일찍 용문에 올라 벼슬이 한림에 처하였더니 명망이 조정의 으뜸 되매, 전하 그 덕망을 승히 여기사 벼슬을 돋우어 이조판서로 좌의정을 하게 하시니, 승상이 국은을 감동하여 갈충보국하니 사방에 일이 업고 도적이 없으매 시화연풍하여 나라가 태평하더라.<end_of_turn>\n"
          ]
        }
      ],
      "source": [
        "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n",
        "\n",
        "from datasets import load_dataset\n",
        "\n",
        "ds = load_dataset(\n",
        "    \"bebechien/HongGildongJeon\",\n",
        "    split=\"train\",\n",
        ")\n",
        "print(ds)\n",
        "data = ds.with_format(\n",
        "    \"np\", columns=[\"original\", \"modern translation\"], output_all_columns=False\n",
        ")\n",
        "train = []\n",
        "\n",
        "for x in data:\n",
        "    item = f\"<start_of_turn>user\\n{x['original']}<end_of_turn>\\n<start_of_turn>model\\n{x['modern translation']}<end_of_turn>\"\n",
        "    length = len(tokenizer(item))\n",
        "    # skip data if the token length is longer than our limit\n",
        "    if length < token_limit:\n",
        "        train.append(item)\n",
        "\n",
        "print(len(train))\n",
        "print(train[0])\n",
        "print(train[1])\n",
        "print(train[2])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhZm69AoAFlT"
      },
      "source": [
        "## Load Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HynvFo3HAG_O"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,614,341,888\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "It looks like you've entered some Korean text.  \n",
            "\n",
            "Unfortunately, I can't understand the meaning of this text.  \n",
            "\n",
            "**Here's why:**\n",
            "\n",
            "* **Korean characters:**  The characters you entered are Korean, but without context, I can't decipher their meaning. \n",
            "* **Possible typos:** It's possible there are typos in the text. \n",
            "\n",
            "**To help me understand, please provide:**\n",
            "\n",
            "* **The context:** Where did you find this text? What was the situation?\n",
            "* **The intended meaning:** Do you know what you're trying to say? \n",
            "* **The intended language:** Are you sure it's Korean?\n",
            "\n",
            "\n",
            "Let me know, and I'll do my best to help! \n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 25.60s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "This appears to be a Korean sentence written in a very archaic style.  It's difficult to understand without more context. \n",
            "\n",
            "Here's why it's challenging and what we can try:\n",
            "\n",
            "* **Archaic Language:** The sentence uses a very old Korean dialect, which is difficult to decipher for modern speakers. \n",
            "* **Lack of Context:** Without knowing the source or the intended meaning, it's hard to interpret the sentence accurately. \n",
            "* **Possible Errors:** There might be typos or errors in the original text.\n",
            "\n",
            "**To help me understand, please provide more information:**\n",
            "\n",
            "* **Source:** Where did you find this sentence? (e.g., a book, a website, a historical document)\n",
            "* **Purpose:** What is the context of this sentence? (e.g., a poem, a historical account, a conversation)\n",
            "* **Any other clues:** Are there any other words or phrases that might help understand the\n",
            "TOTAL TIME ELAPSED: 10.61s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "I understand you're trying to express something, but the text you provided is in a language I don't recognize. \n",
            "\n",
            "It's possible that it's a mix of Korean and English, or perhaps a code or cipher. \n",
            "\n",
            "To help me understand, could you please:\n",
            "\n",
            "* **Tell me what language you're using.**\n",
            "* **Explain what you're trying to say.**\n",
            "* **If it's a code, can you give me any hints about how it works?**\n",
            "\n",
            "The more information you give me, the better I can help! \n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 6.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "I'm sorry, but I don't understand what you're asking. \n",
            "\n",
            "It seems like you're trying to communicate in a language I don't recognize.  Could you please:\n",
            "\n",
            "* **Rephrase your question in English?** \n",
            "* **Provide more context?** What are you trying to do or ask?\n",
            "* **Use a different language?** I can understand many languages.\n",
            "\n",
            "\n",
            "I'm here to help, so please let me know how I can assist you! \n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 6.12s\n"
          ]
        }
      ],
      "source": [
        "import keras\n",
        "import keras_nlp\n",
        "\n",
        "import time\n",
        "\n",
        "gemma = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n",
        "gemma.summary()\n",
        "\n",
        "tick_start = 0\n",
        "\n",
        "\n",
        "def tick():\n",
        "    global tick_start\n",
        "    tick_start = time.time()\n",
        "\n",
        "\n",
        "def tock():\n",
        "    print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n",
        "\n",
        "\n",
        "def text_gen(prompt):\n",
        "    tick()\n",
        "    input = f\"<start_of_turn>user\\n{prompt}<end_of_turn>\\n<start_of_turn>model\\n\"\n",
        "    output = gemma.generate(input, max_length=token_limit)\n",
        "    print(\"\\nGemma output:\")\n",
        "    print(output)\n",
        "    tock()\n",
        "\n",
        "\n",
        "text_gen(\"ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ\")\n",
        "text_gen(\n",
        "    \"금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니\"\n",
        ")\n",
        "text_gen(\n",
        "    \"이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ\"\n",
        ")\n",
        "text_gen(\"ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G9kTUHx1AkrI"
      },
      "source": [
        "## LoRA Fine-tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KTPbb1oFAmrT"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,617,270,528\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> (9.75 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,928,640</span> (11.17 MB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Enable LoRA for the model and set the LoRA rank (4, 8 or 16).\n",
        "gemma.backbone.enable_lora(rank=lora_rank)\n",
        "gemma.summary()\n",
        "\n",
        "# Limit the input sequence length (to control memory usage).\n",
        "gemma.preprocessor.sequence_length = token_limit\n",
        "# Use AdamW (a common optimizer for transformer models).\n",
        "optimizer = keras.optimizers.AdamW(\n",
        "    learning_rate=lr_value,\n",
        "    weight_decay=0.01,\n",
        ")\n",
        "# Exclude layernorm and bias terms from decay.\n",
        "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n",
        "\n",
        "gemma.compile(\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    optimizer=optimizer,\n",
        "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_M1J-ewJm--"
      },
      "source": [
        "## Save LoRA for each epoch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-27fD7pUJaxO"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 945ms/step - loss: 2.6158 - sparse_categorical_accuracy: 0.3294\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "다작하여 그 토성치허지거나라<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 16.91s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "금두거이품에 드러뵈니 일정자자를 나히로다하여 더니 과연 그 대부터 이태하여십삭이 차니<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.78s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "이태남사월초팔일이라날밤에오사극구름이집을두루고향내진동하며션녀한백이촉을들고드러와김사장대려니르다<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.73s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "죄송하지만, \n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.93s\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m348s\u001b[0m 1s/step - loss: 2.6141 - sparse_categorical_accuracy: 0.3296 \n",
            "Epoch 2/5\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 857ms/step - loss: 1.6853 - sparse_categorical_accuracy: 0.4656\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "대작하여 그 송치 흠터지거늘,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.53s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "금두거이 품에 들어보니 일정자자를 나흐리로다하더니, 과연 그 대부터 이태하여십삭이 차니,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.00s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "이태남사월초팔일이라,\n",
            "날밤에 오사가 깊은 집을 두루고 향내 진동하며 션녀 한방이 촉을 들고 드러와 김사당려니르다.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 4.03s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "자히 길너 탐정을 어기지 말으소서 이아희 배필은 낙양이 아니 상서 집아 제니.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.68s\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m291s\u001b[0m 895ms/step - loss: 1.6848 - sparse_categorical_accuracy: 0.4657\n",
            "Epoch 3/5\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 870ms/step - loss: 1.4655 - sparse_categorical_accuracy: 0.5155\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "대작하여 그 송치 흣터지거늘,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.51s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "금두거에 품이 드러 보니 일정자자를 나히로다하더니, 과연 그 대부터 이태하여십삭이 차니,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.00s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "이때 나산 월초팔일이 류날밤에 오성구름이 집을 두루고 향내 진동하며 션녀 한방이 촉을 들고 드러와 김성대려니르다.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.87s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "제히 길너를 징정을 어기지 말으소서, 이아희 배필은 낙양이 아니 상서 집아제니.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.81s\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m274s\u001b[0m 908ms/step - loss: 1.4651 - sparse_categorical_accuracy: 0.5156\n",
            "Epoch 4/5\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 853ms/step - loss: 1.3549 - sparse_categorical_accuracy: 0.5461\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "대작하여 그 송치 폭터지거늘,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.39s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "금두거에 품 의 드러 보니 일정하지 찰자 널로 다하더니, 과연 그 대부터 이태하여십삭이 차니,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.07s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "이때 나산 월초팔일이 류날 밤에 사적구름이 집을 두루고 향내 진동하며 선녀 한방이 촉을 들고 들어와 김성대려니르다.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "제히 길너를 정을 어기지 말으소서. 이 아희 배필은 낙양이 아니 상서 집에 자니.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.67s\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m269s\u001b[0m 890ms/step - loss: 1.3546 - sparse_categorical_accuracy: 0.5462\n",
            "Epoch 5/5\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 854ms/step - loss: 1.2749 - sparse_categorical_accuracy: 0.5708\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "대작하여 그 폭치 폭터지거늘,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.39s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "금두거피 품에 들어보니 일정하지 찰발로 다가 하더니, 과연 그 대부터 이태하여 십삭이 차니,<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.01s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ<end_of_turn>\n",
            "<start_of_turn>model\n",
            "이때는 사월 초팔일이라, 밤에 석구름이 집을 두루고 향내 진동하며 션녀 한방이 촉을 들고 들어와 김성대려니르더라.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.90s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니<end_of_turn>\n",
            "<start_of_turn>model\n",
            "제히 길너 펴 정을 어기지 말으소서. 이 아히 바필은 낙양이 아니 상서 집에 자니.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.74s\n",
            "\u001b[1m302/302\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m269s\u001b[0m 891ms/step - loss: 1.2746 - sparse_categorical_accuracy: 0.5709\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6TUlEQVR4nO3deXxU5aH/8e9Mlsk+JIRsJBAgEAgohFVAZQdRufLr/dWlXpdqrV7hFktd4PZ3sbb3Xty3yrX2tmpbtdQNbFEhAWSRRdmiQCAESEIgJCzZ92Tm/P4IjkYIJCHJmZl83q/X/EHmjPN9ejrky5nnPI/FMAxDAAAAJrGaHQAAAHRvlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKaijAAAAFNRRgAAgKl8zQ7QGk6nUwUFBQoNDZXFYjE7DgAAaAXDMFRRUaG4uDhZrS1f//CIMlJQUKCEhASzYwAAgHbIz89XfHx8i897RBkJDQ2V1DSYsLAwk9MAAIDWKC8vV0JCguv3eEs8oox889VMWFgYZQQAAA9zqSkWTGAFAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKaijAAAAFNRRgAAgKkoIwAAwFTduozszC3WHX/8QqfKa82OAgBAt9Vty4hhGPrPjw9oc/YZvbw+2+w4AAB0W922jFgsFi2aPViStPzLfOWcqTI5EQAA3VO3LSOSdFX/npqc3EuNTkPPpmWZHQcAgG6pW5cRSXp01mBZLNLHX5/U18dLzY4DAEC30+3LSEpcmOaO6C1Jemr1QZPTAADQ/XT7MiJJC2cMkr+PVVsOn9Xm7NNmxwEAoFuhjEhKiAjS7Vf1kSQ9+elBOZ2GyYkAAOg+KCPnzJ+SpBCbr/YXlGvV3pNmxwEAoNugjJzTM8Smn17bX5L0XFqW6hudJicCAKB7oIx8x71X91NkiE15Z6u1fMcxs+MAANAtUEa+I9jmqwXTkiRJL6/LVlVdo8mJAADwfpSR77l1bB/17RmkM5X1+sPmHLPjAADg9Sgj3+PnY9XDM5MlSb/fdERnK+tMTgQAgHejjFzADVfEaljvMFXVO/Tb9YfNjgMAgFejjFyA1WrRouuGSJLe/iJP+cXVJicCAMB7UUZacPXASF2dFKkGh6Hn2EQPAIBOQxm5iMeuGyxJ+uirAmUWlJucBgAA70QZuYgr4u268cpYGYb09Bo20QMAoDNQRi7h4ZnJ8rVatCHrtLYdOWt2HAAAvA5l5BISI4N129hzm+itPijDYBM9AAA6EmWkFf5tWpIC/Xz0VX6pVu8rNDsOAABehTLSClGhAbrvmn6SpGfWZKnRwSZ6AAB0FMpIK913bX9FBPvr6JkqvbvzuNlxAADwGpSRVgoN8NP8KU2b6L249pBq6h0mJwIAwDtQRtrg9qv6KD48UKcq6vT6FjbRAwCgI1BG2sDm66NfzBwkSfrdxiMqra43OREAAJ6PMtJGNw3vrcExoaqobdT/bDhidhwAADweZaSNrFaLHpvdtEz8m1tzdaK0xuREAAB4NspIO0we1Evj+kWovtGpF9MPmR0HAACPRhlpB4vl26sjH+w+rkNFFSYnAgDAc1FG2mlkn3BdNzRGTkN6enWW2XEAAPBYlJHL8PCsZFkt0toDRdqRW2x2HAAAPBJl5DIkRYXoljEJkqSnPmUTPQAA2oMycpkWTBskm69VO/NKtPbAKbPjAADgcSgjlynGHqB7rm7aRO/p1QflcHJ1BACAtqCMdIAHJg2QPdBP2acq9cFuNtEDAKAt2lRGli5dqjFjxig0NFRRUVGaO3eusrIufSfJe++9p8GDBysgIEBXXHGFPvnkk3YHdkf2QD/NmzJAkvRi+iHVNrCJHgAArdWmMrJx40bNmzdP27dvV3p6uhoaGjRz5kxVVVW1+JqtW7fqtttu07333qs9e/Zo7ty5mjt3rvbt23fZ4d3JneMTFWsPUEFZrf6yLc/sOAAAeAyLcRm3gJw+fVpRUVHauHGjrr322gsec8stt6iqqkqrVq1y/eyqq67SiBEj9Lvf/a5V71NeXi673a6ysjKFhYW1N26ne3dHvh794GvZA/206dEpsgf6mR0JAADTtPb392XNGSkrK5MkRUREtHjMtm3bNH369GY/mzVrlrZt23Y5b+2WfjCytwZGhaispkGvbWQTPQAAWqPdZcTpdOqhhx7SxIkTNWzYsBaPKywsVHR0dLOfRUdHq7CwsMXX1NXVqby8vNnDE/j6WPXIrGRJ0utbclRUXmtyIgAA3F+7y8i8efO0b98+LV++vCPzSGqaKGu3212PhISEDn+PzjIjJVqj+oartsGpF9dmmx0HAAC3164yMn/+fK1atUqfffaZ4uPjL3psTEyMioqKmv2sqKhIMTExLb5m8eLFKisrcz3y8/PbE9MUFotFi85tovfuznwdOV1pciIAANxbm8qIYRiaP3++VqxYofXr16tfv36XfM348eO1bt26Zj9LT0/X+PHjW3yNzWZTWFhYs4cnGZMYoelDouRwGnp2DZvoAQBwMW0qI/PmzdNbb72ld955R6GhoSosLFRhYaFqampcx9x5551avHix688LFizQ6tWr9dxzz+ngwYP61a9+pZ07d2r+/PkdNwo39MiswbJYpE/3FWrPsRKz4wAA4LbaVEZeffVVlZWVafLkyYqNjXU9/va3v7mOOXbsmE6ePOn684QJE/TOO+/o97//vYYPH673339fK1euvOikV2+QHBOqfx7Z9BXWU6vZRA8AgJZc1jojXcVT1hn5vhOlNZry7AbVNzr15o/HaHJylNmRAADoMl2yzggurnePQN01vq8k6anVWXKyiR4AAOehjHSyBycnKdTmqwMny/X3rwrMjgMAgNuhjHSy8GB/PTC5aRO9Z9OyVNfIJnoAAHwXZaQL/HhioqJCbTpeUqN3vjhmdhwAANwKZaQLBPn7asH0gZKk364/rIraBpMTAQDgPigjXeTm0QnqHxms4qp6/e/mHLPjAADgNigjXcTPx6qHz22i94fNR3W6os7kRAAAuAfKSBeaPSxGwxN6qLreod+uZxM9AAAkykiXslgseuy6pqsj73xxTHlnq0xOBACA+SgjXWzCgEhNGtRLjU5Dz6YdMjsOAACmo4yY4NFzV0f+8VWB9p0oMzkNAADmooyYYGicXTeNiJPUtIkeAADdGWXEJL+YkSw/H4s2Z5/R59lnzI4DAIBpKCMm6dMzSLeP+2YTvYNsogcA6LYoIyaaPzVJwf4+2nuiTJ/sO2l2HAAATEEZMVFkiE33XdtfkvTsmiw1OJwmJwIAoOtRRkz2k2v6KzLEX7lnq7V8R77ZcQAA6HKUEZOF2Hz1b1ObNtF7aW22quoaTU4EAEDXooy4gdvG9lGfiCCdqazT65+ziR4AoHuhjLgBf1+rfjFzkCTptU1HVVxVb3IiAAC6DmXETcy5Mk5D48JUWdeoV9YfNjsOAABdhjLiJqxWix67brAk6a3tecovrjY5EQAAXYMy4kauGRipCQN6qt7h1AvpbKIHAOgeKCNuxGL59urIiowTOnCy3OREAAB0PsqImxme0EM3XBErw5CeZhM9AEA3QBlxQw/PSpaP1aLPsk5r+9GzZscBAKBTUUbcUL/IYN06JkGS9OSnB2UYbKIHAPBelBE3tWDaQAX6+Sgjv1Rr9heZHQcAgE5DGXFTUWEBuvfqfpKkZ9YcVCOb6AEAvBRlxI39dFJ/hQf56cjpKr2/67jZcQAA6BSUETcWFuCneVOSJEkvrs1WTb3D5EQAAHQ8yoib+5er+qp3j0AVltfqza25ZscBAKDDUUbcXICfjxbOaNpE79UNh1VazSZ6AADvQhnxAHNTeys5OlTltY16dcMRs+MAANChKCMewMdq0WOzkyVJb2zNVUFpjcmJAADoOJQRDzElOUpjEyNU3+jUi2vZRA8A4D0oIx7CYrHosdlNm+i9v+u4sosqTE4EAEDHoIx4kFF9wzUzJVpOQ3pmTZbZcQAA6BCUEQ/z6HXJslqktMwi7corNjsOAACXjTLiYZKiQvXDUU2b6D31aRab6AEAPB5lxAM9NGOgbL5WfZlbrPUHT5kdBwCAy0IZ8UCx9kDdPTFRkvT06iw5nFwdAQB4LsqIh3pwUpLCAnyVVVShFXtOmB0HAIB2o4x4KHuQnx48t4neC+mHVNvAJnoAAM9EGfFgd09IVExYgE6U1uit7XlmxwEAoF0oIx4swM9HP58xUJL0ymeHVV7bYHIiAADajjLi4f55ZLwG9ApWaXWDXtvIJnoAAM9DGfFwvj5WPXpd0zLxf/w8R6fKa01OBABA21BGvMDMlGiN7NNDtQ1OvbQu2+w4AAC0CWXEC1gsFj127urI8h35Onq60uREAAC0HmXES4zr31NTB0fJ4TT0XNohs+MAANBqlBEv8uh1ybJYpI/3ntRX+aVmxwEAoFUoI15kcEyY/k9qb0nSk58eZBM9AIBHoIx4mYUzBsnfx6ptR89qU/YZs+MAAHBJlBEvEx8epDvG95UkPfXpQTnZRA8A4OYoI15o3pQkhdp8lXmyXP/4usDsOAAAXBRlxAtFBPvr/kn9JUnPpR1SfaPT5EQAALSMMuKl7rm6n3qF2nSsuFp//fKY2XEAAGgRZcRLBfn7asG0pk30Xl6Xrcq6RpMTAQBwYZQRL3bLmAT1iwzW2ap6/WHzUbPjAABwQZQRL+bnY9UvZg6SJP3vpqM6U1lnciIAAM5HGfFy1w+L1ZXxdlXVO/TK+sNmxwEA4DxtLiObNm3SnDlzFBcXJ4vFopUrV17yNW+//baGDx+uoKAgxcbG6p577tHZs2fbkxdtZLV+u4ne21/k6djZapMTAQDQXJvLSFVVlYYPH65ly5a16vgtW7bozjvv1L333qv9+/frvffe05dffqn77ruvzWHRPhOTInXNwEg1OAw9l55ldhwAAJrxbesLZs+erdmzZ7f6+G3btikxMVE/+9nPJEn9+vXT/fffr6eeeqqtb43L8Nh1g7U5+3N9lFGg+67pr2G97WZHAgBAUhfMGRk/frzy8/P1ySefyDAMFRUV6f3339f111/f4mvq6upUXl7e7IHLM6y3Xf80PE6S9PQaro4AANxHp5eRiRMn6u2339Ytt9wif39/xcTEyG63X/RrnqVLl8put7seCQkJnR2zW/jFzEHytVq06dBpbT3MJnoAAPfQ6WUkMzNTCxYs0JIlS7Rr1y6tXr1aubm5euCBB1p8zeLFi1VWVuZ65Ofnd3bMbqFvz2DdPq6PJOmp1QdlGGyiBwAwX5vnjLTV0qVLNXHiRD3yyCOSpCuvvFLBwcG65ppr9J//+Z+KjY097zU2m002m62zo3VL86cO1Hu7juur42X6dF+hrr/i/P/9AQDoSp1+ZaS6ulpWa/O38fHxkST+ZW6CXqE2/eSapk30nlmTpQYHm+gBAMzV5jJSWVmpjIwMZWRkSJJycnKUkZGhY8eaNmNbvHix7rzzTtfxc+bM0YcffqhXX31VR48e1ZYtW/Szn/1MY8eOVVxcXMeMAm1y3zX91DPYXzlnqvTuTr4CAwCYq81lZOfOnUpNTVVqaqokaeHChUpNTdWSJUskSSdPnnQVE0m6++679fzzz+uVV17RsGHD9MMf/lDJycn68MMPO2gIaKvQAD/Nn5okSXpxbbaq69lEDwBgHovhAd+VlJeXy263q6ysTGFhYWbH8Qp1jQ5Nf36j8otr9MisZM2bkmR2JACAl2nt72/2pummbL4++sWMZEnS7zYcUUlVvcmJAADdFWWkG/un4XEaEhumirpGLfuMTfQAAOagjHRjTZvoNV0d+fO2PB0vYRM9AEDXo4x0c5MG9dL4/j1V73DqhfRss+MAALohykg3Z7FY9NjswZKkD/cc18FC9gECAHQtygg0IqGHrr8iRoYhPbOaTfQAAF2LMgJJ0sMzk+VjtWjdwVP6MqfY7DgAgG6EMgJJUv9eIbp5dNPuyE9+eoCl+gEAXYYyApeHpg9UgJ9Vu4+VKj2zyOw4AIBugjICl+iwAN0zsZ8k6ek1WWpkEz0AQBegjKCZ+ycNUI8gPx0+VakPd58wOw4AoBugjKAZe6Cf5k1u2qfm+fRDqm1wmJwIAODtKCM4zx3j+yrOHqDC8lr9aWuu2XEAAF6OMoLzBPj56OczBkmS/mfDEZVVN5icCADgzSgjuKAfjIzXoOgQldU06NWNR8yOAwDwYpQRXJCP1aJHZzUtE//GlhwVltWanAgA4K0oI2jRtCFRGt03XHWNTr249pDZcQAAXooyghZZLBYtOreJ3rs783X4VKXJiQAA3ogygosanRih6UOi5TSkZ9YcNDsOAMALUUZwSY9elyyrRVqzv0i7j5WYHQcA4GUoI7ikQdGh+ueR8ZKkJz89yCZ6AIAORRlBq/x8xiD5+1r1ZU6xNmSdNjsOAMCLUEbQKnE9AnX3hERJ0lOrD8rh5OoIAKBjUEbQag9OHqDQAF8dLKzQRxlsogcA6BiUEbRajyB//evkAZKk59IOqa6RTfQAAJePMoI2+fGEfooOs+lEaY3e2n7M7DgAAC9AGUGbBPr76KHpTZvovbI+W+W1bKIHALg8lBG02Q9Hxat/r2CVVDfofzcdNTsOAMDDUUbQZr4+Vj06K1mS9IfNOTpVwSZ6AID2o4ygXWYNjdGIhB6qaXDo5XXZZscBAHgwygja5bub6C3/Ml85Z6pMTgQA8FSUEbTbVf17anJyLzU6DT2blmV2HACAh6KM4LI8OmuwLBbp469Pau/xMrPjAAA8EGUElyUlLkxzR/SW1LRMPAAAbUUZwWVbOGOQ/H2s+vzwGW3OZhM9AEDbUEZw2RIignT7VX0kNV0dcbKJHgCgDSgj6BDzpyQpxOarfSfKtWrvSbPjAAA8CGUEHaJniE0/vba/JOm5tCzVNzpNTgQA8BSUEXSYe6/up8gQm/LOVmv5DjbRAwC0DmUEHSbY5qsF05IkSS+vy1ZVXaPJiQAAnoAygg5169g+6tszSGcq6/WHzTlmxwEAeADKCDqUn49VD89s2kTv95uO6GxlncmJAADujjKCDnfDFbEa1jtMVfUOvfLZYbPjAADcHGUEHc5qtWjRdUMkSW9tz1N+cbXJiQAA7owygk5x9cBIXZ0UqQaHoefTD5kdBwDgxigj6DSPXTdYkrQy44QyC8pNTgMAcFeUEXSaK+LtuvHKWBmG9PQaNtEDAFwYZQSd6uGZyfK1WrQh67S2HTlrdhwAgBuijKBTJUYG67axTZvoPbn6oAyDTfQAAM1RRtDp/m1akgL9fPRVfqlW7ys0Ow4AwM1QRtDpokIDdN81/SRJz6zJUqODTfQAAN+ijKBL3Hdtf0UE++vomSq9u/O42XEAAG6EMoIuERrgp/lTmjbRe3HtIdXUO0xOBABwF5QRdJnbr+qj+PBAnaqo0xtb2UQPANCEMoIuY/P10cIZgyRJr244otLqepMTAQDcAWUEXeqmEb01OCZUFbWN+p8NR8yOAwBwA5QRdCkfq8W1TPybW3N1orTG5EQAALNRRtDlJif30rh+EapvdOpFNtEDgG6PMoIuZ7FY9NjspqsjH+w+rkNFFSYnAgCYiTICU4zsE67rhsbIaUhPr84yOw4AwESUEZjm4VnJslqktQeKtCO32Ow4AACTUEZgmqSoEN0yJkGS9NSnbKIHAN1Vm8vIpk2bNGfOHMXFxclisWjlypWXfE1dXZ1++ctfqm/fvrLZbEpMTNTrr7/enrzwMgumDZLN16qdeSVad+CU2XEAACZocxmpqqrS8OHDtWzZsla/5uabb9a6dev0xz/+UVlZWfrrX/+q5OTktr41vFCMPUD3XN20id7Taw7K4eTqCAB0N75tfcHs2bM1e/bsVh+/evVqbdy4UUePHlVERIQkKTExsa1vCy/2wKQBeueLYzpUVKkPdx/XD0cnmB0JANCFOn3OyN///neNHj1aTz/9tHr37q1Bgwbp4YcfVk1Ny4td1dXVqby8vNkD3sse6KcHJw+QJL2Qfki1DWyiBwDdSaeXkaNHj+rzzz/Xvn37tGLFCr344ot6//339eCDD7b4mqVLl8put7seCQn8S9nb3TUhUbH2ABWU1eov2/LMjgMA6EKdXkacTqcsFovefvttjR07Vtdff72ef/55/elPf2rx6sjixYtVVlbmeuTn53d2TJgswM9HP5/etIneK58dVllNg8mJAABdpdPLSGxsrHr37i273e762ZAhQ2QYho4fP37B19hsNoWFhTV7wPv9YGRvDYwKUVlNg17byCZ6ANBddHoZmThxogoKClRZWen62aFDh2S1WhUfH9/Zbw8P4utj1SOzmu6yen1LjorKa01OBADoCm0uI5WVlcrIyFBGRoYkKScnRxkZGTp27Jikpq9Y7rzzTtfxP/rRj9SzZ0/9+Mc/VmZmpjZt2qRHHnlE99xzjwIDAztmFPAaM1KiNapvuGobnHpxbbbZcQAAXaDNZWTnzp1KTU1VamqqJGnhwoVKTU3VkiVLJEknT550FRNJCgkJUXp6ukpLSzV69GjdfvvtmjNnjl5++eUOGgK8icVi0aJzm+i9uzNfR05XXuIVAABPZzE8YA3u8vJy2e12lZWVMX+km/jJn3Zo7YFTmj0sRq/+yyiz4wAA2qG1v7/ZmwZu6ZFZg2WxSJ/uK9SeYyVmxwEAdCLKCNxSckyofpDaNMH5qdVsogcA3owyAre1cOYg+ftatf1osTYeOm12HABAJ6GMwG317hGoO6/qK0l6anWWnGyiBwBeiTICtzZvSpJCbb46cLJcf/+qwOw4AIBOQBmBWwsP9tcD5zbRezYtS3WNbKIHAN6GMgK39+OJiYoKtel4SY3e+eLYpV8AAPAolBG4vSB/Xy2YPlCS9Nv1h1VRyyZ6AOBNKCPwCDePTlD/yGAVV9XrfzfnmB0HANCBKCPwCH4+Vj18bhO9P2w+qtMVdSYnAgB0FMoIPMbsYTEaHm9Xdb1Dv13PJnoA4C0oI/AYFotFj53bRO+dL44p72yVyYkAAB2BMgKPMmFApK4d1EuNTkPPph0yOw4AoANQRuBxHruuae7IP74q0L4TZSanAQBcLsoIPM7QOLtuGhEnqWkTPQCAZ6OMwCP9Ykay/Hws2px9Rp9nnzE7DgDgMlBG4JH69AzS7eOaNtFb+ukBldWwEBoAeCrKCDzW/KlJCvb30f6Cck19doPe/iJPDnb2BQCPQxmBx4oMsen1u8eof69gna2q1y9X7NMNL2/W1iN8bQMAnsRiGIbb/1OyvLxcdrtdZWVlCgsLMzsO3EyDw6m/bMvTi2sPqby2UZI0a2i0fnl9ivr0DDI5HQB0X639/U0ZgdcorqrXC+mH9PYXeXIakr+PVfdc3U/zpyYpxOZrdjwA6HYoI+i2sgor9JtVmfr8cNPXNZEhNj06K1n/d1S8rFaLyekAoPugjKBbMwxDaw+c0n99nKncs9WSpGG9w7TkxqEa2y/C5HQA0D1QRgBJdY0O/Xlrnl5el62Kuqb5JDdcGavFswcrPpz5JADQmSgjwHecqazTc2mHtHzHMRmGZPO16qfX9tcDkwYomPkkANApKCPABWQWlOvXq/Zr+9FiSVJ0mE2PXTdYc0f0Zj4JAHQwygjQAsMwtGZ/of7rkwPKL66RJA1P6KHH56RoZJ9wk9MBgPegjACXUNvg0OtbcrRs/WFV1TskSTeNiNOi2YMVaw80OR0AeD7KCNBKp8pr9cyaLL2/+7gMQwrws+qBSQN0/7UDFOjvY3Y8APBYlBGgjfYeL9MT/9ivnXklkqQ4e4Aemz1Y/zQ8ThYL80kAoK0oI0A7GIahVV+f1JOfHtSJ0qb5JKP6hmvJjSkantDD3HAA4GEoI8BlqG1w6PebjurVDUdU09A0n+SfR8br0euSFR0WYHI6APAMlBGgAxSW1erp1Qf14Z4TkqQgfx/Nm5Kke6/upwA/5pMAwMVQRoAOtOdYiX69KlN7jpVKkuLDA/Xv1w/R7GExzCcBgBZQRoAO5nQa+vtXBXry04MqLK+VJI3tF6ElN6ZoWG+7yekAwP1QRoBOUl3fqN9tPKrXNh5RXaNTFot0y+gE/WJmsnqF2syOBwBugzICdLITpTV68tOD+sdXBZKkEJuv/m1qku6emCibL/NJAIAyAnSRnbnFeuIfmdp7okyS1LdnkP79+iGamRLNfBIA3RplBOhCTqehD3Yf19NrsnS6ok6SNDGpp/7jxhQNjuH/swC6J8oIYILKukb9z2eH9YfPc1Tf6JTVIt02to8WzhikniHMJwHQvVBGABPlF1dr6acH9MneQklSaICvFkwbqDvHJ8rf12pyOgDoGpQRwA1sP3pWv/5HpjJPlkuS+kcG65c3DNHUwVHMJwHg9SgjgJtwOA29tzNfz6Zl6UxlvSTpmoGRWnJjigZGh5qcDgA6D2UEcDMVtQ16Zf1hvb4lRw0OQz5Wi/5lXB/9fMYg9QjyNzseAHQ4ygjgpnLPVOm/Pjmg9MwiSZI90E8/nz5Qt1/VV34+zCcB4D0oI4Cb23L4jH79j0xlFVVIkpKiQvQfN6Zo0qBeJicDgI5BGQE8QKPDqeU78vVcWpZKqhskSVMHR+mXNwzRgF4hJqcDgMtDGQE8SFlNg15el60/bc1Vo9OQr9WiuyYk6mfTBsoe6Gd2PABoF8oI4IGOnK7Uf318QOsPnpIkRQT7a+GMQbp1TIJ8mU8CwMNQRgAPtvHQaf1mVaYOn6qUJA2OCdWSG1M0ISnS5GQA0HqUEcDDNTicent7nl5Ym62ymqb5JDNTovXv1w9RYmSwyekA4NIoI4CXKKmq14trD+mtL47J4TTk72PVjycmav7UJIUGMJ8EgPuijABe5lBRhX6zKlObs89IkiJD/PXwzGT9cHSCfKwsLQ/A/VBGAC9kGIbWHzyl//z4gHLOVEmShsaFacmNKRrXv6fJ6QCgOcoI4MXqG53687ZcvbQuWxW1jZKk66+I0eLZQ5QQEWRyOgBoQhkBuoGzlXV6Lv2Qln95TE5D8ve16r5r+unByUkKtvmaHQ9AN0cZAbqRAyfL9ZtVmdp65KwkKSrUpkevG6wfpPaWlfkkAExCGQG6GcMwlJZZpP/+5IDyzlZLkobH27VkTopG9Y0wOR2A7ogyAnRTdY0OvbElV6+sP6zKuqb5JP80PE6LZg9WXI9Ak9MB6E4oI0A3d7qiTs+uydK7u/JlGFKAn1X3XztAD0waoEB/H7PjAegGKCMAJEn7TpTp16sy9WVOsSQp1h6gx64brJtGxMliYT4JgM7T2t/fbd55a9OmTZozZ47i4pr+Ilu5cmWrX7tlyxb5+vpqxIgRbX1bAO00rLddf/vpVfqf20cqPjxQJ8tq9dDfMvSDV7cqI7/U7HgA0PYyUlVVpeHDh2vZsmVtel1paanuvPNOTZs2ra1vCeAyWSwWXX9FrNYunKRHZiUryN9He46Vau6yLVr4twwVltWaHRFAN3ZZX9NYLBatWLFCc+fOveSxt956qwYOHCgfHx+tXLlSGRkZrX4fvqYBOlZRea2eXp2lD3YflyQF+vnowckDdN+1/RXgx3wSAB2j076maY833nhDR48e1eOPP96q4+vq6lReXt7sAaDjRIcF6Lmbh+ujeRM1qm+4ahocei79kKY9t1Grvi6QB0wlA+BFOr2MZGdna9GiRXrrrbfk69u6FSGXLl0qu93ueiQkJHRySqB7Gp7QQ+8/MF4v3TpCsfYAnSit0fx39uiW17Zr34kys+MB6CY6tYw4HA796Ec/0hNPPKFBgwa1+nWLFy9WWVmZ65Gfn9+JKYHuzWKx6KYRvbX+F5P10PSBCvCz6svcYs155XM98t5XOlXBfBIAnatT54yUlpYqPDxcPj7ffgftdDplGIZ8fHyUlpamqVOnXvJ9mDMCdJ2C0ho9tfqgPsookCQF+/to3tQk3TOxH/NJALSJW8wZCQsL0969e5WRkeF6PPDAA0pOTlZGRobGjRvXmW8PoB3iegTqpVtT9cG/TtDweLuq6h16enWWZrywUav3nWQ+CYAO1+ZtPSsrK3X48GHXn3NycpSRkaGIiAj16dNHixcv1okTJ/TnP/9ZVqtVw4YNa/b6qKgoBQQEnPdzAO5lVN9wrXhwolbsOaGnVh9UfnGNHnhrt67qH6ElNw5VShxXKQF0jDZfGdm5c6dSU1OVmpoqSVq4cKFSU1O1ZMkSSdLJkyd17Nixjk0JwBRWq0X/PCpenz08Wf82NUk2X6u2Hy3Wjb/drMUf7tWZyjqzIwLwAiwHD6DVjpdUa+mnB/Xx1yclSaE2X/1s2kDdNSFR/r5dslIAAA/C3jQAOs2XOcX69ar92neiaQ2gfpHB+uX1QzRtSBT73QBwoYwA6FQOp6EPdh3X02uyXF/XXDMwUv9xY4oGRYeanA6AO6CMAOgSlXWNWvbZYf1xc47qHU75WC26fVwf/Xz6IIUH+5sdD4CJKCMAutSxs9X6708OaPX+QklSWICvHpo+SHeM7ys/H+aTAN0RZQSAKbYeOaNf/yNTBwsrJEkDegXr/92YoinJUSYnA9DVKCMATONwGvrbjnw9l5als1X1kqTJyb30/25IUVJUiMnpAHQVyggA05XXNui367L15tZcNTgM+VotumN8Xz00bZDsQX5mxwPQySgjANxGzpkq/dfHmVp74JQkKTzITwtnDNJtY/vIl/kkgNeijABwO5uzT+s3qzJ1qKhSkpQcHar/uDFFVw+MNDkZgM5AGQHglhodTr3z5TE9n35IpdUNkqQxieGaNTRGM1Ni1KdnkMkJAXQUyggAt1ZaXa8X12brL9vz5HB++9fQ4JhQzUiJ1syUGA3rHcaKroAHo4wA8AgFpTVas79QafuL9GVucbNiEmsP0IyUaM1Iida4fj3Z/wbwMJQRAB6ntLpe6w+eUnpmkTYeOq3qeofrudAAX01JjtLModGaNKiXQgO4Gwdwd5QRAB6ttsGhrUfOKG1/kdYeKNKZynrXc34+Fk0YEOm6ahIdFmBiUgAtoYwA8BoOp6GM/BKlZRYpfX+Rjp6pavb88IQempkSrZkp0UqKCmGeCeAmKCMAvNbhU5VKyyxUemaR9hwrbfZcv8jgcxNgo5XaJ1w+VooJYBbKCIBu4VR5rdYeOKW0zEJtPXxW9Q6n67mewf6aPqTpq5yrB0YqwM/HxKRA90MZAdDtVNY1amPWaaVnFmrdwVOqqG10PRfo56NrB0VqZkqMpg6OUniwv4lJge6BMgKgW2twOPVlTrHS9jd9nVNQVut6zsdq0ZjEcM1IidHMlGglRLDQGtAZKCMAcI5hGNpfUK60zCKl7S/UwcKKZs8PjgnVzKFNxWRoHAutAR2FMgIALcgvrm66MyezUF/mFOs766wp7txCazOHxmhsvwj5sZEf0G6UEQBohZKqpoXW0jILtenQGdU0fLvQWliAr6YOjtKMlBhNSu6lEJuviUkBz0MZAYA2qm1w6PPsM0rPbFpo7WzVtwut+ftYNSGpp2amxGj6kChFsdAacEmUEQC4DA6noT3HSlzzTHLPVjd7PrVPD9eGfklRISalBNwbZQQAOohhGOcWWitSWmaRvsovbfZ8/8hgzRh6bqG1hHBZWWgNkEQZAYBOU1Req/TMIqVnFmnrkTNqcHz712hkiE3ThzRt6DdhAAutoXujjABAF6iobdDGQ6eVtr9In2U1X2gtyN9Hkwb10oyUaE0dHKUeQSy0hu6FMgIAXay+0akvcs4qbX/TVZPC8uYLrY1NjNDMoU3L08eHs9AavB9lBABMZBiG9p4oc32d8/2F1lJiw1zFJCWWhdbgnSgjAOBGjp2tVlpmodIyi7Qzt/lCa717BLp2Gh7DQmvwIpQRAHBTxVX1Wneg6c6czdmnVdvw7U7D9kA/TR0cpZkp0bp2UC8Fs9AaPBhlBAA8QE29Q5uzTys9s0jrDp5S8XcXWvO16uqkSM1Mida0IdHqFWozMSnQdpQRAPAwDqehXXklSj/3dU7edxZas1ikkX3CXV/n9O/FQmtwf5QRAPBghmHoUFGlq5h8fbys2fMDegVr5tAYzUiJ1oj4Hiy0BrdEGQEAL3KyrEZrz60Au+3IWTV+ZwZsr1Cbpg+JPrfQWk/ZfFloDe6BMgIAXqq8tkEbsk4rbX+hNmSdVmXdtwutBfv7aHJylGakRGtKcpTsQX4mJkV3RxkBgG6grtGh7UeLlZ5ZqPTMIhWV17me87VaNK5/RNNOwynR6t0j0MSk6I4oIwDQzTidTQutpZ0rJoeKKps9P6x3mGYMidHModEaHBPKQmvodJQRAOjmcs9UKT2zSGmZhdqZV6Lv/m0fHx6omSlNE2DHJIbLl4XW0AkoIwAAlzOVdVp/4JRrobW6xm8XWusR9M1CazG6dlCkgvxZaA0dgzICALig6vpGbc4+o7T9RVp/sEgl1Q2u52y+Vl0zMFIzzi20FhnCQmtoP8oIAOCSGh1O7cwrcX2dk19c43rOYpFG9Qk/t6FfjPpFBpuYFJ6IMgIAaBPDMJRVVKG0/U07De890XyhtYFRIU0rwA6N0ZW97Sy0hkuijAAALktBaY3WHihS2v4ibT/afKG1qFCbZqREa0ZKtMaz0BpaQBkBAHSYspoGbchqmgC74eApVdU7XM+F2Hw1OjFcYxIjNLpvuIYn9FCAH+UElBEAQCepa3Ro25GzSsss0trMIp2qqGv2vJ+PRVf0tmtMYoRG9Q3X6MQIRQT7m5QWZqKMAAA6ndNpKPNkuXbkFmtnbom+zC3W6e+VE6lpY78xiREafe7qSd+eQSy61g1QRgAAXc4wDOUX1zSVk7ymgpJ9qvK84yJDbBqT2HTVZExiuIbEhsmPhde8DmUEAOAWSqrqtSuvRDvOlZO9x8tU73A2OybQz0epfXq4yklqn3CF2Fh8zdNRRgAAbqm2waG9J8pcX+3syitRWU1Ds2OsFmlIbNi5r3bCNbpvhGLsASYlRntRRgAAHsHpNHT4dKWrnOzILdbxkprzjosPD3SVkzGJEUrqFcJaJ26OMgIA8FiFZbWuOSc7cot14GS5nN/7bWUP9Dt3t05TObmit51bit0MZQQA4DUqahuUkV+qHbkl2plbrD3HSlXT4Gh2jL+PVVfE25vKSd+m24rDuaXYVJQRAIDXanA4lVlQrp15TeVkR26JzlSef0vxwKgQ15yTMYkRSogI5JbiLkQZAQB0G4ZhKO9stXbkFjfduZNbrCOnq847LirU5lqMbUxihIbEhsqXW4o7DWUEANCtna2s0668EtfVk70nytTgaP4rL8jfRyP7hLuunqT26aFgbinuMJQRAAC+o7bBoa/yS7Xz3JWTXXklqqhtbHaMj9WilNiwZnvtRIVxS3F7UUYAALgIp9PQoVMVrkmxO3NLdKL0/FuK+0QEucrJmMRw9Y/kluLWoowAANBGBaU1zSbFHiws1/d/S/YI8tPovt8uZT+st102X24pvhDKCAAAl6m8tkG780pck2Iz8ktV29B8KXt/X6uGx9td5WRUnwjZg/xMSuxeKCMAAHSw+kan9heUaWduiWtRtrNV9ecdNyg6xFVORveNUHx497ylmDICAEAnMwxDOWeqXCvF7sor0dEz599SHBMWoFGJ4Rpz7uudIbFh8ukG8046rYxs2rRJzzzzjHbt2qWTJ09qxYoVmjt3bovHf/jhh3r11VeVkZGhuro6DR06VL/61a80a9asDh8MAABmO1NZ13TlJLdYO/NKtO9EmRq/t5Z9iM23aZfivk1XT0b06aEgf++7pbi1v7/bPPKqqioNHz5c99xzj37wgx9c8vhNmzZpxowZ+u///m/16NFDb7zxhubMmaMvvvhCqampbX17AADcWmSITdcNi9F1w2IkSTX1DmXklzZNis0r0Z68ElXUNWpz9hltzj4jqemW4mFxYRp97nbiUYnhigrtPrcUX9bXNBaL5ZJXRi5k6NChuuWWW7RkyZJWHc+VEQCAt3A4DWUVVmhnXrHrtuKTZbXnHZfYM+jbSbF9IzSgV7DHzTvptCsjl8vpdKqiokIREREtHlNXV6e6um/3GCgvL++KaAAAdDofq0UpcWFKiQvTneMTJUknSmvO3U7cNCk2q6hCuWerlXu2Wu/vOi5Jigj2P7eMfdO8k2Fxdvn7esdS9l1eRp599llVVlbq5ptvbvGYpUuX6oknnujCVAAAmKd3j0D1HtFbN43oLUkqq2nQ7mPfrneSkV+q4qp6pWcWKT2zSJJk87VqeEIPVzkZ2Sdc9kDPvKW4S7+meeedd3Tffffpo48+0vTp01s87kJXRhISEviaBgDQLdU1OrTvRLlrUuzO3GKVVDc0O8ZikZKjQ79dyj4xQr17BJqUuInbfU2zfPly/eQnP9F777130SIiSTabTTabrYuSAQDg3my+PhrVN1yj+obrfjXdUnzkdJXrysmuvGLlnq3WwcIKHSys0Fvbj0mS4uwBGvWd9U6SY0Ld8pbiLikjf/3rX3XPPfdo+fLluuGGG7riLQEA8FoWi0VJUSFKigrRrWP7SJJOVdRqV25J06TYvGLtLyhXQVmtCr4q0D++KpAkhdp8NbJvuGs5+xEJPRTob/5S9m0uI5WVlTp8+LDrzzk5OcrIyFBERIT69OmjxYsX68SJE/rzn/8sqemrmbvuuksvvfSSxo0bp8LCQklSYGCg7HZ7Bw0DAIDuLSo0QLOviNXsK2IlSVV1jfoqv9RVTnafu6V446HT2njotCTJ12rRsN52je4brrmpvTWstzm/l9s8Z2TDhg2aMmXKeT+/66679Oabb+ruu+9Wbm6uNmzYIEmaPHmyNm7c2OLxrcGtvQAAXJ5Gh1MHCytc653szC1WUfm38zOfv3m4fjAyvkPfk+XgAQBAiwzD0PGSGtd6Jw9OHqD48KAOfQ+3m8AKAADch8ViUUJEkBIigvR/Ujv2ikhbecdqKQAAwGNRRgAAgKkoIwAAwFSUEQAAYCrKCAAAMBVlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqSgjAADAVJQRAABgKsoIAAAwlUfs2msYhqSmrYgBAIBn+Ob39je/x1viEWWkoqJCkpSQkGByEgAA0FYVFRWy2+0tPm8xLlVX3IDT6VRBQYFCQ0NlsVg67L9bXl6uhIQE5efnKywsrMP+u+7E28fI+Dyft4/R28cnef8YGV/7GYahiooKxcXFyWpteWaIR1wZsVqtio+P77T/flhYmFf+H+y7vH2MjM/zefsYvX18kvePkfG1z8WuiHyDCawAAMBUlBEAAGCqbl1GbDabHn/8cdlsNrOjdBpvHyPj83zePkZvH5/k/WNkfJ3PIyawAgAA79Wtr4wAAADzUUYAAICpKCMAAMBUlBEAAGAqry8jy5YtU2JiogICAjRu3Dh9+eWXFz3+vffe0+DBgxUQEKArrrhCn3zySRclbb+2jPHNN9+UxWJp9ggICOjCtG2zadMmzZkzR3FxcbJYLFq5cuUlX7NhwwaNHDlSNptNSUlJevPNNzs9Z3u1dXwbNmw47/xZLBYVFhZ2TeA2Wrp0qcaMGaPQ0FBFRUVp7ty5ysrKuuTrPOVz2J7xedpn8NVXX9WVV17pWhBr/Pjx+vTTTy/6Gk85f1Lbx+dp5+/7nnzySVksFj300EMXPa6rz6FXl5G//e1vWrhwoR5//HHt3r1bw4cP16xZs3Tq1KkLHr9161bddtttuvfee7Vnzx7NnTtXc+fO1b59+7o4eeu1dYxS0yp7J0+edD3y8vK6MHHbVFVVafjw4Vq2bFmrjs/JydENN9ygKVOmKCMjQw899JB+8pOfaM2aNZ2ctH3aOr5vZGVlNTuHUVFRnZTw8mzcuFHz5s3T9u3blZ6eroaGBs2cOVNVVVUtvsaTPoftGZ/kWZ/B+Ph4Pfnkk9q1a5d27typqVOn6qabbtL+/fsveLwnnT+p7eOTPOv8fdeOHTv02muv6corr7zocaacQ8OLjR071pg3b57rzw6Hw4iLizOWLl16weNvvvlm44Ybbmj2s3Hjxhn3339/p+a8HG0d4xtvvGHY7fYuStexJBkrVqy46DGPPvqoMXTo0GY/u+WWW4xZs2Z1YrKO0ZrxffbZZ4Yko6SkpEsydbRTp04ZkoyNGze2eIwnfg6/0ZrxefJn8Bvh4eHGH/7whws+58nn7xsXG5+nnr+Kigpj4MCBRnp6ujFp0iRjwYIFLR5rxjn02isj9fX12rVrl6ZPn+76mdVq1fTp07Vt27YLvmbbtm3NjpekWbNmtXi82dozRkmqrKxU3759lZCQcMl/AXgaTzuH7TVixAjFxsZqxowZ2rJli9lxWq2srEySFBER0eIxnnwOWzM+yXM/gw6HQ8uXL1dVVZXGjx9/wWM8+fy1ZnySZ56/efPm6YYbbjjv3FyIGefQa8vImTNn5HA4FB0d3ezn0dHRLX6/XlhY2KbjzdaeMSYnJ+v111/XRx99pLfeektOp1MTJkzQ8ePHuyJyp2vpHJaXl6umpsakVB0nNjZWv/vd7/TBBx/ogw8+UEJCgiZPnqzdu3ebHe2SnE6nHnroIU2cOFHDhg1r8ThP+xx+o7Xj88TP4N69exUSEiKbzaYHHnhAK1asUEpKygWP9cTz15bxeeL5W758uXbv3q2lS5e26ngzzqFH7NqLjjN+/PhmjX/ChAkaMmSIXnvtNf3mN78xMRlaIzk5WcnJya4/T5gwQUeOHNELL7ygv/zlLyYmu7R58+Zp3759+vzzz82O0ilaOz5P/AwmJycrIyNDZWVlev/993XXXXdp48aNLf7C9jRtGZ+nnb/8/HwtWLBA6enpbj3R1mvLSGRkpHx8fFRUVNTs50VFRYqJibnga2JiYtp0vNnaM8bv8/PzU2pqqg4fPtwZEbtcS+cwLCxMgYGBJqXqXGPHjnX7X/Dz58/XqlWrtGnTJsXHx1/0WE/7HEptG9/3ecJn0N/fX0lJSZKkUaNGaceOHXrppZf02muvnXesJ56/tozv+9z9/O3atUunTp3SyJEjXT9zOBzatGmTXnnlFdXV1cnHx6fZa8w4h177NY2/v79GjRqldevWuX7mdDq1bt26Fr8LHD9+fLPjJSk9Pf2i3x2aqT1j/D6Hw6G9e/cqNja2s2J2KU87hx0hIyPDbc+fYRiaP3++VqxYofXr16tfv36XfI0nncP2jO/7PPEz6HQ6VVdXd8HnPOn8teRi4/s+dz9/06ZN0969e5WRkeF6jB49WrfffrsyMjLOKyKSSeew06bGuoHly5cbNpvNePPNN43MzEzjpz/9qdGjRw+jsLDQMAzDuOOOO4xFixa5jt+yZYvh6+trPPvss8aBAweMxx9/3PDz8zP27t1r1hAuqa1jfOKJJ4w1a9YYR44cMXbt2mXceuutRkBAgLF//36zhnBRFRUVxp49e4w9e/YYkoznn3/e2LNnj5GXl2cYhmEsWrTIuOOOO1zHHz161AgKCjIeeeQR48CBA8ayZcsMHx8fY/Xq1WYN4aLaOr4XXnjBWLlypZGdnW3s3bvXWLBggWG1Wo21a9eaNYSL+td//VfDbrcbGzZsME6ePOl6VFdXu47x5M9he8bnaZ/BRYsWGRs3bjRycnKMr7/+2li0aJFhsViMtLQ0wzA8+/wZRtvH52nn70K+fzeNO5xDry4jhmEYv/3tb40+ffoY/v7+xtixY43t27e7nps0aZJx1113NTv+3XffNQYNGmT4+/sbQ4cONT7++OMuTtx2bRnjQw895Do2OjrauP76643du3ebkLp1vrmV9fuPb8Z01113GZMmTTrvNSNGjDD8/f2N/v37G2+88UaX526tto7vqaeeMgYMGGAEBAQYERERxuTJk43169ebE74VLjQ2Sc3OiSd/DtszPk/7DN5zzz1G3759DX9/f6NXr17GtGnTXL+oDcOzz59htH18nnb+LuT7ZcQdzqHFMAyj8667AAAAXJzXzhkBAACegTICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKaijAAAAFP9fxXxJp5RKO+SAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "class CustomCallback(keras.callbacks.Callback):\n",
        "  def on_epoch_end(self, epoch, logs=None):\n",
        "    model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n",
        "    gemma.backbone.save_lora_weights(model_name)\n",
        "\n",
        "    # Evaluate\n",
        "    text_gen(\"ᄃᆡ작ᄒᆞ여그ᄭᅩᆺ치흣터지거ᄂᆞᆯ\")\n",
        "    text_gen(\n",
        "      \"금두겁이품의드러뵈니일졍ᄌᆡᄌᆞᄅᆞᆯ나흐리로다ᄒᆞ더니과연그ᄃᆞᆯ부터잉ᄐᆡᄒᆞ여십삭이차니\"\n",
        "    )\n",
        "    text_gen(\n",
        "      \"이ᄯᆡᄂᆞᆫᄉᆞ월초팔일이라이날밤의오ᄉᆡᆨ구룸이집을두루고향ᄂᆡ진동ᄒᆞ며션녀ᄒᆞᆫᄡᅣᆼ이촉을들고드러와김ᄉᆡᆼᄃᆞ려니르ᄃᆡ\"\n",
        "    )\n",
        "    text_gen(\"ᄌᆡ히길너텬졍을어긔지말으소셔이아희ᄇᆡ필은낙양니샹셔집아ᄌᆡ니\")\n",
        "\n",
        "history = gemma.fit(train, epochs=train_epoch, batch_size=1, callbacks=[CustomCallback()])\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.plot(history.history['loss'])\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3FMPzVKFgRT"
      },
      "source": [
        "## Expansion Idea\n",
        "\n",
        "To achieve similar tasks, you can replicate the same structure. Below are some examples:\n",
        "\n",
        "* American English <-> British English datasets\n",
        "\n",
        "Various everyday objects and concepts have different names depending on the region. For example, in American English (AmE), people use terms like \"elevator,\" \"truck,\" \"cookie,\" and \"french fries,\" while in British English (BrE), the equivalent words are \"lift,\" \"lorry,\" \"biscuit,\" and \"chips,\" respectively.\n",
        "\n",
        "Apart from vocabulary differences, spelling variations also exist. For instance, in AmE, words ending in \"-or\" are often spelled with \"-our\" in BrE. Examples include \"color\" (AmE) and \"colour\" (BrE), or \"humor\" (AmE) and \"humour\" (BrE).\n",
        "\n",
        "Another spelling variation is the \"-ize\" versus \"-ise\" distinction. In AmE, words like \"organize\" and \"realize\" are commonly spelled with a \"z,\" whereas in BrE, the preferred spelling is \"organise\" and \"realise,\" using an \"s\" instead.\n",
        "\n",
        "With the help of AI tools like Gemma, it is possible to create a style transfer from one English dialect to another, allowing seamless transitions between American and British English writing styles.\n",
        "\n",
        "* Kansai-ben datasets\n",
        "\n",
        "In the Kansai region of Japan, there is a distinct group of dialects known as Kansai-ben. Compared to the standard Japanese language, Japanese speakers perceive Kansai-ben as being both more melodic and harsher in its pronunciation and intonation.\n",
        "\n",
        "Utilizing the capabilities of Gemma, you can create a dialect translator by preparing a substantial quantity of Kansai-ben datasets.\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Translator_of_Old_Korean_Literature.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}