llama3_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NIMs for LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Load the .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Get the value of the key\n",
    "nvapi_key = os.getenv('nvapi')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(\n",
    "  base_url = \"https://integrate.api.nvidia.com/v1\",\n",
    "  api_key = nvapi_key\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Llama 3.1_405B !!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def llama3_1_405b(input):\n",
    "  completion = client.chat.completions.create(\n",
    "    model=\"meta/llama-3.1-405b-instruct\",\n",
    "    messages=[{\"role\":\"user\",\n",
    "              \"content\":input}],\n",
    "    temperature=0.2,\n",
    "    top_p=0.7,\n",
    "    max_tokens=1024,\n",
    "    stream=True\n",
    "  )\n",
    "\n",
    "  for chunk in completion:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "      print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Puedo entender y generar texto en varios idiomas, incluyendo pero no limitado a:\n",
      "\n",
      "* Español\n",
      "* Inglés\n",
      "* Francés\n",
      "* Alemán\n",
      "* Italiano\n",
      "* Portugués\n",
      "* Chino (simplificado y tradicional)\n",
      "* Japonés\n",
      "* Coreano\n",
      "* Ruso\n",
      "* Árabe\n",
      "* Hebreo\n",
      "* Hindi\n",
      "\n",
      "Sin embargo, mi capacidad para comprender y generar texto en cada idioma puede variar dependiendo de la complejidad del texto y la cantidad de datos de entrenamiento que he recibido en ese idioma.\n",
      "\n",
      "En general, puedo:\n",
      "\n",
      "* Entender y responder a preguntas básicas en la mayoría de los idiomas mencionados anteriormente.\n",
      "* Generar texto coherente y gramaticalmente correcto en inglés, español, francés, alemán, italiano y portugués.\n",
      "* Generar texto con cierta precisión en chino, japonés, coreano y ruso, aunque puede haber errores gramaticales o de vocabulario.\n",
      "* Entender y generar texto en árabe, hebreo y hindi, aunque mi capacidad en estos idiomas es limitada y puede requerir más tiempo y esfuerzo para generar respuestas precisas.\n",
      "\n",
      "Es importante tener en cuenta que mi capacidad para comprender y generar texto en diferentes idiomas es un área en constante desarrollo, y estoy en proceso de mejorar mi habilidad para comunicarme en varios idiomas."
     ]
    }
   ],
   "source": [
    "llama3_1_405b(\"Cuantos idiomas hablas?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Llama 3.1 8B \n",
    "**(probably more than enough..)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def llama3_1_8b(input):\n",
    "  completion = client.chat.completions.create(\n",
    "    model=\"meta/llama-3.1-8b-instruct\",\n",
    "    messages=[{\"role\":\"user\",\n",
    "              \"content\":input}],\n",
    "    temperature=0.2,\n",
    "    top_p=0.7,\n",
    "    max_tokens=1024,\n",
    "    stream=True\n",
    "  )\n",
    "\n",
    "  for chunk in completion:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "      print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parlo l'italiano, l'inglese, il francese e il tedesco."
     ]
    }
   ],
   "source": [
    "llama3_1_8b(\"quante lingue parli?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### On Nvidia NIMs, by Llama 3.1..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nvidia NIMs (Nvidia Instant Neurons) are a technology that allows for the acceleration of AI models, such as LLaMA 3.1 8b and 405b, on NVIDIA GPUs. Here's a high-level overview of how they work:\n",
      "\n",
      "1. **Model pruning**: NIMs are a type of model pruning, where a large AI model is reduced in size while maintaining its accuracy. This is done by removing redundant or unnecessary parameters, which reduces the model's computational requirements.\n",
      "2. **Knowledge distillation**: The pruned model is then distilled into a smaller, more efficient model that can be run on a GPU. This process involves training the smaller model to mimic the behavior of the original, larger model.\n",
      "3. **NIMs as a lookup table**: The distilled model is then converted into a lookup table, which is essentially a large array of precomputed values. This lookup table is used to accelerate the model's computations.\n",
      "4. **GPU acceleration**: When you run the NIM on a GPU, the lookup table is used to accelerate the model's computations. The GPU performs a series of matrix multiplications and other operations on the lookup table, which allows for fast and efficient inference.\n",
      "\n",
      "Now, regarding your question about computer specs:\n",
      "\n",
      "**Yes, your computer specs matter**:\n",
      "\n",
      "1. **GPU**: You'll need a high-end NVIDIA GPU with sufficient memory (at least 24 GB) and compute power (e.g., NVIDIA A100, A40, or RTX 3080 Ti). The more powerful the GPU, the faster the NIM will run.\n",
      "2. **Memory**: You'll need sufficient system memory (RAM) to accommodate the NIM's lookup table, which can be quite large (e.g., 10-20 GB or more). The more RAM, the better.\n",
      "3. **CPU**: While the CPU is not the primary bottleneck, a fast CPU can help with other tasks, such as loading the NIM and preparing the input data.\n",
      "4. **Storage**: You'll need a fast storage drive (e.g., NVMe SSD) to load the NIM and other dependencies quickly.\n",
      "\n",
      "To give you a rough idea, here are some estimated requirements for running LLaMA 3.1 8b and 405b on a NVIDIA GPU:\n",
      "\n",
      "* LLaMA 3.1 8b: 24-32 GB of GPU memory, 16-32 GB of system RAM, and a high-end NVIDIA GPU (e.g., A100 or A40)\n",
      "* LLaMA 3.1 405b: 32-48 GB of GPU memory, 32-64 GB of system RAM, and a high-end NVIDIA GPU (e.g., A100 or A40)\n",
      "\n",
      "Keep in mind that these are rough estimates, and the actual requirements may vary depending on the specific use case and implementation.\n",
      "\n",
      "If you're planning to run LLaMA 3.1 8b and 405b locally using NIMs, I recommend checking the official NVIDIA documentation and the LLaMA repository for more detailed information on the required hardware specifications."
     ]
    }
   ],
   "source": [
    "llama3_1_8b(\"How do Nvidia NIMs work? If I were to be using nims to run Llama 3.1 8b and 405b locally, do my computer specs matter at all?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nims-llama",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}