Real_Time_Energy_Data_Low_Carbon_Training_Project_Summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Real Time Energy Data For Low Carbon Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "- Select a training location with the lowest real time carbon intensity to perform low carbon training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Process\n",
    "\n",
    "1. Retrieve real time carbon intensity data by API\n",
    "\n",
    "2. Select the region with the lowest carbon intensity for training \n",
    "\n",
    "3. Perform low carbon training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detail Process\n",
    "\n",
    "1. Set up GCP\n",
    "\n",
    "    - Account info\n",
    "\n",
    "    - Project info, i.e. project ID\n",
    "\n",
    "    - Credentials\n",
    "\n",
    "2. Initialize Vertex AI\n",
    "\n",
    "    - `from google.cloud import aiplatform`\n",
    "    \n",
    "    - Initialize with project ID and credentials\n",
    "\n",
    "3. Set up training environment\n",
    "\n",
    "    - Import environment\n",
    "\n",
    "    - Load environment\n",
    "\n",
    "    - Import request\n",
    "\n",
    "    - Import helper\n",
    "\n",
    "    - Import json\n",
    "\n",
    "4. Load the Electricity Maps API\n",
    "    - [Free personal tier API](https://api-portal.electricitymaps.com/)\n",
    "\n",
    "5. Select the training location with the real time lowest carbon intensity\n",
    "\n",
    "\n",
    "    - Method 1: Define carbon intensity function with selected training location\n",
    "\n",
    "        - Define an url by getting latitude and longitude on Google Map\n",
    "\n",
    "        - Define get request\n",
    "\n",
    "        - Load json with API returned content\n",
    "\n",
    "            - Carbon intensity\n",
    "\n",
    "            - Datetime\n",
    "\n",
    "            - UpdatedAt time\n",
    "\n",
    "            - CreatedAt time\n",
    "\n",
    "            - Renewable energy type and its amount\n",
    "            \n",
    "            - Power consumption amount\n",
    "    \n",
    "    - Method 2: Define cleanest function with region list\n",
    "\n",
    "        - Return the training location with the real time lowest carbon intensity out of region list\n",
    "\n",
    "6. Define task.py as a training job\n",
    "\n",
    "    - Import libraries\n",
    "\n",
    "    - Create/ load a dataset\n",
    "    \n",
    "    - Create model\n",
    "\n",
    "7. Define storage bucket\n",
    "\n",
    "    - `from google.cloud import storage`\n",
    "\n",
    "    - Define with project ID and credentials\n",
    "\n",
    "8. Define custom training job\n",
    "\n",
    "    - Include selected training location\n",
    "\n",
    "    - Include task.py training job\n",
    "\n",
    "    - Include storage bucket\n",
    "    \n",
    "    - Include display job name\n",
    "\n",
    "9. Train model in selected training location\n",
    "\n",
    "10. Delete bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Out of Scope\n",
    "\n",
    "- Carbon footprint tracking report per project and cloud account"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remarks \n",
    "\n",
    "- One alternative way is to define Power_stats to view both real time carbon intensity and power breakdown data of a selected training location in one step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import helper, requests, json, numpy as np\n",
    "def power_stats(lat,lon, api_key=helper.load_emaps_api_key()):\n",
    "    coordinates = { \"lat\": lat, \"lon\": lon }\n",
    "\n",
    "    url_intensity = f\"https://api.electricitymap.org/v3/carbon-intensity/latest?lat={coordinates['lat']}&lon={coordinates['lon']}\"\n",
    "    request_intensity = requests.get(url_intensity, headers={\"auth-token\": api_key})\n",
    "    intensity = json.loads(request_intensity.content)\n",
    "\n",
    "    url_breakdown = f\"https://api.electricitymap.org/v3/power-breakdown/latest?lat={coordinates['lat']}&lon={coordinates['lon']}\"\n",
    "    request_breakdown = requests.get(url_breakdown, headers={\"auth-token\": api_key})\n",
    "    breakdown = json.loads(request_breakdown.content)\n",
    "\n",
    "    breakdown_abridged = {\n",
    "        'renewablePercentage': breakdown['renewablePercentage'],\n",
    "        'fossilFreePercentage': breakdown['fossilFreePercentage'],\n",
    "        'powerConsumptionBreakdown': breakdown['powerConsumptionBreakdown'],\n",
    "        'consumption_percent': {\n",
    "            k: np.round((v/breakdown['powerConsumptionTotal']) * 100) \n",
    "            for k, v \n",
    "            in breakdown['powerConsumptionBreakdown'].items()\n",
    "        },\n",
    "    }\n",
    "    \n",
    "    return intensity, breakdown_abridged"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}