U3.04 - [TALLER] TensorFlow Low Level API.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-cache -O init.py -q https://raw.githubusercontent.com/rramosp/2020.deeplearning/master/init.py\n",
    "from init import init; init(force_download=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    print (\"Using TF2 in Google Colab\")\n",
    "except:\n",
    "    pass\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "%load_ext tensorboard\n",
    "\n",
    "from sklearn.datasets import *\n",
    "from local.lib import mlutils\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**A multilayer perceptron**\n",
    "\n",
    "assuming $n$ layers, the output at layer $i$\n",
    "\n",
    "$$\\mathbf{a}_i = \\text{activation}(\\mathbf{a}_{i-1} \\cdot \\mathbf{W}_i + \\mathbf{b}_i)$$\n",
    "\n",
    "at the first layer\n",
    "\n",
    "$$\\mathbf{a}_0 = \\text{activation}(\\mathbf{X} \\cdot \\mathbf{W}_0 + \\mathbf{b}_0)$$\n",
    "\n",
    "and the layer prediction is the output of the last layer:\n",
    "\n",
    "$$\\hat{\\mathbf{y}} = \\mathbf{a}_{n-1}$$ \n",
    "\n",
    "with $\\text{activation}$ being an activation function, such as $\\text{sigmoid}(z) = \\frac{1}{1+e^{-z}}$, $\\text{tanh}$, $\\text{ReLU}$, etc.\n",
    "\n",
    "\n",
    "**Cost (with regularization)**\n",
    "\n",
    "\n",
    "$$J(\\mathbf{b}_1, b_2, \\mathbf{W}_1, \\mathbf{W}_2) = \\frac{1}{m}\\sum_{i=0}^{m-1} (\\hat{y}-y)^2 + \\lambda \\sum_{i=0}^{n-1} \\bigg[ \\| \\mathbf{b}_i\\|^2 + \\|\\mathbf{W}_i\\|^2 \\bigg]$$\n",
    "\n",
    "\n",
    "$\\lambda$ regulates the participation of the regularization terms. Given a vector or matrix $\\mathbf{T}$, its squared norm is denoted by $||\\mathbf{T}||^2 \\in \\mathbb{R}$ and it's computed by squaring all its elements and summing them all up. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Multilayer perceptron model with low level API\n",
    "\n",
    "build a custom model (`build`, `call`) using standard optimization (`model.compile`, etc.)\n",
    "\n",
    "your class must:\n",
    "\n",
    "- accept in the constructor:\n",
    "   - `neurons`: a list containing the number of neurons of each layer\n",
    "   - `activations`: a list of strings containing one of `'sigmoid'`, `'tanh'`, `'linear'`, `'relu'` so that the corresponding `tf.keras.activations` function is used.\n",
    "\n",
    "- **include Tensorboard callbacks for BOTH LOSS AND ACCURACY. See the [example here](https://www.tensorflow.org/tensorboard/get_started)**. You will have to add the appropriate [keras metric](https://www.tensorflow.org/api_docs/python/tf/keras/metrics) according to your model output.\n",
    "\n",
    "- create a **custom loss function** to include the regularization parameter and **use it when compiling the model**. Look in the internet for tutorials, for example [here](https://towardsdatascience.com/advanced-keras-constructing-complex-custom-losses-and-metrics-c07ca130a618), and many others\n",
    "\n",
    "for instance, the following code:\n",
    "\n",
    "    mlp = MLP(neurons=[10,1], activations=[\"tanh\",\"sigmoid\"])\n",
    "\n",
    "must create a MLP with two layers with 10 and 1 neurons each, and `tanh` and `sigmoid` activation correspondingly.\n",
    "\n",
    "\n",
    "Observe that:\n",
    "\n",
    "- as you are following the Keras class API (`call`+ `build`) your model should work as any other model (`compile`, `fit`, etc.).\n",
    "\n",
    "- the `loss` method of your MLP instance must be passed on when calling `compile`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.activations import relu, sigmoid, tanh, linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from progressbar import progressbar as pbar\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.activations import relu, sigmoid, tanh, linear\n",
    "\n",
    "class MLP(Model):\n",
    "    def __init__(self, neurons, activations, reg=0.):\n",
    "        super(MLP, self).__init__()\n",
    "        assert len(neurons)==len(activations), \"must have the same number of neurons and activations\"\n",
    "        \n",
    "        ... # YOUR CODE HERE\n",
    "        \n",
    "    def build(self, input_shape):\n",
    "        \n",
    "        ... # YOUR CODE HERE\n",
    "        \n",
    "    @tf.function\n",
    "    def call(self, X):\n",
    "        ... # YOUR CODE HERE\n",
    "\n",
    "    @tf.function\n",
    "    def loss(self, y_true, y_pred):\n",
    "        ... # YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_moons(200, noise=.35)\n",
    "X, y = X.astype(np.float32), y.astype(np.float32).reshape(-1,1)\n",
    "plt.scatter(X[:,0][y[:,0]==0], X[:,1][y[:,0]==0], color=\"red\", label=\"class 0\")\n",
    "plt.scatter(X[:,0][y[:,0]==1], X[:,1][y[:,0]==1], color=\"blue\", label=\"class 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf logs\n",
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs/no_regularization\")\n",
    "\n",
    "mlp = MLP(neurons=[10,1], activations=[\"tanh\",\"sigmoid\"])\n",
    "\n",
    "mlp.compile(... # YOUR CODE HERE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp.fit(X,y, epochs=400, batch_size=16, verbose=0, \n",
    "        callbacks=[tensorboard_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict = lambda X: (mlp.predict(X)[:,0]>0.5).astype(int)\n",
    "mlutils.plot_2Ddata_with_boundary(predict, X, y.reshape(-1));\n",
    "plt.title(\"accuracy %.2f\"%np.mean(predict(X)==y.reshape(-1)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "regularization must work!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs/with_regularization\")\n",
    "mlp = MLP(neurons=[10,1], activations=[\"tanh\",\"sigmoid\"], reg=0.001)\n",
    "\n",
    "mlp.compile(... # YOUR CODE HERE)\n",
    "\n",
    "mlp.fit(X,y, epochs=400, batch_size=10, verbose=0, callbacks=[tensorboard_callback])\n",
    "mlutils.plot_2Ddata_with_boundary(predict, X, y.reshape(-1))\n",
    "plt.title(\"accuracy %.2f\"%np.mean(predict(X)==y.reshape(-1)));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir logs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Multilayer perceptron model AND optimization loop with low level API\n",
    "\n",
    "build a custom model such as in the exercise above and implement your optimization loop in `.fit`\n",
    " \n",
    "observe that you will have to:\n",
    "- use whichever method you choose from the corresponding notebook (custom SGD, `apply_gradients`, `train_on_batch`)\n",
    "- write by hand loss and accuracy to Tensorboard. See  how[`tf.summary`](https://www.tensorflow.org/api_docs/python/tf/summary) works\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class MLP(Model):\n",
    "    def __init__(self, neurons, activations, reg=0.):\n",
    "        super(MLP, self).__init__()\n",
    "        assert len(neurons)==len(activations), \"must have the same number of neurons and activations\"\n",
    "        \n",
    "        ... # YOUR CODE HERE\n",
    "        \n",
    "    def build(self, input_shape):\n",
    "        \n",
    "        ... # YOUR CODE HERE\n",
    "\n",
    "                \n",
    "    @tf.function\n",
    "    def call(self, X):\n",
    "        ... # YOUR CODE HERE\n",
    "\n",
    "\n",
    "    @tf.function\n",
    "    def loss(self, y_true, y_pred):\n",
    "        ... # YOUR CODE HERE\n",
    "    \n",
    "    def fit(self, X, y, epochs, batch_size):\n",
    "        ... # YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "... use tensorboard!!! ...\n",
    "\n",
    "mlp = MLP(neurons=[10,1], activations=[\"tanh\",\"sigmoid\"])\n",
    "mlp.compile(... # YOUR CODE HERE)\n",
    "\n",
    "mlp.fit(X,y, epochs=400, batch_size=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlutils.plot_2Ddata_with_boundary(predict, X, y.reshape(-1));\n",
    "plt.title(\"accuracy %.2f\"%np.mean(predict(X)==y.reshape(-1)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "regularization must work!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "... use tensorboard!!! ...\n",
    "\n",
    "mlp = MLP(neurons=[10,1], activations=[\"tanh\",\"sigmoid\"], reg=0.01)\n",
    "mlp.compile(... # YOUR CODE HERE)\n",
    "\n",
    "mlp.fit(X,y, epochs=400, batch_size=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlutils.plot_2Ddata_with_boundary(predict, X, y.reshape(-1));\n",
    "plt.title(\"accuracy %.2f\"%np.mean(predict(X)==y.reshape(-1)));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%tensorboard --logdir logs\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "p37",
   "language": "python",
   "name": "p37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}