Ch06_Multilayer_Perceptrons/Weight_Decay.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weight Decay\n",
    "\n",
    "Now that we have characterized the problem of overfitting\n",
    "and motivated the need for capacity control,\n",
    "we can begin discussing some of the popular techniques\n",
    "used to these ends in practice.\n",
    "Recall that we can always mitigate overfitting\n",
    "by going out and collecting more training data,\n",
    "that can be costly and time consuming,\n",
    "typically making it impossible in the short run.\n",
    "For now, let's assume that we have already obtained\n",
    "as much high-quality data as our resources permit\n",
    "and focus on techniques aimed at limiting the capacity\n",
    "of the function classes under consideration.\n",
    "\n",
    "In our toy example,\n",
    "we saw that we could control the complexity of a polynomial\n",
    "by adjusting its degree.\n",
    "However, most of machine learning\n",
    "does not consist of polynomial curve fitting.\n",
    "And moreover, even when we focus on polynomial regression,\n",
    "when we deal with high-dimensional data,\n",
    "manipulating model capacity by tweaking the degree $d$ is problematic.\n",
    "To see why, note that for multivariate data\n",
    "we must generalize the concept of polynomials\n",
    "to include *monomials*, which are simply\n",
    "products of powers of variables.\n",
    "For example, $x_1^2 x_2$, and $x_3 x_5^2$ are both monomials of degree $3$.\n",
    "The number of such terms with a given degree $d$\n",
    "blows up as a function of the degree $d$.\n",
    "\n",
    "Concretely, for vectors of dimensionality $D$,\n",
    "the number of monomials of a given degree $d$ is ${D -1 + d} \\choose {D-1}$.\n",
    "Hence, a small change in degree, even from say $1$ to $2$ or $2$ to $3$\n",
    "would entail a massive blowup in the complexity of our model.\n",
    "Thus, tweaking the degree is too blunt a hammer.\n",
    "Instead, we need a more fine-grained tool\n",
    "for adjusting function complexity.\n",
    "\n",
    "## Squared Norm Regularization\n",
    "\n",
    "*Weight decay* (commonly called *L2* regularization),\n",
    "might be the most widely-used technique\n",
    "for regularizing parametric machine learning models.\n",
    "The basic intuition behind weight decay is\n",
    "the notion that among all functions $f$,\n",
    "the function $f = 0$ is the simplest.\n",
    "Intuitively, we can then measure functions by their proximity to zero.\n",
    "But how precisely should we measure\n",
    "the distance between a function and zero?\n",
    "There is no single right answer.\n",
    "In fact, entire branches of mathematics,\n",
    "e.g. in functional analysis and the theory of Banach spaces\n",
    "are devoted to answering this issue.\n",
    "\n",
    "For our present purposes, a very simple interpretation will suffice:\n",
    "We will consider a linear function\n",
    "$f(\\mathbf{x}) = \\mathbf{w}^\\top \\mathbf{x}$\n",
    "to be simple if its weight vector is small.\n",
    "We can measure this via $||\\mathbf{w}||^2$.\n",
    "One way of keeping the weight vector small\n",
    "is to add its norm as a penalty term\n",
    "to the problem of minimizing the loss.\n",
    "Thus we replace our original objective,\n",
    "*minimize the prediction error on the training labels*,\n",
    "with new objective,\n",
    "*minimize the sum of the prediction error and the penalty term*.\n",
    "Now, if the weight vector becomes too large,\n",
    "our learning algorithm will find more profit in\n",
    "minimizing the norm $|| \\mathbf{w} ||^2$\n",
    "versus minimizing the training error.\n",
    "That's exactly what we want.\n",
    "To illustrate things in code, let's revive our previous example\n",
    "from linear regression.\n",
    "There, our loss was given by\n",
    "\n",
    "$$l(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n",
    "\n",
    "Recall that $\\mathbf{x}^{(i)}$ are the observations,\n",
    "$y^{(i)}$ are labels, and $(\\mathbf{w}, b)$\n",
    "are the weight and bias parameters respectively.\n",
    "To arrive at a new loss function\n",
    "that penalizes the size of the weight vector,\n",
    "we need to add $||\\mathbf{w}||^2$, but how much should we add?\n",
    "To address this, we need to add a new hyperparameter,\n",
    "that we will call the *regularization constant* and denote by $\\lambda$:\n",
    "\n",
    "$$l(\\mathbf{w}, b) + \\frac{\\lambda}{2} \\|\\boldsymbol{w}\\|^2$$\n",
    "\n",
    "This non-negative parameter $\\lambda \\geq 0$\n",
    "governs the amount of regularization.\n",
    "For $\\lambda = 0$, we recover our original loss function,\n",
    "whereas for $\\lambda > 0$ we ensure that $\\mathbf{w}$ cannot grow too large. The astute reader might wonder why we are squaring\n",
    "the norm of the weight vector.\n",
    "We do this for two reasons.\n",
    "First, we do it for computational convenience.\n",
    "By squaring the L2 norm, we remove the square root,\n",
    "leaving the sum of squares of each component of the weight vector.\n",
    "This is convenient because it is easy to compute derivatives of a sum of terms (the sum of derivatives equals the derivative of the sum).\n",
    "\n",
    "Moreover, you might ask, why the L2 norm in the first place and not the L1 norm, or some other distance function.\n",
    "In fact, several other choices are valid\n",
    "and are popular throughout statistics.\n",
    "While L2-regularized linear models constitute\n",
    "the classic *ridge regression* algorithm\n",
    "L1-regularizaed linear regression\n",
    "is a similarly fundamental model in statistics\n",
    "popularly known as *lasso regression*.\n",
    "\n",
    "One mathematical reason for working with the L2 norm and not some other norm,\n",
    "is that it penalizes large components of the weight vector\n",
    "much more than it penalizes small ones.\n",
    "This encourages our learning algorithm to discover models\n",
    "which distribute their weight across a larger number of features,\n",
    "which might make them more robust in practice\n",
    "since they do not depend precariously on a single feature.\n",
    "The stochastic gradient descent updates for L2-regularied regression\n",
    "are as follows:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "w & \\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right) \\mathbf{w} - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right),\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "As before, we update $\\mathbf{w}$ based on the amount\n",
    "by which our estimate differs from the observation.\n",
    "However, we also shrink the size of $\\mathbf{w}$ towards $0$.\n",
    "That's why the method is sometimes called \"weight decay\":\n",
    "because the penalty term literally causes our optimization algorthm\n",
    "to *decay* the magnitude of the weight at each step of training.\n",
    "This is more convenient than having to pick\n",
    "the number of parameters as we did for polynomials.\n",
    "In particular, we now have a continuous mechanism\n",
    "for adjusting the complexity of $f$.\n",
    "Small values of $\\lambda$ correspond to unconstrained $\\mathbf{w}$,\n",
    "whereas large values of $\\lambda$ constrain $\\mathbf{w}$ considerably.\n",
    "Since we don't want to have large bias terms either,\n",
    "we often add $b^2$ as a penalty, too.\n",
    "\n",
    "## High-dimensional Linear Regression\n",
    "\n",
    "For high-dimensional regression it is difficult\n",
    "to pick the 'right' dimensions to omit.\n",
    "Weight-decay regularization is a much more convenient alternative.\n",
    "We will illustrate this below.\n",
    "First, we will generate some synthetic data as before\n",
    "\n",
    "$$y = 0.05 + \\sum_{i = 1}^d 0.01 x_i + \\epsilon \\text{ where }\n",
    "\\epsilon \\sim \\mathcal{N}(0, 0.01)$$\n",
    "\n",
    "representing our label as a linear function of our inputs,\n",
    "corrupted by Gaussian noise with zero mean and variance 0.01.\n",
    "To observe the effects of overfitting more easily,\n",
    "we can make our problem high-dimensional,\n",
    "setting the data dimension to $d = 200$\n",
    "and working with a relatively small number of training examples—here we'll set the sample size to 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "\n",
    "%matplotlib inline\n",
    "import d2l\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.utils.data\n",
    "\n",
    "n_train, n_test, num_inputs = 20, 100, 200\n",
    "true_w, true_b = torch.ones(size=(num_inputs, 1)) * 0.01, torch.tensor(0.05)\n",
    "\n",
    "features = torch.randn(size=(n_train + n_test, num_inputs))\n",
    "labels = torch.matmul(features, true_w) + true_b\n",
    "labels.add_(torch.empty(size=labels.shape).normal_(0, std = 0.01))\n",
    "train_features, test_features = features[:n_train, :], features[n_train:, :]\n",
    "train_labels, test_labels = labels[:n_train], labels[n_train:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation from Scratch\n",
    "\n",
    "Next, we will show how to implement weight decay from scratch.\n",
    "All we have to do here is to add the squared $\\ell_2$ penalty\n",
    "as an additional loss term added to the original target function.\n",
    "The squared norm penalty derives its name from the fact\n",
    "that we are adding the second power $\\sum_i w_i^2$.\n",
    "The $\\ell_2$ is just one among an infinite class of norms call p-norms,\n",
    "many of which you might encounter in the future.\n",
    "In general, for some number $p$, the $\\ell_p$ norm is defined as\n",
    "\n",
    "$$\\|\\mathbf{w}\\|_p^p := \\sum_{i=1}^d |w_i|^p$$\n",
    "\n",
    "### Initialize Model Parameters\n",
    "\n",
    "First, we'll define a function to randomly initialize our model parameters and set `requires_grad`as `True` on each to allocate memory for the gradients we will calculate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [],
   "source": [
    "def init_params():\n",
    "    w = torch.randn(size=(num_inputs, 1),requires_grad=True)\n",
    "    b = torch.zeros((1,),requires_grad=True)\n",
    "    return [w, b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define $\\ell_2$ Norm Penalty\n",
    "\n",
    "Perhaps the most convenient way to implement this penalty\n",
    "is to square all terms in place and summ them up.\n",
    "We divide by $2$ by convention\n",
    "(when we take the derivative of a quadratic function,\n",
    "the $2$ and $1/2$ cancel out, ensuring that the expression\n",
    "for the update looks nice and simple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    }
   },
   "outputs": [],
   "source": [
    "def l2_penalty(w):\n",
    "    return torch.sum(w.pow(2)) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Training and Testing\n",
    "\n",
    "The following code defines how to train and test the model\n",
    "separately on the training data set and the test data set.\n",
    "Unlike the previous sections, here, the $\\ell_2$ norm penalty term\n",
    "is added when calculating the final loss function.\n",
    "The linear network and the squared loss\n",
    "haven't changed since the previous chapter,\n",
    "so we'll just import them via `d2l.linreg` and `d2l.squared_loss`\n",
    "to reduce clutter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [],
   "source": [
    "batch_size, num_epochs, lr = 1, 100, 0.003\n",
    "net, loss = d2l.linreg, d2l.squared_loss\n",
    "train_iter = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(train_features, train_labels), batch_size, shuffle=True)\n",
    "\n",
    "def fit_and_plot(lambd):\n",
    "    w, b = init_params()\n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            with torch.enable_grad():\n",
    "                # The L2 norm penalty term has been added\n",
    "                y_hat = net(X, w, b)\n",
    "                l = loss(y_hat, y) + lambd * l2_penalty(w)\n",
    "            l.backward()\n",
    "            d2l.sgd([w, b], lr, batch_size)\n",
    "        with torch.no_grad():    \n",
    "            train_ls.append(torch.mean(loss(net(train_features, w, b),train_labels)).item())\n",
    "            test_ls.append(torch.mean(loss(net(test_features, w, b),test_labels)).item())\n",
    "    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('l2 norm of w:', torch.norm(w).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training without Regularization\n",
    "\n",
    "Next, let's train and test the high-dimensional linear regression model.\n",
    "When `lambd = 0` we do not use weight decay.\n",
    "As a result, while the training error decreases, the test error does not.\n",
    "This is a perfect example of overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "8"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 261.778125 184.15625\" width=\"261.778125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 261.778125 184.15625 \r\n",
       "L 261.778125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 251.078125 146.6 \r\n",
       "L 251.078125 10.7 \r\n",
       "L 55.778125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m299a673e26\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(59.680759 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.367278 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.235046 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(164.102815 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(199.970584 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#m299a673e26\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(232.657102 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(135.595313 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mc28b376f45\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc28b376f45\" y=\"124.878746\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-10}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 128.677964)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(231.391602 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc28b376f45\" y=\"97.827215\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-7}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 8.203125 72.90625 \r\n",
       "L 55.078125 72.90625 \r\n",
       "L 55.078125 68.703125 \r\n",
       "L 28.609375 0 \r\n",
       "L 18.3125 0 \r\n",
       "L 43.21875 64.59375 \r\n",
       "L 8.203125 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-55\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(25.278125 101.626434)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-55\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc28b376f45\" y=\"70.775685\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-4}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 74.574904)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc28b376f45\" y=\"43.724155\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 47.523374)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc28b376f45\" y=\"16.672625\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(31.178125 20.471844)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#p22d525c269)\" d=\"M 64.655398 24.149477 \r\n",
       "L 66.448786 30.2812 \r\n",
       "L 68.242175 35.379695 \r\n",
       "L 70.035563 40.292765 \r\n",
       "L 71.828951 44.966575 \r\n",
       "L 73.62234 49.615542 \r\n",
       "L 75.415728 54.036645 \r\n",
       "L 77.209117 58.279455 \r\n",
       "L 79.002505 62.430257 \r\n",
       "L 80.795894 66.489236 \r\n",
       "L 82.589282 70.485675 \r\n",
       "L 84.38267 74.458646 \r\n",
       "L 86.176059 78.256129 \r\n",
       "L 87.969447 82.118076 \r\n",
       "L 89.762836 85.882267 \r\n",
       "L 91.556224 89.576142 \r\n",
       "L 93.349613 93.221696 \r\n",
       "L 95.143001 96.780424 \r\n",
       "L 96.936389 100.473599 \r\n",
       "L 98.729778 104.019105 \r\n",
       "L 100.523166 107.514955 \r\n",
       "L 102.316555 111.028708 \r\n",
       "L 104.109943 114.475221 \r\n",
       "L 105.903332 118.026404 \r\n",
       "L 107.69672 121.374561 \r\n",
       "L 109.490108 124.63286 \r\n",
       "L 111.283497 127.443351 \r\n",
       "L 113.076885 129.809162 \r\n",
       "L 114.870274 131.150141 \r\n",
       "L 116.663662 132.629809 \r\n",
       "L 118.457051 133.779111 \r\n",
       "L 120.250439 136.025567 \r\n",
       "L 122.043827 136.086902 \r\n",
       "L 123.837216 136.153411 \r\n",
       "L 125.630604 136.738141 \r\n",
       "L 127.423993 136.446314 \r\n",
       "L 129.217381 136.579127 \r\n",
       "L 131.01077 136.575985 \r\n",
       "L 132.804158 136.671452 \r\n",
       "L 134.597546 136.881839 \r\n",
       "L 136.390935 136.573061 \r\n",
       "L 138.184323 136.627785 \r\n",
       "L 139.977712 137.920297 \r\n",
       "L 141.7711 138.030297 \r\n",
       "L 143.564489 138.173679 \r\n",
       "L 145.357877 137.98142 \r\n",
       "L 147.151265 137.670589 \r\n",
       "L 148.944654 137.782501 \r\n",
       "L 150.738042 137.920743 \r\n",
       "L 152.531431 138.580698 \r\n",
       "L 154.324819 138.731909 \r\n",
       "L 156.118208 138.70369 \r\n",
       "L 157.911596 138.557314 \r\n",
       "L 159.704985 138.551648 \r\n",
       "L 161.498373 138.704108 \r\n",
       "L 163.291761 139.171128 \r\n",
       "L 165.08515 138.966199 \r\n",
       "L 166.878538 138.915465 \r\n",
       "L 168.671927 138.826864 \r\n",
       "L 170.465315 138.620883 \r\n",
       "L 172.258704 138.198332 \r\n",
       "L 174.052092 137.935772 \r\n",
       "L 175.84548 138.803114 \r\n",
       "L 177.638869 139.371249 \r\n",
       "L 179.432257 139.577802 \r\n",
       "L 181.225646 138.950776 \r\n",
       "L 183.019034 138.419533 \r\n",
       "L 184.812423 138.945258 \r\n",
       "L 186.605811 138.960228 \r\n",
       "L 188.399199 139.349392 \r\n",
       "L 190.192588 139.565454 \r\n",
       "L 191.985976 139.747502 \r\n",
       "L 193.779365 139.629315 \r\n",
       "L 195.572753 139.958258 \r\n",
       "L 197.366142 140.070981 \r\n",
       "L 199.15953 139.426754 \r\n",
       "L 200.952918 139.281649 \r\n",
       "L 202.746307 139.266093 \r\n",
       "L 204.539695 139.577772 \r\n",
       "L 206.333084 139.563339 \r\n",
       "L 208.126472 139.761035 \r\n",
       "L 209.919861 138.761785 \r\n",
       "L 211.713249 138.91847 \r\n",
       "L 213.506637 140.12019 \r\n",
       "L 215.300026 140.422727 \r\n",
       "L 217.093414 140.302069 \r\n",
       "L 218.886803 140.069902 \r\n",
       "L 220.680191 139.556561 \r\n",
       "L 222.47358 139.654918 \r\n",
       "L 224.266968 139.465133 \r\n",
       "L 226.060356 139.032148 \r\n",
       "L 227.853745 138.184797 \r\n",
       "L 229.647133 138.382542 \r\n",
       "L 231.440522 138.44864 \r\n",
       "L 233.23391 138.545178 \r\n",
       "L 235.027299 138.77425 \r\n",
       "L 236.820687 138.755151 \r\n",
       "L 238.614075 138.761215 \r\n",
       "L 240.407464 138.945664 \r\n",
       "L 242.200852 139.75007 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#p22d525c269)\" d=\"M 64.655398 16.877273 \r\n",
       "L 66.448786 16.989726 \r\n",
       "L 68.242175 17.019696 \r\n",
       "L 70.035563 17.034238 \r\n",
       "L 71.828951 17.041466 \r\n",
       "L 73.62234 17.046239 \r\n",
       "L 75.415728 17.048125 \r\n",
       "L 77.209117 17.049092 \r\n",
       "L 79.002505 17.04958 \r\n",
       "L 80.795894 17.049817 \r\n",
       "L 82.589282 17.049984 \r\n",
       "L 84.38267 17.050085 \r\n",
       "L 86.176059 17.050131 \r\n",
       "L 87.969447 17.050166 \r\n",
       "L 89.762836 17.050185 \r\n",
       "L 91.556224 17.050197 \r\n",
       "L 93.349613 17.050206 \r\n",
       "L 95.143001 17.050211 \r\n",
       "L 96.936389 17.050214 \r\n",
       "L 98.729778 17.050215 \r\n",
       "L 100.523166 17.050216 \r\n",
       "L 102.316555 17.050216 \r\n",
       "L 104.109943 17.050216 \r\n",
       "L 105.903332 17.050217 \r\n",
       "L 107.69672 17.050216 \r\n",
       "L 109.490108 17.050216 \r\n",
       "L 111.283497 17.050217 \r\n",
       "L 113.076885 17.050217 \r\n",
       "L 114.870274 17.050217 \r\n",
       "L 116.663662 17.050217 \r\n",
       "L 118.457051 17.050217 \r\n",
       "L 120.250439 17.050217 \r\n",
       "L 122.043827 17.050217 \r\n",
       "L 123.837216 17.050217 \r\n",
       "L 125.630604 17.050217 \r\n",
       "L 127.423993 17.050217 \r\n",
       "L 129.217381 17.050217 \r\n",
       "L 131.01077 17.050217 \r\n",
       "L 132.804158 17.050217 \r\n",
       "L 134.597546 17.050217 \r\n",
       "L 136.390935 17.050217 \r\n",
       "L 138.184323 17.050217 \r\n",
       "L 139.977712 17.050217 \r\n",
       "L 141.7711 17.050217 \r\n",
       "L 143.564489 17.050217 \r\n",
       "L 145.357877 17.050217 \r\n",
       "L 147.151265 17.050217 \r\n",
       "L 148.944654 17.050217 \r\n",
       "L 150.738042 17.050217 \r\n",
       "L 152.531431 17.050217 \r\n",
       "L 154.324819 17.050217 \r\n",
       "L 156.118208 17.050217 \r\n",
       "L 157.911596 17.050217 \r\n",
       "L 159.704985 17.050217 \r\n",
       "L 161.498373 17.050217 \r\n",
       "L 163.291761 17.050217 \r\n",
       "L 165.08515 17.050217 \r\n",
       "L 166.878538 17.050217 \r\n",
       "L 168.671927 17.050217 \r\n",
       "L 170.465315 17.050217 \r\n",
       "L 172.258704 17.050217 \r\n",
       "L 174.052092 17.050217 \r\n",
       "L 175.84548 17.050217 \r\n",
       "L 177.638869 17.050217 \r\n",
       "L 179.432257 17.050217 \r\n",
       "L 181.225646 17.050217 \r\n",
       "L 183.019034 17.050217 \r\n",
       "L 184.812423 17.050217 \r\n",
       "L 186.605811 17.050217 \r\n",
       "L 188.399199 17.050217 \r\n",
       "L 190.192588 17.050217 \r\n",
       "L 191.985976 17.050217 \r\n",
       "L 193.779365 17.050217 \r\n",
       "L 195.572753 17.050217 \r\n",
       "L 197.366142 17.050217 \r\n",
       "L 199.15953 17.050217 \r\n",
       "L 200.952918 17.050217 \r\n",
       "L 202.746307 17.050217 \r\n",
       "L 204.539695 17.050217 \r\n",
       "L 206.333084 17.050217 \r\n",
       "L 208.126472 17.050217 \r\n",
       "L 209.919861 17.050217 \r\n",
       "L 211.713249 17.050217 \r\n",
       "L 213.506637 17.050217 \r\n",
       "L 215.300026 17.050217 \r\n",
       "L 217.093414 17.050217 \r\n",
       "L 218.886803 17.050217 \r\n",
       "L 220.680191 17.050217 \r\n",
       "L 222.47358 17.050217 \r\n",
       "L 224.266968 17.050217 \r\n",
       "L 226.060356 17.050217 \r\n",
       "L 227.853745 17.050217 \r\n",
       "L 229.647133 17.050217 \r\n",
       "L 231.440522 17.050217 \r\n",
       "L 233.23391 17.050217 \r\n",
       "L 235.027299 17.050217 \r\n",
       "L 236.820687 17.050217 \r\n",
       "L 238.614075 17.050217 \r\n",
       "L 240.407464 17.050217 \r\n",
       "L 242.200852 17.050217 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 55.778125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 251.078125 146.6 \r\n",
       "L 251.078125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 251.078125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 55.778125 10.7 \r\n",
       "L 251.078125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 188.803125 94.828125 \r\n",
       "L 244.078125 94.828125 \r\n",
       "Q 246.078125 94.828125 246.078125 92.828125 \r\n",
       "L 246.078125 64.471875 \r\n",
       "Q 246.078125 62.471875 244.078125 62.471875 \r\n",
       "L 188.803125 62.471875 \r\n",
       "Q 186.803125 62.471875 186.803125 64.471875 \r\n",
       "L 186.803125 92.828125 \r\n",
       "Q 186.803125 94.828125 188.803125 94.828125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_14\">\r\n",
       "     <path d=\"M 190.803125 70.570312 \r\n",
       "L 210.803125 70.570312 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_15\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(218.803125 74.070312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_16\">\r\n",
       "     <path d=\"M 190.803125 85.248437 \r\n",
       "L 210.803125 85.248437 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_17\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(218.803125 88.748437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p22d525c269\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l2 norm of w: 14.817721366882324\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot(lambd=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Using Weight Decay\n",
    "\n",
    "The example below shows that even though the training error increased,\n",
    "the error on the test set decreased.\n",
    "This is precisely the improvement that we expect from using weight decay.\n",
    "While not perfect, overfitting has been mitigated to some extent.\n",
    "In addition, the $\\ell_2$ norm of the weight $\\mathbf{w}$\n",
    "is smaller than without using weight decay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 257.378125 184.15625\" width=\"257.378125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 257.378125 184.15625 \r\n",
       "L 257.378125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 246.678125 146.6 \r\n",
       "L 246.678125 10.7 \r\n",
       "L 51.378125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m3d6db75334\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(55.280759 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(87.967278 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(123.835046 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(159.702815 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(195.570584 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#m3d6db75334\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(228.257102 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(131.195312 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mba26dfef46\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"125.07027\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-3}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-51\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 128.869488)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"102.436476\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 106.235695)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"79.802683\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 83.601902)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"57.168889\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 60.968108)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"34.535096\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 38.334315)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mba26dfef46\" y=\"11.901303\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 15.700521)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -2 0 \r\n",
       "\" id=\"m865020f87d\" style=\"stroke:#000000;stroke-width:0.6;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"140.890612\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_8\">\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"136.904999\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_9\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"134.077162\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_10\">\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"131.88372\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_11\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"130.091548\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_12\">\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"128.576289\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_13\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"127.263711\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_14\">\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"126.105935\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_15\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"118.256819\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_16\">\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"114.271206\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_17\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"111.443368\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_18\">\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"109.249927\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_19\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"107.457755\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_20\">\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"105.942495\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_21\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"104.629917\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_22\">\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"103.472142\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_23\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"95.623025\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_24\">\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"91.637412\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_25\">\r\n",
       "     <g id=\"line2d_31\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"88.809575\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_26\">\r\n",
       "     <g id=\"line2d_32\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"86.616134\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_27\">\r\n",
       "     <g id=\"line2d_33\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"84.823962\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_28\">\r\n",
       "     <g id=\"line2d_34\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"83.308702\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_29\">\r\n",
       "     <g id=\"line2d_35\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"81.996124\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_30\">\r\n",
       "     <g id=\"line2d_36\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"80.838348\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_31\">\r\n",
       "     <g id=\"line2d_37\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"72.989232\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_32\">\r\n",
       "     <g id=\"line2d_38\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"69.003619\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_33\">\r\n",
       "     <g id=\"line2d_39\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"66.175781\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_34\">\r\n",
       "     <g id=\"line2d_40\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"63.98234\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_35\">\r\n",
       "     <g id=\"line2d_41\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"62.190168\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_36\">\r\n",
       "     <g id=\"line2d_42\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"60.674908\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_37\">\r\n",
       "     <g id=\"line2d_43\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"59.362331\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_38\">\r\n",
       "     <g id=\"line2d_44\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"58.204555\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_39\">\r\n",
       "     <g id=\"line2d_45\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"50.355439\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_40\">\r\n",
       "     <g id=\"line2d_46\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"46.369825\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_41\">\r\n",
       "     <g id=\"line2d_47\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"43.541988\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_42\">\r\n",
       "     <g id=\"line2d_48\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"41.348547\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_43\">\r\n",
       "     <g id=\"line2d_49\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"39.556375\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_44\">\r\n",
       "     <g id=\"line2d_50\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"38.041115\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_45\">\r\n",
       "     <g id=\"line2d_51\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"36.728537\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_46\">\r\n",
       "     <g id=\"line2d_52\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"35.570762\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_47\">\r\n",
       "     <g id=\"line2d_53\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"27.721645\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_48\">\r\n",
       "     <g id=\"line2d_54\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"23.736032\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_49\">\r\n",
       "     <g id=\"line2d_55\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"20.908194\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_50\">\r\n",
       "     <g id=\"line2d_56\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"18.714753\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_51\">\r\n",
       "     <g id=\"line2d_57\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"16.922581\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_52\">\r\n",
       "     <g id=\"line2d_58\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"15.407322\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_53\">\r\n",
       "     <g id=\"line2d_59\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"14.094744\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_54\">\r\n",
       "     <g id=\"line2d_60\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m865020f87d\" y=\"12.936968\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_61\">\r\n",
       "    <path clip-path=\"url(#pa2c4e5f9d6)\" d=\"M 60.255398 35.653267 \r\n",
       "L 62.048786 50.944367 \r\n",
       "L 63.842175 64.893602 \r\n",
       "L 65.635563 78.271524 \r\n",
       "L 67.428951 91.426728 \r\n",
       "L 69.22234 103.927247 \r\n",
       "L 71.015728 114.844492 \r\n",
       "L 72.809117 122.619278 \r\n",
       "L 74.602505 127.701644 \r\n",
       "L 76.395894 128.225636 \r\n",
       "L 78.189282 127.906638 \r\n",
       "L 79.98267 128.552347 \r\n",
       "L 81.776059 130.639235 \r\n",
       "L 83.569447 130.651671 \r\n",
       "L 85.362836 130.603946 \r\n",
       "L 87.156224 129.36019 \r\n",
       "L 88.949613 130.270679 \r\n",
       "L 90.743001 131.614055 \r\n",
       "L 92.536389 129.140601 \r\n",
       "L 94.329778 128.010175 \r\n",
       "L 96.123166 130.429246 \r\n",
       "L 97.916555 131.216089 \r\n",
       "L 99.709943 130.184983 \r\n",
       "L 101.503332 129.836339 \r\n",
       "L 103.29672 132.417514 \r\n",
       "L 105.090108 131.544072 \r\n",
       "L 106.883497 133.024908 \r\n",
       "L 108.676885 133.087047 \r\n",
       "L 110.470274 132.279005 \r\n",
       "L 112.263662 132.118054 \r\n",
       "L 114.057051 130.386704 \r\n",
       "L 115.850439 131.559352 \r\n",
       "L 117.643827 132.284418 \r\n",
       "L 119.437216 133.594013 \r\n",
       "L 121.230604 132.117812 \r\n",
       "L 123.023993 133.231244 \r\n",
       "L 124.817381 133.247632 \r\n",
       "L 126.61077 133.295217 \r\n",
       "L 128.404158 135.037912 \r\n",
       "L 130.197546 132.813386 \r\n",
       "L 131.990935 135.81274 \r\n",
       "L 133.784323 133.982303 \r\n",
       "L 135.577712 134.388257 \r\n",
       "L 137.3711 134.81621 \r\n",
       "L 139.164489 135.294633 \r\n",
       "L 140.957877 135.079545 \r\n",
       "L 142.751265 133.670505 \r\n",
       "L 144.544654 134.899166 \r\n",
       "L 146.338042 134.644511 \r\n",
       "L 148.131431 135.571416 \r\n",
       "L 149.924819 133.499737 \r\n",
       "L 151.718208 135.483956 \r\n",
       "L 153.511596 135.055382 \r\n",
       "L 155.304985 133.558713 \r\n",
       "L 157.098373 134.740148 \r\n",
       "L 158.891761 134.642528 \r\n",
       "L 160.68515 135.904546 \r\n",
       "L 162.478538 134.615512 \r\n",
       "L 164.271927 135.963153 \r\n",
       "L 166.065315 134.320864 \r\n",
       "L 167.858704 136.322542 \r\n",
       "L 169.652092 137.633579 \r\n",
       "L 171.44548 136.102793 \r\n",
       "L 173.238869 134.190863 \r\n",
       "L 175.032257 135.461901 \r\n",
       "L 176.825646 134.667961 \r\n",
       "L 178.619034 137.906081 \r\n",
       "L 180.412423 136.719228 \r\n",
       "L 182.205811 136.656233 \r\n",
       "L 183.999199 138.479853 \r\n",
       "L 185.792588 136.313399 \r\n",
       "L 187.585976 135.388283 \r\n",
       "L 189.379365 135.855524 \r\n",
       "L 191.172753 136.078648 \r\n",
       "L 192.966142 136.627109 \r\n",
       "L 194.75953 136.112453 \r\n",
       "L 196.552918 137.161787 \r\n",
       "L 198.346307 137.983266 \r\n",
       "L 200.139695 138.395723 \r\n",
       "L 201.933084 137.167052 \r\n",
       "L 203.726472 136.160718 \r\n",
       "L 205.519861 137.562766 \r\n",
       "L 207.313249 137.144048 \r\n",
       "L 209.106637 137.492716 \r\n",
       "L 210.900026 138.036412 \r\n",
       "L 212.693414 138.35429 \r\n",
       "L 214.486803 137.778694 \r\n",
       "L 216.280191 136.866886 \r\n",
       "L 218.07358 137.619066 \r\n",
       "L 219.866968 137.714264 \r\n",
       "L 221.660356 136.19453 \r\n",
       "L 223.453745 137.310702 \r\n",
       "L 225.247133 137.111515 \r\n",
       "L 227.040522 135.306314 \r\n",
       "L 228.83391 140.422727 \r\n",
       "L 230.627299 140.244031 \r\n",
       "L 232.420687 138.001165 \r\n",
       "L 234.214075 137.313182 \r\n",
       "L 236.007464 138.139726 \r\n",
       "L 237.800852 137.251221 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_62\">\r\n",
       "    <path clip-path=\"url(#pa2c4e5f9d6)\" d=\"M 60.255398 16.877273 \r\n",
       "L 62.048786 20.384298 \r\n",
       "L 63.842175 23.879492 \r\n",
       "L 65.635563 27.422402 \r\n",
       "L 67.428951 30.957695 \r\n",
       "L 69.22234 34.50839 \r\n",
       "L 71.015728 38.059254 \r\n",
       "L 72.809117 41.616414 \r\n",
       "L 74.602505 45.171442 \r\n",
       "L 76.395894 48.722315 \r\n",
       "L 78.189282 52.262434 \r\n",
       "L 79.98267 55.793678 \r\n",
       "L 81.776059 59.305794 \r\n",
       "L 83.569447 62.789881 \r\n",
       "L 85.362836 66.221185 \r\n",
       "L 87.156224 69.598145 \r\n",
       "L 88.949613 72.85556 \r\n",
       "L 90.743001 75.998889 \r\n",
       "L 92.536389 78.998967 \r\n",
       "L 94.329778 81.771856 \r\n",
       "L 96.123166 84.288128 \r\n",
       "L 97.916555 86.483349 \r\n",
       "L 99.709943 88.359402 \r\n",
       "L 101.503332 89.924225 \r\n",
       "L 103.29672 91.155927 \r\n",
       "L 105.090108 92.160145 \r\n",
       "L 106.883497 92.885563 \r\n",
       "L 108.676885 93.465734 \r\n",
       "L 110.470274 93.89289 \r\n",
       "L 112.263662 94.201707 \r\n",
       "L 114.057051 94.481967 \r\n",
       "L 115.850439 94.691914 \r\n",
       "L 117.643827 94.850216 \r\n",
       "L 119.437216 95.019397 \r\n",
       "L 121.230604 95.12898 \r\n",
       "L 123.023993 95.218387 \r\n",
       "L 124.817381 95.357054 \r\n",
       "L 126.61077 95.447855 \r\n",
       "L 128.404158 95.537109 \r\n",
       "L 130.197546 95.6353 \r\n",
       "L 131.990935 95.714263 \r\n",
       "L 133.784323 95.843967 \r\n",
       "L 135.577712 95.949157 \r\n",
       "L 137.3711 96.005673 \r\n",
       "L 139.164489 96.117965 \r\n",
       "L 140.957877 96.200296 \r\n",
       "L 142.751265 96.272987 \r\n",
       "L 144.544654 96.372869 \r\n",
       "L 146.338042 96.463002 \r\n",
       "L 148.131431 96.528281 \r\n",
       "L 149.924819 96.598062 \r\n",
       "L 151.718208 96.692633 \r\n",
       "L 153.511596 96.781741 \r\n",
       "L 155.304985 96.882213 \r\n",
       "L 157.098373 96.947087 \r\n",
       "L 158.891761 97.002964 \r\n",
       "L 160.68515 97.076735 \r\n",
       "L 162.478538 97.159631 \r\n",
       "L 164.271927 97.21757 \r\n",
       "L 166.065315 97.286779 \r\n",
       "L 167.858704 97.381936 \r\n",
       "L 169.652092 97.407788 \r\n",
       "L 171.44548 97.473165 \r\n",
       "L 173.238869 97.59108 \r\n",
       "L 175.032257 97.648514 \r\n",
       "L 176.825646 97.696511 \r\n",
       "L 178.619034 97.753814 \r\n",
       "L 180.412423 97.84241 \r\n",
       "L 182.205811 97.881954 \r\n",
       "L 183.999199 97.923201 \r\n",
       "L 185.792588 98.008888 \r\n",
       "L 187.585976 98.046403 \r\n",
       "L 189.379365 98.111586 \r\n",
       "L 191.172753 98.179996 \r\n",
       "L 192.966142 98.215053 \r\n",
       "L 194.75953 98.256508 \r\n",
       "L 196.552918 98.311652 \r\n",
       "L 198.346307 98.363641 \r\n",
       "L 200.139695 98.413744 \r\n",
       "L 201.933084 98.454149 \r\n",
       "L 203.726472 98.510546 \r\n",
       "L 205.519861 98.557111 \r\n",
       "L 207.313249 98.597413 \r\n",
       "L 209.106637 98.616404 \r\n",
       "L 210.900026 98.691489 \r\n",
       "L 212.693414 98.735223 \r\n",
       "L 214.486803 98.748308 \r\n",
       "L 216.280191 98.824847 \r\n",
       "L 218.07358 98.830954 \r\n",
       "L 219.866968 98.872948 \r\n",
       "L 221.660356 98.939887 \r\n",
       "L 223.453745 98.967021 \r\n",
       "L 225.247133 98.993022 \r\n",
       "L 227.040522 99.053413 \r\n",
       "L 228.83391 99.022308 \r\n",
       "L 230.627299 99.080286 \r\n",
       "L 232.420687 99.111412 \r\n",
       "L 234.214075 99.143509 \r\n",
       "L 236.007464 99.193448 \r\n",
       "L 237.800852 99.205401 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 51.378125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 146.6 \r\n",
       "L 246.678125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 246.678125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 10.7 \r\n",
       "L 246.678125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 48.05625 \r\n",
       "L 239.678125 48.05625 \r\n",
       "Q 241.678125 48.05625 241.678125 46.05625 \r\n",
       "L 241.678125 17.7 \r\n",
       "Q 241.678125 15.7 239.678125 15.7 \r\n",
       "L 184.403125 15.7 \r\n",
       "Q 182.403125 15.7 182.403125 17.7 \r\n",
       "L 182.403125 46.05625 \r\n",
       "Q 182.403125 48.05625 184.403125 48.05625 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_63\">\r\n",
       "     <path d=\"M 186.403125 23.798437 \r\n",
       "L 206.403125 23.798437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_64\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(214.403125 27.298437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_65\">\r\n",
       "     <path d=\"M 186.403125 38.476562 \r\n",
       "L 206.403125 38.476562 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_66\"/>\r\n",
       "    <g id=\"text_16\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 41.976562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pa2c4e5f9d6\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l2 norm of w: 0.031043482944369316\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot(lambd=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concise Implementation\n",
    "\n",
    "Because weight decay is ubiquitous in neural network optimization,\n",
    "Pytorch makes it especially convenient,\n",
    "integrating weight decay into the optimization algorithm itself\n",
    "for easy use in combination with any loss function.\n",
    "Moreover, this integration serves a computational benefit,\n",
    "allowing implementation tricks to add weight decay to the algorithm,\n",
    "without any additional computational overhead.\n",
    "Since the weight decay portion of the update\n",
    "depdends only on the current value of each parameter,\n",
    "and the optimizer must to touch each parameter once anyway.\n",
    "\n",
    "In the following code, we specify\n",
    "the weight decay hyper-parameter directly\n",
    "through the `weight_decay` parameter when instantiating our `Optimizer`.\n",
    "By default, Pytorch decays both weights and biases simultaneously.\n",
    "Per-Parameter or Parameters with different lr,weight_decay can be specified \n",
    "in the optimizer function using an iterable of `dict`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_and_plot_pytorch(wd):\n",
    "    net = nn.Sequential(nn.Linear(200, 1))\n",
    "    #Random Uniform Initialization of the Model parameters\n",
    "    for param in net.parameters():\n",
    "        param.data.uniform_()\n",
    "    \n",
    "    loss = nn.MSELoss()\n",
    "    # The weight parameter has been decayed with its weight_decay set as wd\n",
    "    # The weight_decay for bias is by default zero (no decay)\n",
    "    optimizer = torch.optim.SGD([\n",
    "        {\"params\":net[0].weight,'weight_decay': wd},\n",
    "        {\"params\":net[0].bias}\n",
    "    ],lr=lr)\n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            with torch.enable_grad():\n",
    "                optimizer.zero_grad()\n",
    "                l = loss(net(X), y)\n",
    "            l.backward()\n",
    "            # Call the step function to\n",
    "            # update the weight and bias\n",
    "            optimizer.step()\n",
    "        train_ls.append(torch.mean(loss(net(train_features),\n",
    "                             train_labels)).item())\n",
    "        test_ls.append(torch.mean(loss(net(test_features),\n",
    "                            test_labels)).item())\n",
    "    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('L2 norm of w:', net[0].weight.norm().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plots look just the same as when we implemented weight decay from scratch\n",
    "but they run a bit faster and are easier to implement,\n",
    "a benefit that will become more pronounced for large problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 261.778125 184.15625\" width=\"261.778125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 261.778125 184.15625 \r\n",
       "L 261.778125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 251.078125 146.6 \r\n",
       "L 251.078125 10.7 \r\n",
       "L 55.778125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m212b3d8e31\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(59.680759 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.367278 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.235046 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(164.102815 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(199.970584 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#m212b3d8e31\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(232.657102 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(135.595313 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m8334cebf43\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m8334cebf43\" y=\"129.337389\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-12}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 133.136607)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(231.391602 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m8334cebf43\" y=\"104.977225\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-9}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.984375 1.515625 \r\n",
       "L 10.984375 10.5 \r\n",
       "Q 14.703125 8.734375 18.5 7.8125 \r\n",
       "Q 22.3125 6.890625 25.984375 6.890625 \r\n",
       "Q 35.75 6.890625 40.890625 13.453125 \r\n",
       "Q 46.046875 20.015625 46.78125 33.40625 \r\n",
       "Q 43.953125 29.203125 39.59375 26.953125 \r\n",
       "Q 35.25 24.703125 29.984375 24.703125 \r\n",
       "Q 19.046875 24.703125 12.671875 31.3125 \r\n",
       "Q 6.296875 37.9375 6.296875 49.421875 \r\n",
       "Q 6.296875 60.640625 12.9375 67.421875 \r\n",
       "Q 19.578125 74.21875 30.609375 74.21875 \r\n",
       "Q 43.265625 74.21875 49.921875 64.515625 \r\n",
       "Q 56.59375 54.828125 56.59375 36.375 \r\n",
       "Q 56.59375 19.140625 48.40625 8.859375 \r\n",
       "Q 40.234375 -1.421875 26.421875 -1.421875 \r\n",
       "Q 22.703125 -1.421875 18.890625 -0.6875 \r\n",
       "Q 15.09375 0.046875 10.984375 1.515625 \r\n",
       "z\r\n",
       "M 30.609375 32.421875 \r\n",
       "Q 37.25 32.421875 41.125 36.953125 \r\n",
       "Q 45.015625 41.5 45.015625 49.421875 \r\n",
       "Q 45.015625 57.28125 41.125 61.84375 \r\n",
       "Q 37.25 66.40625 30.609375 66.40625 \r\n",
       "Q 23.96875 66.40625 20.09375 61.84375 \r\n",
       "Q 16.21875 57.28125 16.21875 49.421875 \r\n",
       "Q 16.21875 41.5 20.09375 36.953125 \r\n",
       "Q 23.96875 32.421875 30.609375 32.421875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-57\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(25.278125 108.776443)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-57\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m8334cebf43\" y=\"80.617061\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-6}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 84.416279)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m8334cebf43\" y=\"56.256897\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{-3}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-51\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(25.278125 60.056115)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m8334cebf43\" y=\"31.896733\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(31.178125 35.695951)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#pf4fb47be37)\" d=\"M 64.655398 27.502922 \r\n",
       "L 66.448786 33.440861 \r\n",
       "L 68.242175 40.280269 \r\n",
       "L 70.035563 47.022634 \r\n",
       "L 71.828951 53.642364 \r\n",
       "L 73.62234 59.309939 \r\n",
       "L 75.415728 65.622829 \r\n",
       "L 77.209117 72.373453 \r\n",
       "L 79.002505 79.355075 \r\n",
       "L 80.795894 87.875911 \r\n",
       "L 82.589282 92.975612 \r\n",
       "L 84.38267 101.127098 \r\n",
       "L 86.176059 109.855722 \r\n",
       "L 87.969447 118.460909 \r\n",
       "L 89.762836 124.066922 \r\n",
       "L 91.556224 129.728644 \r\n",
       "L 93.349613 133.983323 \r\n",
       "L 95.143001 135.601204 \r\n",
       "L 96.936389 134.298748 \r\n",
       "L 98.729778 136.034793 \r\n",
       "L 100.523166 134.263752 \r\n",
       "L 102.316555 134.166493 \r\n",
       "L 104.109943 134.545716 \r\n",
       "L 105.903332 135.241981 \r\n",
       "L 107.69672 132.057811 \r\n",
       "L 109.490108 136.081924 \r\n",
       "L 111.283497 136.295547 \r\n",
       "L 113.076885 136.306548 \r\n",
       "L 114.870274 135.444457 \r\n",
       "L 116.663662 137.789335 \r\n",
       "L 118.457051 137.920803 \r\n",
       "L 120.250439 134.560149 \r\n",
       "L 122.043827 136.321552 \r\n",
       "L 123.837216 136.110668 \r\n",
       "L 125.630604 136.379331 \r\n",
       "L 127.423993 136.822155 \r\n",
       "L 129.217381 137.278826 \r\n",
       "L 131.01077 139.027875 \r\n",
       "L 132.804158 135.613948 \r\n",
       "L 134.597546 135.93907 \r\n",
       "L 136.390935 139.32816 \r\n",
       "L 138.184323 137.173234 \r\n",
       "L 139.977712 137.280315 \r\n",
       "L 141.7711 135.862276 \r\n",
       "L 143.564489 135.917499 \r\n",
       "L 145.357877 133.591481 \r\n",
       "L 147.151265 132.034265 \r\n",
       "L 148.944654 133.957687 \r\n",
       "L 150.738042 136.525055 \r\n",
       "L 152.531431 136.992641 \r\n",
       "L 154.324819 137.164576 \r\n",
       "L 156.118208 135.86377 \r\n",
       "L 157.911596 137.955802 \r\n",
       "L 159.704985 135.47646 \r\n",
       "L 161.498373 133.748363 \r\n",
       "L 163.291761 135.225517 \r\n",
       "L 165.08515 138.083449 \r\n",
       "L 166.878538 137.836101 \r\n",
       "L 168.671927 138.388502 \r\n",
       "L 170.465315 138.084852 \r\n",
       "L 172.258704 137.848327 \r\n",
       "L 174.052092 136.735915 \r\n",
       "L 175.84548 136.134263 \r\n",
       "L 177.638869 136.939846 \r\n",
       "L 179.432257 135.319916 \r\n",
       "L 181.225646 136.730495 \r\n",
       "L 183.019034 135.765002 \r\n",
       "L 184.812423 138.739881 \r\n",
       "L 186.605811 136.601928 \r\n",
       "L 188.399199 137.152347 \r\n",
       "L 190.192588 138.515145 \r\n",
       "L 191.985976 139.260926 \r\n",
       "L 193.779365 137.593151 \r\n",
       "L 195.572753 137.230772 \r\n",
       "L 197.366142 137.35368 \r\n",
       "L 199.15953 135.937288 \r\n",
       "L 200.952918 135.372196 \r\n",
       "L 202.746307 135.635836 \r\n",
       "L 204.539695 138.250037 \r\n",
       "L 206.333084 138.864271 \r\n",
       "L 208.126472 138.198456 \r\n",
       "L 209.919861 138.892977 \r\n",
       "L 211.713249 139.227137 \r\n",
       "L 213.506637 139.038279 \r\n",
       "L 215.300026 138.688996 \r\n",
       "L 217.093414 138.257397 \r\n",
       "L 218.886803 136.342324 \r\n",
       "L 220.680191 135.984672 \r\n",
       "L 222.47358 138.442961 \r\n",
       "L 224.266968 136.586296 \r\n",
       "L 226.060356 138.572685 \r\n",
       "L 227.853745 136.61732 \r\n",
       "L 229.647133 135.755565 \r\n",
       "L 231.440522 134.408342 \r\n",
       "L 233.23391 136.058674 \r\n",
       "L 235.027299 136.807146 \r\n",
       "L 236.820687 137.121957 \r\n",
       "L 238.614075 140.422727 \r\n",
       "L 240.407464 139.739933 \r\n",
       "L 242.200852 139.306929 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#pf4fb47be37)\" d=\"M 64.655398 16.877273 \r\n",
       "L 66.448786 17.000286 \r\n",
       "L 68.242175 16.959185 \r\n",
       "L 70.035563 16.973451 \r\n",
       "L 71.828951 16.968278 \r\n",
       "L 73.62234 16.969971 \r\n",
       "L 75.415728 16.969452 \r\n",
       "L 77.209117 16.969612 \r\n",
       "L 79.002505 16.969634 \r\n",
       "L 80.795894 16.969594 \r\n",
       "L 82.589282 16.969605 \r\n",
       "L 84.38267 16.969604 \r\n",
       "L 86.176059 16.969605 \r\n",
       "L 87.969447 16.969605 \r\n",
       "L 89.762836 16.969605 \r\n",
       "L 91.556224 16.969605 \r\n",
       "L 93.349613 16.969605 \r\n",
       "L 95.143001 16.969605 \r\n",
       "L 96.936389 16.969605 \r\n",
       "L 98.729778 16.969605 \r\n",
       "L 100.523166 16.969605 \r\n",
       "L 102.316555 16.969605 \r\n",
       "L 104.109943 16.969605 \r\n",
       "L 105.903332 16.969605 \r\n",
       "L 107.69672 16.969605 \r\n",
       "L 109.490108 16.969605 \r\n",
       "L 111.283497 16.969605 \r\n",
       "L 113.076885 16.969605 \r\n",
       "L 114.870274 16.969605 \r\n",
       "L 116.663662 16.969605 \r\n",
       "L 118.457051 16.969605 \r\n",
       "L 120.250439 16.969605 \r\n",
       "L 122.043827 16.969605 \r\n",
       "L 123.837216 16.969605 \r\n",
       "L 125.630604 16.969605 \r\n",
       "L 127.423993 16.969605 \r\n",
       "L 129.217381 16.969605 \r\n",
       "L 131.01077 16.969605 \r\n",
       "L 132.804158 16.969605 \r\n",
       "L 134.597546 16.969605 \r\n",
       "L 136.390935 16.969605 \r\n",
       "L 138.184323 16.969605 \r\n",
       "L 139.977712 16.969605 \r\n",
       "L 141.7711 16.969605 \r\n",
       "L 143.564489 16.969605 \r\n",
       "L 145.357877 16.969605 \r\n",
       "L 147.151265 16.969605 \r\n",
       "L 148.944654 16.969605 \r\n",
       "L 150.738042 16.969605 \r\n",
       "L 152.531431 16.969605 \r\n",
       "L 154.324819 16.969605 \r\n",
       "L 156.118208 16.969605 \r\n",
       "L 157.911596 16.969605 \r\n",
       "L 159.704985 16.969605 \r\n",
       "L 161.498373 16.969605 \r\n",
       "L 163.291761 16.969605 \r\n",
       "L 165.08515 16.969605 \r\n",
       "L 166.878538 16.969605 \r\n",
       "L 168.671927 16.969605 \r\n",
       "L 170.465315 16.969605 \r\n",
       "L 172.258704 16.969605 \r\n",
       "L 174.052092 16.969605 \r\n",
       "L 175.84548 16.969605 \r\n",
       "L 177.638869 16.969605 \r\n",
       "L 179.432257 16.969605 \r\n",
       "L 181.225646 16.969605 \r\n",
       "L 183.019034 16.969605 \r\n",
       "L 184.812423 16.969605 \r\n",
       "L 186.605811 16.969605 \r\n",
       "L 188.399199 16.969605 \r\n",
       "L 190.192588 16.969605 \r\n",
       "L 191.985976 16.969605 \r\n",
       "L 193.779365 16.969605 \r\n",
       "L 195.572753 16.969605 \r\n",
       "L 197.366142 16.969605 \r\n",
       "L 199.15953 16.969605 \r\n",
       "L 200.952918 16.969605 \r\n",
       "L 202.746307 16.969605 \r\n",
       "L 204.539695 16.969605 \r\n",
       "L 206.333084 16.969605 \r\n",
       "L 208.126472 16.969605 \r\n",
       "L 209.919861 16.969605 \r\n",
       "L 211.713249 16.969605 \r\n",
       "L 213.506637 16.969605 \r\n",
       "L 215.300026 16.969605 \r\n",
       "L 217.093414 16.969605 \r\n",
       "L 218.886803 16.969605 \r\n",
       "L 220.680191 16.969605 \r\n",
       "L 222.47358 16.969605 \r\n",
       "L 224.266968 16.969605 \r\n",
       "L 226.060356 16.969605 \r\n",
       "L 227.853745 16.969605 \r\n",
       "L 229.647133 16.969605 \r\n",
       "L 231.440522 16.969605 \r\n",
       "L 233.23391 16.969605 \r\n",
       "L 235.027299 16.969605 \r\n",
       "L 236.820687 16.969605 \r\n",
       "L 238.614075 16.969605 \r\n",
       "L 240.407464 16.969605 \r\n",
       "L 242.200852 16.969605 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 55.778125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 251.078125 146.6 \r\n",
       "L 251.078125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 55.778125 146.6 \r\n",
       "L 251.078125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 55.778125 10.7 \r\n",
       "L 251.078125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 188.803125 94.828125 \r\n",
       "L 244.078125 94.828125 \r\n",
       "Q 246.078125 94.828125 246.078125 92.828125 \r\n",
       "L 246.078125 64.471875 \r\n",
       "Q 246.078125 62.471875 244.078125 62.471875 \r\n",
       "L 188.803125 62.471875 \r\n",
       "Q 186.803125 62.471875 186.803125 64.471875 \r\n",
       "L 186.803125 92.828125 \r\n",
       "Q 186.803125 94.828125 188.803125 94.828125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_14\">\r\n",
       "     <path d=\"M 190.803125 70.570312 \r\n",
       "L 210.803125 70.570312 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_15\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(218.803125 74.070312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_16\">\r\n",
       "     <path d=\"M 190.803125 85.248437 \r\n",
       "L 210.803125 85.248437 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_17\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(218.803125 88.748437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pf4fb47be37\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 7.739922523498535\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_pytorch(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 257.378125 184.15625\" width=\"257.378125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 257.378125 184.15625 \r\n",
       "L 257.378125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 246.678125 146.6 \r\n",
       "L 246.678125 10.7 \r\n",
       "L 51.378125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"me9945cf851\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(55.280759 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(87.967278 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(123.835046 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(159.702815 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(195.570584 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#me9945cf851\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(228.257102 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(131.195312 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m3f113c4cb4\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"143.381195\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-4}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 147.180414)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"120.825859\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-3}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-51\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 124.625077)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"98.270522\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 102.069741)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"75.715185\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 79.514404)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"53.159849\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 56.959067)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3f113c4cb4\" y=\"30.604512\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 34.403731)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -2 0 \r\n",
       "\" id=\"mf753930439\" style=\"stroke:#000000;stroke-width:0.6;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"146.875061\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_8\">\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"145.567033\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_9\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"144.413271\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_10\">\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"136.591362\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_11\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"132.619565\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_12\">\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"129.801529\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_13\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"127.615692\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_14\">\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"125.829732\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_15\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"124.319724\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_16\">\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"123.011697\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_17\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"121.857934\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_18\">\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"114.036026\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_19\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"110.064228\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_20\">\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"107.246193\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_21\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"105.060355\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_22\">\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"103.274395\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_23\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"101.764388\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_24\">\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"100.45636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_25\">\r\n",
       "     <g id=\"line2d_31\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"99.302598\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_26\">\r\n",
       "     <g id=\"line2d_32\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"91.480689\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_27\">\r\n",
       "     <g id=\"line2d_33\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"87.508891\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_28\">\r\n",
       "     <g id=\"line2d_34\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"84.690856\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_29\">\r\n",
       "     <g id=\"line2d_35\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"82.505018\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_30\">\r\n",
       "     <g id=\"line2d_36\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"80.719059\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_31\">\r\n",
       "     <g id=\"line2d_37\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"79.209051\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_32\">\r\n",
       "     <g id=\"line2d_38\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"77.901023\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_33\">\r\n",
       "     <g id=\"line2d_39\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"76.747261\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_34\">\r\n",
       "     <g id=\"line2d_40\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"68.925352\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_35\">\r\n",
       "     <g id=\"line2d_41\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"64.953555\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_36\">\r\n",
       "     <g id=\"line2d_42\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"62.13552\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_37\">\r\n",
       "     <g id=\"line2d_43\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"59.949682\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_38\">\r\n",
       "     <g id=\"line2d_44\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"58.163722\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_39\">\r\n",
       "     <g id=\"line2d_45\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"56.653715\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_40\">\r\n",
       "     <g id=\"line2d_46\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"55.345687\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_41\">\r\n",
       "     <g id=\"line2d_47\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"54.191924\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_42\">\r\n",
       "     <g id=\"line2d_48\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"46.370016\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_43\">\r\n",
       "     <g id=\"line2d_49\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"42.398218\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_44\">\r\n",
       "     <g id=\"line2d_50\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"39.580183\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_45\">\r\n",
       "     <g id=\"line2d_51\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"37.394345\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_46\">\r\n",
       "     <g id=\"line2d_52\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"35.608385\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_47\">\r\n",
       "     <g id=\"line2d_53\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"34.098378\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_48\">\r\n",
       "     <g id=\"line2d_54\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"32.79035\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_49\">\r\n",
       "     <g id=\"line2d_55\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"31.636588\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_50\">\r\n",
       "     <g id=\"line2d_56\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"23.814679\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_51\">\r\n",
       "     <g id=\"line2d_57\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"19.842882\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_52\">\r\n",
       "     <g id=\"line2d_58\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"17.024846\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_53\">\r\n",
       "     <g id=\"line2d_59\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"14.839008\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_54\">\r\n",
       "     <g id=\"line2d_60\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"13.053049\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_55\">\r\n",
       "     <g id=\"line2d_61\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"11.543041\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_56\">\r\n",
       "     <g id=\"line2d_62\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mf753930439\" y=\"10.235013\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_63\">\r\n",
       "    <path clip-path=\"url(#p72999dce6d)\" d=\"M 60.255398 47.584164 \r\n",
       "L 62.048786 70.860388 \r\n",
       "L 63.842175 92.958205 \r\n",
       "L 65.635563 106.918433 \r\n",
       "L 67.428951 115.966446 \r\n",
       "L 69.22234 116.782039 \r\n",
       "L 71.015728 115.983624 \r\n",
       "L 72.809117 116.662792 \r\n",
       "L 74.602505 113.703189 \r\n",
       "L 76.395894 116.089854 \r\n",
       "L 78.189282 117.208997 \r\n",
       "L 79.98267 118.820464 \r\n",
       "L 81.776059 120.278553 \r\n",
       "L 83.569447 119.99516 \r\n",
       "L 85.362836 120.394649 \r\n",
       "L 87.156224 119.331133 \r\n",
       "L 88.949613 119.337436 \r\n",
       "L 90.743001 116.282245 \r\n",
       "L 92.536389 119.069907 \r\n",
       "L 94.329778 118.115554 \r\n",
       "L 96.123166 122.647419 \r\n",
       "L 97.916555 121.545843 \r\n",
       "L 99.709943 122.045448 \r\n",
       "L 101.503332 122.595258 \r\n",
       "L 103.29672 120.902769 \r\n",
       "L 105.090108 123.594112 \r\n",
       "L 106.883497 120.921906 \r\n",
       "L 108.676885 122.061173 \r\n",
       "L 110.470274 121.525848 \r\n",
       "L 112.263662 120.862411 \r\n",
       "L 114.057051 120.002104 \r\n",
       "L 115.850439 127.606315 \r\n",
       "L 117.643827 123.025663 \r\n",
       "L 119.437216 126.365779 \r\n",
       "L 121.230604 122.305243 \r\n",
       "L 123.023993 122.784972 \r\n",
       "L 124.817381 124.276264 \r\n",
       "L 126.61077 121.504671 \r\n",
       "L 128.404158 127.441311 \r\n",
       "L 130.197546 123.636545 \r\n",
       "L 131.990935 128.576142 \r\n",
       "L 133.784323 123.90095 \r\n",
       "L 135.577712 127.125945 \r\n",
       "L 137.3711 125.504792 \r\n",
       "L 139.164489 126.198074 \r\n",
       "L 140.957877 129.360427 \r\n",
       "L 142.751265 125.050497 \r\n",
       "L 144.544654 129.271511 \r\n",
       "L 146.338042 129.182692 \r\n",
       "L 148.131431 128.942682 \r\n",
       "L 149.924819 130.454564 \r\n",
       "L 151.718208 125.940013 \r\n",
       "L 153.511596 132.180284 \r\n",
       "L 155.304985 127.473868 \r\n",
       "L 157.098373 130.591676 \r\n",
       "L 158.891761 130.558285 \r\n",
       "L 160.68515 128.802342 \r\n",
       "L 162.478538 128.635576 \r\n",
       "L 164.271927 129.812493 \r\n",
       "L 166.065315 126.49616 \r\n",
       "L 167.858704 132.5452 \r\n",
       "L 169.652092 126.125656 \r\n",
       "L 171.44548 131.092279 \r\n",
       "L 173.238869 128.880233 \r\n",
       "L 175.032257 129.870568 \r\n",
       "L 176.825646 133.198071 \r\n",
       "L 178.619034 131.187342 \r\n",
       "L 180.412423 131.26806 \r\n",
       "L 182.205811 133.346131 \r\n",
       "L 183.999199 132.838191 \r\n",
       "L 185.792588 133.183007 \r\n",
       "L 187.585976 134.415414 \r\n",
       "L 189.379365 132.36638 \r\n",
       "L 191.172753 135.731453 \r\n",
       "L 192.966142 131.205195 \r\n",
       "L 194.75953 132.454407 \r\n",
       "L 196.552918 133.141524 \r\n",
       "L 198.346307 136.299881 \r\n",
       "L 200.139695 132.37436 \r\n",
       "L 201.933084 130.449698 \r\n",
       "L 203.726472 131.179709 \r\n",
       "L 205.519861 136.338493 \r\n",
       "L 207.313249 132.970296 \r\n",
       "L 209.106637 136.92862 \r\n",
       "L 210.900026 133.241203 \r\n",
       "L 212.693414 132.157372 \r\n",
       "L 214.486803 133.835214 \r\n",
       "L 216.280191 137.958004 \r\n",
       "L 218.07358 134.799548 \r\n",
       "L 219.866968 139.450878 \r\n",
       "L 221.660356 140.422727 \r\n",
       "L 223.453745 130.927755 \r\n",
       "L 225.247133 138.113545 \r\n",
       "L 227.040522 137.683865 \r\n",
       "L 228.83391 138.11358 \r\n",
       "L 230.627299 136.690467 \r\n",
       "L 232.420687 137.052558 \r\n",
       "L 234.214075 136.692737 \r\n",
       "L 236.007464 135.076535 \r\n",
       "L 237.800852 135.386261 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_64\">\r\n",
       "    <path clip-path=\"url(#p72999dce6d)\" d=\"M 60.255398 16.877273 \r\n",
       "L 62.048786 20.360338 \r\n",
       "L 63.842175 24.009522 \r\n",
       "L 65.635563 27.63192 \r\n",
       "L 67.428951 31.269881 \r\n",
       "L 69.22234 34.894627 \r\n",
       "L 71.015728 38.547621 \r\n",
       "L 72.809117 42.174654 \r\n",
       "L 74.602505 45.787347 \r\n",
       "L 76.395894 49.371407 \r\n",
       "L 78.189282 52.84937 \r\n",
       "L 79.98267 56.292404 \r\n",
       "L 81.776059 59.497806 \r\n",
       "L 83.569447 62.473284 \r\n",
       "L 85.362836 65.083799 \r\n",
       "L 87.156224 67.387935 \r\n",
       "L 88.949613 69.161917 \r\n",
       "L 90.743001 70.699767 \r\n",
       "L 92.536389 71.719842 \r\n",
       "L 94.329778 72.484086 \r\n",
       "L 96.123166 72.97503 \r\n",
       "L 97.916555 73.215413 \r\n",
       "L 99.709943 73.533194 \r\n",
       "L 101.503332 73.673734 \r\n",
       "L 103.29672 73.833069 \r\n",
       "L 105.090108 73.87301 \r\n",
       "L 106.883497 74.059614 \r\n",
       "L 108.676885 74.085175 \r\n",
       "L 110.470274 74.253674 \r\n",
       "L 112.263662 74.4104 \r\n",
       "L 114.057051 74.561357 \r\n",
       "L 115.850439 74.659881 \r\n",
       "L 117.643827 74.785619 \r\n",
       "L 119.437216 74.983683 \r\n",
       "L 121.230604 75.154654 \r\n",
       "L 123.023993 75.226492 \r\n",
       "L 124.817381 75.55395 \r\n",
       "L 126.61077 75.714228 \r\n",
       "L 128.404158 75.777091 \r\n",
       "L 130.197546 75.992129 \r\n",
       "L 131.990935 76.1682 \r\n",
       "L 133.784323 76.370048 \r\n",
       "L 135.577712 76.522907 \r\n",
       "L 137.3711 76.769875 \r\n",
       "L 139.164489 76.942662 \r\n",
       "L 140.957877 77.107982 \r\n",
       "L 142.751265 77.343023 \r\n",
       "L 144.544654 77.542191 \r\n",
       "L 146.338042 77.669645 \r\n",
       "L 148.131431 77.982053 \r\n",
       "L 149.924819 78.089915 \r\n",
       "L 151.718208 78.304694 \r\n",
       "L 153.511596 78.493596 \r\n",
       "L 155.304985 78.605961 \r\n",
       "L 157.098373 78.814321 \r\n",
       "L 158.891761 79.00735 \r\n",
       "L 160.68515 79.268625 \r\n",
       "L 162.478538 79.30893 \r\n",
       "L 164.271927 79.564297 \r\n",
       "L 166.065315 79.705287 \r\n",
       "L 167.858704 79.911118 \r\n",
       "L 169.652092 80.204831 \r\n",
       "L 171.44548 80.241427 \r\n",
       "L 173.238869 80.411926 \r\n",
       "L 175.032257 80.582911 \r\n",
       "L 176.825646 80.794323 \r\n",
       "L 178.619034 80.90788 \r\n",
       "L 180.412423 81.116372 \r\n",
       "L 182.205811 81.229533 \r\n",
       "L 183.999199 81.394033 \r\n",
       "L 185.792588 81.617901 \r\n",
       "L 187.585976 81.676498 \r\n",
       "L 189.379365 81.915697 \r\n",
       "L 191.172753 82.037841 \r\n",
       "L 192.966142 82.145513 \r\n",
       "L 194.75953 82.323637 \r\n",
       "L 196.552918 82.420659 \r\n",
       "L 198.346307 82.529038 \r\n",
       "L 200.139695 82.72384 \r\n",
       "L 201.933084 82.887685 \r\n",
       "L 203.726472 82.984832 \r\n",
       "L 205.519861 83.089752 \r\n",
       "L 207.313249 83.285491 \r\n",
       "L 209.106637 83.376727 \r\n",
       "L 210.900026 83.5132 \r\n",
       "L 212.693414 83.647654 \r\n",
       "L 214.486803 83.825971 \r\n",
       "L 216.280191 83.860584 \r\n",
       "L 218.07358 83.942596 \r\n",
       "L 219.866968 84.014751 \r\n",
       "L 221.660356 84.155759 \r\n",
       "L 223.453745 84.321686 \r\n",
       "L 225.247133 84.400002 \r\n",
       "L 227.040522 84.494128 \r\n",
       "L 228.83391 84.6727 \r\n",
       "L 230.627299 84.750715 \r\n",
       "L 232.420687 84.868401 \r\n",
       "L 234.214075 84.997061 \r\n",
       "L 236.007464 85.09853 \r\n",
       "L 237.800852 85.186064 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 51.378125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 146.6 \r\n",
       "L 246.678125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 146.6 \r\n",
       "L 246.678125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 10.7 \r\n",
       "L 246.678125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 48.05625 \r\n",
       "L 239.678125 48.05625 \r\n",
       "Q 241.678125 48.05625 241.678125 46.05625 \r\n",
       "L 241.678125 17.7 \r\n",
       "Q 241.678125 15.7 239.678125 15.7 \r\n",
       "L 184.403125 15.7 \r\n",
       "Q 182.403125 15.7 182.403125 17.7 \r\n",
       "L 182.403125 46.05625 \r\n",
       "Q 182.403125 48.05625 184.403125 48.05625 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_65\">\r\n",
       "     <path d=\"M 186.403125 23.798437 \r\n",
       "L 206.403125 23.798437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_66\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(214.403125 27.298437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_67\">\r\n",
       "     <path d=\"M 186.403125 38.476562 \r\n",
       "L 206.403125 38.476562 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_68\"/>\r\n",
       "    <g id=\"text_16\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 41.976562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p72999dce6d\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.044153060764074326\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_pytorch(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we only touched upon one notion of\n",
    "what constitutes a simple *linear* function.\n",
    "For nonlinear functions, what constitutes *simplicity*\n",
    "can be a far more complex question.\n",
    "For instance, there exist [Reproducing Kernel Hilbert Spaces (RKHS)](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space)\n",
    "which allow one to use many of the tools\n",
    "introduced for linear functions in a nonlinear context.\n",
    "Unfortunately, RKHS-based algorithms\n",
    "do not always scale well to massive amounts of data.\n",
    "For the purposes of this book, we limit ourselves\n",
    "to simply summing over the weights for different layers,\n",
    "e.g. via $\\sum_l \\|\\mathbf{w}_l\\|^2$,\n",
    "which is equivalent to weight decay applied to all layers.\n",
    "\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Regularization is a common method for dealing with overfitting. It adds a penalty term to the loss function on the training set to reduce the complexity of the learned model.\n",
    "* One particular choice for keeping the model simple is weight decay using an $\\ell_2$ penalty. This leads to weight decay in the update steps of the learning algorithm.\n",
    "* Pytorch provides automatic weight decay functionality in the optimizer by setting the hyperparameter `weight_decay`.\n",
    "* You can have different lr,Weight_decay for different sets of parameters in an optimizer. They need to be passed as dicts.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Experiment with the value of $\\lambda$ in the estimation problem in this page. Plot training and test accuracy as a function of $\\lambda$. What do you observe?\n",
    "1. Use a validation set to find the optimal value of $\\lambda$. Is it really the optimal value? Does this matter?\n",
    "1. What would the update equations look like if instead of $\\|\\mathbf{w}\\|^2$ we used $\\sum_i |w_i|$ as our penalty of choice (this is called $\\ell_1$ regularization).\n",
    "1. We know that $\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$. Can you find a similar equation for matrices (mathematicians call this the [Frobenius norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm))?\n",
    "1. Review the relationship between training error and generalization error. In addition to weight decay, increased training, and the use of a model of suitable complexity, what other ways can you think of to deal with overfitting?\n",
    "1. In Bayesian statistics we use the product of prior and likelihood to arrive at a posterior via $p(w|x) \\propto p(x|w) p(w)$. How can you identify $p(w)$ with regularization?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}