works/cPSE/periodic_spectral_ergodicity_dnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "# Periodic Spectral Ergodicity: A measure for Neural Architecture Search\n",
    "\n",
    "(c) 2019   \n",
    "    Developed by  \n",
    "    Mehmet Süzen        \n",
    "    suzen at acm dot org  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Sketch\n",
    "\n",
    "* Introduce a measure for spectral ergodicity on the set of different length square matrices.  \n",
    "  This is called `periodic spectral ergodicity (PSE)`.\n",
    "* Report `periodic spectral ergodicity (PSE)` measure for pre-trained networks.\n",
    "    * ResNet/VGG variants with top-1/top-5 test errors vs. PSE.\n",
    "        * VGG-11, VGG-13, VGG-16, VGG-19, VGG-11-bn, VGG-13-bn, VGG-16-bn, VGG-19-bn,   \n",
    "          RestNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, ResNeXt-101-32x8d, \n",
    "          resnext50_32x4d\n",
    "        * extract 4d/2d weights.\n",
    "        * reshape 4d to 2d.\n",
    "    * Report ensemble sizes/weight matrix sizes\n",
    "* Answering questions:\n",
    "  * Are there any relationship between network predictive performance and PSE?\n",
    "  * Can we say anyting about PSE causing better architecure?\n",
    "  * How can we use PSE in Neural Architecture Search (NAS)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "('numpy version:',\n",
    " '1.17.2',\n",
    " 'matplotlib :',\n",
    " '3.1.1',\n",
    " 'Python version:',\n",
    " '3.7.3 (default, Mar 27 2019, 16:54:48) \\n[Clang 4.0.1 (tags/RELEASE_401/final)]',\n",
    " 'Bristol version',\n",
    " '0.2.5',\n",
    " 'pytorch:',\n",
    " '1.3.0',\n",
    " 'torchvision:',\n",
    " '0.4.1a0+d94043a')\n",
    " ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext lab_black\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import sys\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import bristol\n",
    "from bristol.spectral import Ergodicity\n",
    "import json\n",
    "\n",
    "ergo = Ergodicity()\n",
    "\n",
    "import torchvision\n",
    "import torchvision.models as models\n",
    "import torch\n",
    "\n",
    "\"numpy version:\", np.__version__, \"matplotlib :\", matplotlib.__version__, \"Python version:\", sys.version, \"Bristol version\", bristol.__version__, \"pytorch:\", torch.__version__, \"torchvision:\", torchvision.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Links\n",
    "\n",
    "* [pytorch models](https://pytorch.org/docs/stable/torchvision/models.html)   \n",
    "  Pytorch model pre-trained with top1 top5 errors.\n",
    "* pedrodiamel [visualise layers and networks](https://github.com/pedrodiamel/nettutorial/blob/master/pytorch/pytorch_visualization.ipynb)\n",
    "* Charles Martin's extraction of weight matrices\n",
    "  | [slice CNN](https://github.com/CalculatedContent/ImplicitSelfRegularization/blob/master/All-pytorch-models-wCNNs-Slices.ipynb) | [pytorch CV calls](https://github.com/CalculatedContent/WeightWatcher/blob/master/WeightWatcher-Full-PyTorchCV.ipynb) | \n",
    "      * `Charles pulls slices from a single layer multiple times on convolutional layers, we will do a reshape and \n",
    "        get a single weight matrix per layer rather than slices.`\n",
    "* Cyclic list for PES computation \n",
    "  [iterools cycle overflow](https://stackoverflow.com/questions/23416381/circular-list-iterator-in-python)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Python Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Layer Matrix Set of pretrained network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_layer_matrix_set(pmodel):\n",
    "    \"\"\"\n",
    "    \n",
    "    Return layer matrix set of a given pre-trained model \n",
    "    \n",
    "    Input\n",
    "    \n",
    "    pmodel : pytorch torchvision pre-trained model\n",
    "    \n",
    "    Returns:\n",
    "    \n",
    "    A tuple (A_set, A_set_N, A_set_types)\n",
    "        A_set       : A list of 2D np-array, weight matrices\n",
    "        A_set_N     : Shape of NxN matrices. \n",
    "        A_set_types : Layer type, pytorch object type that is\n",
    "                      extracted as 2D weight matrix.\n",
    "    \n",
    "    \"\"\"\n",
    "    A_set = []\n",
    "    A_set_N = []\n",
    "    A_set_types = []\n",
    "    for x in pmodel.modules():\n",
    "        type_mod = str(type(x))  # module/method name\n",
    "        if \"torch.nn.modules\" in type_mod:\n",
    "            try:\n",
    "                layer_weights = torch.Tensor(x.weight)\n",
    "                shape_layer = list(layer_weights.shape)\n",
    "                len_shape = len(shape_layer)\n",
    "                if len_shape >= 2:\n",
    "                    N = shape_layer[0]\n",
    "                    M = np.prod(shape_layer[1:])\n",
    "                    if N > 1 and M > 1:\n",
    "                        X = layer_weights\n",
    "                        Ap = np.array(X.reshape(N, M).detach().numpy())\n",
    "                        A = np.matmul(Ap, np.transpose(Ap))\n",
    "                        A_set.append(A)\n",
    "                        A_set_N.append(A.shape)\n",
    "                        A_set_types.append(type_mod)\n",
    "            except:\n",
    "                pass\n",
    "    return (A_set, A_set_N, A_set_types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "### Get Layer Matrix Set of pretrained network, test with resnet18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pmodel = models.resnet18(pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_t = get_layer_matrix_set(pmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Eigenvalues of Layer Matrix set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_eigenvals_layer_matrix_set(A_set):\n",
    "    \"\"\"\n",
    "    \n",
    "    Compute eigenvalues of given set of matrices\n",
    "    \n",
    "    Input: \n",
    "    \n",
    "    A_set : list of 2D ndarrays, square real \n",
    "    \n",
    "    Output\n",
    "    eigenvals_set : List of list of eigenvalues\n",
    "    \n",
    "    \n",
    "    \"\"\"\n",
    "    eigenvals_set = []\n",
    "    for A in A_set:\n",
    "        eigen_values = np.linalg.eigvals(np.matmul(A, np.transpose(A)))\n",
    "        eigenvals_set.append(eigen_values)\n",
    "    return eigenvals_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Eigenvalues of Layer Matrix set test with resnet18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eset = get_eigenvals_layer_matrix_set(A_t[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# len(eset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# [len(ei) for ei in eset]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert layer matrix set eigenvalues to periodic set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import cycle\n",
    "\n",
    "\n",
    "def list2plist(lst, upper_bound):\n",
    "    \"\"\"\n",
    "    \n",
    "    Given list lst ans upper_bound.\n",
    "    Return period_lst, cycle. \n",
    "    \n",
    "    \"\"\"\n",
    "    pool = cycle(lst)\n",
    "    c = 1\n",
    "    lst_period = []\n",
    "    for item in pool:\n",
    "        c = c + 1\n",
    "        if isinstance(item, np.complex64):\n",
    "            item = item.real  # catch for numerical small-unstable numbers\n",
    "        lst_period.append(abs(item))\n",
    "        if c > upper_bound:\n",
    "            break\n",
    "    return lst_period\n",
    "\n",
    "\n",
    "def eigenvals_set_to_periodic(layer_eigens):\n",
    "    \"\"\"\n",
    "    \n",
    "    Layer matrix set eigenvalues to periodic set\n",
    "    \n",
    "    \"\"\"\n",
    "    upper_bound = np.max([len(e) for e in eset])\n",
    "    eset_period = [list2plist(e, upper_bound) for e in eset]\n",
    "    return eset_period"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert layer matrix set eigenvalues to periodic set test with resnet18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eset_per = eigenvals_set_to_periodic(eset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(eset_per)[0][999]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute PSE using periodic set coming from pretrained network\n",
    "\n",
    "These methods actuall do not know if eigenvalues comes from pretrained network.\n",
    "\n",
    "PSE is quantified by symmetric distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def d_layers_pse(eset_per):\n",
    "    \"\"\"\n",
    "    \n",
    "    Progression of D_layers given periodic set\n",
    "    \n",
    "    \"\"\"\n",
    "    nlayers = len(eset_per) - 1  # minus 1 for the last layer\n",
    "    print(nlayers)\n",
    "    N = len(eset_per[0])\n",
    "    D_layer = []\n",
    "    for l in np.arange(1, nlayers):\n",
    "        eigen_l = np.ravel(np.array(eset_per[0:l]))\n",
    "        l1 = l + 1\n",
    "        eigen_l1 = np.ravel(np.array(eset_per[0:l1]))\n",
    "        omega_l = ergo.thirumalai_mountain(eigen_l, l, N)\n",
    "        omega_l1 = ergo.thirumalai_mountain(eigen_l1, l1, N)\n",
    "        dl = ergo.kl_distance_symmetric(omega_l, omega_l1)\n",
    "        D_layer.append(dl)\n",
    "    return D_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute PSE using periodic set coming from pretrained network test with resnet18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Data generate and results\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate data for list of pretrained networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "netnames = [\n",
    "    \"vgg11\",\n",
    "    \"vgg13\",\n",
    "    \"vgg16\",\n",
    "    \"vgg19\",\n",
    "    \"vgg11_bn\",\n",
    "    \"vgg13_bn\",\n",
    "    \"vgg16_bn\",\n",
    "    \"vgg19_bn\",\n",
    "    \"resnet18\",\n",
    "    \"resnet34\",\n",
    "    \"resnet50\",\n",
    "    \"resnet101\",\n",
    "    \"resnet152\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d_layers_dict = {}\n",
    "for netname in netnames:\n",
    "    print(\"d_layer for \", netname)\n",
    "    pmodel = getattr(models, netname)(pretrained=True)\n",
    "    print(type(pmodel))\n",
    "    A_t = get_layer_matrix_set(pmodel)\n",
    "    eset = get_eigenvals_layer_matrix_set(A_t[0])\n",
    "    eset_per = eigenvals_set_to_periodic(eset)\n",
    "    d_layers = d_layers_pse(eset_per)\n",
    "    d_layers_dict[netname] = d_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"/Users/msuzen/data/d_layers_dict.json\", \"w\") as fp:\n",
    "    json.dump(d_layers_dict, fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read\n",
    "with open(\"/Users/msuzen/data/d_layers_dict.json\", \"r\") as fp:\n",
    "    d_layers_dict = json.load(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d_layers_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resnet results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "font = {\"family\": \"normal\", \"weight\": \"bold\", \"size\": 14}\n",
    "\n",
    "plt.rc(\"font\", **font)\n",
    "Dl_18 = d_layers_dict[\"resnet18\"]\n",
    "Dl_18l = np.log10(Dl_18)\n",
    "m = len(Dl_18)\n",
    "plt.plot(np.arange(1, m + 1), Dl_18l, \"-\", label=\"resnet18\")\n",
    "Dl_34 = d_layers_dict[\"resnet34\"]\n",
    "Dl_34l = np.log10(Dl_34)\n",
    "m = len(Dl_34)\n",
    "plt.plot(np.arange(1, m + 1), Dl_34l, \"x-\", label=\"resnet34\")\n",
    "Dl_50 = d_layers_dict[\"resnet50\"]\n",
    "Dl_50l = np.log10(Dl_50)\n",
    "m = len(Dl_50)\n",
    "plt.plot(np.arange(1, m + 1), Dl_50l, \"o-\", label=\"resnet50\")\n",
    "Dl_101 = d_layers_dict[\"resnet101\"]\n",
    "Dl_101l = np.log10(Dl_101)\n",
    "m = len(Dl_101)\n",
    "plt.plot(np.arange(1, m + 1), Dl_101l, \"--\", label=\"resnet101\")\n",
    "Dl_152 = d_layers_dict[\"resnet152\"]\n",
    "Dl_152l = np.log10(Dl_152)\n",
    "m = len(Dl_152)\n",
    "plt.plot(np.arange(1, m + 1), Dl_152l, \".-\", label=\"resnet152\")\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.xlabel(\"Network layer depth\", **font)\n",
    "plt.ylabel(\"Approach to PSE:  $\\log_{10} D_{pse}$\", **font)\n",
    "plt.title(\"Resnet Pre-trained Architectures  \", **font)\n",
    "plt.savefig(\n",
    "    \"plots/resnet_symmetric_resnet.eps\", format=\"eps\", dpi=1000, bbox_inches=\"tight\"\n",
    ")\n",
    "# plt.cla()\n",
    "# plt.clf()\n",
    "# plt.gca()\n",
    "# plt.gcf()\n",
    "# plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse = [\n",
    "    np.mean(Dl_18l),\n",
    "    np.mean(Dl_34l),\n",
    "    np.mean(Dl_50l),\n",
    "    np.mean(Dl_101l),\n",
    "    np.mean(Dl_152l),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top1_err = [30.24, 26.70, 23.85, 22.63, 21.69]\n",
    "# https://pytorch.org/docs/stable/torchvision/models.html\n",
    "top5_err = [10.92, 8.58, 7.13, 6.44, 5.94]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top1_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top1_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(\n",
    "    mean_pse, [1.1, 4.5, 2.3, 2.6, 0.5]\n",
    ")  # some random seq. to see the correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "### VGG results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "font = {\"family\": \"normal\", \"weight\": \"bold\", \"size\": 14}\n",
    "\n",
    "# 'vgg11', 'vgg13', 'vgg16', 'vgg19', 'vgg11_bn', 'vgg13_bn', 'vgg16_bn', 'vgg19_bn'\n",
    "\n",
    "plt.rc(\"font\", **font)\n",
    "Dl = d_layers_dict[\"vgg11\"]\n",
    "Dl_11 = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_11, \"-\", label=\"vgg11\")\n",
    "\n",
    "Dl = d_layers_dict[\"vgg13\"]\n",
    "Dl_13 = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_13, \"x-\", label=\"vgg13\")\n",
    "\n",
    "Dl = d_layers_dict[\"vgg16\"]\n",
    "Dl_16 = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_16, \"o-\", label=\"vgg16\")\n",
    "\n",
    "Dl = d_layers_dict[\"vgg19\"]\n",
    "Dl_19 = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_19, \"--\", label=\"vgg19\")\n",
    "\n",
    "\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.xlabel(\"Network layer depth\", **font)\n",
    "plt.ylabel(\"Approach to PSE: $\\log_{10} D_{pse}$\", **font)\n",
    "plt.title(\"VGG Pre-trained Architectures\", **font)\n",
    "plt.savefig(\"plots/vgg_symmetric_pse.eps\", format=\"eps\", dpi=1000, bbox_inches=\"tight\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "font = {\"family\": \"normal\", \"weight\": \"bold\", \"size\": 14}\n",
    "\n",
    "# 'vgg11', 'vgg13', 'vgg16', 'vgg19', 'vgg11_bn', 'vgg13_bn', 'vgg16_bn', 'vgg19_bn'\n",
    "\n",
    "Dl = d_layers_dict[\"vgg11_bn\"]\n",
    "Dl_11_bn = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_11_bn, \"-\", label=\"vgg11_bn\")\n",
    "\n",
    "plt.rc(\"font\", **font)\n",
    "Dl = d_layers_dict[\"vgg13_bn\"]\n",
    "Dl_13_bn = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_13_bn, \"x-\", label=\"vgg13_bn\")\n",
    "\n",
    "Dl = d_layers_dict[\"vgg16_bn\"]\n",
    "Dl_16_bn = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_16_bn, \"o-\", label=\"vgg16_bn\")\n",
    "\n",
    "Dl = d_layers_dict[\"vgg19_bn\"]\n",
    "Dl_19_bn = np.log10(Dl)\n",
    "m = len(Dl)\n",
    "plt.plot(np.arange(1, m + 1), Dl_19_bn, \"--\", label=\"vgg19_bn\")\n",
    "\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.xlabel(\"Network layer depth\", **font)\n",
    "plt.ylabel(\"Approach to PSE:  $\\log_{10} D_{pse}$\", **font)\n",
    "plt.title(\"VGG Pre-trained Architectures with BN\", **font)\n",
    "plt.savefig(\n",
    "    \"plots/vgg_symmetric_pse_bn.eps\", format=\"eps\", dpi=1000, bbox_inches=\"tight\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse = [np.mean(Dl_11), np.mean(Dl_13), np.mean(Dl_16), np.mean(Dl_19)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top1_err = [30.98, 30.07, 28.41, 27.62]\n",
    "# https://pytorch.org/docs/stable/torchvision/models.html\n",
    "top5_err = [11.37, 10.75, 9.62, 9.12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top1_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse = [np.mean(Dl_11_bn), np.mean(Dl_13_bn), np.mean(Dl_16_bn), np.mean(Dl_19_bn)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top5_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse = [np.mean(Dl_11_bn), np.mean(Dl_13_bn), np.mean(Dl_16_bn), np.mean(Dl_19_bn)]\n",
    "top1_err = [29.62, 28.45, 26.63, 25.76]\n",
    "# https://pytorch.org/docs/stable/torchvision/models.html\n",
    "top5_err = [10.19, 9.63, 8.5, 8.15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_pse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top1_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mean_pse, top5_err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compiled results table\n",
    "\n",
    "```\n",
    "\\begin{table}[]\n",
    "\\centering\n",
    "\\begin{tabular}{|l|l|l|l|}\n",
    "\\hline\n",
    "Architecture  & Top-1 error  &  Top-5 error  & cPSE   \\\\ \\hline\n",
    "vgg11         & 30.98        &   11.37       & 0.04   \\\\ \\hline\n",
    "vgg13         & 30.07        &   10.75       & 0.41   \\\\ \\hline\n",
    "vgg16         & 28.41        &    9.63       & 0.14   \\\\ \\hline\n",
    "vgg19         & 27.62        &    9.12       &-0.10   \\\\ \\hline\n",
    "vgg11bn       & 29.62        &   10.19       & 0.38   \\\\ \\hline\n",
    "vgg13bn       & 28.45        &    9.63       & 0.36   \\\\ \\hline\n",
    "vgg16bn       & 26.63        &    8.50       & 0.18   \\\\ \\hline\n",
    "vgg19bn       & 25.76        &    8.15       &-0.07   \\\\ \\hline\n",
    "resnet18      & 30.24        &   10.92       &-0.19   \\\\ \\hline\n",
    "resnet34      & 26.70        &    8.58       &-0.74   \\\\ \\hline\n",
    "resnet50      & 23.85        &    7.13       &-1.03   \\\\ \\hline\n",
    "resnet101     & 22.63        &    6.44       &-1.77   \\\\ \\hline\n",
    "resnet152     & 21.69        &    5.94       &-2.29   \\\\ \\hline\n",
    "\\end{tabular}\n",
    "\\\n",
    "\\caption{Classification performance and cPSE of investigated architectures. The correlation between\n",
    "both classification performances and cPSE for ResNet ($\\rho=0.94$)  for VGG ($\\rho=0.44$ and $\\rho_{bn}=0.93$\n",
    "with batch normalisation.}\n",
    "\\label{corr}\n",
    "\\end{table}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}